av9ash/GitBugs huggingface dataset:
Bug: ClassCastException when trying to persist the result of a join between two Iceberg tables
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/21 23:19
Affects Version/s: 3.1.2

I am gettin an error when I try to persist the results on a Join operation. Note that both tables to be joined and the output table are Iceberg tables.

SQL code to repro. 
{code}
String sqlJoin = String.format(
        "SELECT * from " +
                "((select %s from %s.%s where %s ) lllll " +
                "join (select %s from %s.%s where %s ) rrrrr " +
                "using (%s))",
        ........);
spark.sql(sqlJoin).writeTo("ciptest.ttt").option("write-format", "parquet").createOrReplace();
{code}

My exception stack is:
{code}
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
	at org.apache.spark.sql.execution.UnsafeRowSerializerInstance$$anon$1.writeValue(UnsafeRowSerializer.scala:64)
	at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:249)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:158)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
	at ….
{code}

Explain on the Sql statement gets the following plan:
{code}
== Physical Plan ==
Project [ ... ]
+- SortMergeJoin […], Inner
  :- Sort […], false, 0
  : +- Exchange hashpartitioning(…, 10), ENSURE_REQUIREMENTS, [id=#38]
  :   +- Filter (…)
  :    +- BatchScan[... ] left [filters=…]
  +- *(2) Sort […], false, 0
   +- Exchange hashpartitioning(…, 10), ENSURE_REQUIREMENTS, [id=#47]
     +- *(1) Filter (…)
      +- BatchScan[…] right [filters=…] 
{code}

Note that several variations of this fail. Besides the repro code listed above I have tried doing CTAS and trying to write the result into parquet files without making a table out of it.
Bug: Anonymized/obfuscated query plan
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/21 00:07
Affects Version/s: 3.1.2

I would like to share a query plan for a specific issue(https://issues.apache.org/jira/browse/SPARK-37609), but can't without at least anonymising the column names. If I could call {{explain}} that would anonymise/obfuscate the column names that would be useful.
Bug: Refactor antlr4 syntax file
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/21 07:39
Affects Version/s: 3.3.0

Refer to the ShardingSphere project, split into multiple files by type, very clear



https://github.com/apache/shardingsphere/tree/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-mysql/src/main/antlr4/imports/mysql
Bug: Can anyone help me to compile a scala file
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Dec/21 18:37
Affects Version/s: 2.2.3

I am working with spark SQL, trying to read data from SAS using JDBC. I have to use customized JdbcDialect to handle special format of the SAS data. Here is the code:
{code:java}
import org.apache.spark.sql.jdbc.JdbcDialectobject 

object SasDialect extends JdbcDialect {
   override def canHandle(url: String): Boolean = url.startsWith("jdbc:sharenet")
   override def quoteIdentifier(colName: String): String = "\"" + colName + "\"n" 
} 
{code}
It worked well in Scala, but my major language is Python, so I need to compile this into jar file, so I can call it from python, like
{code:python}
from py4j.java_gateway import java_import

gw = spark.sparkContext._gateway
java_import(gw.jvm, "com.me.SasJDBCDialect")
gw.jvm.org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(
  gw.jvm.com.me.SasJDBCDialect()){code}
But I am a newbie of Scala, and don't know how to make it a jar file so it can be call as "com.me.MyJDBCDialect". Can anyone here do me a favour to generate a jar file for me?

 

Thanks!
Bug: null values should be saved as nothing rather than quoted empty Strings "" with default settings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/21 05:20
Affects Version/s: 2.4.0

As mentioned in sql migration guide([https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-23-to-24]),
{noformat}
Since Spark 2.4, empty strings are saved as quoted empty strings "". In version 2.3 and earlier, empty strings are equal to null values and do not reflect to any characters in saved CSV files. For example, the row of "a", null, "", 1 was written as a,,,1. Since Spark 2.4, the same row is saved as a,,"",1. To restore the previous behavior, set the CSV option emptyValue to empty (not quoted) string.{noformat}
But actually, both empty strings and null values are saved as quoted empty Strings "" rather than "" (for empty strings) and nothing(for null values)。

code:
{code:java}
val data = List("spark", null, "").toDF("name")
data.coalesce(1).write.csv("spark_csv_test")
{code}
 actual result:
{noformat}
line1: spark
line2: ""
line3: ""{noformat}
expected result:
{noformat}
line1: spark
line2: 
line3: ""
{noformat}
Bug: sequence over dates with month interval is producing incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/21 06:09
Affects Version/s: 3.1.1

Sequence function with dates and step interval in months producing unexpected results.

Here is a sample using Spark 3.2 (though the behavior is the same in 3.1.1 and presumably earlier):

{{scala> spark.sql("select sequence(date '2021-01-01', date '2022-01-01', interval '3' month) x, date '2021-01-01' + interval '3' month y").collect()}}
{{res1: Array[org.apache.spark.sql.Row] = Array([WrappedArray(2021-01-01, {color:#FF0000}*2021-03-31, 2021-06-30, 2021-09-30,* {color}{color:#172b4d}2022-01-01{color}),2021-04-01])}}

Expected result of adding 3 months to the 2021-01-01 is 2021-04-01, while sequence returns 2021-03-31.

At the same time sequence over timestamps works as expected:

{{scala> spark.sql("select sequence(timestamp '2021-01-01 00:00', timestamp '2022-01-01 00:00', interval '3' month) x").collect()}}
{{res2: Array[org.apache.spark.sql.Row] = Array([WrappedArray(2021-01-01 00:00:00.0, *2021-04-01* 00:00:00.0, *2021-07-01* 00:00:00.0, *2021-10-01* 00:00:00.0, 2022-01-01 00:00:00.0)])}}

 

A similar issue was reported in the past - [SPARK-31654] sequence producing inconsistent intervals for month step - ASF JIRA (apache.org)
It's marked resolved, but the problem is either resurfaced or was never actually fixed.
Bug: mypy breaks on pyspark.pandas.plot.core.Bucketizer
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/21 01:13
Affects Version/s: 3.2.0

Mypy breaks on a project with pyspark 3.2.0 dependency (worked fine for 3.1.2), see stacktrace below:

{noformat}
Traceback (most recent call last):
  File "/Users/rav/.cache/pre-commit/repoytd0jvc_/py_env-python3.9/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/Users/rav/.cache/pre-commit/repoytd0jvc_/py_env-python3.9/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "mypy/main.py", line 87, in main
  File "mypy/main.py", line 165, in run_build
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 254, in _build
  File "mypy/build.py", line 2697, in dispatch
  File "mypy/build.py", line 3021, in process_graph
  File "mypy/build.py", line 3138, in process_stale_scc
  File "mypy/build.py", line 2288, in write_cache
  File "mypy/build.py", line 1475, in write_cache
  File "mypy/nodes.py", line 313, in serialize
  File "mypy/nodes.py", line 3149, in serialize
  File "mypy/nodes.py", line 3083, in serialize
AssertionError: Definition of pyspark.pandas.plot.core.Bucketizer is unexpectedly incomplete
{noformat}
Bug: Create Spark interface for creating UDFs
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Dec/21 15:07
Affects Version/s: 3.2.0

Currently, based on [spark documentation|https://spark.apache.org/docs/3.2.0/sql-ref-syntax-ddl-create-function.html], if you need to create a custom UDF with specific logic implemented in (let's say) Scala, you have to extend your class from:
 # UDF or UDAF from org.apache.hadoop.hive.ql.exec package
 # AbstractGenericUDAFResolver, GenericUDF, or GenericUDTF from org.apache.hadoop.hive.ql.exec package
 # UserDefinedAggregateFunction in org.apache.spark.sql.expressions (which is deprecated)

The third option is deprecated, and (I'm not sure) this option is used only for agg user functions. So, for plain UDF we have to follow these first two options. That means we MUST add a dependency for the hive-exec. Maybe, we can bring spark native interface for creating UDFs.

WDYT guys? 
Thanks in advance
Bug: Support TimedeltaIndex creation (from Series/Index) and TimedeltaIndex.astype
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/21 05:50
Affects Version/s: 3.3.0

 

To solve the issues below:


{code:java}
>>> idx = ps.TimedeltaIndex([timedelta(1), timedelta(microseconds=2)])
>>> idx
TimedeltaIndex(['1 days 00:00:00', '0 days 00:00:00.000002'], dtype='timedelta64[ns]', freq=None)
>>> ps.TimedeltaIndex(idx)
Traceback (most recent call last):
...
    raise TypeError("astype can not be applied to %s." % self.pretty_name)
TypeError: astype can not be applied to timedelta.
 {code}
 

 

 
{code:java}
>>> s = ps.Series([timedelta(1), timedelta(microseconds=2)], index=[10, 20])
>>> s
10          1 days 00:00:00
20   0 days 00:00:00.000002
dtype: timedelta64[ns]
>>> ps.TimedeltaIndex(s)
Traceback (most recent call last):
...
    raise TypeError("astype can not be applied to %s." % self.pretty_name)
TypeError: astype can not be applied to timedelta.
 {code}
Bug: Add the startswith() and endswith() string functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/21 12:20
Affects Version/s: 3.3.0

startswith() and endswith() are popular in other systems:

Snowflake:

[https://docs.snowflake.com/en/sql-reference/functions/startswith.html]

[https://docs.snowflake.com/en/sql-reference/functions/endswith.html]

Cosmos DB:

[https://docs.microsoft.com/en-us/azure/cosmos-db/sql/sql-query-startswith]

[https://docs.microsoft.com/en-us/azure/cosmos-db/sql/sql-query-endswith]

I would like to propose such functions to Spark SQL to make migration easier.
Bug: Support tuple for non-MultiIndex column name.
Status: Open | Priority: Major | Resolution: 
Created: 23/Dec/21 07:03
Affects Version/s: 3.3.0

pandas API on Spark doesn't support tuple as column name for non-MultiIndex column.
{code:java}
>>> psdf = ps.DataFrame({"A": [1, 2, 3]})
>>> psdf
A
0 1
1 2
2 3
>>> psdf[('a', 'b')] = [4, 5, 6]
Traceback (most recent call last):
...
KeyError: 'Key length (2) exceeds index depth (1)'
{code}
As pandas support this, we should follow the behavior.
{code:java}
>>> pdf = pd.DataFrame({"A": [1, 2, 3]})
>>> pdf[('a', 'b')] = [4, 5, 6]
>>> pdf
   A  (a, b)
0  1       4
1  2       5
2  3       6{code}
Bug: Spark shell sets log level to INFO by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/21 01:56
Affects Version/s: 3.3.0

{code}
./bin/spark-shell
{code}

{code}
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/12/31 10:55:04 INFO SignalUtils: Registering signal handler for INT
21/12/31 10:55:08 INFO HiveConf: Found configuration file null
21/12/31 10:55:08 INFO SparkContext: Running Spark version 3.3.0-SNAPSHOT
...

21/12/31 10:55:09 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, ..., None)
...
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0-SNAPSHOT
      /_/

Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_291)
Type in expressions to have them evaluated.
Type :help for more information.
{code}
Bug: [PYSPARK] Fix mlflow doctest
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/21 07:18
Affects Version/s: 3.3.0

{code:java}
**********************************************************************
1324File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 149, in pyspark.pandas.mlflow.load_model
1325Failed example:
1326    mlflow.set_experiment("my_experiment")
1327Expected nothing
1328Got:
1329    <Experiment: artifact_location='file:///__w/spark/spark/python/target/5d5d5841-43b7-41c4-bc0a-1c5763b355ca/tmpfhad7kycpandas_on_spark_mlflow/0', experiment_id='0', lifecycle_stage='active', name='my_experiment', tags={}>
1330**********************************************************************
1331   1 of  26 in pyspark.pandas.mlflow.load_model
1332***Test Failed*** 1 failures. {code}
Bug: Enable matplotlib test for pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/21 02:15
Affects Version/s: 3.3.0

Since the matplotlib is not installed in CI, so the related tests are skipped.

([https://app.codecov.io/gh/apache/spark/blob/master/python/pyspark/pandas/tests/plot/test_series_plot_matplotlib.py)]

We should enable for better test coverage.
Bug: plot.hist throws AttributeError on pandas=1.3.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/21 12:13
Affects Version/s: 3.2.0

plot.hist from PySpark throws AttributeError exception when pyspark.pandas is used with pandas=1.3.5.

Pandas in commit [https://github.com/pandas-dev/pandas/commit/029907c9d69a0260401b78a016a6c4515d8f1c40]
replaced MPLPlot._add_legend_handle with MPLPlot._append_legend_handles_labels.

I've attached PR on github which replaces use of MPLPlot._add_legend_handle in PySpark with MPLPlot._append_legend_handles_labels.

 

Code:

 
{code:java}
import pyspark.pandas as ps
from matplotlib import pyplot as plt

ps.set_option("plotting.backend", "matplotlib")

df = ps.DataFrame({'data': [4, 5, 5, 6, 8, 9]})
df['data'].plot.hist()

plt.show()
 {code}
 

 

Truncated traceback:
{code:java}
Traceback (most recent call last):                                              
  File "/home/develop/Documents/sparkbug/code.py", line 6, in <module>
    df['data'].plot.hist()
  ...
  File "/mnt/transient/develop/miniconda3/envs/testenv/lib/python3.9/site-packages/pyspark/pandas/plot/matplotlib.py", line 403, in _make_plot
    self._add_legend_handle(artists[0], label, index=i)
AttributeError: 'PandasOnSparkHistPlot' object has no attribute '_add_legend_handle' {code}
Bug: Write session time zone in the Parquet file metadata so that rebase can use it instead of JVM timezone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/21 12:47
Affects Version/s: 3.3.0

We could write session time zone in the Parquet file that was used to write timestamps so we can use the same time zone to reconstruct the values instead of using JVM time zone (which could be different).
Bug: Migrate to M1 machines in Jenkins
Status: Open | Priority: Major | Resolution: 
Created: 26/Dec/21 07:09
Affects Version/s: 3.3.0

See https://mail-archives.apache.org/mod_mbox/spark-dev/202112.mbox/%3CCACdU-dTLuB--1GzAv6XfS-pCrcihhvDpUMrGe%3DfJXUYJpqiX9Q%40mail.gmail.com%3E.

We should revisit all related Jenkins specific codes when M1 machines are ready.
Bug: Skip PIP packaging test if Python version is lower than 3.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/21 03:57
Affects Version/s: 3.3.0

{code}
Writing pyspark-3.3.0.dev0/setup.cfg
Creating tar archive
removing 'pyspark-3.3.0.dev0' (and everything under it)
Installing dist into virtual env
Obtaining file:///home/jenkins/workspace/SparkPullRequestBuilder%402/python
pyspark requires Python '>=3.7' but the running Python is 3.6.8
Cleaning up temporary directory - /tmp/tmp.CCragmNU1X
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/dev/run-pip-tests ; received return code 1
Attempting to post to GitHub...
{code}

After we drop Python 3.6 at SPARK-37632, PIP packaging test fails intermittently https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/146255/console. Apparently, different Python versions are installed. We should skip the test in these cases.
Bug: Revisit codes in PySpark, SparkR and other docs
Status: Open | Priority: Major | Resolution: 
Created: 26/Dec/21 07:17
Affects Version/s: 3.3.0

{code}
R/check-cran.sh:# Jenkins installs arrow. See SPARK-29339.
R/pkg/tests/run-all.R:  # CRAN machines. For Jenkins we should already have SPARK_HOME set.
R/run-tests.sh:    # We have 2 NOTEs: for RoxygenNote and one in Jenkins only "No repository set"
dev/run-pip-tests:# Jenkins has PySpark installed under user sitepackages shared for some reasons.
python/pyspark/sql/tests/test_streaming.py:            # Jenkins is very slow, we don't assert it. If there is something wrong, "lastProgress"
python/pyspark/streaming/tests/test_kinesis.py:        # Don't start the StreamingContext because we cannot test it in Jenkins
dev/create-release/release-build.sh:# This is a band-aid fix to avoid the failure of Maven nightly snapshot in some Jenkins
docs/building-spark.md:## Running Jenkins tests with GitHub Enterprise
docs/building-spark.md:To run tests with Jenkins:
dev/tests/pr_merge_ability.sh:# found at dev/run-tests-jenkins.
dev/tests/pr_merge_ability.sh:# known as `ghprbActualCommit` in `run-tests-jenkins`
dev/tests/pr_merge_ability.sh:# known as `sha1` in `run-tests-jenkins`
dev/tests/pr_public_classes.sh:# found at dev/run-tests-jenkins.
dev/tests/pr_public_classes.sh:# known as `ghprbActualCommit` in `run-tests-jenkins`
docs/building-spark.md:    ./dev/run-tests-jenkins
{code}
Bug: Revisit codes in testing scripts for Jenkins
Status: Open | Priority: Major | Resolution: 
Created: 26/Dec/21 07:11
Affects Version/s: 3.3.0

e.g.)

https://github.com/apache/spark/blob/master/dev/run-tests-jenkins
https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py
Jenkins specific logics at https://github.com/apache/spark/blob/master/dev/run-tests.py
Bug: Spark catalog and Delta tables
Status: Open | Priority: Major | Resolution: 
Created: 14/Dec/21 21:38
Affects Version/s: 3.1.2

I'm using Spark with Delta tables, while tables are created, there are no columns in the table.

Steps to reproduce:
1. Start spark-shell 
{code:java}
spark-shell --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" --conf "spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY"{code}
2. Create delta table
{code:java}
spark.range(10).write.format("delta").option("path", "tmp/delta").saveAsTable("delta"){code}
3. Make sure table exists 
{code:java}
spark.catalog.listTables.show{code}
4. Find out that columns are not
{code:java}
spark.catalog.listColumns("delta").show{code}

This is critical for Delta integration with different BI tools such as Power BI or Tableau, as they are querying spark catalog for the metadata and we are getting errors that no columns are found. 
Discussion can be found in Delta repository - https://github.com/delta-io/delta/issues/695
Bug: Make it easier to convert numpy arrays to Spark Dataframes
Status: Open | Priority: Major | Resolution: 
Created: 20/Dec/21 16:50
Affects Version/s: 3.1.2

Make it easier to convert numpy arrays to dataframes.


Often we receive errors:

 
{code:java}
df = spark.createDataFrame(numpy.arange(10))
Can not infer schema for type: <class 'numpy.int64'>
{code}
 

OR
{code:java}
df = spark.createDataFrame(numpy.arange(10.))
Can not infer schema for type: <class 'numpy.float64'>
{code}
 

Today (Spark 3.x) we have to:
{code:java}
spark.createDataFrame(pd.DataFrame(numpy.arange(10.))) {code}
Make this easier with a direct conversion from Numpy arrays to Spark Dataframes.
Bug: Unify v1 and v2 ALTER TABLE .. SET LOCATION tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/21 18:40
Affects Version/s: 3.2.0

Extract ALTER TABLE .. SET LOCATION tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Reset numFailures when one of task attempts succeeds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/21 09:52
Affects Version/s: 3.3.0

In production environment, we found some logic leak about TaskSetManager abort. For example:

If one task has failed 3 times(max failed threshold is 4 in default), and there is a retry task and speculative task both in running state, then one of these 2 task attempts succeed and to cancel another. But executor which task need to be cancelled lost(oom in our situcation), this task marked as failed, and TaskSetManager handle this failed task attempt, it has failed 4 times so abort this stage and cause job failed.

I created the patch for this bug and will soon be sent the pull request.
Bug: when charVarcharAsString is true, char datatype partition table query incorrect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/21 09:30
Affects Version/s: 3.1.2

This ticket aim at fixing the bug that does not apply right-padding for char types column when charVarcharAsString is true and partition data length is less than defined length.
For example, a query below returns nothing in master, but a correct result is `abc`.
{code:java}
scala> sql("set spark.sql.legacy.charVarcharAsString=true")
scala> sql("CREATE TABLE tb01(i string, c char(5)) USING parquet partitioned by (c)")
scala> sql("INSERT INTO tb01 values(1, 'abc')")
scala> sql("select c from tb01 where c = 'abc'").show
+---+
|  c|
+---+
+---+{code}
This is because `ApplyCharTypePadding` rpad the expr to charLength. We should handle this consider conf spark.sql.legacy.charVarcharAsString value.
Bug: Translate more standard aggregate functions for pushdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/21 03:27
Affects Version/s: 3.3.0

Currently, Spark aggregate pushdown will translate some standard aggregate functions, so that compile these functions suitable specify database.

After this job, users could override JdbcDialect.compileAggregate to implement some aggregate functions supported by some database.
Bug: Fine tune logic to demote Broadcast hash join in DynamicJoinSelection 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Dec/21 17:25
Affects Version/s: 2.4.8

In the current implementation of {{DynamicJoinSelection}} the logic checks if one side of the join has high ratio of empty partitions and adds a NO_BROADCAST hint on that side since  a shuffle join can short-circuit the local joins where one side is empty.

This logic is doesn't make sense for all join type.  For example, a Left Outer Join cannot short circuit if RHS is empty so we should not inhibit BHJ.  On the other hand a LOJ executed as a shuffle join where the LHS has many empty can short circuit the local join so we should inhibit the BHJ because BHJ will use {{OptimizeShuffleWithLocalRead}} which will re-assemble LHS partitions as the were before the shuffle and thus may not have many empty ones any more.

This supersedes SPARK-37193
Bug: Timedelta support in pandas API on Spark
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 03/Dec/21 03:01
Affects Version/s: 3.3.0

Since DayTimeIntervalType is supported in PySpark, we may add TimedeltaIndex support in pandas API on Spark accordingly.

We shall approach it in steps below:
- introduce
- properties
- functions and basic operations
- creation (from Series/Index, generic methods)
- type conversion (astype)