av9ash/GitBugs huggingface dataset:
Bug: Change description-file to description_file
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/23 21:29
Affects Version/s: 4.0.0

+ cp -r /home/bjorn/spark/data /home/bjorn/spark/dist
+ '[' true == true ']'
+ echo 'Building python distribution package'
Building python distribution package
+ pushd /home/bjorn/spark/python
+ rm -rf pyspark.egg-info
+ python3 setup.py sdist
/usr/lib/python3.11/site-packages/setuptools/dist.py:745: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!

        ********************************************************************************
        Usage of dash-separated 'description-file' will not be supported in future
        versions. Please use the underscore name 'description_file' instead.

        This deprecation is overdue, please update your project and remove deprecated
        calls to avoid build errors in the future.

        See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
        ********************************************************************************

!!
Bug: Fix `pyspark.sql.tests.connect.test_connect_basic` in Python 3.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/23 23:06
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/6914662405/job/18812759788

{code}
======================================================================
ERROR [3.529s]: test_recursion_handling_for_plan_logging (pyspark.sql.tests.connect.test_connect_basic.SparkConnectBasicTests.test_recursion_handling_for_plan_logging)
SPARK-45852 - Test that we can handle recursion in plan logging.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/sql/tests/connect/test_connect_basic.py", line 171, in test_recursion_handling_for_plan_logging
    self.assertIsNotNone(cdf.schema)
                         ^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/sql/connect/dataframe.py", line 1735, in schema
    return self._session.client.schema(query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 924, in schema
    schema = self._analyze(method="schema", plan=plan).schema
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1110, in _analyze
    self._handle_error(error)
  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1499, in _handle_error
    self._handle_rpc_error(error)
  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1570, in _handle_rpc_error
    raise SparkConnectGrpcException(str(rpc_error)) from None
pyspark.errors.exceptions.connect.SparkConnectGrpcException: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INTERNAL
	details = "Exception serializing request!"
	debug_error_string = "None"
>

----------------------------------------------------------------------
Ran 141 tests in 86.259s

FAILED (errors=1)
{code}
Bug: spark-build-info.ps1 missing the docroot property
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/23 05:34
Affects Version/s: 4.0.0

https://github.com/apache/spark/blob/master/build/spark-build-info.ps1#L38-L44
https://github.com/apache/spark/blob/master/build/spark-build-info#L30-L36
Bug: Filters are not pushed down across lateral view
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/23 07:24
Affects Version/s: 3.5.0

{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.config("spark.sql.catalogImplementation", "hive").appName("Write ORC File").getOrCreate()
spark.sql('drop TABLE if exists test').show()
spark.sql('CREATE EXTERNAL TABLE test (request struct<characteristic:array<struct<id:string,value:string>>>)'
'ROW FORMAT SERDE "org.apache.hadoop.hive.ql.io.orc.OrcSerde" '
'STORED AS INPUTFORMAT "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat" '
'OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat" '
'LOCATION "testfolder"').show()
spark.sql("select request from test lateral view explode(request.characteristic) cTable as c where c.value='79640000000'").explain()
{code}

shows
{code}
== Physical Plan ==
*(1) Project [request#2]
+- *(1) Filter (isnotnull(c#4.value) AND (c#4.value = 79640000000))
   +- *(1) Generate explode(request#2.characteristic), [request#2], false, [c#4]
      +- *(1) ColumnarToRow
         +- FileScan orc spark_catalog.default.test[request#2] Batched: true, DataFilters: [], Format: ORC, Location: InMemoryFileIndex(1 paths)[file:/Users/paf/Downloads/spark-warehouse/testfolder], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<request:struct<characteristic:array<struct<id:string,value:string>>>>
{code}

Which is extremely slow.

Suppose I search for a column value, which is totally out of min/max statistics range.

Search could have been much faster, but no.
Bug: Provide partitioning expressions in Java as same as Scala
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/23 06:42
Affects Version/s: 4.0.0

See https://github.com/apache/spark/pull/43858.

Once Scala 3 is out, we can support the same way of partitioning expressions such as:

{code}
import static org.apache.spark.sql.functions.partitioning.*;
{code}
Bug: Enhancement  of XSDToSchema API: Change to HDFS API for cloud storage accessibility
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/23 23:26
Affects Version/s: 4.0.0

Previously, it utilized `java.nio.path`, which limited file reading to local file systems only. By changing this to an HDFS-compatible API, we now enable the XSDToSchema function to access files in cloud storage.
Bug: Document `spark.master.*` configurations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/23 22:45
Affects Version/s: 3.4.2

Currently, `spark.master.*` configurations are undocumented.
{code:java}
$ git grep 'ConfigBuilder("spark.master'
core/src/main/scala/org/apache/spark/internal/config/UI.scala:  val MASTER_UI_DECOMMISSION_ALLOW_MODE = ConfigBuilder("spark.master.ui.decommission.allow.mode")
core/src/main/scala/org/apache/spark/internal/config/package.scala:  private[spark] val MASTER_REST_SERVER_ENABLED = ConfigBuilder("spark.master.rest.enabled")
core/src/main/scala/org/apache/spark/internal/config/package.scala:  private[spark] val MASTER_REST_SERVER_PORT = ConfigBuilder("spark.master.rest.port")
core/src/main/scala/org/apache/spark/internal/config/package.scala:  private[spark] val MASTER_UI_PORT = ConfigBuilder("spark.master.ui.port")
core/src/main/scala/org/apache/spark/internal/config/package.scala:    ConfigBuilder("spark.master.ui.historyServerUrl")
core/src/main/scala/org/apache/spark/internal/config/package.scala:    ConfigBuilder("spark.master.useAppNameAsAppId.enabled") {code}
Bug: (Scala) Support different retry policies for connect client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/23 10:50
Affects Version/s: 4.0.0

Support multiple retry policies defined at the same time. Each policy determines which error types it can retry and how exactly.

For instance, networking errors should generally be retried differently that
remote resource being available.

Relevant python ticket: SPARK-45733
Bug: Allow non-deterministic Python UDFs in MapInPandas/MapInArrow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/23 04:04
Affects Version/s: 4.0.0

Currently if a Python udf is non-deterministic, the analyzer will fail with this error:[INVALID_NON_DETERMINISTIC_EXPRESSIONS] The operator expects a deterministic expression, but the actual expression is "pyUDF()", "a". SQLSTATE: 42K0E;
Bug: Leaked file streams in ParquetFileFormat
Status: Open | Priority: Major | Resolution: 
Created: 15/Nov/23 23:25
Affects Version/s: 4.0.0

- [https://github.com/apache/spark/actions/runs/6859020738/job/18650698085]
 - [https://github.com/apache/spark/actions/runs/6872747886/job/18691717269]
{code:java}
Cause: java.lang.IllegalStateException: There are 1 possibly leaked file streams.
29975[info]   at org.apache.spark.DebugFilesystem$.assertNoOpenStreams(DebugFilesystem.scala:54)
29976[info]   at org.apache.spark.sql.test.SharedSparkSessionBase.$anonfun$afterEach$1(SharedSparkSession.scala:165)
...
29977[info]   at 
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatSuite.eventually(ParquetFileFormatSuite.scala:31)
29984[info]   at org.apache.spark.sql.test.SharedSparkSessionBase.afterEach(SharedSparkSession.scala:164)
29985[info]   at org.apache.spark.sql.test.SharedSparkSessionBase.afterEach$(SharedSparkSession.scala:158)
29986[info]   at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormatSuite.afterEach(ParquetFileFormatSuite.scala:31)
29987[info]   at org.scalatest.BeforeAndAfterEach.$anonfun$runTest$1(BeforeAndAfterEach.scala:247)
29988[info]   at 
...
30025[info]   Cause: java.lang.Throwable:
30026[info]   at org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:35)
30027[info]   at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:75)
30028[info]   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:997)
30029[info]   at org.apache.parquet.hadoop.util.HadoopInputFile.newStream(HadoopInputFile.java:69)
30030[info]   at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:796)
30031[info]   at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:666)
30032[info]   at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:85)
30033[info]   at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:76)
30034[info]   at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:450) {code}
Bug: Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec
Status: Closed | Priority: Major | Resolution: Not A Bug
Created: 14/Nov/23 18:42
Affects Version/s: 3.5.1

while writing bug test for  [SPARK-45866|https://issues.apache.org/jira/projects/SPARK/issues/SPARK-45866], found that canonicalization of SubqueryAdaptiveBroadcastExec is broken in the sense that buildPlan : LogicalPlan is not canonicalized which causes batchscans to differ when reuse of exchange check happens in AQE.
Moreover the equivalence of SubqueryAdaptiveBroadcastExec and SubqueryBroadcastExec  is not there which also aggravates the re-use of exchange in aqe broken.
Bug: SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 14/Nov/23 18:49
Affects Version/s: 3.5.1

A created stage may contain SubqueryAdaptiveBroadcastExec while incominng exchange may contain SubqueryBroadcastExec and though they are equivalent , the match does not happen because equals/hashCode do not match , resulting in non re-use of exchange.
Bug: Fix 'Super method + is deprecated.'
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Nov/23 03:45
Affects Version/s: 4.0.0

  @deprecated("Consider requiring an immutable Map or fall back to Map.concat.", "2.13.0")
  def + [V1 >: V](kv: (K, V1)): CC[K, V1] =
    mapFactory.from(new View.Appended(this, kv))
Bug: Refactor the optimizer plan validation to decouple validateSchemaOutput and validateExprIdUniqueness
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/23 22:26
Affects Version/s: 3.4.0

Currently, the expressionIDUniqueness validation is closely coupled with output schema validation. 

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala#L403C7-L411C8

Some refactoring can improve readability and reuse.
Bug: Improve exception message when no remote url is set 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Nov/23 22:33
Affects Version/s: 3.5.0

When "SPARK_CONNECT_MODE_ENABLED" but no spark remote url is set, PySpark currently throws this exception:

AttributeError: 'NoneType' object has no attribute 'startswith'

We should improve this.
Bug: Different column orders lead to OOM
Status: Open | Priority: Major | Resolution: 
Created: 13/Nov/23 07:39
Affects Version/s: 2.3.2

 
Problem description:
 I am using spark to generate HFile, OOM occurs in the repartitionAndSortWithinPartitions stage, and the input data is the result of sql . When I swap column b and column c , It will not occurs OOM. 

 

The sql will occurs OOM
{code:java}
-- colum "a" and "b" are longType，column "c" is a json string   
select a, b , c from table{code}
The sql will not occurs OOM
{code:java}
-- colum "a" and "b" are longType，column "c" is a json string   
select a, c , b from table{code}
 

The code logic is as follows
{code:java}
// code placeholder

  val family = Bytes.toBytes("cf")  
  // colum "a" and "b" are longType，column "c" is a json string
  val sql = "select a, b , c from table "
  spark.sql(sql).flatMap(row => {
    val buffer = new ArrayBuffer[Tuple2[KeyFamilyQualifier, ImmutableBytesWritable]]
    val v1 = row.getLong(0)
    val v2 = row.getLong(1)
    val v3 = row.getString(2)    

    val rowkey = Bytes.toBytes(v1)
    
    val kfq1 = new KeyFamilyQualifier(rowkey, family, Bytes.toBytes("v1"))
    buffer += Tuple2(kfq1, new ImmutableBytesWritable(Bytes.toBytes(v1)))  
   
    val kfq2 = new KeyFamilyQualifier(rowkey, family, Bytes.toBytes("v2"))
    buffer += Tuple2(kfq2, new ImmutableBytesWritable(Bytes.toBytes(v2)))    

    val kfq3 = new KeyFamilyQualifier(rowkey, family, Bytes.toBytes("v3"))
    buffer += Tuple2(kfq3, new ImmutableBytesWritable(Bytes.toBytes(v3)))    

    buffer.toArray
  })
  .repartitionAndSortWithinPartitions(new BulkLoadPartitioner(startKeys))
  .map(tuple => {
    val kfq = tuple._1
    val value = tuple._2
    
    val rowkey = new ImmutableBytesWritable(kfq.getRowKey)
    val keyValue = new KeyValue(kfq.getRowKey(), kfq.getFamily(), kfq.getQualifier(), value.get())
    (rowkey, keyValue)
  })
  .saveAsNewAPIHadoopFile(
    tmpPath, 
    classOf[ImmutableBytesWritable],
    classOf[KeyValue],
    classOf[HFileOutputFormat2],
    job.getConfiguration()
  ) 


{code}
 

Now, even with a small amount of data, this program will still cause OOM during the shuffle stage. If  I swap the positions of fields b and c in the SQL statement, it will not cause OOM. the new sql is "select a, c, b from table".

 

I used mat to analyse the jvm dump file of executor, the result is 

!image-2023-11-13-15-46-16-868.png!

The JVM is filled with PartitionedPairBuffer$$data 

!image-2023-11-13-15-47-01-764.png!
Bug: Expand hash functionalities from to include XXH3
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/23 23:28
Affects Version/s: 3.5.0

I often work in projects that require deterministic randomness, especially when creating surrogate keys. For small volumes of data xxhash64 works well however this functionality doesn't scale well - with a 64-bit hash code, the chance of collision is one in a million when you hash just six million items increasing sharply due to the birthday paradox.

Currently there are a few ways to handle this
 - hash: 32-bit output (>50% chance of collision at least one for tables larger than 77000 rows)
 - xxhash64: 64-bit output (>50% chance of collision at least one for tables larger than 5 billion rows)
 - shaXXX/md5: single binary column input, string output, quite computationally expensive.

I'd suggest adding the newest algorithm in the xxhash64 family, XXH3. The XXH3 family is a modern 64-bit and 128-bit hash function family that provides improved strength and performance across the board.

I'd imagine this would be a new function named xxhash3 and take 64 bit, and 128bit bit lengths. For usability I believe the bit length should default to 128bits to provide the best experience to reduce accidental collisions and leave users to set the bit length to 64 as an override if they need to for additional performance or interop reasons. (given the benchmarks, this would likely be quite rare)

References:
 * [Documentation|https://xxhash.com/]
 * xxHash64 Ticket (https://issues.apache.org/jira/browse/SPARK-27099)
 * [Existing xxHash64 logic|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/XXH64.java]
Bug: Combine multiple like to like all
Status: Open | Priority: Major | Resolution: 
Created: 11/Nov/23 12:42
Affects Version/s: 4.0.0


{code:scala}
   spark.sql("create table t(a string, b string, c string) using parquet")
    spark.sql(
      """
        |select * from t where
        |substr(a, 1, 5) like '%a%' and
        |substr(a, 1, 5) like '%b%'
        |""".stripMargin).explain(true)
{code}

We can optimize the query to:
{code:scala}
    spark.sql(
      """
        |select * from t where
        |substr(a, 1, 5) like all('%a%', '%b%')
        |""".stripMargin).explain(true)
{code}
Bug: Implement push-down filter with partition ID and grouping key (if possible) for state data source reader
Status: Open | Priority: Major | Resolution: 
Created: 10/Nov/23 14:47
Affects Version/s: 4.0.0

If the query filters the state data via partition ID, it is a good chance for state data source to avoid spinning all state store instances and wasting resource. We can spin state store instances for only necessary partitions.

Same thing applies to grouping keys, although the criteria on distribution is bound to the operator rather than the key in state store, hence it could be very tricky unless we can follow the same criteria on distribution for the operator.
Bug: ConcurrentModificationException in CliSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/23 07:47
Affects Version/s: 4.0.0

{code:java}
// code placeholder

java.util.ConcurrentModificationException: mutation occurred during iteration
[info]   at scala.collection.mutable.MutationTracker$.checkMutations(MutationTracker.scala:43)
[info]   at scala.collection.mutable.CheckedIndexedSeqView$CheckedIterator.hasNext(CheckedIndexedSeqView.scala:47)
[info]   at scala.collection.IterableOnceOps.addString(IterableOnce.scala:1247)
[info]   at scala.collection.IterableOnceOps.addString$(IterableOnce.scala:1241)
[info]   at scala.collection.AbstractIterable.addString(Iterable.scala:933)
[info]   at scala.collection.IterableOnceOps.mkString(IterableOnce.scala:1191)
[info]   at scala.collection.IterableOnceOps.mkString$(IterableOnce.scala:1189)
[info]   at scala.collection.AbstractIterable.mkString(Iterable.scala:933)
[info]   at scala.collection.IterableOnceOps.mkString(IterableOnce.scala:1204)
[info]   at scala.collection.IterableOnceOps.mkString$(IterableOnce.scala:1204)
[info]   at scala.collection.AbstractIterable.mkString(Iterable.scala:933)
[info]   at org.apache.spark.sql.hive.thriftserver.CliSuite.runCliWithin(CliSuite.scala:205)
[info]   at org.apache.spark.sql.hive.thriftserver.CliSuite.$anonfun$new$20(CliSuite.scala:501) {code}
Bug: Unable to set compression codec for Hive CTAS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/23 15:43
Affects Version/s: 3.4.0

Hi,

We've discovered code that worked in Spark 3.3.0 doesn't in 3.4.0. I can't find anything in the release notes to indicate why, so I wonder if this is a bug. Thank you for looking.

Here we're using our own custom codec, but we noticed we can't set gzip either.

{{  SparkConf conf = spark.sparkContext().conf();}}
{{  conf.set("hive.exec.compress.output", "true");}}
{{  conf.set("mapred.output.compression.codec", D2Codec.class.getName()); }}
{{  spark.sql("CREATE TABLE b AS SELECT id FROM a");}}

This will create the table, but it writes uncompressed files, where Spark 3.3.0 would write compressed files. 

Any advice is appreciated and I can help run tests. We run Spark on K8S using the stackable.tech distribution.
Bug: Add Iceberg and Hudi to third party projects
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/23 14:02
Affects Version/s: 4.0.0


{noformat}
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.util.concurrent.ExecutionException: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: iceberg. Please find packages at `https://spark.apache.org/third-party-projects.html`.
	at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:262)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:166)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:41)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:166)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:161)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:175)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
{noformat}
Bug: Provide an interface for Streaming sources to add _metadata columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/23 02:45
Affects Version/s: 3.5.1

Currently, only the native V1 file-based streaming source can read the `_metadata` column: [https://github.com/apache/spark/blob/370870b7a0303e4a2c4b3dea1b479b4fcbc93f8d/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala#L63]

 

Our goal is to create an interface that allows other streaming sources to add `{{{}_metadata`{}}} columns. For instance, we would like the Delta Streaming source, which you can find here: [https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/sources/DeltaDataSource.scala#L49], to extend this interface and provide the `{{{}_metadata`{}}} column for its underlying storage format, such as Parquet.
Bug: spark.sql.optimizeNullAwareAntiJoin should respect spark.sql.autoBroadcastJoinThreshold
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/23 00:02
Affects Version/s: 4.0.0

Normally broadcast join can be disabled when users set {{spark.sql.autoBroadcastJoinThreshold}} to -1. However this doesn't apply to {{spark.sql.optimizeNullAwareAntiJoin}}:

{code}
      case j @ ExtractSingleColumnNullAwareAntiJoin(leftKeys, rightKeys) =>
        Seq(joins.BroadcastHashJoinExec(leftKeys, rightKeys, LeftAnti, BuildRight,
          None, planLater(j.left), planLater(j.right), isNullAwareAntiJoin = true))
{code}