av9ash/GitBugs huggingface dataset:
Bug: Thrift server support
Status: Open | Priority: Major | Resolution: 
Created: 03/Mar/24 17:51
Affects Version/s: 4.0.0

Thrift server tests fail with following error:
repo: build/sbt -Phive-thriftserver "hive-thriftserver/testOnly org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite -- -z collations.sql

error:
```
```
[info] - collations.sql *** FAILED *** (1 second, 144 milliseconds)

[13016](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13017)[info] Expected "[aaa aaa]", but got "[java.lang.IllegalArgumentException

[13017](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13018)[info] Unrecognized type name: string COLLATE 'UCS_BASIC_LCASE']" Result did not match for query #7

[13018](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13019)[info] select * from t1 where ucs_basic = 'aaa' (ThriftServerQueryTestSuite.scala:226)

[13019](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13020)[info] org.scalatest.exceptions.TestFailedException:

[13020](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13021)[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)

[13021](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13022)[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)

[13022](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13023)[info] at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1564)

[13023](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13024)[info] at org.scalatest.Assertions.assertResult(Assertions.scala:847)

[13024](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13025)[info] at org.scalatest.Assertions.assertResult$(Assertions.scala:842)

[13025](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13026)[info] at org.scalatest.funsuite.AnyFunSuite.assertResult(AnyFunSuite.scala:1564)

[13026](https://github.com/dbatomic/spark/actions/runs/8129693575/job/22217101481#step:10:13027)[info] at org.apache.spark.sql.hive.thriftserver.ThriftServerQueryTestSuite.$anonfun$runQueries$6(ThriftServerQueryTestSuite.scala:226)
```
Bug: Support `spark.shutdown.timeout` config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Mar/24 21:50
Affects Version/s: 4.0.0

org.apache.spark.util.ShutdownHookManager is used to register custom shutdown operations. This is not easily configurable. The underlying org.apache.hadoop.util.ShutdownHookManager has a default timeout of 30 seconds.  It can be configured by setting hadoop.service.shutdown.timeout, but this must be done in the core-site.xml/core-default.xml because a new hadoop conf object is created and there is no opportunity to modify it.

org.apache.hadoop.util.ShutdownHookManager provides an overload to pass a custom timeout. Spark should use that and allow a user defined timeout to be used.

This is useful because we see timeouts during shutdown and want to give some extra time for the event queues to drain to avoid log data loss.
Bug: Provide to PySpark a functionality to get estimated size of DataFrame in bytes
Status: Open | Priority: Minor | Resolution: 
Created: 10/Mar/24 17:13
Affects Version/s: 4.0.0

Something equal to sessionState().executePlan(...).optimizedPlan().stats().sizeInBytes() in JVM-Spark. It may be done via simple call of `_jsparkSession` in a regular PySpark and via a plugin for Spark Connect.

 

This functionality is useful when one need to check a possibility of broadcast join without modifying global broadcast threshold.

 

The function in PySpark API may looks like: `DataFrame.estimate_size_in_bytes() -> float` or `DataFrame.estimateSizeInBytes() -> float`.
Bug: Replace commands with relations in a few tests in SparkConnectClientSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Mar/24 11:23
Affects Version/s: 4.0.0

A few [tests|https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/SparkConnectClientSuite.scala#L481-L527] in SparkConnectClientSuite attempt to test the result collection of a reattachable execution through the use of a SQL command. The SQL command, on a real server, is not executed eagerly (since it is a select command) and thus, is not entirely accurate. The test itself is non-problematic since a dummy server with dummy responses is used but a small improvement here would be to construct a relation rather than a command.
Bug: Assign classes to DEFAULT value errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:27
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_13[44-46]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign classes to interval errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:08
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[09-14]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign error classes to SHOW CREATE TABLE errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:04
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_127[0-5]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Rename errorClass to errorCondition and subClass to subCondition
Status: Open | Priority: Minor | Resolution: 
Created: 16/Mar/24 16:29
Affects Version/s: 4.0.0

We've agreed on the parent task to rename {{errorClass}} to align it more closely with the SQL standard, and take advantage of the opportunity to break backwards compatibility offered by the Spark version change from 3.5 to 4.0.

This ticket also covers renaming {{subClass}} as well.

This is a subtask so the changes are in their own PR and easier to review apart from other things.
Bug: Assign classes to Parquet converter errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:24
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_22[38-40]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign error classes to ALTER COLUMN errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:02
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_105[3-4]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Assign classes to Parquet type errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:23
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_11[72-74]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Adding new 'Silent' ExplainMode
Status: Open | Priority: Minor | Resolution: 
Created: 18/Mar/24 16:49
Affects Version/s: 4.0.0

While investigating unit test duration we found that org.apache.spark.sql.execution.QueryExecution.explainString () takes approximately 14% time. This method generates the string representation of the execution plan. The string is often used for logging purposes. This is also called for each AQE job so it can save prod execution time too. While SPARK-44485 does exist to help optimize the prod execution time, the main purpose of this PR is to save time during unit testing.

I've added a silent mode to ExplainMode to try and mitigate this issue.
Bug:  Assign classes to Row to JSON errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 13:22
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_32[49-51]* defined in {*}core/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: XML: Ignore row tags in CDATA Tokenizer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Mar/24 00:28
Affects Version/s: 4.0.0

The current parser does not recognize CDATA sections and thus will read row tags that are enclosed within a CDATA section. The expected behavior is for none of the following rows to be read, but they are all read. 
{code:java}
// BUG:  rowTag in CDATA section
val xmlString="""<?xml version="1.0" encoding="UTF-8" ?>
<test><![CDATA[
    <elem id="1" />
    <elem id="2" > </elem>
    <elem> <id>3</id> </elem>
]]>
</test>
{code}
Bug:  AuthEngine key exchange needs additional KDF round
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Mar/24 17:27
Affects Version/s: 3.4.0

AuthEngine implements a bespoke [key exchange protocol |[https://github.com/apache/spark/tree/master/common/network-common/src/main/java/org/apache/spark/network/crypto]|https://github.com/apache/spark/tree/master/common/network-common/src/main/java/org/apache/spark/network/crypto].] based on the NNpsk0 Noise pattern and using X25519.

The Spark code improperly uses the derived shared secret directly, which is an encoded X coordinate. This should be passed into a KDF rather than used directly.
Bug: Allow LiveEventBus to stop without the completely draining of event queue
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 04:44
Affects Version/s: 4.0.0

#Problem statement:
The SparkContext.stop() hung a long time on LiveEventBus.stop() when listeners slow

#User scenarios:
We have a centralized service with multiple instances to regularly execute user's scheduled tasks.
For each user task within one service instance, the process is as follows:

1.Create a Spark session directly within the service process with an account defined in the task.
2.Instantiate listeners by class names and register them with the SparkContext. The JARs containing the listener classes are uploaded to the service by the user.
3.Prepare resources.
4.Run user logic (Spark SQL).
5.Stop the Spark session by invoking SparkSession.stop().

In step 5, it will wait for the LiveEventBus to stop, which requires the remaining events to be completely drained by each listener.


Since the listener is implemented by users and we cannot prevent some heavy stuffs within the listener on each event, there are cases where a single heavy job has over 30,000 tasks,
and it could take 30 minutes for the listener to process all the remaining events, because within the listener, it requires a coarse-grained global lock and update the internal status to the remote database.


This kind of delay affects other user tasks in the queue. Therefore, from the server side perspective, we need the guarantee that the stop operation finishes quickly.
Bug: try_to_number fails with NPE for malformed input
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Mar/24 07:20
Affects Version/s: 3.5.1

{code}
scala>     import org.apache.spark.sql.functions._
     |     val df = spark.createDataset(spark.sparkContext.parallelize(Seq("11")))
     |     df.select(try_to_number($"value", lit("$99.99"))).show()
24/03/29 16:19:16 ERROR Executor: Exception in task 10.0 in stage 2.0 (TID 15)
java.lang.NullPointerException: Cannot invoke "org.apache.spark.sql.types.Decimal.toPlainString()" because "<local7>" is null
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:50)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:894)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:894)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:368)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:332)
{code}
Bug: count_distinct ignores null values
Status: Open | Priority: Minor | Resolution: 
Created: 14/Mar/24 15:12
Affects Version/s: 3.4.1

The documentation states, that in group by and count statements, null values will not be ignored / form their own groups.

!image-2024-03-14-16-13-03-107.png|width=491,height=373!
However, the behavior of count_distinct does not account for nulls. 
Either the documentation or the implementation is wrong here...

!image-2024-03-14-16-12-35-267.png!
Bug: Assign error classes to FILTER expression errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Mar/24 12:59
Affects Version/s: 4.0.0

Choose a proper name for the error class *_LEGACY_ERROR_TEMP_102[4-7]* defined in {*}common/utils/src/main/resources/error/error-classes.json{*}. The name should be short but complete (look at the example in error-classes.json).

Add a test which triggers the error from user code if such test still doesn't exist. Check exception fields by using {*}checkError(){*}. The last function checks valuable error fields only, and avoids dependencies from error text message. In this way, tech editors can modify error format in error-classes.json, and don't worry of Spark's internal tests. Migrate other tests that might trigger the error onto checkError().

If you cannot reproduce the error from user space (using SQL query), replace the error by an internal error, see {*}SparkException.internalError(){*}.

Improve the error message format in error-classes.json if the current is not clear. Propose a solution to users how to avoid and fix such kind of errors.

Please, look at the PR below as examples:
 * [https://github.com/apache/spark/pull/38685]
 * [https://github.com/apache/spark/pull/38656]
 * [https://github.com/apache/spark/pull/38490]
Bug: Primary resource jar added to spark.jars twice under k8s cluster mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 20/Mar/24 23:44
Affects Version/s: 3.4.0

{*}Context{*}:
To submit spark jobs to Kubernetes under cluster mode, the {{spark-submit}} will be triggered twice. 
The first time {{SparkSubmit}} will run under k8s cluster mode, it will append primary resource to {{spark.jars}} and call {{KubernetesClientApplication::start}} to create a driver pod. 
The driver pod will run {{spark-submit}} again with the same primary resource jar. However this time the {{SparkSubmit}} will run under client mode with {{spark.kubernetes.submitInDriver}} as {{true}}, plus the updated {{spark.jars}}. Under this mode, {{SparkSubmit}} will download all the jars in {{spark.jars}} to driver and those {{spark.jars}} urls will be replaced by the driver local paths. 
Then SparkSubmit will append the same primary resource to spark.jars again. So in this case, {{spark.jars}} will have 2 paths of duplicate copies of primary resource, one with the original url user submit with, the other with the driver local file path. 
Later when driver starts the SparkContext, it will copy all the {{spark.jars}} to {{spark.app.initial.jar.urls}}, and replace the driver local jars paths in {{spark.app.initial.jar.urls}} with driver file service paths. 
Now all the jars in the {{--jars}} or `spark.jars` in the original user submission will be replaced with a driver file service url and added to  {{spark.app.initial.jar.urls}}. And the primary resource jar in the original submission will show up in {{spark.app.initial.jar.urls}} twice: one with the original path in the user submission, the other with a driver file service url.
When executors start, they will download all the jars in the {{spark.app.initial.jar.urls}}. 

*Issue*:
The executor will download 2 duplicate copies of primary resource, one with the original url user submit with, the other with the driver local file path, which leads to resource waste. This is also reported previously [here|https://github.com/apache/spark/pull/37417#issuecomment-1517797912].