av9ash/GitBugs huggingface dataset:
Bug: Support Kryo Serialization with AgnosticEncoders
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/24 20:27
Affects Version/s: 4.0.0

Add support for Kryo based serialization to Agnostic encoders. This will allow us to port the entire Encoders class from sql/core to sql/api.

Unfortunately supporting connect is not really possible at this moment. We cannot share the configuration of the Kryo objects between the server and connect. This is not possible due to the fact that connect - by design - does not have a all classes needed on its classpath. This makes constructing the same configuration (with the same class ids) almost impossible. On top of this, backwards compatibility will be a problem.

For connect the only way forward is to have a separately configured version of the kryo serializer that leverages hard coded class ids. That is probably going to need some form of configurability.
Bug: PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/24 18:21
Affects Version/s: 4.0.0

 *_{{SimplifyConditionals}}_* has a simplification code where if any of the (not first) branches  in _*{{CaseWhen}}*_ is *_{{TrueLiteral}}_* we remove all the remaining branches including the *_{{{}elseValue{}}}._*
{code:java}
case CaseWhen(branches, ) if branches.exists(._1 == TrueLiteral) => // a branch with a true condition eliminates all following branches,
// these branches can be pruned away
val (h, t) = branches.span(_._1 != TrueLiteral)
CaseWhen( h :+ t.head, None)}}{code}
Now, the nullability check of *_{{CaseWhen}}_* checks that 
(1) either of the branches including *_{{elseValue}}_* is {{nullable}} or 
(2) *_{{elseValue}}_* is {{{}None{}}}.

 

The above simplification makes the *_{{elseValue}}_* as {{{}None{}}}. Combined with this nullability check makes the *_{{CaseWhen}}_* switch from {{non-nullable}} to {{{}nullable{}}}.

This ticket aims to fix this issue and remain the nullability of the expression by replacing the *_elsevalue_* to the value of the TrueLiteral expression.
Bug: Fix `docker-image-tool.sh` to be up-to-date
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/24 21:46
Affects Version/s: 4.0.0

Apache Spark 4 dropped Java 11 support. So, we should fix the following.
{code}
-  - Build and push Java11-based image with tag "v3.4.0" to docker.io/myrepo
+  - Build and push Java17-based image with tag "v4.0.0" to docker.io/myrepo
{code}

Apache Spark 4 requires JDK instead of JRE. So, we should fix the following.
{code}
-    $0 -r docker.io/myrepo -t v3.4.0 -b java_image_tag=11-jre build
+    $0 -r docker.io/myrepo -t v4.0.0 -b java_image_tag=17 build
{code}

Lastly, `3.4.0` is too old because it's released on April 13, 2023. We had better use v4.0.0.
{code}
-    $0 -r docker.io/myrepo -t v3.4.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
+    $0 -r docker.io/myrepo -t v4.0.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
{code}
Bug: Poor performance in ProjectingInternalRow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/24 11:27
Affects Version/s: 3.5.2

In {*}ProjectingInternalRow{*}, the *colOrdinals* is passed as a {_}List{_}. According to the Scala documentation, the _{{apply}}_ method for _{{List}}_ has a linear time complexity, and it is used in all methods of ProjectingInternalRow for every row. This can have a significant impact on performance.

The following flame graph was captured in a {*}merge into sql{*}. A considerable amount of time was spent on {{{}List.apply{}}}. Changing this to _{{IndexedSeq}}_ would improve the performance.

 

[^20240827-172739.html]

[https://docs.scala-lang.org/overviews/collections-2.13/performance-characteristics.html]
Bug: ReattachExecute does not supply server-side session id
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/24 08:27
Affects Version/s: 4.0.0

Related to SPARK-47380.

 

One example.

Driver restart -> session re created -> OPERATION_NOT_FOUND instead of SESSION_NOT_FOUND because the first attempt to reattach a session does not contain the last time observed server side session id.

E.g.,

Start execution (client_observed_server_side_session_id is set) -> driver restart -> reattach (client_observed_server_side_session_id is not set) -> OPERATION_NOT_FOUND instead of SESSION_NOT_FOUND.
Bug: Handle null metrics in ConnectProgressExecutionListenerSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/24 23:32
Affects Version/s: 3.5.0

On task end, `TaskMetrics` may be `null`, as in the case of task failure (see [here|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala#L83]).

We should not throw an NPE in this case and instead handle gracefully.
Bug: Fix return type nullness of the xpath expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/24 17:11
Affects Version/s: 4.0.0

The `xpath` expression incorrectly marks its return type as array of non-null strings. However, it can actually return an array containing nulls. This can cause NPE in code generation, such as query `select coalesce(xpath(repeat('<a></a>', id), 'a')[0], '') from range(1, 2)`.
Bug: Base64 nullability change
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/24 20:10
Affects Version/s: 3.5.2

In [https://github.com/apache/spark/pull/47303] , the Base64 function was updated to use `StaticInvoke` instead of directly calling the underlying encode function. This change also updated Base64 of a non-nullable child to be nullable instead of non-nullable. This was backported to Spark 3.5 as well ([https://github.com/apache/spark/pull/47325)] and has caused issues with failing state schema checks for stateful streaming jobs. This should be fixed to have the nullability based on the child.
Bug: Add expression for `java.util.Arrays.binarySearch`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/24 03:15
Affects Version/s: 4.0.0

Add a dedicated expression 
{code:java}
ArrayBinarySearch(array, value) {code}
for binary search, the behavior should be the same as `java.util.Arrays.binarySearch`, so that we can use it to implement histogram plot in the client side (no longer need to depend on mllib's Bucketizer)

 

This expression is for internal purposes, should not be exposed to end users.

It assume the array is already sorted.

if array or value is null, returns null.
Bug: NullPointerException from SparkThrowableHelper.isInternalError method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/24 06:45
Affects Version/s: 3.5.2

The SparkThrowableHelper.isInternalError method doesn't handle null input, and it could lead to NullPointerException.

Example stacktrace from our environment with Spark 3.5.1:
{code:java}
 Caused by: java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because "errorClass" is null
    at org.apache.spark.SparkThrowableHelper$.isInternalError(SparkThrowableHelper.scala:64)
    at org.apache.spark.SparkThrowableHelper.isInternalError(SparkThrowableHelper.scala)
    at org.apache.spark.SparkThrowable.isInternalError(SparkThrowable.java:50)
    at org.apache.spark.SparkException.isInternalError(SparkException.scala:27)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:688)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:772)
    ... 30 more  {code}
Bug: Fix example props override in helm chart operatorConfiguration.dynamicConfig.data
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 29/Aug/24 23:14
Affects Version/s: kubernetes-operator-0.1.0

Existing values include an example override line for `operatorConfiguration.dynamicConfig.data` :

 

```
spark.kubernetes.operator.reconciler.intervalSeconds: 60
```

 

This could cause failures at installation:

 

```

Error: INSTALLATION FAILED: ConfigMap in version "v1" cannot be handled as a ConfigMap: json: cannot unmarshal number into Go struct field ConfigMap.data of type string

```

 

In fact, all key-value pairs are consumed as strings by operator.
Bug: Support collations with get_json_object and json_tuple
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/24 08:26
Affects Version/s: 4.0.0

Make get_json_object and json_tuple collation aware. 

 

Currently get_json_object takes in 2 inputs, a JSON object, and a JSON path expression.

Example:

Note that ǆ is a single codepoint(U+01F3), and so is ﬀ(U+FB00).

Query:
{code:java}
select get_json_object('{"abcǆ": { "abﬀ": "something" } }', '$.abcǆ')
{code}
Output:
{code:java}
+----------------------------------------------------------+
|get_json_object({"abcǆ": { "abﬀ": "something" } }, $.abcǆ)|
+----------------------------------------------------------+
|                                       {"abﬀ":"something"}|
+----------------------------------------------------------+{code}
 

We would want to support collations in such a way that every .<name>(traversing, through the JSON tree, to a node by its node name) is collation aware. Here is an example of what would we want to support(in this example, with respect to UNICODE_CI_AI collation):

Query:
{code:java}
select get_json_object('{"abcǆ": { "abﬀ": "something" } }' collate UNICODE_CI_AI, '$.abcdz.abFf'){code}
Output:
{code:java}
+----------------------------------------------------------------------------------------+
|get_json_object(collate({"abcǆ": { "abﬀ": "something" } }, UNICODE_CI_AI), $.abcdz.abFf)|
+----------------------------------------------------------------------------------------+
|                                                                               something|
+----------------------------------------------------------------------------------------+{code}
 

 

With json_tuple we would want a similar thing.

json_tuple takes in a JSON object, and multiple names, and for each name, it returns its JSON object. Basically, for each name given as a query, it scans the children of the root, and if there is a child named like the queried name, then that childs JSON object is returned as an output for the queried name. Outputs for all the names are then return as an output for json_tuple.

Example:

Query:
{code:java}
select json_tuple('{\"abcǆ\": { \"abﬀ\": \"something\" } , \"abc\": \"something2\"}', 'abcǆ', 'abc') {code}
Output:
{code:java}
+-------------------+----------+
|                 c0|        c1|
+-------------------+----------+
|{"abﬀ":"something"}|something2|
+-------------------+----------+ {code}
Similarly like with get_json_object, we would want the following behavior:

Query:
{code:java}
select json_tuple('{\"abcǆ\": { \"abﬀ\": \"something\" } , \"abc\": \"something2\"}' collate 'UNICODE_CI_AI', 'abcdz', 'aBc') {code}
Output:
{code:java}
+-------------------+----------+
|                 c0|        c1|
+-------------------+----------+
|{"abﬀ":"something"}|something2|
+-------------------+----------+ {code}
Bug: Split case classes implementing a datasource into pairs of abstract base classes and case classes
Status: Open | Priority: Major | Resolution: 
Created: 31/Aug/24 13:38
Affects Version/s: 4.0.0

Currently, to extend an existing V2 datasource, we should inherit to common base classes, mostly duplicating the existing implementation. This happen because the datasource logic is implemented in case classes. E.g., to extend {{{}v2.parquet{}}}datasource Scan with{{{}SupportsReportPartitioning{}}} we should duplicate about hundred lines of {{ParquetScan}} adding only a single method of our own. Then, we should keep an eye on the duplicated code when we upgrade spark version: maybe there are improvements.

I propose to split the above mentioned case classes into pairs of abstract base classes and case classes. A base class should contain the logic that may be inherited and a case class should contain only minimal logic, if any.
Bug: Spark Driver process not terminating on Kubernetes
Status: Open | Priority: Major | Resolution: 
Created: 30/Aug/24 18:56
Affects Version/s: 3.2.0

We are migrating from YARN to Kubernetes (K8s). The simple program below experiences a driver hang when running in K8s but completes successfully when running in YARN.

 
{code:java}
object TestSparkJobHanging {  
  def main(args: Array[String]): Unit = {  
    val spark = SparkSession  
      .builder()  
      .getOrCreate()  
  
    // executorService is not daemon thread  
    val executorService = Executors.newFixedThreadPool(6)  
    // submit 1 task to executor  
    val runnableTask: Runnable = new Runnable {  
      def run(): Unit = {  
        spark.read.json(args(0)).write.parquet(args(1))  
      }  
    }  
    val future = executorService.submit(runnableTask)  
    future.get()  
  }  
}   {code}
 

The root cause of this issue lies in the architectural differences between YARN and Kubernetes. When Spark runs in cluster mode on YARN, the driver program terminates by calling System.exit(). However, in K8s, the driver needs to wait for all non-daemon threads to exit.

Questions:
 # Has this issue been addressed in the latest Spark version?

 # If not, is this the expected behavior? Do we have plans to make the behavior consistent between YARN and Kubernetes (K8s)?
Bug: CONNECT_SESSION_PLAN_CACHE_SIZE is too small for certain programming patterns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/24 14:54
Affects Version/s: 4.0.0

Example:

 

```

df_1 = df_a.filter(col('X').isNotNull())

df_2 = df_b.filter(col('SAFE_SU_Conv').isNotNull())

....

df_x = ...

for _ in range(0, 5):

    df_x = df_x.select(...)

...

df_3 = df_1.join(df_2, ...)

```

=> df_x completely invalidates all the cached entries.
Bug: Break apart the Structured Streaming Programming Guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Aug/24 15:00
Affects Version/s: 4.0.0

As discussed in [this email thread,|https://lists.apache.org/thread/tbqxg28w4njsp4ws5gbssfckx5zydbdj] we should break apart the Structured Streaming programming guide to make it easier for readers to consume (it will also help SEO).
Bug: Decouple TaskSetExcludelist creation from HealthTracker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Aug/24 19:14
Affects Version/s: 3.5.3

TaskSetManager creates TaskSetExcludelist based on if TaskScheduler has a HealthTracker available.

This task tracks the effort to decouple task/stage level excludeList and application level healthTracker creation, so they can be enabled independently.
Bug: Spark website doesn't properly scroll to hash links 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/24 05:40
Affects Version/s: 4.0.0

On the version-specific Spark documentation, if you click a header, the page will scroll past the actual content, hiding it. For example, if you go to [this link|https://spark.apache.org/docs/latest/#downloading], you'll probably notice the page scroll past "Downloads".
Bug: Add unit tests for potential RocksDB state store SST file mismatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Aug/24 23:49
Affects Version/s: 4.0.0

Recently we discovered some RocksDB state store file version ID mismatch issues. Although it happens to have been fixed by other change, we don't have test coverage for it. Add unit tests for them.
Bug: Avoid double normalization in the cache process
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/24 04:05
Affects Version/s: 4.0.0

There is an issue introduced in [#46465|https://github.com/apache/spark/pull/46465], which is that normalization is applied twice during the cache process. Some normalization rules may not be idempotent, so applying them repeatedly may break the plan shape and cause an unexpected cache miss.
Bug: A query running under ckpt version v1 should be able to read a STATELESS query running under ckptv2
Status: Open | Priority: Major | Resolution: 
Created: 29/Aug/24 18:22
Affects Version/s: 4.0.0

Follow up to [https://github.com/apache/spark/pull/47932]

The validate version method force checks if the version persisted in the commit log 

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CommitLog.scala#L59]

 

But the V2 checkpoint is only meaningful to stateful queries, so for a stateless query, a CommitLog running under V1 (old code) should be able to process a persisted commit log that has V2 format (new log)
Bug: Complete Metadata requests on each micro batch causing Kafka issues
Status: Open | Priority: Major | Resolution: 
Created: 28/Aug/24 05:10
Affects Version/s: 3.3.2

We have noticed that spark does complete metadata requests on each micro batch and this is causing high metadata requests on small micro batch intervals .

 

For example Kafka with 1900 partitions and 10 sec micro batch we are seeing order of 
~{*}360K{*} metadata requests / sec 

Same with job with 60 sec micro batch we are observing *~60K* meta data requests.

 

Metadata requests are controlled by *metadata.max.age.ms* but these config have no effect on spark consumers by default its 5 mins still we are seeing these huge requests.
Bug: Fix Binder integration in PySpark documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Aug/24 05:02
Affects Version/s: 3.4.3

https://mybinder.org/v2/gh/apache/spark/bb7846dd487?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb is broken.
Bug: Support Custom configuration of exposed types: ClusterIP, Nodeport, LoadBalancer, Ingress
Status: Open | Priority: Major | Resolution: 
Created: 27/Aug/24 10:38
Affects Version/s: 4.0.0

The exposed rest service could be used to access the Spark’s Web UI，Similar to flink kubernetes.rest-service.expose.type: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#kubernetes-rest-service-exposed-type
Bug: Handle surrogate pairs properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/24 06:50
Affects Version/s: 4.0.0

Fix the following string expressions to handle one-to-many case mapping properly:
 * StringReplace
 * StringInstr
 * StringLocate
 * SubstringIndex
 * StringTrim
 * StringTrimLeft
 * StringTrimRight

 

Examples of incorrect results (under {{ICU}} collations):
{code:java}
StringReplace("😄a", "a", "b") // returns: "😄ab" (incorrect), instead of: "😄b" (correct){code}
 
{code:java}
StringInstr("😄a", "a") // returns: 3 (incorrect), instead of: 2 (correct){code}
 
{code:java}
StringLocate("a", "😄a") // returns: 3 (incorrect), instead of: 2 (correct){code}
 
{code:java}
SubstringIndex("😄a", "a") // returns: "😄a" (incorrect), instead of: "😄" (correct){code}
 
{code:java}
StringTrim("😄", "😄") // returns: "😄" (incorrect), instead of: "" (correct){code}
 
{code:java}
StringTrimLeft("😄", "😄") // returns: "😄" (incorrect), instead of: "" (correct){code}
 
{code:java}
StringTrimRight("😄", "😄") // returns: "😄" (incorrect), instead of: "" (correct){code}
Bug: hive.exec.compress.output not working as expected
Status: Open | Priority: Major | Resolution: 
Created: 26/Aug/24 16:04
Affects Version/s: 3.4.0

reproduction:
{code:java}
import org.apache.spark.sql.SaveMode

spark.conf.set("hive.exec.compress.output", "true")
spark.conf.set("mapred.output.compression.codec", "org.apache.hadoop.io.compress.BZip2Codec")
spark.conf.set("mapred.output.compression.type", "BLOCK")

val data = Seq(
  (1, "Test1", 100),
  (2, "Test2", 200),
  (3, "Test3", 300),
  (4, "Test4", 400),
  (5, "Test5", 500)
)

val data = spark.createDataFrame(data).toDF("a", "b", "c")

val databaseName = "test_database"
val tableName = "test_table"

spark.sql(s"DROP DATABASE IF EXISTS $databaseName CASCADE")
spark.sql(s"CREATE DATABASE $databaseName")

data.write.format("hive")
  .mode(SaveMode.Overwrite)
  .saveAsTable(s"$databaseName.$tableName")

{code}
Expectation/Behaviour in Spark != 3.4:

{noformat}
$ hadoop fs -ls /user/spark/warehouse/test_database.db/test_table 
Found 5 items
-rwxr-xr-x   1 hadoop spark         53 2024-08-23 15:39 /user/spark/warehouse/test_database.db/test_table/part-00000-c2f5bb2d-afbe-4f28-bc5b-b076211c3c15-c000.bz2
-rwxr-xr-x   1 hadoop spark         53 2024-08-23 15:39 /user/spark/warehouse/test_database.db/test_table/part-00001-c2f5bb2d-afbe-4f28-bc5b-b076211c3c15-c000.bz2
...
{noformat}

In Spark 3.4 the {{.bz2}} extension is missing and the files aren't compressed.
 
This bug was introduced in 3.4.0 by the refactoring in SPARK-41708 - pull request #39277 and "accidentally" fixed in 3.5 by SPARK-43186

The patch which breaks the functionality relies on the "pass by pointer" nature of Scala/Java to populate the compression parameters in an object

{code}
val fileSinkConf = new FileSinkDesc(hiveTempPath.externalTempPath.toString, tableDesc, false)
setupHadoopConfForCompression(fileSinkConf, hadoopConf, sparkSession)
{code}

However, in Spark 3.4, the {{fileSinkConf}} is wrapped in a shim which makes a copy when translating a {{ShimFileSinkDesc}} to a {{FileSinkDesc}}. This means that the {{setupHadoopConfForCompression}} operates on the copy rather than the object being used afterwards.

This is fixed in 3.5 because the shim (and thus the copy) was removed.
Bug: Large INTERVAL SECOND values cannot be cast to decimal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Aug/24 22:39
Affects Version/s: 4.0.0

Spark fails when trying to cast an `interval second` value to decimal where the number of microseconds requires 19 digits.



{code:java}
scala> sql("select 1000000000000.000000::interval second").show(false)

+---------------------------------------------+

|CAST(1000000000000.000000 AS INTERVAL SECOND)|

+---------------------------------------------+

|INTERVAL '1000000000000' SECOND              |

+---------------------------------------------+

 

 

scala> sql("select 1000000000000.000000::interval second::decimal(38, 10)").show(false)

org.apache.spark.SparkArithmeticException: [NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION]  0 cannot be represented as Decimal(18, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003
{code}