av9ash/GitBugs huggingface dataset:
Bug: NullPointerException in InMemoryTableScanExec
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 13:39
Affects Version/s: 4.0.0

{{}}
{code:java}
java.lang.NullPointerException: null  at org.apache.spark.sql.execution.columnar.InMemoryTableScanExec.filteredCachedBatches(InMemoryTableScanExec.scala:156)  at org.apache.spark.sql.execution.columnar.InMemoryTableScanExec.inputRDD$lzycompute(InMemoryTableScanExec.scala:98)  at org.apache.spark.sql.execution.columnar.InMemoryTableScanExec.inputRDD(InMemoryTableScanExec.scala:84)  at org.apache.spark.sql.execution.columnar.InMemoryTableScanExec.doExecute(InMemoryTableScanExec.scala:163)  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:195)  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:191)  at org.apache.spark.sql.execution.InputAdapter.inputRDD(WholeStageCodegenExec.scala:527)  at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs(WholeStageCodegenExec.scala:455)  at org.apache.spark.sql.execution.InputRDDCodegen.inputRDDs$(WholeStageCodegenExec.scala:454)  at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:498)  at org.apache.spark.sql.execution.ProjectExec.inputRDDs(basicPhysicalOperators.scala:51)  at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:751)  at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:195)  at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)  at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)  at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:191)  at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:364)  at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:445)  at org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:4218)  at org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:3459)  at org.apache.spark.sql.Dataset.$anonfun$withAction$2(Dataset.scala:4208)  at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:526)  ... 23 more{code}
Bug: ClassCastException in PySpark when writing to DynamoDB (MapWritable to DynamoDBItemWritable)
Status: Open | Priority: Major | Resolution: 
Created: 13/Dec/24 11:55
Affects Version/s: 3.5.2

Writing data to a DynamoDB table using PySpark results in the following error:
{code:java}
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.MapWritable cannot be cast to org.apache.hadoop.dynamodb.DynamoDBItemWritable
    at org.apache.hadoop.dynamodb.write.DefaultDynamoDBRecordWriter.convertValueToDynamoDBItem(DefaultDynamoDBRecordWriter.java:22){code}
However, when using the same logic written in Scala, the write operation works as expected.

This indicates a potential issue in the PySpark logic responsible for converting RDD elements before they are written to DynamoDB via `DefaultDynamoDBRecordWriter`. This prevents users from leveraging PySpark for integration with DynamoDb.
 * Potential area of interest: [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala#L183-L186]

 

*Broad steps to reproduce the issue:*
 * Create a DynamoDB table with the required schema.
 * Load some data into S3 to serve as the source data.
 * Read from S3 and write to the DynamoDB table using PySpark.
 ** You should encounter the error mentioned above.
 * Repeat the same steps using Scala.
 ** This time, the data will be successfully written to the DynamoDB table with no errors encountered.
Bug: Broken support for disabling hostname verification based on insecure-skip-tls-verify in KUBECONFIG
Status: Open | Priority: Major | Resolution: 
Created: 12/Dec/24 17:35
Affects Version/s: 3.2.0

SPARK-33720 via [PR #30684|https://github.com/apache/spark/pull/30684] broke usage of {{insecure-skip-tls-verify}} via KUBECONFIG file when using cluster context with auto configure.

PR for this issue overwrites {{insecure-skip-tls-verify}} from KUBECONFIG set by autoconfigure when new KUBERNETES_TRUST_CERTIFICATES property is missing because it defaults to false. With this bug it is impossible to drive hostname verification via KUBECONFIG when multiple clusters are present with different requirements.

Correct implementation should do the following:
 # If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure isTrustCerts=true then no hostname verification
 # If KUBERNETES_TRUST_CERTIFICATES is not set AND Config from autoconfigure isTrustCerts=false then do hostname verfication
 # If KUBERNETES_TRUST_CERTIFICATES is set use this as an override

I have verified autoconfigure with insecure-skip-tls-verify set to true works in 3.1.3 before this change.

Relevant code from Spark and kuubernetes-client:

 
{code:java}
val KUBERNETES_TRUST_CERTIFICATES =
    ConfigBuilder("spark.kubernetes.trust.certificates")
      .createWithDefault(false)
val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
      .withTrustCerts(sparkConf.get(KUBERNETES_TRUST_CERTIFICATES))
// kubernetes-client Config.autoConfigure
config.setTrustCerts(currentCluster.getInsecureSkipTlsVerify() != null && currentCluster.getInsecureSkipTlsVerify());
config.setDisableHostnameVerification(currentCluster.getInsecureSkipTlsVerify() != null && currentCluster.getInsecureSkipTlsVerify())
{code}
Bug: Add loging support for Spark dataset API show()
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/24 20:24
Affects Version/s: 3.5.3

Currently Spark Dataset API supports show operation and send the output to console. But if the container restarted then this console output is not available. So there is a need to build the support for logging with Show method.

 
{code:java}
def show(numRows: Int, truncate: Boolean): Unit = if (truncate) {
  println(showString(numRows, truncate = 20))
} else {
  println(showString(numRows, truncate = 0))
} {code}
We will introduce a new parameter to use logging instead of sending to console.
Bug: [Connect] Support sc.broadcast on Connect 
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/24 18:19
Affects Version/s: 4.0.0

Spark Context methods are not supported in Spark Connect. Broadcast is a common API used by developers to send over some data to all executors. Eg. machine learning model weights can be broadcasted, so no need to fetch individually.
 
This makes migration of workloads to Spark Connect complex.
 
I will try to come up with a design doc & PR after that if approach looks good.
Bug: Change partitioning to SinglePartition if partition number is 1
Status: Open | Priority: Major | Resolution: 
Created: 05/Dec/24 07:56
Affects Version/s: 4.0.0

For node `RebalancePartitions` and `RebalanceByExpression`, if partition number is 1 we can use `SinglePartition` instead of other `Partitioning`.
h4.
Bug: Reading partition values from parquet over multiple batches fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/24 23:46
Affects Version/s: 4.0.0

https://github.com/apache/spark/commit/800faf0abfa368ad0a5ef1e0fa44b74dbaab724e cleans up column vector resources in column to row. However, `ConstantColumnVector` holds the same value over multiple batches and shouldn't be cleaned up.
Bug: [M1] Fix bug where empty BEGIN END blocks throw error in loops 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/24 21:25
Affects Version/s: 4.0.0

There is a bug in SQL scripting, where compound statements throw an error if their body is consisted solely of empty BEGIN END blocks. Examples:

{{WHILE 1 = 1 DO}}

{{  BEGIN}}

{{  END;}}
{{END WHILE;}}
Bug: Operations on a dataset with void/nulltype fail
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/24 19:49
Affects Version/s: 3.5.0

If a column is of data type `void` / `NullType()` , basic operations fail. These should behave as columns of entirely `null` data of a bottom type, implicitly or explicitly castable to any other type in the event of an insert/union/etc.

 

Something like select, union, or save raises `[INTERNAL_ERROR]` with the column not found, or methods over dataframes like `isEmpty()` fail, with errors similar to

 

{{Py4JJavaError{color:#555555}: An error occurred while calling o2303.isEmpty. : org.apache.spark.SparkException: [INTERNAL_ERROR] Couldn't find COLUMNNAME#REFNUMBER in [...,_databricks_internal_edge_computed_column_skip_row#17633] SQLSTATE: XX000{color}}}
Bug: MERGE fails with INVALID_NON_DETERMINISTIC_EXPRESSIONS if MATCHED case is presented
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/24 14:06
Affects Version/s: 3.5.3

Such query runs well:
{code:sql}
MERGE INTO "2" AS target
USING (
  select name from "1"
) AS source
ON true
WHEN MATCHED THEN UPDATE SET name = source."name"
WHEN NOT MATCHED THEN INSERT (name) VALUES (source.name)
{code}

But if some non-detemenistic filter applied, e.x.
{code:sql}
MERGE INTO "2" AS target
USING (
  select name 
  from "1"
  qualify
    row_number() over (
      partition by name
      order by rand()
    ) = 1
) AS source
ON true
WHEN MATCHED THEN UPDATE SET name = source."name"
WHEN NOT MATCHED THEN INSERT (name) VALUES (source.name)
{code}

then INVALID_NON_DETERMINISTIC_EXPRESSIONS error appears:
{code}
[INVALID_NON_DETERMINISTIC_EXPRESSIONS] The operator expects a deterministic expression, but the actual expression is "true", "exists()".
{code}


1. After removing WHEN MATCHED case query compiles again.
2. Merge condition doesn't matter.

I didn't find any explicit requirements regarding MERGE's input, so it feels like a bug.
Bug: Add default value for driver/executor CPU limit configuration
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/24 08:34
Affects Version/s: 3.5.1

 

K8s pod.yaml
{code:java}
limits:
    memory: 24320Mi
requests:
    cpu: "4"
    memory: 24320Mi{code}
If *spark.kubernetes.driver.limit.cores* and *spark.kubernetes.executor.limit.cores* are not specified when submitting a Spark job on Kubernetes, the CPU limit for the Kubernetes pod will be empty. By default, it would be more appropriate to align the values with *driver cores requests* and *executor cores request*
Bug: Disable repartition by Variant columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Dec/24 20:38
Affects Version/s: 4.0.0

Currently, repartition operations on Variant columns like `df.repartition(5, "v")` or `sql("...distribute by v")` are not blocked during analysis. This should be blocked since Variant equality is not defined.
Bug: df.rdd.countApprox() : PySpark API does not respect the timeout.
Status: Open | Priority: Major | Resolution: 
Created: 09/Dec/24 09:38
Affects Version/s: 3.5.0

The 


{code:java}
# Create a large table with dummy data
df = spark.range(100000000)
df.write.format("delta").saveAsTable("large_table")

import time
df = spark.read.format("delta").table("large_table")
start_time = time.time()
df.cache()
print(f"count = {df.rdd.countApprox(timeout=1)}")
print(f"Time taken: {time.time() - start_time} seconds")
{code}
===> runs until the count is fetched (2 min)

{code:java}

%scala
   // Read the Delta table into a DataFrame
    val df = spark.read.format("delta").table("large_table")
    val startTime = System.nanoTime()
    df.cache()
    val approxCount = df.rdd.countApprox(timeout = 10)
    val endTime = System.nanoTime()
    val duration = (endTime - startTime) / 1e9 // Convert nanoseconds to seconds
    println(s"Time taken: $duration seconds")

{code}

===> times out in 10 ms

This is also commented in the below doc, but I could not find a corresponding bug for this.
https://mlflow.org/docs/latest/_modules/mlflow/data/spark_dataset.html
 # Use Spark RDD countApprox to get approximate count since count() may be expensive.
            # Note that we call the Scala RDD API because the PySpark API does not respect the
            # specified timeout. Reference code:
            # https://spark.apache.org/docs/3.4.0/api/python/_modules/pyspark/rdd.html
            # #RDD.countApprox. This is confirmed to work in all Spark 3.x versions
Bug: Spark build failure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/24 07:15
Affects Version/s: 3.5.3

DeduplicateWithinWatermark
 
overriding lazy value references in class QueryPlan of type org.apache.spark.sql.catalyst.expressions.AttributeSet; method references needs to be a stable, immutable value
 
old
{code:java}
override def references: AttributeSet = AttributeSet(keys) ++
  AttributeSet(child.output.filter(_.metadata.contains(EventTimeWatermark.delayKey))) {code}
 
 
change to 
{code:java}
override lazy val references: AttributeSet = AttributeSet(keys) ++
AttributeSet(child.output.filter(_.metadata.contains(EventTimeWatermark.delayKey))) {code}
Bug: Fix java.util.NoSuchElementException when event time column is dropped after dropDuplicatesWithinWatermark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/24 22:56
Affects Version/s: 3.5.0

Consider the following query:

```
val result = inputData.toDF()
.select("_1", "_2")
.withColumn("timestamp", to_timestamp($"_2", "yyyy-MM-dd HH:mm:ss"))
.withWatermark("timestamp", "24 hours")
.dropDuplicatesWithinWatermark("timestamp")
.select("_1")[]
```
 
Currently, the ColumnPruning optimization will prune the `timestamp` column since it is not selected in the final Project, leading to a `java.util.NoSuchElementException` when we try to get the event time column in DeduplicateWithinWatermarkExec.
 
We need to update the references for the DeduplicateWithinWatermark logical plan node so that the event time column is included in the references.
Bug: Support pushing DeserializeToObject through Union
Status: Open | Priority: Major | Resolution: 
Created: 05/Dec/24 20:28
Affects Version/s: 4.0.0

When migrating some RDD code to Dataset we noticed a significant performance degradation. The degradation was caused by an extra serialize/deserialize round trip.

Consider the following plan
{code:java}
val ds1 = spark.createDataset(Seq((1,2)))
val ds2 = sc.parallelize(Seq((1,2))).toDS().filter(_._1 > 1)
ds1.union(ds2).map(identity).explain(){code}
{code:java}
== Physical Plan ==
*(2) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#353, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#354]
+- *(2) MapElements $line43.$read$$iw$$Lambda$4181/0x00007f3d4d10ae10@24287e1f, obj#352: scala.Tuple2
   +- *(2) DeserializeToObject newInstance(class scala.Tuple2), obj#351: scala.Tuple2
      +- Union
         :- LocalTableScan [_1#318, _2#319]
         +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#326, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#327]
            +- *(1) Filter $line40.$read$$iw$$Lambda$4179/0x00007f3d4d10ecc8@d06af57.apply
               +- Scan[obj#325]{code}
if we pushed the DeserializeToObject through the Union we be able to cancel it against the SerializeFromObject and we could perform the union on the deserialized objects instead.
Bug: Deprecated no-op `spark.shuffle.spill` config
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Dec/24 18:29
Affects Version/s: 4.0.0

Previously, Spark shows a warning when `spark.shuffle.spill=false`. 

After this, Spark will show a deprecation warning additionally when `spark.shuffle.spill=true` is explicitly used.
{code:java}
24/12/03 10:36:34 WARN SparkConf: The configuration key 'spark.shuffle.spill' has been deprecated as of Spark 1.6 and may be removed in the future. Not used anymore. {code}
Bug: Block usage of collations for maps' key
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Dec/24 11:34
Affects Version/s: 3.5.4

Now it is possible to set collations on keys of map type. But it could lead to unexpected bugs related to uniqueness of keys, so after collation change existing data may become unreadable. For example, it happens when table is created with
{code:java}
map('A' -> 1, 'a' -> 2){code}
and then collation switched to UTF8_LCASE.

I propose to block collations on keys for now. Other systems rejects such changes as well.
Probably int he future we could discuss about blocking changing collation but not blocking creation tables with collated keys.
Bug: Improve warning logging in CacheManager
Status: Open | Priority: Minor | Resolution: 
Created: 20/Dec/24 18:25
Affects Version/s: 4.0.0

{{CacheManager}} currently logs warning when there is an attempt to add dataframe with the logical plan that already present in the cache. There is no warning message logged if there is an attempt to remove dataframe from the cache when the logical plan is not present in the cache. The request is to:
 # Add information about what logical plan is added/removed from the cache
 # Add missing warning message for the above case

While there is an ability to enable detailed debug logging for the {{CacheManager}}, such debug logging is not enabled by default as it logs large amount of data.

Consider the following code that leads to memory leaks:

{noformat}
Dataset<Row> dataset = ...
Dataset<Row> dataset1 = dataset.withColumn(...);
Dataset<Row> dataset2 = dataset1.withColumn(...);
dataset.persist(); // OK
dataset1.persist(); // OK
dataset.persist(); // currently logs warning without logical plan details
dataset.unpersist(); // OK
dataset.unpersist(); // no warning
dataset2.unpersist(); // no warning, the actual call should be on dataset1
{noformat}
Bug: Add quote builtin function
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Dec/24 20:09
Affects Version/s: 4.1.0

Hive 4.0.0 introduced a new builtin function, quote.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27362046#LanguageManualUDF-StringFunctions

For interoperability, it might be nice to introduce the function to Spark SQL.
Bug: Add File Extension Option to CSV DataSource Writer
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Dec/24 22:01
Affects Version/s: 3.5.3

h3. What changes were proposed in this pull request?

The existing CSV DataSource allows one to set the delimiter/separator but does not allow the changing of the file extension. This means that a file can have values separated by tabs but me marked as a ".csv" file. This change allows one to change the file extension to match the delimiter/separator (e.g. ".tsv" for a tab separated value file).

PR: [https://github.com/apache/spark/pull/49233]
h3. Why are the changes needed?

This PR adds an additional option to set the fileExtension. The end result is that when a separator is set that is not a comma that the output file has a file extension that matches the separator (e.g. file.tsv, file.psv, etc...).

Notes on Previous Pull Request [#17973|https://github.com/apache/spark/pull/17973]
A pull request adding this option was discussed 7 years ago. One reason it wasn't added was:
"I would like to suggest to leave this out if there is no better reason for now. Downside of this is, it looks this allows arbitrary name and it does not gurantee the extention is, say, tsv when the delmiter is a tab. It is purely up to the user."

I don't believe this is a good reason to not let the user set the extension. If we let them set the delimiter/separator to an arbitrary string/char then why not let the user also set the file extension to specify the separator that the file uses (e.g. tsv, psv, etc...). This addition keeps the "csv" file extension as the default and has the benefit of allowing other separators to match the file extension.
h3. Does this PR introduce _any_ user-facing change?

Yes. This PR adds one row to the options table for the CSV DataSource documentation to include the "fileExtension" option.
h3. How was this patch tested?

One unit test was added to validate a file is written with the new extension.
h3. Was this patch authored or co-authored using generative AI tooling?

No
Bug: Error compiling spark-protobuf module using user-defined protoc
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Dec/24 05:58
Affects Version/s: 4.0.0

I'm using user-defined protoc since default version of protoc and protoc-gen-grpc-java do not work well on my macOS 11.
 
Unfortunately, this results in below error when compiling spark-protobuf module:
 
{code:java}
$ export SPARK_PROTOC_EXEC_PATH=/Users/foobar/dev/protoc-4.29.1/protoc
$ export CONNECT_PLUGIN_EXEC_PATH=/Users/foobar/dev/protoc-gen-grpc-java-1.59.1/protoc-gen-grpc-java
$ ./build/mvn -DskipTests clean package -Puser-defined-protoc -DskipDefaultProtoc
 ...
[ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (default) on project spark-protobuf_2.13: Execution default of goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run failed: Cannot read the array length because "<local6>" is null{code}
Bug: Provide spark docker images based on ubi-minimal
Status: Open | Priority: Minor | Resolution: 
Created: 11/Dec/24 14:30
Affects Version/s: 3.5.4

Assuming it doesn't interfere with any existing requirement from spark/spark-docker itself, it would be beneficial to offer spark-docker images based on ubi-minimal (e.g. `eclipse-temurin:17-jdk-ubi9-minimal`.

The ubi-minimal images have 0 vulnerabilities in docker-hub, as opposed to the ubuntu-based images, which have 4-5 medium and 10-21 low known vulnerabilities.

If not instead of the current images, these could be created and published in tandem with them.
Bug: Local spark under scalatest hangs writing to local disk
Status: Open | Priority: Minor | Resolution: 
Created: 20/Dec/24 00:08
Affects Version/s: 4.0.0

This may be nothing. I apologize if it is.

Testing out spark 4.0.0 preview 2 and finding an issue with a local integration test writing to local disk.

Env:
scala 2.13.15
jdk 17.0.11 (Azul) 
sbt 1.10.6

I tried jdk 21 as well; no love either.

build.sbt dependencies:
{code:scala}
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % Provided,
  "org.apache.spark" %% "spark-sql" % sparkVersion  % Provided,
  "org.scalatest" %% "scalatest" % scalatestVersion % Test
)
{code}
Here's my test case:
{code:scala}
test("this is a test") {
    val spark: SparkSession = SparkSession
        .builder()
        .master("local[*]")
        .config("spark.executor.memory", "1g") // Adjust as needed
        .config("spark.driver.memory", "1g")
        .getOrCreate()

    import spark.implicits._
    spark.sparkContext.setLogLevel("TRACE")

    val x = List(
      (1, 2L,"a", "2022-01-01"),
      (1, 2L, "b", "2022-01-02"),
      (1, 2L, "c", "2022-01-03")
    ).toDF("a", "b", "c", "dt")
    x.show(10)


    x.write
      .mode("append")
      .csv("/tmp/sandbox")
  }
{code}
It hangs. No job or stage in the UI. No activity in the console. TRACE loglevel just gives me this:
{code:java}
{"ts":"2024-12-19T23:49:48.764Z","level":"TRACE","msg":"Checking for hosts with no recent heartbeats in HeartbeatReceiver.","logger":"HeartbeatReceiver"}
{code}
I've confirmed the path is writeable and even at TRACE loglevel I can't seem to get it to write and no clue why.

When I ctrl-c I get some spam afterwards from scalatest.

Maybe I'm daft, it's probable. Spark 3.5 and lower I've had no issues with this, so it's quite a bit more likely that it's me.