av9ash/GitBugs huggingface dataset:
Bug: Optimize complex type insertion performance
Status: Open | Priority: Major | Resolution: 
Created: 17/Jul/24 03:01
Affects Version/s: 3.5.0

During the upgrade from Spark 3.1.1 to 3.5.0, I found a performance regression in map type inserts.

There are some extra conversion expressions in project before insert, which doesn't seem to be always necessary.
{code:java}
map_from_arrays(transform(map_keys(map#516), lambdafunction(lambda key#652, lambda key#652, false)), transform(map_values(map#516), lambdafunction(lambda value#654, lambda value#654, false))) AS map#656 {code}
Bug: Upgrade the kubernetes-client dependency okhttp version 4.12
Status: Open | Priority: Major | Resolution: 
Created: 11/Jul/24 06:43
Affects Version/s: 3.5.2

Right now Fabric8 Kubernetes Client doesn't work with IPv6 based Kubernetes Clusters due to an issue in OkHttp [square/okhttp#5889|https://github.com/square/okhttp/pull/5889]. The issue is solved for OkHttp 4, but Fabric8 Kubernetes Client depends on OkHttp 3.x.

 

https://github.com/fabric8io/kubernetes-client/issues/2632

[https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md]
Bug: Support recursive reference of Avro schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/24 23:55
Affects Version/s: 4.0.0

Recursive reference denotes the case that the type of a field can be defined before in the parent nodes. A simple example is:
{code:java}
{
  "type": "record",
  "name": "LongList",
  "fields" : [
    {"name": "value", "type": "long"},
    {"name": "next", "type": ["null", "LongList"]}
  ]
}
{code}
This is written in Avro Schema DSL and represents a linked list data structure. Spark currently will throw an error on this schema. Many users used schema like this, so we should support it.
Bug: Accumulate task-level execution memory in stage and display in Spark UI
Status: Open | Priority: Major | Resolution: 
Created: 03/Jul/24 00:08
Affects Version/s: 4.0.0

As a follow for https://issues.apache.org/jira/browse/SPARK-48628, after we have task-level onheap/offheap execution memory metrics, we should accumulate them in stage and make them available in Spark UI.
Bug: Spark UI is showing wrong timing metrics for min and median time.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/24 09:05
Affects Version/s: 3.4.0

In Spark 3.4.0, there is inconsistency in Spark UI and Spark History server UI for timing metrics.

Spark UI view:

!image-2024-07-29-14-35-23-597.png!

Spark History server view:

!image-2024-07-29-14-34-46-506.png!
Bug: Use isolated JobArtifactSet for each spark session
Status: Open | Priority: Major | Resolution: 
Created: 10/Jul/24 11:38
Affects Version/s: 4.0.0

Consider multi user use same spark cluster with independent spark session, Kyuubi group session share level for example:
 # user may use different udf jar to register function, but may has same name udf in different udf jar
 # the valid udf class is always the class from the first load udf jar, this not meet users expections
Bug: Maintenance thread pool error should not cause the entire executor to crash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/24 22:55
Affects Version/s: 4.0.0

Today, it's possible for an exception within a thread in the maintenance pool to cause the entire executor to crash. Here's how:
 # An error occurs in a maintenance pool thread
 # It gets passed to the maintenance task thread, which `throw`s it
 # That gets caught by `onError`, which `.stop()`s the maintenance thread pool
 # If any of the maintenance pool threads are waiting on a lock, they will receive an `InterruptedException` (this happens if they are verifying if the their state store instance is active)
 # This `InterruptedException` is not caught, which is not `NonFatal`
 # This uncaught exception bubbles all the way to the `SparkUncaughtExceptionHandler`, causing the executor to exit

A fix that is better is to modify the maintenance thread pool to only `unload` providers that experience errors, not stop the entire thread pool.
Bug: Cleanup shuffle dependencies for all SQL executions
Status: Open | Priority: Major | Resolution: 
Created: 10/Jul/24 22:04
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-47764 added a shuffle cleanup mechanism that applies to a `QueryExecution`, but it only enables it for Spark Connect.

We'd like to enable this for all query executions via configuration. The configurations that were added in https://issues.apache.org/jira/browse/SPARK-47764 are generic (did not have "connect" in the name), so we hope we can just use the same config: `spark.sql.shuffleDependency.fileCleanup.enabled`. SPARK-47764 also added: `spark.sql.shuffleDependency.skipMigration.enabled`, and I don't understand in which cases it is useful yet, but given the interface as it is, it could also be done for all queries.
Bug: Inject Runtime Filter for multi-level join with function condition
Status: Open | Priority: Major | Resolution: 
Created: 09/Jul/24 03:55
Affects Version/s: 3.5.1

InjectRuntimeFilter is effective when there is a single join with function condition, but it will not be effective when there are multiple joins.

 

Add test sqls in `InjectRuntimeFilterSuite.test("Runtime bloom filter join: two joins")` to reproduce:
{code:java}
-- bf1 as creation side and inject runtime filter for bf3.
select * from bf1 join bf3 on bf3.c3 = if(bf1.c1 < 5, 5, bf1.c1) where bf1.a1 = 5

-- bf1 will not inject runtime filter for bf3.
select * from bf1 join bf2 on bf1.c1 = bf2.c2 join bf3 on bf3.c3 = if(bf1.c1 < 5, 5, bf1.c1) where bf1.a1 = 5{code}
Bug: Add support for interval types in the Variant spec
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/24 19:51
Affects Version/s: 4.0.0

The Variant spec does not have support for the [YearMonthIntervalType|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/YearMonthIntervalType.html] and [DayTimeIntervalType|https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/DayTimeIntervalType.html] data types. We aim to add support for these types in the variant spec.
Bug: Reduce Cloud Store List API cost for state store maintenance task
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/24 21:19
Affects Version/s: 3.4.3

Currently, during the state store maintenance process, we find which old version files of the RocksDB state store to delete by listing all existing snapshotted version files in the checkpoint directory every 1 minute by default. The frequent list calls in the cloud can result in high costs. To address this concern and reduce the cost associated with state store maintenance, we should aim to minimize the frequency of listing object stores inside the maintenance task. To minimize the frequency, we will try to accumulate versions to delete and only call list when the number of versions to delete reaches a configured threshold.
Bug: "Externally Managed Environment" error when building PySpark Docker image 
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/24 03:27
Affects Version/s: 3.5.1

When trying to build Docker image based on PySpark Dockerfile in Ubuntu 20.04, I got the following error:
{code}
#7 19.13 error: externally-managed-environment
#7 19.13
#7 19.13 × This environment is externally managed
#7 19.13 ╰─> To install Python packages system-wide, try apt install
#7 19.13     python3-xyz, where xyz is the package you are trying to
#7 19.13     install.
#7 19.13
#7 19.13     If you wish to install a non-Debian-packaged Python package,
#7 19.13     create a virtual environment using python3 -m venv path/to/venv.
#7 19.13     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
#7 19.13     sure you have python3-full installed.
#7 19.13
#7 19.13     If you wish to install a non-Debian packaged Python application,
#7 19.13     it may be easiest to use pipx install xyz, which will manage a
#7 19.13     virtual environment for you. Make sure you have pipx installed.
#7 19.13
#7 19.13     See /usr/share/doc/python3.12/README.venv for more information.
#7 19.13
#7 19.13 note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
#7 19.13 hint: See PEP 668 for the detailed specification.
#7 ERROR: process "/bin/sh -c apt-get update &&     apt install -y python3 python3-pip &&     rm -rf /usr/lib/python3.11/EXTERNALLY-MANAGED &&     pip3 install --upgrade pip setuptools &&     rm -rf /root/.cache && rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 1
{code}

Looking at the [Dockerfile|https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile], it does the following:
{code}
RUN apt-get update && \
    apt install -y python3 python3-pip && \
    pip3 install --upgrade pip setuptools && \
    # Removed the .cache to save space
    rm -rf /root/.cache && rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
{code}

If {{pip}} was installed by the system package manager, and then we are trying to overwrite it via {{pip3 install}}, the error could happen.

A simple solution would be to create a virtual environment first, install the latest pip there, and then update {{PATH}} to use that instead. 

Wonder if anyone else has encountered the same issue and whether it is a good idea to fix it.
Bug: Corrupt data from parquet scans
Status: Open | Priority: Major | Resolution: 
Created: 19/Jul/24 21:46
Affects Version/s: 3.5.0

Its very rare and non-deterministic but since Spark 3.5.0 we have started seeing a correctness bug in parquet scans when using the vectorized reader. 

We've noticed this on double type columns where occasionally small groups (typically 10s to 100s) of rows are replaced with crazy values like `-1.29996470e+029, 3.56717569e-184, 7.23323243e+307, -1.05929677e+045, -7.60562076e+240, -3.18088886e-064, 2.89435993e-116`. I think this is the result of interpreting uniform random bits as a double type. Most of my testing has been on an array of double type column but we have also seen it on un-nested plain double type columns. 

I've been testing this by adding a filter that should return zero results but will return non-zero if the parquet scan has problems. I've attached screenshots of this from the Spark UI. 

I did a `git bisect` and found that the problem starts with [https://github.com/apache/spark/pull/39950], but I haven't yet understood why. Its possible that this change is fine but it reveals a problem elsewhere? I did also notice  [https://github.com/apache/spark/pull/44853] which appears to be a different implementation of the same thing so maybe that could help. 

Its not a major problem by itself but another symptom appears to be that Parquet scan tasks fail at a rate of approximately 0.03% with errors like those in the attached `example_task_errors.txt`. If I revert [https://github.com/apache/spark/pull/39950] I get exactly 0 task failures on the same test. 

 

The problem seems to be a bit dependant on how the parquet files happen to be organised on blob storage so I don't yet have a reproduce that I can share that doesn't depend on private data. 

I tested on a pre-release 4.0.0 and the problem was still present.
Bug: Add SQL syntax in create/replace table to create an identity column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jul/24 17:54
Affects Version/s: 4.0.0

Add SQL support for creating identity columns. Identity Column syntax should be flexible such that users can specify 
 * whether identity values are always generated by the system
 * (optionally) the starting value of the column 
 * (optionally) the increment/step of the column

The SQL syntax support should also allow flexible ordering of the increment and starting values, as both variants are used in the wild by other systems (e.g. [PostgreSQL|https://www.postgresql.org/docs/current/sql-createsequence.html] [Oracle).|https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/CREATE-SEQUENCE.html#GUID-E9C78A8C-615A-4757-B2A8-5E6EFB130571] That is, we should allow both
{code:java}
START WITH <start> INCREMENT BY <step>{code}
and 
{code:java}
INCREMENT BY <step> START WITH <start>{code}
.

For example, we should be able to define
{code:java}
CREATE TABLE default.example (
  id LONG GENERATED ALWAYS AS IDENTITY,
  id2 LONG GENERATED BY DEFAULT START WITH 0 INCREMENT BY -10,
  id3 LONG GENERATED ALWAYS AS IDENTITY INCREMENT BY 2 START WITH -8,
  value LONG
)
{code}
This will enable defining identity columns in Spark SQL for data sources that support it.
Bug: toJSON produces wrong values if DecimalType information is lost in as[Product]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/24 18:39
Affects Version/s: 3.1.1

Consider this example:
{code:scala}
package com.jetbrains.jetstat.etl

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types.DecimalType

object A {
  case class Example(x: BigDecimal)

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .master("local[1]")
      .getOrCreate()

    import spark.implicits._

    val originalRaw = BigDecimal("123.456")
    val original = Example(originalRaw)

    val ds1 = spark.createDataset(Seq(original))
    val ds2 = ds1
      .withColumn("x", $"x" cast DecimalType(12, 6))

    val ds3 = ds2
      .as[Example]

    println(s"DS1: schema=${ds1.schema}, encoder.schema=${ds1.encoder.schema}")
    println(s"DS2: schema=${ds1.schema}, encoder.schema=${ds2.encoder.schema}")
    println(s"DS3: schema=${ds1.schema}, encoder.schema=${ds3.encoder.schema}")

    val json1 = ds1.toJSON.collect().head
    val json2 = ds2.toJSON.collect().head
    val json3 = ds3.toJSON.collect().head

    val collect1 = ds1.collect().head
    val collect2_ = ds2.collect().head
    val collect2 = collect2_.getDecimal(collect2_.fieldIndex("x"))
    val collect3 = ds3.collect().head

    println(s"Original: $original (scale = ${original.x.scale}, precision = ${original.x.precision})")
    println(s"Collect1: $collect1 (scale = ${collect1.x.scale}, precision = ${collect1.x.precision})")
    println(s"Collect2: $collect2 (scale = ${collect2.scale}, precision = ${collect2.precision})")
    println(s"Collect3: $collect3 (scale = ${collect3.x.scale}, precision = ${collect3.x.precision})")
    println(s"json1: $json1")
    println(s"json2: $json2")
    println(s"json3: $json3")
  }
}
{code}
Running it you'd see that json3 contains very much wrong data. After a bit of debugging, and sorry since I'm bad with Spark internals, I've found that:
 * In-memory representation of the data in this example used {{UnsafeRow}}, whose {{.getDecimal}} uses compression to store small Decimal values as longs, but doesn't remember decimal sizing parameters,
 * However, there are at least two sources for precision & scale to pass to that method: {{Dataset.schema}} (which is based on query execution, always contains 38,18 for me) and {{Dataset.encoder.schema}} (that gets updated in `ds2` to 12,6 but then is reset in `ds3`). Also, there is a {{Dataset.deserializer}} that seems to be combining those two non-trivially.
 * This doesn't seem to affect {{Dataset.collect()}} methods since they use {{deserializer}}, but {{Dataset.toJSON}} only uses the first schema.

Seems to me that either {{.toJSON}} should be more aware of what's going on or {{.as[]}} should be doing something else.
Bug: Purge OperatorStateMetadataV2 files and StateSchemaV3 files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/24 16:33
Affects Version/s: 4.0.0

Currently, OperatorStateMetadataV2 and StateSchemaV3 files are written for every new query run. We need to purge these files so state files don't build up indefinitely.
Bug: Enabling deleteIfExists operator in TWS with Virtual Column Families
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/24 21:17
Affects Version/s: 4.0.0

Fully integrating the TransformWithState operator with Virtual Column Families by adding support for the deleteIfExists operator. In RocksDB.scala, return the columnFamilyId if it exists in {{removeColFamilyIfExists}} so that we can remove all the keys for the corresponding columnFamily once deleteIfExists is called.
Bug: Kafka Offsets Not Committed to Consumer Group in Spark Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 31/Jul/24 03:30
Affects Version/s: 3.5.1

h3. Summary

Unable to commit Kafka offsets back to Kafka consumer group when using Spark Structured Streaming.
h3. Description

I am using Spark Structured Streaming to read data from a Kafka topic, process it, and write the results to the console. While the streaming job reads and processes data from Kafka correctly, it fails to commit the offsets back to the Kafka consumer group. As a result, Kafka does not recognize where the consumer left off, causing potential reprocessing of messages.
h3. Configuration

Here is the relevant part of my Spark application configuration:
{code:java}
spark = (
        SparkSession.builder
        .appName("Test-KafkaSparkStructuredStreaming")
        .config("spark.ui.enabled", "true")
        .config("spark.ui.port", test_port)
        .config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1")
        .config("spark.dynamicAllocation.enabled", "true")
        .config("spark.dynamicAllocation.minExecutors", "1")
        .config("spark.dynamicAllocation.maxExecutors", "1")
        .config("spark.kafka.consumer.commit.offsets.on.checkpoint", "true")
        .getOrCreate()
    )

df = (
        spark.readStream.format("kafka")
        .option("kafka.bootstrap.servers", kafka_bootstrap_servers)
        .option("subscribe", kafka_topic)
        .option("auto.offset.reset", "earliest")
        .option("group.id", "Test_group")
        .load()
    )

# Spark Streaming Process code ...

query = (
        result_df.writeStream.outputMode("append")
        .format("console")
        .option(
            "checkpointLocation",
            "spark-checkpoint",
        )
        .start()
    )

query.awaitTermination()

spark.stop(){code}
h3. Steps to Reproduce
 # Set up a Kafka topic with data.
 # Configure a Spark Structured Streaming job to read from the Kafka topic using the above configuration.
 # Start the streaming job.
 # Observe that data is read and processed correctly, but offsets are not committed back to Kafka.

h3. Expected Behavior

Kafka offsets should be committed back to the Kafka consumer group, allowing Kafka to track the last consumed offsets.
h3. Actual Behavior

Kafka offsets are not committed back to the Kafka consumer group. As a result, Kafka does not recognize where the consumer left off, potentially causing reprocessing of messages.
Bug: Implement IndentingXMLStreamWriter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jul/24 18:38
Affects Version/s: 4.0.0

SPARK-44790 added a usage of `com.sun.xml.txw2.output.IndentingXMLStreamWriter` to Apache Spark repository.

This issue aims to investigate the workaround to remove usages.
Bug: Improve error message with invalid unresolved column reference in UDTF call
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/24 18:53
Affects Version/s: 4.0.0

This bug covers improving an error message in the event of invalid UDTF calls. For example:

{{select * from udtf(}}
{{  observed => TABLE(select column from t),}}
{{  value_col => classic_dollars}}
{{)}}

Currently we get:

{{Unsupported subquery expression: Table arguments are used in a function where they are not supported:}}
{{'UnresolvedTableValuedFunction [udtf], observed => table-argument#68918 [], value_col => 'classic_dollars, false}}
{{   +- Project ...}}
{{       +- SubqueryAlias ...}}
{{          +- Relation ...}}

But the real error is that the user passed column identifier classic_dollars rather than string "classic_dollars" into the string argument.

The core reason is that {{CheckAnalysis}} checks the analyzer output tree for unresolved nodes at the end rather than reporting the error at the time of attempted resolving of the corresponding expression or operator (in this case, the unresolved attribute reference resulting from the unquoted {{{}classic_dollars{}}}).
Bug: Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/24 19:18
Affects Version/s: 3.4.4

Currently very large "INSERT INTO ... VALUES" statements result into disproportionally large parse trees as each literal will need to remain in the parse tree, until it eventually gets evaluated into a LocalTable, once the appropriate analyzer/optimizer rules have been applied.

 

This results in increased memory pressure on the driver, when such large statements are generated, that can lead to OOMs and GC pauses. It also results in suboptimal runtime performance as the time it takes to apply analyzer/optimizer rules is typically proportional to the size of the parse tree.

 

Both these issues can be resolved by applying the functions that evaluate the unresolved table into a local table eagerly from the AST Builder, thus short-circuiting the evaluation of such statements.
Bug: Infinite loop with GlobalLimit/BindParameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/24 10:45
Affects Version/s: 3.5.0

When using Spark connect and with the following snippet:



{{{}df = spark.sql("SELECT * FROM VALUES (TIMESTAMP('2022-12-25 10:30:00'), 1) as tab(date, val) where val = :val", args = \{'val': 1}){}}}{{{}df.first().collect(){}}}

 

Spark enters the infinite loop in the parser.