av9ash/GitBugs huggingface dataset:
Bug: Add option snapshotStartBatchId and snapshotPartitionId to state data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/24 20:14
Affects Version/s: 4.0.0

Define two new options, _snapshotStartBatchId_ and _snapshotPartitionId_, for the existing state reader. Both of them should be provided at the same time.
 # When there is no snapshot file at that batch (note there is an off-by-one issue between version and batch Id), throw an exception.
 # Otherwise, the reader should continue to rebuild the state by reading delta files only, and ignore all snapshot files afterwards.
 # Note that if a batchId option is already specified. That batchId is the ending batchId, we should then end at that batchId.
Bug: Performance Improvement for BIN function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/24 07:07
Affects Version/s: 4.0.0

{code:java}
--- a/sql/core/benchmarks/MathFunctionBenchmark-results.txt
+++ b/sql/core/benchmarks/MathFunctionBenchmark-results.txt
@@ -2,5 +2,5 @@ OpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
 Apple M2 Max
 encode:                                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-BIN                                                2657           2661           5          3.8         265.7       1.0X
+BIN                                                1524           1567          61          6.6         152.4       1.0X {code}
Bug: Make `IndexShuffleBlockResolver` have explicit constructors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/24 20:49
Affects Version/s: 4.0.0

Although `IndexShuffleBlockResolver` is `private` and there is no contract to keep class constructor signitures, we had better reduce the following situations with the old libraries built against old Spark versions.

{quote}Cause: java.lang.NoSuchMethodError: 'void org.apache.spark.shuffle.IndexShuffleBlockResolver.<init>(org.apache.spark.SparkConf, org.apache.spark.storage.BlockManager)'
     [info]   at org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager.<init>(CometShuffleManager.scala:64)
{quote}
Bug: Remove lock acquisition in doMaintenance() by making a deep copy of file mappings in RocksDBFileManager in load()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/24 17:39
Affects Version/s: 3.4.3

Currently the lock of the *RocksDB* state store is acquired when uploading the snapshot inside maintenance tasks when change log checkpointing is enabled, which causes lock contention between query processing tasks and state maintenance thread. We want to eliminate lock contention to decrease latency of streaming queries so lock acquisition inside maintenance tasks should be avoided. This can introduce race conditions between task and maintenance threads. By making a deep copy of {{versionToRocksDBFiles}} and {{localFilesToDfsFiles}} in {*}RocksDBFileManager{*}, we can ensure that the file manager state is not updated by task thread when background snapshot uploading tasks attempt to upload a snapshot.
Bug: Log entry should be constructed only once
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/24 22:35
Affects Version/s: 4.0.0

In the current implementation:

 
{code:java}
class LogEntry(messageWithContext: => MessageWithContext) {
  def message: String = messageWithContext.message
  def context: java.util.HashMap[String, String] = messageWithContext.context
}

def logInfo(entry: LogEntry): Unit = {
  if (log.isInfoEnabled) {
    withLogContext(entry.context) {
      log.info(entry.message)
    }
  }
}{code}
 

 

The field `messageWithContext` is constructed twice, one from `entry.context` and another one from `entry.message`.

We should improve this.
Bug: Create to_avro and from_avro SQL functions to match PySpark equivalent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/24 21:54
Affects Version/s: 4.0.0

The PySpark API is here: https://github.com/apache/spark/blob/d5c33c6bfb5757b243fc8e1734daeaa4fe3b9b32/python/pyspark/sql/avro/functions.py#L35
Bug: Wrong FS error when InsertIntoHadoopFSCommand with HadoopMapreduceCommitterProtocol when table path and partition path are in different FS
Status: Open | Priority: Major | Resolution: 
Created: 27/Jun/24 18:48
Affects Version/s: 3.4.3

If table path and partition path are in different FS (for example different gcs bucket), the staging directory is under the table path, HadoopMapreduceCommitProtocol create fs object using this staging path, so the operation on the partition paths will fail with 'Wrong FS' error.
Bug: Remove three unnecessary type registrations from KryoSerializer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 05:59
Affects Version/s: 4.0.0

The types `None`, `Nil`, `cala.collection.immutable.$colon$colon` are already registered in [https://github.com/twitter/chill/blame/0a34e50f742ab74598c81f153cfe8ad90bf3a859/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L238-L241], so there is no need to register them again in Spark.
Bug: Explicitly use worker_main when it starts with pyspark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 02:30
Affects Version/s: 4.0.0

After SPARK-44380, Python worker, the provided arguments must be used as a worker main. It might break other cases such as https://github.com/getsentry/sentry-python/blob/master/sentry_sdk/integrations/spark/spark_worker.py
Bug: Python UDF in higher order functions should not throw internal error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 03:15
Affects Version/s: 4.0.0

{code}
from pyspark.sql.functions import transform, udf, col, array
spark.range(1).select(transform(array("id"), lambda x: udf(lambda y: y)(x))).collect()
{code}

throws an internal error:

{code}
	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotEvaluateExpressionError(QueryExecutionErrors.scala:73)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval(Expression.scala:507)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval$(Expression.scala:506)
{code}
Bug: Got incastable error when deserializer in cogroup is resolved during application of DeduplicateRelation rule
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/24 00:52
Affects Version/s: 4.0.0

When running the following commands:

 

 
{code:java}
    val lhs = spark.createDataFrame(
      List(Row(123L)).asJava,
      StructType(Seq(StructField("GROUPING_KEY", LongType)))
    )
    val rhs = spark.createDataFrame(
      List(Row(0L, 123L)).asJava,
      StructType(Seq(StructField("ID", LongType), StructField("GROUPING_KEY", LongType)))
    )    val lhsKV = lhs.groupByKey((r: Row) => r.getAs[Long]("GROUPING_KEY"))
    val rhsKV = rhs.groupByKey((r: Row) => r.getAs[Long]("GROUPING_KEY"))
    val cogrouped = lhsKV.cogroup(rhsKV)(
      (a: Long, b: Iterator[Row], c: Iterator[Row]) => Iterator(0L)
    )
    val joined = rhs.join(cogrouped, col("ID") === col("value"), "left") {code}
 

 

It gets an error:
{code:java}
java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.objects.AssertNotNull cannot be cast to org.apache.spark.sql.catalyst.analysis.UnresolvedDeserializer {code}
Bug: Improve performance of ParserUtils.unescapeSQLString
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/24 22:03
Affects Version/s: 4.0.0

The `ParserUtils.unescapeSQLString` method is currently implemented using regexes for part of the parsing, but is costly because multiple Matcher instances are created for every input cahracter.

I think that we should optimize this to remove the use of regexes.

I will submit a PR for this.
Bug: RocksDB setLevelCompactionDynamicLevelBytes should be set to false
Status: Open | Priority: Major | Resolution: 
Created: 25/Jun/24 05:03
Affects Version/s: 4.0.0

In RocksDB 8.4.0, they changed the default of `level_compaction_dynamic_level_bytes` to true. This means that any query that runs with this enabled will have an L6 in the LSM tree, so if a user rolls back to a previous version of Spark that doesn't have RocksDB 8.4.0, then their L6 of the LSM tree will not be compacted. We should set this conf to be false so that this rolling back works in Spark.
Bug: Improve performance of SparkConf.get(ConfigEntry)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/24 23:46
Affects Version/s: 4.0.0

While analyzing some performance profiling results, I spotted some unexpected hotspots inside of configuration readers. I think that we can optimize the performance for the typical case by adding fast-paths to short circuit costly regexes and collection operations in certain cases. I will submit a PR for this.
Bug: Reading lengthy JSON results in a corrupted record.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/24 00:28
Affects Version/s: 3.5.1

When reading a data frame from a JSON file including a very long string, spark will incorrectly make it a corrupted record even though the format is correct. Here is a minimal example with PySpark:


{{import json}}
{{import tempfile}}
{{{}from pyspark.sql import SparkSession{}}}{{{}# Create a Spark session{}}}
{{spark = (SparkSession.builde}}
{{    .appName("PySpark JSON Example")}}
{{    .getOrCreate()}}
{{{}){}}}{{{}# Define the JSON content{}}}
{{data = {}}
{{    "text": "a" * 100000000}}
{{{}}{}}}{{{}# Create a temporary file{}}}
{{with tempfile.NamedTemporaryFile(delete=False, suffix=".json", mode="w") as tmp_file:}}
{{    # Write the JSON content to the temporary file}}
{{    tmp_file.write(json.dumps(data) + "\n")}}
{{    tmp_file_path = tmp_file.name}}{{    # Load the JSON file into a PySpark DataFrame}}
{{    df = spark.read.json(tmp_file_path)}}{{    # Print the schema}}
{{    print(df)}}
Bug: PySpark MinHashLSH when used with CountVectorizer doesn't meet requirements
Status: Open | Priority: Major | Resolution: 
Created: 21/Jun/24 21:40
Affects Version/s: 3.5.1

I'm facing an issue when trying to use the MinHashLSH model, where the model is complaining about having only zero values in some rows although I do apply a filter before using the model. Here is a sample code to demonstrate using pyspark:


```python
@F.udf(returnType=types.BooleanType())
def is_non_zero_vector(vector: SparseVector) -> bool:
"""
Returns True if the vector has at least one non zero element
"""
return vector.numNonzeros() > 0
 
df_text = df.select("id", "text")

tokenizer=Tokenizer(inputCol="text", outputCol="text_tokens")
df_text=tokenizer.transform(df_text).select("id", "text_tokens")

ngram=NGram(inputCol="text_tokens", outputCol="text_ngrams", n=self.min_hash_lsh_ngram_size)
df_text=ngram.transform(df_text).select("id", "text_ngrams")

count_vectorizer=CountVectorizer(inputCol="text_ngrams", outputCol="text_count_vector").fit(df_text)
df_text=count_vectorizer.transform(df_text).select("id", "text_count_vector")

# Keep only the non zero vectors
df_text=df_text.filter(is_non_zero_vector(F.col("text_count_vector")))

min_hash_lsh=MinHashLSH(
inputCol="text_count_vector",
outputCol="text_hashes",
seed=self.min_hash_lsh_num_hash_tables,
numHashTables=self.min_hash_lsh_num_hash_tables,
).fit(df_text)
df_text=min_hash_lsh.transform(df_text)

# Calculate the distance between all pairs of vectors and keep only the pairs with a distance > 0 (that are duplicates)
pairs_df=min_hash_lsh.approxSimilarityJoin(df_text, df_text, 0.6, distCol="vector_distance")
pairs_df=pairs_df.filter("vector_distance != 0")

```

I've also analyzed the dataframe and there is in fact no rows without at least 1 non-zero index.
Bug: Cache table doesn't work with collated column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/24 14:52
Affects Version/s: 4.0.0

Following sequence of queries produces the error:
{code:java}
>  cache lazy table t as select col from values ('a' collate utf8_lcase) as (col);
> select col from t;
org.apache.spark.SparkException: not support type: org.apache.spark.sql.types.StringType@1.
        at org.apache.spark.sql.errors.QueryExecutionErrors$.notSupportTypeError(QueryExecutionErrors.scala:1069)
        at org.apache.spark.sql.execution.columnar.ColumnBuilder$.apply(ColumnBuilder.scala:200)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.$anonfun$next$1(InMemoryRelation.scala:85)
        at scala.collection.immutable.List.map(List.scala:247)
        at scala.collection.immutable.List.map(List.scala:79)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.next(InMemoryRelation.scala:84)
        at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.next(InMemoryRelation.scala:82)
        at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anon$2.next(InMemoryRelation.scala:296)
        at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anon$2.next(InMemoryRelation.scala:293)
... {code}
This is also the problem on non-lazy cached tables.
Bug: ArrayIndexOutOfBoundsException in CartesianRDD getPartitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/24 00:12
Affects Version/s: 4.0.0

```val rdd1 = spark.sparkContext.parallelize(Seq(1, 2, 3), numSlices = 65536)
val rdd2 = spark.sparkContext.parallelize(Seq(1, 2, 3), numSlices = 65536)rdd2.cartesian(rdd1).partitions```

Throws `ArrayIndexOutOfBoundsException: 0` at CartesianRDD.scala:69 because `s1.index * numPartitionsInRdd2 + s2.index` overflows and wraps to 0. We should provide a better error message which indicates the number of partition overflows so it's easier for the user to debug.
Bug: Assign specific exit code for executors killed by TaskReaper
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/24 14:25
Affects Version/s: 4.0.0

When TaskReaper kills an executor, the executor loss reason would be "Command exited with code 50", which is the exit code for default uncaught exception.

We would like to have a specific exit code for executors killed by TaskReaper, so that we can better monitor these cases.
Bug: Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/24 04:48
Affects Version/s: 4.0.0

When having a table directory with invalid partitions such as:
{code:java}
table/
  invalid/...
  part=1/...
  part=2/...
  part=3/...{code}
a SQL query reading all of the partitions would fail with 
{code:java}
java.lang.AssertionError: assertion failed: Conflicting directory structures detected. Suspicious paths: 
 table 
 table/invalid {code}
 

I propose to add a data source option and Spark SQL config to ignore invalid partition paths. The config will be disabled by default to retain the current behaviour.
{code:java}
spark.conf.set("spark.sql.files.ignoreInvalidPartitionPaths", "true"){code}
{code:java}
spark.read.format("parquet").option("ignoreInvalidPartitionPaths", "true").load(...)  {code}
Bug: Fix test case "error during accessing host local dirs for executors"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/24 12:49
Affects Version/s: 4.0.0

There is a logical error in test case "error during accessing host local dirs for executors" in ShuffleBlockFetcherIteratorSuite.

It tries to test fetching host-local blocks, but the host-local BlockManagerId is configured incorrectly, and ShuffleBlockFetcherIterator will treat those blocks as remote blocks instead.
Bug: Fix support for StructTypes with collations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/24 15:04
Affects Version/s: 4.0.0

While adding expression walker it was noticed that StructType support is broken. One problem is that `CollationsTypeCasts` is doing a cast in all BinaryExpressions which includes ExtractValue. Consequently, we are unable to extract value if we do a cast there, as ExtractValue only supports nonNullLiterals as extracting keys.
Bug: Perf improvement for parsing hex string
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/24 09:43
Affects Version/s: 4.0.0

{code:java}
================================================================================================
Hex Comparison
================================================================================================OpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
Apple M2 Max
Cardinality 1000000:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Apache                                             5050           5100          86          0.2        5050.1       1.0X
Spark                                              3822           3840          30          0.3        3821.6       1.3X
Java                                               2462           2522          87          0.4        2462.1       2.1XOpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
Apple M2 Max
Cardinality 2000000:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Apache                                            10020          10828        1154          0.2        5010.1       1.0X
Spark                                              6875           6966         144          0.3        3437.7       1.5X
Java                                               4999           5092          89          0.4        2499.3       2.0XOpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
Apple M2 Max
Cardinality 4000000:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Apache                                            20090          20433         433          0.2        5022.5       1.0X
Spark                                             13389          13620         229          0.3        3347.2       1.5X
Java                                              10023          10069          42          0.4        2505.6       2.0XOpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
Apple M2 Max
Cardinality 8000000:                      Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Apache                                            40277          43453        2755          0.2        5034.7       1.0X
Spark                                             27145          27380         311          0.3        3393.1       1.5X
Java                                              19980          21198        1473          0.4        2497.5       2.0X {code}
Bug: [Bug] Partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jun/24 19:18
Affects Version/s: 4.0.0

There is a bug that results in an internal error with some combination of the Python UDTF "select" and "partitionBy" options of the "analyze" method.

To reproduce:
{code:java}
from pyspark.sql.functions import (
    AnalyzeArgument,
    AnalyzeResult,
    PartitioningColumn,
    SelectedColumn,
    udtf
)

from pyspark.sql.types import (
    DoubleType,
    StringType,
    StructType,
)

@udtf
class TestTvf:
    @staticmethod
    def analyze(observed: AnalyzeArgument) -> AnalyzeResult:
        out_schema = StructType()
        out_schema.add("partition_col", StringType())
        out_schema.add("double_col", DoubleType())

        return AnalyzeResult(
            schema=out_schema,
            partitionBy=[PartitioningColumn("partition_col")],
            select=[
                SelectedColumn("partition_col"),
                SelectedColumn("double_col"),
            ],
        )

    def eval(self, *args, **kwargs):
        pass

    def terminate(self):
        for _ in range(10):
            yield {
                "partition_col": None,
                "double_col": 1.0,
            }


spark.udtf.register("serialize_test", TestTvf) 

# Fails
(
    spark
    .sql(
        """
        SELECT * FROM serialize_test(
            TABLE(
                SELECT
                    5 AS unused_col,
                    'hi' AS partition_col,
                    1.0 AS double_col
                
                UNION ALL

                SELECT
                    4 AS unused_col,
                    'hi' AS partition_col,
                    1.0 AS double_col
            )
        )
        """
    )
    .toPandas()
){code}
Bug: Support scalar subquery with group-by on column equal to constant
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/24 18:29
Affects Version/s: 4.0.0

For example, this query is currently unsupported but can be allowed, because the filter y2 = 1 guarantees that the group-by has only one value.
{code:java}
select *, (select count(*) from y where x1 = y1 and y2 = 1 group by y2) from x; {code}
Bug: Perf improvement for binary to HEX_DISCRETE strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/24 05:31
Affects Version/s: 4.0.0

{code:java}
+OpenJDK 64-Bit Server VM 17.0.10+0 on Mac OS X 14.5
+Apple M2 Max
+Cardinality 100000:                       Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+------------------------------------------------------------------------------------------------------------------------
+Spark                                             42210          43595        1207          0.0      422102.9       1.0X
+Java                                                238            243           2          0.4        2381.9     177.2X {code}
Bug: Avoid storage amplification when accessing sub-Variant
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/24 17:48
Affects Version/s: 4.0.0

When a variant_get expression returns a Variant, or a nested type containing Variant, we just return the sub-slice of the Variant value along with the full metadata, even though most of the metadata is probably unnecessary to represent the value. This may be very inefficient if the value is then written to disk (e.g. shuffle file or parquet). We should instead rebuild the value with minimal metadata.
Bug: Support Column type for several SQL functions in scala and python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/24 14:14
Affects Version/s: 3.5.1

Currently, several SQL functions accept both native types and Columns, but only accept native types in their scala/python APIs:

* array_remove (works in SQL, scala, not in python)
* array_position(works in SQL, scala, not in python)
* map_contains_key (works in SQL, scala, not in python)
* substring (works only in SQL)

For example, this is possible in SQL:

{code:python}
spark.sql("select array_remove(col1, col2) from values(array(1,2,3), 2)")
{code}

But not in python:
{code:python}
df.select(F.array_remove(F.col("col1"), F.col("col2"))
{code}