av9ash/GitBugs huggingface dataset:
Bug: Fix Spark Connect collation issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/24 10:17
Affects Version/s: 4.0.0

Collated expression "SELECT 'abc' COLLATE 'UCS_BASIC_LCASE'" is failing when connecting to sever using Spark Connect:
{code:java}
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (org.apache.spark.sql.connect.common.InvalidPlanInput) Does not support convert string(UCS_BASIC_LCASE) to connect proto types.{code}
When using default collation "UCS_BASIC", the error is not occurring.
Bug: Fix `core` module to succeed SBT tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/24 21:23
Affects Version/s: 3.4.0

This happens at branch-3.4 only. branch-3.3/branch-3.5/master are okay.
{code:java}
$ build/sbt "core/testOnly *.DAGSchedulerSuite"
[info] DAGSchedulerSuite:
[info] - [SPARK-3353] parent stage should have lower stage id *** FAILED *** (439 milliseconds)
[info]   java.lang.IllegalStateException: Could not initialize plugin: interface org.mockito.plugins.MockMaker (alternate: null)
...
[info] *** 1 SUITE ABORTED ***
[info] *** 118 TESTS FAILED ***
[error] Error during tests:
[error] 	org.apache.spark.scheduler.DAGSchedulerSuite
[error] (core / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 48 s, completed Feb 27, 2024, 1:26:27 PM {code}
 

MAVEN
{code:java}
$ build/mvn dependency:tree -pl core | grep byte-buddy
...
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.12.10:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.12.10:test
{code}
SBT
{code:java}
$ build/sbt "core/test:dependencyTree" | grep byte-buddy
[info]   | | | | +-net.bytebuddy:byte-buddy:1.12.10 (evicted by: 1.12.18)
[info]   | | | | +-net.bytebuddy:byte-buddy:1.12.18
{code}
Bug: Preformance issue on thrift API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/24 08:10
Affects Version/s: 3.4.1

This new complexity was introduced in SPARK-39041.

before this PR the code was:
{code:java}
while (curRow < maxRows && iter.hasNext) {
  val sparkRow = iter.next()
  val row = ArrayBuffer[Any]()
  var curCol = 0
  while (curCol < sparkRow.length) {
    if (sparkRow.isNullAt(curCol)) {
      row += null
    } else {
      addNonNullColumnValue(sparkRow, row, curCol, timeFormatters)
    }
    curCol += 1
  }
  resultRowSet.addRow(row.toArray.asInstanceOf[Array[Object]])
  curRow += 1
}{code}
 foreach without the _*O(n^2)*_ complexity so this change just return the state to what it was before.

 

In class `RowSetUtils` there is a loop that has _*O(n^2)*_ complexity:
{code:scala}
...
 while (i < rowSize) {
          val row = rows(I)
          ...
{code}
It can be easily converted back into _*O( n )*_ complexity.
Bug: Introduce abstraction that facilitates more flexible management of file listings within the system.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/24 00:11
Affects Version/s: 3.5.0

The introduction of these constructs is crucial for defining a standardized API for file listing operations, regardless of the underlying representation that's used to represent files and partitions. By introducing said abstractions we improve the modularity of the code and enable future improvements that can prove to be beneficial both for runtime and memory improvements.
Bug: Fix `kafka-0-10-sql` to use `ResetSystemProperties` if `KafkaTestUtils` is used
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/24 10:58
Affects Version/s: 3.0.0



https://github.com/apache/spark/blob/ee312ecb40ea5b5303fc794a3d494b6f27cda923/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala#L290
Bug: Upgrade `aircompressor` to 0.26
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Feb/24 17:05
Affects Version/s: 3.5.0

`aircompressor` is a transitive dependency from Apache ORC and Parquet.

`aircompressor` v0.26 reported the following bug fixes recently.
 - [Fix out of bounds read/write in Snappy decompressor]([https://github.com/airlift/aircompressor/commit/b89db180bb97debe025b640dc40ed43816e8c7d2])
 - [Fix ZstdOutputStream corruption on double close]([https://github.com/airlift/aircompressor/commit/b89db180bb97debe025b640dc40ed43816e8c7d2])
Bug: Use larger memory for Maven builds
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 21/Feb/24 06:42
Affects Version/s: 4.0.0

{code}
*** RUN ABORTED ***
An exception or error caused a run to abort: unable to create native thread: possibly out of memory or process/resource limits reached 
  java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
  at java.base/java.lang.Thread.start0(Native Method)
  at java.base/java.lang.Thread.start(Thread.java:1553)
  at java.base/java.lang.System$2.start(System.java:2577)
  at java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
  at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
  at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
  at org.apache.spark.rpc.netty.SharedMessageLoop.$anonfun$threadpool$1(MessageLoop.scala:128)
  at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
  at org.apache.spark.rpc.netty.SharedMessageLoop.<init>(MessageLoop.scala:127)
  at org.apache.spark.rpc.netty.Dispatcher.sharedLoop$lzycompute(Dispatcher.scala:46)
  ...
Warning:  The requested profile "volcano" could not be activated because it does not exist.
Warning:  The requested profile "hive" could not be activated because it does not exist.
Error:  Failed to execute goal org.scalatest:scalatest-maven-plugin:2.2.0:test (test) on project spark-core_2.13: There are test failures -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :spark-core_2.13
Error: Process completed with exit code 1.
{code}

https://github.com/HyukjinKwon/spark/actions/runs/7984135094/job/21800463337
Bug: RocksDB versionID Mismatch in SST files with Compaction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/24 21:06
Affects Version/s: 3.5.0

RocksDB compaction can result in version Id mismatch errors if the same version is committed twice from the same executor. (Multiple commits can happen due to Spark Stage/task retry).

A particular scenario where this can happen is provided below: 
 # Version V1 is loaded on executor A, RocksDB State Store has 195.sst, 196.sst, 197.sst and 198.sst files. 
2. State changes are made, which result in creation of a new table file 200.sst. 
3. State store is committed as version V2. The SST file 200.sst (as 000200-8c80161a-bc23-4e3b-b175-cffe38e427c7.sst) is uploaded to DFS, and previous 4 files are reused. A new metadata file is created to track the exact SST files with unique IDs, and uploaded with RocksDB Manifest as part of V1.zip.
4. Rocks DB compaction is triggered at the same time. The compaction creates a new L1 file (201.sst), and deletes existing 5 SST files.
5. Spark Stage is retried. 
6. Version V1 is reloaded on the same executor. The local files are inspected, and 201.sst is deleted. The 4 SST files in version V1 are downloaded again to local file system. 
7. Any local files which are deleted (as part of version load) are also removed from local → DFS file upload tracking. **However, the files already deleted as a result of compaction are not removed from tracking. This is the bug which resulted in the failure.**
8. State store is committed as version V1. However, the local mapping of SST files to DFS file path still has 200.sst in its tracking, hence the SST file is not re-uploaded. A new metadata file is created to track the exact SST files with unique IDs, and uploaded with the new RocksDB Manifest as part of V2.zip. (The V2.zip file is overwritten here atomically)
9. A new executor tried to load version V2. However, the SST files in (1) are now incompatible with Manifest file in (6) resulting in the version Id mismatch failure.

 

We need to ensure that any files deleted from local filesystem post compaction are not tracked in uploadedDFSFiles mapping if the same version is loaded again.
Bug: Avoid noisy RejectedExecutionExceptions during StandaloneSchedulerBackend shutdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 19:55
Affects Version/s: 3.5.0

While it is in the process of shutting down, the StandaloneSchedulerBackend might throw RejectedExecutionExceptions when RPC handler `onDisconnected` methods attempt to submit new tasks to a stopped executorDelayRemoveThread executor service. We can reduce log and uncaught exception noise by catching and ignoring these exceptions if they occur during shudown.
Bug: Install proper Python version in SparkR Windows build to avoid warnings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 07:32
Affects Version/s: 4.0.0

https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830

{code}
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\__init__.py", line [53](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:54), in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\rdd.py", line [54](https://github.com/HyukjinKwon/spark/actions/runs/7985005685/job/21802732830#step:10:55), in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\java_gateway.py", line 33, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\serializers.py", line 69, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\__init__.py", line 1, in <module>
  File "D:\a\spark\spark\python\lib\pyspark.zip\pyspark\cloudpickle\cloudpickle.py", line 80, in <module>
ImportError: cannot import name 'CellType' from 'types' (C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\types.py)
{code}
Bug: Uses proper options for command script in macos-14 build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/24 23:57
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/7964626045/job/21742573399

It fails as below:

{code}
Run # Fix for TTY related issues when launching the Ammonite REPL in tests.
/usr/bin/script: illegal option -- c
usage: script [-aeFkpqr] [-t time] [file [command ...]]
       script -p [-deq] [-T fmt] [file]
Error: Process completed with exit code 1.
{code}

See https://man.freebsd.org/cgi/man.cgi?script(1)
https://man7.org/linux/man-pages/man1/script.1.html
Bug: Separate state tracking variables from MicroBatchExecution/StreamExecution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/24 23:26
Affects Version/s: 4.0.0

To improve code clarity and maintainability, I propose that we move all the variables that track mutable state and metrics for streaming query into a separate class.  With this refactor, it would be easy to track and find all the mutable state a microbatch can have.
Bug: Upgrade Python to 3.11 in Maven builds
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/24 00:12
Affects Version/s: 4.0.0

{code}
  Error: dyld[4738]: Library not loaded: /usr/local/opt/gettext/lib/libintl.8.dylib
    Referenced from: <E2FD7085-038F
  Error: -3B8E-94C6-6649527BFDBE> /Users/runner/hostedtoolcache/Python/3.9.18/x64/bin/python3.9
    Reason: tried: '/usr/local/opt/gettext/lib/libintl.8.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/opt/gettext/lib/libintl.8.dylib' (no such file), '/usr/local/opt/gettext/lib/libintl.8.dylib' (no such file), '/usr/local/lib/libintl.8.dylib' (no such file), '/usr/lib/libintl.8.dylib' (no such file, not in dyld cache)
  ./setup.sh: line 52:  4738 Abort trap: 6           ./python -m ensurepip
{code}

https://github.com/apache/spark/actions/runs/7964626045/job/21742574260

https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md
Bug: Raise Spark's exception with an error class in config value check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/24 18:40
Affects Version/s: 4.0.0

Currently, Spark throws *IllegalArgumentException* in `checkValue` of ConfigBuilder. Need to overload `checkValue` to throw `SparkIllegalArgumentException` with an error class. This should improve user experience with Spark SQL, and impressions of Spark's errors.
Bug: Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/24 18:35
Affects Version/s: 4.0.0

This has been observed multiple times.
{code:java}
[info] - incomplete apps get refreshed *** FAILED *** (15 seconds, 450 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 43 times over 10.22918722 seconds. Last failure message: 0 did not equal 4. (HistoryServerSuite.scala:564) {code}
Bug: Use Scala 2.13 Spark distribution in HiveExternalCatalogVersionsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Feb/24 17:26
Affects Version/s: 4.0.0

SPARK-45314 makes Scala 2.13 is the default Scala version.
As one of migration paths, the users choose Apache Spark 3.5.0 (Scala 2.13) and Apache Spark 3.4.2 (Scala 2.13).
We had better focus on Scala 2.13 testing.
Bug: Wrong result from flatMapGroups using Scala 2.13.x
Status: Open | Priority: Major | Resolution: 
Created: 15/Feb/24 12:30
Affects Version/s: 3.5.0

Using Scala 2.13 and `KeyValueGroupedDataset::flatMapGroups` produces wrong results. All rows produced by flatMapGroups have the values from the last entry in the returned iterator.
 * Downgrading to Scala 2.12 fixes the issue.
 * Using `mapGroups` followed by `flatMap` also fixes the issue.

 

Test-Setup:
{code:scala}
import org.apache.spark.sql.SparkSession
object Main {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[*]").getOrCreate()
    import spark.implicits._

    // using flatMapGroups
    spark.createDataset(Seq(1, 2))
      .groupByKey(x => x)
      .flatMapGroups((x, _) => Seq(10 + x, 20 + x, 30 + x)).show()

    // second code using map, then flatMap ~> should yield the same result    
    spark.createDataset(Seq(1, 2))
      .groupByKey(x => x)
      .mapGroups((x, _) => Seq(10 + x, 20 + x, 30 + x))
      .flatMap(x => x).show()
   }
} {code}
We map the key 1 to the Sequence (11, 21, 31). Analogously the key 2 is mapped to (12, 22, 32). Both computations should produce the following (identical) result:
{code:java}
+-----+
|value|
+-----+
|   11|
|   21|
|   31|
|   12|
|   22|
|   32|
+-----+ {code}
This was run using Scala 2.12 with Spark 3.5 - using the following `build.sbt`:
{code:java}
ThisBuild / scalaVersion := "2.12.18"
libraryDependencies += "org.apache.spark" % "spark-sql_2.12" % "3.5.0"
{code}
 

Problem: By upgrading to Scala 2.13 we instead get the following result:
{code:java}
+-----+
|value|
+-----+
|   31|
|   31|
|   31|
|   32|
|   32|
|   32|
+-----+ {code}
Using this new `build.sbt`. The Code was not modified.
{code:java}
ThisBuild / scalaVersion := "2.13.10"
libraryDependencies += "org.apache.spark" % "spark-sql_2.13" % "3.5.0" {code}
[The test-case is inspired by this StackOverflow question.|https://stackoverflow.com/questions/74633091/why-does-keyvaluegroupeddatasets-flatmapgroups-give-incorrect-result-when-runni]
Bug: join between cached temp tables result in missing entries
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/24 19:12
Affects Version/s: 3.5.0

we create several temp tables (views) by loading several delta tables and joining between them. 
those views are used for calculation of different metrics. each metric requires different views to be used. some of the more popular views are cached for better performance. 

we have noticed that once we upgraded from spark 3.4.2  to spark 3.5.0 some of the join started to fail.

we can reproduce a case were we have 2 data frames (views) (this is not the real names  / values we use. this is just for the example)
 # users with the column user_id, campaign_id, user_name.
we make sure it has a single entry
'111111', '22222', 'Jhon Doe'
 # actions with the column user_id, campaign_id, action_id, action count
we make sure it has a single entry
'111111', '22222', 'clicks', 5

 
 # users view can be filtered for user_id = '111111' or/and campaign_id = '22222' and it will find the existing single row
 # actions view can be filtered for user_id = '111111' or/and campaign_id = '22222' and it will find the existing single row
 # users and actions can be inner join by user_id *OR* campaign_id and the join will be successful. 
 # users and actions can *not* be inner join by user_id *AND* campaign_id. The join results in no entry.

 # if we write both of the views to S3 and read them back to new data frames, suddenly the join is working.
 # if we disable AQE the join is working
 # running checkpoint on the views does not make join #4 work
Bug: Sum of floats/doubles may be incorrect depending on partitioning
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 12/Feb/24 17:32
Affects Version/s: 3.3.4

I found this problem using [Hypothesis|https://hypothesis.readthedocs.io/en/latest/].

Here's a reproduction that fails on {{{}master{}}}, 3.5.0, 3.4.2, and 3.3.4 (and probably all prior versions as well):
{code:python}
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum

SUM_EXAMPLE = [
    (1.0,),
    (0.0,),
    (1.0,),
    (9007199254740992.0,),
]

spark = (
    SparkSession.builder
    .config("spark.log.level", "ERROR")
    .getOrCreate()
)


def compare_sums(data, num_partitions):
    df = spark.createDataFrame(data, "val double").coalesce(1)
    result1 = df.agg(sum(col("val"))).collect()[0][0]
    df = spark.createDataFrame(data, "val double").repartition(num_partitions)
    result2 = df.agg(sum(col("val"))).collect()[0][0]
    assert result1 == result2, f"{result1}, {result2}"


if __name__ == "__main__":
    print(compare_sums(SUM_EXAMPLE, 2))
{code}
This fails as follows:
{code:python}
AssertionError: 9007199254740994.0, 9007199254740992.0
{code}
I suspected some kind of problem related to code generation, so tried setting all of these to {{{}false{}}}:
 * {{spark.sql.codegen.wholeStage}}
 * {{spark.sql.codegen.aggregate.map.twolevel.enabled}}
 * {{spark.sql.codegen.aggregate.splitAggregateFunc.enabled}}

But this did not change the behavior.

Somehow, the partitioning of the data affects the computed sum.
Bug: ALTER COLUMN DROP DEFAULT test fails with JSON data sources
Status: Open | Priority: Major | Resolution: 
Created: 12/Feb/24 19:32
Affects Version/s: 3.5.0

Enabling the JSON data source causes a test case to fail:
{code:java}
[info] - SPARK-39557 INSERT INTO statements with tables with map defaults *** FAILED *** (1 second, 498 milliseconds)
[info]   Results do not match for query:
[info]   Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
[info]   Timezone Env:
[info]
[info]   == Parsed Logical Plan ==
[info]   'UnresolvedRelation [t], [], false
[info]
[info]   == Analyzed Logical Plan ==
[info]   i: int, s: struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>, t: array<map<string,boolean>>
[info]   SubqueryAlias spark_catalog.default.t
[info]   +- Relation spark_catalog.default.t[i#13929,s#13930,t#13931] json
[info]
[info]   == Optimized Logical Plan ==
[info]   Relation spark_catalog.default.t[i#13929,s#13930,t#13931] json
[info]
[info]   == Physical Plan ==
[info]   FileScan json spark_catalog.default.t[i#13929,s#13930,t#13931] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex(1 paths)[file:/home/mark.jarvin/photon/spark/sql/core/spark-warehouse/org.apach..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<i:int,s:struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>,t:array<map<stri...
[info]
[info]   == Results ==
[info]
[info]   == Results ==
[info]   !== Correct Answer - 4 ==                                                        == Spark Answer - 4 ==
[info]   !struct<>                                                                        struct<i:int,s:struct<x:array<struct<a:int,b:int>>,y:array<map<string,boolean>>>,t:array<map<string,boolean>>>
[info]   ![1,[List([1,2]),List(Map(def -> false, jkl -> true))],List(Map(xyz -> true))]   [1,[ArraySeq([1,2]),ArraySeq(Map(def -> false, jkl -> true))],ArraySeq(Map(xyz -> true))]
[info]   ![2,null,List(Map(xyz -> true))]                                                 [2,[ArraySeq([1,2]),ArraySeq(Map(def -> false, jkl -> true))],ArraySeq(Map(xyz -> true))]
[info]   ![3,[List([3,4]),List(Map(mno -> false, pqr -> true))],List(Map(xyz -> true))]   [3,[ArraySeq([3,4]),ArraySeq(Map(mno -> false, pqr -> true))],ArraySeq(Map(xyz -> true))]
[info]   ![4,[List([3,4]),List(Map(mno -> false, pqr -> true))],List(Map(xyz -> true))]   [4,[ArraySeq([3,4]),ArraySeq(Map(mno -> false, pqr -> true))],ArraySeq(Map(xyz -> true))] (QueryTest.scala:267){code}