av9ash/GitBugs huggingface dataset:
Bug: Use Magic Committer for all S3 buckets by default
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Mar/24 21:11
Affects Version/s: 4.1.0

This issue aims to use Apache Hadoop `Magic Committer` for all S3 buckets by default in Apache Spark 4.0.0.

Apache Hadoop `Magic Committer` has been used for S3 buckets to get the best performance since [S3 becomes fully consistent on December 1st, 2020|https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/].
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel
bq. Amazon S3 provides strong read-after-write consistency for PUT and DELETE requests of objects in your Amazon S3 bucket in all AWS Regions. This behavior applies to both writes to new objects as well as PUT requests that overwrite existing objects and DELETE requests. In addition, read operations on Amazon S3 Select, Amazon S3 access controls lists (ACLs), Amazon S3 Object Tags, and object metadata (for example, the HEAD object) are strongly consistent.
Bug: Precision issues with sum of floats/doubles leads to incorrect data after repartition stage retry
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/24 08:08
Affects Version/s: 3.3.2

We discovered an important correctness issue directly linked to SPARK-47024

Even if SPARK-47024 has been considered 'Not a Problem' since it is linked directly to floats and double precision, it can still have drastic impacts combined to spark.sql.execution.sortBeforeRepartition set to true (the default)

We consistently reproduced the issue doing a GROUP BY with a SUM of float or double aggregation, followed by a repartition (common case to produce bigger files as output, either triggered by SQL hints or extensions like kyuubi). 

If the repartition stage fails with Fetch Failed Exception for only few tasks, spark decides to recompute the partitions from the previous stage for which output could not be fetched and will retry only the failed partitions downstream.

Because block fetch order is indeterministic, the new upstream partition computation can provide a slightly different value for a float/double sum aggregation. We noticed a 1 bit difference is UnsafeRow backing byte array in all of our attempts. The sort performed before repartition uses UnsafeRow.hashcode for the row prefix which will be completely different even with such 1 bit difference, leading to the sort being completely different in the new upstream partition and thus target downstream partition for the shuffled rows completely different as well.

Because sort becomes undeterministic and since only the failed dowstream tasks are retried the resulting repartition will lead to duplicate rows as well as missing rows. The solution brought by SPARK-23207 is broken.

So far, we can only deactivate spark.sql.execution.sortBeforeRepartition to make the entire job fail instead of producing incorrect data. The default for spark currently leads to silent correctness issue.
Bug: Set spark.hadoop.fs.s3a.connection.establish.timeout to 30s
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/24 21:54
Affects Version/s: 4.0.0

To suppress like HADOOP-19097
{code}
24/03/25 14:46:21 WARN ConfigurationHelper: Option fs.s3a.connection.establish.timeout is too low (5,000 ms). Setting to 15,000 ms instead
{code}
Bug: CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan
Status: Open | Priority: Major | Resolution: 
Created: 27/Mar/24 01:15
Affects Version/s: 3.5.1

This issue became apparent while bringing my PR 
[https://github.com/apache/spark/pull/49124]

in synch with latest master.

Though that PR is meant to do early collapse of projects so that the tree size is kept at minimum when projects keep getting added , in the analyzer phase itself.

But as part of the work, the CacheManager lookup also needed to be modified.

One of the newly added test in master failed. On analysis of failure it turns out that the cache manager is not picking cached InMemoryRelation for a subplan.

This shows up in following existing test

org.apache.spark.sql.DatasetCacheSuite
{quote}test("SPARK-26708 Cache data and cached plan should stay consistent") {
val df = spark.range(0, 5).toDF("a")
val df1 = df.withColumn("b", $"a" + 1)
val df2 = df.filter($"a" > 1)

df.cache()
// Add df1 to the CacheManager; the buffer is currently empty.
df1.cache()
{color:#4c9aff}// After calling collect(), df1's buffer has been loaded.{color}
df1.collect()
// Add df2 to the CacheManager; the buffer is currently empty.
df2.cache()

// Verify that df1 is a InMemoryRelation plan with dependency on another cached plan.
assertCacheDependency(df1)
val df1InnerPlan = df1.queryExecution.withCachedData
.asInstanceOf[InMemoryRelation].cacheBuilder.cachedPlan
// Verify that df2 is a InMemoryRelation plan with dependency on another cached plan.
assertCacheDependency(df2)

df.unpersist(blocking = true)

{color:#00875a}// Verify that df1's cache has stayed the same, since df1's cache already has data{color}
// before df.unpersist().
val df1Limit = df1.limit(2)
val df1LimitInnerPlan = df1Limit.queryExecution.withCachedData.collectFirst
Unknown macro: \{ case i}
assert(df1LimitInnerPlan.isDefined && df1LimitInnerPlan.get == df1InnerPlan)

// Verify that df2's cache has been re-cached, with a new physical plan rid of dependency
// on df, since df2's cache had not been loaded before df.unpersist().
val df2Limit = df2.limit(2)
val df2LimitInnerPlan = df2Limit.queryExecution.withCachedData.collectFirst
Unknown macro: \{ case i}
{quote}
{quote}*{color:#de350b}// This assertion is not right{color}*
assert(df2LimitInnerPlan.isDefined &&
!df2LimitInnerPlan.get.exists(_.isInstanceOf[InMemoryTableScanExec]))
}
{quote}
 

Since df1 exists in the cache as InMemoryRelation,

val df = spark.range(0, 5).toDF("a")
val df1 = df.withColumn("b", $"a" + 1)
val df2 = df.filter($"a" > 1)

df2 is derivable from the cached df1.

So when val df2Limit = df2.limit(2), is created,  it should utilize the cached df1 .

 

The pull request for the same is 

[https://github.com/apache/spark/pull/49124|https://github.com/apache/spark/pull/49124]
Bug: Use Java 21 instead of 21-jre in K8s Dockerfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Mar/24 22:57
Affects Version/s: 4.0.0

{code}
$ docker run -it --rm azul/zulu-openjdk:21-jre jmap
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "jmap": executable file not found in $PATH: unknown.
{code}
Bug: Spark 3.3 produces invalid base64
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/24 17:23
Affects Version/s: 3.3.0

SPARK-37820 was introduced in Spark 3.3 and breaks behavior of {{base64}} (which is fine but shouldn't happen between minor version).

{code:title=Spark 3.2}
>>> spark.sql(f"""SELECT base64('{'a' * 58}') AS base64""").collect()[0][0]
'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYQ=='
{code}

Note the different output in Spark 3.3 (the addition of {{\r\n}} newlines).

{code:title=Spark 3.3}
>>> spark.sql(f"""SELECT base64('{'a' * 58}') AS base64""").collect()[0][0]
'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFh\r\nYQ=='
{code}

The former decodes fine with the {{base64}} on my machine but the latter does not:
{code}
$ pbpaste | base64 --decode
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%

$ pbpaste | base64 --decode
base64: stdin: (null): error decoding base64 input stream
{code}
Bug: Addition of Indeterminate Collation Support
Status: Open | Priority: Major | Resolution: 
Created: 28/Mar/24 09:24
Affects Version/s: 4.0.0

{{INDETERMINATE_COLLATION}} should only be thrown on comparison operations and memory storing of data, and we should be able to combine different implicit collations for certain operations like concat and possible others in the future.
This is why we have to add another predefined collation id named {{INDETERMINATE_COLLATION_ID}} which means that the result is a combination of conflicting non-default implicit collations. Right now it would an id of -1 so it fail if it ever goes to the {{{}CollatorFactory{}}}.
Bug: Support GROUP BY for MapType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Mar/24 09:31
Affects Version/s: 4.0.0

Currently, Spark GROUP BY doesn't allow map data types, otherwise the plan analysis fails: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala#L197-L203

Using implemented map_sort from https://issues.apache.org/jira/browse/SPARK-47007 we should implement GROUP BY for map data types.
Bug: Driver stuck when thread pool is not shut down 
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/24 12:21
Affects Version/s: 4.0.0

The app example：

 
{code:java}
object SparkTest {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("zt-test")
      .config("spark.logConf",true)
      .getOrCreate()
    spark.sparkContext.setLogLevel("INFO")
    val threadPool = Executors.newFixedThreadPool(5)

    for (i <- 0 until 10) {
      threadPool.execute(new Task("Task " + i))
      // do not shut down threadPool
    }
    val rdd = spark.sparkContext.makeRDD(Seq(1,2,4))
    val res = rdd.collect()
    // throw 
    throw new Throwable()
  }
}

class Task(private var name: String) extends Runnable {
  override def run(): Unit = {
    System.out.println("Executing task: " + name + " by " + Thread.currentThread.getName)
  }
}{code}
 

when app is running on yarn with cluster mode, even if thread pool is not closed, driver will shut down, which can not lead container not to stop. However, when running on k8s, if thread pool is not closed, the driver pod will be stuck, and will not release resource. 

With yarn-cluster mode, the ApplicationMaster wrapped with 'System.exit', like this
{code:java}
ugi.doAs(new PrivilegedExceptionAction[Unit]() {
  override def run(): Unit = System.exit(master.run()) 
}) {code}
so, when threads are parking, exitcode can also be passed to System#exit. In this sutiation, AM can stop.

When driver is on k8s with client mode, if encounters exception and thread pool is not closed, driver pod may stuck.
Bug: StringTranslate (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 12:45
Affects Version/s: 4.0.0

Enable collation support for the *StringTranslate* built-in string function in Spark. First confirm what is the expected behaviour for this function when given collated strings, and then move on to implementation and testing. One way to go about this is to consider using {_}StringSearch{_}, an efficient ICU service for string matching. Implement the corresponding unit tests (CollationStringExpressionsSuite) and E2E tests (CollationSuite) to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *StringTranslate* function so it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class, as well as _StringSearch_ using the [ICU user guide|https://unicode-org.github.io/icu/userguide/collation/string-search.html] and [ICU docs|https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/StringSearch.html]. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: spark driver process hangs due to "unable to create new native thread"
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 03:12
Affects Version/s: 3.1.1

we encounter that spark driver hangs for about 11 hours,  and finall killed by user. In the driver log there is an error log: 
{quote}16:42:40 151 ERROR (org.apache.spark.rpc.netty.Inbox:94) - An error happened while processing message in the inbox for CoarseGrainedScheduler
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:719)
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
        at org.apache.spark.scheduler.TaskResultGetter.enqueueSuccessfulTask(TaskResultGetter.scala:61)
        at org.apache.spark.scheduler.TaskSchedulerImpl.liftedTree2$1(TaskSchedulerImpl.scala:769)
        at org.apache.spark.scheduler.TaskSchedulerImpl.statusUpdate(TaskSchedulerImpl.scala:745)
        at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anonfun$receive$1.applyOrElse(CoarseGrainedSchedulerBackend.scala:144)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
{quote}
 

After detailed analysis, we found that, the driver submitted task 0.0 at "16:40:50" to executor 4, and executor 4 finished the task 0.0 at "16:42:39", then executor 4 sent results to the driver. But in the same time, there is not sufficient memory in the the server that running the driver, the driver "unable to create new native thread" to handle the successful result of task 0.0, then the driver think task 0.0 has not finished and waiting for the "missed result" forever.

 

driver submit task 0.0

!driver_submit_task.png!

 

executor 4 task 0.0

!executor_4.png!

 

oom-killer:

!oom-killer.png!
Bug: Introduce `_LEGACY_ERROR_UNKNOWN` for default error class
Status: Open | Priority: Major | Resolution: 
Created: 11/Mar/24 07:58
Affects Version/s: 4.0.0

Currently, in Spark, when an {{ErrorClass}} is not explicitly defined for an exception, the method {{getErrorClass}} returns {{{}null{}}}. This behavior can lead to ambiguity and makes debugging more challenging by not providing a clear indication that the error class was not set.
Bug: Spark Remote Connect to multiple Spark Sessions
Status: Open | Priority: Major | Resolution: 
Created: 29/Mar/24 14:28
Affects Version/s: 3.5.0

Current Spark Remote connects looks to have a single spark remote connect session. Can it be extended to do multiple Spark Sessions. This would help in creating decentralized Kubernetes/Custom Cloud Environment and reduce the compute of the current Spark Session.

Making Spark to work like a full remote API call for multiple datasets where requried.
Bug: Skip transfer the last spilled shuffle data
Status: Open | Priority: Major | Resolution: 
Created: 22/Mar/24 07:25
Affects Version/s: 4.0.0

If there is only one spill data file, spark will transfer that spill file to the final data file.
{code:java}
@Override
  public void transferMapSpillFile(
      File mapSpillFile,
      long[] partitionLengths,
      long[] checksums) throws IOException {
    // The map spill file already has the proper format, and it contains all of the partition data.
    // So just transfer it directly to the destination without any merging.
    File outputFile = blockResolver.getDataFile(shuffleId, mapId);
    File tempFile = Utils.tempFileWith(outputFile);
    Files.move(mapSpillFile.toPath(), tempFile.toPath());
    blockResolver
      .writeMetadataFileAndCommit(shuffleId, mapId, partitionLengths, checksums, tempFile);
  }
{code}
But if that spill file and the final data file are on different disks,  there will still be a heavy data transfer.
{code:java}
sun.nio.fs.UnixCopyFile.transfer(Native Method)
sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:251)
sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:471)
sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
java.nio.file.Files.move(Files.java:1395)
org.apache.spark.shuffle.sort.io.LocalDiskSingleSpillMapOutputWriter.transferMapSpillFile(LocalDiskSingleSpillMapOutputWriter.java:52)
org.apache.spark.shuffle.sort.UnsafeShuffleWriter.mergeSpills(UnsafeShuffleWriter.java:280)
org.apache.spark.shuffle.sort.UnsafeShuffleWriter.closeAndWriteOutput(UnsafeShuffleWriter.java:224)
org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:180)
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
org.apache.spark.scheduler.Task.run(Task.scala:131)
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
org.apache.spark.executor.Executor$TaskRunner$$Lambda$453/980524593.apply(Unknown Source)
org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1465)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
{code}

We can optimize this step by writing the final spill file to the disk which the final data file will be used.
Bug: Add hooks to release the ANTLR DFA cache after parsing SQL
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/24 21:58
Affects Version/s: 4.0.0

ANTLR builds a DFA cache while parsing to speed up parsing of similar future inputs. However, this cache is never cleared and can only grow. Extremely large SQL inputs can lead to very large DFA caches (>20GiB in one extreme case I've seen).

Spark’s ANTLR SQL parser is derived from the Presto ANTLR SQL Parser, and Presto has added hooks to be able to clear this DFA cache. I think Spark should have similar hooks.

References:
 * [https://github.com/antlr/antlr4/blob/f08a19bbb202b02a521f84d99e661e386bea8625/runtime/Java/src/org/antlr/v4/runtime/atn/ParserATNSimulator.java#L163-L171]

 * [https://stackoverflow.com/questions/28017135/why-antlr4-parsers-accumulates-atnconfig-objects?rq=2]

 * [https://github.com/antlr/antlr4/issues/499]

 * [https://github.com/trinodb/trino/pull/3186/files#diff-75b81ed5837578d1af42fcc91e4094a247138e5da6edb9d9e4b67d53247b8ca9]
Bug: Mode expression for strings (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 12:24
Affects Version/s: 4.0.0

Enable collation support for the *Mode* expression in Spark. First confirm what is the expected behaviour for this expression when given collated strings, then move on to the implementation that would enable handling strings of all collation types. Implement the corresponding unit tests and E2E SQL tests to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *Mode* expression so it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

Examples:

With UTF8_BINARY collation, the query
SELECT mode(col) FROM VALUES (‘a’), (‘a’), (‘a’), (‘B’), (‘B’), (‘b’), (‘b’) AS tab(col);
should return 'a'.

With UTF8_BINARY_LCASE collation, the query
SELECT mode(col) FROM VALUES (‘a’), (‘a’), (‘a’), (‘B’), (‘B’), (‘b’), (‘b’) AS tab(col);
should return either 'B' or 'b'.

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Levenshtein (all collations)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/24 12:05
Affects Version/s: 4.0.0

Enable collation support for the *Levenshtein* built-in string function in Spark. First confirm what is the expected behaviour for this function when given collated strings, and then move on to implementation and testing. Implement the corresponding unit tests and E2E sql tests to reflect how this function should be used with collation in SparkSQL, and feel free to use your chosen Spark SQL Editor to experiment with the existing functions to learn more about how they work. In addition, look into the possible use-cases and implementation of similar functions within other other open-source DBMS, such as [PostgreSQL|https://www.postgresql.org/docs/].

 

The goal for this Jira ticket is to implement the *Levenshtein* function so it supports all collation types currently supported in Spark. To understand what changes were introduced in order to enable full collation support for other existing functions in Spark, take a look at the Spark PRs and Jira tickets for completed tasks in this parent (for example: Contains, StartsWith, EndsWith).

 

Read more about ICU [Collation Concepts|http://example.com/] and [Collator|http://example.com/] class. Also, refer to the Unicode Technical Standard for string [searching|https://www.unicode.org/reports/tr10/#Searching] and [collation|https://www.unicode.org/reports/tr35/tr35-collation.html#Collation_Type_Fallback].
Bug: Add PySpark DataFrame method to return iterator of PyArrow RecordBatches
Status: Open | Priority: Major | Resolution: 
Created: 19/Mar/24 16:58
Affects Version/s: 3.5.1

As a follow-up to SPARK-47365:

{{toArrow()}} is useful when the data is relatively small. For larger data, the best way to return the contents of a PySpark DataFrame in Arrow format is to return an iterator of [PyArrow RecordBatches|https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html].