av9ash/GitBugs huggingface dataset:
Bug: Subquery changes the output schema of the outer query
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 17/Oct/23 21:13
Affects Version/s: 3.3.3

A query can have an incorrect output schema because of a subquery.

Assume this data:
{noformat}
create or replace temp view t1(a) as values (1), (2), (3), (7);
create or replace temp view t2(c1) as values (1), (2), (3);
create or replace temp view t3(col1) as values (3), (9);
cache table t1;
cache table t2;
cache table t3;
{noformat}
When run in {{spark-sql}}, the following query has a superfluous boolean column:
{noformat}
select *
from t1
where exists (
  select c1
  from t2
  where a = c1
  or a in (select col1 from t3)
);

1	false
2	false
3	true
{noformat}
The result should be:
{noformat}
1
2
3
{noformat}
When executed via the {{Dataset}} API, you don't see the incorrect result, because the Dataset API truncates the right-side of the rows based on the analyzed plan's schema (it's the optimized plan's schema that goes wrong).

However, even with the {{Dataset}} API, this query goes wrong:
{noformat}
select (
  select *
  from t1
  where exists (
    select c1
    from t2
    where a = c1
    or a in (select col1 from t3)
  )
  limit 1
)
from range(1);

java.lang.AssertionError: assertion failed: Expects 1 field, but got 2; something went wrong in analysis
	at scala.Predef$.assert(Predef.scala:279)
	at org.apache.spark.sql.execution.ScalarSubquery.updateResult(subquery.scala:88)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$waitForSubqueries$1(SparkPlan.scala:276)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$waitForSubqueries$1$adapted(SparkPlan.scala:275)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
        ...
{noformat}
Other queries that have the wrong schema:
{noformat}
select *
from t1
where a in (
  select c1
  from t2
  where a in (select col1 from t3)
);
{noformat}
and
{noformat}
select *
from t1
where not exists (
  select c1
  from t2
  where a = c1
  or a in (select col1 from t3)
);
{noformat}
Bug: Broken backward compatibility in PySpark: StreamingQueryListener due to the addition of onQueryIdle
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Oct/23 06:13
Affects Version/s: 3.5.0

We observed the issue that existing StreamingQueryListener implementations in PySpark start to fail after upgrading to 3.5.0. We identified that the root cause is https://issues.apache.org/jira/browse/SPARK-43183 since onQueryIdle was defined as `@abstractmethod`, requiring all exiting implementations to implement the method.

It seems like abstractmethod annotation should be removed to retain backward compatibility.
Bug: client will wait forever if session in spark connect server is evicted
Status: Resolved | Priority: Critical | Resolution: Implemented
Created: 31/Oct/23 11:10
Affects Version/s: 3.5.0

Step1. start a spark connect server

Step2. submit a spark job which will run long
{code:java}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote(f"sc://HOST:PORT/;user_id=job").create()
spark.sql("A SQL will run longer than creating 100 sessions").show() {code}
 

Step3. create more than 100 sessions

Tips: Run concurrently with step2
{code:java}
for i in range(0, 200):
    spark = SparkSession.builder.remote(f"sc://HOST:PORT/;user_id={i}").create()
    spark.sql("show databases") {code}
 

*When the python code in step3 is executed, the session created in step2 will be evicted, and the client will wait forever*
Bug: Upgrade AWS SDK to v2 for Spark Kinesis connector
Status: Open | Priority: Major | Resolution: 
Created: 30/Oct/23 09:32
Affects Version/s: 3.5.0

Sub-task of [SPARK-44124|https://issues.apache.org/jira/browse/SPARK-44124]. In this issue, we focus on the AWS SDK v2 upgrade in Kinesis connector
Bug: Canonicalization of DynamicPruningSubquery is broken
Status: Open | Priority: Major | Resolution: 
Created: 25/Oct/23 04:59
Affects Version/s: 3.5.0

The canonicalization of (buildKeys: Seq[Expression]) in the class DynamicPruningSubquery is broken, as the buildKeys are canonicalized just by calling 
buildKeys.map(_.canonicalized)
The  above would result in incorrect canonicalization as it would not be normalizing the exprIds relative to buildQuery output
The fix is to use the buildQuery : LogicalPlan's output to normalize the buildKeys expression
as given below, using the standard approach.

buildKeys.map(QueryPlan.normalizeExpressions(_, buildQuery.output)),
Bug: Extremely slow execution of sum of columns in Spark 3.4.1
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 31/Oct/23 21:59
Affects Version/s: 3.4.1

We are in the process of upgrading some pySpark jobs from Spark 3.1.2 to Spark 3.4.1 and some code that was running fine is now basically never ending even for small dataframes.

We have simplified the problematic piece of code and the minimum pySpark example below shows the issue:
{code:java}
n_cols = 50
data = [{f"col{i}": i for i in range(n_cols)} for _ in range(5)]
df_data = sql_context.createDataFrame(data)

df_data = df_data.withColumn(
    "col_sum", sum([F.col(f"col{i}") for i in range(n_cols)])
)
df_data.show(10, False) {code}
Basically, this code with Spark 3.1.2 runs fine but with 3.4.1 the computation time seems to explode when the value of `n_cols` is bigger than about 25 columns. A colleague suggested that it could be related to the limit of 22 elements in a tuple in Scala 2.13 (https://www.scala-lang.org/api/current/scala/Tuple22.html), since the 25 columns are suspiciously close to this. Is there any known defect in the logical plan optimization in 3.4.1? Or is this kind of operations (sum of multiple columns) supposed to be implemented differently?
Bug: Fix yarn distribution build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/23 20:27
Affects Version/s: 4.0.0

[https://github.com/apache/spark/pull/43164] introduced a regression in:

 

```

./dev/make-distribution.sh --tgz -Phive -Phive-thriftserver -Pyarn

```

 

this needs to be fixed
Bug: Use `LazyList` instead of `Stream`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/23 02:17
Affects Version/s: 4.0.0

* class Stream in package immutable is deprecated (since 2.13.0)
 * object Stream in package immutable is deprecated (since 2.13.0)
 * type Stream in package scala is deprecated (since 2.13.0)
 * value Stream in package scala is deprecated (since 2.13.0)
 * method append in class Stream is deprecated (since 2.13.0)
 * method toStream in trait IterableOnceOps is deprecated (since 2.13.0)

 
{code:java}
[warn] /Users/yangjie01/SourceCode/git/spark-mine-sbt/sql/core/src/test/scala/org/apache/spark/sql/GenTPCDSData.scala:49:20: class Stream in package immutable is deprecated (since 2.13.0): Use LazyList (which is fully lazy) instead of Stream (which has a lazy tail only)
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.BlockingLineStream.BlockingStreamed.stream, origin=scala.collection.immutable.Stream, version=2.13.0
[warn]     val stream: () => Stream[T])
[warn]                    ^ {code}
Bug: Add feature to evaluate subquery before push down filter Optimizer rule
Status: Open | Priority: Major | Resolution: 
Created: 21/Oct/23 05:10
Affects Version/s: 3.3.2

Some queries can benefit from having it's scalar subquery in the filter evaluated while planning so that the scalar result (from the subquery) can be push down. 

This adds a new feature(which is disabled by default to maintain current behavior) that would evaluate scalar subqueries in the Optimizer before rule to push down filter. 

For example, a query like 
{code:java}
select * from t2 where b > (select max(a) from t1) {code}
where t1 is a small table but t2 is a very large table can benefit if we first evaluate the subquery then push down the result to the pushed filter (instead of having the subquery in the post scan filter)
Bug: Executor hangs indefinitely due to decommissioner errors
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/23 20:18
Affects Version/s: 3.5.0

During Spark executor decommission, the fallback storage uploads can fail due to some race conditions even though we check the actual file exists: 

java.io.FileNotFoundException: No file: /var/data/spark-ab14b716-630d-435e-a92a-1403f6206dd8/blockmgr-7f9ab4d7-1340-4b39-9558-fde994a82090/0b/shuffle_175_66754_0.index
	at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.checkSource(CopyFromLocalOperation.java:314)
	at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.execute(CopyFromLocalOperation.java:167)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFromLocalFile$26(S3AFileSystem.java:3854)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2480)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2499)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:3847)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2558)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2520)
	at org.apache.spark.storage.FallbackStorage.copy(FallbackStorage.scala:67)
	at org.apache.spark.storage.BlockManagerDecommissioner$ShuffleMigrationRunnable.$anonfun$run$12(BlockManagerDecommissioner.scala:146)
	at org.apache.spark.storage.BlockManagerDecommissioner$ShuffleMigrationRunnable.$anonfun$run$12$adapted(BlockManagerDecommissioner.scala:146)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.storage.BlockManagerDecommissioner$ShuffleMigrationRunnable.run(BlockManagerDecommissioner.scala:146)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

This will block the executor from exiting properly because the decommissioner doesn't think shuffle migration is complete.
Bug: Retry mvn deploy failures
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/23 11:52
Affects Version/s: 4.0.0

It is common to see {{408 Request Timeout}} and {{502 Proxy Error}} when deploying artifacts to Apache snapshot repository: https://github.com/apache/spark/actions/runs/6437635317

{quote}
2023-10-07T01:09:51.1719360Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project spark-streaming_2.13: ArtifactDeployerException: Failed to deploy artifacts: Could not transfer artifact org.apache.spark:spark-streaming_2.13:jar:tests:3.3.4-20231007.005815-57 from/to apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots): transfer failed for https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-streaming_2.13/3.3.4-SNAPSHOT/spark-streaming_2.13-3.3.4-20231007.005815-57-tests.jar, status: 502 Proxy Error -> [Help 1]
{quote}

{quote}
2023-10-07T01:11:48.5651501Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project spark-connect-common_2.12: Failed to deploy artifacts: Could not transfer artifact org.apache.spark:spark-connect-common_2.12:xml:cyclonedx:3.4.2-20231007.001102-103 from/to apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots): transfer failed for https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-connect-common_2.12/3.4.2-SNAPSHOT/spark-connect-common_2.12-3.4.2-20231007.001102-103-cyclonedx.xml, status: 408 Request Timeout -> [Help 1]
{quote}

Such errors should be retried by `mvn deploy`.
Bug: Mapped TINYINT to ShortType for MsSqlServerDialect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/23 18:30
Affects Version/s: 3.5.0

TINYINT of SQL server is not correctly mapped to spark types when using JDBC connector. For now, it is mapped to the IntegerType, which is not accurate. It should be mapped to ShortType.
Bug: Publish all snapshots or none
Status: Open | Priority: Major | Resolution: 
Created: 27/Oct/23 12:01
Affects Version/s: 4.0.0

We have seen failures during publishing snapshots due to build failures (SPARK-45651). As a consequence, some packages have been published and others haven't. This leads to an inconsistency across the latest Spark snapshots, which can cause {{ClassNotFoundException}}s like in https://github.com/apache/spark/pull/43102#issuecomment-1759541718.

Ideally, packages are published once *all* packages have been built, not once individual packages have been built.
Bug: SPJ: Handle empty input partitions after dynamic filtering
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Oct/23 16:14
Affects Version/s: 3.4.1

When the number of input partitions become 0 after dynamic filtering, in {{BatchScanExec}}, currently SPJ will fail with error:
{code}
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions$lzycompute(BatchScanExec.scala:108)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.filteredPartitions(BatchScanExec.scala:65)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD$lzycompute(BatchScanExec.scala:136)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.inputRDD(BatchScanExec.scala:135)
{code}

This is because {{groupPartitions}} will return {{None}} for this case.
Bug: Upgrade AWS SDK to v2 for Hadoop dependency
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 30/Oct/23 09:36
Affects Version/s: 3.5.0

Hadoop is planning to ship the SDKv2 upgrade in 3.4.0, as shown in this [HADOOP-18073|https://issues.apache.org/jira/browse/HADOOP-18073]. One of the Hadoop modules that Spark relies on is *hadoop-aws*, which comes with S3A connector that allows Spark to access data in S3 buckets. Hadoop-aws contains dependency on AWS SDKv1, thus we should also update the Hadoop version to 3.4.0.
Bug: Task fraction resource request is not expected
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/23 01:56
Affects Version/s: 3.2.1

 
{code:java}
test("SPARK-XXX") {
  import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

  withTempDir { dir =>
    val scriptPath = createTempScriptWithExpectedOutput(dir, "gpuDiscoveryScript",
      """{"name": "gpu","addresses":["0"]}""")

    val conf = new SparkConf()
      .setAppName("test")
      .setMaster("local-cluster[1, 12, 1024]")
      .set("spark.executor.cores", "12")
    conf.set(TASK_GPU_ID.amountConf, "0.08")
    conf.set(WORKER_GPU_ID.amountConf, "1")
    conf.set(WORKER_GPU_ID.discoveryScriptConf, scriptPath)
    conf.set(EXECUTOR_GPU_ID.amountConf, "1")
    sc = new SparkContext(conf)
    val rdd = sc.range(0, 100, 1, 4)
    var rdd1 = rdd.repartition(3)
    val treqs = new TaskResourceRequests().cpus(1).resource("gpu", 1.0)
    val rp = new ResourceProfileBuilder().require(treqs).build
    rdd1 = rdd1.withResources(rp)
    assert(rdd1.collect().size === 100)
  }
} {code}
In the above test, the 3 tasks generated by rdd1 are expected to be executed in sequence as we expect "new TaskResourceRequests().cpus(1).resource("gpu", 1.0)" should override "conf.set(TASK_GPU_ID.amountConf, "0.08")". However, those 3 tasks are run in parallel in fact.

The root cause is that ExecutorData#ExecutorResourceInfo#numParts is static. In this case, the "gpu.numParts" is initialized with 12 (1/0.08) and won't change even if there's a new task resource request (e.g., resource("gpu", 1.0) in this case). Thus, those 3 tasks are able to be executed in parallel.
Bug: Upgrade jetty to  9.4.53.v20231009
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Oct/23 12:58
Affects Version/s: 4.0.0

fix 
 * [CVE-2023-44487|https://github.com/advisories/GHSA-qppj-fm5r-hxr3]
 * [CVE-2023-36478|https://github.com/advisories/GHSA-wgh7-54f2-x98r]
Bug: Make InheritableThread inherit the active session
Status: Open | Priority: Major | Resolution: 
Created: 04/Oct/23 17:21
Affects Version/s: 3.5.0

Repro:

{code:java}
# repro.py
from multiprocessing.pool import ThreadPool
from pyspark import inheritable_thread_target
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Test").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

def f(i, spark):
    print(f"{i} spark = {spark}")
    print(f"{i} active session = {SparkSession.getActiveSession()}")
    print(f"{i} local property foo = {spark.sparkContext.getLocalProperty('foo')}")
    spark = SparkSession.builder.appName("Test").getOrCreate()
    print(f"{i} spark = {spark}")
    print(f"{i} active session = {SparkSession.getActiveSession()}")

pool = ThreadPool(4)
spark.sparkContext.setLocalProperty("foo", "bar")
pool.starmap(inheritable_thread_target(f), [(i, spark) for i in range(4)]){code}

Run as: {{./bin/spark-submit repro.py}}

{{getOrCreate()}} doesn't set the active session either. The only way is calling the Java function directly: {{spark._jsparkSession.setActiveSession(spark._jsparkSession)}}.
Bug: Ensure the continuity of rolling log index
Status: Open | Priority: Major | Resolution: 
Created: 26/Oct/23 03:21
Affects Version/s: 3.5.0

Current the log file index will increment before running `initLogFile()`. When running `rollEventlogFile()`, the index will increment first, then create new log file to init event log writer.
 
If the log file creation fails, `initLogFile()` will throw an exception and stop running the method. The log file index will still increment next time `rollEventLogFile()` is called,
which will cause the file index to become discontinuous. EventLogFileReader can not read the event log files normally.
 
Therefore, we need to update the logic here to ensure the continuity of rolling log index
Bug: Upgrade dropwizard metrics 4.2.21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/23 19:28
Affects Version/s: 4.0.0

[https://github.com/dropwizard/metrics/releases/tag/v4.2.21]

[https://github.com/dropwizard/metrics/releases/tag/v4.2.20]
Bug:  Auto repartition before writing data into partitioned or bucket table
Status: Open | Priority: Major | Resolution: 
Created: 18/Oct/23 15:03
Affects Version/s: 4.0.0

Now, when writing data into partitioned table, there will be at least *dynamicPartitions * ShuffleNum* files; when writing data into bucket table, there will be at least *bucketNums * shuffleNum* files.
We can shuffle by the dynamic partitions or bucket columns before writing data into the table and will create ShuffleNum files.
Bug: Converting timestamp_ntz to array<timestamp_ntz> can cause NPE or SEGFAULT on parquet vectorized reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/23 07:47
Affects Version/s: 3.5.0

Repro:

```

val path = "/tmp/sample_parquet_file"

spark.sql("SELECT CAST('2019-01-01' AS TIMESTAMP_NTZ) AS field").write.parquet(path)
spark.read.schema("field ARRAY<TIMESTAMP_NTZ>").parquet(path).collect()

```

Depending on the memory mode, it will throw an NPE on OnHeap mode and SEGFAULT on OffHeap mode.
Bug: Collapse repartition operators with a project
Status: Open | Priority: Major | Resolution: 
Created: 19/Oct/23 08:44
Affects Version/s: 4.0.0

We can collapse two repartition operators with a project between them.

For example:

df.repartition($"a").select($"a", $"b", $"a" + $"b").repartition($"b")

is same to

df.select($"a", $"b", $"a" + $"b").repartition($"b")
Bug: The last Task may get stuck in multi resource-profile
Status: Open | Priority: Major | Resolution: 
Created: 13/Oct/23 08:16
Affects Version/s: 4.0.0

Description:
In scenarios involving multiple resource profiles (e.g., prof1 and prof2), when a taskset (prof1) has only one remaining task (task0) awaiting scheduling, and there are executors (executor0 with prof0 and executor1 with prof1), if executor1 fails to run task0, executor1 gets blacklisted. Consequently, task0 becomes unschedulable, leading to a blockage in the task scheduling process.

Example Code:


{code:java}
val rprof = new ResourceProfileBuilder()
val ereqs = new ExecutorResourceRequests()
ereqs.memory("4g")
ereqs.memoryOverhead("2g").offHeapMemory("1g")
val resourceProfile = rprof.require(ereqs).build()
val rdd = sc.parallelize(1 to 10, 1).withResources(resourceProfile)
rdd.map(num => {
  if (TaskContext.get().attemptNumber() == 0) {
    throw new RuntimeException("First attempt encounters an error")
  } else {
    num / 2
  }
}).collect()
{code}

Issue:
The issue arises when the taskSet becomes unschedulable. The logic attempts to find a task that cannot be scheduled on any executor across all profiles. However, when determining whether an executor can schedule the task, there is no distinction made based on the resource profile. This leads to an incorrect assumption that executor0 (prof0) can schedule the task, which is not the case.

Relevant Code:

{code:java}
if (!launchedAnyTask) {
  taskSet.getCompletelyExcludedTaskIfAny(hostToExecutors) ........
}

def getCompletelyExcludedTaskIfAny(
      hostToExecutors: HashMap[String, HashSet[String]]): Option[Int] = {
........
  pendingTask.find { indexInTaskSet =>
    // try to find some executor this task can run on.
    // It's possible that some *other* task isn't schedulable anywhere,
    // but we will discover that in some later call, when that unschedulable task is the last task remaining.
    hostToExecutors.forall { case (host, execsOnHost) =>
      // Check if the task can run on the node
      val nodeExcluded =
        appHealthTracker.isNodeExcluded(host) ||
          taskSetExcludelist.isNodeExcludedForTaskSet(host) ||
          taskSetExcludelist.isNodeExcludedForTask(host, indexInTaskSet)
      if (nodeExcluded) {
        true
      } else {
        // Check if the task can run on any of the executors
        execsOnHost.forall { exec =>
          appHealthTracker.isExecutorExcluded(exec) ||
            taskSetExcludelist.isExecutorExcludedForTaskSet(exec) ||
            taskSetExcludelist.isExecutorExcludedForTask(exec, indexInTaskSet)
        }
      }
    }
  }
}
{code}