av9ash/GitBugs huggingface dataset:
Bug: Make `BlockManager` warn before `removeBlockInternal`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/24 18:45
Affects Version/s: 4.0.0

{code}
24/03/18 18:40:46 WARN BlockManager: Putting block broadcast_0 failed due to exception java.nio.file.NoSuchFileException: /data/spark/blockmgr-56a6c418-90be-4d89-9707-ef45f7eaf74c/0e.
24/03/18 18:40:46 WARN BlockManager: Block broadcast_0 was not removed normally.
24/03/18 18:40:46 INFO TaskSchedulerImpl: Cancelling stage 0
24/03/18 18:40:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
24/03/18 18:40:46 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) failed in 0.264 s due to Job aborted due to stage failure: Task serialization failed: java.nio.file.NoSuchFileException: /data/spark/blockmgr-56a6c418-90be-4d89-9707-ef45f7eaf74c/0e
java.nio.file.NoSuchFileException: /data/spark/blockmgr-56a6c418-90be-4d89-9707-ef45f7eaf74c/0e
{code}
Bug: Support trailing commas in select list
Status: Open | Priority: Major | Resolution: 
Created: 15/Mar/24 23:39
Affects Version/s: 4.0.0

DuckDb has popularized allowing trailing commas in the SELECT list.
The benefit of this ability is that it is easy to add, remove, comment out expressions from the select list:
{noformat}

SELECT c1,
             /* c2 */
   FROM T;

vs
SELECT  c1
        /* ,  c2 */
   FROM T; 
{noformat}
Recently Snowflake jumped onto this usability feature as well.
Bug: spark-sql does not recognize expressions in repartition hint
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 15/Mar/24 21:02
Affects Version/s: 3.4.1

In Scala, it is possible to do this, to create a bucketed table to not have many small files.

 
{code:scala}
df.repartition(expr("pmod(hash(user_id), 200)"))
  .write
  .mode(SaveMode.Overwrite)
  .bucketBy(200, "user_id")
  .option("path", output_path)
  .saveAsTable("bucketed_table")
{code}
Found [this small trick|https://towardsdatascience.com/best-practices-for-bucketing-in-spark-sql-ea9f23f7dd53] to have the same # files as buckets.

However, the equivalent does not work in spark-sql (using repartition hint)
{code:sql}
create table bucketed_table stored as parquet
clustered by (user_id) into 200 buckets
select /*+ repartition (pmod(hash(user_id),200)) */ * from df_table
{code}
{{REPARTITION Hint parameter should include columns, but 'pmod('hash('user_id), 200) found.}}

When I instead make a virtual column and use that, Spark is not respecting the repartition anymore
{code:sql}
create table bucketed_table stored as parquet
clustered by (user_id) into 200 buckets
select /*+ repartition (bkt) */
*, pmod(hash(user_id),200) as bkt
from df_table
{code}
{code:bash}
$ hdfs dfs -ls -h /user/spark/warehouse/bucket_test.db/bucketed_table| head
Found 101601 items
...
{code}

The same behavior is seen even with DISTRIBUTE BY clause

{code:sql}
create table bucketed_table stored as parquet
clustered by (user_id) into 200 buckets
select *
from df_table
distribute by pmod(hash(user_id),200)
{code}

Can the behavior of repartition hint be changed to work like the Scala/Python equivalent?

Thank you
Bug: Remove the test case workaround for JDK 8
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/24 09:51
Affects Version/s: 4.0.0

Spark SQL test case in ExpressionEncoderSuite fails in windows operation system.

{code:java}
Internal error (java.io.FileNotFoundException): D:\Users\gja\git-forks\spark\sql\catalyst\target\scala-2.13\test-classes\org\apache\spark\sql\catalyst\encoders\ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$OuterLevelWithVeryVeryVeryLongClassName2$OuterLevelWithVeryVeryVeryLongClassName3$OuterLevelWithVeryVeryVeryLongClassName4$OuterLevelWithVeryVeryVeryLongClassName5$OuterLevelWithVeryVeryVeryLongClassName6$.class (文件名、目录名或卷标语法不正确。)
java.io.FileNotFoundException: D:\Users\gja\git-forks\spark\sql\catalyst\target\scala-2.13\test-classes\org\apache\spark\sql\catalyst\encoders\ExpressionEncoderSuite$OuterLevelWithVeryVeryVeryLongClassName1$OuterLevelWithVeryVeryVeryLongClassName2$OuterLevelWithVeryVeryVeryLongClassName3$OuterLevelWithVeryVeryVeryLongClassName4$OuterLevelWithVeryVeryVeryLongClassName5$OuterLevelWithVeryVeryVeryLongClassName6$.class (文件名、目录名或卷标语法不正确。)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at com.intellij.openapi.util.io.FileUtil.loadFileBytes(FileUtil.java:211)
	at org.jetbrains.jps.incremental.scala.local.LazyCompiledClass.$anonfun$getContent$1(LazyCompiledClass.scala:18)
	at scala.Option.getOrElse(Option.scala:201)
	at org.jetbrains.jps.incremental.scala.local.LazyCompiledClass.getContent(LazyCompiledClass.scala:17)
	at org.jetbrains.jps.incremental.instrumentation.BaseInstrumentingBuilder.performBuild(BaseInstrumentingBuilder.java:38)
	at org.jetbrains.jps.incremental.instrumentation.ClassProcessingBuilder.build(ClassProcessingBuilder.java:80)
	at org.jetbrains.jps.incremental.IncProjectBuilder.runModuleLevelBuilders(IncProjectBuilder.java:1569)
	at org.jetbrains.jps.incremental.IncProjectBuilder.runBuildersForChunk(IncProjectBuilder.java:1198)
	at org.jetbrains.jps.incremental.IncProjectBuilder.buildTargetsChunk(IncProjectBuilder.java:1349)
	at org.jetbrains.jps.incremental.IncProjectBuilder.buildChunkIfAffected(IncProjectBuilder.java:1163)
	at org.jetbrains.jps.incremental.IncProjectBuilder$BuildParallelizer$1.run(IncProjectBuilder.java:1129)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:244)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:30)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.executeFirstTaskAndHelpQueue(BoundedTaskExecutor.java:222)
	at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:218)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:210)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:842)
{code}
Bug: Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/24 23:38
Affects Version/s: 4.0.0

In the new operator for arbitrary state-v2, we cannot rely on the session/encoder being available since the initialization for the various state instances happens on the executors. Also, we can only support limited state types with the available encoders. Hence, for the state serialization, we propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging SQL encoder can speed up the serialization.
Bug: spark jdbc one insert with multiple values
Status: Open | Priority: Major | Resolution: 
Created: 14/Mar/24 08:25
Affects Version/s: 4.0.0

Many databases support a single insert sql to write multiple rows of data. Write performance is more efficient than batch execution of multiple sql files

 

https://github.com/apache/spark/blob/9986462811f160eacd766da8a4e14a9cbb4b8710/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L725

 

example:

 
{code:java}
INSERT INTO Customers (Name, Age, Active) ('Name1',21,1) INSERT INTO Customers (Name, Age, Active) ('Name2',21,1)
Vs
INSERT INTO Customers (Name, Age, Active) ('Name1',21,1), ('Name2',21,1)
{code}
Bug: Tuple encoder produces wrong results with Option inputs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/24 02:31
Affects Version/s: 3.3.4

 

The behavior of tupled encoders on the Option type was changed by https://github.com/apache/spark/pull/40755.
{code:java}
import org.apache.spark.sql.{Encoders, Encoder} 

case class Required(name: String) 
case class Optional(name: String) 

implicit val enc: Encoder[(Required, Option[Optional])] = Encoders.tuple(Encoders.product[Required], Encoders.product[Option[Optional]]) 
 
spark.createDataFrame(Seq( 
(Required("1"), Some(Optional("1"))), 
(Required("2"), None) 
)).as[(Required, Option[Optional])].collect(){code}
Before the PR, the result is:
{code:java}
Array((Required(1),Some(Optional(1))), (Required(2),None)){code}
After the PR, the result is:
{code:java}
Array((Required(1),Some(Optional(1))), (Required(2),null)) {code}
which is incorrect because the original input is None rather than null.
Bug: Fix performance regression in JDK 17 caused from RocksDB logging
Status: Open | Priority: Major | Resolution: 
Created: 12/Mar/24 21:04
Affects Version/s: 3.3.0

JDK 17 has a performance regression in the JNI's AttachCurrentThread and DetachCurrentThread calls, as reported here: [https://bugs.openjdk.org/browse/JDK-8314859]. You can find a minimal reproduction of the JDK issue in that bug report. I have marked as affected versions 3.3.0^ since that is when JDK 17 started being offered in Spark.

For context, every time RocksDB logs, it currently [attaches itself to the JVM|https://github.com/facebook/rocksdb/blob/main/java/rocksjni/loggerjnicallback.cc#L140], invokes the RocksDB [logging callback that we specify|https://github.com/apache/spark/blob/8fcef1657a02189f91d5485eabb5b165706cdce9/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L839], and then [detaches itself from the JVM|https://github.com/facebook/rocksdb/blob/main/java/rocksjni/loggerjnicallback.cc#L170]. These attach/detach calls regressed, causing JDK 17 SS queries to run up to 10-15% slower than their respective JDK 8 queries.

For example, a 100K record/second dropDuplicates had a p95 latency regression of 12%. A regression of 12% and 21% (at the p95) was observed for a query with 1M record/second, 100K keys, 10 second windows, and 0 second watermark.

Because the Hotspot folks marked this as "Won't fix," one way to fix this is to avoid the JNI entirely and write the RocksDB to stderr. RocksDB [8.11.3 natively supports this|https://github.com/facebook/rocksdb/wiki/Logging-in-RocksJava#configuring-a-native-logger] (I implemented that feature in RocksJava). We can configure our RocksDB logger to do its logging this way.
Bug: Execute immediate throws NPE when `sqlString` variable value is null
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/24 13:38
Affects Version/s: 4.0.0

Execute immediate throws NPE when `sqlString` variable value is null

 

`EXECUTE IMMEDIATE sqlString` throws NPE when `sqlString` is null.

Throw proper error when variable value is null
Bug: Remove inferTimestampNTZ config check in ParquetRowConverter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 20:55
Affects Version/s: 3.4.0

The configuration `spark.sql.parquet.inferTimestampNTZ.enabled` is not related the parquet row converter. We should remove the config check in the ParquetRowConverter, otherwise reading TimestampNTZ columns may fail when `spark.sql.parquet.inferTimestampNTZ.enabled` is disabled.
Bug: Change "collate" in StringType typename to lowercase
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/24 10:48
Affects Version/s: 4.0.0

Change [https://github.com/apache/spark/blob/37bdf5179e01f6c8cb8c83445a155646794aabef/sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala#L50] "COLLATE" to lowercase in order to follow convention. Example from DecimalType https://github.com/apache/spark/blob/2a51242b1bdb4a1d469fc81dc0d21c4cde305753/sql/api/src/main/scala/org/apache/spark/sql/types/DecimalType.scala#L59-L63.
Bug: new test suite for UTF8String
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/24 11:53
Affects Version/s: 4.0.0

{{CollationSuite}} is a bit over cluttered and it should be used for for E2E testing. Create {{UTF8StringSuiteWithCollation}} suite which should contain unit string-level tests for {{startsWith}} and {{endsWith}} functions.

 

Changes originally proposed in [this PR|https://github.com/apache/spark/pull/45421#discussion_r1519451854].
Bug: MapState Implementation for State V2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 20:03
Affects Version/s: 4.0.0

This task adds changes for MapState implementation in State Api v2. This implementation adds a new encoder/decoder to encode grouping key and user key into a composite key to be put into RocksDB so that we could retrieve key-value pair by user specified user key by one rocksdb get.
Bug: Enhance the console sink to provide watermark and state information
Status: Open | Priority: Major | Resolution: 
Created: 12/Mar/24 16:03
Affects Version/s: 4.0.0

As [discussed in the dev mailing list|https://lists.apache.org/thread/3wd23w0tldk3d8dbjznc11ybc1p3v3hh], we should enhance the console sink for Structured Streaming to additionally provide information about:
 * The stream's watermark at the end of the batch
 * The rows in state at the end of each batch

This will be enabled via an `option` on the sink. Since both of these additions are for stateful queries only, the option will not affect stateless queries. To make parsing the output easier, timestamps will be duration-rendered (i.e. "1 second" instead of the ISO 8601 extened timestamp). For joins, just the KeyWithIndexToValue will be shown. If there are multiple stateful operators, we'll print a state table for each.

These considerations are up to discussion either in this thread or in the PR.
Bug: Moving tests to related Suites
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Mar/24 11:58
Affects Version/s: 4.0.0

*What changes were proposed in this pull request?*
Tests from `QueryCompilationErrorsSuite` were moved to `DDLSuite` and `JDBCTableCatalogSuite`.

*Why are the changes needed?*
We should move tests to related test suites in order to improve testing.
Bug: DataType __repr__ change breaks datatype checking (anit-)pattern
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 17:26
Affects Version/s: 3.4.1

This pr: [https://github.com/apache/spark/pull/34320]

Made reprs for datatype eval-able. This is kind of nice, but we have a ton of users doing stuff like:

 
{code:java}
if str(data_type) == "StringType":
   ...
{code}
 

Which breaks.

 

What would people think of adding a __str__ to the base class that returns the old behaviour so we can have the best of both worlds.
Bug: SQL identifiers must not start with digit
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/24 01:49
Affects Version/s: 4.0.0

In Spark SQL, we accept identifiers as `[a-zA-Z0-9_]+`, with a few exceptions:
 * It cannot be all digits
 * It cannot be special number literals like 0d, 0f

The second exception is a bit fragile. What if we add new number literals in the future? It will be a breaking change every single time. In addition, most mainstream databases do not allow these as identifiers can not start with a digit. We should follow them and unify the exceptions: the identifier can not start with digit.
Bug: taskSummary API should include the task state as a parameter
Status: Open | Priority: Major | Resolution: 
Created: 07/Mar/24 10:49
Affects Version/s: 3.5.1

After the JIRA, https://issues.apache.org/jira/browse/SPARK-26119, task summary only contain successful task.

In our scenario, utilize the REST API [/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary] to obtain the taskSummary metric. Use the Max(schedule delay) to diagnose why the stage duration is excessively long. but in some case the slowest task is a failed state task, so task summary metrics is not included this task.

expect：

the rest api [/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary] task state as a parameter.
Bug: LATERAL regresses correlation name resolution
Status: Open | Priority: Major | Resolution: 
Created: 06/Mar/24 19:53
Affects Version/s: 3.3.0

{code:java}
CREATE TABLE persons(name STRING);
INSERT INTO persons VALUES('Table: persons');
CREATE OR REPLACE TABLE women(name STRING);
INSERT INTO women VALUES('Table: women');

-- This works:
SELECT (SELECT max(folk.id) 
          FROM persons AS men(id),
       (SELECT name) AS folk(id))
  FROM women;
Table: women

-- This does not:
SELECT (SELECT max(folk.id) 
          FROM persons AS men(id), 
        LATERAL (SELECT name) AS folk(id))
  FROM women;
[UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name `name` cannot be resolved.  SQLSTATE: 42703;{code}

This is weird. LATERAL should be strictly additive to name resolution rules.
Bug: Explicit repartitioning is ignored when reading bucketed table but with bucketing disabled
Status: Open | Priority: Major | Resolution: 
Created: 06/Mar/24 13:32
Affects Version/s: 3.4.2

When reading a bucketed table with `spark.sql.sources.bucketing.autoBucketedScan.enabled = true` (default), Spark may decide to not use the bucketing during the read. However it seems like some code may still optimize e.g. an exchange away because there is bucketing, making the exchange "unnecessary".

Maybe one optimization rule first removes the Exchange thinking that the bucketing provides the same distribution, and then after the DisableUnnecessaryBucketedScan rule disables the bucketing as there is no more Exchange?

See the below for repro, tested on 3.5.1, 3.5.0 and 3.4.2:
{noformat}
val df_no_buckets = spark.range(10)
df_no_buckets.write.bucketBy(4, "id").saveAsTable("df_buckets")
val df_buckets = spark.read.table("df_buckets")

# Repartitioning the non-bucketed table works as expected
scala> df_no_buckets.repartition(4, col("id")).explain() 
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Exchange hashpartitioning(id#0L, 4), REPARTITION_BY_NUM, [plan_id=24]
   +- Range (0, 10, step=1, splits=12)


scala> df_no_buckets.repartition(4, col("id")).rdd.getNumPartitions
res2: Int = 4

# But repartition is lost when reading a bucketed table, leading to a wrong number of partitions in the end
scala> df_buckets.repartition(4, col("id")).explain()
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- FileScan parquet spark_catalog.default.df_buckets[id#3L] Batched: true, Bucketed: false (disabled by query planner), DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>

scala> df_buckets.repartition(4, col("id")).rdd.getNumPartitions
res4: Int = 10 

# After disabling auto bucketed scan, the repartition stays away but output partitioning is correct as the read keeps the bucketing
scala> spark.conf.set("spark.sql.sources.bucketing.autoBucketedScan.enabled", false)

scala> df_buckets.repartition(4, col("id")).explain()
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- FileScan parquet spark_catalog.default.df_buckets[id#3L] Batched: true, Bucketed: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 4 out of 4

scala> df_buckets.repartition(4, col("id")).rdd.getNumPartitions
res7: Int = 4{noformat}
Bug: OptimizeSkewInRebalanceRepartitions should support ProjectExec(_,ShuffleQueryStageExec)
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 06/Mar/24 04:03
Affects Version/s: 3.5.1

Current OptimizeSkewInRebalanceRepartitions only support match ShuffleQueryStageExec, this case only support SQL query, can't work when insert since there have a project between ShuffleQueryStageExec and insert command
{code:java}
plan transformUp {
  case p @ ProjectExec(_, stage: ShuffleQueryStageExec) if isSupported(stage.shuffle) =>
    p.copy(child = tryOptimizeSkewedPartitions(stage))
  case stage: ShuffleQueryStageExec if isSupported(stage.shuffle) =>
    tryOptimizeSkewedPartitions(stage)
} {code}
Bug: Add Parquet file reader metrics to scan
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 20:28
Affects Version/s: 4.0.0

With the addition of external metrics support in Parquet ([PARQUET-2374|https://issues.apache.org/jira/browse/PARQUET-2374]), it is now possible to gather file level scan metrics and have them displayed in a Parquet Scan's metrics. This can be done for both DSV1 and DSV2 implementations by providing a ParquetMetricsCallback implementation
Bug: We should ensure enough parallelism when ShuffleExchangeLike join with specs without shuffle
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 09:28
Affects Version/s: 4.0.0

The following case is introduced by https://issues.apache.org/jira/browse/SPARK-35703

// When choosing specs, we should consider those children with no `ShuffleExchangeLike` node
// first. For instance, if we have:
// A: (No_Exchange, 100) <---> B: (Exchange, 120)
// it's better to pick A and change B to (Exchange, 100) instead of picking B and insert a
// new shuffle for A.


*But we'd better improve it in some cases, for example:*
A: (No_Exchange, 2) <---> B: (Exchange, 100)

The current logic will change to:
A: (No_Exchange, 2) <---> B: (Exchange,2)

It actually not ensure enough parallelism, it will reduce the performance i think.
Bug: 'parseTableIdentifier' fails when a catalog name is provided
Status: Open | Priority: Major | Resolution: 
Created: 05/Mar/24 08:51
Affects Version/s: 3.5.0

{code:scala}
spark.sessionState.sqlParser.parseTableIdentifier(
  "`my catalog`.`my database`.`my table`"
)
{code}
fails with
{code:scala}
org.apache.spark.sql.catalyst.parser.ParseException: 
[PARSE_SYNTAX_ERROR] Syntax error at or near '.'.(line 1, pos 26)

== SQL ==
`my catalog`.`my database`.`my table`
--------------------------^^^

	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(parsers.scala:257)
	at org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:98)
	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:54)
	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableIdentifier(AbstractSqlParser.scala:41)
{code}
 

Note: It works as expected on Databricks clusters (verified with Spark 3.3.2 and 3.5.0).
Bug: Bump ap-loader 3.0(v8) to support for async-profiler 3.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/24 06:31
Affects Version/s: 4.0.0

ap-loader 3.0(v8) has already been released, which supports for async-profier 3.0. The release guide refers to [Loader for 3.0 (v8): Binary launcher and AsyncGetCallTrace replacement|https://github.com/jvm-profiling-tools/ap-loader/releases/tag/3.0-8].
Bug: Upgrade jetty to 11.0.20
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/24 11:30
Affects Version/s: 4.0.0

fix 
 * [CVE-2024-22201|https://github.com/advisories/GHSA-rggv-cv7r-mw98] - HTTP/2 connection not closed after idle timeout when TCP congested
Bug: All executions end up being put in the abandoned tombstone cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/24 12:59
Affects Version/s: 3.5.2

All executions upon close end up in the abandoned tombstone cache due to a missing `if` guard [here|https://github.com/apache/spark/blob/262ed5bcab0ba750b089b0693dbb1a59ef6fd11f/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala#L124]