av9ash/GitBugs huggingface dataset:
Bug: OOM killer may leave SparkContext in broken state causing ConnectionRefusedError
Status: Open | Priority: Major | Resolution: 
Created: 25/Jun/24 10:33
Affects Version/s: 3.5.0

Related to https://issues.apache.org/jira/browse/SPARK-18523, and https://github.com/apache/spark/pull/15961. I'm currently on:

{code}
pyspark                   3.5.0              pyhd8ed1ab_0    conda-forge
py4j                      0.10.9.7           pyhd8ed1ab_0    conda-forge
{code}

When Spark JVM process gets OOM-Killed, `SparkContext.stop` fails with `ConnectionRefusedError`, which leaves the `SparkSession/Context` in a "dirty" state. https://issues.apache.org/jira/browse/SPARK-18523 addressed this by catching the {{Py4JError}} it looks like the code now raises {{ConnectionRefusedError}}:

{code}
Traceback (most recent call last):
  ...
  File "<TRUNC>/lib/python3.11/site-packages/pyspark/sql/session.py", line 1796, in stop
    self._sc.stop()
  File "<TRUNC>/lib/python3.11/site-packages/pyspark/context.py", line 654, in stop
    self._jsc.stop()
  File "<TRUNC>/lib/python3.11/site-packages/py4j/java_gateway.py", line 1321, in __call__
    answer = self.gateway_client.send_command(command)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<TRUNC>/lib/python3.11/site-packages/py4j/java_gateway.py", line 1036, in send_command
    connection = self._get_connection()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "<TRUNC>/lib/python3.11/site-packages/py4j/clientserver.py", line 284, in _get_connection
    connection = self._create_new_connection()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<TRUNC>/lib/python3.11/site-packages/py4j/clientserver.py", line 291, in _create_new_connection
    connection.connect_to_java_server()
  File "<TRUNC>/lib/python3.11/site-packages/py4j/clientserver.py", line 438, in connect_to_java_server
    self.socket.connect((self.java_address, self.java_port))
ConnectionRefusedError: [Errno 111] Connection refused
{code}
Bug: Patch executor pod without optimistic locking
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/24 10:00
Affects Version/s: 3.5.0

With fabric8 version 6.5, Apache Spark patch executor pod with optimistic locking, which means requestObject of patch contains resourceVersion. 
Other components attempting to issue patch requests to the executor pod during this period will encounter error 409 conflicts.
Bug: Arrow python UDFS didn't support UDT as output type
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/24 03:18
Affects Version/s: 3.4.3

{code:java}
df.select(udf(lambda x: x, returnType=ExamplePointUDT(), useArrow=useArrow)("point")), {code}
 
{code:java}
 java.lang.AssertionError: assertion failed: Invalid schema from pandas_udf: expected org.apache.spark.sql.test.ExamplePointUDT@49ccc723, StructType(StructField(st,StructType(StructField(tt,TimestampType,true)),true)), got ArrayType(DoubleType,false)
 {code}
Bug: K8s resource name prefix follows DNS Subdomain Names rule
Status: Open | Priority: Major | Resolution: 
Created: 20/Jun/24 07:57
Affects Version/s: 3.5.1

In SPARK-39614, we extended the allowed name length from 63 to 253 for executor pod and config map.

However, when the pod name is exceeded length 253, we don't truncate it. This leads to error when creating the Spark pods.

Error example:
{code:java}
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/foo/pods. Message: Pod "some-super-long-spark-pod-name-exceeded-length-253-driver" is invalid: metadata.name: Invalid value: "some-super-long-spark-pod-name-exceeded-length-253-driver": must be no more than 253 characters. {code}
Bug: Remove ExplainUtils.processPlan synchronize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/24 00:45
Affects Version/s: 4.0.0

[https://github.com/apache/spark/pull/45282] introduced synchronize to `ExplainUtils.processPlan`  to avoid race condition when multiple queries refers to same cached plan.

The granularity of lock is too large. We can try to fix the root cause of this concurrency issue by refactoring the usage of mutable OP_ID_TAG, which is not a good practice in terms of immutable nature of SparkPlan.  Instead, we can use an auxiliary id map, with object identity as the key.
Bug: Add consistency check and fallback for mapIds in push-merged block meta
Status: Open | Priority: Major | Resolution: 
Created: 11/Jun/24 02:15
Affects Version/s: 3.2.0

When push-based shuffle enabled, 0.03% of the spark application in our cluster experienced shuffle data loss. The metrics of Exchange as follows:

!image-2024-06-11-10-19-57-227.png|width=405,height=170!

We eventually found some WARN logs on the shuffle server:
 
{code:java}
WARN shuffle-server-8-216 org.apache.spark.network.shuffle.RemoteBlockPushResolver: Application application_xxxx shuffleId 0 shuffleMergeId 0 reduceId 133 update to index/meta failed{code}
 

And analyzed the cause from the code：

The merge metadata obtained by the reduce side from the driver comes from the {{mapTracker}} in the server's memory, while the actual reading of chunk data is based on the records in the shuffle server's {{{}metaFile{}}}. There is no consistency check between the two.
Bug: Fix GROUP BY MapType feature
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/24 12:50
Affects Version/s: 3.4.3

There are issues in the current implementation, which causes broken plans to be produced:
 * The InsertMapSortInGroupingExpressions transformation only replaces the grouping expressions with mapsort, not the aggregate expressions. This causes broken plans like select c from t group by mapsort(c), where c now becomes a non-aggregating expression.

 * Parent operators referencing the transformed columns are not updated, which means that we know have broken references.
Bug: Perf Improvement for Encode with empty string and UTF-8 charset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 12:24
Affects Version/s: 4.0.0

 Apple M2 Max
 encode:                                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
 ------------------------------------------------------------------------------------------------------------------------
-UTF-8                                              3672           3697          22          5.4         183.6       1.0X
+UTF-8                                             79270          79698         448          0.3        3963.5       1.0X
Bug: [Arbitrary State Support] Remove TimeMode None
Status: Open | Priority: Major | Resolution: 
Created: 03/Jun/24 15:04
Affects Version/s: 4.0.0

A structured streaming query works in either Processing time mode, or event Time mode depending on whether eventTime has been specified, the presence of TImeMode.None is confusing. This Jira targets removing TimeMode None from the experimental API. 

Note that if eventTimeColumn is specified for output dataset in transformWithState operator. TimeMode defaults to EventTime.
Bug: Add opt-in flag to have SparkSubmit automatically call System.exit after user code main method exits
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/24 23:14
Affects Version/s: 4.0.0

This PR proposes to add a new flag, `spark.submit.callSystemExitOnMainExit` (default false), which when true will instruct SparkSubmit to call System.exit() in the JVM once the user code's main method has exited (for Java / Scala jobs) or once the user's Python or R script has exited.

This is intended to address a longstanding issue where SparkSubmit invocations might hang after user code has completed:

[According to Java’s java.lang.Runtime docs|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Runtime.html#shutdown]:
{quote}The Java Virtual Machine initiates the _shutdown sequence_ in response to one of several events:
 # when the number of [live|https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Thread.html#isAlive()] non-daemon threads drops to zero for the first time (see note below on the JNI Invocation API);

 # when the {{Runtime.exit}} or {{System.exit}} method is called for the first time; or

 # when some external event occurs, such as an interrupt or a signal is received from the operating system.{quote}
For Python and R programs, SparkSubmit’s PythonRunner and RRunner will call {{System.exit()}} if the user program exits with a non-zero exit code (see [python|https://github.com/apache/spark/blob/d5c33c6bfb5757b243fc8e1734daeaa4fe3b9b32/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala#L101-L104] and [R|https://github.com/apache/spark/blob/d5c33c6bfb5757b243fc8e1734daeaa4fe3b9b32/core/src/main/scala/org/apache/spark/deploy/RRunner.scala#L109-L111] runner code).

But for Java and Scala programs, plus any _successful_ R or Python programs, Spark will _not_ automatically call System.exit.

In those situation, the JVM will only shutdown when, via event (1), all non-[daemon|https://stackoverflow.com/questions/2213340/what-is-a-daemon-thread-in-java] threads have exited (unless the job is cancelled and sent an external interrupt / kill signal, corresponding to event (3)).

Thus, *non-daemon* threads might cause logically-completed spark-submit jobs to hang rather than completing.

The non-daemon threads are not always under Spark's own control and may not necessarily be cleaned up by SparkContext.stop().

Thus, it is useful to have an opt-in functionality to have SparkSubmit automatically call `System.exit()` upon main method exit (which usually, but not always, corresponds to job completion): this option will allow users and data platform operators to enforce System.exit() calls without having to modify individual jobs' code.
Bug: Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/24 21:49
Affects Version/s: 4.0.0

[kubernetes-client v6.13.0|https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.0]
Bug: Support analyze column stats for tables with collated columns
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/24 12:52
Affects Version/s: 4.0.0

Following sequence fails:
{code:java}
> create table t(s string collate utf8_lcase) using parquet;
> insert into t values ('A');
> analyze table t compute statistics for all columns;
[UNSUPPORTED_FEATURE.ANALYZE_UNSUPPORTED_COLUMN_TYPE] The feature is not supported: The ANALYZE TABLE FOR COLUMNS command does not support the type "STRING COLLATE UTF8_LCASE" of the column `s` in the table `spark_catalog`.`default`.`t`. SQLSTATE: 0A000
 {code}
Users should be able to run ANALYZE commands on tables which have columns with collated type.
Bug: Add task peak on/off heap execution memory metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/24 07:45
Affects Version/s: 4.0.0

Currently there is no task on/off heap execution memory metrics. There is a [peakExecutionMemory|https://github.com/apache/spark/blob/3cd35f8cb6462051c621cf49de54b9c5692aae1d/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala#L114]  metrics, however, the semantic is a confusing: it only cover the execution memory used by shuffle/join/aggregate/sort, which is accumulated in specific operators.

 

We can easily maintain the whole task-level peak memory in TaskMemoryManager, assuming *acquireExecutionMemory* is the only one narrow waist for acquiring execution memory.

 

Also it's nice to cleanup/deprecate that poorly-named `peakExecutionMemory`.

 

Creating two followup sub tickets:
 * https://issues.apache.org/jira/browse/SPARK-48788 :accumulate task metrics in stage, and display in Spark UI
 * https://issues.apache.org/jira/browse/SPARK-48789 : deprecate `peakExecutionMemory` once we have replacement for it.
Bug: Reduce the number of accesses to S3 object storage
Status: Open | Priority: Major | Resolution: 
Created: 08/Jun/24 13:50
Affects Version/s: 3.5.0

If we access a Spark table on an object storage file system with parquet files, the object storage suffers many requests that seem to be unnecessary. To explain this I will do it with an example:

I have created a simple table, with 3 files:

*business/t_filter/country=ES/data_date_part=2023-09-27/part-00000-0f52aae9-2db8-415e-93f3-8331539c0ead.c000*
*business/t_filter/country=ES/data_date_part=2023-06-01/part-00000-0f52aae9-2db8-415e-93f3-8331539c0ead.c000*    
*business/t_filter/country=ES/data_date_part=2023-09-27/part-00000-f10096c1-53bc-4e2f-bc56-eba65acfa44a.c000*    

and I have put a table that represents business/t_filter with country and data_date_part partitions, you have the following requests.

If you use versions prior to Spark 3.5 or Hadoop 3.4, in my case it is exactly Spark 3.2 and Hadoop 3.1, the number of requests you have are the following -> IMAGE Spark 3.2 Hadoop 3.1

In this image we can see all the requests where we can find the following errors:
 * Two HEAD and two LIST are made with the implementation of S3, of the folders where the files are located, which could only be resolved with a single list. This bug has already been resolved in -> HADOOP-18073 -> Result : IMAGE 2 Spark 3.2 Hadoop 3.4
 * For each file, the parquet footing is listed twice. This bug is resolved in ->SPARK-42388 -> Result : IMAGE Spark 3.5 Hadoop 3.1
 * A Head Object is launched twice each time a file is read, this could be reduced by implementing the FileSystem interface so that it could receive the FileStatus that has already been calculated above.
 ** HADOOP-15229
 ** PARQUET-2493
 * The requests could be reduced when reading the parquet footer, since first you have to read the size of the schema and then the schema, which implies two HTTP/HTTPS requests to S3. It would be nice if there was a minimum threshold, for example 100KB, in which, if the file is smaller than that, it would not have to make two requests, and the entire file would be brought, since bringing 100 KB will take less time in one request to bring 8 B in a request and then another request for x KB. Even so, I don't know if this task makes sense.
 ** It would be to change this implementation, with an environment variable, that if it is set to -1 it does the same, but if it has a threshold set, up to that threshold you do not have to call the seek function twice, which repeats a GET Object.  [https://github.com/apache/parquet-java/blob/apache-parquet-1.14.0/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileReader.java]

 

With all these improvements, updating to the latest version of Spark and Hadoop would go from more than 30 requests to 11 in the proposed example.
Bug: Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/24 23:44
Affects Version/s: 4.0.0

This query is not legal and should give an error, but instead we incorrectly allow it and it returns wrong results.
{code:java}
create table x(x1 int, x2 int);
insert into x values (1, 1);
create table y(y1 int, y2 int);
insert into y values (2, 2), (3, 3);

select *, (select count(*) from y where y1 > x1 group by y1) from x; {code}
It returns two rows, even though there's only one row of x.

The correct result is an error: more than one row returned by a subquery used as an expression (as seen in postgres for example)

 

This is a longstanding bug. The bug is in CheckAnalysis in {{{}checkAggregateInScalarSubquery{}}}. It allows grouping columns that are present in correlation predicates, but doesn’t check whether those predicates are equalities -  because when that code was written, non-equality correlation wasn’t allowed. Therefore, it looks like this bug has existed since non-equality correlation was added (~2 years ago).

 

Various other expressions that are not equi-joins between the inner and outer fields hit this too, e.g. `where y1 + y2 = x1 group by y1`.

Another bugged case is if the correlation condition is an equality but it's under another operator like an OUTER JOIN or UNION.
Bug: A filter should not be pushed down if it contains Unevaluable expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/24 22:07
Affects Version/s: 4.0.0

We should avoid pushing down Unevaluable expression as it can cause unexpected failures. For example, the code snippet below (assuming there is a table {{_t_}} with a partition column {{{_}p{_})}}
{code:java}
from pyspark import SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import StringType

import pyspark.sql.functions as f

def getdata(p: str) -> str:
    return "data"

NEW_COLUMN = 'new_column'
P_COLUMN = 'p'

f_getdata = f.udf(getdata, StringType())
rows = spark.sql("select * from default.t")

table = rows.withColumn(NEW_COLUMN, f_getdata(f.col(P_COLUMN)))

df = table.alias('t1').join(table.alias('t2'), (f.col(f"t1.{NEW_COLUMN}") == f.col(f"t2.{NEW_COLUMN}")), how='inner')

df.show(){code}
will cause an error like:
{code:java}
org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot evaluate expression: getdata(input[0, string, true])#16
    at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
    at org.apache.spark.SparkException$.internalError(SparkException.scala:96)
    at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotEvaluateExpressionError(QueryExecutionErrors.scala:66)
    at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval(Expression.scala:391)
    at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval$(Expression.scala:390)
    at org.apache.spark.sql.catalyst.expressions.PythonUDF.eval(PythonUDF.scala:71)
    at org.apache.spark.sql.catalyst.expressions.IsNotNull.eval(nullExpressions.scala:384)
    at org.apache.spark.sql.catalyst.expressions.InterpretedPredicate.eval(predicates.scala:52)
    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$1(ExternalCatalogUtils.scala:166)
    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.$anonfun$prunePartitionsByFilter$1$adapted(ExternalCatalogUtils.scala:165) {code}
Bug: Python foreachBatch streaming query cannot be stopped gracefully after pin thread mode is enabled and is running spark queries 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/24 23:58
Affects Version/s: 4.0.0

Followup of https://issues.apache.org/jira/browse/SPARK-39218

 

It only considered the InterruptedException is thrown when time.sleep(10) is intercepted. But when a spark query is executing:
{code:java}
def func(batch_df, batch_id):
    batch_df.sparkSession.range(10000000).write.saveAsTable("oops")
    print(batch_df.count()) {code}
the actual error would be:

 
{code:java}
py4j.protocol.Py4JJavaError: An error occurred while calling o2141502.saveAsTable.  
: org.apache.spark.SparkException: Job aborted.  
<REDACTED>
<REDACTED>
<REDACTED>
...
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$2(StreamExecution.scala:262)  
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)  
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:262)  
*Caused by: java.lang.InterruptedException  
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1000)*  
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1308)   {code}
We should also add consideration to this scenario
Bug: Fine-grained State Data Source
Status: Open | Priority: Major | Resolution: 
Created: 11/Jun/24 19:25
Affects Version/s: 4.0.0

The current state reader API replays the state store rows from the latest snapshot and newer delta files if any. The issue with this mechanism is that sometimes, the snapshot files could be wrongly constructed, or user want to know the change of state across batches. We need to improve the State Reader so that it can handle a variety of fine-grained requirements. For example, reconstruct a state based on arbitrary snapshot; support CDC mode for state evolution.
Bug: Virtual Column Family for RocksDB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/24 21:35
Affects Version/s: 4.0.0

Introducing virtual column family to RocksDB. We attach an 8-byte-Id prefix as column family identifier for each of the key row that is put into RocksDB. The encoding and decoding of the virtual column family prefix happens at the {{RocksDBKeyEncoder}} layer as we can pre-allocate extra 8 bytes and avoid additional memcpy.
Bug: Wrong Result in regr_slope&regr_intercept Aggregate with Tuples has NULL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/24 03:13
Affects Version/s: 3.4.0

When calculate slope and intercept using regr_slope & regr_intercept aggregate:

(using Java api)
{code:java}
spark.sql("drop table if exists tab");
spark.sql("CREATE TABLE tab(y int, x int) using parquet");
spark.sql("INSERT INTO tab VALUES (1, 1)");
spark.sql("INSERT INTO tab VALUES (2, 3)");
spark.sql("INSERT INTO tab VALUES (3, 5)");
spark.sql("INSERT INTO tab VALUES (NULL, 3)");
spark.sql("INSERT INTO tab VALUES (3, NULL)");
spark.sql("SELECT " +
        "regr_slope(x, y), " +
        "regr_intercept(x, y)" +
        "FROM tab").show(); {code}
Spark result:
{code:java}
+------------------+--------------------+
|  regr_slope(x, y)|regr_intercept(x, y)|
+------------------+--------------------+
|1.4545454545454546| 0.09090909090909083|
+------------------+--------------------+ {code}
The correct answer should be 2.0 and -1.0 obviously.

 

Reason:

In sql/catalyst/expressions/aggregate/linearRegression.scala,

 
{code:java}
case class RegrSlope(left: Expression, right: Expression) extends DeclarativeAggregate
  with ImplicitCastInputTypes with BinaryLike[Expression] {

  private val covarPop = new CovPopulation(right, left)

  private val varPop = new VariancePop(right)
...... {code}
CovPopulation will filter tuples which right *OR* left is NULL

But VariancePop will only filter null right expression.

This will cause wrong result when some of the tuples' left is null (and right is not null).

{*}Same reason with RegrIntercept{*}.

 

A possible fix:
{code:java}
case class RegrSlope(left: Expression, right: Expression) extends DeclarativeAggregate
  with ImplicitCastInputTypes with BinaryLike[Expression] {

  private val covarPop = new CovPopulation(right, left)

  private val varPop = new VariancePop(If(And(IsNotNull(left), IsNotNull(right)),
    right, Literal.create(null, right.dataType))) 
.....{code}
*same fix to RegrIntercept*
Bug: Give snapshotStartBatchId and snapshotPartitionId to the state data source
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 05/Jun/24 16:58
Affects Version/s: 4.0.0

Right now, to read a version of the state data, the state source will try to find the first snapshot file before the given version and construct it using the delta files. In some debugging scenarios, users need more granular control on how to reconstruct the given state, for example they want to start from a specific snapshot instead of the closest one. One use case is to find whether a snapshot has been corrupted after committing.

This task introduces two options {{snapshotStartBatchId}} and {{snapshotPartitionId}} to the state data source. By specifying them, users can control the starting batch id of the snapshot and partition id of the state.