av9ash/GitBugs huggingface dataset:
Bug: Support Avro encoding for TransformWithState operator - ValueState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/24 20:18
Affects Version/s: 4.0.0

Currently, we use UnsafeRow to store all state in the StateStores. This change will add StateStore APIs that support byte array operations on the StateStore, and to enable Avro encoding to be used for the TransformWithState operator, only for ValueState. Subsequent state variable types will be supported in future PRs.
Bug: Fix Misreporting of Fatal Errors as Uncaught Exceptions in SparkUncaughtExceptionHandler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Oct/24 17:33
Affects Version/s: 4.0.0

In {{{}Executor.scala{}}}, an exception is considered fatal (determined by isFatalError()) if any exception in the chain (or its causes) is fatal. This behavior is controlled by the {{spark.executor.killOnFatalError.depth}} config, which limits the depth to which the chain is inspected. If a fatal error is found, {{SparkUncaughtExceptionHandler}} is called.



However, currently {{SparkUncaughtExceptionHandler}} only considers the top-level exception when reporting the exit code, rather than traversing the full exception chain to identify the true fatal cause. As a result, some fatal errors, such as {{{}OutOfMemoryError{}}}, are mistakenly reported as uncaught exceptions.

For instance, if we have an OOM exception with the following structure:

RuntimeException
 - Caused by: RuntimeException
 - Caused by: java.lang.OutOfMemory

{{SparkUncaughtExceptionHandler}} would quit the executor with error code SparkExitCode.UNCAUGHT_EXCEPTION, when the true cause is an OOM error.



This change intends to modify {{SparkUncaughtExceptionHandler}} to:
 * Inspect the exception chain (up to the configured depth).
 * Ensure that the actual fatal error is correctly identified and reflected in the exit code.
Bug: Support Variant in Spark Connect Scala client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 23:02
Affects Version/s: 4.0.0

Currently Spark Connect Scala client doesn't support Variant type when collecting data.

{code:scala}
scala> spark.sql("select parse_json('1')").collect()
org.apache.spark.SparkUnsupportedOperationException: [UNSUPPORTED_DATATYPE] Unsupported data type "VARIANT". SQLSTATE: 0A000
  org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError(ExecutionErrors.scala:159)
  org.apache.spark.sql.errors.ExecutionErrors.unsupportedDataTypeError$(ExecutionErrors.scala:156)
  org.apache.spark.sql.errors.ExecutionErrors$.unsupportedDataTypeError(ExecutionErrors.scala:254)
  org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:370)
...
 {code}
Bug: Flexible configuration of javaoptions.
Status: Open | Priority: Major | Resolution: 
Created: 31/Oct/24 14:12
Affects Version/s: 3.5.3

In Yarn mode, the javaOptions configuration item is executed as the bash -c command, such as *spark.executor.extraJavaOptions，* *spark.executor.defaultJavaOptions,*  *spark.executor.extraJavaOptions,*  {*}spark.driver.defaultJavaOptions{*}, which may cause arbitrary command execution risks. Can we add a configuration item to eliminate risks when the javaOptions configuration is not required?

 

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala

 
{code:java}
/**
 * Escapes a string for inclusion in a command line executed by Yarn. Yarn executes commands
 * using either
 *
 * (Unix-based) `bash -c "command arg1 arg2"` and that means plain quoting doesn't really work.
 * The argument is enclosed in single quotes and some key characters are escaped.
 *
 * (Windows-based) part of a .cmd file in which case windows escaping for each argument must be
 * applied. Windows is quite lenient, however it is usually Java that causes trouble, needing to
 * distinguish between arguments starting with '-' and class names. If arguments are surrounded
 * by ' java takes the following string as is, hence an argument is mistakenly taken as a class
 * name which happens to start with a '-'. The way to avoid this, is to surround nothing with
 * a ', but instead with a ".
 *
 * @param arg A single argument.
 * @return Argument quoted for execution via Yarn's generated shell script.
 */ {code}
Bug: Fix PostgreSQL connector behaviour for multidimensional arrays
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/24 17:40
Affects Version/s: 4.0.0

There is a bug introduced in this PR [https://github.com/apache/spark/pull/46006]. This PR fixed the behaviour for PostgreSQL connector for multidimensional arrays since we have mapped all arrays to 1D arrays.

This PR has introduced a bug. Following scenario is broken:
 * User has a table t1 on Postgres and does CTAS command to create table t2 with same data.
 * PR 46006 is resolving the dimensionality of column by reading the metadata from pg_attribute table and attndims column.
 * This query returns correct dimensionality for table t1, but for table t2 that is created via CTAS it returns 0 always. This leads to all of the arrays being mapped to 0-D array which is the type itself (for example int)

As a solution, we can query array_ndims function on given column that will return the dimension of the column. It works for CTAS-like-created tables too.
Bug: [M1] Unique label names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Oct/24 09:21
Affects Version/s: 4.0.0

Check for label uniqueness should be added.

Check should work on the scope level, i.e. we can have multiple same labels within the one script, but they cannot belong to the same scope.

Shouldn’t be too complicated, each visit method has a corresponding enter/exit method for each node that can be overridden as well and thus we can easily handle scopes.
Bug: Consolidate configurable schema verification of createDataFrame
Status: Open | Priority: Major | Resolution: 
Created: 28/Oct/24 06:48
Affects Version/s: 4.0.0

{{verifySchema}} parameter of createDataFrame decides whether to verify data types of every row against schema.
h3. In Spark Classic

Now it only takes effect for with createDataFrame with
 * regular Python instances

We propose to make it work with createDataFrame with
 * {{pyarrow.Table}}
 * {{pandas.DataFrame}} with Arrow optimization
 * {{pandas.DataFrame}} without Arrow optimization

h3. In Spark Connect

Now it does not take effect.

We propose to make it work with all inputs.
Bug: ArtifactManager follow-up: better/lighter registration logic during cloning
Status: Open | Priority: Major | Resolution: 
Created: 22/Oct/24 15:32
Affects Version/s: 4.0.0

In https://github.com/apache/spark/pull/48320 the `ArtifactManager.clone` method, we re-register all existing artifacts with new Session UUID to the new SparkContext. This current logic is a bit heavy and unnecessary because all artifacts are already in place and have passed all necessary checks.
Ideally, we want to modify the SparkContext.addedJars map directly to insert these artifacts with new session UUIDs - or something that is lightweight.
Bug: Moving Avro files to sql/core so they can be used by TransformWithState operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Oct/24 03:46
Affects Version/s: 4.0.0

With the introduction of the TransformWithState operator, we are going to persist all state information in the StateStore with Avro encoding, instead of UnsafeRow. This is because UnsafeRow is not backwards compatible and can change between Spark releases, potentially causing StateStore corruptions. Avro is backwards-compatible and can be reliably used. This change will move the necessary files to the sql/core directory so Avro encoding can be used by stateful streaming operators.
Bug: Address the limitation of metrics calculation with DSv1 source in streaming query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/24 05:35
Affects Version/s: 4.0.0

In streaming query, we calculate the number of output rows per stream, via collecting the metric from the source nodes in the executed plan.

For DSv2 data sources, the source nodes in the executed plan are always MicroBatchScanExec, and these nodes contain the stream information.

But for DSv1 data sources, the logical node and the physical node representing the scan of the source are technically arbitrary (any logical node and any physical node), hence Spark makes an assumption that the leaf nodes for initial logical plan <=> logical plan for batch N <=> physical plan for batch N are the same so that we can associate these nodes. This is fragile and we have non-trivial number of reports of broken metric.

This ticket aims to address the limitation for DSv1 streaming source; the idea is to scope the logical/physical nodes to the "widely-used set" and pass the stream information into these nodes, so that we can use the same approach of calculating metrics with DSv2 to DSv1 streaming sources.
Bug: RocksDB Hardening: Fix new file mapping version advancement and ineffective file reuse bug
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/24 19:32
Affects Version/s: 4.0.0

There are 2 bugs in the recently added new approach for RocksDB SST file mapping in this PR: [https://github.com/apache/spark/pull/47875]
 # The file mapping version is not properly advancing and only advances when we are reopening the RocksDB. This causes ineffective file reuse and we end up not reusing files that should have been reused. Leading to a lot of unnecessary file upload/download, and in the worst case it will make us act like file reuse is disabled.
 # Ineffective file reuse when creating a checkpoint. We currently will not reuse the files for creating checkpoint, if it was added in the current version i.e. if you do load(v1) -> save(v2), the SST files loaded in v1 will not be reused in v2, and we will upload them again.

These two bugs were not caught even though we have tests to catch them, because there was also a bug in the test.

NOTE: these bugs will not cause corruption and more of a performance bug. They just make file reuse ineffective. And end up working like there's no file reuse enabled. i.e. A lot of file upload/download
Bug: Fix Streaming RocksDB AcquireLock double release
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/24 21:51
Affects Version/s: 3.5.3

The RocksDB implementation will run into a race condition with the acquireLock due to the completion listener in the following scenario: 
||Task 1||Task 2||
|load() - acquire| |
|commit() - release| |
| |load() - acquire()|
|completionListener - release| |

Now RocksDB is thread unsafe as task 2 believes it holds the lock though the lock was released.
Bug: Use latest Python dependencies for Python client3.5 <> 4.0server job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/24 07:33
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/11543780375/job/32128570166 fails apparently because of different grpc versions. While it is a legitimate failure to fix, that CI does not target to test all different dependency versions. It only targets 3.5cleint <> 4.0server. We should better make the test on its purpose.
Bug: Move UnresolvedCatalogRelation resolution to a separate method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/24 09:00
Affects Version/s: 4.0.0

Factor out `UnresolvedCatalogRelation` resolution into a separate `FindDataSourceTable` method to reuse it in the single-pass Analyzer.

Context: https://issues.apache.org/jira/browse/SPARK-49834
Bug: Connect session is not released if not calling spark.stop() explicitly
Status: Open | Priority: Major | Resolution: 
Created: 30/Oct/24 03:21
Affects Version/s: 4.0.0

Hi,

I found that the Spark Connect session will not be released if not calling spark.stop() explicitly.
h2. *Repro:*

I have a python file with below code 

test.py
{code:java}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
spark.range(10).show(){code}
After executing it by 
{code:java}
python test.py{code}
I found the corresponding connect session is still alive from spark webui. See this session id 96260131-a22c-4342-92df-8dc7ace5d1de item in below table

But if I have `spark.stop() been called explicitly in the python file
{code:java}
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
spark.range(10).show()
spark.stop(){code}
The connect session will be released. See the 4e8ffb4e-7684-4fa7-b750-814f9a23f2d0 item

 
||[User|http://localhost:4040/connect/?&sessionstat.sort=User&sessionstat.pageSize=100#sessionstat]||[Session ID|http://localhost:4040/connect/?&sessionstat.sort=Session+ID&sessionstat.pageSize=100#sessionstat]||[Start Time ▾|http://localhost:4040/connect/?&sessionstat.sort=Start+Time&sessionstat.desc=false&sessionstat.pageSize=100#sessionstat]||[Finish Time|http://localhost:4040/connect/?&sessionstat.sort=Finish+Time&sessionstat.pageSize=100#sessionstat]||[Duration|http://localhost:4040/connect/?&sessionstat.sort=Duration&sessionstat.pageSize=100#sessionstat]||[Total Execute|http://localhost:4040/connect/?&sessionstat.sort=Total+Execute&sessionstat.pageSize=100#sessionstat]||
|xxx|[4e8ffb4e-7684-4fa7-b750-814f9a23f2d0|http://localhost:4040/connect/session/?id=4e8ffb4e-7684-4fa7-b750-814f9a23f2d0]|2024/10/30 11:05:25|2024/10/30 11:05:25|78 ms|1|
|xxx|[96260131-a22c-4342-92df-8dc7ace5d1de|http://localhost:4040/connect/session/?id=96260131-a22c-4342-92df-8dc7ace5d1de]|2024/10/30 11:04:41| |13 minutes 55 seconds|0|


So I'm wondering if this is per-design or a potential bug? Since if the connect session is not released, the connect server will
still hold the caches which will not be freed. That could blow up the connect server/driver memory.
Bug: Spark Kubernetes support requires vulnerable okhttp dependency
Status: Open | Priority: Major | Resolution: 
Created: 29/Oct/24 21:57
Affects Version/s: 3.4.1

We have been trying to resolve the following CVE flagged in our Spark install: [https://www.cve.org/CVERecord?id=CVE-2023-0833]

The vulnerability is on {{{}okhttp-3.12.12.jar{}}}.  It's pulled in by the fabric8 kubernetes client: {{{}kubernetes-client-6.7.2.jar{}}}.

[The fabric8 kubernetes client does support using other HTTP clients|https://blog.marcnuri.com/kubernetes-client-6-httpclient-how-to#vanilla-java-jdk-HttpClient]. However, Spark is explicitly initializing it with a bespoke okhttp client instance. We are using 3.4.1, but the problem is still there in the master branch.
 # [Creating an okhttp dispatcher with a custom apache thread pool|https://github.com/apache/spark/blob/cfe14c9c275b75f45b8d810333174cfd08c3af61/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala#L83C1-L84C1]
 # [Building an okhttp client factory with that dispatcher|https://github.com/apache/spark/blob/cfe14c9c275b75f45b8d810333174cfd08c3af61/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala#L120C1-L121C1]
 # [Initializing fabric8 kubernetes client with the custom okhttp client factory|https://github.com/apache/spark/blob/95b2d27079c2e012ab5bfb8c1dd83b11d7848258/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala#L127C1-L128C1]

Furthermore, we are unable to upgrade the okhttp client library itself.
The earliest version of okhttp that does not have the CVE, and is also not dependent on a version of the Kotlin standard library that doesn't have CVEs, appears to be {{okhttp:4.10.0}} (with {{{}kotlin-stdlib:1.6.20{}}}). However, the latest release of the fabric8 kubernetes client ({{{}}kubernetes-client:6.13.4{}}}) is still dependent on okhttp 3.12.12. It cannot run with okhttp 4.10.

In summary: Because it's hard-coded into Spark we are unable to replace okhttp, and because fabric8 kubernetes client stopped support for newer versions of okhttp we are unable to upgrade it to a non-vulnerable version.
Bug: Add TryMakeTimestampLTZ and TryMakeTimestampNTZ expression alternatives
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/24 12:42
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-49637

 

In order to improve suggestions for the linked ticket, we need to add these expressions to Spark.
Bug: [Possible bug] Incorrect watermark advancement if watermark node is lost/pruned
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 01:18
Affects Version/s: 4.0.0

This does not happen in current optimization rules, but it was mostly a luck and we were silently dropping CollectMetrics node, hence it'd be ideal to address the issue in prior.

WatermarkTracker only looks at the physical plan during calculation of the new watermark value. It determines the watermark node by index, hence we have various issues when the watermark node is lost on the optimization.

1) watermark advancement is made even there is one node to be dropped (should be considered as no data from that node)

2) watermark tracker incorrectly update the memory map of the previous value of watermark node (index is not a stable key)
Bug: Support optional "bw_method" parameter in kde/density plots
Status: Open | Priority: Major | Resolution: 
Created: 17/Oct/24 03:06
Affects Version/s: 4.0.0

Support optional "bw_method" parameter in kde plots for both Pandas on Spark and PySpark.

Currently, users have to specify:

 

{{bw_method : int or float}}
{{            The method used to calculate the estimator bandwidth.}}

 

We should investigate the possibilities to set it optional like in native pandas.
Bug: Introduce a new pattern object for LogicalRelation to help avoiding default pattern (full parameters)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/24 22:28
Affects Version/s: 4.0.0

During my work (work-in-progress), I found it very difficult to add a param into LogicalRelation, because a bunch of places use pattern matching with LogicalRelation, and all of them are using default pattern with specifying all params (despite wildcards).

I looked into the patterns and realized most of the patterns just need relation and catalog table inside the LogicalRelation. output and isStreaming are used infrequently. That said, introducing a new pattern object extracting BaseRelation and CatalogTable from LogicalRelation would cover most of the patterns we have for LogicalRelation, and this would help a lot to introduce a new param into LogicalRelation in future.
Bug: Negative AQE caching in SQL cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Oct/24 23:29
Affects Version/s: 4.0.0

When we use a cached an AQE plan, it will do `cachedPlan.execute` to build the RDD, which will execute all AQE stages except the result stage. If any of them failed, the failure will be cached by lazy RDD val. So the next time when we reuse that cached plan (even by a totally irrelevant caller) it will fail immediately.

We need to re-cache the AQE plan upon failure.
Bug: metrics defined by observe API are lost when the physical node is pruned out
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Oct/24 08:52
Affects Version/s: 3.4.3

When user defines the metrics via observe API, they expect the metrics to be retrieved via Observation (batch query) or update event of StreamingQueryListener.

But when the node (CollectMetrics) is lost in any reason (e.g. subtree is pruned by PruneFilters), Spark does behave like the metrics were not defined, instead of providing default values.

Spark should give a best effort to provide default values - when the node is pruned out from optimizer, it is mostly logically equivalent that there were no input being processed with the node (except the bug in analyzer/optimizer/etc which drop the node incorrectly), hence it's valid to just have default value.
Bug: Make pysaprk-ml-connect tests passing without optional dependencies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 23:39
Affects Version/s: 4.0.0

{code}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/__w/spark/spark/python/pyspark/ml/tests/connect/test_connect_evaluation.py", line 38, in <module>
    class EvaluationTestsOnConnect(EvaluationTestsMixin, unittest.TestCase):
                                   ^^^^^^^^^^^^^^^^^^^^
NameError: name 'EvaluationTestsMixin' is not defined
{code}

https://github.com/apache/spark/actions/runs/11447673972/job/31849621508
Bug: Handle or document `NumPy 2.1` difference in Python 3.13
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 04/Oct/24 22:06
Affects Version/s: 4.0.0

Although SPARK-48710 fixed to use NumPy 2.0 compatible types. Python 3.13 requires NumPy 2.1 (SPARK-49869) and seems to reveal another instances of differences.

New `NumPy 2.1` seems to have a different output style in Python 3.13. Of course, the values are correct.
- https://github.com/apache/spark/actions/runs/11186188886/job/31100777649
{code}
**********************************************************************
File "/__w/spark/spark/python/pyspark/core/rdd.py", line 2463, in __main__.RDD.sampleStdev
Failed example:
    sc.parallelize([1, 2, 3]).sampleStdev()
Expected:
    1.0
Got:
    np.float64(1.0)
**********************************************************************
File "/__w/spark/spark/python/pyspark/core/rdd.py", line 2436, in __main__.RDD.stdev
Failed example:
    sc.parallelize([1, 2, 3]).stdev()
Expected:
    0.816...
Got:
    np.float64(0.816496580927726)
**********************************************************************
{code}
Bug: Clarification regarding Spark v3.4 and EOL status
Status: Open | Priority: Major | Resolution: 
Created: 23/Oct/24 13:54
Affects Version/s: 3.4.3

Hi there.

I'd like to enquire the current support status of Spark v3.4. The versioning policy documentation, states that releases are typically supported for 18 months, and the last minor release is considered LTS and supported for longer.

At the time of writing, there appears to be three versions of Apache Spark, which are being built, and have release tags cut in GitHub:
 * *v4.0 (preview, not GA)*
 * *v3.5.x*
 * *v3.4.x*

+Could you help me determine how long v3.4.x is supported for?+ It still looks to be active and I see -rc tags being cut in advance of the next v3.4x patch release.

I'd also like to work with [endoflife.date|https://endoflife.date/apache-spark], to make the necessary updates to v3.4.
Bug: RegExpReplace does not catch runtime errors when it fails on certain rows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/24 01:47
Affects Version/s: 4.0.0

Currently, RegExprReplace does not catch any exceptions during runtime. It lets the underlying Java libraries throw whichever error they do. It would be good if Spark caught the underlying exceptions and threw a formal SparkException.
Bug: Spark Connect should support empty ndarray
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/24 05:15
Affects Version/s: 4.0.0

 

In [4]: spark.range(1).select(sf.lit(np.array([], np.int32))).schema
Out[4]: StructType([StructField('array()', ArrayType(NullType(), False), False)])

 

it should be ArrayType(IntegerType...)
Bug: Remove support for interval types from the Variant spec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Oct/24 01:46
Affects Version/s: 4.0.0

I implemented interval support for Variant before, but because the Variant spec type is supposed to be open and compatible with other engines which may not support all the ANSI Interval types, more thought needs to be put into the design of these intervals in Variant.