av9ash/GitBugs huggingface dataset:
Bug: TPCDSV1_4_PlanStabilitySuite fails
Status: Open | Priority: Minor | Resolution: 
Created: 06/May/22 19:30
Affects Version/s: 3.2.1

Tests tpcds-v1.4/q4 and tpcds-v1.4/q5 failed in [https://github.com/apache/spark/pull/36458] and [https://github.com/apache/spark/pull/31349] at least.

 

Local run succeeds.
Bug: Replcace double negation in exists with forall
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/22 14:18
Affects Version/s: 3.4.0

Some code in Spark as follows:
{code:java}
!Seq(1, 2).exists(x => !condition(x)) {code}
can replace with 
{code:java}
Seq(1, 2).forall(x => condition(x)) {code}
for code simplification
Bug: SparkContext.addFile trigger backend exception if it tries to add empty Hadoop directory
Status: In Progress | Priority: Minor | Resolution: 
Created: 05/May/22 05:56
Affects Version/s: 3.2.1

_spark.sparkContext.addFile("hdfs://xxxx/empty_dir", true)_ will result a backend failure with a vague error message:
{code:java}
java.nio.file.NoSuchFileException: /data/nvme1n1/nm-local-dir/usercache/jdai/appcache/application_1650999286466_24833/spark-c9e4de1a-5932-4c9b-bcac-138c8770960d/9368705081651698621617_cache
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
	at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
	at java.nio.file.Files.copy(Files.java:1274)
	at org.apache.spark.util.Utils$.copyRecursive(Utils.scala:734)
	at org.apache.spark.util.Utils$.copyFile(Utils.scala:705)
	at org.apache.spark.util.Utils$.fetchFile(Utils.scala:542)
	at org.apache.spark.executor.Executor.$anonfun$updateDependencies$4(Executor.scala:935)
	at org.apache.spark.executor.Executor.$anonfun$updateDependencies$4$adapted(Executor.scala:932)
	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
	at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
	at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
	at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:932)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}
This happens when executor localize the resource, it will invoke org.apache.spark.util.Utils.fetchFile, inside the method, Spark will first invoke *doFetchFile* to first download the file from HDFS to staging, then invoke *copyFile* to copy the staging file to final destination. However, if it is an empty HDFS folder, doFetchFile invokes fetchHcfsFile, which skips creating the empty staging folder. However, *copyFile* would assume the staging folder exist, thus trigger the exception.
Bug: Change to use `seq.indices` constructing ranges
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/May/22 03:27
Affects Version/s: 3.4.0

Some code in Spark constructing ranges of collection indices manually as follows:

 
{code:java}
for (i <- 0 until shuffledOffers.size) {
      val execId = shuffledOffers(i).executorId
      val host = shuffledOffers(i).host
      val taskSetRpID = taskSet.taskSet.resourceProfileId {code}
can use `shuffledOffers.indices` instead of `0 until shuffledOffers.size`
Bug: Replace map.getOrElse(false/true) with exists/forall
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 25/May/22 12:58
Affects Version/s: 3.3.0

Replace _map(_.toBoolean).getOrElse(false)_ with _exists(_.toBoolean)_

Replace _map(_.toBoolean).getOrElse(true)_ with _exists(_.toBoolean)_
Bug: Documentation for the decode function has an incorrect reference
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/May/22 07:53
Affects Version/s: 3.2.1

The documentation for the decode function introduced in SPARK-33527 refers erroneously to Oracle. It appears that the documentation string has been in large parts copied from [https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/DECODE.html#GUID-39341D91-3442-4730-BD34-D3CF5D4701CE]

This proposes to update the documentation for the decode function to fix the issue.
Bug: Replace If-Else branch with bitwise operators in roundNumberOfBytesToNearestWord
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 25/May/22 02:57
Affects Version/s: 3.2.1

[https://github.com/apache/spark/blob/a6dd6076d708713d11585bf7f3401d522ea48822/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java#L40-L47]

 

Here，can use bitwise operators to avoid If-Else branch and improve computation performance.
Bug: Use slice instead of take and drop
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/May/22 08:41
Affects Version/s: 3.4.0

{code:java}
seq.drop(n).take(m) -> seq.slice(n, n + m)
seq.take(m).drop(n) -> seq.slice(n, m) {code}
Bug: How do I read  parquet with python object
Status: Resolved | Priority: Trivial | Resolution: Won't Do
Created: 09/May/22 13:17
Affects Version/s: 2.4.5

{{python:}}

 

import pandas as pd

a=pd.DataFrame([[1,[2.3,1.2]]],columns=['a','b'])

a.to_parquet('a.parquet')

 

pyspark:

 

d2 = spark.read.parquet('a.parquet')

 

will return error:

An error was encountered: An error occurred while calling o277.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 14 in stage 9.0 failed 4 times, most recent failure: Lost task 14.2 in stage 9.0 (TID 63, 10.169.0.196, executor 15): java.lang.IllegalArgumentException: Illegal Capacity: -221

how can I fix it?

Thanks.
Bug: Inaccurate Decimal multiplication and division results
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 04/Nov/23 00:35
Affects Version/s: 3.2.4

Decimal multiplication and division results may be inaccurate due to rounding issues.
h2. Multiplication:
{code:scala}
scala> sql("select  -14120025096157587712113961295153.858047 * -0.4652").show(truncate=false)
+----------------------------------------------------+                          
|(-14120025096157587712113961295153.858047 * -0.4652)|
+----------------------------------------------------+
|6568635674732509803675414794505.574764              |
+----------------------------------------------------+
{code}
The correct answer is
{quote}6568635674732509803675414794505.574763
{quote}

Please note that the last digit is 3 instead of 4 as

 
{code:scala}
scala> java.math.BigDecimal("-14120025096157587712113961295153.858047").multiply(java.math.BigDecimal("-0.4652"))
val res21: java.math.BigDecimal = 6568635674732509803675414794505.5747634644
{code}
Since the factional part .574763 is followed by 4644, it should not be rounded up.
h2. Division:
{code:scala}
scala> sql("select -0.172787979 / 533704665545018957788294905796.5").show(truncate=false)
+-------------------------------------------------+
|(-0.172787979 / 533704665545018957788294905796.5)|
+-------------------------------------------------+
|-3.237521E-31                                    |
+-------------------------------------------------+
{code}
The correct answer is
{quote}-3.237520E-31
{quote}

Please note that the last digit is 0 instead of 1 as

 
{code:scala}
scala> java.math.BigDecimal("-0.172787979").divide(java.math.BigDecimal("533704665545018957788294905796.5"), 100, java.math.RoundingMode.DOWN)
val res22: java.math.BigDecimal = -3.237520489418037889998826491401059986665344697406144511563561222578738E-31
{code}
Since the factional part .237520 is followed by 4894..., it should not be rounded up.
Bug: Document state data source - reader
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 08/Nov/23 04:55
Affects Version/s: 4.0.0

Once the code for SPARK-45511 is landed, we need to put some effort on documenting the usage. Since we are open to further improvement e.g. state writer, we may want to have a separate page like Kafka integration.

We can include the usage of state metadata source into this page, as state metadata would be mainly useful for state data source.
Bug: Revisit and Improve Spark Standalone Cluster
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 10/Nov/23 01:30
Affects Version/s: 4.0.0

*Spark Standalone Cluster* has been supported for a long time as one of the resource managers.

As a part of Apache Spark 4.0.0, we revisit all layers of `Spark Standalone Cluster` as a long running subsystem inside K8s environment.
 # Spark Master, Worker, History Server Web UI Layer
 # Spark Master HA and Recovery Layer
 # Spark Master REST API Layer (including Cluster Utilization monitoring)
 # Spark Job Scheduling Layer
 # Spark Worker Management by exposing Cluster Utilization monitoring for Elastic Cluster Management
 # Spark Master/Worker dependency and classpath audit
 # Security
 # Documentation
Bug: Fix documentation of spark.executor.maxNumFailures
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 15/Nov/23 14:59
Affects Version/s: 3.5.0

https://issues.apache.org/jira/browse/SPARK-41210 added support for spark.executor.maxNumFailures on Kubernetes, it made this config generic and deprecated the yarn version.  This config isn't documented and defaults are not documented.

 

[https://github.com/apache/spark/commit/40872e9a094f8459b0b6f626937ced48a8d98efb]

\

It also added {color:#0a3069}spark.executor.failuresValidityInterval.{color}

 

{color:#0a3069}Both need to have default values specified for yarn and k8s, it also needs to remove the yarn documentation for equivalent configs spark.yarn.max.executor.failures configuration{color}
Bug: SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation
Status: Open | Priority: Major | Resolution: 
Created: 16/Nov/23 20:55
Affects Version/s: 3.5.1

Though documentation clearly recommends to add all columns in a single shot, but in reality is difficult to expect customer to modify their code, as in spark2  the rules in analyzer were such that  they did not do deep tree traversal.  Moreover in Spark3 , the plans are cloned before giving to analyzer , optimizer etc which was not the case in Spark2.
All these things have resulted in query time being increased from 5 min to 2 - 3 hrs.
Many times the columns are added to plan via some for loop logic which just keeps adding new computation based on some rule.

So,  my suggestion is to Collapse the Projects early, once the analysis of the logical plan is done, but before the plan gets assigned to the field variable in QueryExecution.
The PR for the above is ready for review.
The major change is in the way the lookup is performed in CacheManager.
I have described the logic in the PR and have added multiple tests.
Bug: Support Variant data type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Nov/23 19:23
Affects Version/s: 4.0.0

I propose to add a Variant data type in Spark. It is used to efficiently represent semi-structured values without a user-specified schema. Currently, many users are depending on JSON expressions to handle JSON data, which can often lead to repeated JSON parsing and degraded performance. One of the major goals of the Variant type is to use a more efficient binary representation internally and avoid repeated JSON parsing. At the same time, it keeps the flexibility of schemaless JSON data.
Bug: Support Executor JVM Profiling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/23 20:03
Affects Version/s: 4.0.0

To profile a Spark application a user or developer has to run a spark job locally on the development machine and use a tool like Java flight recorder, Yourkit, or async-profiler to record profiling information. Because profiling can be expensive, the profiler is typically attached to the Spark jvm process after the process has started and stopped once sufficient profiling data is collected.

The developers environment is frequently different from the production environment and may not yield accurate information.

However, the profiling process is hard when a Spark application runs as a distributed job on a cluster where the developer may have limited access to the actual nodes where the executor processes are running.  Also, in environments like Kubernetes where the executor pods may be removed as soon as the job completes, retrieving the profiling information from each executor pod can become quite tricky.

This feature is to add a low overhead sampling profiler like async-profiler as a built in capability to the Spark job that can be turned on using only user configurable parameters (async-profiler is a low overhead profiler that can be invoked programmatically and is available as a single multi-platform jar (for linux, and mac).

In addition, for convenience, the feature would save profiling output files to the distributed file system so that information from all executors can be available in a single place.

The feature would add an executor plugin that does not add any overhead unless enabled and can be configured to accept profiler arguments as a configuration parameter.
Bug: Add missing parameters for Pandas API on Spark
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/23 06:40
Affects Version/s: 4.0.0

When we looks at the "Missing parameters" from [Supported pandas API|https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/supported_pandas_api.html], there are still many parameters are missing in many APIs from Pandas API on Spark. We should add those missing parameters to improve usability.
Bug: Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/23 20:39
Affects Version/s: 3.5.1

In certain types of queries for eg TPCDS Query 14b,  the reuse of exchange does not happen in AQE , resulting in perf degradation.
The spark TPCDS tests are unable to catch the problem, because the InMemoryScan used for testing do not implement the equals & hashCode correctly , in the sense , that they do take into account the pushed down run time filters.

In concrete Scan implementations, for eg iceberg's SparkBatchQueryScan , the equality check , apart from other things, also involves Runtime Filters pushed ( which is correct).

In spark the issue is this:
For a given stage being materialized,  just before materialization starts, the run time filters are confined to the BatchScanExec level.
Only when the actual RDD corresponding to the BatchScanExec, is being evaluated,  do the runtime filters get pushed to the underlying Scan.

Now if a new stage is created and it checks in the stageCache using its canonicalized plan to see if a stage can be reused, it fails to find the r-usable  stage even if the stage exists, because the canonicalized spark plan present in the stage cache, has now the run time filters pushed to the Scan , so the incoming canonicalized spark plan does not match the key as their underlying scans differ . that is incoming spark plan's scan does not have runtime filters , while the canonicalized spark plan present as key in the stage cache has the scan with runtime filters pushed.

The fix as I have worked is to provide, two methods in the SupportsRuntimeV2Filtering interface ,
default boolean equalToIgnoreRuntimeFilters(Scan other) {
    return this.equals(other);
  }

  default int hashCodeIgnoreRuntimeFilters() {
    return this.hashCode();
  }

In the BatchScanExec, if the scan implements SupportsRuntimeV2Filtering, then instead of batch.equals, it should call scan.equalToIgnoreRuntimeFilters

And the underlying Scan implementations should provide equality which excludes run time filters.

Similarly the hashCode of BatchScanExec, should use scan.hashCodeIgnoreRuntimeFilters instead of ( batch.hashCode).

Will be creating a PR with bug test for review.
Bug: The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly 
Status: Open | Priority: Major | Resolution: 
Created: 14/Nov/23 19:00
Affects Version/s: 3.5.1

These   InMemoryV2FilterBatchScan and InMemoryBatchScan test classes are not implementing hashCode and equals correctly as they are not taking into account the pushed runtime filters. As a result they are unable to expose the TPCDS test issues which can show whether the reuse of exchange is happening correctly or not.
If these classes implement equals and hashCode taking into account the pushed runtime filters,  we would see that TPCDS Q14b which should ideally be reusing the exchange containing Union ,  is not happening due to multiple bugs which surface in AQE.

Actual V2 DataSources  like iceberg correctly implement equals and hashCode taking into account pushed runtime filters , which also expose the same issue of reuse of exchnage not happening
Bug: spark.catalog.listTables fails with ParseException after upgrading to Spark 3.4.1 from 3.3.1
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/23 14:36
Affects Version/s: 3.4.0

After upgrading to Spark 3.4.1, the listTables() method in PySpark now throws a ParseException with the message "Syntax error at or near end of input.". This did not occur in previous versions of Spark, such as 3.3.1.

Install Spark version 3.4.1.
 
Run pyspark
```bash
{{pyspark --packages io.delta:delta-core_2.12:2.4.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"}}
```
 
Attempt to list tables using
```console
{{spark.range(1).createTempView("test_view")}}
{{spark.catalog.listTables()}}
```
Expected result: The listTables() method should return a list of tables without throwing any exceptions.

Actual result: 
{{Traceback (most recent call last):}}
{{File "<stdin>", line 1, in <module>}}
{{File ".venv/lib/python3.10/site-packages/pyspark/sql/catalog.py", line 302, in listTables}}
{{iter = self._jcatalog.listTables(dbName).toLocalIterator()}}
{{File ".venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in _{_}call{_}_}}
{{File ".venv/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py", line 175, in deco}}
{{raise converted from None}}
{{pyspark.errors.exceptions.captured.ParseException:}}
{{[PARSE_SYNTAX_ERROR] Syntax error at or near end of input.(line 1, pos 0)}}

== SQL ==

^^^

>>>

The same code worked correctly in Spark version 3.3.1.
No changes were made to the code aside from upgrading Spark.

Thank you for considering this issue! Any assistance in resolving it would be greatly appreciated.

Best regards,
Andrej
Bug: Upgrade `protobuf-java` to 3.25.1 to match with protobuf 4.25.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/23 07:00
Affects Version/s: 4.0.0

- https://github.com/protocolbuffers/protobuf/pull/14617

{code}
- set(protobuf_VERSION_STRING "4.24.0")
+ set(protobuf_VERSION_STRING "4.25.0")
{code}

{code}
- <version>3.24.0</version>
+ <version>3.25.0</version>
{code}
Bug: Upgrade `protobuf` to 4.25.1 to support `Python 3.11`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/23 23:27
Affects Version/s: 4.0.0

`protobuf` starts to support Python 3.11 since v4.25.0.

- https://pypi.org/project/protobuf/4.25.0/
Bug: Spark Kubernetes Operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/23 17:38
Affects Version/s: kubernetes-operator-0.1.0

We would like to develop a Java-based Kubernetes operator for Apache Spark. Following the operator pattern (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/), Spark users may manage applications and related components seamlessly using native tools like kubectl. The primary goal is to simplify the Spark user experience on Kubernetes, minimizing the learning curve and operational complexities and therefore enable users to focus on the Spark application development.

Ideally, it would reside in a separate repository (like Spark docker or Spark connect golang) and be loosely connected to the Spark release cycle while supporting multiple Spark versions.

SPIP doc: [https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE|https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE/edit#heading=h.hhham7siu2vi]

Dev email discussion : [https://lists.apache.org/thread/wdy7jfhf7m8jy74p6s0npjfd15ym5rxz]
Bug: Upgrade `Volcano` to 1.8.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Nov/23 16:40
Affects Version/s: 4.0.0

To bring the latest feature and bug fixes in addition to the test coverage for Volcano scheduler 1.8.1.

[https://github.com/volcano-sh/volcano/releases/tag/v1.8.1]

 

[https://github.com/volcano-sh/volcano/pull/3101 |https://github.com/volcano-sh/volcano/pull/3101](volcano adapt k8s v1.27 volcano-sh/volcano#3101)
Bug: Add separate flag for privateKeyPassword
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/23 22:00
Affects Version/s: 4.0.0

Right now with config inheritance we support:
 * JKS with password A, PEM with password B
 * JKS with no password, PEM with password A
 * JKS and PEM with no password

 

But we do not support the case where JKS has a password and PEM does not. If we set keyPassword we will attempt to use it, and cannot set `spark.ssl.rpc.keyPassword` to null. So let's make it a separate flag as the easiest workaround.

 

This was noticed while migrating some existing deployments to the RPC SSL support where we use openssl support for RPC and use a key with no password
Bug: SPIP: Structured Streaming - Arbitrary State API v2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/23 20:23
Affects Version/s: 4.0.0

SPIP: Structured Streaming - Arbitrary State API v2

 

We are planning to introduce a new operator for [Spark Structured Streaming|https://spark.apache.org/streaming/] around [arbitrary stateful operations|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#arbitrary-stateful-operations]. The operator(s) we have today ([mapGroupsWithState/flatMapGroupsWithState|https://spark.apache.org/docs/3.1.3/api/java/org/apache/spark/sql/streaming/GroupState.html]) have a bunch of limitations (as noted in Q3 below) that prevent the use of the APIs for complex/advanced use-cases for stateful stream processing. We plan to remove these limitations with the introduction of a new API to handle state manipulation for streaming queries.

 

As part of this effort, we want to address:
 * composite types

 * multiple state variables (flexible data modeling)

 * state TTL

 * callback/timer improvements

 * schema evolution

 * initial state

 * side outputs

 * output modes

 * co-process functions

 * chaining operators after new operator etc