av9ash/GitBugs huggingface dataset:
Bug: migrate kafka consumer offset information in spark to new MSK cluster
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/24 05:25
Affects Version/s: 4.0.0

I have spark job that reads messages from kafka , using

 

            

 

{{```spark.readStream(.....)```}}

 

method. This is deployed in AWS as sparkapp and it reads data from MSK kafka. Recently the devOPs decided to migrate to a different MSK cluster for cost reduction.

There were some go Lang services which use their kafka offsets maintained by kafka server itself and the all the consumer offsets were automaticallly migrated to new MSK cluster since it is being managed by kafka itself.

However for spark jobs we have structured streaming which has checkpointing which is being managed by spark job itself. So when this spark job is being made to listen to new kafka URLs , they fail with missing offsets error. To resolve this we need to set failOnDataLoss = false , or delete checkpoint, in either case we have data loss and state gets lost, if the job has some custom state logic .

I have 2 requirements
 # Is there a way that kafka offsets are migrated automatically for spark consumers too whenever there is a migration of MSK, assuming that spark job has structured streaming.

      2. Is there a way that we can delete checkpoint, so that we can remove the offsets tracked by checkpoint, in order to reprocess with "earliest" offset, however without losing the Spark state .
Bug: Support DataFrame conversion to table argument
Status: Open | Priority: Major | Resolution: 
Created: 22/Nov/24 05:21
Affects Version/s: 4.0.0

Table-Valued Functions (TVFs) and User-Defined Table Functions (UDTFs) are widely used in Spark workflows. These functions often require a table argument, which Spark internally represents as a Catalyst expression. While Spark SQL supports constructs like TABLE(<query>) for this purpose, **there is no direct API in PySpark or Scala to convert a DataFrame into a table argument**. So we propose to support DataFrame conversion to table arguments (in Spark Classic first), and enable UDTFs to accept table arguments in both PySpark and Scala..
Bug: Refactor docker image for testing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/24 03:16
Affects Version/s: 4.0.0

currently we only have single testing image ({_}dev/infra/Dockerfile{_}), for jobs {{{}pyspark{}}}, {{{}sparkr{}}}, {{lint}} and {{{}docs{}}}, it has two major issues:
 * {*}disk space limitation{*}: we are adding more and more packages in it, the disk space left for testing is very limited, and cause {{No space left on device}} from time to time;
 * {*}environment conflicts{*}: for example, even though we already install some packages for {{docs}} in the docker file, we still need to install some additional python packages in {{{}build_and_test{}}}, due to the conflicts between {{docs}} and {{{}pyspark{}}}. It is hard to maintain because the related packages are installed in different places.

 

so we want to split existing base image to multiple ones, so that:
 * completely cache all the dependencies for each job;
 * centralize related installations for each job;
 * free up disk space on the base image;
 * introduce new dev tools based on new images;
Bug: Executor memory config is incorrect when multiple stage level resource profiles works
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/24 03:35
Affects Version/s: 3.1.3

https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature to help adjust stage's level resource profile(abbv rp), we are observing some config is not as expected when multiple resource profile works. 
When one stage assigned rp2 different with default rp, the `UnifiedMemoryManager` will still get old conf as the default rp, which will cause  maxOffHeapMemory is still the old value!

{code:scala}
protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
{code}
Bug: Command Execution in JavaOptions
Status: Open | Priority: Major | Resolution: 
Created: 06/Nov/24 08:06
Affects Version/s: 3.5.1

We found that Spark allow user to upload jar packet with`extraJavaOptions` ,

 

When instantiating a JAR package, use the javaOption configuration construct to execute the command if you configure the advanced optional parameters:

{color:#808080}spark.executor.extraJavaOption:{color}
{code:java}
\'touch\$IFS/tmp/zzz123|'{code}
The instance executes the jar package, uses the Yarn mode, and injects 'touch /tmp/zzz123' into the bash -c execution command.

Command as:
{code:java}
spark-submit --class JavaWordCount --master yarn --deploy-mode client --conf spark.executor.extraJavaOptions="\`touch\$IFS/tmp/zzz123\`" test.jar {code}
for java:
{code:java}
sparkLauncher.setConf("spark.executor.extraJavaOptions", "`touch$IFS/tmp/zzz123`"); {code}
Spark execution location:

/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
Bug: Limit number of subdirectories that FallbackStorage creates per shuffle
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/24 19:34
Affects Version/s: 4.0.0

The {{FallbackStorage}} copies shuffle data during executor decommissioning to a distributed or cloud storage like S3 or HDFS. In https://github.com/apache/spark/pull/34762, a hash has been added to the path of a file in order to reduce the number of files per directory (prefix in S3 terms). This creates as many directories per shuffle as files are transferred, while each directory contains a single file.

While this might be useful for S3, it may pose challenges for other filesystems. A shuffle of 100,000 partitions creates 100,000 directories, each containing a single file. The number of directories should be configurable to be able to adjust this behavior for the specific filesystem used.
Bug: No need to set initialInputBufferOffset when Aggregate mode is Complete
Status: Open | Priority: Major | Resolution: 
Created: 19/Nov/24 11:39
Affects Version/s: 3.5.3

Here:
[https://github.com/apache/spark/blob/87a5b37ec3c4b383a5938144612c07187d597ff8/sql/core/src/main/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAgg.scala#L73-L78]
{code:java}
            if isHashBasedAggWithKeys(partialAgg) && isPartialAgg(partialAgg, hashAgg) =>
            if (SortOrder.orderingSatisfies(
                partialAgg.child.outputOrdering, sortAgg.requiredChildOrdering.head)) {
              sortAgg.copy(
                aggregateExpressions = sortAgg.aggregateExpressions.map(_.copy(mode = Complete)),
                child = partialAgg.child) {code}
 

Just a small flaw.

No need to set initialInputBufferOffset when Aggregate mode is Complete.
Bug: Clean up ColumnVector resource after processing all rows in ColumnarToRowExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Nov/24 22:13
Affects Version/s: 3.4.4

Currently we only assign null to ColumnarBatch object but it doesn't release the resources hold by the vectors in the batch. For OnHeapColumnVector, the Java arrays may be automatically collected by JVM, but for OffHeapColumnVector, the allocated off-heap memory will be leaked.

For custom ColumnVector implementations like Arrow-based, it also possibly causes issues on memory safety if the underlying buffers are reused across batches. Because when ColumnarToRowExec begins to fill values for next batch, the arrays in previous batch are still hold.
Bug: [k8s] Apply for executor pods in parallel
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/24 08:19
Affects Version/s: 3.5.1

The performance of spark on k8s is worse than that of yarn. It is found that the application of executor pod is executed sequentially. The k8s interface for applying pod is kubernetesClient.pods().inNamespace(namespace).resource(podWithAttachedContainer).create(), which is asynchronous. However, each execution still takes an average of 62.57ms. Applying 280 pods takes 17520ms, which means that the speed of applying pod is about 15-16 pods/second. If a job requires more executors, this speed will become a bottleneck. I would like to ask whether this logic can be changed to concurrently apply for executor pods, and whether there will be any negative impact.

The logic of applying for executor is in method： org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator#requestNewExecutors
Bug: Add missing schema check for createDataFrame from numpy ndarray on Spark Connect
Status: Open | Priority: Major | Resolution: 
Created: 15/Nov/24 03:58
Affects Version/s: 4.0.0

Enforce schema for createDataFrame from numpy ndarray

 

Currently, the conversion from ndarray to pa.table doesn’t consider the schema at all ([for e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L649]), whereas pa.table or pd.DataFrame to pa.table does take the schema into account ([for e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L616]).
Bug: [M1] Fix grammar allowing empty bodies for loops, IF and CASE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/24 12:30
Affects Version/s: 4.0.0

SQL Scripting grammar currently allows for loops, IF and CASE to have empty bodies. Example:

{{WHILE 1 = 1 DO END WHILE;}}

This is not intended and should be changed so that empty bodies aren't allowed
Bug: load-spark-env.sh test for terminal or pipe broken / non-portable
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/24 17:56
Affects Version/s: 3.5.3

shell script `bin/load-spark-env.sh` uses a non-portable way of testing whether environment is a terminal or a pipe.

For details, see https://github.com/apache/spark/pull/48937
Bug: Update docs with YuniKorn 1.6.0 and first-time installation guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/24 18:40
Affects Version/s: kubernetes-operator-0.1.0

The README getting-started guide - YuniKorn section needs a one-liner guide for first-time user.

if the reader has not yet installed YuniKorn charts locally before, they would need to add the repo first.
Bug: Replace assertEqual with assertDataFrameEqual
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/24 07:03
Affects Version/s: 4.0.0

We have old patterns:

{{assertEqual(df.collect(), [rows])}}

only checks row values but doesn’t validate the DataFrame’s schema.

 

assertDataFrameEqual is explicitly built to handle DataFrame-specific comparisons, including schema.

So we propose to replace assertEqual with assertDataFrameEqual
Bug: Prohibit set operations on Variant type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/24 17:09
Affects Version/s: 4.0.0

Currently, the operations `DISTINCT`, `INTERSECT` and `EXCEPT` are allowed on Variant columns. However, since Variant is not orderable, and Variant equality is not defined, we should not allow this behavior. Sometimes, these queries fail during execution. However, they should fail during analysis with a well defined error message. Example:

select distinct (parse_json(case when id = 0 then 'null' else '1' end) as v, id % 5 as id) from range(0, 100, 1, 5)
Bug: mapPartitions fails when called in streaming mode from Python
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 28/Nov/24 23:47
Affects Version/s: 3.5.0

Calling the mapPartitions API in Python like this,

 

mapped_rdd = dataset.rdd.mapPartitions(process_partitions)

 

in Streaming mode results in the following error,

 

AnalysisException: Queries with streaming sources must be executed with writeStream.start();

the same API works if I call it from Scala, or if I execute outside Streaming Mode.
Bug: Spark creates unnecessary copies of data during explode operations
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/24 14:49
Affects Version/s: 3.5.0

+*Observed Behavior:*+

When some explode() operation is performed and copies of elements in rows are created, the memory grows proportionally to the number of new copies.

+*Expected Behavior:*+

Memory should be kept constant regardless of the new rows because elements in rows are {*}+immutable+{*}, so there's no point in keeping multiple copies.

At most, memory should grow with the number of partitions.

 

+*Discussion:*+

This is especially important when the data you're duplicating is a heavy buffer, for example a binary file. You don't want multiple copies be created in memory for the binary file. Also, it's not necessary because explode() operations don't affect the value of elements.

Check this example,
{code:java}
 
from pyspark.sql import Row
from pyspark.sql.functions import lit, col, explode, array

my_df = spark.read.format("binaryFile").load(my_200MB_file)


# Memory consumption should be independent of this one
K = 14


# Create a range column [0, 1, ..., K-1] for each row in the DataFrame
ranges = array([lit(i) for i in range(K)])  


# Add a new column with the range array and explode it
expanded_df = (
    my_df
    .withColumn("range", ranges)  # Add range array column
    .select(col("content"), explode(col("range")).alias("index"))  
)


# Do something with the DF
expanded_df.write.parquet("./pepe.parquet")
 {code}
The number of copies of the binary buffer shouldn't change with 'K'.
Bug: Add support for location for managed tables on database
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/24 14:37
Affects Version/s: 4.0.0

Consider the changes done under HIVE-22995, for HMS4 support. 

Wiki - [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ManagedandExternalTables] 

Behaviour change - With HIVE4 , MANAGEDLOCATION has been introduced with database. And LOCATION now refers to the path of external table.

 

So, when a database is created using sparkSQL, it dont have managed location reference and all the tables created inside it is of external table type. Impacts functionality like TRUNC, .....

 

Reference logs - 

metastore.MetastoreDefaultTransformer (MetastoreDefaultTransformer.java:transformDatabase(811)) - Starting translation for transformDatabase for processor null with null on database truncate_insert_overwrite locationUri=hdfs://madzilla/warehouse/tablespace/managed/hive/truncate_insert_overwrite.db {*}managedLocationUri=null{*}{*}{*}2024-11-27T15:42:20,320 INFO  [Metastore-Handler-Pool: Thread-3187]: ** metastore.MetastoreDefaultTransformer (MetastoreDefaultTransformer.java:transformDatabase(827)) - Database's location is a managed location, setting to a new default path based on external warehouse path:hdfs://madzilla/warehouse/tablespace/external/hive/truncate_insert_overwrite.db

 

Results in external table type.

!image-2024-11-29-20-12-52-808.png!
Bug: Parametrized identifiers not working when referencing CTEs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/24 18:03
Affects Version/s: 4.0.0

Parametrized are not working when referencing a CTE:

Example query that is failing:
{code:java}
sql("with tt1 as (select 1) select * from identifier(:cte)", Map("cte" -> "tt1")) {code}
Bug: Fixing Maven build errors introduced by Guava cache in RocksDBStateStoreProvider
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/24 23:53
Affects Version/s: 4.0.0

There are maven errors introduced by the guava dependency in `sql/core`, as we use the Guava cache to store the Avro encoders, outlined in this comment: [https://github.com/apache/spark/pull/48401#issuecomment-2504353098]
Change the NonFateSharingCache to not use the Guava cache

It looks like the PR CI doesn't run the suites with Maven, which caused this build error to go unnoticed on merge, but was caught in post-merge. To test this, we need to run the following commands:
{code:java}
build/mvn clean install -DskipTests -pl sql/core -am
build/mvn test -pl sql/core -DwildcardSuites=none -Dtest=test.org.apache.spark.sql.JavaDatasetSuite{code}
Bug: Use Guava cache to store Avro encoders
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/24 23:49
Affects Version/s: 4.0.0

There are maven errors introduced by the guava dependency in `sql/core`, as we use the Guava cache to store the Avro encoders, outlined in this comment: [https://github.com/apache/spark/pull/48401#issuecomment-2504353098]
Change the NonFateSharingCache to not use the Guava cache

It looks like the PR CI doesn't run the suites with Maven, which caused this build error to go unnoticed on merge, but was caught in post-merge. To test this, we need to run the following commands:
{code:java}
build/mvn clean install -DskipTests -pl sql/core -am
build/mvn test -pl sql/core -DwildcardSuites=none -Dtest=test.org.apache.spark.sql.JavaDatasetSuite{code}
Re-introduce use of Guava cache without build errors
Bug: Executors of stage failed still alive even through stage has been retried
Status: Open | Priority: Major | Resolution: 
Created: 12/Nov/24 07:21
Affects Version/s: 3.3.0

We are executing spark dataframe API with foreachPartition and we observed something that we are not able to explain.

In the figure that we provided, you can see that stage 19 has been retried due to some ShuffleOutputNotFound error and stage 19 has been retried 1.

!stage19_stage19retry1.png|width=1150,height=603!

However, we found that there are still some executors allocated for stage 19 in state RUNNING e.g. partition 371 and 200. In addition, as not yet finished, partition 371 and 200 have been resubmitted again in stage 19 (retry 1). 

!trigger_duplicate_process.png|width=1144,height=402!

Is there any configuration can help us make sure that all executor from the stage failed will be terminated before retry staged been triggered.
Bug: Executor memory config is incorrect when multiple stage level resource profiles works
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 26/Nov/24 03:31
Affects Version/s: 3.1.3

https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature to help adjust stage's level resource profile(abbv rp), we are observing some config is not as expected when multiple resource profile works. 
When one stage assigned rp2 different with default rp, the `UnifiedMemoryManager` will still get old conf as the default rp, which will cause  maxOffHeapMemory is still the old value!

{code:scala}
protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
{code}
Bug: Executor memory config is incorrect when multiple stage level resource profiles works
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 26/Nov/24 03:32
Affects Version/s: 3.1.3

https://issues.apache.org/jira/browse/SPARK-27495 has imported a great feature to help adjust stage's level resource profile(abbv rp), we are observing some config is not as expected when multiple resource profile works. 
When one stage assigned rp2 different with default rp, the `UnifiedMemoryManager` will still get old conf as the default rp, which will cause  maxOffHeapMemory is still the old value!

{code:scala}
protected[this] val maxOffHeapMemory = conf.get(MEMORY_OFFHEAP_SIZE)
{code}
Bug: Reduce parallelism in Pure Python library build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/24 06:49
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/workflows/build_python_connect.yml
https://github.com/apache/spark/actions/workflows/build_python_connect35.yml

To avoid OOM
Bug: [CONNECT][PYTHON] Provide the ability to reconfigure the connect logger
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/24 09:51
Affects Version/s: 4.0.0

We currently offer the mechanism to configure the connect-specific logger based on the environment variable SPARK_CONNECT_LOG_LEVEL.

The logger is configured once at the the time of "module load". In some specific cases, some Python frameworks (eg. IPythonKernel) can modify the Python log level after the fact.

There is no good way to restore the logger to restore its previous functionality to honour the environment variable configured.
Bug: TransformWithState secondary index sizes should be proportional to primary indexes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/24 20:05
Affects Version/s: 4.0.0

Currently, the TWS operator handles TTL state variables in the same approximate way:
 # Upsert the value into the primary index
 # Upsert the expiration timestamp into the secondary index, where the expiration timestamp is the `batchTimestampMs`

The issue with this approach is that if the same state variable is updated across two different micro-batches, there exists 1 entry in the primary index, while there exists _two_ entries in the secondary index. Consider the following example for a state variable `foo` with value `v1`, and TTL delay of 500:

Batch 0, `batchTimestampMs = 100`, `foo` updates to `v1`:
 * Primary index: `[foo -> (v1, 600)]`
 * Secondary index: `[(600, foo) -> EMPTY]`

Batch 1: `batchTimestampMs = 200`, `foo` updates to `v2`:
 * Primary index: `[foo -> (v2, 700)]`
 * Secondary index: `[(600, foo) -> EMPTY, (700, foo) -> EMPTY]`

You'll notice that the secondary index now has size 2, even though the primary index only has size 1. When we clean up `(600, foo)`, we actually _don't_ delete it, since we do [another lookup in the primary index to determine if the secondary index entry we're dealing with is stale|https://github.com/apache/spark/blob/05508cf7cb9da3042fa4b17645102a6406278695/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ValueStateImplWithTTL.scala#L109-L113].

So, to summarize, the write path is always write to the primary _and_ secondary index, and then for deletion, we go through everything in the secondary index, and do a lookup on the primary index to see whether to delete it.

While this may not seem like a huge issue, things get way worse for `ListState`. Our cleanup logic for list state is as follows:
 # Grab an iterator for the entire list
 # Clear the entire list
 # For each element in the iterator that isn't expired, merge it back into the list

This means that having an erroneous entry in your secondary index means that you will go through the entire list _several_ times, which will negatively impact performance. We should most definitely make sure that the secondary index has only as many elements as the primary index, which will prevent us from doing unnecessary work during cleanup.

Solutions to this problem will be proposed in the PR.
Bug: Spark Connect as the default API in Spark 4
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/24 13:37
Affects Version/s: 4.0.0

We propose to enable Spark Connect as the default interface for Scala and Python applications in Spark 4. With this change, we will ensure that by default, all Scala and Python applications will run using the Spark Connect API so that developers can 1) benefit from stability, upgradability, and ease of use and 2) use Spark from non-JVM languages.

As a lot of cases are using the full Spark API,  we want to make it simple to switch back for compatibility reasons. We propose to add a new Spark configuration spark.api.mode with the following options:
 * connect:  Spark uses the Spark Connect as the default API (default)
 * classic: Spark uses classic mode. The classic mode is today’s default. 

With this configuration, we open up Spark to a new generation of developers while ensuring all existing workloads continue to run without any impact!

SPIP: https://docs.google.com/document/d/1C0kuQEliG78HujVwdnSk0wjNwHEDdwo2o8aVq7kbhTo/edit?tab=t.0#heading=h.r2c3xrbiklu3
Bug: Dependencies for Data Source API are missing in `pyspark[sql]`
Status: Open | Priority: Major | Resolution: 
Created: 23/Nov/24 20:02
Affects Version/s: 4.0.0

If I install `pyspark[sql]` from pypi, and try to use [Data Source API|https://spark.apache.org/docs/preview/api/python/user_guide/sql/python_data_source.html], then it fails because Python dependencies aren't satisfied - grpcio, grpcio-status and googleapis-common-protos are missing...