av9ash/GitBugs huggingface dataset:
Bug: Failed to remove shuffle ${shuffleId} - null when using Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/22 09:49
Affects Version/s: 3.2.1

test sql/core module using Scala 2.13 with maven , there are many log as follows，although these did not cause UTs failure

 
{code:java}
17:44:09.957 WARN org.apache.spark.storage.BlockManagerMaster: Failed to remove shuffle 87 - null
java.lang.NullPointerException
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1(MapOutputTracker.scala:882)
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1$adapted(MapOutputTracker.scala:881)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.MapOutputTrackerMaster.unregisterShuffle(MapOutputTracker.scala:881)
	at org.apache.spark.storage.BlockManagerStorageEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerStorageEndpoint.scala:59)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
	at org.apache.spark.storage.BlockManagerStorageEndpoint.$anonfun$doAsync$1(BlockManagerStorageEndpoint.scala:89)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
17:44:09.958 ERROR org.apache.spark.ContextCleaner: Error cleaning shuffle 94
java.lang.NullPointerException
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1(MapOutputTracker.scala:882)
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1$adapted(MapOutputTracker.scala:881)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.MapOutputTrackerMaster.unregisterShuffle(MapOutputTracker.scala:881)
	at org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:241)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$3(ContextCleaner.scala:202)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$3$adapted(ContextCleaner.scala:195)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$1(ContextCleaner.scala:195)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1432)
	at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:189)
	at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:79) {code}
Bug: Allow file splitting in combination with row index generation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/22 16:10
Affects Version/s: 3.3.0

This issue is a follow up for SPARK-37980

Because of a bug in parquet-mr https://issues.apache.org/jira/browse/PARQUET-2161 it is currently impossible to generate row indexes for parquet files if they are split into multiple pieces. Instead, each file must be read in a single task. 

Once the version of parquet-mr with the fix is included in Spark, we should remove the workarounds from the code (marked with this ticket number) from the code, so that parquet files are splittable even when the row indexes need to be generated.
Bug: when using df.astype("str") on pyspark dataframe. None are converted "None"
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/22 12:51
Affects Version/s: 3.3.0

when using df.astype("str") on pyspark dataframe. None are converted "None".

 
 - Not able to keep Null values as None. instead it is converted to string "None" for whole column.

 

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/9431837223032/3368288770184753/3794760114602748/latest.html
Bug: Upgrade mysql-connector-java to 8.0.29
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/22 08:32
Affects Version/s: 3.4.0

Improper Handling of Insufficient Permissions or Privileges in MySQL Connectors Java.

Vulnerability in the MySQL Connectors product of Oracle MySQL (component: Connector/J). Supported versions that are affected are 8.0.27 and prior. Difficult to exploit vulnerability allows high privileged attacker with network access via multiple protocols to compromise MySQL Connectors. Successful attacks of this vulnerability can result in takeover of MySQL Connectors. CVSS 3.1 Base Score 6.6 (Confidentiality, Integrity and Availability impacts). CVSS Vector: (CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:H/I:H/A:H).

[CVE-2022-21363|https://nvd.nist.gov/vuln/detail/CVE-2022-21363]
Bug: The inconsistent timezone in Spark History Server UI
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/22 19:28
Affects Version/s: 3.2.1

The spark history server is running in UTC timezone. But we are trying to view history server in IST timezone.
The history server landing page shows time in IST but Jobs, Stages, Storage, Environment, Executors tabs shows time in UTC.
Bug: SPIP: Asynchronous Offset Management in Structured Streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/22 20:59
Affects Version/s: 3.3.0

Currently in Structured Streaming, at the beginning of every micro-batch the offset to process up to for the current batch is persisted to durable storage.  At the end of every micro-batch, a marker to indicate the completion of this current micro-batch is persisted to durable storage. For pipelines such as one that read from Kafka and write to Kafka, end-to-end exactly once is not support and latency is sensitive, we can allow users to configure offset commits to be written asynchronously thus this commit operation will not contribute to the batch duration and effectively lowering the overall latency of the pipeline.

 

SPIP Doc: 

 

https://docs.google.com/document/d/1iPiI4YoGCM0i61pBjkxcggU57gHKf2jVwD7HWMHgH-Y/edit?usp=sharing
Bug: Add safe.directory for container based job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/22 02:06
Affects Version/s: 3.4.0

{code:java}
```
fatal: unsafe repository ('/__w/spark/spark' is owned by someone else)
To add an exception for this directory, call:
    git config --global --add safe.directory /__w/spark/spark
fatal: unsafe repository ('/__w/spark/spark' is owned by someone else)
To add an exception for this directory, call:
    git config --global --add safe.directory /__w/spark/spark
Error: Process completed with exit code 128.
``` {code}
https://github.blog/2022-04-12-git-security-vulnerability-announced/
[https://github.com/actions/checkout/issues/760]

```yaml
    - name: Github Actions permissions workaround
      run: |
        git config --global --add safe.directory ${GITHUB_WORKSPACE}
```
Bug: PySpark support numpy 1.23.X
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/22 02:09
Affects Version/s: 3.4.0

{code:java}
======================================================================
ERROR [2.102s]: test_arithmetic_op_exceptions (pyspark.pandas.tests.test_series_datetime.SeriesDateTimeTest) 
----------------------------------------------------------------------
Traceback (most recent call last):
File "/__w/spark/spark/python/pyspark/pandas/tests/test_series_datetime.py", line 99, in test_arithmetic_op_exceptions self.assertRaisesRegex(TypeError, expected_err_msg, lambda: other / psser) File "/usr/lib/python3.9/unittest/case.py", line 1276, in assertRaisesRegex return context.handle('assertRaisesRegex', args, kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 201, in handle callable_obj(*args, **kwargs)
File "/__w/spark/spark/python/pyspark/pandas/tests/test_series_datetime.py", line 99, in <lambda> self.assertRaisesRegex(TypeError, expected_err_msg, lambda: other / psser)
File "/__w/spark/spark/python/pyspark/pandas/base.py", line 465, in __array_ufunc__ 
raise NotImplementedError(NotImplementedError: pandas-on-Spark objects currently do not support <ufunc 'divide'>.
----------------------------------------------------------------------
{code}
Bug: Add Apache Spark infra GA image cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/22 01:31
Affects Version/s: 3.4.0

We should probably replace the base image (https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L302, https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage) to plain ubunto image w/ Docker image cache. See also https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md
Bug: partitionng by datestamp leads to wrong query on backend?
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 27/Jun/22 21:11
Affects Version/s: 3.3.0

Hello,

I am new to Apache spark, so please bear with me. I would like to report what seems to me a bug, but may be I am just not understanding something.

My goal is to run data analysis on a spark cluster. Data is stored in PostgreSQL DB. Tables contained timestamped entries (timestamp with time zone).

The code look like:

 {code:python}

from pyspark.sql import SparkSession

spark = SparkSession \
        .builder \
        .appName("foo") \
        .config("spark.jars", "/opt/postgresql-42.4.0.jar") \
        .getOrCreate()

df = spark.read \
     .format("jdbc") \
     .option("url", "jdbc:postgresql://example.org:5432/postgres") \
     .option("dbtable", "billing") \
     .option("user", "user") \
     .option("driver", "org.postgresql.Driver") \
     .option("numPartitions", "4") \
     .option("partitionColumn", "datestamp") \
     .option("lowerBound", "2022-01-01 00:00:00") \
     .option("upperBound", "2022-06-26 23:59:59") \
     .option("fetchsize", 1000000) \
     .load()

t0 = time.time()
print("Number of entries is =====> ", df.count(), " Time to execute ", time.time()-t0)
...
{code}

datestamp is timestamp with time zone. 

I see this query on DB backend:


{code:java}
SELECT 1 FROM billinginfo  WHERE "datestamp" < '2022-01-02 11:59:59.9375' or "datestamp" is null
{code}


The table is huge and entries go way back before 
 2022-01-02 11:59:59. So what ends up happening - all workers but one complete and one remaining continues to process that query which, to me, looks like it wants to get all the data before 2022-01-02 11:59:59. Which is not what I intended. 

I remedies this by changing to:


{code:python}
     .option("dbtable", "(select * from billinginfo where datestamp > '2022 01-01-01 00:00:00') as foo") \
{code}


And that seem to have solved the issue. But this seems kludgy. Am I doing something wrong or there is a bug in the way partitioning queries are generated?
Bug: Asynchronous State Checkpointing in Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 21:00
Affects Version/s: 3.3.0

We can reduce the latency of stateful pipelines in Structured Streaming by making state checkpoints asynchronous.  One of the major contributors of latency for stateful pipelines in Structured Streaming can be checkpointing the state changes of every micro-batch.  If we make the state checkpointing asynchronous, we can potentially significantly lower the latency of the pipeline as the state checkpointing won’t or will contribute less to the batch latency.
Bug: Configurable State Checkpointing Frequency in Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 21:01
Affects Version/s: 3.3.0

Currently, for stateful pipelines state changes are checkpointed for every micro-batch. State checkpoints can contribute significantly to the latency of a micro-batch.  If state is checkpointed less frequently, its effect on batch latency can be amortized.  This can be used in conjunction with asynchronous state checkpointing to further reduce the cost in latency state checkpointing may incur.
Bug: Invoking .repartition(100000) in a unit test causes the unit test to take >20 minutes.
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/22 05:07
Affects Version/s: 3.3.0

Here's a proof of concept: 

{code}
val result = spark
      .createDataset(List("test"))
      .rdd
      .repartition(100000)
      .map { _ =>
        "test"
      }
      .collect()
      .toList
 
    println(result)
{code}

This code takes a very long time in unit test.

We aim to test for correctness in unit test... not testing the repartition. 

Is there a way to make it faster? (e.g. disable partition in test)
Bug: Dataset planning in a unit test takes a very long time to finish (e.g. >8mins for complex job)
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/22 05:11
Affects Version/s: 3.3.0

At Stripe, we have a very complex data job. The unit test was running fine when we used RDD.

After we switched to Dataset, the unit test takes considerably longer (e.g. > 8 mins just for planning).

Most of our unit tests only process 1-2 records.

We have tried to investigate it a bit, and we are somewhat sure it's the planning phrase.

We tried disabling almost all optimizers except the ~10 optimizers that can't be disabled. It doesn't impact the test run time at all.

Is there a way to make dataset plan faster in unit test.

Thank you!

(Please excuse us. I may use inaccurate term.)
Bug: Asynchronous I/O API support in Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 20:58
Affects Version/s: 3.3.0

Often in streaming pipelines used to support ETL use cases, API calls to external systems can be found.  To better support and improve performance for such use cases, an asynchronous processing API should be added to Structured Streaming so that external requests can be handled in a more efficient manner.
Bug: PySpark df.count() operation works fine on DBR 7.3 LTS but fails in DBR 10.4 LTS
Status: Open | Priority: Major | Resolution: 
Created: 26/Jun/22 08:49
Affects Version/s: 3.2.1

I have a job that infers schema from mongodb and does operations such as flattening and unwinding because there are nested fields. After performing various transformations, finally when the count() is performed, it works perfectly fine in databricks runtime version 7.3 LTS but fails to perform the same in 10.4 LTS.

*Below is the image that shows successful run in 7.3 LTS:*

!https://docs.microsoft.com/answers/storage/attachments/215035-image.png|width=630,height=75!

*Below is the image that shows failure in 10.4 LTS:*

!image-2022-06-27-11-00-50-119.png|width=624,height=64!

And I have validated that there is no field in our schema that has NullType. In fact when the schema was inferred, there were Null & void type fields which were converted to string using my UDF. This issue will persists even when I infer schema on complete dataset, that is, samplePoolSize is on full data set.
Bug: Multiple Stateful Operators in Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 20:50
Affects Version/s: 3.3.0

Currently, Structured Streaming only supports one stateful operator per pipeline.  This constraint excludes Structured Streaming to be used for many use cases such ones involving chained time window aggregations, chained stream-stream outer equality join, stream-stream time interval join followed by time window aggregation.  Enabling multiple stateful operators within a pipeline in Structured Streaming will open up many additional use cases for its usage as well as be on par with other stream processing engines.
Bug: Advanced Windowing in Structured Streaming
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 20:51
Affects Version/s: 3.3.0

Currently in Structure Streaming there does not exist a user friendly method to define custom windowing strategies. Users can only choose from an existing set of built-in window strategies such tumbling, sliding, session windows.  To improve flexibility of the engine and to support more advanced use cases, Structured Streaming needs to provide an intuitive API that allows users to define custom window boundaries, to trigger windows when they are complete, and, if necessary, evict elements from windows.
Bug: Schema Evolution for Stateful Pipelines in Structured Streaming 
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/22 20:55
Affects Version/s: 3.3.0

Support for schema evolution of stateful operators is non-existent. There is no clear path to evolve the schema for existing built-in stateful operators.  For use defined stateful operators such as map/flatMapGroupsWithState, there is no path for evolving state as well.
Bug: Recover branch-3.2 build broken by SPARK-39273 and SPARK-39252
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 08:56
Affects Version/s: 3.2.1

https://github.com/apache/spark/runs/6584105591?check_suite_focus=true
https://github.com/apache/spark/runs/6584981916?check_suite_focus=true
Bug: Use JAVA_OPTS for AppVeyer build to increase the memory properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/22 02:39
Affects Version/s: 3.4.0


https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/43740704

AppVeyor build is being failed because of the lack of memory.
Should use JAVA_OPTS to make the memory configured properly.
Bug: Test failure in SPARK-39387 with JDK 11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/22 01:23
Affects Version/s: 3.4.0


{code}
[info] - SPARK-39387: BytesColumnVector should not throw RuntimeException due to overflow *** FAILED *** (3 seconds, 393 milliseconds)
[info]   org.apache.spark.SparkException: Job aborted.
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:593)
[info]   at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:279)
[info]   at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186)
[info]   at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
[info]   at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
[info]   at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
[info]   at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
[info]   at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:111)
[info]   at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:171)
[info]   at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
[info]   at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
[info]   at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
[info]   at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
[info]   at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
[info]   at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
[info]   at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
{code}

https://github.com/apache/spark/runs/6919076419?check_suite_focus=true
Bug: How to get access to the data created in different Spark Applications
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 21/Jun/22 21:25
Affects Version/s: 3.3.0

I am working on a project using PySpark and I am blocked because I want to share data between different Spark applications. The situation is that we have a running java server which can handles incoming requests with a thread pool, and each thread has a corresponding python process. We want to use pandas on Spark, but have it so that any of the python processes can access the same data in spark. For example, in a python process, we created a SparkSession, read some data, modified the data using pandas on Spark api and we want to get access to that data in a different python process. The core problem is how to share data between different SparkSession or how to let different python process connect to the same SparkSession. I researched a bit but it seems impossible to share data between different python process without using external DB or connect to the same SparkSession. Generally, is this possible / what would be the recommended way to do this with the least impact on performance?
Bug: Define each workflow for each scheduled job in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jun/22 01:28
Affects Version/s: 3.4.0

it is very difficult to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.

Since GitHub added the feature to import other workflow, we should leverage this feature, see also https://github.com/apache/spark/blob/master/.github/workflows/build_and_test_ansi.yml. Once we can separate them, it will be defined as a separate workflows. We might have to clean up existing build history to make it easier to read (it requires committer-only permission to clean them up).

Namely, each scheduled job should be classified under "All workflows" at https://github.com/apache/spark/actions so other developers can easily track them.

See also https://docs.github.com/en/actions/using-workflows/reusing-workflows
Bug: Move checkout and sync steps into re-usable composite action
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 20/Jun/22 10:15
Affects Version/s: 3.4.0

Steps "Checkout Spark repository" and "Sync the current branch with the latest in Apache Spark" are defined 9 times in `build-and-test.yml`. They should be defined in a composite action once, that is than reused. Simplifies workflow file and reduces code duplication.
Bug: Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/22 15:37
Affects Version/s: 3.4.0

trying to build with the hadoop-2 profile, which uses hadoop 2.7 version fails due to:

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:454: value DECOMMISSIONING is not a member of object org.apache.hadoop.yarn.api.records.NodeState

 

DECOMMISSIONING was only added in hadoop 2.8.

This was added in https://issues.apache.org/jira/browse/SPARK-30835
Bug: Support function: percentile_approx、percentile_count、percentile_disc
Status: Open | Priority: Major | Resolution: 
Created: 17/Jun/22 02:55
Affects Version/s: 3.4.0

[https://docs.databricks.com/spark/latest/spark-sql/language-manual/functions/percentile_approx.html]

[https://docs.databricks.com/spark/latest/spark-sql/language-manual/functions/percentile_cont.html]

[https://docs.databricks.com/spark/latest/spark-sql/language-manual/functions/percentile_disc.html]