av9ash/GitBugs huggingface dataset:
Bug: Install torchvision for Python 3.12 build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/23 02:38
Affects Version/s: 4.0.0

{code}
======================================================================
ERROR [0.001s]: test_end_to_end_run_distributedly (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorDistributedUnitTestsOnConnect.test_end_to_end_run_distributedly)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 495, in test_end_to_end_run_distributedly
    train_fn = create_training_function(self.mnist_dir_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function
    from torchvision import transforms, datasets
ModuleNotFoundError: No module named 'torchvision'

======================================================================
ERROR [0.001s]: test_end_to_end_run_locally (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorLocalUnitTestsIIOnConnect.test_end_to_end_run_locally)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 402, in test_end_to_end_run_locally
    train_fn = create_training_function(self.mnist_dir_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function
    from torchvision import transforms, datasets
ModuleNotFoundError: No module named 'torchvision'

======================================================================
ERROR [0.001s]: test_end_to_end_run_locally (pyspark.ml.tests.connect.test_parity_torch_distributor.TorchDistributorLocalUnitTestsOnConnect.test_end_to_end_run_locally)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 402, in test_end_to_end_run_locally
    train_fn = create_training_function(self.mnist_dir_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 60, in create_training_function
    from torchvision import transforms, datasets
ModuleNotFoundError: No module named 'torchvision'

----------------------------------------------------------------------
Ran 23 tests in 50.860s
{code}
Bug: Enhancing PySpark documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/23 20:37
Affects Version/s: 4.0.0

Currently, the PySpark documentation is unable to utilize many of the advanced features available in recent Sphinx versions due to older package versions. This limitation hinders the documentation's visual appeal and usability, particularly when compared to other projects like Pandas which have already adopted these enhancements. By updating the Sphinx and Jinja2 versions, we can significantly improve the documentation's layout, design, and interactive features, thereby enhancing the end-user experience.
Bug: Adding "Q&A Support" and "Mailing Lists" link into PySpark doc homepage.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 05:40
Affects Version/s: 4.0.0

It is aimed at improving user engagement and providing quick access to community support and discussions. This approach is inspired by the [Pandas documentation](https://pandas.pydata.org/docs/index.html), which effectively uses a similar section for community engagement.

The "Q&A Support" will lead users to a curated list of StackOverflow questions tagged with `pyspark`, while the mailing lists will offer platforms for deeper discussions and insights within the Spark community.
Bug: YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/23 09:53
Affects Version/s: 3.1.3

We meet a case that user call sc.stop() after run all custom code, but stuck in some place. 

Cause below situation
 # User call sc.stop()
 # sc.stop() stuck in some process, but SchedulerBackend.stop was called
 # Since tarn ApplicationMaster didn't finish， still call YarnAllocator.allocateResources()
 # Since driver endpoint stop new allocated executor failed to register
 # untll trigger Max number of executor failures

Caused by 

Before call CoarseGrainedSchedulerBackend.stop() will call YarnSchedulerBackend.requestTotalExecutor() to clean request info

!image-2023-11-20-17-56-56-507.png|width=898,height=297!

 

From the log we make sure that CoarseGrainedSchedulerBackend.stop()  was called

 

 

When YarnAllocator handle then empty resource request,  since resourceTotalExecutorsWithPreferedLocalities is empty, miss clean targetNumExecutorsPerResourceProfileId.

!image-2023-11-20-17-56-45-212.png|width=708,height=379!
Bug: SQL on streaming Temp view fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/23 18:59
Affects Version/s: 4.0.0

The following code fails in the last step with Spark Connect.

The root cause is that Connect server triggers physical plan on a streaming Dataframe [in SparkConnectPlanner.scala|https://github.com/apache/spark/blob/334d952f9555cbfad8ef84987d6f978eb6b37b9b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L2591]. Better to avoid that entirely, but at least for streaming it should be avoided since it cannot be done with a batch execution engine.
{code:java}
df = spark.readStream.format("rate").option("numPartitions", "1").load()
df.createOrReplaceTempView("temp_view")

view_df = spark.sql("SELECT * FROM temp_view") // FAILS{code}
Bug: Using brighter color for document title for better visibility
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 09:03
Affects Version/s: 4.0.0

With the increasing popularity of dark mode for its eye comfort and energy-saving benefits, it's important to ensure that our documentation is easily readable in both light and dark settings. The current title font color in dark mode is not optimal for readability, which can hinder user experience. By adjusting the color, we aim to enhance the overall accessibility and readability of the PySpark documentation in dark mode.
Bug: Update `pydata_sphinx_theme` version requirement to >=0.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 03:38
Affects Version/s: 4.0.0

Updating to `>=0.13` ensures that we can benefit from the latest improvements and bug fixes in `pydata_sphinx_theme` without being restricted to a specific patch version. It helps maintain the documentation with the latest features available in the theme.
Bug: Prune keys or values from Generate if it is a map type
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/23 14:50
Affects Version/s: 4.0.0

CREATE TABLE t(id STRING,  features MAP<STRING, DOUBLE>) USING parquet;
Rewrite:
```
SELECT id, key FROM (SELECT id, explode(features) FROM t) tmp
```
to
```
SELECT id, key FROM (SELECT explode(keys) AS key FROM (select map_keys(features) AS keys FROM t) t) t
```
Bug: Restore documentation for DSv2 API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/23 23:53
Affects Version/s: 3.4.1

DSv2 documentation is mistakenly gone after https://github.com/apache/spark/pull/38392. It used to exist in 3.3.0: https://spark.apache.org/docs/3.3.0/api/scala/org/apache/spark/sql/connector/catalog/index.html
Bug: Precompile regex patterns in SparkDateTimeUtils.getZoneId
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 10:03
Affects Version/s: 3.5.0

SparkDateTimeUtils.getZoneId uses String.replaceFirst method, that internally does a Pattern.compile(regex). This method is called once for each dataset row when using functions like from_utc_timestamp.
Bug: CTE reference node does not inherit the flag `isStreaming` from CTE definition node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 02:28
Affects Version/s: 3.3.2

Looks like this is a long standing bug.

We figured out that CTE reference node would never set the isStreaming flag to true, regardless of the value for flag in CTE definition. The node cannot determine the right value of isStreaming flag by itself (likewise it cannot determine about resolution by itself) but it has no parameter in constructor, hence always takes the default (no children, so batch one).

This may impact some rules which behaves differently depending on isStreaming flag. It would no longer be a problem once CTE reference is replaced with CTE definition at some point in "optimization phase", but all rules in analyzer and optimizer being triggered before the rule takes effect may be impacted.

We probably couldn't sync the flag in real time, but we should sync the flag when we mark CTE reference to be "resolved". The rule `ResolveWithCTE` will be a good place to do that.
Bug: Add the test party for reattach test case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 01:07
Affects Version/s: 4.0.0

We need the same test "ReleaseSession releases all queries and does not allow more requests in the session" added in SPARK-45798 to identify an issue like SPARK-46042.

This is caused by SPARK-46039
Bug: EliminateEventTimeWatermark does not consider the fact that isStreaming flag can change for current child during resolution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 05:05
Affects Version/s: 3.3.2

Looks like this is a long standing bug.

The object `EliminateEventTimeWatermark` is implemented as a rule, but it is not registered in analyzer/optimizer. Instead, it is called directly when withWatermark method is called, which means the rule is applied immediately against the child, regardless whether child is resolved or not.

It is not an issue for the usage of pure DataFrame API because streaming sources have the flag isStreaming set to true even it is yet resolved, but mix-up of SQL and DataFrame API would expose the issue; we may not know the exact value of isStreaming flag on unresolved node and it is subject to change upon resolution.

We should register EliminateEventTimeWatermark as a rule on analysis (or pre-optimization) instead, and do not apply the elimination if the child is not yet resolved.
Bug: Improve error message when using a string literal where only an identifier can go
Status: Open | Priority: Major | Resolution: 
Created: 23/Nov/23 07:50
Affects Version/s: 3.4.2

The following:
{noformat}
*nospark-sql (default)> select * from "t";


[PARSE_SYNTAX_ERROR] Syntax error at or near '"t"'. SQLSTATE: 42601 (line 1, pos 14)


== SQL ==
select * from "t"
--------------^^^

{noformat}
.. is confusing if one is used to double quotes for identifiers.
Similarly, it is easy to mix up ' and `.
So we would like to return an error that clearly states that a string was given where an identifier was expected. We can also propose using spark.sql.ansi.double_quoted_identifiers in that case.
Bug: Install `six==1.16.0` explicitly for `pandas` in Python 3.12
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 00:13
Affects Version/s: 4.0.0

This happens in Python 3.12 CI only.
- https://github.com/apache/spark/actions/runs/6959106836/job/18935673389
{code}
Starting test(python3.12): pyspark.streaming.tests.test_context (temp output: /__w/spark/spark/python/target/73ed28d0-ae18-426e-9760-d03bea982a9b/python3.12__pyspark.streaming.tests.test_context__l4z6a7a2.log)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/__w/spark/spark/python/pyspark/streaming/tests/test_context.py", line 23, in <module>
    from pyspark.testing.streamingutils import PySparkStreamingTestCase
  File "/__w/spark/spark/python/pyspark/testing/__init__.py", line 19, in <module>
    from pyspark.testing.pandasutils import assertPandasOnSparkEqual
  File "/__w/spark/spark/python/pyspark/testing/pandasutils.py", line 58, in <module>
    import pyspark.pandas as ps
  File "/__w/spark/spark/python/pyspark/pandas/__init__.py", line 33, in <module>
    require_minimum_pandas_version()
  File "/__w/spark/spark/python/pyspark/sql/pandas/utils.py", line 27, in require_minimum_pandas_version
    import pandas
  File "/usr/local/lib/python3.12/dist-packages/pandas/__init__.py", line 46, in <module>
    from pandas.core.api import (
  File "/usr/local/lib/python3.12/dist-packages/pandas/core/api.py", line 1, in <module>
    from pandas._libs import (
  File "/usr/local/lib/python3.12/dist-packages/pandas/_libs/__init__.py", line 18, in <module>
    from pandas._libs.interval import Interval
  File "interval.pyx", line 1, in init pandas._libs.interval
  File "hashtable.pyx", line 1, in init pandas._libs.hashtable
  File "missing.pyx", line 42, in init pandas._libs.missing
AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)
{code}
Bug: Upgrade ZSTD-JNI to 1.5.5-10
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/23 12:31
Affects Version/s: 4.0.0

[https://github.com/luben/zstd-jni/releases/tag/v1.5.5-8]

[https://github.com/luben/zstd-jni/releases/tag/v1.5.5-9]
Bug: Isolated classloader per spark session
Status: Open | Priority: Major | Resolution: 
Created: 22/Nov/23 03:33
Affects Version/s: 3.5.0

Hello,

We use spark 3.5.0 and were wondering if the following is achievable using spark-core

Our use case involves spinning up a spark cluster wherein the driver application loads user jars on-the-fly ( the user jar is not the spark driver/application ) but merely a catalog of transformations. A single spark application can load multiple jars in its lifetime with potential of classpath conflict if care is not taken by the framework

The driver needs to load the jar, add the jar to the executor & call a predefined class.method to trigger the transformation

Each transformation runs in its own spark session inside the same spark application

AFAIK, on the executor side, isolated classloader per session is only possible when using the spark-connect facilities. Is it possible to do this without using spark connect?

Spark connect is the only facility that adds the jar into a sessionUUID directory of executor and when an executor runs tasks of a job from that session, it sets a ChildFirstClassLoader pointing to the sessionUUID directory

 

Thank you
Bug: support grouping set operation in dataframe api
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/23 02:58
Affects Version/s: 3.4.1

I am using spark dataframe api for complex calculations. When I need to use the grouping sets function, I can only convert the expression to sql via analyzedPlan and then splice these sql into a complex sql to execute. In some cases, this operation generates an extremely complex sql. executing this complex sql, antlr4 continues to consume a large amount of memory, similar to a memory leak scenario. If you can and rollup, cube function through the dataframe api to calculate these operations will be much simpler.
Bug: Fix Pearson correlation calculation more stable
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 08/Nov/23 05:38
Affects Version/s: 3.5.0

Spark uses the formula {{ck / sqrt(xMk * yMk)}} to calculate the Pearson Correlation Coefficient. If {{xMk}} and {{yMk}} are very small, it can lead to double multiplication overflow, resulting in a denominator of 0. This leads to an Infinity result in the calculation.

For example, when calculating the correlation for the same columns a and b in the following table, the result will be Infinity, but the correlation for identical columns should be 1.0 instead.
||a||b||
|1e-200|1e-200|
|1e-200|1e-200|
|1e-100|1e-100|

Modifying the formula to {{ck / sqrt(xMk) / sqrt(yMk)}} can indeed solve this issue and improve the stability of the calculation. The benefit of this modification is that it splits the square root of the denominator into two parts: {{sqrt(xMk)}} and {{{}sqrt(yMk){}}}. This helps avoid multiplication overflow or cases where the product of extremely small values becomes zero.
Bug: Gracefully deal with recursion exception during Spark Connect logging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/23 11:44
Affects Version/s: 3.5.0

```
from google.protobuf.text_format import MessageToString
from pyspark.sql.functions import col, lit

df = spark.range(10)

for x in range(800):
  df = df.withColumn(f"next{x}", lit(1))
  MessageToString(df._plan.to_proto(spark._client), as_one_line=True)

```
Bug: Show proper dependency requirement messages for Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/23 01:17
Affects Version/s: 4.0.0


{code}
./bin/pyspark --remote local
{code}

We should improve the error messages below.

{code}
/.../pyspark/shell.py:57: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../pyspark/shell.py", line 52, in <module>
    spark = SparkSession.builder.getOrCreate()
  File "/.../pyspark/sql/session.py", line 476, in getOrCreate
    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
  File "/.../pyspark/sql/connect/session.py", line 53, in <module>
    from pyspark.sql.connect.client import SparkConnectClient, ChannelBuilder
  File "/.../pyspark/sql/connect/client/__init__.py", line 22, in <module>
    from pyspark.sql.connect.client.core import *  # noqa: F401,F403
  File "/.../pyspark/sql/connect/client/core.py", line 51, in <module>
    import google.protobuf.message
ModuleNotFoundError: No module named 'google
{code}

{code}
/.../pyspark/shell.py:57: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/.../pyspark/shell.py", line 52, in <module>
    spark = SparkSession.builder.getOrCreate()
  File "/.../pyspark/sql/session.py", line 476, in getOrCreate
    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
  File "/.../pyspark/sql/connect/session.py", line 53, in <module>
    from pyspark.sql.connect.client import SparkConnectClient, ChannelBuilder
  File "/.../pyspark/sql/connect/client/__init__.py", line 22, in <module>
    from pyspark.sql.connect.client.core import *  # noqa: F401,F403
  File "/.../pyspark/sql/connect/client/core.py", line 52, in <module>
    from grpc_status import rpc_status
ModuleNotFoundError: No module named 'grpc_status'
{code}
Bug: Only do the thread interruption check for putIterator on executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Nov/23 21:27
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-45025 

introduces a peaceful thread interruption handling. However, there is an edge case: when a streaming query is stopped on the driver, it interrupts the stream execution thread. If the streaming query is doing memory store operations on driver and performs {{doPutIterator}} at the same time, the [unroll process will be broken|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L224] and [returns used memory|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L245-L247].

This can result in {{closeChannelException}} as it falls into this [case clause|https://github.com/apache/spark/blob/aa646d3050028272f7333deaef52f20e6975e0ed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1614-L1622] which opens an I/O channel and persists the data into the disk. However, because the thread is interrupted, the channel will be closed at the begin: [https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L172] and throws out {{closeChannelException}}

On executors, [the task will be killed if the thread is interrupted|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L374], however, we don't do it on the driver.
Bug: Fix `pyspark.ml.torch.tests.test_distributor` in Python 3.11
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/23 23:05
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/6914662405/job/18812759511

{code}
======================================================================
FAIL [0.000s]: test_local_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests.test_local_training_succeeds) [subtest: 1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 384, in test_local_training_succeeds
    self.assertEqual(
AssertionError: '1' != '0'
- 1
+ 0


======================================================================
FAIL [0.142s]: test_local_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTests.test_local_training_succeeds) [subtest: 2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 384, in test_local_training_succeeds
    self.assertEqual(
AssertionError: '1,2,0' != '0,1,2'
- 1,2,0
+ 0,1,2


======================================================================
FAIL [0.000s]: test_local_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTestsII.test_local_training_succeeds) [subtest: 1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 384, in test_local_training_succeeds
    self.assertEqual(
AssertionError: '1' != '0'
- 1
+ 0


======================================================================
FAIL [0.139s]: test_local_training_succeeds (pyspark.ml.torch.tests.test_distributor.TorchDistributorLocalUnitTestsII.test_local_training_succeeds) [subtest: 2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/ml/torch/tests/test_distributor.py", line 384, in test_local_training_succeeds
    self.assertEqual(
AssertionError: '1,2,0' != '0,1,2'
- 1,2,0
+ 0,1,2


----------------------------------------------------------------------
Ran 23 tests in 166.741s

FAILED (failures=4)
{code}