av9ash/GitBugs huggingface dataset:
Bug: Refactor `StreamingDataSourceRelation` into `StreamingDataSourceRelation` and `StreamingDataSourceScanRelation` for parity with batch scan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/24 18:18
Affects Version/s: 4.0.0

To prepare for the incoming structured streaming operator pushdown, we'd like to refactor some catalyst object relationships first.
Bug: Bad SQLSTATE "ID001" for INVALID_INVERSE_DISTRIBUTION_FUNCTION 
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/24 01:16
Affects Version/s: 4.0.0

 INVALID_INVERSE_DISTRIBUTION_FUNCTION 
The "ID" SQLSTATE class is undefined and it is heavy handed to consume a class for a function.
In Spark we use the K** subclass to expand the existing range with Spark private states 
Since this error appears to be compile time I propose "42" as a class.
the next free slot in 42K** is:
42K0K
Bug: Do not register state store provider into coordinator in state data source - reader
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/24 07:19
Affects Version/s: 4.0.0

Currently we initialize state store instance via StateStore.get() which initializes state store provider instance and registers the provider instance to the coordinator. This involves unnecessary overheads e.g. maintenance task could be triggered for this provider.

The state data source should just initialize the provider and manage the provider instance by itself.
Bug: InMemoryTableScanExec#outputPartitioning does not update the KeyGroupedPartitioning#expressions
Status: Open | Priority: Major | Resolution: 
Created: 19/Jan/24 08:08
Affects Version/s: 3.5.0

Only expression-based partitionings get updated, so the KeyGroupedPartitioning has old attribute ExprId and EnsureRequirements has to reshuffle.

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala#L140]

I think this is not trivial to reproduce, as no source exposes the KeyGroupedPartitioning yet, but if you have a DataFrame with outputPartitioning: KeyGroupedPartitioning, apply .cache() to it and try to sort-merge-join it with other DataFrame partitioned the same way, the data-frames will be reshuffled.
Bug: dropDuplicatesWithinWatermark throws error on canonicalizing plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/24 08:54
Affects Version/s: 3.5.0

Simply said, this test code fails:
{code:java}
test("SPARK-XXXXX: canonicalization of StreamingDeduplicateWithinWatermarkExec should work") {
  withTempDir { checkpoint =>
    val dedupeInputData = MemoryStream[(String, Int)]
    val dedupe = dedupeInputData.toDS()
      .withColumn("eventTime", timestamp_seconds($"_2"))
      .withWatermark("eventTime", "10 second")
      .dropDuplicatesWithinWatermark("_1")
      .select($"_1", $"eventTime".cast("long").as[Long])

    testStream(dedupe, Append)(
      StartStream(checkpointLocation = checkpoint.getCanonicalPath),

      AddData(dedupeInputData, "a" -> 1),
      CheckNewAnswer("a" -> 1),

      Execute { q =>
        // This threw out error!
        q.lastExecution.executedPlan.canonicalized
      }
    )
  }
} {code}
with below error:
{code:java}
[info] - SPARK-XXXXX: canonicalization of StreamingDeduplicateWithinWatermarkExec should work *** FAILED *** (1 second, 237 milliseconds)
[info]   Assert on query failed: Execute: None.get
[info]   scala.None$.get(Option.scala:627)
[info]       scala.None$.get(Option.scala:626)
[info]       org.apache.spark.sql.execution.streaming.StreamingDeduplicateWithinWatermarkExec.<init>(statefulOperators.scala:1101)
[info]       org.apache.spark.sql.execution.streaming.StreamingDeduplicateWithinWatermarkExec.copy(statefulOperators.scala:1092)
[info]       org.apache.spark.sql.execution.streaming.StreamingDeduplicateWithinWatermarkExec.withNewChildInternal(statefulOperators.scala:1148)
[info]       org.apache.spark.sql.execution.streaming.StreamingDeduplicateWithinWatermarkExec.withNewChildInternal(statefulOperators.scala:1087)
[info]       org.apache.spark.sql.catalyst.trees.UnaryLike.withNewChildrenInternal(TreeNode.scala:1210)
[info]       org.apache.spark.sql.catalyst.trees.UnaryLike.withNewChildrenInternal$(TreeNode.scala:1208)
[info]       org.apache.spark.sql.execution.streaming.BaseStreamingDeduplicateExec.withNewChildrenInternal(statefulOperators.scala:949)
[info]       org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$withNewChildren$2(TreeNode.scala:323) {code}
Bug: SHOW CREATE TABLE shows incomplete information for STRUCT type fields.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/24 02:48
Affects Version/s: 4.0.0

`SHOW CREATE TABLE` does not return struct fields `NOT NULL` qualifiers as well as comments.

E.g.
```
CREATE TABLE test(a STRUCT<b: INT NOT NULL, c: STRING COMMENT 'comment'>);
SHOW CREATE TABLE test;
```

Result:
```
CREATE TABLE test(a STRUCT<b: INT, c: STRING)
```
Bug: In ResourceProfileManager, function calls should occur after variable declarations.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/24 06:36
Affects Version/s: 3.5.0

As the title suggests, in *ResourceProfileManager*, function calls should be made after variable declarations. When determining *isSupport*, all variables are uninitialized, with booleans defaulting to false and objects to null. While the end result is correct, the evaluation process is abnormal.
Bug: Use default ORC compression in data source benchmarks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/24 02:37
Affects Version/s: 4.0.0

{code}
$ git grep OrcCompressionCodec | grep Benchmark
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BuiltInDataSourceWriteBenchmark.scala:import org.apache.spark.sql.execution.datasources.orc.OrcCompressionCodec
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BuiltInDataSourceWriteBenchmark.scala:      OrcCompressionCodec.SNAPPY.lowerCaseName())
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala:import org.apache.spark.sql.execution.datasources.orc.OrcCompressionCodec
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala:      OrcCompressionCodec.SNAPPY.lowerCaseName()).orc(dir)
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala:import org.apache.spark.sql.execution.datasources.orc.OrcCompressionCodec
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala:      .setIfMissing("orc.compression", OrcCompressionCodec.SNAPPY.lowerCaseName())
{code}
Bug: Fix add in SQLMetric
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 01:51
Affects Version/s: 3.5.0

A previous refactor mistakenly used `isValid` for add. Since `defaultValidValue` was always `0`, this didn't cause any correctness issues.

What we really want to do for add (and merge) is `if (isZero) _value = 0`.
Bug: Inject LocalLimitExec when matching OffsetAndLimit or LimitAndOffset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/24 04:14
Affects Version/s: 3.5.0

For queries containing both a LIMIT and an OFFSET in a subquery, physical translation will drop the `LocalLimit` planned in the optimizer stage by mistake; this manifests as larger than necessary shuffle sizes for `GlobalLimitExec`. Fix to not drop this node.
Bug: ReattachExecute requests do not refresh aliveness of SessionHolder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 20:04
Affects Version/s: 4.0.0

In the first executePlan request, creating the {{ExecuteHolder}} triggers  {{getOrCreateIsolatedSession}} which refreshes the aliveness of {{{}SessionHolder{}}}. However in {{ReattachExecute}} , we fetch the {{ExecuteHolder}} directly without going through the {{SessionHolder}} (and hence making it seem like the {{SessionHolder}} is idle).

 

This would result in long-running queries (which do not send release execute requests since that refreshes aliveness) failing because the {{SessionHolder}} would expire during active query execution.
Bug: Support `s.c.immutable.ArraySeq` as `customCollectionCls` in `MapObjects`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/24 03:08
Affects Version/s: 4.0.0

{code:java}
val myUdf = udf((a: immutable.ArraySeq[Int]) =>
    immutable.ArraySeq.unsafeWrapArray[Int](Array(a.head + 99)))
  checkAnswer(Seq(Array(1))
    .toDF("col")
    .select(myUdf(Column("col"))),
    Row(ArrayBuffer(100))) {code}
will test failed in Spark 4.0 when using Scala 2.13
Bug: CoGroup.applyInPandas/Arrow should pass arguments properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/24 22:18
Affects Version/s: 3.5.0

In Spark Connect, {{CoGroup.applyInPandas/Arrow}} doesn't take arguments properly, so the arguments of the UDF can be broken:
{noformat}
>>> import pandas as pd
>>>
>>> df1 = spark.createDataFrame(
...     [(1, 1.0, "a"), (2, 2.0, "b"), (1, 3.0, "c"), (2, 4.0, "d")], ("id", "v1", "v2")
... )
>>> df2 = spark.createDataFrame([(1, "x"), (2, "y"), (1, "z")], ("id", "v3"))
>>>
>>> def summarize(left, right):
...     return pd.DataFrame(
...         {
...             "left_rows": [len(left)],
...             "left_columns": [len(left.columns)],
...             "right_rows": [len(right)],
...             "right_columns": [len(right.columns)],
...         }
...     )
...
>>> df = (
...     df1.groupby("id")
...     .cogroup(df2.groupby("id"))
...     .applyInPandas(
...         summarize,
...         schema="left_rows long, left_columns long, right_rows long, right_columns long",
...     )
... )
>>>
>>> df.show()
+---------+------------+----------+-------------+
|left_rows|left_columns|right_rows|right_columns|
+---------+------------+----------+-------------+
|        2|           1|         2|            1|
|        2|           1|         1|            1|
+---------+------------+----------+-------------+
{noformat}

The result should be:

{noformat}
+---------+------------+----------+-------------+
|left_rows|left_columns|right_rows|right_columns|
+---------+------------+----------+-------------+
|        2|           3|         2|            2|
|        2|           3|         1|            2|
+---------+------------+----------+-------------+
{noformat}
Bug: Interrupt when executing ANALYSIS phase
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 04/Jan/24 04:05
Affects Version/s: 3.4.0

I have a long-running spark app on which I will start many tasks. When I am executing complex tasks, I may spend a lot of time in the ANALYSIS phase or the OPTIMIZATION phase or run into oom exceptions. I will cancel the task when timeout is detected by calling cancelJobGroup method. However, the task is not interrupted and the execution plan is still being generated. I{*}s there a way to interrupt these phases?{*}
Bug: Streaming UI hover-over shows incorrect value
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/24 21:58
Affects Version/s: 4.0.0

Running a simple streaming query:

val df = spark.readStream.format("rate").option("rowsPerSecond", "50000000").load()

val q = df.writeStream.format("noop").start()

 

The hover-over value is incorrect in the streaming ui (shows 321.00 at undefined)

 
!Screenshot 2024-01-08 at 1.55.57 PM.png!
Bug: Pyspark throwing TypeError while collecting a RDD
Status: Open | Priority: Major | Resolution: 
Created: 09/Jan/24 15:32
Affects Version/s: 3.4.1

Im trying to collect a RDD after applying a filter on it but its throwing an error.

 

Error can be reproduced from below code
{code:java}
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("Practice").getOrCreate()

sc = spark.sparkContext

data = [1,2,3,4,5,6,7,8,9,10,11,12]
dataRdd = sc.parallelize(data)

dataRdd = dataRdd.filter(lambda a: a%2==0)

dataRdd.collect() {code}
Below is the error that its throwing:

 
{code:java}
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[18], line 1 ----> 1 dataRdd.collect() 
File ~\anaconda3\envs\spark_latest\Lib\site-packages\pyspark\rdd.py:
1814, in RDD.collect(self)  
1812 with SCCallSiteSync(self.context):  
1813 assert self.ctx._jvm is not None 
-> 1814 sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())  
1815 return list(_load_from_socket(sock_info, self._jrdd_deserializer)) 
File ~\anaconda3\envs\spark_latest\Lib\site-packages\pyspark\rdd.py:
5441, in PipelinedRDD._jrdd(self)  
438 else:  
5439 profiler = None 
-> 5441 wrapped_func = _wrap_function(  
5442 self.ctx, self.func, self._prev_jrdd_deserializer, self._jrdd_deserializer, profiler  
5443 )  
5445 assert self.ctx._jvm is not None  
5446 python_rdd = self.ctx._jvm.PythonRDD(  
5447 self._prev_jrdd.rdd(), wrapped_func, self.preservesPartitioning, self.is_barrier  
5448 ) 
File ~\anaconda3\envs\spark_latest\Lib\site-packages\pyspark\rdd.py:
5243, in _wrap_function(sc, func, deserializer, serializer, profiler)  
5241 pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)  
5242 assert sc._jvm is not None 
-> 5243 return sc._jvm.SimplePythonFunction(  
5244 bytearray(pickled_command),  
5245 env,  
5246 includes,  
5247 sc.pythonExec,  
5248 sc.pythonVer,  
5249 broadcast_vars,  
5250 sc._javaAccumulator,  
5251 ) 

TypeError: 'JavaPackage' object is not callable{code}
Bug: AccumulatorV2 isZero doesn't do what its name implies
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/24 23:32
Affects Version/s: 3.5.0

`AccumulatorV2`'s `isZero` doesn't do what the name or comment implies - it actually checks if the accumulator hasn't been updated.

The comment implies that for a `LongAccumulator`, for example, a value of `0` would cause `isZero` to be `true`. But if we were to `add(0)`, then the value would still be `0` but `isZero` would return `false`.

Propose to rename this to `isUpdated` so the name matches the meaning more closely.
Bug: Disallow re-registration of statically registered data sources
Status: Open | Priority: Major | Resolution: 
Created: 08/Jan/24 06:32
Affects Version/s: 4.0.0

This is a follow-up for SPARK-46522. Currently, Spark allows the re-registration of both statically and dynamically registered Python data sources. However, for (statically) registered Java/Scala data sources, Spark currently throws exceptions if users try to register the data soruce with the same name.

We should make this behavior consistent: either allow re-registration of all statically loaded data sources, or disallow them.
Bug: Python data source options should be a case insensitive dictionary
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/24 11:49
Affects Version/s: 4.0.0

Data source options are stored as a `CaseInsensitiveStringMap` in Scala, however, its behavior is inconsistent in Python:
{code:java}
class MyDataSource(DataSource):
    def __init__(self, options):
        self.api_key = options.get("API_KEY") # <- This is None

spark.read.format(..).option("API_KEY", my_key).load(...){code}
Currently, options will not have this "API_KEY" as everything is converted to lowercase on the Scala side. This can be confusing to users.
Bug: Remove invalid attachCleanupResourceChecker in JoinSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/24 01:58
Affects Version/s: 3.5.0

AttachCleanupResourceChecker in JoinSuite is invalid:

1. The matching of SortExec needs to be in `QueryExecution.executePlan` not `QueryExecution.sparkPlan`, The correct way is `foreachUp(df.queryExecution.executedPlan)\{f()}`.
2. `Mockito` counts the number of function calls, only for objects after `spy`. Calls to the original object are not counted. Therefore, when using df.queryExecution.executedPlan correctly to match, count is always 0.
3. Not all SortMergeJoin joinTypes will trigger cleanupResources(), such as 'full outer join'.
Bug: Refine getMessage for Spark Connect
Status: Open | Priority: Major | Resolution: 
Created: 04/Jan/24 01:24
Affects Version/s: 4.0.0

We added getMessage API from [https://github.com/apache/spark/pull/44292] to provide a simplified error message for users, but it's not supported in the same way for Spark Connect server side. We should match the behavior to fill the gap between two.
Bug: View simpleString not follow conf spark.sql.debug.maxToStringFields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/24 08:22
Affects Version/s: 3.2.4

{code:java}
override def simpleString(maxFields: Int): String = {
    s"View (${desc.identifier}, ${output.mkString("[", ",", "]")})"
  }
{code}
Will show all columns info
Bug: Extract a helper method to eliminate the duplicate code in `GrpcExceptionConverter` that retrieves `MessageParameters` from `ErrorParams`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/24 05:55
Affects Version/s: 4.0.0

{code:java}
params.errorClass match {
  case Some(_) => params.messageParameters
  case None => Map("message" -> params.message)
} {code}
The above code pattern appears 7 times in `GrpcExceptionConverter`.
Bug: FutureWarning for interpolate with object dtype
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/24 01:11
Affects Version/s: 4.0.0

>>> pdf.interpolate()
<stdin>:1: FutureWarning: DataFrame.interpolate with object dtype is deprecated and will raise in a future version. Call obj.infer_objects(copy=False) before interpolating instead.
   A  B
0  a  1
1  b  2
2  c  3
Bug: InferFiltersFromConstraint rule is creating a redundant filter
Status: Reopened | Priority: Minor | Resolution: 
Created: 11/Jan/24 05:13
Affects Version/s: 3.5.0

while bring my old PR which uses a different approach to  the ConstraintPropagation algorithm ( [SPARK-33152|https://issues.apache.org/jira/browse/SPARK-33152]) in synch with current master, I noticed a test failure in my branch for SPARK-33152:
The test which is failing is
InferFiltersFromConstraintSuite:
{code}
  test("SPARK-43095: Avoid Once strategy's idempotence is broken for batch: Infer Filters") {
    val x = testRelation.as("x")
    val y = testRelation.as("y")
    val z = testRelation.as("z")

    // Removes EqualNullSafe when constructing candidate constraints
    comparePlans(
      InferFiltersFromConstraints(x.select($"x.a", $"x.a".as("xa"))
        .where($"xa" <=> $"x.a" && $"xa" === $"x.a").analyze),
      x.select($"x.a", $"x.a".as("xa"))
        .where($"xa".isNotNull && $"x.a".isNotNull && $"xa" <=> $"x.a" && $"xa" === $"x.a").analyze)

    // Once strategy's idempotence is not broken
    val originalQuery =
      x.join(y, condition = Some($"x.a" === $"y.a"))
        .select($"x.a", $"x.a".as("xa")).as("xy")
        .join(z, condition = Some($"xy.a" === $"z.a")).analyze

    val correctAnswer =
      x.where($"a".isNotNull).join(y.where($"a".isNotNull), condition = Some($"x.a" === $"y.a"))
        .select($"x.a", $"x.a".as("xa")).as("xy")
        .join(z.where($"a".isNotNull), condition = Some($"xy.a" === $"z.a")).analyze

    val optimizedQuery = InferFiltersFromConstraints(originalQuery)
    comparePlans(optimizedQuery, correctAnswer)
    comparePlans(InferFiltersFromConstraints(optimizedQuery), correctAnswer)
  }
{code}

In the above test, I believe the below assertion is not proper.
There is a redundant filter which is getting created.
Out of these two isNotNull constraints,  only one should be created.

$"xa".isNotNull && $"x.a".isNotNull 
 Because "xa" is an alias of x."a" , so only one isNullConstraint is needed.

  // Removes EqualNullSafe when constructing candidate constraints
    comparePlans(
      InferFiltersFromConstraints(x.select($"x.a", $"x.a".as("xa"))
        .where($"xa" <=> $"x.a" && $"xa" === $"x.a").analyze),
      x.select($"x.a", $"x.a".as("xa"))
        .where($"xa".isNotNull && $"x.a".isNotNull && $"xa" <=> $"x.a" && $"xa" === $"x.a").analyze) 

This is not a big issue, but it highlights the need to take a relook at the code of ConstraintPropagation and related code.

I am filing this jira so that constraint code can be tightened/made more robust.
Bug: `Cast` displayed different results between Regular Spark and Spark Connect
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jan/24 01:30
Affects Version/s: 4.0.0

The following doctest will throw an error in the tests of the pyspark-connect module
{code:java}
Example 5: Decrypt data with key.

>>> import pyspark.sql.functions as sf
>>> df = spark.createDataFrame([(
...     "83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94",
...     "0000111122223333",)],
...     ["input", "key"]
... )
>>> df.select(sf.try_aes_decrypt(
...     sf.unhex(df.input), df.key
... ).cast("STRING")).show(truncate=False) # doctest: +SKIP
+------------------------------------------------------------------+
|CAST(try_aes_decrypt(unhex(input), key, GCM, DEFAULT, ) AS STRING)|
+------------------------------------------------------------------+
|Spark                                                             |
+------------------------------------------------------------------+ {code}
{code:java}
df.select(sf.try_aes_decrypt(
4170        sf.unhex(df.input), df.key
4171    ).cast("STRING")).show(truncate=False)
4172Expected:
4173    +------------------------------------------------------------------+
4174    |CAST(try_aes_decrypt(unhex(input), key, GCM, DEFAULT, ) AS STRING)|
4175    +------------------------------------------------------------------+
4176    |Spark                                                             |
4177    +------------------------------------------------------------------+
4178Got:
4179    +--------------------------------------------------+
4180    |try_aes_decrypt(unhex(input), key, GCM, DEFAULT, )|
4181    +--------------------------------------------------+
4182    |Spark                                             |
4183    +--------------------------------------------------+{code}
!screenshot-1.png|width=671,height=222!
Bug: Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/24 09:19
Affects Version/s: 3.5.0

When we added onQueryIdle to StreamingQueryListener, we missed to add backward compatibility test(s) into StreamingQueryListenerSuite.

We added the compatibility test in pyspark when we fixed an issue on pyspark specific via https://issues.apache.org/jira/browse/SPARK-45631, but we didn't add the symmetric test in Spark Connect (both Scala and PySpark). We should add the test.
Bug: Eliminate JDK Requirement in PySpark Installation
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Jan/24 22:54
Affects Version/s: 3.5.0

PySpark requires users to have the correct JDK version (JDK 8+ for Spark<4; JDK 17+ for Spark>=4) installed locally.

We can make the Spark installation script install the JDK, so users don’t need to do this step manually.
h1. Details
 # When the entry point for a Spark class is invoked, the spark-class script checks if Java is installed in the user environment.

 # If Java is not installed, the user is prompted to select whether they want to install JDK 17.

 # If the user selects yes, JDK 17 is installed (using the [install-jdk library|https://pypi.org/project/install-jdk/]) and JAVA_HOME variable and RUNNER are set appropriately. The Spark build will now work!

 # If the user selects no, we provide them a brief description of how to install JDK manually.