av9ash/GitBugs huggingface dataset:
Bug: Return null when overflowing during casting from timestamp to integers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Nov/23 05:52
Affects Version/s: 3.3.3

Spark cast works in two modes: ansi and non-ansi. When overflowing during casting, the common behavior under non-ansi mode is to return null. However, casting from Timestamp to Int/Short/Byte returns a wrapping value now. The behavior to silently overflow doesn't make sense.
Bug: Implement lint check for PySpark custom errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 03:01
Affects Version/s: 4.0.0

Currently, in the PySpark codebase, there is an inconsistency in the usage of exceptions. In some instances, PySpark-specific exceptions are utilized, while in others, generic Python built-in exceptions are used. This inconsistency can lead to confusion and difficulty in maintaining and debugging the code. See [https://github.com/apache/spark/pull/44024] related work to fix such a case.

The goal of this ticket is to establish a standardized practice for error handling in PySpark by mandating the use of PySpark-specific exceptions where applicable. This will ensure that all exceptions thrown within PySpark adhere to a consistent format and standard, making them more informative and easier to handle.
Bug: Skipping trimAll call in stringToDate functions to avoid needless string copy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Nov/23 16:04
Affects Version/s: 3.0.0

In StringToDate function call we currently first call trimAll to remove any whitespace and isocontrol characters. Trimming results in copying the input string which is not really needed given that we can do all the parsing in place by just skipping the whitespace/isocontrol chars.

Given that we have customers complaining about speed of stringtodate function, especially when input string is long/potentially malformed proposal is to skip trimAll call and do parsing in place.
Bug: Overflow in Parquet row group filter creation causes incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/23 14:43
Affects Version/s: 3.5.0

While the parquet readers don't support reading parquet values into larger Spark types, it's possible to trigger an overflow when creating a Parquet row group filter that will then incorrectly skip row groups and bypass the exception in the reader,

Repro:
{code:java}
Seq(0).toDF("a").write.parquet(path)
spark.read.schema("a LONG").parquet(path).where(s"a < ${Long.MaxValue}").collect(){code}
This succeeds and returns no results. This should either fail if the Parquet reader doesn't support the upcast from int to long or produce result `[0]` if it does.
Bug: Shuffle data lost on decommissioned executor caused by race condition between lastTaskRunningTime and lastShuffleMigrationTime
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/23 03:09
Affects Version/s: 3.4.1

We recently identified a very tricky race condition in decommissioned node, which could lead to shuffle data lost even data migration is enabled:

* At 04:30:51, RDD block refresh happened, and found no pending works
* Shortly after that (a few milliseconds), the shutdownThread in CoarseGrainedExecutorBackend found 1 running task, so lastTaskRunningTime updated to the current system nano time
* Shortly after that, Shuffle block refresh happened, and found no pending works
* Shortly after that, a task finished on the decommissioned executor, and generated new shuffle blocks
* One second later, the shutdownThread in CoarseGrainedExecutorBackend found no running task, lastTaskRunningTime would not be updated, and the executor didn’t exit because min(lastRDDMigrationTime, lastShuffleMigrationTime) <  lastTaskRunningTime
* After 30 seconds, at 04:31:21, RDD block refresh happened, and found no pending works, lastRDDMigrationTime updated to the current system nano time
* At this exact moment, all known blocks are migrated, and min(lastRDDMigrationTime, lastShuffleMigrationTime) > lastTaskRunningTime
* shutdownThread is triggered, and asked to stop the executor
* Shuffle block refresh thread was still sleeping, and got interrupted by the stop command, so it didn’t have the chance to discover the shuffle blocks generated by the previously finished task
* Eventually, the executor exited, and the output of the task was lost, Spark need to recompute that partition

The root cause for the race condition is that the Shuffle block refresh happened between lastTaskRunningTime was updated and task finished, in that case the shutdownThread could request to stop the executor before the BlockManagerDecommissioner discover the new shuffle blocks generated by the latest finished task.
Bug: STORE_ASSIGNMENT_POLICY should be reset in HiveCompatibilitySuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/23 15:02
Affects Version/s: 4.0.0

STORE_ASSIGNMENT_POLICY was not reset in HiveCompatibilitySuite, causing subsequent test cases to fail.

Details: https://github.com/wForget/spark/actions/runs/6902668865/job/18779862759
Bug: Restrict charsets in encode()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/23 05:36
Affects Version/s: 4.0.0

Currently the list of supported charsets in encode() is not stable and fully depends on the used JDK version. So, sometimes user code might not work because a devop changed Java version in Spark cluster. The ticket aims to restrict the list of supported charsets by:

{code}
'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'
{code}
Bug: Split scheduled Python build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/23 01:35
Affects Version/s: 4.0.0

Python 3.12 build fails as below:

{code}
/__w/spark/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas Series is expected to be small.
  warnings.warn(message, PandasAPIOnSparkAdviceWarning)
/__w/spark/spark/python/pyspark/testing/pandasutils.py:401: FutureWarning: `assertPandasOnSparkEqual` will be removed in Spark 4.0.0. Use `ps.testing.assert_frame_equal`, `ps.testing.assert_series_equal` and `ps.testing.assert_index_equal` instead.
  warnings.warn(
/__w/spark/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: `to_pandas` loads all data into the driver's memory. It should only be used if the resulting pandas DataFrame is expected to be small.
  warnings.warn(message, PandasAPIOnSparkAdviceWarning)
ok (15.809s)
  test_groupby_rolling_sum (pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby_rolling.OpsOnDiffFramesGroupByRollingParityTests.test_groupby_rolling_sum) ... ERROR StatusCo
Had test failures in pyspark.pandas.tests.connect.test_parity_ops_on_diff_frames_groupby_rolling with python3.12; see logs.
Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-pandas-connect-part2 --parallelism=1 --python-executables=pypy3,python3.10,python3.11,python3.12 ; received return code 255
Error: Process completed with exit code 19.
{code}

https://github.com/apache/spark/actions/runs/7034467411/job/19154767856

I suspect this by OOM. We could split the build.
Bug: Escape the single quote, _ and % for DS V2 pushdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/23 08:05
Affects Version/s: 3.4.0

Spark supports push down startsWith, endWith and contains to JDBC database with DS V2 pushdown.
But the V2ExpressionSQLBuilder didn't escape the single quote, _ and %, it can cause unexpected result.
Bug: Reduce stack depth by replace Option.isDefined with Option.isEmpty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/23 10:04
Affects Version/s: 4.0.0

There are a lot of Option.isDefined used with !, this behavior increase the stack length.
The reason is isDefined calls isEmpty.
final def isDefined: Boolean = !isEmpty

We also get the compile waring Replace with . isEmpty
Bug: Various Pandas functions fail in interpreted mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/23 21:58
Affects Version/s: 3.4.1

Various Pandas functions ({{kurt}}, {{var}}, {{skew}}, {{cov}}, and {{stddev}}) fail with an unboxing-related exception when run in interpreted mode.

Here are some reproduction cases for pyspark interactive mode:
{noformat}
spark.sql("set spark.sql.codegen.wholeStage=false")
spark.sql("set spark.sql.codegen.factoryMode=NO_CODEGEN")

import numpy as np
import pandas as pd

import pyspark.pandas as ps

pser = pd.Series([1, 2, 3, 7, 9, 8], index=np.random.rand(6), name="a")
psser = ps.from_pandas(pser)

# each of the following actions gets an unboxing error
psser.kurt()
psser.var()
psser.skew()

# set up for covariance test
pdf = pd.DataFrame([(1, 2), (0, 3), (2, 0), (1, 1)], columns=["a", "b"])
psdf = ps.from_pandas(pdf)

# this gets an unboxing error
psdf.cov()

# set up for stddev resr
from pyspark.pandas.spark import functions as SF
from pyspark.sql.functions import col
from pyspark.sql import Row
df = spark.createDataFrame([Row(a=1), Row(a=2), Row(a=3), Row(a=7), Row(a=9), Row(a=8)])

# this gets an unboxing error
df.select(SF.stddev(col("a"), 1)).collect()
{noformat}
Exception from the first case ({{psser.kurt()}}) is
{noformat}
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
	at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:112)
	at org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.compare(PhysicalDataType.scala:184)
	at scala.math.Ordering.lt(Ordering.scala:98)
	at scala.math.Ordering.lt$(Ordering.scala:98)
	at org.apache.spark.sql.catalyst.types.PhysicalDoubleType$$anonfun$2.lt(PhysicalDataType.scala:184)
	at org.apache.spark.sql.catalyst.expressions.LessThan.nullSafeEval(predicates.scala:1196)
{noformat}
Bug: Fix the CSS of Spark doc's generated tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/23 21:10
Affects Version/s: 3.5.1

After [https://github.com/apache/spark/pull/40269], there is no boarder in the generated tables of Spark doc.  We should fix it.

!image-2023-11-30-13-11-01-796.png!
Bug: ANSI Double quoted identifiers do not work in Python threads
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/23 22:30
Affects Version/s: 3.4.0

Enabling and using `spark.sql.ansi.doubleQuotedIdentifiers` does not work correctly in Python threads

The following example shows how applying a filter, "\"status\" = 'Unchanged'", leads to empty results when run in a thread. I believe this is because the "status" field is interpreted as a literal in the thread, but as an attribute outside of it.
{code:python}
from concurrent import futures
from pyspark import sql

spark = (
  sql.SparkSession.builder.master("local[*]")
  .config("spark.sql.ansi.enabled", "true")
  .config("spark.sql.ansi.doubleQuotedIdentifiers", "true")
  .getOrCreate()
)

def demonstrate_issue(spark):
  # Path to JSON file with contents:
  # [{"status": "Unchanged"}, {"status": "Changed"}]
  df = spark.read.json("data/example.json")
  df.filter("\"status\" = 'Unchanged'").show()

# Shows 1 record, expected
demonstrate_issue(spark)

with futures.ThreadPoolExecutor(1) as executor:
  # Shows 0 records, unexpected
  executor.submit(demonstrate_issue, spark)
 {code}
 

Additional testing notes:
 * When parsing the expression with `sql.functions.expr` in Java via Py4J, the "status" field is interpreted as a literal value from the thread, not an attribute
 * Using double quotes with `spark.sql` does work in the thread
 * Using a dataframe created in memory does work in the thread
 * Tested in versions 3.4.0, 3.4.1, & 3.5.0 on Windows and Mac

 

The original PR that added this option is here: [https://github.com/apache/spark/pull/38022]
Bug: [KUBERNETES] Respect the existing kubernetes container SPARK_LOCAL_DIRS env
Status: Open | Priority: Major | Resolution: 
Created: 24/Nov/23 13:48
Affects Version/s: 3.5.0

Respect the user defined SPARK_LOCAL_DIRS container env when setup local dirs.

 

For example, we use hostPath for spark local dir.

But we do not mount the sub disks directly to the pod, we mount a root path for spark driver/executor pod.

 

For example, the root path is `/hadoop`.

 

And there are sub disks under that, likes `hadoop/1, /hadoop/2, /hadoop/3, /hadoop4`.

 

And we want to define the SPARK_LOCAL_DIRS in the driver/executor pod env.

 

But now, the user specified SPARK_LOCAL_DIRS does not work.
Bug: Fail INSERT INTO ... REPLACE statement if the condition contains subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/23 21:08
Affects Version/s: 4.0.0

For the following query:
{code:java}
INSERT INTO tbl REPLACE WHERE id = (select c2 from values(1) as t(c2)) SELECT * FROM source{code}
There will be an analysis error:
{code:java}
[UNRESOLVED_COLUMN.WITHOUT_SUGGESTION] A column, variable, or function parameter with name `c2` cannot be resolved.  SQLSTATE: 42703; line 1 pos 51;
'OverwriteByExpression RelationV2[id#27L, data#28] testcat.tbl testcat.tbl, (id#27L = scalar-subquery#26 []), false {code}
The error message is confusing. The actually reason is the OverwriteByExpression plan doesn't support subqueries. While supporting the feature is non-trivial, we should improve the error message.
Bug: Union of two kryo-encoded Dataset[Object] produces rows of bytes
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/23 00:00
Affects Version/s: 3.5.0

[https://github.com/apache/spark/blob/9987cab84bb05a61a5c0b43f94a733561a2e074a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L4459] treats Dataset[X] as Dataset[Row] if X can take Row. Unfortunately Object can take row, so a try to union two Dataset[Object] produces pretty strange result.

The workaround is to wrap Object into something that can't take Row (e.g. Array)

Reproduction:
{code:java}
val sql = new org.apache.spark.sql.SQLContext(sc)
implicit val encoder = org.apache.spark.sql.Encoders.kryo[Object]
val ds = sql.createDataset(Seq("2".asInstanceOf[Object]))
val unionDs = ds.union(ds)
unionDs.takeAsList(10) {code}
Produces in notebook
{code:java}
sql: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@76e23eff
encoder: org.apache.spark.sql.Encoder[Object] = class[value[0]: binary]
ds: org.apache.spark.sql.Dataset[Object] = [value: binary]
unionDs: org.apache.spark.sql.Dataset[Object] = [value: binary]
res20: java.util.List[Object] = [[[B@387e604f], [[B@601d8331]] {code}
Workaround:
{code:java}
implicit val arrayEncoder = org.apache.spark.sql.Encoders.kryo[Array[Object]]
val workaround = ds.map(x => Array(x))
val workaroundUnionDs = workaround.union(workaround).map(x => x(0))
workaroundUnionDs.takeAsList(10) {code}
Produces expected result:
{code:java}
arrayEncoder: org.apache.spark.sql.Encoder[Array[Object]] = class[value[0]: binary]
workaround: org.apache.spark.sql.Dataset[Array[Object]] = [value: binary]
workaroundUnionDs: org.apache.spark.sql.Dataset[Object] = [value: binary]
res19: java.util.List[Object] = [2, 2] {code}
Bug: Remove treatEmptyValuesAsNulls and use nullValue option instead in XML
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/23 23:22
Affects Version/s: 4.0.0

Today, we offer two available options to handle null values. To enhance user clarity and simplify usage, we propose consolidating these into a single option. We recommend retaining the {{nullValue}} option due to its broader semantic scope.
Bug: can't return prediction that has different length than ml input
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/23 19:15
Affects Version/s: 3.4.1

I'm using

 
from pyspark.ml.functions import predict_batch_udf
to construct a pandas udf that runs a computer vision model to predict classification labels for images. The model takes a 4D array as input and returns a 4D array as output (Batch, Channels, Height, Width)
 
However I'd like to run some additional processing in the pandas_udf to convert the 4D output array (floats) to text labels since this is a more useful output and we are trying to register pandas_udfs ahead of time for spark.sql users.
 
 
When I set the return type to a StringType though I get an error
 
```
23/11/29 02:43:04 WARN TaskSetManager: Lost task 0.0 in stage 8.0 (TID 16) (172.18.0.2 executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/spark/python/pyspark/ml/functions.py", line 809, in predict yield _validate_and_transform_prediction_result( File "/opt/spark/python/lib/pyspark.zip/pyspark/ml/functions.py", line 331, in _validate_and_transform_prediction_result raise ValueError("Prediction results must have same length as input data.") ValueError: Prediction results must have same length as input data.
```
 
This seems like an unnecessary limitation, since it is common for ML models, especially in computer vision, to take input shapes different from output shapes.
 
I think the issue is here: [https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/functions.html]
 
Can this check that enforces same shape be removed?
 
 
to illustrate the problem, here are my StructTypes. The Raw one works but the Str one does not
 
```
from collections import namedtuple
from pyspark.sql.types import ArrayType, IntegerType, StringType,StructType, StructField, FloatType

Task = namedtuple('TaskSchema', ['inference_input', 'inference_result'])

SingleLabelClassificationRaw = Task(
inference_input=StructType([
StructField("array", ArrayType(IntegerType()), nullable=False),
StructField("shape", ArrayType(IntegerType()), nullable=False)
]),
inference_result=ArrayType(FloatType())
)

SingleLabelClassificationStr = Task(
inference_input=StructType([
StructField("array", ArrayType(IntegerType()), nullable=False),
StructField("shape", ArrayType(IntegerType()), nullable=False)
]),
inference_result=StringType()
)
```
Bug: raise_error should only take the error class parameter for internal usage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/23 13:16
Affects Version/s: 4.0.0

In [https://github.com/apache/spark/pull/42985] , we extended the `raise_error` function to take an extra error-class parameter. However, this is too powerful as users may use it to throw special internal errors that we don't expect them to throw.

It's useful to have this `raise_error` extension for internal usage. We should create an `ExpressionBuilder` for `raise_error` function, so that we only allow end users to pass the error message, not error class.
Bug: Fix QueryExecutionErrorsSuite with Java 21
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/23 13:05
Affects Version/s: 4.0.0

[https://github.com/apache/spark/actions/runs/7014008773/job/19081075487]
{code:java}
[info] - FAILED_EXECUTE_UDF: execute user defined function with registered UDF *** FAILED *** (42 milliseconds)
15247[info]   java.lang.IllegalArgumentException: For parameter 'reason' value 'java.lang.StringIndexOutOfBoundsException: Range [5, 6) out of bounds for length 5' does not match: java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5
15248[info]   at org.apache.spark.SparkFunSuite.$anonfun$checkError$2(SparkFunSuite.scala:357)
15249[info]   at org.apache.spark.SparkFunSuite.$anonfun$checkError$2$adapted(SparkFunSuite.scala:352)
15250[info]   at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
15251[info]   at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
15252[info]   at scala.collection.AbstractIterable.foreach(Iterable.scala:933) {code}
Bug: Enhancing readability of PySpark API reference by hiding verbose typehints.
Status: Open | Priority: Major | Resolution: 
Created: 27/Nov/23 06:31
Affects Version/s: 4.0.0

Currently, the PySpark API documentation displays all type hints in the signatures, which can make the documentation appear cluttered and less readable. By setting `autodoc_typehints` to 'none', we can achieve a cleaner and more concise presentation of our API, similar to how the Pandas documentation handles type hints. This approach has been effective in Pandas, making the documentation more approachable and easier to understand, especially for newcomers.
Bug: [CONNECT][SCALA] Insufficient details in error when a UDF fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/23 16:46
Affects Version/s: 3.5.0

Currently, when a UDF fails the connect client does not receive the actual error that caused the failure. 

As an example, the error message looks like -
{code:java}
Exception in thread "main" org.apache.spark.SparkException: grpc_shaded.io.grpc.StatusRuntimeException: INTERNAL: Job aborted due to stage failure: Task 2 in stage 0.0 failed 4 times, most recent failure: Lost task 2.3 in stage 0.0 (TID 10) (10.68.141.158 executor 0): org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (` (Main$$$Lambda$4770/1714264622)`: (int) => int). SQLSTATE: 39000 {code}
In this case, the actual error was a {{{}java.lang.NoClassDefFoundError{}}}.
Bug: df.emptyDataFrame shows 1 if we repartition(1) in Spark 3.3.x and above
Status: Open | Priority: Major | Resolution: 
Created: 26/Nov/23 06:23
Affects Version/s: 3.3.3

{color:#FF0000}Version: 3.3.3{color}

 

{color:#FF0000}scala> val df = spark.emptyDataFrame{color}
{color:#FF0000}df: org.apache.spark.sql.DataFrame = []{color}

{color:#FF0000}scala> df.rdd.getNumPartitions{color}
{color:#FF0000}res0: Int = 0{color}

{color:#FF0000}scala> df.repartition(1).rdd.getNumPartitions{color}
{color:#FF0000}res1: Int = 1{color}

{color:#FF0000}scala> df.repartition(1).rdd.isEmpty(){color}
{color:#FF0000}[Stage 1:>                                                          (0 + 1) /                                                                             res2: Boolean = true{color}

Version: 3.2.4

scala> val df = spark.emptyDataFrame
df: org.apache.spark.sql.DataFrame = []

scala> df.rdd.getNumPartitions
res0: Int = 0

scala> df.repartition(1).rdd.getNumPartitions
res1: Int = 0

scala> df.repartition(1).rdd.isEmpty()
res2: Boolean = true

 

{color:#FF0000}Version: 3.5.0{color}

{color:#FF0000}scala> val df = spark.emptyDataFrame{color}
{color:#FF0000}df: org.apache.spark.sql.DataFrame = []{color}

{color:#FF0000}scala> df.rdd.getNumPartitions{color}
{color:#FF0000}res0: Int = 0{color}

{color:#FF0000}scala> df.repartition(1).rdd.getNumPartitions{color}
{color:#FF0000}res1: Int = 1{color}

{color:#FF0000}scala> df.repartition(1).rdd.isEmpty(){color}
{color:#FF0000}[Stage 1:>                                                          (0 + 1) /                                                                             res2: Boolean = true{color}

 

When we do repartition of 1 on an empty dataframe, the resultant partition is 1 in version 3.3.x and 3.5.x whereas when I do the same in version 3.2.x, the resultant partition is 0. May i know why this behaviour is changed from 3.2.x to higher versions. 

 

The reason for raising this as a bug is I have a scenario where my final dataframe returns 0 records in EKS(local spark) with single node(driver and executor on the sam node) but it returns 1 in EMR both uses a same spark version 3.3.3. I'm not sure why this behaves different in both the environments. As a interim solution, I had to repartition a empty dataframe if my final dataframe is empty which returns 1 for 3.3.3. Would like to know if this really a bug or this behaviour exists in the future versions and cannot be changed?

 

Because, If we go for a spark upgrade and this behaviour is changed, we will face the issue again. 

Please confirm on this.