av9ash/GitBugs huggingface dataset:
Bug: SparkR cannot collect dataframe with NA in a date column along with another timestamp column
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 12/Jan/23 03:56
Affects Version/s: 3.3.0

This issue seems to be related with https://issues.apache.org/jira/browse/SPARK-17811, which was resolved by [https://github.com/apache/spark/pull/15421] .

If there exists a column of data type `date` which is completely NA, and another column of data type `timestamp`, then SparkR cannot collect that Spark dataframe into R dataframe.

The reproducible code snippet is below. 
{code:java}
df <- data.frame(x = as.Date(NA), y = as.POSIXct("2022-01-01"))
SparkR::collect(SparkR::createDataFrame(df))

#> Error in handleErrors(returnStatus, conn): org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 1 times, most recent failure: Lost task 0.0 in stage 25.0 (TID 25) (ip-10-172-210-194.us-west-2.compute.internal executor driver): java.lang.IllegalArgumentException: Invalid type N
#> at org.apache.spark.api.r.SerDe$.readTypedObject(SerDe.scala:94)
#> at org.apache.spark.api.r.SerDe$.readObject(SerDe.scala:68)
#> at #> org.apache.spark.sql.api.r.SQLUtils$.$anonfun$bytesToRow$1(SQLUtils.scala:129)
#> at org.apache.spark.sql.api.r.SQLUtils$.$anonfun$bytesToRow$1$adapted(SQLUtils.scala:128)
#> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
#> at scala.collection.immutable.Range.foreach(Range.scala:158)
#> ...{code}
This issue does not appear If the column of `date` data type is {_}not missing{_}. Or if there _does not exist_ any other column with data type as `timestamp`.
{code:java}
df <- data.frame(x = as.Date("2022-01-01"), y = as.POSIXct("2022-01-01"))
SparkR::collect(SparkR::createDataFrame(df))

#>            x             y                                                         
#> 1     2022-01-01    2022-01-01{code}
or
{code:java}
df <- data.frame(x = as.Date(NA), y = as.character("2022-01-01"))
SparkR::collect(SparkR::createDataFrame(df))

#>            x             y
#> 1        <NA>       2022-01-01{code}
Bug: Schema pruning fails on non-foldable array index or map key
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/23 23:08
Affects Version/s: 3.2.3

Schema pruning tries to extract selected fields from struct extractors. It looks through GetArrayItem/GetMapItem, but when doing so, it ignores the index/key, which may itself be a struct field. If it is a struct field that is not otherwise selected, and some other field of the same attribute is selected, then pruning will drop the field, resulting in an optimizer error.
Bug: Introduce SparkPath to address paths and URIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/23 21:56
Affects Version/s: 3.4.0

Today, Spark represents file paths in various ways. Sometimes they are Hadoop `Path`s, sometimes they are `Path.toString`s, and sometimes they are `Path.toUri.toString`s.

This discrepancy means that Spark does not always work when user provided strings have special characters. Sometimes Spark will try to create a URI with an unescaped string; sometimes Spark will double-escape a path and try to access the wrong file.

 

This issue proposes a new `SparkPath` class meant to provide type safety when Spark is dealing with paths.
Bug: The DATATYPE_MISMATCH error class contains inappropriate and duplicating subclasses
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/23 21:56
Affects Version/s: 3.4.0

subclass WRONG_NUM_ARGS (with suggestions) semantically does not belong into DATATYPE_MISMATCH and there is an error class with that same name.
We should rea the subclasses for this errorclass, which seems to have become a bit of a dumping ground...
Bug: `createDataFrame` doesn't handle None/NaN properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 03:47
Affects Version/s: 3.4.0

{code:python}
        data = [Row(id=1, value=float("NaN")), Row(id=2, value=42.0), Row(id=3, value=None)]

        # +---+-----+
        # | id|value|
        # +---+-----+
        # |  1|  NaN|
        # |  2| 42.0|
        # |  3| null|
        # +---+-----+

        cdf = self.connect.createDataFrame(data)
        sdf = self.spark.createDataFrame(data)

        print()
        print()
        print(cdf._show_string(100, 100, False))
        print()
        print(cdf.schema)
        print()
        print(sdf._jdf.showString(100, 100, False))
        print()
        print(sdf.schema)

        self.compare_by_show(cdf, sdf)
{code}



{code:java}
+---+-----+
| id|value|
+---+-----+
|  1| null|
|  2| 42.0|
|  3| null|
+---+-----+


StructType([StructField('id', LongType(), True), StructField('value', DoubleType(), True)])

+---+-----+
| id|value|
+---+-----+
|  1|  NaN|
|  2| 42.0|
|  3| null|
+---+-----+


StructType([StructField('id', LongType(), True), StructField('value', DoubleType(), True)])

{code}



this issue is due to that `createDataFrame` can't handle None/NaN properly:

1, in the conversion from local data to pd.DataFrame, it automatically converts both None and NaN to NaN
2, then in the conversion from pd.DataFrame to pa.Table, it always converts NaN to null
Bug: Typo in StreamingQuery.scala
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/23 05:40
Affects Version/s: 3.2.3

{color:#172b4d}/**{color}
{color:#172b4d} * Returns the unique id of this run of the query. That is, every start/restart of a query *will*{color}
{color:#172b4d} ** *generated* a unique runId. Therefore, every time a query is restarted from{color}
{color:#172b4d} * checkpoint, it will have the same [[id]] but different [[runId]]s.{color}
{color:#172b4d} */{color}
{color:#172b4d}def runId: UUID{color}
Bug: Add `SparkConnectIllegalArgumentException` to handle Spark Connect error precisely.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jan/23 12:02
Affects Version/s: 3.5.0

Add `SparkConnectIllegalArgumentException` to handle Spark Connect error precisely by catching `IllegalArgumentException` before unexpectedly raise `SparkConnectGrpcException`.
Bug: Parity in Error types between pyspark and connect functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:26
Affects Version/s: 3.4.0

PySpark throws Py4JJavaError where as connect throws SparkConnectException
{code:java}
from pyspark.sql.functions import assert_true

df = self.spark.range(3)

self.assertEqual(
    df.select(assert_true(df.id < 3)).toDF("val").collect(),
    [Row(val=None), Row(val=None), Row(val=None)],
)

with self.assertRaises(Py4JJavaError) as cm:
    df.select(assert_true(df.id < 2, "too big")).toDF("val").collect(){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 950, in test_assert_true
    df.select(assert_true(df.id < 2, "too big")).toDF("val").collect()
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1076, in collect
    table = self._session.client.to_table(query)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 414, in to_table
    table, _ = self._execute_and_fetch(req)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 586, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 629, in _handle_error
    raise SparkConnectException(status.message, info.reason) from None
pyspark.sql.connect.client.SparkConnectException: (java.lang.RuntimeException) too big {code}
Bug: Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jan/23 19:59
Affects Version/s: 3.1.2

We run into an error after an upgrade from Spark 3.1 to Spark 3.2 cased by SPARK-34674 which closed the SparkContext right after the application start. This application was a spark job server built on top of springboot so all the job submits were outside of the main method.
Bug: Failed to test ClientE2ETestSuite with maven
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/23 12:54
Affects Version/s: 3.5.0

run the following commands
{code:java}
build/mvn clean
build/mvn clean install -pl connector/connect/server -am -DskipTests
build/mvn clean install -pl connector/connect/client/jvm -am -DskipTests
build/mvn test -pl connector/connect/client/jvm {code}
{code:java}
ClientE2ETestSuite:
Using jar: /${basedir}/spark-source/connector/connect/server/target/spark-connect_2.12-3.5.0-SNAPSHOT.jar
Error: Failed to load class org.apache.spark.repl.Main. {code}
Bug: Omit "Task Executor Metrics" field in JsonProtocol output if values are all zero
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jan/23 00:40
Affects Version/s: 3.0.0

SPARK-26329 added "Task Executor Metrics" to JsonProtocol SparkListenerTaskEnd JSON. With the default `spark.executor.metrics.pollingInterval = 0` configuration these metric values are only updated when heartbeats occur. If a task launches and finishes between executor heartbeats then all of these metric values will be zero. For jobs with large numbers of short tasks, this contributes to significant event log bloat.

JsonProtocol already knows how to handle the absence of the "Task Executor Metrics" field, so I think it's safe for us to omit this field when all values are zero.
Bug: Fix DataFrame.sample parameters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:42
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 401, in pyspark.sql.connect.dataframe.DataFrame.sample
Failed example:
    df.sample(0.5, 3).count()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.sample[1]>", line 1, in <module>
        df.sample(0.5, 3).count()
    TypeError: DataFrame.sample() takes 2 positional arguments but 3 were given
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 411, in pyspark.sql.connect.dataframe.DataFrame.sample
Failed example:
    df.sample(False, fraction=1.0).count()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.sample[6]>", line 1, in <module>
        df.sample(False, fraction=1.0).count()
    TypeError: DataFrame.sample() got multiple values for argument 'fraction'{code}
Bug: Implement DataFrameReader.text
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:45
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 276, in pyspark.sql.connect.functions.input_file_name
Failed example:
    df = spark.read.text(path)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.input_file_name[2]>", line 1, in <module>
        df = spark.read.text(path)
    AttributeError: 'DataFrameReader' object has no attribute 'text'{code}
Bug: IPv6 address mapping can fail with sparse addresses
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/23 20:11
Affects Version/s: 3.4.0

e.g. the `addressToExecutorId` ends up storing `2602:fcb1::1337:12` but when it comes time to handle the `onDisconnect` all of the implicit zeros are expanded out.
Bug: Force SBT protobuf version to match Maven on branch 3.2 and 3.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/23 14:41
Affects Version/s: 3.2.3

Update SparkBuild.scala to force SBT use of protobuf-java to match the Maven version.  The Maven dependencyManagement section forces protobuf-java to use 2.5.0, but SBT is using 3.14.0.

Snippet from Maven dependency tree

 
{noformat}
[INFO] +- com.google.crypto.tink:tink:jar:1.6.0:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile    <--- 2.x
[INFO] |  \- com.google.code.gson:gson:jar:2.8.6:compile{noformat}
  Snippet from SBT dependency tree
{noformat}
[info]   +-com.google.crypto.tink:tink:1.6.0
[info]   | +-com.google.code.gson:gson:2.8.6
[info]   | +-com.google.protobuf:protobuf-java:3.14.0               <--- 3.x{noformat}
The fix is updating SparkBuild.scala just like SPARK-11538 did with guava.  In addition we should comment on the need to keep the top-level pom.xml and SparkBuild.scala in sync as was done in SPARK-41247
Bug: Make SparkR able to stop properly when the connection is timed-out
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/23 12:25
Affects Version/s: 3.5.0

{code}
./bin/sparkR --conf spark.r.backendConnectionTimeout=10
{code}

wait 10 secs.

{code}

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.0-SNAPSHOT
      /_/


SparkSession Web UI available at http://192.168.35.219:4040
SparkSession available as 'spark'(master = local[*], app id = local-1674649482367).
> 23/01/25 21:24:53 WARN RBackendHandler: Ignoring read timeout in RBackendHandler
spark.session()
Error in spark.session() : could not find function "spark.session"
> spark.session.stop()
Error in spark.session.stop() :
  could not find function "spark.session.stop"
{code}
Bug: Scalar Inline Python UDF in Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/23 04:01
Affects Version/s: 3.4.0

Support scalar inline user-defined function of Python(a.k.a., unregistered Python UDF) in Spark Connect.

Currently, the user-specified return type must be of `pyspark.sql.types.DataType`.
Bug: Cast boolean to timestamp fails with ClassCastException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/23 00:49
Affects Version/s: 3.3.1

When casting a boolean value to timestamp, the following error is thrown:
{code:java}
[info]   java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
[info]   at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:107)
[info]   at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getWriter$5(InternalRow.scala:178)
[info]   at org.apache.spark.sql.catalyst.InternalRow$.$anonfun$getWriter$5$adapted(InternalRow.scala:178) {code}
Bug: Spark 3.3.0 binary breaking change missing from release notes
Status: Open | Priority: Major | Resolution: 
Created: 18/Jan/23 10:28
Affects Version/s: 3.3.0

SPARK-37929 contains a binary breaking change in the SupportsNamespaces API


See: [https://github.com/apache/spark/pull/35246/files#r792289685]

 

There is no mention in the [release notes|https://spark.apache.org/releases/spark-release-3-3-0.html]
Bug: Introduce sasl retry count in RetryingBlockTransferor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/23 14:00
Affects Version/s: 3.4.0

Previously a boolean variable, saslTimeoutSeen, was used in RetryingBlockTransferor. However, the boolean variable wouldn't cover the following scenario:

1. SaslTimeoutException
2. IOException
3. SaslTimeoutException
4. IOException

Even though IOException at #2 is retried (resulting in increment of retryCount), the retryCount would be cleared at step #4.
Since the intention of saslTimeoutSeen is to undo the increment due to retrying SaslTimeoutException, we should keep a counter for SaslTimeoutException retries and subtract the value of this counter from retryCount.
Bug: Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/23 14:24
Affects Version/s: 3.2.2

Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.

eg:

{{spark-submit --files myfile --master k8s.. --deploy-mode cluster` will upload the files to /tmp/spark-uuid/myfile}}

The issue happens because [Utils.createTempDir()|https://github.com/apache/spark/blob/v3.3.1/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L344] creates a directory with a uuid in the directory name. This issue does not affect the --archives option, because we `unarchive` the archives into the destination directory which is relative to the working dir. This bug affects file access pre & post app creation. For example if we distribute python dependencies with pex, we need to use --files to attach the pex file and change the spark.pyspark.python to point to this file. But the file location can not be determined before submitting the app. On the other hand, after the app is created, referencing the files without using `SparkFiles.get` also does not work
Bug: Fix wrong error message for `MUST_AGGREGATE_CORRELATED_SCALAR_SUBQUERY`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 03:01
Affects Version/s: 3.4.0

The current error message

"Correlated scalar subqueries in the GROUP BY clause must also be in the aggregate expressions<treeNode>"

is incorrect since it's not related with group by clause.

We should fix it.
Bug: Integrate all errors that causes re-create of a view failing into one error class
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/23 08:17
Affects Version/s: 3.4.0

Should integrate errors that causes re-create of view failing into single error class, and make the subclasses for distinct cases.

e.g.
 * CANNOT UPCAST
 * TABLE NOT FOUND/UNRESOLVED
 * UNRESOLVED FUNCTION
 * DATATYPE MISMATCH
Bug: Restore the signature of Utils#setStringField  to `setStringField(input: String, f: String => MessageOrBuilder)`
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 21/Jan/23 09:45
Affects Version/s: 3.4.0

Restore the signature of `f` when `f` is declared as `f: String => MessageOrBuilder` and `setStringField` used by other modules not cause maven build failure
Bug: Add missing options for Protobuf functions.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/23 18:02
Affects Version/s: 3.4.0

 e should be able to pass options for both {{from_protobuf()}} and {{{}to_protobuf(){}}}.
Currently there are some gaps:
 * In Scala {{to_protobuf()}} does not have a way to pass options.
 * In Scala {{from_protobuf()}} that takes Java class name does not allow options.
 * In Python, {{from_protobuf()}} that uses Java class name does not propagate options.
 * In Python {{to_protobuf()}} does not pass options.
Bug: createDataFrame supports column with map type.
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 11/Jan/23 12:59
Affects Version/s: 3.4.0

Currently, Connect API createDataFrame does not support create dataframe with map type.
For example, 

{code:java}
    >>> df = spark.createDataFrame(
    ...     [(1, ["foo", "bar"], {"x": 1.0}), (2, [], {}), (3, None, None)],
    ...     ("id", "an_array", "a_map")
    ... )
{code}

The above code want create a dataframe with column 'a_map' which is map type.
But pyarrow recognize {"x": 1.0} as a struct not map.
pyarrow supports map with format [('x', 1.0)]

Because the data frame's schema is not correct, so the other sequence operator will be impacted.
For example:

{code:java}
    df.select("id", "a_map", posexplode_outer("an_array")).show()
{code}
Expected:

{code:java}
    +---+----------+----+----+
    | id|     a_map| pos| col|
    +---+----------+----+----+
    |  1|{x -> 1.0}|   0| foo|
    |  1|{x -> 1.0}|   1| bar|
    |  2|        {}|null|null|
    |  3|      null|null|null|
    +---+----------+----+----+
{code}

Got:

{code:java}
    +---+------+----+----+
    | id| a_map| pos| col|
    +---+------+----+----+
    |  1| {1.0}|   0| foo|
    |  1| {1.0}|   1| bar|
    |  2|{null}|null|null|
    |  3|  null|null|null|
    +---+------+----+----+
    <BLANKLINE>
{code}
Bug: Fix `count(expr("*"))` function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:29
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 801, in pyspark.sql.connect.functions.count
Failed example:
    df.select(count(expr("*")), count(df.alphabets)).show()
Expected:
    +--------+----------------+
    |count(1)|count(alphabets)|
    +--------+----------------+
    |       4|               3|
    +--------+----------------+
Got:
    +----------------+----------------+
    |count(alphabets)|count(alphabets)|
    +----------------+----------------+
    |               3|               3|
    +----------------+----------------+
    <BLANKLINE> {code}
Bug: Make `count(*)` work correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/23 06:37
Affects Version/s: 3.4.0

cdf.select(CF.count("*"), CF.count(cdf.alphabets)).collect()


{code:java}
pyspark.sql.connect.client.SparkConnectAnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `*` cannot be resolved. Did you mean one of the following? [`alphabets`]
Plan: 'Aggregate [unresolvedalias('count('*), None), count(alphabets#32) AS count(alphabets)#35L]
+- Project [alphabets#30 AS alphabets#32]
   +- LocalRelation [alphabets#30]

{code}
Bug: Address General Fixes
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/23 22:40
Affects Version/s: 3.4.0

We want the distributor to have the ability to run multiple torchrun processes per task if task.gpu.amount > 1.

We want to add a check to see if `import torch` doesn't raise an ImportError since the TorchDistributor requires torch. If it raises an ImportError, we will give the user more details.
Bug: Scala Client Proper logging for the client
Status: Open | Priority: Major | Resolution: 
Created: 20/Jan/23 15:26
Affects Version/s: 3.4.0

Introduce proper logging for the client and change [https://github.com/apache/spark/pull/39541/files/2a589543bdec80f4cf806af0a8566d2de8c04140#r1082062813] to use the client logging.