av9ash/GitBugs huggingface dataset:
Bug: Use AppendData.byName for SQL INSERT INTO by name for DSV2 and block ambiguous queries with static partitions columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/23 23:01
Affects Version/s: 3.4.0

Currently for INSERT INTO by name we reorder the value list and convert it to INSERT INTO by ordinal. Since DSv2 logical nodes have the isByName flag we don't need to do this. The current approach is limiting in that
 # Users must provide the full list of table columns (this limits the functionality for features like generated columns see SPARK-41290)
 # It allows ambiguous queries such as INSERT OVERWRITE t PARTITION (c='1') (c) VALUES ('2') where the user provides both the static partition column 'c' and the column 'c' in the column list. We should check that the static partition column is not in the column list.
Bug: DataFrame.groupBy requires all cols be Column or str
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:30
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 302, in pyspark.sql.connect.dataframe.DataFrame.groupBy
Failed example:
    df.groupBy(["name", df.age]).count().sort("name", "age").show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.groupBy[4]>", line 1, in <module>
        df.groupBy(["name", df.age]).count().sort("name", "age").show()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 251, in groupBy
        raise TypeError(
    TypeError: groupBy requires all cols be Column or str, but got list ['name', Column<'ColumnReference(age)'>]{code}
Bug: Enable test_add_months_function, test_array_repeat, test_dayofweek, test_first_last_ignorenulls, test_function_parity, test_inline, test_window_time, test_reciprocal_trig_functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 04:36
Affects Version/s: 3.4.0
Bug: Implement Dataframe.sort,sortWithinPartitions Ordering
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:37
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 422, in pyspark.sql.connect.dataframe.DataFrame.sort
Failed example:
    df.orderBy(["age", "name"], ascending=[False, False]).show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.sort[8]>", line 1, in <module>
        df.orderBy(["age", "name"], ascending=[False, False]).show()
    TypeError: DataFrame.sort() got an unexpected keyword argument 'ascending'
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 379, in pyspark.sql.connect.dataframe.DataFrame.sortWithinPartitions
Failed example:
    df.sortWithinPartitions("age", ascending=False)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.sortWithinPartitions[1]>", line 1, in <module>
        df.sortWithinPartitions("age", ascending=False)
    TypeError: DataFrame.sortWithinPartitions() got an unexpected keyword argument 'ascending'{code}
Bug: Fix DataFrame.describe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:02
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 898, in pyspark.sql.connect.dataframe.DataFrame.describe
Failed example:
    df.describe(['age']).show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.describe[1]>", line 1, in <module>
        df.describe(['age']).show()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 832, in describe
        raise TypeError(f"'cols' must be list[str], but got {type(s).__name__}")
    TypeError: 'cols' must be list[str], but got list {code}
Bug: DataFrame hint parameter can be str, float or int
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 02:59
Affects Version/s: 3.4.0

{code:java}
df = self.spark.range(10e10).toDF("id")
such_a_nice_list = ["itworks1", "itworks2", "itworks3"]
hinted_df = df.hint("my awesome hint", 1.2345, "what", such_a_nice_list){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 556, in test_extended_hint_types
    hinted_df = df.hint("my awesome hint", 1.2345, "what", such_a_nice_list)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 482, in hint
    raise TypeError(
TypeError: param should be a int or str, but got float 1.2345{code}
Bug: DataFrame.show formatting int as double
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:13
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 650, in pyspark.sql.connect.dataframe.DataFrame.fillna
Failed example:
    df.na.fill(50).show()
Expected:
    +---+------+-----+----+
    |age|height| name|bool|
    +---+------+-----+----+
    | 10|  80.5|Alice|null|
    |  5|  50.0|  Bob|null|
    | 50|  50.0|  Tom|null|
    | 50|  50.0| null|true|
    +---+------+-----+----+
Got:
    +----+------+-----+----+
    | age|height| name|bool|
    +----+------+-----+----+
    |10.0|  80.5|Alice|null|
    | 5.0|  50.0|  Bob|null|
    |50.0|  50.0|  Tom|null|
    |50.0|  50.0| null|true|
    +----+------+-----+----+
    <BLANKLINE>{code}
Bug: Column.isNull returns nan instead of None
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 08:25
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/column.py", line 99, in pyspark.sql.connect.column.Column.isNull
Failed example:
    df.filter(df.height.isNull()).collect()
Expected:
    [Row(name='Alice', height=None)]
Got:
    [Row(name='Alice', height=nan)]
{code}
Bug: --packages may not work on Windows 11
Status: Open | Priority: Major | Resolution: 
Created: 04/Jan/23 17:11
Affects Version/s: 3.2.1

Gastón Ortiz reported an issue when using spark 3.2.1 and hadoop 2.7 in windows 11. See [https://github.com/delta-io/delta/issues/1059]

Looks like executor cannot fetch the jar files. See the critical stack trace below (the full stack trace is in [https://github.com/delta-io/delta/issues/1059] ):
{code:java}
org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:366) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:762) at org.apache.spark.util.Utils$.fetchFile(Utils.scala:549) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13(Executor.scala:962) at org.apache.spark.executor.Executor.$anonfun$updateDependencies$13$adapted(Executor.scala:954) at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985) at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149) at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237) at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44) at scala.collection.mutable.HashMap.foreach(HashMap.scala:149) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$updateDependencies(Executor.scala:954) at org.apache.spark.executor.Executor.<init>(Executor.scala:247) at  {code}
This is not a Delta Lake issue, as this can be reproduced by running `pyspark --packages org.apache.kafka:kafka-clients:2.8.1` as well.

I don't have a Windows 11 environment to debug. Hence I help Gastón Ortiz create this ticket and it would be great if anyone who has a Windows 11 environment can help this.
Bug: Implement creating empty Dataframe
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:34
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 99, in pyspark.sql.connect.dataframe.DataFrame.isEmpty
Failed example:
    df_empty = spark.createDataFrame([], 'a STRING')
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.isEmpty[0]>", line 1, in <module>
        df_empty = spark.createDataFrame([], 'a STRING')
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", line 186, in createDataFrame
        raise ValueError("Input data cannot be empty")
    ValueError: Input data cannot be empty{code}
Bug: Fix `nanvl` function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:51
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 313, in pyspark.sql.connect.functions.nanvl
Failed example:
    df.select(nanvl("a", "b").alias("r1"), nanvl(df.a, df.b).alias("r2")).collect()
Expected:
    [Row(r1=1.0, r2=1.0), Row(r1=2.0, r2=2.0)]
Got:
    [Row(r1=1.0, r2=1.0), Row(r1=nan, r2=nan)]{code}
Bug: Fix `isnan` function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:46
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 288, in pyspark.sql.connect.functions.isnan
Failed example:
    df.select("a", "b", isnan("a").alias("r1"), isnan(df.b).alias("r2")).show()
Expected:
    +---+---+-----+-----+
    |  a|  b|   r1|   r2|
    +---+---+-----+-----+
    |1.0|NaN|false| true|
    |NaN|2.0| true|false|
    +---+---+-----+-----+
Got:
    +----+----+-----+-----+
    |   a|   b|   r1|   r2|
    +----+----+-----+-----+
    | 1.0|null|false|false|
    |null| 2.0|false|false|
    +----+----+-----+-----+
    <BLANKLINE>{code}
Bug: Fix `pmod` function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/23 02:53
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 622, in pyspark.sql.connect.functions.pmod
Failed example:
    df.select(pmod("a", "b")).show()
Expected:
    +----------+
    |pmod(a, b)|
    +----------+
    |       NaN|
    |       NaN|
    |       1.0|
    |       NaN|
    |       1.0|
    |       2.0|
    |      -5.0|
    |       7.0|
    |       1.0|
    +----------+
Got:
    +----------+
    |pmod(a, b)|
    +----------+
    |      null|
    |      null|
    |       1.0|
    |      null|
    |       1.0|
    |       2.0|
    |      -5.0|
    |       7.0|
    |       1.0|
    +----------+
    <BLANKLINE>{code}
Bug: Column.eqNullSafe fails on NaN comparison
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 08:22
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/column.py", line 115, in pyspark.sql.connect.column.Column.eqNullSafe
Failed example:
    df2.select(
        df2['value'].eqNullSafe(None),
        df2['value'].eqNullSafe(float('NaN')),
        df2['value'].eqNullSafe(42.0)
    ).show()
Expected:
    +----------------+---------------+----------------+
    |(value <=> NULL)|(value <=> NaN)|(value <=> 42.0)|
    +----------------+---------------+----------------+
    |           false|           true|           false|
    |           false|          false|            true|
    |            true|          false|           false|
    +----------------+---------------+----------------+
Got:
    +----------------+---------------+----------------+
    |(value <=> NULL)|(value <=> NaN)|(value <=> 42.0)|
    +----------------+---------------+----------------+
    |            true|          false|           false|
    |           false|          false|            true|
    |            true|          false|           false|
    +----------------+---------------+----------------+
{code}
Bug: Implement Dataframe.rdd getNumPartitions
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 02/Jan/23 16:35
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 243, in pyspark.sql.connect.dataframe.DataFrame.coalesce
Failed example:
    df.coalesce(1).rdd.getNumPartitions()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.coalesce[1]>", line 1, in <module>
        df.coalesce(1).rdd.getNumPartitions()
    AttributeError: 'function' object has no attribute 'getNumPartitions'{code}
Bug: Implement Dataframe.readStream
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 02/Jan/23 17:27
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line ?, in pyspark.sql.connect.dataframe.DataFrame.isStreaming
Failed example:
    df = spark.readStream.format("rate").load()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.dataframe.DataFrame.isStreaming[0]>", line 1, in <module>
        df = spark.readStream.format("rate").load()
    AttributeError: 'SparkSession' object has no attribute 'readStream'{code}
Bug: Implement `intX2` function
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Jan/23 02:27
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 2332, in pyspark.sql.connect.functions.call_udf
Failed example:
    df.select(call_udf("intX2", "id")).show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.call_udf[4]>", line 1, in <module>
        df.select(call_udf("intX2", "id")).show()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 534, in show
        print(self._show_string(n, truncate, vertical))
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 423, in _show_string
        ).toPandas()
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1031, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 619, in _handle_error
        raise SparkConnectAnalysisException(
    pyspark.sql.connect.client.SparkConnectAnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `intX2` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].
    Plan: {code}
Bug: SparkSession.createDataFrame does not respect the column names in the dictionary
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 07:17
Affects Version/s: 3.4.0

{code}
Failed example:
    with tempfile.TemporaryDirectory() as d:
        # Write a DataFrame into a JSON file
        spark.createDataFrame(
            [{"age": 100, "name": "Hyukjin Kwon"}]
        ).write.mode("overwrite").format("json").save(d)

        # Read the JSON file as a DataFrame.
        spark.read.format('json').load(d).show()
Expected:
    +---+------------+
    |age|        name|
    +---+------------+
    |100|Hyukjin Kwon|
    +---+------------+
Got:
    +---+------------+
    | _1|          _2|
    +---+------------+
    |100|Hyukjin Kwon|
    +---+------------+
{code}
Bug: UnsafeHashedRelation read method needs to confirm the correctness of the data
Status: Open | Priority: Major | Resolution: 
Created: 02/Jan/23 08:20
Affects Version/s: 2.4.8

Recently, we have encountered the thread-safe issue of[ [SPARK-31511]|https://issues.apache.org/jira/browse/SPARK-31511] in production.  The version 2.4.3 we use has not been fixed yet, which leads to data errors.  I think this is a serious error, the data broadcast by the Driver is inconsistent with the data of the Executor.  The Executor side should confirm the correctness of the data when reading the data.  The numKeys and numValues read from the file header should be consistent with the real data read.  This judgment should be added to prevent wrong data from being calculated.
Bug: Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/Jan/23 04:38
Affects Version/s: 3.0.0

After Jira SPARK-28209 and PR [25007|[https://github.com/apache/spark/pull/25007]], the new shuffle writer api is proposed. All shuffle writers(BypassMergeSortShuffleWriter, SortShuffleWriter, UnsafeShuffleWriter) are based on LocalDiskShuffleMapOutputWriter to write local disk shuffle files. The config spark.shuffle.unsafe.file.output.buffer used in LocalDiskShuffleMapOutputWriter was only used in UnsafeShuffleWriter before. 
 
It's better to rename it and make it more suitable.
Bug: Log when the K8s Exec Pods Allocator Stalls
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Jan/23 18:34
Affects Version/s: 3.4.0

Sometimes if the K8s APIs are being slow the ExecutorPods allocator can stall and it would be good for us to log this (and how long we've stalled for) so folks can tell more clearly why Spark is unable to reach the desired target number of executors.

 

This is _somewhat_ related to SPARK-36664 which logs the time spent waiting for executor allocation but goes a step further for K8s and logs when we've stalled because we have too many pending pods.
Bug: Implement code generation for `to_csv` function (StructsToCsv)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Jan/23 13:30
Affects Version/s: 3.4.0

Implement code generation for `to_csv` function instead of extending it from CodegenFallback trait.
{code:java}
org.apache.spark.sql.catalyst.expressions.StructsToCsv.doGenCode(...){code}
 

This is good to have from performance point of view.
Bug: pyspark.sql.functions should not expose typing.cast
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Jan/23 14:33
Affects Version/s: 3.3.1

In pyspark, the `pyspark.sql.functions` modules imports and exposes the method `typing.cast`.

This may lead to errors from users that can be hard to spot.


*Example*
It took me a few minutes to understand why the following code:

 
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql import functions as f

spark = SparkSession.builder.getOrCreate()
df = spark.sql("""SELECT 1 as a""")
df.withColumn("a", f.cast("STRING", f.col("a"))).printSchema()  {code}
which executes without any problem, gives the following result:

 

 
{code:java}
root
|-- a: integer (nullable = false){code}

This is because `f.cast` here calls `typing.cast, and the correct syntax is:
{code:java}
df.withColumn("a", f.col("a").cast("STRING")).printSchema(){code}
 
which indeed gives:

{code:java}
root
 |-- a: string (nullable = false) {code}

*Suggestion of solution*

Option 1: The methods imported in the module `pyspark.sql.functions` could be obfuscated to prevent this. For instance:
{code:java}
from typing import cast as _cast{code}

Option 2: only import `typing` and replace all occurrences of `cast` with `typing.cast`
Bug: Different column name of lit(np.int8)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/23 06:35
Affects Version/s: 3.4.0


{code:java}
('1', 'tinyint')
('CAST(1 AS TINYINT)', 'tinyint')

- [('1', 'tinyint')]
+ [('CAST(1 AS TINYINT)', 'tinyint')]

{code}