av9ash/GitBugs huggingface dataset:
Bug: Fix DataFrame createDataframe handling of None
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 03:05
Affects Version/s: 3.4.0

{code:java}
row = self.spark.createDataFrame([("Alice", None, None, None)], schema).fillna(True).first()
self.assertEqual(row.age, None){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 231, in test_fillna
    self.assertEqual(row.age, None)
AssertionError: nan != None{code}
 
{code:java}
row = (
    self.spark.createDataFrame([("Alice", 10, None)], schema)
    .replace(10, 20, subset=["name", "height"])
    .first()
)
self.assertEqual(row.name, "Alice")
self.assertEqual(row.age, 10)
self.assertEqual(row.height, None) {code}
{code:java}
Traceback (most recent call last):   File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 372, in test_replace     self.assertEqual(row.height, None) AssertionError: nan != None
{code}
Bug: Shuffle output location refetch during shuffle migration in decommission
Status: Open | Priority: Major | Resolution: 
Created: 10/Jan/23 01:33
Affects Version/s: 3.3.1

When shuffle migration enabled during spark decommissionm, shuffle data will be migrated into live executors, then update latest location to MapOutputTracker. It has some issues:
 # Executors only do map output location fetch in the beginning of the reduce stage, so any shuffle output location change in the middle of reduce will cause FetchFailed as reducer fetch from old location. Even stage retries could solve this, this still cause lots of resource waste as all shuffle read and compute happened before FetchFailed partition will be wasted.
 # During stage retries, less running tasks cause more executors to be decommissioned and shuffle data location keep changing. In the worst case, stage could need lots of retries, further breaking SLA.

So I propose to support refetch map output location during reduce phase if shuffle migration is enabled and FetchFailed is caused by a decommissioned dead executor. The detailed steps as below:
 # When `BlockTransferService` fetch blocks failed from a decommissioned dead executor, ExecutorDeadException(isDecommission as true) will be thrown.
 # Make MapOutputTracker support fetch latest output without epoch provided.
 # `ShuffleBlockFetcherIterator` will refetch latest output from MapOutputTrackMaster. For all the shuffle blocks on this decommissioned, there should be a new location on another executor. If not, throw exception as current. If yes, create new local and remote requests to fetch these migrated shuffle blocks. The flow will be similar as failback fetch when push merged fetch failed.
Bug: Unify the schema or datatype in protos
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/23 02:55
Affects Version/s: 3.4.0

this ticket only focus on the protos sent from client to server.

we normally use 

{code:java}
  oneof schema {

    DataType datatype = 2;

    // Server will use Catalyst parser to parse this string to DataType.
    string datatype_str = 3;
  }
{code}

to represent a schema or datatype.

actually, we can simplify it with just a string. In the server, we can easily parse a DDL-formatted schema or a JSON formatted one.


{code:java}
  // (Optional) The schema of local data.
  // It should be either a DDL-formatted type string or a JSON string.
  //
  // The server side will update the column names and data types according to this schema.
  // If the 'data' is not provided, then this schema will be required.
  optional string schema = 2;
{code}
Bug: Fix Function `nth_value` functions output
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 05/Jan/23 17:51
Affects Version/s: 3.4.0

{code:java}
from pyspark.sql import Window
from pyspark.sql.functions import nth_value

df = self.spark.createDataFrame(
    [
        ("a", 0, None),
        ("a", 1, "x"),
        ("a", 2, "y"),
        ("a", 3, "z"),
        ("a", 4, None),
        ("b", 1, None),
        ("b", 2, None),
    ],
    schema=("key", "order", "value"),
)
w = Window.partitionBy("key").orderBy("order")

rs = df.select(
    df.key,
    df.order,
    nth_value("value", 2).over(w),
    nth_value("value", 2, False).over(w),
    nth_value("value", 2, True).over(w),
).collect()

expected = [
    ("a", 0, None, None, None),
    ("a", 1, "x", "x", None),
    ("a", 2, "x", "x", "y"),
    ("a", 3, "x", "x", "y"),
    ("a", 4, "x", "x", "y"),
    ("b", 1, None, None, None),
    ("b", 2, None, None, None),
]

for r, ex in zip(sorted(rs), sorted(expected)):
    self.assertEqual(tuple(r), ex[: len(r)]){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 755, in test_nth_value
    self.assertEqual(tuple(r), ex[: len(r)])
AssertionError: Tuples differ: ('a', 1, 'x', None) != ('a', 1, 'x', 'x')

First differing element 3:
None
'x'

- ('a', 1, 'x', None)
?               ^^^^

+ ('a', 1, 'x', 'x')
?               ^^^
 {code}
Bug: Current microbatch model is insufficient to efficiently handle partitioned logs with offsets
Status: Open | Priority: Major | Resolution: 
Created: 08/Jan/23 16:27
Affects Version/s: 3.2.3

Spark microbatch only allows unidirectional flow of information from the driver to the worker, which has multiple negative effects for partition and offset based sources.

If you have large gaps in the offsets between the current offset and the next message (such as when there is garbage collection, compaction or a newly-created subscription with a 0 offset seek point), the driver will need to work through all of the empty ranges, despite the worker knowing each time that the next message it will receive is at a much later offset.

In addition, because the driver can only specify a range of work in terms of offsets, the size of the range selected must be highly tuned to the size of messages in the backlog- when the size changes or is variable, there can be no correct selection that both allows high throughput and doesn't overload the driver when messages are large.

To resolve this, there needs to be a channel to communicate information about what messages were processed back to the driver code. This would allow simply resolving both of the above issues by 1) sending the offset of the last read message back to the driver and 2) structuring reads as <start offset, head offset, byte limit> instead of <start offset, end offset>, which would allow repeatable reads that also can respond to messages of different sizes by making different sized batches, and would allow skipping large offset gaps.

This would have the added advantage of allowing non-offset based systems (such as google Pub/Sub) to implement the microbatch API, by propagating information about which messages were read back to the driver code.
Bug: Reuse expressions in WindowSpecDefinition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jan/23 17:37
Affects Version/s: 3.4.0

For complex expressions in window spec definition, we extract it and replace it with an AttributeReference (with an internal column name, e.g. "_w0").
We can reuse the extracted expressions to avoid generator unnecessary Window node.

For example: the logical plan of query
{code:sql}
SELECT max(value) over(partition by (key) order by cast(ts as timestamp)) as w1,
       min(value) over(partition by (key) order by cast(ts as timestamp)) as w2
FROM t1
{code}
contains two Window node.
Bug: Infer IsNotNull constraints for complex join expressions
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Jan/23 08:49
Affects Version/s: 3.4.0

Infer IsNotNull constraints for complex join expressions could help filter a lot of rows before join.
For example,
{code:sql}
CREATE TABLE t(id int, text string) using parquet;

SELECT *
FROM t as t1
LEFT JOIN t as t2
ON t1.text = get_json_object(t2.text, '$.not_exists_col');
{code}
we can infer *IsNotNull(get_json_object(t2.text, '$.not_exists_col'))* constraint along with *IsNotNull(t2.text)* to filter a lot of rows for the right side of the join.
Bug: DataFrame.createDataFrame converting int to bigint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:34
Affects Version/s: 3.4.0

{code:java}
dt = datetime.date(2021, 12, 27)

# Note; number var in Python gets converted to LongType column;
# this is not supported by the function, so cast to Integer explicitly
df = self.spark.createDataFrame([Row(date=dt, add=2)], "date date, add integer")

self.assertTrue(
    all(
        df.select(
            date_add(df.date, df.add) == datetime.date(2021, 12, 29),
            date_add(df.date, "add") == datetime.date(2021, 12, 29),
            date_add(df.date, 3) == datetime.date(2021, 12, 30),
        ).first()
    )
){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 391, in test_date_add_function
    ).first()
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 246, in first
    return self.head()
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 310, in head
    rs = self.head(1)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 312, in head
    return self.take(n)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 317, in take
    return self.limit(num).collect()
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1076, in collect
    table = self._session.client.to_table(query)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 414, in to_table
    table, _ = self._execute_and_fetch(req)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 586, in _execute_and_fetch
    self._handle_error(rpc_error)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", line 625, in _handle_error
    raise SparkConnectAnalysisException(
pyspark.sql.connect.client.SparkConnectAnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "date_add(date, add)" due to data type mismatch: Parameter 2 requires the ("INT" or "SMALLINT" or "TINYINT") type, however "add" has the type "BIGINT".
Plan: 'GlobalLimit 1
+- 'LocalLimit 1
   +- 'Project [unresolvedalias('`==`(date_add(date#753, add#754L), 2021-12-29), None), unresolvedalias('`==`(date_add(date#753, add#754L), 2021-12-29), None), (date_add(date#753, 3) = 2021-12-30) AS (date_add(date, 3) = DATE '2021-12-30')#759]
      +- Project [date#753, add#754L]
         +- Project [date#749 AS date#753, add#750L AS add#754L]
            +- LocalRelation [date#749, add#750L]{code}
Bug: Function `from_json` should support non-literal expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 07:16
Affects Version/s: 3.4.0


{code:java}
File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1351, in pyspark.sql.connect.functions.from_json
Failed example:
    df.select(from_json(df.value, schema).alias("json")).collect()
Exception raised:
    Traceback (most recent call last):
      File "/Users/ruifeng.zheng/.dev/miniconda3/envs/spark_dev/lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.from_json[12]>", line 1, in <module>
        df.select(from_json(df.value, schema).alias("json")).collect()
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1019, in collect
        pdf = self.toPandas()
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/dataframe.py", line 1054, in toPandas
        return self._session.client.to_pandas(query)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 413, in to_pandas
        return self._execute_and_fetch(req)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 573, in _execute_and_fetch
        self._handle_error(rpc_error)
      File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/client.py", line 625, in _handle_error
        raise SparkConnectException(status.message) from None
    pyspark.sql.connect.client.SparkConnectException: Schema in from_json should be literal string, but got 'schema_of_json({"a": 0})

{code}
Bug: Implement DataFrame.explain format to be similar to PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 17:08
Affects Version/s: 3.4.0

{code:java}
df = spark.createDataFrame([(14, "Tom"), (23, "Alice"), (16, "Bob")], ["age", "name"]) 
df.explain()
df.explain(True)
df.explain(mode="formatted")
df.explain("cost"){code}
{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1296, in pyspark.sql.connect.dataframe.DataFrame.explain
Failed example:
    df.explain()
Expected:
    == Physical Plan ==
    *(1) Scan ExistingRDD[age...,name...]
Got:
    == Physical Plan ==
    LocalTableScan [age#1148L, name#1149]
    <BLANKLINE>
    <BLANKLINE>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", line 1314, in pyspark.sql.connect.dataframe.DataFrame.explain
Failed example:
    df.explain(mode="formatted")
Expected:
    == Physical Plan ==
    * Scan ExistingRDD (...)
    (1) Scan ExistingRDD [codegen id : ...]
    Output [2]: [age..., name...]
    ...
Got:
    == Physical Plan ==
    LocalTableScan (1)
    <BLANKLINE>
    <BLANKLINE>
    (1) LocalTableScan
    Output [2]: [age#1170L, name#1171]
    Arguments: [age#1170L, name#1171]
    <BLANKLINE>
    <BLANKLINE>{code}
Bug: Window.rowsBetween should handle `float("-inf")` and `float("+inf")` as argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:30
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame([(1, "1"), (2, "2"), (1, "2"), (1, "2")], ["key", "value"])
w = Window.partitionBy("value").orderBy("key")
from pyspark.sql import functions as F

sel = df.select(
    df.value,
    df.key,
    F.max("key").over(w.rowsBetween(0, 1)),
    F.min("key").over(w.rowsBetween(0, 1)),
    F.count("key").over(w.rowsBetween(float("-inf"), float("inf"))),
    F.row_number().over(w),
    F.rank().over(w),
    F.dense_rank().over(w),
    F.ntile(2).over(w),
)
rs = sorted(sel.collect()){code}
{code:java}
Traceback (most recent call last):   File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 821, in test_window_functions     F.count("key").over(w.rowsBetween(float("-inf"), float("inf"))),   File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/window.py", line 152, in rowsBetween     raise TypeError(f"start must be a int, but got {type(start).__name__}") TypeError: start must be a int, but got float {code}
Bug: Enable spark.sql.orc.enableNestedColumnVectorizedReader by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/23 19:10
Affects Version/s: 3.4.0

Like `spark.sql.parquet.enableNestedColumnVectorizedReader`, this issue aims to enable `spark.sql.orc.enableNestedColumnVectorizedReader` by default.
Bug: Refine the naming in proto messages
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/23 02:51
Affects Version/s: 3.4.0

normally, we name the fields after the corresponding LogiclalPlan or DataFrame API, but they are not consistent in protos, for example, the column name:


{code:java}
  message UnresolvedRegex {
    // (Required) The column name used to extract column with regex.
    string col_name = 1;
  }
{code}


{code:java}
  message Alias {
    // (Required) The expression that alias will be added on.
    Expression expr = 1;

    // (Required) a list of name parts for the alias.
    //
    // Scalar columns only has one name that presents.
    repeated string name = 2;

    // (Optional) Alias metadata expressed as a JSON map.
    optional string metadata = 3;
  }
{code}



{code:java}
// Relation of type [[Deduplicate]] which have duplicate rows removed, could consider either only
// the subset of columns or all the columns.
message Deduplicate {
  // (Required) Input relation for a Deduplicate.
  Relation input = 1;

  // (Optional) Deduplicate based on a list of column names.
  //
  // This field does not co-use with `all_columns_as_keys`.
  repeated string column_names = 2;

  // (Optional) Deduplicate based on all the columns of the input relation.
  //
  // This field does not co-use with `column_names`.
  optional bool all_columns_as_keys = 3;
}
{code}


{code:java}
// Computes basic statistics for numeric and string columns, including count, mean, stddev, min,
// and max. If no columns are given, this function computes statistics for all numerical or
// string columns.
message StatDescribe {
  // (Required) The input relation.
  Relation input = 1;

  // (Optional) Columns to compute statistics on.
  repeated string cols = 2;
}
{code}


we probably should unify the naming:

single column -> `column`

multi columns -> `columns`
Bug: Remove `optional` notation in proto
Status: Open | Priority: Major | Resolution: 
Created: 05/Jan/23 18:51
Affects Version/s: 3.4.0

Every field in proto3 has a default value. We should revisit existing proto field to understand if the default value can be used without tell the field is set and not set, and remove `optional` as much as possible from Spark Connect proto surface.
Bug: Function `slice` should handle string in params
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:53
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame(
    [
        (
            [1, 2, 3],
            2,
            2,
        ),
        (
            [4, 5],
            2,
            2,
        ),
    ],
    ["x", "index", "len"],
)

expected = [Row(sliced=[2, 3]), Row(sliced=[5])]
self.assertTrue(
    all(
        [
            df.select(slice(df.x, 2, 2).alias("sliced")).collect() == expected,
            df.select(slice(df.x, lit(2), lit(2)).alias("sliced")).collect() == expected,
            df.select(slice("x", "index", "len").alias("sliced")).collect() == expected,
        ]
    )
)

self.assertEqual(
    df.select(slice(df.x, size(df.x) - 1, lit(1)).alias("sliced")).collect(),
    [Row(sliced=[2]), Row(sliced=[4])],
)
self.assertEqual(
    df.select(slice(df.x, lit(1), size(df.x) - 1).alias("sliced")).collect(),
    [Row(sliced=[1, 2]), Row(sliced=[4])],
){code}
{code:java}
 Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 596, in test_slice
    df.select(slice("x", "index", "len").alias("sliced")).collect() == expected,
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/utils.py", line 332, in wrapped
    return getattr(functions, f.__name__)(*args, **kwargs)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 1525, in slice
    raise TypeError(f"start should be a Column or int, but got {type(start).__name__}")
TypeError: start should be a Column or int, but got str{code}
Bug: Handle Function `rand() `
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jan/23 17:56
Affects Version/s: 3.4.0

{code:java}
df = self.df
from pyspark.sql import functions

rnd = df.select("key", functions.rand()).collect()
for row in rnd:
    assert row[1] >= 0.0 and row[1] <= 1.0, "got: %s" % row[1]
rndn = df.select("key", functions.randn(5)).collect()
for row in rndn:
    assert row[1] >= -4.0 and row[1] <= 4.0, "got: %s" % row[1]

# If the specified seed is 0, we should use it.
# https://issues.apache.org/jira/browse/SPARK-9691
rnd1 = df.select("key", functions.rand(0)).collect()
rnd2 = df.select("key", functions.rand(0)).collect()
self.assertEqual(sorted(rnd1), sorted(rnd2))

rndn1 = df.select("key", functions.randn(0)).collect()
rndn2 = df.select("key", functions.randn(0)).collect()
self.assertEqual(sorted(rndn1), sorted(rndn2)){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_functions.py", line 299, in test_rand_functions
    rnd = df.select("key", functions.rand()).collect()
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/dataframe.py", line 2917, in select
    jdf = self._jdf.select(self._jcols(*cols))
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/dataframe.py", line 2537, in _jcols
    return self._jseq(cols, _to_java_column)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/dataframe.py", line 2524, in _jseq
    return _to_seq(self.sparkSession._sc, cols, converter)
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/column.py", line 86, in _to_seq
    cols = [converter(c) for c in cols]
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/column.py", line 86, in <listcomp>
    cols = [converter(c) for c in cols]
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/column.py", line 65, in _to_java_column
    raise TypeError(
TypeError: Invalid argument, not a string or column: Column<'rand()'> of type <class 'pyspark.sql.connect.column.Column'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
{code}
Bug: DataFrame dropDuplicates should throw error on non list argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/23 02:51
Affects Version/s: 3.4.0

{code:java}
df = self.spark.createDataFrame([("Alice", 50), ("Alice", 60)], ["name", "age"])

# shouldn't drop a non-null row
self.assertEqual(df.dropDuplicates().count(), 2)

self.assertEqual(df.dropDuplicates(["name"]).count(), 1)

self.assertEqual(df.dropDuplicates(["name", "age"]).count(), 2)

type_error_msg = "Parameter 'subset' must be a list of columns"
with self.assertRaisesRegex(TypeError, type_error_msg):
    df.dropDuplicates("name"){code}
{code:java}
Traceback (most recent call last):
  File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/tests/test_dataframe.py", line 128, in test_drop_duplicates
    with self.assertRaisesRegex(TypeError, type_error_msg):
AssertionError: TypeError not raised{code}
Bug: DataFrame.show(): 'Column' object is not callable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/23 22:50
Affects Version/s: 3.4.0

{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", line 855, in pyspark.sql.connect.functions.first
Failed example:
    df.groupby("name").agg(first("age", ignorenulls=True)).orderBy("name").show()
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.functions.first[3]>", line 1, in <module>
        df.groupby("name").agg(first("age", ignorenulls=True)).orderBy("name").show()
    TypeError: 'Column' object is not callable{code}