av9ash/GitBugs huggingface dataset:
Bug: Compatibility of string representation in Column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/22 01:44
Affects Version/s: 3.4.0

Doctest in pyspark.sql.connect.column.Columnfails with the error below:
{code:java}
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 120, in pyspark.sql.connect.column.Column
Failed example:
    df.name
Expected:
    Column<'name'>
Got:
    Column<'ColumnReference(name)'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 122, in pyspark.sql.connect.column.Column
Failed example:
    df["name"]
Expected:
    Column<'name'>
Got:
    Column<'ColumnReference(name)'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 127, in pyspark.sql.connect.column.Column
Failed example:
    df.age + 1
Expected:
    Column<'(age + 1)'>
Got:
    Column<'+(ColumnReference(age), Literal(1))'>
**********************************************************************
File "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/column.py", line 129, in pyspark.sql.connect.column.Column
Failed example:
    1 / df.age
Expected:
    Column<'(1 / age)'>
Got:
    Column<'/(Literal(1), ColumnReference(age))'> {code}
 

We should enable this back after fixing the issue in Spark Connect
Bug: Implement Baseline API Code
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/22 00:16
Affects Version/s: 3.4.0

Creating a baseline API so that we can agree on how the users will interact with the code. This was determined in this [Design Document|https://docs.google.com/document/d/1_nhUP46cHnYmnZoyirySXvuY1KDMU3vdHRx9MngSVtA/edit] and can be updated as necessary.
Bug: SPJ: Avoid shuffle when partition keys mismatch, but join expressions are compatible
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/22 20:42
Affects Version/s: 3.3.1

Currently when checking whether two sides of a Storage Partitioned Join are compatible, we requires both the partition expressions as well as the partition keys are compatible. However, this condition could be relaxed so that we only require the former. In the case that the latter is not compatible, we can calculate a common superset of keys and push down the information to both sides of the join, and use empty partitions for the missing keys.
Bug: Assign name to _LEGACY_ERROR_TEMP_1130
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Support for protocol extensions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/22 14:41
Affects Version/s: 3.4.0

Spark comes with many different extension points. Many of those simply become available through the shared classpath between Spark and the user application. To be able to support arbitrary plugins e.g. for Delta or Iceberg, we need a way to make the Spark Connect protocol extensible and let users register their own handlers.
Bug: Union of tables with and without metadata column fails when used in join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Dec/22 07:19
Affects Version/s: 3.3.2

Here is a test case that can be added to {{MetadataColumnSuite}} to demonstrate the issue
{code:scala}
  test("SPARK-41557: Union of tables with and without metadata column should work") {
    withTable(tbl) {
      sql(s"CREATE TABLE $tbl (id bigint, data string) PARTITIONED BY (id)")
      checkAnswer(
        spark.sql(
          s"""
            SELECT b.*
            FROM RANGE(1)
              LEFT JOIN (
                SELECT id FROM $tbl
                UNION ALL
                SELECT id FROM RANGE(10)
              ) b USING(id)
          """),
        Seq(Row(0))
      )
    }
  }
 {code}

Here a table with metadata columns {{$tbl}} is unioned with a table without metdata columns {{RANGE(10)}}. If this result is later used in a join, query analysis fails saying mismatch in the number of columns of the union caused by the metadata columns. However, here we can see that we explicitly project only one column during the union, so the union should be valid.

{code}
org.apache.spark.sql.AnalysisException: [NUM_COLUMNS_MISMATCH] UNION can only be performed on inputs with the same number of columns, but the first input has 3 columns and the second input has 1 columns.; line 5 pos 16;
'Project [id#26L]
+- 'Project [id#26L, id#26L]
   +- 'Project [id#28L, id#26L]
      +- 'Join LeftOuter, (id#28L = id#26L)
         :- Range (0, 1, step=1, splits=None)
         +- 'SubqueryAlias b
            +- 'Union false, false
               :- Project [id#26L, index#30, _partition#31]
               :  +- SubqueryAlias testcat.t
               :     +- RelationV2[id#26L, data#27, index#30, _partition#31] testcat.t testcat.t
               +- Project [id#29L]
                  +- Range (0, 10, step=1, splits=None)
{code}
Bug: Assign name to _LEGACY_ERROR_TEMP_2009
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Assign name to _LEGACY_ERROR_TEMP_1249
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: SparkSession.createDataFrame does not support nested datatypes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:13
Affects Version/s: 3.4.0

{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in pyspark.sql.connect.group.GroupedData.pivot
Failed example:
    df2 = spark.createDataFrame([
        Row(training="expert", sales=Row(course="dotNET", year=2012, earnings=10000)),
        Row(training="junior", sales=Row(course="Java", year=2012, earnings=20000)),
        Row(training="expert", sales=Row(course="dotNET", year=2012, earnings=5000)),
        Row(training="junior", sales=Row(course="dotNET", year=2013, earnings=48000)),
        Row(training="expert", sales=Row(course="Java", year=2013, earnings=30000)),
    ])
Exception raised:
    Traceback (most recent call last):
      File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest pyspark.sql.connect.group.GroupedData.pivot[3]>", line 1, in <module>
        df2 = spark.createDataFrame([
      File "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line 196, in createDataFrame
        table = pa.Table.from_pandas(pdf)
      File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 611, in dataframe_to_arrays
        arrays = [convert_column(c, f)
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 611, in <listcomp>
        arrays = [convert_column(c, f)
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 598, in convert_column
        raise e
      File "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", line 592, in convert_column
        result = pa.array(col, type=type_, from_pandas=True, safe=safe)
      File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
      File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
      File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status
    pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", 'Conversion failed for column 1 with type object')
{code}
Bug: Assign name to _LEGACY_ERROR_TEMP_2054
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Support generator function explode/explode_outer in the FROM clause
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Dec/22 01:24
Affects Version/s: 3.4.0

Currently, the table-valued generator function explode/explode_outer can only be used in the SELECT clause of a query:

SELECT explode(array(1, 2))

This task is to allow table-valued functions to be used in the FROM clause of a query:

SELECT * FROM explode(array(1, 2))
Bug: `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Dec/22 03:15
Affects Version/s: 3.3.0

{code:scala}
scala> spark.sql("select regexp_replace('', '[a\\\\d]{0,2}', 'x')").show
+----------------------------------+
|regexp_replace(, [a\d]\{0,2}, x, 1)|
+----------------------------------+
|                                 x|
+----------------------------------+
 
 
scala> spark.sql("select regexp_replace('', '[a\\\\d]{0, 2}', 'x')").show
org.apache.spark.SparkException: The Spark SQL phase optimization failed with an internal error. Please, fill a bug report in, and provide the full stack trace.
{code}
Bug: Assign name to _LEGACY_ERROR_TEMP_1230
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:10
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Resolve dtypes inconsistencies of date/timestamp functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/22 23:52
Affects Version/s: 3.4.0

When implementing date/timestamp functions, we notice inconsistent dtypes with PySpark, as shown below.

{code:python}
>> sdf.select(SF.current_timestamp()).toPandas().dtypes
current_timestamp()    datetime64[ns]
dtype: object
>>> cdf.select(CF.current_timestamp()).toPandas().dtypes
current_timestamp()    datetime64[ns, America/Los_Angeles]
{code}

Affected functions include:

{code:python}
to_timestamp, from_utc_timestamp, to_utc_timestamp, timestamp_seconds, current_timestamp, date_trunc
{code}

We may have to implement `is_timestamp_ntz_preferred` for Connect.

After the fix, tests of those date/timestamp functions which use `compare_by_show` should be switched to `toPandas` comparison.
Bug: Reorder fields to use consecutive field numbers
Status: Open | Priority: Major | Resolution: 
Created: 29/Dec/22 01:31
Affects Version/s: 3.4.0

make IDs consecutive

```
    RepartitionByExpression repartition_by_expression = 27;

    // NA functions
    NAFill fill_na = 90;
    NADrop drop_na = 91;
    NAReplace replace = 92;
```
Bug: Metadata column should be appended at the end of project list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Dec/22 00:03
Affects Version/s: 3.3.2

For the following query:

 
{code:java}
CREATE TABLE table_1 (
  a ARRAY<STRING>,
 s STRUCT<id: STRING>)
USING parquet;

CREATE VIEW view_1 (id)
AS WITH source AS (
    SELECT * FROM table_1
),
renamed AS (
    SELECT
     s.id
    FROM source
)
SELECT id FROM renamed;

with foo AS (
  SELECT 'a' as id
),
bar AS (
  SELECT 'a' as id
)
SELECT
  1
FROM foo
FULL OUTER JOIN bar USING(id)
FULL OUTER JOIN view_1 USING(id)
WHERE foo.id IS NOT NULL{code}
There will be the following error:

 
{code:java}
class org.apache.spark.sql.types.ArrayType cannot be cast to class org.apache.spark.sql.types.StructType (org.apache.spark.sql.types.ArrayType and org.apache.spark.sql.types.StructType are in unnamed module of loader 'app')
java.lang.ClassCastException: class org.apache.spark.sql.types.ArrayType cannot be cast to class org.apache.spark.sql.types.StructType (org.apache.spark.sql.types.ArrayType and org.apache.spark.sql.types.StructType are in unnamed module of loader 'app')
    at org.apache.spark.sql.catalyst.expressions.GetStructField.childSchema$lzycompute(complexTypeExtractors.scala:108)
    at org.apache.spark.sql.catalyst.expressions.GetStructField.childSchema(complexTypeExtractors.scala:108)
    at org.apache.spark.sql.catalyst.expressions.GetStructField.dataType(complexTypeExtractors.scala:114)
    at org.apache.spark.sql.catalyst.expressions.Alias.toAttribute(namedExpressions.scala:193)
    at org.apache.spark.sql.catalyst.expressions.AliasHelper$$anonfun$getAliasMap$1.applyOrElse(AliasHelper.scala:50)
    at org.apache.spark.sql.catalyst.expressions.AliasHelper$$anonfun$getAliasMap$1.applyOrElse(AliasHelper.scala:50)
    at scala.collection.immutable.List.collect(List.scala:315)
    at org.apache.spark.sql.catalyst.expressions.AliasHelper.getAliasMap(AliasHelper.scala:50)
    at org.apache.spark.sql.catalyst.expressions.AliasHelper.getAliasMap$(AliasHelper.scala:47)
    at org.apache.spark.sql.catalyst.optimizer.CollapseProject$.getAliasMap(Optimizer.scala:992)
    at org.apache.spark.sql.catalyst.optimizer.CollapseProject$.canCollapseExpressions(Optimizer.scala:1029){code}
This is caused by the inconsistent metadata column positions in the following two nodes:
 * Table relation: at the ending position
 * Project list: at the beginning position

When the InlineCTE rule executes, the metadata column in project is wrongly combined with the table output.
Bug: Identify aggregation expression in the nodePatterns of PythonUDF
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 20/Dec/22 20:28
Affects Version/s: 3.4.0

When a PythonUDF is evaluated as SQL_GROUPED_AGG_PANDAS_UDF, we can mark it as AGGREGATE_EXPRESSION in the `nodePatterns`. So that we can check whether an expression contains aggregation with a handy way: `expr.containsPattern(AGGREGATE_EXPRESSION)`
Bug: Create distinct metadata attributes for metadata that is constant or file and metadata that is generated during the scan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Dec/22 14:15
Affects Version/s: 3.3.1

There are two types or Metadata in Spark
 * Metadata that is constant per file (file_name, file_size, ...)
 * Metadata that is not contant (currently only row_index)

The two types are generated differently
 * File constant metadata is appended to the output after scan
 * non-constant metadata is generated during the scan

The proposal here is to create different metadata attributes to distinguish those different types throughout the code
Bug: Set TRANSFORM reader and writer's format correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Dec/22 13:12
Affects Version/s: 3.3.1

We'll get wrong data when transform only specify reader or writer 's row format delimited, the reason is using the wrong format to feed/fetch data to/from running script now.  In theory, writer uses inFormat to feed to input data into the running script and reader uses outFormat to read the output from the running script, but inFormat and outFormat are set wrong value currently in the following code:
{code:java}
val (inFormat, inSerdeClass, inSerdeProps, reader) =
  format(
    inRowFormat, "hive.script.recordreader",
    "org.apache.hadoop.hive.ql.exec.TextRecordReader")

val (outFormat, outSerdeClass, outSerdeProps, writer) =
  format(
    outRowFormat, "hive.script.recordwriter",
    "org.apache.hadoop.hive.ql.exec.TextRecordWriter") {code}
 

Example SQL:
{code:java}
spark-sql> CREATE TABLE t1 (a string, b string); 

spark-sql> INSERT OVERWRITE t1 VALUES("1", "2"), ("3", "4");

spark-sql> SELECT TRANSFORM(a, b)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         >   USING 'cat'
         >   AS (c)
         > FROM t1;
c

spark-sql> SELECT TRANSFORM(a, b)
         >   USING 'cat'
         >   AS (c)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         > FROM t1;
c
1    23    4{code}
 

The same sql in hive:
{code:java}
hive> SELECT TRANSFORM(a, b)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    >   USING 'cat'
    >   AS (c)
    > FROM t1;
c
1,2
3,4

hive> SELECT TRANSFORM(a, b)
    >   USING 'cat'
    >   AS (c)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    > FROM t1;
c
1    2
3    4 {code}
Bug: Assign name to _LEGACY_ERROR_TEMP_2137
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Assign name to _LEGACY_ERROR_TEMP_2051
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 06:04
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Add new config to clean up `spark.ui.store.path` directory when SparkContext.stop()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Dec/22 08:34
Affects Version/s: 3.4.0

{{spark.ui.store.path}} directory not clean up when {{SparkContext.stop() now:}}
 # {{{}{}}}The disk space occupied by the {{spark.ui.store.path}} directory will continue to grow.
 # When submitting new App and reusing the {{spark.ui.store.path}} directory, we will see the content related to the previous App, which is a bit strange
Bug: Assign name to _LEGACY_ERROR_TEMP_2136
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 05:59
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time
Bug: Test parity: enable doctests in Spark Connect
Status: Resolved | Priority: Major | Resolution: Done
Created: 21/Dec/22 07:22
Affects Version/s: 3.4.0

We should actually run the doctests of Spark Connect.

We should add something like https://github.com/apache/spark/blob/master/python/pyspark/sql/column.py#L1227-L1247 to Spark Connect modules, and add the module into https://github.com/apache/spark/blob/master/dev/sparktestsupport/modules.py#L507
Bug: InterpretedUnsafeProjection doesn't properly handle an array of UDTs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/22 23:23
Affects Version/s: 3.4.0

Reproduction steps:
{noformat}
// create a file of vector data
import org.apache.spark.ml.linalg.{DenseVector, Vector}

case class TestRow(varr: Array[Vector])
val values = Array(0.1d, 0.2d, 0.3d)
val dv = new DenseVector(values).asInstanceOf[Vector]

val ds = Seq(TestRow(Array(dv, dv))).toDS
ds.coalesce(1).write.mode("overwrite").format("parquet").save("vector_data")

// this works
spark.read.format("parquet").load("vector_data").collect

sql("set spark.sql.codegen.wholeStage=false")
sql("set spark.sql.codegen.factoryMode=NO_CODEGEN")

// this will get an error
spark.read.format("parquet").load("vector_data").collect
{noformat}
The error varies each time you run it, e.g.:
{noformat}
Sparse vectors require that the dimension of the indices match the dimension of the values.
You provided 2 indices and  6619240 values.
{noformat}
or
{noformat}
org.apache.spark.SparkRuntimeException: Error while decoding: java.lang.NegativeArraySizeException
{noformat}
or
{noformat}
java.lang.OutOfMemoryError: Java heap space
  at org.apache.spark.sql.catalyst.expressions.UnsafeArrayData.toDoubleArray(UnsafeArrayData.java:414)
{noformat}
or
{noformat}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x00000001120c9d30, pid=64213, tid=0x0000000000001003
#
# JRE version: Java(TM) SE Runtime Environment (8.0_311-b11) (build 1.8.0_311-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.311-b11 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.dylib+0xc9d30]  acl_CopyRight+0x29
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /<my-local-directory>/hs_err_pid64213.log
Compiled method (nm)  582142 11318     n 0       sun.misc.Unsafe::copyMemory (native)
 total in heap  [0x000000011efa8890,0x000000011efa8be8] = 856
 relocation     [0x000000011efa89b8,0x000000011efa89f8] = 64
 main code      [0x000000011efa8a00,0x000000011efa8be8] = 488
Compiled method (nm)  582142 11318     n 0       sun.misc.Unsafe::copyMemory (native)
 total in heap  [0x000000011efa8890,0x000000011efa8be8] = 856
 relocation     [0x000000011efa89b8,0x000000011efa89f8] = 64
 main code      [0x000000011efa8a00,0x000000011efa8be8] = 488
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
{noformat}
Bug: SparkSession.createDataFrame does not respect the column names in the row
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Dec/22 12:12
Affects Version/s: 3.4.0



{code}
File "/.../spark/python/pyspark/sql/connect/group.py", line 173, in pyspark.sql.connect.group.GroupedData.pivot
Failed example:
    df1.show()
Differences (ndiff with -expected +actual):
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    - |course|year|earnings|
    + |    _1|  _2|   _3|
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    - |dotNET|2012|   10000|
    ?              ---
    + |dotNET|2012|10000|
    - |  Java|2012|   20000|
    ?              ---
    + |  Java|2012|20000|
    - |dotNET|2012|    5000|
    ?               ---
    + |dotNET|2012| 5000|
    - |dotNET|2013|   48000|
    ?              ---
    + |dotNET|2013|48000|
    - |  Java|2013|   30000|
    ?              ---
    + |  Java|2013|30000|
    - +------+----+--------+
    ?                   ---
    + +------+----+-----+
    + <BLANKLINE>
{code}
Bug: Assign name to _LEGACY_ERROR_TEMP_2310
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/22 05:59
Affects Version/s: 3.4.0

We should use proper error class name rather than `_LEGACY_ERROR_TEMP_xxx`.

 

*NOTE:* Please reply to this ticket before start working on it, to avoid working on same ticket at a time