av9ash/GitBugs huggingface dataset:
Bug: Streaming python data source API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jan/24 19:38
Affects Version/s: 4.0.0

This is a follow up of https://issues.apache.org/jira/browse/SPARK-44076. The idea is to enable Python developers to develop streaming data sources in python. The goal is to make a Python-based API that is simple and easy to use, thus making Spark more accessible to the wider Python developer community.

 

Design doc: https://docs.google.com/document/d/1cJ-w1hGPOBFp-5DLmf68sTLsAOwb55oW6SAuuAUFEM4/edit
Bug: CacheTable AsSelect should inherit from CTEInChildren to make sure it can be matched
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/24 03:47
Affects Version/s: 4.0.0

Current code since CaheTableAsSelelct not inherit CETInChildren,  still return forceInline.  Make cache result can't be matched

!image-2024-01-17-11-48-28-867.png|width=859,height=363!
Bug: Support coalesce partition through cartesian product
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 22/Jan/24 07:07
Affects Version/s: 3.5.0

Like `Union`, It's also better to support partial optimize with `CartesianProduct`.
{code:java}
CartesianProduct
   HashAggregate
      ShuffleQueryStage
   FileScan
{code}
Bug: Coalesce partiton assert error after skew join optimization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jan/24 08:36
Affects Version/s: 3.3.1

Recently when we were testing TPCDS Q71, we found that if `spark.sql.shuffle.partitions` and `spark.sql.adaptive.coalescePartitions.initialPartitionNum` are both set to the number of executor cores, an AssertError may be reported in coalescePartition due to the partitionSpecs of joins after skew are different.
Bug: Exclude unittest-xml-reporting in Python 3.12 image
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 02:31
Affects Version/s: 4.0.0

unittest-xml-reporting seems not supporting, and this seems hiding the real error:

{code}
  File "/__w/spark/spark/python/pyspark/streaming/tests/test_kinesis.py", line 118, in <module>
    unittest.main(testRunner=testRunner, verbosity=2)
  File "/usr/lib/python3.12/unittest/main.py", line 105, in __init__
    self.runTests()
  File "/usr/lib/python3.12/unittest/main.py", line 281, in runTests
    self.result = testRunner.run(self.test)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/xmlrunner/runner.py", line 67, in run
    test(result)
  File "/usr/lib/python3.12/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.12/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.12/unittest/case.py", line 692, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/unittest/case.py", line 662, in run
    result.stopTest(self)
  File "/usr/local/lib/python3.12/dist-packages/xmlrunner/result.py", line 327, in stopTest
    self.callback()
  File "/usr/local/lib/python3.12/dist-packages/xmlrunner/result.py", line 235, in callback
    test_info.test_finished()
  File "/usr/local/lib/python3.12/dist-packages/xmlrunner/result.py", line 180, in test_finished
    self.test_result.stop_time - self.test_result.start_time
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: '_XMLTestResult' object has no attribute 'start_time'. Did you mean: 'stop_time'?

{code}

This is optional dependency in testing so we can exclude this.
Bug: String function support (parent)
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/24 17:26
Affects Version/s: 4.0.0

TODO: List of all functions that need to be updated for collation support:

 
|Feature/function|Priority|Type|
|Shuffle|0|comparison|
|Delta Columns|0|storage|
|Partition key|0|storage|
|Comparison operators|0|comparison|
|IN list|0|comparison|
|GROUP BY |0|comparison|
|MERGE, HASH joins|0|comparison|
|ORDER BY |0|sorting|
|Aggregation|0|comparison|
|like|0|comparison|
|regexp_*|0|matching|
|concat|0|Pass through|
|substr|0|Pass through|
|between|0|comparison|
|coalesce |0|Pass through|
|Is distinct|0|comparison|
|*trim*|0|Pass through|
|instr|0|comparison|
|lcase|0|Pass through, modify|
|lower|0|Pass through modify|
|replace|0|comparison|
|ucase|0|modify , pass through|
|upper|0|Modify, pass through|
|count(distinct )|0|comparison|
|min/max|0|Comparison pass through|
|array|0|Pass through|
|case|0|Pass through|
|decode|0|Pass through|
|elt|0|Comparison, pass through|
|Nullif, nvl, nvl2|0|Pass through, comparison|
| | | |
|Session variables|1|storage|
|SQL UDF|1|Storage, pass through|
|Python UDF|1|Storage|
|Array element|1|storage|
|Map key|1|Storage, comparison|
|Map value|1|Storage|
|Struct field|1|storage|
|least/greatest|1|Comparison, pass through|
|if/iff/ifnull|1|Pass through, comparison|
|mapExpr [ keyExpr ]|1|Comparison, pass through|
|concat_ws|1|Pass through|
|contains|1|comparison|
|left|1|Pass through|
|*pad|1|Pass through|
|repeat|1|Pass through|
|reverse|1|Pass through|
|translate|1|Comparison, Pass through|
|array_agg|1|Pass through|
|first/last/any|1|Pass through|
|mode|1|Comparison, pass through|
|array_*|1|Pass through, dedup (array distinct)|
| | | |
|*explode*|2|Pass through|
|filter|2|Pass through|
|flatten|2|Pass through|
|inline*|2|Pass through|
|reduce|2|Pass through|
|reverse|2|Pass through|
|shuffle|2|Pass through|
|Slice|2|Pass through|
|sort_array|2|Comparison, pass through|
|transform|2|Pass through|
|zip*|2|Pass through|
|map|2|Pass through|
|map_*|2|Pass through|
|str_to_map|2|Comparison, pass through|
|transform*|2|Pass through|
|stack|2|Pass through|
|describe|2|display|
|ilike|2|matching|
|charindex|2|comparison|
|endswith|2|comparison|
|startswith|2|comparison|
|find_in_set|2|comparison|
|initcap|2|Pass through, modify|
|locate|2|comparison|
|mask|2|Pass through|
|overlay|2|Pass through|
|position|2|comparison|
|sentences|2|Comparison, pass through|
|split|2|Comparison, pass through|
|split_part|2|Comparison, pass through|
|collect_list|2|Pass through|
|collect_set|2|Pass through|
|min_by/max_by|2|Comparison, pass through|
|Element_at, []|2|Pass through|
|aggregate|2|Pass through|
Bug: Overwrite partitions with custom location should reset partition locations
Status: Open | Priority: Major | Resolution: 
Created: 14/Jan/24 14:21
Affects Version/s: 3.5.0

In hive metastore we support partitions to be outside of corresponding table location.

When overwrite such partition with hive, the overwritten partitions should be recreated under table location.

Also, currently if a partition is on a different fileystem from the table, when overwriting spark will throw exception.
Bug: df.show() of pyspark displayed different results between Regular Spark and Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/24 05:46
Affects Version/s: 4.0.0

The following doctest will throw an error in the tests of the pyspark-connect module
{code:java}
Example 2: Converting a complex StructType to a CSV string    

    >>> from pyspark.sql import Row, functions as sf
    >>> data = [(1, Row(age=2, name='Alice', scores=[100, 200, 300]))]
    >>> df = spark.createDataFrame(data, ("key", "value"))
    >>> df.select(sf.to_csv(df.value)).show(truncate=False) # doctest: +SKIP
    +-----------------------+
    |to_csv(value)          |
    +-----------------------+
    |2,Alice,"[100,200,300]"|
    +-----------------------+{code}
{code:java}
**********************************************************************
3953File "/__w/spark/spark/python/pyspark/sql/connect/functions/builtin.py", line 2232, in pyspark.sql.connect.functions.builtin.to_csv
3954Failed example:
3955    df.select(sf.to_csv(df.value)).show(truncate=False)
3956Expected:
3957    +-----------------------+
3958    |to_csv(value)          |
3959    +-----------------------+
3960    |2,Alice,"[100,200,300]"|
3961    +-----------------------+
3962Got:
3963    +--------------------------------------------------------------------------+
3964    |to_csv(value)                                                             |
3965    +--------------------------------------------------------------------------+
3966    |2,Alice,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@99c5e30f|
3967    +--------------------------------------------------------------------------+
3968    <BLANKLINE>
3969**********************************************************************
3970   1 of  18 in pyspark.sql.connect.functions.builtin.to_csv
3971***Test Failed*** 1 failures. {code}
Bug: Count bug introduced for scalar subquery when using TEMPORARY VIEW, as compared to using table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/24 05:28
Affects Version/s: 3.5.0

Using the temp view reproduces COUNT bug, returns nulls instead of 0.

With a table:
{code:java}
scala> spark.sql("""CREATE TABLE outer_table USING parquet AS SELECT * FROM VALUES
     |     (1, 1),
     |     (2, 1),
     |     (3, 3),
     |     (6, 6),
     |     (7, 7),
     |     (9, 9) AS inner_table(a, b)""")

val res6: org.apache.spark.sql.DataFrame = []

scala> spark.sql("CREATE TABLE null_table USING parquet AS SELECT CAST(null AS int) AS a, CAST(null as int) AS b ;")

val res7: org.apache.spark.sql.DataFrame = []

scala> spark.sql("""SELECT ( SELECT COUNT(null_table.a) AS aggAlias FROM null_table WHERE null_table.a = outer_table.a) FROM outer_table""").collect()

val res8: Array[org.apache.spark.sql.Row] = Array([0], [0], [0], [0], [0], [0]) {code}
With a view:

 
{code:java}
spark.sql("CREATE TEMPORARY VIEW outer_view(a, b) AS VALUES (1, 1), (2, 1),(3, 3), (6, 6), (7, 7), (9, 9);")

spark.sql("CREATE TEMPORARY VIEW null_view(a, b) AS SELECT CAST(null AS int), CAST(null as int);")

spark.sql("""SELECT ( SELECT COUNT(null_view.a) AS aggAlias FROM null_view WHERE null_view.a = outer_view.a) FROM outer_view""").collect()

val res2: Array[org.apache.spark.sql.Row] = Array([null], [null], [null], [null], [null], [null]){code}
Bug: Build and Run with Java 21
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 23/Jan/24 14:49
Affects Version/s: 3.4.3

Apache Spark supports Java 8, Java 11 (LTS) and Java 17 (LTS). The next Java LTS version is {*}21{*}.
||Version||Release Date||
|Java 21 (LTS)|19th September 2023|

Apache Spark has a release plan and Spark Code freeze along with the release branch cut details published here,
 - [https://spark.apache.org/versioning-policy.html]

Supporting new Java version is considered as a new feature which we cannot allow to backport.
Bug: IDENTIFIER clause does not work with CTE reference
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jan/24 17:51
Affects Version/s: 3.3.4

IDENTIFIER clause does not pick up CTE 

DECLARE agg = 'max';
DECLARE col = 'c1';
DECLARE tab = 'T';

WITH S(c1, c2) AS (VALUES(1, 2), (2, 3)),
      T(c1, c2) AS (VALUES ('a', 'b'), ('c', 'd'))
SELECT IDENTIFIER(agg)(IDENTIFIER(col)) FROM IDENTIFIER(tab);

[TABLE_OR_VIEW_NOT_FOUND] The table or view `T` cannot be found. Verify the spelling and correctness of the schema and catalog.

If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.

To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 3 pos 45;

'Project [unresolvedalias(expressionwithunresolvedidentifier('agg, org.apache.spark.sql.catalyst.parser.AstBuilder$$Lambda$2785/0x0000009002071490@126688a7))]

+- 'UnresolvedRelation [T], [], false
Bug: Clean up the broadcast data generated when sql execution ends
Status: Open | Priority: Major | Resolution: 
Created: 14/Jan/24 05:43
Affects Version/s: 4.0.0

Broadcast data cleaning can only rely on cleaning when GC is triggered, which may lead to a lot of waste of memory usage , and may also cause query instability if a single GC takes too long.
Actually we can clean the broadcast data generated during the execution of the sql after sql execution to reduce memory on driver or executor.
Bug: Add WindowExec SQLMetrics
Status: Open | Priority: Major | Resolution: 
Created: 09/Jan/24 22:39
Affects Version/s: 4.0.0

Currently, WindowExec Physical Operator has only spillSize SQLMetric. This jira aims to add following SQLMetrics to provide more information from WindowExec usage during query execution:
{code:java}
numOfOutputRows: Number of total output rows.
numOfPartitions: Number of processed input partitions.
numOfWindowPartitions: Number of generated window partitions.
spilledRows: Number of total spilled rows.
spillSizeOnDisk: Total spilled data size on disk.{code}
As an example use-case, WindowExec spilling behavior depends on multiple factors and it can sometime cause {{SparkOutOfMemoryError}} instead of spilling to disk so it is hard to track without SQL Metrics such as:
*1-* WindowExec creates ExternalAppendOnlyUnsafeRowArray (internal ArrayBuffer) per task (a.k.a child RDD partition)
*2-* When ExternalAppendOnlyUnsafeRowArray size exceeds spark.sql.windowExec.buffer.in.memory.threshold=4096, ExternalAppendOnlyUnsafeRowArray switches to UnsafeExternalSorter as spillableArray by moving its all buffered rows into UnsafeExternalSorter and ExternalAppendOnlyUnsafeRowArray (internal ArrayBuffer) is cleared. In this case, WindowExec starts to write UnsafeExternalSorter' s buffer (a.k.a UnsafeInMemorySorter).
*3-* UnsafeExternalSorter is being created per window partition. When UnsafeExternalSorter' buffer size exceeds spark.sql.windowExec.buffer.spill.threshold=Integer.MAX_VALUE, it starts to write to disk and get cleared all buffer (a.k.a UnsafeInMemorySorter) content. In this case, UnsafeExternalSorter will continue to buffer next records until exceeding spark.sql.windowExec.buffer.spill.threshold.

*New WindowExec SQLMetrics Sample Screenshot:*
!WindowExec SQLMetrics.png|width=257,height=152!
Bug: Language support for collations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/24 17:30
Affects Version/s: 4.0.0

Languages and localization for collations are supported by ICU library. Collation naming format is as follows:
{code:java}
<2-letter language code>[_<4-letter script>][_<3-letter country code>][_specifier_specifier...]{code}
Locale specifier consists of the first part of collation name (language + script + country). Locale specifiers need to be stable across ICU versions; to keep existing ids and names invariant we introduce golden file will locale table which should case CI failure on any silent changes.

Currently supported optional specifiers:
 * CS/CI - case sensitivity, default is case-sensitive; supported by configuring ICU collation levels
 * AS/AI - accent sensitivity; default is accent-sensitive; supported by configuring ICU collation levels
 * <unspecified>/LCASE/UCASE - case conversion performed prior to comparisons; supported by internal implementation relying on ICU locale-aware conversions

User can use collation specifiers in any order except of locale which is mandatory and must go first. There is a one-to-one mapping between collation ids and collation names defined in CollationFactory.
Bug: Introduce throwable field to mark expressions that may throw a runtime exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/24 23:50
Affects Version/s: 3.5.0

For predicate pushdown, currently it is possible that we push down a filter that ends up being evaluated on more rows than before it was pushed down (e.g. if we push the filter through a selective join). In this case, it is possible that we now evaluate the filter on a row that will cause a runtime error to be thrown, when prior to pushing this would not have happened.

If we mark certain expressions as throwable, we can avoid pushing these through joins, filters, and aggregations to avoid this.
Bug: Handle a non-existent error class
Status: Open | Priority: Major | Resolution: 
Created: 04/Jan/24 07:44
Affects Version/s: 4.0.0

Check existence of error class in Scala connect client while converting errors from proto to Spark exceptions. And when an error class doesn't exists, use some legacy error class. This will allow to avoid internal errors when the client receives non-existent error class from the connect server.
Bug: Session level config was not loaded when isolation is enable.
Status: Open | Priority: Major | Resolution: 
Created: 02/Jan/24 08:49
Affects Version/s: 3.5.0

I setup thriftserver based on v3.5.0, when I execute command, will throw this error:
{code:java}
15:10:53.400 [HiveServer2-Handler-Pool: Thread-293] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
## ignore very long stack trace ## {code}
With some debugging and analysis, I found that proxyuser should use token to access metastore, but actually uses kerberos. The direct reason is that "hive.metastore.token.signature" is lost.

In fact, we have set "hive.metastore.token.signature" to "HiveServer2ImpersonationToken" for config when construct HiveSessionImplwithUGI, and store the config in HiveSessionImplwithUGI::sessionHive and HiveSessionImplwithUGI::sessionState
When session is acquire, we should set sessionState and sessionHive to thread-level variables. Then the execution statements will use their own sessionHive and sessionState, so use the right config.

But if isolation is enable, a new SessionState and Hive will be constructed using the specified hive version. Config is not passed from HiveSessionImplwithUGI::sessionState to this SessionState. And config is not passed from HiveSessionImplwithUGI::sessionHive to new Hive. So hive.metastore.token.signature is missing.

How to fix?

For `spark.sql.hive.metastore.jars` is 'builtin', we can directly obtain the session-level config which is threadlocal variable by SessionState.get() or Hive.get().

For `spark.sql.hive.metastore.jars` is 'maven' or 'path', we will use IsolatedClientLoader to reload Hive metastore related class, It means the thread-local variable in SessionState and Hive will be missing. So we need a new structure to store threadlocal  config.
Bug: Revert S3A endpoint fixup logic of SPARK-35878
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 22/Jan/24 11:58
Affects Version/s: 3.4.3

The v2 SDK does its region resolution "differently", and the changes of SPARK-35878 actually create problems.

That PR went in to fix  a regression in Hadoop 3.3.1 which has been fixed since 3.3.2; removing it is not going to cause problems on anyone not using the 3.3.1 release, which is 3 years old and replaced by multiple follow on 3.3.x releases
Bug: Reading a non-empty Avro file with empty blocks returns 0 records
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/24 04:25
Affects Version/s: 3.5.0

When an Avro file contains empty blocks, Spark returns 0 records while "fastavro" and "avro-python-3" both read the file correctly and return records.

 

This is due to the way Spark handles empty blocks (or does not handle). Call to `hasNext` loads the next block and if that block is empty, it returns false. But instead of exiting the loop, we need to probe the next block until sync point is reached.
Bug: quoted strings in a JSON path should support ? characters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/24 17:19
Affects Version/s: 3.0.0

I think this impacts all versions of Spark after SPARK-18677, which made the operator work at all in 2.1.0/2.0.3

I comes down to
{code:java}
 name <- '.' ~> "[^\\.\\[]+".r | "['" ~> "[^\\'\\?]+".r <~ "']"{code}
[https://github.com/apache/spark/blob/01bb1b1a3dbfc68f41d9b13de863d26d587c7e2f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L79]

 

The regular expression/pattern is saying that we want a [' followed by one or more characters that are not a single quote ' or a question mark ? followed by ']. That question mark looks out of place. When I try to put in a question mark in a quoted string it fails to produce any result, but when I put the same data/path into [https://jsonpath.com/] I get a result

 

data
{code:java}
{"?":"QUESTION"} {code}
path
{code:java}
$['?'] {code}
 

I also see no tests validating that a question mark is not allowed so I suspect that it is a long standing bug.
Bug: Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/24 23:57
Affects Version/s: 3.5.0

Issue is also reported on Clickhouse's github: [https://github.com/ClickHouse/clickhouse-java/issues/1505] 
h3. Bug description



When using Scala spark to write an array of string to Clickhouse, the driver throws {{java.lang.IllegalArgumentException: Unknown data type: string}} exception.

Exception is thrown by: [https://github.com/ClickHouse/clickhouse-java/blob/aa3870eadb1a2d3675fd5119714c85851800f076/clickhouse-data/src/main/java/com/clickhouse/data/ClickHouseDataType.java#L238]

This was caused by Spark JDBC Utils tried to cast the type to lower case ({{{}String{}}} -> {{{}string{}}}).
[https://github.com/apache/spark/blob/6b931530d75cb4f00236f9c6283de8ef450963ad/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L639]
h3. Steps to reproduce
 # Create Clickhouse table with String Array field ([https://clickhouse.com/]).
 # Write data to the table with scala Spark, via Clickhouse's JDBC ([https://github.com/ClickHouse/clickhouse-java)] 
{code:java}
   // code extraction, will need to setup a Scala Spark job with clickhouse jdbc
    val clickHouseSchema = StructType(
      Seq(
        StructField("str_array", ArrayType(StringType))
      )
    )
    val data = Seq(
      Row(
        Seq("a", "b")
      )
    )

    val clickHouseDf = spark.createDataFrame(sc.parallelize(data), clickHouseSchema)
   
    val props = new Properties
    props.put("user", "default")
    clickHouseDf.write
      .mode(SaveMode.Append)
      .option("driver", com.clickhouse.jdbc.ClickHouseDriver)
      .jdbc("jdbc:clickhouse://localhost:8123/foo", table = "bar", props) {code}
h2. Fix

 - [https://github.com/apache/spark/pull/44459]