av9ash/GitBugs huggingface dataset:
Bug: Pandas circular import error in Python 3.10 
Status: Open | Priority: Major | Resolution: 
Created: 02/Apr/24 00:14
Affects Version/s: 4.0.0

{{AttributeError: partially initialized module 'pandas' has no attribute '_pandas_datetime_CAPI' (most likely due to a circular import)}}

 

The above error appears in multiple tests with Python 3.10.

Python 3.11, 3.12 and pypy3 don't have the issue.

 

See [https://github.com/apache/spark/actions/runs/8469356110/job/23208894575] for details.
Bug: Use asynchronous callback for execution cleanup
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/24 15:51
Affects Version/s: 3.5.0

Expired sessions are regularly checked and cleaned up by a maintenance thread. However, currently, this process is synchronous. Therefore, in rare cases, interrupting the execution thread of a query in a session can take hours, causing the entire maintenance process to stall, resulting in a large amount of memory not being cleared.

We address this by introducing asynchronous callbacks for execution cleanup, avoiding synchronous joins of execution threads, and preventing the maintenance thread from stalling in the above scenarios. To be more specific, instead of calling {{runner.join()}} in ExecutorHolder.close(), we set a post-cleanup function as the callback through {{{}runner.processOnCompletion{}}}, which will be called asynchronously once the execution runner is completed or interrupted. In this way, the maintenance thread won't get blocked on {{{}join{}}}ing an execution thread.
Bug: Support `three` methods of the log `concatenation` in the `structured logging framework`
Status: Open | Priority: Major | Resolution: 
Created: 02/Apr/24 07:03
Affects Version/s: 4.0.0

Allow log syntax like

- 1.log"... MDC(...) $normal_variables ..."
{code:java}
log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down. $remoteAddress"
{code}

- 2.s"... $normal_variables ..." ++ log"... MDC(...) ..."
{code:java}
log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down." ++ s"$remoteAddress"
{code}

- 3.log"... MDC(...) ..." ++ s"... $normal_variables ..."
{code:java}
s"$remoteAddress" ++ log"Driver terminated with exit code ${MDC(EXIT_CODE, exitCode)}! Shutting down."
{code}
Bug: Use simdjson-java in JSON related UDFs
Status: Open | Priority: Major | Resolution: 
Created: 10/Apr/24 17:10
Affects Version/s: 4.0.0

JSON parsing speed is important.

Right now, functions like GET_JSON_OBJECT, FROM_JSON are slow because they don't use [https://github.com/simdjson/simdjson-java]

 

We should consider adopting [https://github.com/simdjson/simdjson-java] for those UDFs.
Bug: File format of insert overwrite dir does not take effect
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 19/Apr/24 07:40
Affects Version/s: 4.0.0

File format of insert overwrite dir does not take effect.

 

Reproduce:
{code:java}
set hive.default.fileformat=parquet;

-- Not as expected, I want an orc file but written in parquet format
INSERT OVERWRITE DIRECTORY '$tablePath'
STORED AS
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
select 1;

-- Failed
select * from `orc`.`$tablePath`; {code}
Bug: Decouple Spark Go Connect Library versioning from Spark versioning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/24 21:48
Affects Version/s: 3.5.1

There is a recent discussion in Spark community for Spark Operator version naming convention. People like to use version independent of Spark versions. That applies to Spark Connect Go Client as well. Better to start from v1.
Bug: Nullability after join not respected in UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/24 06:57
Affects Version/s: 3.4.3

{code:java}
val ds1 = Seq(1).toDS()
val ds2 = Seq[Int]().toDS()
val f = udf[(Int, Option[Int]), (Int, Option[Int])](identity)
ds1.join(ds2, ds1("value") === ds2("value"), "outer").select(f(struct(ds1("value"), ds2("value")))).show()
ds1.join(ds2, ds1("value") === ds2("value"), "outer").select(struct(ds1("value"), ds2("value"))).show() {code}
outputs
{code:java}
+---------------------------------------+
|UDF(struct(value, value, value, value))|
+---------------------------------------+
|                                 {1, 0}|
+---------------------------------------+

+--------------------+
|struct(value, value)|
+--------------------+
|           {1, NULL}|
+--------------------+ {code}

So when the result is passed to UDF the null-ability after the the join is not respected and we incorrectly end up with a 0 value instead of a null/None value.
Bug: [CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/24 08:04
Affects Version/s: 3.5.0

When the server closes a session, usually after a cluster restart, the client is unaware of this until it receives an error.

Once it does so, there is no way for the client to create a new session since the stale sessions are still recorded as default and active sessions.

The only solution currently is to restart the Python interpreter on the client, or to reach into the session builder and change the active or default session.
Bug: Memory leak when interrupting shuffle write using zstd compression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/24 05:58
Affects Version/s: 3.2.0

When spark.sql.execution.interruptOnCancel=true and spark.io.compression.codec=zstd, a memory leak was found when tasks were cancelled at specific times. The reason for this is that cancelling a task interrupts the shuffle write, which then calls org.apache.spark.storage.DiskBlockObjectWriter#closeResources. this process then only closes the ManualCloseOutputStream, which is wrapped with this ZstdInputStreamNoFinalizer will not be closed. Moreover, ZstdInputStreamNoFinalizer doesn't implement Finalizer so it won't be reclaimed by GC automatically.
Bug: [CONNECT][PYTHON] Session not found error should automatically retry during reattach
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/24 05:03
Affects Version/s: 3.4.3

When an OPERATION_NOT_FOUND error is raised and no prior responses were received, the client retries the ExecutePlan RPC: [https://github.com/apache/spark/blob/e6217c111fbdd73f202400494c42091e93d3041f/python/pyspark/sql/connect/client/reattach.py#L257]

 

Another error SESSION_NOT_FOUND should follow the same logic.
Bug: SPJ: Support Transfrom Expressions for One Side Shuffle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/24 19:09
Affects Version/s: 3.4.3

SPARK-41471 allowed Spark to shuffle just one side and still conduct SPJ, if the other side is KeyGroupedPartitioning.  However, the support was just for a KeyGroupedPartition without any partition transform (day, year, bucket).  It will be useful to add support for partition transform as well, as there are many tables partitioned by those transforms.
Bug: Restrict CAST expression for collations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/24 08:02
Affects Version/s: 4.0.0

Current state of code allows for calls like CAST(1 AS STRING COLLATE UNICODE). We want to restrict CAST expression to only be able to cast to default collation string, and to only allow COLLATE expression to produce explicitly collated strings.
Bug: DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/24 21:24
Affects Version/s: 4.0.0

https://github.com/apache/spark/pull/44263 converted `IllegalStateException` to `InternalError` when a unit for DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd is not from a sanctioned list.

Originally incorrect unit should have been caught in parser before an expression is constructed. However, PySpark introduced PythonSQLUtils that creates expressions without validation. Since then the `INTERNAL_ERROR` is incorrect error class and should be converted to execution error with correct error class instead.
Bug: SQLQueryTestSuite flaky case due to view name conflict
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 14:08
Affects Version/s: 4.0.0

In SQLQueryTestSuite, the test case "Test logic for determining whether a query is semantically sorted" can sometimes fail with an error

{{Cannot create table or view `main`.`default`.`t1` because it already exists.}}

if run concurrently with other sql test cases that also create tables with the same name.
Bug: Add schema evolution options to views 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/24 17:59
Affects Version/s: 4.0.0

We want to provide the ability for views to react to changes in the query resolution in manners differently than just failing the view.
For example we want the view to be able to compensate for type changes by casting the query result to the view column types.
Or to adopt any type of column arity changes into a view.
Bug: Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/24 15:33
Affects Version/s: 4.0.0

While building the DataFrame step by step, each time a new DataFrame is generated with an empty schema, which is lazily computed on access. However, if a user's code frequently accesses the schema of these new DataFrames using methods such as `df.columns`, it will result in a large number of Analyze requests to the server. Each time, the entire plan needs to be reanalyzed, leading to poor performance, especially when constructing highly complex plans.

Now, by introducing plan cache in SparkConnectPlanner, we aim to reduce the overhead of repeated analysis during this process. This is achieved by saving significant computation if the resolved logical plan of a subtree of can be cached.

A minimal example of the problem:
{code:java}
import pyspark.sql.functions as F
df = spark.range(10)
for i in range(200):
  if str(i) not in df.columns: # <-- The df.columns call causes a new Analyze request in every iteration
    df = df.withColumn(str(i), F.col("id") + i)
df.show() {code}
With this patch, the performance of the above code improved from ~110s to ~5s.
Bug: Nested field's nullable value could be invalid after extracted using GetStructField
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Apr/24 05:03
Affects Version/s: 3.4.2

I've got error when append to table using DataFrameWriterV2.

The error was occured in TableOutputResolver.checkNullability. This error occurs when the data type of the schema is the same, but the order of the fields is different.

I found that GetStructField.nullable returns unexpected result.
{code:java}
override def nullable: Boolean = child.nullable || childSchema(ordinal).nullable {code}
Even if nested field has not nullability attribute, it returns true when parent struct has nullability attribute.
||Parent nullability||Child nullability||Result||
|true|true|true|
|{color:#ff0000}true{color}|{color:#ff0000}false{color}|{color:#ff0000}true{color}|
|{color:#172b4d}false{color}|{color:#172b4d}true{color}|{color:#172b4d}true{color}|
|false|false|false|

 

I think the logic should be changed to get just child's nullability, because both of parent and child should be nullable to be considered nullable.

 
{code:java}
override def nullable: Boolean = childSchema(ordinal).nullable  {code}
 

 

 

I want to check current logic is reasonable, or my suggestion can occur other side effect.
Bug: InjectRuntimeFilter for multi-level join should check child join type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/24 08:32
Affects Version/s: 4.0.0

{code:java}
with 
refund_info as (
    select
        loan_id,
        1 as refund_type
    from
        default.table_b
    where grass_date = '2024-04-25'
       
),
next_month_time as (
    select /*+ broadcast(b, c) */
         loan_id
        ,1 as final_repayment_time
    FROM default.table_c
    where grass_date = '2024-04-25'
)
select
        a.loan_id
        ,c.final_repayment_time
        ,b.refund_type    from
        (select
             loan_id
        from
            default.table_a2
        where grass_date = '2024-04-25'
        select
            loan_id
        from
            default.table_a1
        where grass_date = '2024-04-24' 
        ) a
    left join
        refund_info b
    on a.loan_id = b.loan_id
    left join
        next_month_time c
    on a.loan_id = c.loan_id
;
 {code}
!image-2024-04-28-16-38-37-510.png|width=899,height=201!

 

In this query, it inject table_b as table_c's runtime filter, but table_b join condition is LEFT OUTER, causing table_c missing data.

Caused by 

InjectRuntimeFilter.extractSelectiveFilterOverScan(), when handle join, since left plan is a UNION< result is NONE, then zip l/r keys to extract from right. Then cause this issue

!image-2024-04-28-16-41-08-392.png|width=883,height=706!
Bug: Implement SimpleDataSourceStreamReader for python streaming data source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/24 06:56
Affects Version/s: 3.5.1

 SimpleDataSourceStreamReader is a simplified version of the DataStreamReader interface.
 # It doesn’t require developers to reason about data partitioning.
 # It doesn’t require getting the latest offset before reading data.

There are 3 functions that needs to be defined 

1. Read data and return the end offset.

_def read(self, start: Offset) -> (Iterator[Tuple], Offset)_

2. Read data between start and end offset, this is required for exactly once read.

_def read2(self, start: Offset, end: Offset) -> Iterator[Tuple]_

3. initial start offset of the streaming query.

def initialOffset() -> dict

Implementation: Wrap the SimpleDataSourceStreamReader instance in a DataSourceStreamReader internally and make the prefetching and caching transparent to the data source developer. The record prefetched in python process will be sent to JVM as arrow record batches.
Bug: Support Chaining Stateful Operators in TransformWithState
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/24 17:13
Affects Version/s: 4.0.0

This issue tracks adding support to chain stateful operators after the Arbitrary State API, transformWithState. In order to support chaining, we need to allow the user to specify the new eventTimeColumn in the output from StatefulProcessor. Any watermark evaluation expressions downstream after transformWithState would use the user specified eventTimeColumn.
Bug: Do not display the splits parameter in Rang
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/24 08:35
Affects Version/s: 3.5.1

[SQL]

explain extended select * from range(0, 4);

plan

== Parsed Logical Plan ==

'Project [*]

+- 'UnresolvedTableValuedFunction [range], [0, 4]

 

== Analyzed Logical Plan ==

id: bigint

Project [id#11L|#11L]

+- Range (0, 4, step=1, splits=None)

 

== Optimized Logical Plan ==

Range (0, 4, step=1, splits=None)

 

== Physical Plan ==

*(1) Range (0, 4, step=1, splits=1)

 

The splits parameter will only be set during the physical execution phase. But it is also displayed in the logical execution phase as None, which is not very user-friendly. Showing the physical execution plan can help users.
Bug: try_add/try_multiply should not be semantic equal to add/multiply
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/24 02:54
Affects Version/s: 4.0.0

In the current implementation, the following code will return true
{code:java}
val l1 = Literal(1)
val l2 = Literal(2)
val l3 = Literal(3)
val expr1 = Add(Add(l1, l2), l3)
val expr2 = Add(Add(l2, l1, EvalMode.TRY), l3)
expr1.semanticEquals(expr2) {code}
The same applies to Multiply.

When creating MultiCommutativeOp for Add/Multiply, we should ensure all the evalMode are consistent.
Bug: Reduce memory pressure of TreeNode `tags` variable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/24 18:14
Affects Version/s: 4.0.0

By default, the TreeNode `tags` variable is initialized to a `mutable.Map.empty`, which defaults to a map with an internal 16 element hashtable array. For plans with a high number of TreeNodes with empty tags, this adds unnecessary driver memory pressure, especially for large expression trees. Reduce the memory pressure by lazily initializing this variable.
Bug: Enable structured logging in log4j2.properties.template and update `configuration.md`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/24 16:57
Affects Version/s: 4.0.0

# rename the current log4j2.properties.template as log4j2.properties.pattern-layout-template
 # Enable structured logging in log4j2.properties.template
 # Update `configuration.md` on how to configure logging
Bug: SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/24 22:27
Affects Version/s: 3.4.3

If spark.sql.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled is true, then SPJ no longer triggers if there are more join keys than partition keys.  It is triggered only if join keys is equal to , or less than, partition keys.

 

We can relax this constraint, as this case was supported if the flag is not enabled.
Bug: Fix a bug in try_divide function when with decimals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/24 22:50
Affects Version/s: 3.5.2

Binary Arithmetic operators should include the evalMode during makeCopy. Otherwise, the following query will throw DIVIDE_BY_ZERO error instead of returning null

 
{code:java}
SELECT try_divide(1, decimal(0)); {code}
This is caused from the rule DecimalPrecision:
{code:java}
case b @ BinaryOperator(left, right) if left.dataType != right.dataType =>
  (left, right) match {
 ...
    case (l: Literal, r) if r.dataType.isInstanceOf[DecimalType] &&
        l.dataType.isInstanceOf[IntegralType] &&
        literalPickMinimumPrecision =>
      b.makeCopy(Array(Cast(l, DataTypeUtils.fromLiteral(l)), r)) {code}
Bug: XML: Stop ignoring CDATA within rows. 
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Apr/24 21:19
Affects Version/s: 4.0.0

This change ignores CDATA within row tags as well as outside of it. We should only ignore CDATA found outside of row tags as they are considered data within the row.
[https://github.com/apache/spark/pull/45487]

 

NOTE: With the current parser implementation, after not ignoring CDATA elements within row tags there remains the edge case of a matching closing row tag within CDATA which will be parsed as a valid end tag. 
Example:
{code:java}
<row> <![CDATA[ </row> ]]> {code}
after no longer ignoring CDATA within rows, the closing tag in the example above will be matched by the parser which is incorrect.
Bug: Ensure the same hash partitioning scheme/hash function is used across batches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/24 23:18
Affects Version/s: 3.5.1

To really make sure that any changes to hash function / partitioner in Spark doesn’t cause logical correctness issues in existing running streaming queries, we should add a new unit test, to ensure hash function stability is maintained.
Bug: InternalRowComparableWrapper should cache rowOrdering to improve performace
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/24 12:30
Affects Version/s: 3.4.3

InternalRowComparableWrapper recreates row ordering for each output partition when SPJ is enabled. The row ordering is generated via codegen which is quite expensive and the output partitions might be quite large for production table such as hundreds of thousands partitions. We encountered this issue when applying SPJ with multiple large Iceberg tables and the plan phase took tens of minutes to complete.

Attaching a screenshot to provide related stack trace:
  !screenshot-1.png! 

A simple fix for this would be caching the rowOrdering for InternalRowComparableWrapper as the datatype of the InternalRow is immutable
Bug: SparkSession.createDataFrame should warn for unsupported options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/24 04:25
Affects Version/s: 4.0.0

{code}
spark.createDataFrame([1,2,3], verifySchema=True)
{code}

and

{code}
spark.createDataFrame([1,2,3], samplingRatio=0.5)
{code}

Do not work with Spark Connect.
Bug: CREATE TABLE AS SELECT changes behaviour in SPARK 3.4.0
Status: Open | Priority: Major | Resolution: 
Created: 23/Apr/24 17:20
Affects Version/s: 3.4.0

SPARK-41859 changes the behaviour for `CREATE TABLE AS SELECT ...` from OVERWRITE to APPEND when {{spark.sql.legacy.allowNonEmptyLocationInCTAS}} is set to {{{}true{}}}:

{{drop table if exists test_table;}}
{{create table test_table location '/tmp/test_table' stored as parquet as select 1 as col union all select 2 as col;}}
{{drop table if exists test_table;}}
{{create table test_table location '/tmp/test_table' stored as parquet as select 3 as col union all select 4 as col;}}
{{select * from test_table;}}

This produces {3, 4} in Spark <3.4.0 and {1, 2, 3, 4} in Spark 3.4.0 and later. This is a silent change in {{spark.sql.legacy.allowNonEmptyLocationInCTAS}} behaviour which introduces wrong results in the user application
Bug: Support Generated Column expressions that are `RuntimeReplaceable`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/24 00:45
Affects Version/s: 4.0.0

Currently, default columns that have a default of a `RuntimeReplaceable` expression fails.

This is because the `AlterTableCommand` constant folds before replacing expressions with the actual implementation. For example:
```
sql(s"CREATE TABLE t(v VARIANT DEFAULT parse_json('1')) USING PARQUET")
sql("INSERT INTO t VALUES(DEFAULT)")
```

fails because `parse_json` is `RuntimeReplaceable` and is evaluated before the analyzer inserts the correct expression into the plan

This is especially important for Variant types because literal variants are difficult to create - `parse_json` will likely be used the majority of the time.
Bug: ANSI Mode: Trim binary values shall result binary type instead of string type 
Status: Open | Priority: Major | Resolution: 
Created: 19/Apr/24 03:20
Affects Version/s: 4.0.0

```

<trim function> when applied to binary strings is identical in syntax (apart from the default <trim character>) and semantics to the corresponding operation on character strings except that the returned value is a binary string.

```
Bug: Improve UTF8_BINARY_LCASE collation comparison performance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/24 10:17
Affects Version/s: 4.0.0

Current collation [benchmarks|https://github.com/apache/spark/blob/e9f204ae93061a862e4da52c128eaf3512a66c7b/sql/core/benchmarks/CollationBenchmark-results.txt] indicate that UTF8_BINARY_LCASE collation comparisons are order of magnitude slower (~7-10x) than plain binary comparisons. Improve the performance by optimizing lowercase comparison function for UTF8String instances instead of performing full lowercase conversion before binary comparison.
Bug: SQLServer does not support 1 and 0 as boolean values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/24 15:48
Affects Version/s: 3.4.3

Sometimes in Spark, when a column that is generated as CASE WHEN structure is used in comparison filter, output of optimized plan will be: CASE WHEN expression THEN (1 or 0)... which is not supported in SQLServer. Exception is thrown by SQLServer that a "non-boolean expression is given when boolean was expected". For now, we should not support CASE WHEN pushdown in SQLServer.