av9ash/GitBugs huggingface dataset:
Bug: Built-in functions for Zstd compression and decompression
Status: Open | Priority: Major | Resolution: 
Created: 20/May/24 14:26
Affects Version/s: 4.0.0

Some users are using UDFs for Zstd compression and decompression, which results in poor performance. If we provide native functions, the performance will be improved by compressing and decompressing just within the JVM.

 

Now, we are introducing three new built-in functions:
{code:java}
zstd_compress(input: binary [, level: int [, steaming_mode: bool]])

zstd_decompress(input: binary)

try_zstd_decompress(input: binary)
{code}
where
 * `input`: The binary value to compress or decompress.
 * `level`: Optional integer argument that represents the compression level. The compression level controls the trade-off between compression speed and compression ratio. The default level is 3. Valid values: between 1 and 22 inclusive
 * `streaming_mode`: Optional boolean argument that represents whether to use streaming mode to compress. 

Examples:
{code:sql}
> SELECT base64(zstd_compress(repeat("Apache Spark ", 10)));
  KLUv/SCCpQAAaEFwYWNoZSBTcGFyayABABLS+QU=
> SELECT base64(zstd_compress(repeat("Apache Spark ", 10), 3, true));
  KLUv/QBYpAAAaEFwYWNoZSBTcGFyayABABLS+QUBAAA=
> SELECT string(zstd_decompress(unbase64("KLUv/SCCpQAAaEFwYWNoZSBTcGFyayABABLS+QU=")));
  Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark Apache Spark
> SELECT zstd_decompress(zstd_compress("Apache Spark"));
  Apache Spark
> SELECT try_zstd_decompress("invalid input")
  NULL
{code}
These three built-in functions are also available in Python and Scala.
Bug: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.
Status: Open | Priority: Major | Resolution: 
Created: 31/May/24 08:25
Affects Version/s: 3.3.0

The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

for example: The partition field of table A or B is part_dt.  

select a.part_dt,{*}b.part_dt{*}
from a left join b on b.part_dt = a.part_dt 
where a.part_dt in ('2024-05-30','2024-05-31')

During the generation of Dynamic Partition Pruning (DPP), the tree will be traversed recursively more times. 

24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with: part_dt#219 IN (2024-05-30,2024-05-31)
24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with: part_dt#223 IN (2024-05-30,2024-05-31),isnotnull(part_dt#223),dynamicpruning#234 [part_dt#223|#223]
24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with: part_dt#219 IN (2024-05-30,2024-05-31)

The last one is extra：24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with: part_dt#219 IN (2024-05-30,2024-05-31)

When more partitions meet the condition, it will increase by 2 to the power of n, then divided by 2.

for example:

select a.part_dt{*},b.part_dt{*},c.part_dt
from a 
left join b on b.part_dt = a.part_dt 
left join c on c.part_dt = a.part_dt
where a.part_dt in ('2024-05-30','2024-05-31')

result:

24/05/31 16:29:19 INFO ExecuteStatement: Execute in full collect mode
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#249 IN (2024-05-30,2024-05-31)
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#253 IN (2024-05-30,2024-05-31),isnotnull(part_dt#253),dynamicpruning#261 [part_dt#253|#253]
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#257 IN (2024-05-30,2024-05-31),isnotnull(part_dt#257),dynamicpruning#262 [part_dt#257|#257]
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#249 IN (2024-05-30,2024-05-31)
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#249 IN (2024-05-30,2024-05-31)
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#253 IN (2024-05-30,2024-05-31),isnotnull(part_dt#253),dynamicpruning#261 [part_dt#253|#253]
24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with: part_dt#249 IN (2024-05-30,2024-05-31)

The last four lines are all superfluous.

When more than 10 conditions are met, it will cause a memory overflow when generating DPP, and the generated plan will be about 400 times larger.
Bug: JSON objects should not be modified when read as STRING
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/24 20:40
Affects Version/s: 4.0.0

Currently, when reading a JSON like this:


bq. {"a": {"b": -999.99999999999999999999999999999999995}}


With the schema:

bq. a STRING


Spark will yield a result like this:


bq. {"b": -1000.0}


This is due to how we convert a non-string value to a string in JacksonParser
Bug: Fix nested array to respect legacy conf of inferArrayTypeFromFirstElement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/24 02:39
Affects Version/s: 4.0.0

{code}
>>> spark.conf.set("spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled", True)
>>> spark.createDataFrame([[[[1, "a"]]]])
DataFrame[_1: array<array<string>>]
{code}

should infer it as an integer of array
Bug: Late rewrite of PlanWithUnresolvedIdentifier
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/24 13:23
Affects Version/s: 4.0.0

PlanWithUnresolvedIdentifier is rewritten later in analysis which causes rules like 
SubstituteUnresolvedOrdinals to miss the new plan. This causes following queries to fail:

{code:java}
create temporary view identifier('v1') as (select my_col from (values (1), (2), (1) as (my_col)) group by 1);
--
cache table identifier('t1') as (select my_col from (values (1), (2), (1) as (my_col)) group by 1); 
--
create table identifier('t2') as (select my_col from (values (1), (2), (1) 
as (my_col)) group by 1);
insert into identifier('t2') select my_col from (values (3) as (my_col)) group by 1; {code}
Bug: Modify the description of whether dynamic partitioning is enabled in the “  Stage Level Scheduling Overview”
Status: Open | Priority: Major | Resolution: 
Created: 09/May/24 03:50
Affects Version/s: 4.0.0

“  Stage Level Scheduling Overview ” in running-on-yarn and  running-on-kubernetes

The description of dynamic partitioning is inconsistent with the code implementation verification.

In running-on-yarn 

'
 * When dynamic allocation is disabled: It allows users to specify different task resource requirements at the stage level and will use the same executors requested at startup.

'

But the  implementation is：

Class：ResourceProfileManager

Fuc：isSupported

private[spark] def isSupported(rp: ResourceProfile): Boolean = {
assert(master != null)
if (rp.isInstanceOf[TaskResourceProfile] && !dynamicEnabled) {
if ((notRunningUnitTests || testExceptionThrown) &&
!(isStandaloneOrLocalCluster || isYarn || isK8s)) {
throw new SparkException("TaskResourceProfiles are only supported for Standalone, " +
"Yarn and Kubernetes cluster for now when dynamic allocation is disabled.")
}
}

 

The judgment of this code is that it does not support TaskResourceProfile in Yarn and k8s when dynamic partitioning is closed.

 

The description in the document does not match, so the document needs to be modified.
Bug: Support configuring limit control for SQL to use maximum cores
Status: Open | Priority: Major | Resolution: 
Created: 23/May/24 04:16
Affects Version/s: 3.5.1

When there is a long-running shared Spark SQL cluster, there may be a situation where a large SQL occupies all the cores of the cluster, affecting the execution of other SQLs. Therefore, it is hoped that there is a configuration that can limit the maximum cores used by SQL.
Bug: Support push down char and varchar predicates to Hive Metastore
Status: Open | Priority: Major | Resolution: 
Created: 11/May/24 09:25
Affects Version/s: 3.5.1

Hive Metastore supports {{char}} and {{varchar}} types in partition filter since [HIVE-26661|https://issues.apache.org/jira/browse/HIVE-26661], so we can support it in Spark side.
Bug: [M0] Spark Connect investigation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 08:36
Affects Version/s: 4.0.0

Some notebook modes, VS code extension, etc. execute SQL commands through Spark Connect.

We need to:
- Figure out exceptions that we are getting in Spark Connect stack for SQL scripts.

- Understand the Spark Connect stack better, so we can more easily propose design for new API(s) we are going to introduce in the future.

 

For more details, design doc can be found in parent Jira item.
Bug: Nested pythonUDF in groupBy and aggregate result in Binding Exception 
Status: Open | Priority: Major | Resolution: 
Created: 16/May/24 15:05
Affects Version/s: 3.3.2

Steps to Reproduce 

1. Data creation
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, LongType, TimestampType, StringType
from datetime import datetime

# Define the schema
schema = StructType([
    StructField("col1", LongType(), nullable=True),
    StructField("col2", TimestampType(), nullable=True),
    StructField("col3", StringType(), nullable=True)
])

# Define the data
data = [
    (1, datetime(2023, 5, 15, 12, 30), "Discount"),
    (2, datetime(2023, 5, 16, 16, 45), "Promotion"),
    (3, datetime(2023, 5, 17, 9, 15), "Coupon")
]

# Create the DataFrame
df = spark.createDataFrame(data, schema)
df.createOrReplaceTempView("temp_offers")

# Query the temporary table using SQL
# DISTINCT required to reproduce the issue. 
testDf = spark.sql("""
                    SELECT DISTINCT 
                    col1,
                    col2,
                    col3 FROM temp_offers
                    """) {code}
2. UDF registration 
{code:java}
import pyspark.sql.functions as F 
import pyspark.sql.types as T

#Creating udf functions 
def udf1(d):
    return d

def udf2(d):
    if d.isoweekday() in (1, 2, 3, 4):
        return 'WEEKDAY'
    else:
        return 'WEEKEND'

udf1_name = F.udf(udf1, T.TimestampType())
udf2_name = F.udf(udf2, T.StringType()) {code}
3. Adding UDF in grouping and agg
{code:java}
groupBy_cols = ['col1', 'col4', 'col5', 'col3']
temp = testDf \
  .select('*', udf1_name(F.col('col2')).alias('col4')).select('*', udf2_name('col4').alias('col5')) 

result = (temp.groupBy(*groupBy_cols).agg(F.countDistinct('col5').alias('col6'))){code}
4. Result
{code:java}
result.show(5, False) {code}
*We get below error*
{code:java}
An error was encountered:
An error occurred while calling o1079.showString.
: java.lang.IllegalStateException: Couldn't find pythonUDF0#1108 in [col1#978L,groupingPythonUDF#1104,groupingPythonUDF#1105,col3#980,count(pythonUDF0#1108)#1080L]
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
	at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73) {code}
Bug: Add data write time metric to FileFormatDataWriter/BasicWriteJobStatsTracker
Status: Open | Priority: Major | Resolution: 
Created: 23/May/24 06:03
Affects Version/s: 4.0.0

For FileFormatDataWriter we currently record metrics of "task commit time" and "job commit time" in `org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker#metrics`. We may also record the time spent on "data write" (together with the time spent on producing records from the iterator), which is usually one of the major parts of the total duration of a writing operation. It helps us identify the bottleneck and time skew, and also the generic performance tuning.
Bug: Disable a flaky `SparkSessionE2ESuite.interrupt tag` test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/24 03:22
Affects Version/s: 4.0.0

- https://github.com/apache/spark/actions/runs/8962353911/job/24611130573 (Master, 5/5)
- https://github.com/apache/spark/actions/runs/8948176536/job/24581022674 (Master, 5/4)
Bug: Stop am retry, in situations where some errors and retries may not be successful
Status: Open | Priority: Major | Resolution: 
Created: 16/May/24 13:48
Affects Version/s: 4.0.0

In yarn cluster mode, spark.yarn.maxAppAttempts will be configured. In our production environment, it is configured as 2 If the first execution fails, AM will retry. However, in some scenarios, even attempting a second task may fail.

For example:

org. apache. park. SQL AnalysisException: Table or view not found: test.testxxxx_xxxxx; Line 1 pos 14;
Project
+-Unresolved Relationship [bigdata_qa, testxxxxx_xxxxx], [], false

 


Other example:
Caused by: org. apache. hadoop. hdfs. protocol NSQuotaExceededException: The NameSpace quota (directories and files) of directory/tmp/xxx_file/xxxx is exceeded: quota=1000000 file count=1000001


Would it be more appropriate to try capturing these exceptions and stopping retry?
Bug: Add TCP mode to StatsdSink
Status: Open | Priority: Major | Resolution: 
Created: 16/May/24 03:07
Affects Version/s: 4.0.0

Currently, the StatsdSink in Spark supports UDP mode only, which is the default mode of StatsD. However, in real production environments, we often find that a more reliable transmission of metrics is needed to avoid metrics lose in high-traffic systems.

 

TCP mode is already supported by Statsd: [https://github.com/statsd/statsd/blob/master/docs/server.md]

Prometheus' statsd_exporter: [https://github.com/prometheus/statsd_exporter] 

and also many other Statsd-based metrics proxies/receivers.
Bug: Invalid previous reader checks in Vectorized DELTA_BYTE_ARRAY parquet decoder
Status: Open | Priority: Major | Resolution: 
Created: 10/May/24 12:31
Affects Version/s: 3.2.0

The vectorized DELTA_BYTE_ARRAY Parquet decoder can cause read failures when reading columns with varying page encodings and if some pages are encoded using DELTA_BYTE_ARRAY.

Same bug existed in parquet-mr reader but was fixed 3 months ago. There is no separate bug fix commit, it was silently fixed along with other changes. https://github.com/apache/parquet-mr/blob/c241170d9bc2cd8415b04e06ecea40ed3d80f64d/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnReaderBase.java#L732
Bug: [M0] Checks for variable declarations
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 20/May/24 12:52
Affects Version/s: 4.0.0

Add checks to parser (visitCompoundBodyImpl() in AstBuilder) for variable declarations, based on a passed-in flag:
 * Variable can be declared only at the beginning of the compound.
 * Variable cannot be declared in a compound at all - throw exception.

 

For more details, design doc can be found in parent Jira item.
Bug: [M0] Support for IF ELSE statement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 12:55
Affects Version/s: 4.0.0

Add support for IF ELSE statements to SQL scripting parser & interpreter:
 * IF
 * IF / ELSE
 * IF / ELSE IF / ELSE

 

For more details, design doc can be found in parent Jira item.
Bug: CSV parsing failure with char/varchar type columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/24 07:46
Affects Version/s: 3.5.1

CSV table containing char and varchar columns will result in the following error when selecting from the CSV table:
{code:java}
java.lang.IllegalArgumentException: requirement failed: requiredSchema (struct<id:int,name:string>) should be the subset of dataSchema (struct<id:int,name:string>).
    at scala.Predef$.require(Predef.scala:281)
    at org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:56)
    at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.$anonfun$buildReader$2(CSVFileFormat.scala:127)
    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:155)
    at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:140)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:231)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:293)
    at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125){code}
The reason for the error is that the StringType columns in the dataSchema and requiredSchema of UnivocityParser are not consistent. It is due to the metadata contained in the StringType StructField of the dataSchema, which is missing in the requiredSchema. We need to retain the metadata when resolving schema.
Bug: Streaming query listener not working in 3.5 client <> 4.0 server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 10:11
Affects Version/s: 4.0.0

{code}
======================================================================
ERROR [1.488s]: test_listener_events (pyspark.sql.tests.connect.streaming.test_parity_listener.StreamingListenerParityTests.test_listener_events)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/tests/connect/streaming/test_parity_listener.py", line 53, in test_listener_events
    self.spark.streams.addListener(test_listener)
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/streaming/query.py", line 244, in addListener
    self._execute_streaming_query_manager_cmd(cmd)
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/streaming/query.py", line 260, in _execute_streaming_query_manager_cmd
    (_, properties) = self._session.client.execute_command(exec_cmd)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/client/core.py", line 982, in execute_command
    data, _, _, _, properties = self._execute_and_fetch(req)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/client/core.py", line 1283, in _execute_and_fetch
    for response in self._execute_and_fetch_as_iterator(req):
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/client/core.py", line 1264, in _execute_and_fetch_as_iterator
    self._handle_error(error)
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/client/core.py", line 1503, in _handle_error
    self._handle_rpc_error(error)
  File "/home/runner/work/spark/spark-3.5/python/pyspark/sql/connect/client/core.py", line 1539, in _handle_rpc_error
    raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.io.EOFException) 
----------------------------------------------------------------------
{code}
Bug: [M0] Fix SET behavior for scripts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/24 10:11
Affects Version/s: 4.0.0

By standard, SET is used to set variable value in SQL scripts.

On our end, SET is configured to work with some Hive configs, so the grammar is a bit messed up and for that reason it was decided to use SET VAR instead of SET to work with SQL variables.

This is not by standard and we should figure out the way to be able to use SET for SQL variables and forbid setting of Hive configs from SQL scripts.

 

For more details, design doc can be found in parent Jira item.
Bug: [M0] Support for exceptions thrown from parser/interpreter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 13:02
Affects Version/s: 4.0.0

In general, add support for SQL scripting related exceptions.
By the time someone starts working on this item, some exception support might already exist - check if it needs refactoring.

 

Have in mind that for some (all?) exceptions we might need to know which line(s) in the script are responsible for it.
Bug: Add config to switch between client side listener and server side listener 
Status: Open | Priority: Major | Resolution: 
Created: 02/May/24 19:28
Affects Version/s: 4.0.0

We are moving the implementation of Streaming Query Listener from server to client. For clients already running client side listener, to prevent regression, we should add a config to let them decide what type of listener the user want to use.

 

This is only added to 3.5.x published versions. For 4.0 and upwards we only use client side listener.
Bug: Wrap empty relation propagation in a dedicated node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/24 22:43
Affects Version/s: 4.0.0

Currently we replace with a LocalTableScan in case of empty relation propagation, which lost the information about the original query plan and make it less human readable. The idea is to create a dedicated `EmptyRelation` node which is a lead node but wraps the original query plan inside.
Bug: Enable reflect expressions with collated strings
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/24 07:21
Affects Version/s: 4.0.0

As a movement to collated world, we need to make sure all expressions are supported by appropriate collations. Using expression walker testing `CallMethodViaReflection` was found to be erroneous. This expression is used as a replacement for all reflection methods and needs to be improved. This ticket needs to update methods of this expression. Relevant code could be found in these files: `CallMethodViaReflection.scala`, `TypeCoercion.scala`, `AnsiTypeCoercion.scala`, `CollationTypeCasts.scala`, and for testing `Collation*Suite.scala` files.
Bug: [M0] Parser support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/24 12:27
Affects Version/s: 4.0.0

Implement parse for SQL scripting with all supporting changes for upcoming interpreter implementation and future extensions of the parser:
 * Parser - support only compound statements
 * Parser testing

 

For more details, design doc can be found in parent Jira item.
Bug: Preserve nulls in map columns in PyArrow Tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/24 07:08
Affects Version/s: 3.5.1

Because of a limitation in PyArrow, when PyArrow Tables containing MapArray columns with nested fields or timestamps are passed to {{{}spark.createDataFrame(){}}}, null values in the MapArray columns are replaced with empty lists.

The PySpark function where this happens is {{{}pyspark.sql.pandas.types._check_arrow_array_timestamps_localize{}}}.

Also see [https://github.com/apache/arrow/issues/41684].

See the skipped tests and the TODO mentioning SPARK-48302.

[Update] A fix for this has been implemented in PyArrow in [https://github.com/apache/arrow/pull/41757] by adding a {{mask}} argument to {{{}pa.MapArray.from_arrays{}}}. This will be released in PyArrow 17.0.0. Since older versions of PyArrow (which PySpark will still support for a while) won't have this argument, we will need to do a check like:

{{LooseVersion(pa.\_\_version\_\_) >= LooseVersion("17.0.0")}}

or

{{from inspect import signature}}
{{"mask" in signature(pa.MapArray.from_arrays).parameters}}

and only pass {{mask}} if that's true.
Bug: Add E2E test for DropDuplicateWithinWatermark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/24 07:22
Affects Version/s: 4.0.0

Currently we do not have a e2e test for DropDuplicateWithinWatermark, we should add one. We can simply use one of the test written in Scala here (with the testStream API) and replicate it to python:

[https://github.com/apache/spark/commit/0e9e34c1bd9bd16ad5efca77ce2763eb950f3103]

 

The change should happen in [https://github.com/apache/spark/blob/eee179135ed21dbdd8b342d053c9eda849e2de77/python/pyspark/sql/tests/streaming/test_streaming.py#L29]

 

so we can test it in both connect and non-connect.

 

Test with:

```
python/run-tests --testnames pyspark.sql.tests.streaming.test_streaming
python/run-tests --testnames pyspark.sql.tests.connect.streaming.test_parity_streaming
```