av9ash/GitBugs huggingface dataset:
Bug: Enable FrameParityBinaryOpsTests.test_binary_operator_multiply
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 09/Jun/23 02:17
Affects Version/s: 3.5.0


{code}
======================================================================
ERROR [4.378s]: test_binary_operator_multiply (pyspark.pandas.tests.connect.computation.test_parity_binary_ops.FrameParityBinaryOpsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/tests/computation/test_binary_ops.py", line 169, in test_binary_operator_multiply
    self.assert_eq(psdf["a"] * psdf["b"], pdf["a"] * pdf["b"])
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/base.py", line 325, in __mul__
    return self._dtype_op.mul(self, other)
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/data_type_ops/string_ops.py", line 78, in mul
    return column_op(SF.repeat)(left, right)
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/base.py", line 222, in wrapper
    scol = f(
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/spark/functions.py", line 94, in repeat
    n = _to_java_column(n) if isinstance(n, Column) else _create_column_from_literal(n)
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/column.py", line 51, in _create_column_from_literal
    return cast(JVMView, sc._jvm).functions.lit(literal)
  File "/Users/ruifeng.zheng/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1314, in __call__
    args_command, temp_args = self._build_args(*args)
  File "/Users/ruifeng.zheng/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1277, in _build_args
    (new_args, temp_args) = self._get_args(args)
  File "/Users/ruifeng.zheng/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1264, in _get_args
    temp_arg = converter.convert(arg, self.gateway_client)
  File "/Users/ruifeng.zheng/Dev/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_collections.py", line 510, in convert
    for element in object:
  File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/column.py", line 468, in __iter__
    raise PySparkTypeError(
pyspark.errors.exceptions.base.PySparkTypeError: [NOT_ITERABLE] Column is not iterable.

{code}
Bug: Include the name of the UDTF in the error messages generated during the function execution
Status: Open | Priority: Major | Resolution: 
Created: 08/Jun/23 18:16
Affects Version/s: 3.5.0

Improve the error messages throw from the Python worker side by including the name of the UDTF in the error messages.

 

This is a suggestion from this comment: https://github.com/apache/spark/pull/41316#discussion_r1221962150
Bug: Add math functions to Scala and Python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 05:47
Affects Version/s: 3.5.0

Add following functions:

* ceiling
* e
* pi
* ln
* negative
* positive
* power
* sign
* std
* width_bucket

to:

* Scala API
* Python API
* Spark Connect Scala Client
* Spark Connect Python Client
Bug: Add xpath_* functions to Scala and Python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 02:28
Affects Version/s: 3.5.0

Add following functions:

* xpath
* xpath_boolean
* xpath_double
* xpath_float
* xpath_int
* xpath_long
* xpath_number
* xpath_short
* xpath_string

to:

* Scala API
* Python API
* Spark Connect Scala Client
* Spark Connect Python Client
Bug: Add linear regression aggregate functions to Scala and Python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 02:05
Affects Version/s: 3.5.0

Add following functions:

* regr_avgx
* regr_avgy
* regr_count
* regr_intercept
* regr_r2
* regr_slope
* regr_sxx
* regr_sxy
* regr_syy

to:

* Scala API
* Python API
* Spark Connect Scala Client
* Spark Connect Python Client
Bug: Add support for UDAF
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/23 13:20
Affects Version/s: 3.5.0

Reference: [https://github.com/apache/spark/blob/4547c9c90e3d35436afe89b10c794050ed8d04d7/sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala#L136-L175]
Bug: Spark protobuf enums.as.ints raises exception on repeated enum types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/23 15:16
Affects Version/s: 3.4.0

For repeated enum types, the `enums.as.ints` being enabled currently raises an exception when trying to deserialize repeated enum fields. We should fix this behavior so that repeated enum fields work correctly.
Bug: Add unix_* functions to Scala and Python
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 01:43
Affects Version/s: 3.5.0

Add following functions:

* unix_date
* unix_micros
* unix_millis
* unix_seconds

to:

* Scala API
* Python API
* Spark Connect Scala Client
* Spark Connect Python Client
Bug: bad case of connect-jvm-client-mima-check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/23 03:46
Affects Version/s: 3.5.0

run 

```

build/sbt "protobuf/clean"

dev/connect-jvm-client-mima-check

```
{code:java}
Using SPARK_LOCAL_IP=localhost
Using SPARK_LOCAL_IP=localhost
Do connect-client-jvm module mima check ...
Failed to find the jar: spark-protobuf-assembly(.*).jar or spark-protobuf(.*)3.5.0-SNAPSHOT.jar inside folder: /Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/protobuf/target. This file can be generated by similar to the following command: build/sbt package|assembly
finish connect-client-jvm module mima check ...
connect-client-jvm module mima check passed.
 {code}
The check result is wrong,  there are both error messages and checks successful
Bug: Improve error messages: CANNOT_DECODE_URL, CANNOT_MERGE_INCOMPATIBLE_DATA_TYPE, CANNOT_PARSE_DECIMAL, CANNOT_READ_FILE_FOOTER, CANNOT_RECOGNIZE_HIVE_TYPE.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/23 04:34
Affects Version/s: 3.5.0

Improve error message for usability.
Bug: Dropping blocks from memory to disk may result in heartbeat loss
Status: Open | Priority: Major | Resolution: 
Created: 06/Jun/23 05:31
Affects Version/s: 3.3.2

Dropping blocks from memory to disk is executed inside a synchronized block on MemoryManager (see org.apache.spark.storage.memory.MemoryStore#evictBlocksToFreeSpace). Heartbeats include memory metrics that are retrieved from MemoryManager using synchronized methods. When blocks are being dropped, heartbeats cannot be sent as Heartbeater is blocked. If dropping blocks takes longer than the network timeout, heartbeats are considered lost and the executor gets killed by the driver.
Bug: Structured Streaming UI should display failed queries correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/23 22:39
Affects Version/s: 3.1.0

The Structured Streaming UI is designed to be able to show a query's status (active/finished/failed) and if failed, the error message.
Due to a bug in the implementation, the error message in {{QueryTerminatedEvent}} isn't being tracked by the UI data, so in turn the UI always shows failed queries as "finished".

Example:
{code:scala}
implicit val ctx = spark.sqlContext
import org.apache.spark.sql.execution.streaming.MemoryStream

spark.conf.set("spark.sql.ansi.enabled", "true")

val inputData = MemoryStream[(Int, Int)]

val df = inputData.toDF().selectExpr("_1 / _2 as a")

inputData.addData((1, 2), (3, 4), (5, 6), (7, 0))
val testQuery = df.writeStream.format("memory").queryName("kristest").outputMode("append").start
testQuery.processAllAvailable()
{code}

Here we intentionally fail a query, but the Spark UI's Structured Streaming tab would show this as "FINISHED" without any errors, which is wrong.
Bug: Fix the bug doesn't display column's sql for Percentile[Cont|Disc]
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/23 05:13
Affects Version/s: 3.3.0

Last year, I committed Percentile[Cont|Disc] functions for Spark SQL.
Recently, I found the sql method of Percentile[Cont|Disc] doesn't display column's sql suitably.
Bug: RocksDB state store can become corrupt on task retries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 13:07
Affects Version/s: 3.4.0

A couple of our streaming jobs have failed since upgrading to Spark 3.4 with an error such as:

org.rocksdb.RocksDBException: Mismatch in unique ID on table file ###. Expected: [###,###} Actual\{###,###} in file ..../MANIFEST-####

This is due to the change from [https://github.com/facebook/rocksdb/commit/6de7081cf37169989e289a4801187097f0c50fae] that enabled unique ID checks by default, and I finally tracked down the exact sequence of steps that leads to this failure in the way RocksDB state store is used.
 # A task fails after uploading the checkpoint to HDFS. Lets say it uploaded 11.zip to version 11 of the table, but the task failed before it could finish after successfully uploading the checkpoint.
 # The same task is retried and goes back to load version 10 of the table as expected.
 # Cleanup/maintenance is called for this partition, which looks in HDFS for persisted versions and sees up through version 11 since that zip file was successfully uploaded on the previous task.
 # As part of resolving what SST files are part of each table version, versionToRocksDBFiles.put(version, newResolvedFiles) is called for version 11 with its SST files that were uploaded in the first failed task.
 # The second attempt at the task commits and goes to sync its checkpoint to HDFS.
 # versionToRocksDBFiles contains the SST files to upload from step 4, and these files are considered "the same" as what's in the local working dir because the name and file size match.
 # No SST files are uploaded because they matched above, but in reality the unique ID inside the SST files is different (presumably this is just randomly generated and inserted into each SST file?), it just doesn't affect the size.
 # A new METADATA file is uploaded which has the new unique IDs listed inside.
 # When version 11 of the table is read during the next batch, the unique IDs in the METADATA file don't match the unique IDS in the SST files, which causes the exception.

 

This is basically a ticking time bomb for anyone using RocksDB. Thoughts on possible fixes would be:
 * Disable unique ID verification. I don't currently see a binding for this in the RocksDB java wrapper, so that would probably have to be added first.
 * Disable checking if files are already uploaded with the same size, and just always upload SST files no matter what.
 * Update the "same file" check to also be able to do some kind of CRC comparison or something like that.
 * Update the mainteance/cleanup to not update the versionToRocksDBFiles map.
Bug: Upgrade Cloudpickle to 2.2.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/23 08:16
Affects Version/s: 3.3.2

Cloudpickle 2.2.1 has a fix for named tuple issue (https://github.com/cloudpipe/cloudpickle/issues/460). PySpark relies on namedtuple heavily especially for RDD. We should upgrade and fix it.
Bug: Add rule to remove unused CTEDef
Status: Open | Priority: Major | Resolution: 
Created: 02/Jun/23 07:24
Affects Version/s: 3.2.4

{code:java}
// code placeholder
with t1 as (
  select rand() c3
),
t2 as (select * from t1)
select c3 from t1 where c3 > 0 {code}
{code:java}
// code placeholder
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.InlineCTE ===
 WithCTE                                               WithCTE
 :- CTERelationDef 0, false                            :- CTERelationDef 0, false
 :  +- Project [rand(3418873542988342437) AS c3#236]   :  +- Project [rand(3418873542988342437) AS c3#236]
 :     +- OneRowRelation                               :     +- OneRowRelation
!:- CTERelationDef 1, false                            +- Project [c3#236]
!:  +- Project [c3#236]                                   +- Filter (c3#236 > cast(0 as double))
!:     +- CTERelationRef 0, true, [c3#236]                   +- CTERelationRef 0, true, [c3#236]
!+- Project [c3#236]                                   
!   +- Filter (c3#236 > cast(0 as double))             
!      +- CTERelationRef 0, true, [c3#236]             
 {code}
When the above query applies the inlineCTE rule, inline is not possible because the refCount of CTERelationDef 0 is equal to 2.

However, according to the optimized logicalplan, the plan can be further optimized because the refCount of CTERelationDef 0 is equal to 1.

Therefore, we can add the rule *RemoveUnusedCTEDef* to delete the unreferenced CTERelationDef to prevent the refCount from being miscalculated
{code:java}
// code placeholder
Project [c3#236]
+- Filter (c3#236 > cast(0 as double))
   +- Project [rand(-7871530451581327544) AS c3#236]
      +- OneRowRelation {code}
Bug: Add cast alias to Scala and Python
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Jun/23 01:35
Affects Version/s: 3.5.0

add functions for:

*     castAlias("boolean", BooleanType),
*     castAlias("tinyint", ByteType),
*     castAlias("smallint", ShortType),
*     castAlias("int", IntegerType),
*     castAlias("bigint", LongType),
*     castAlias("float", FloatType),
*     castAlias("double", DoubleType),
*     castAlias("decimal", DecimalType.USER_DEFAULT),
*     castAlias("date", DateType),
*     castAlias("timestamp", TimestampType),
*     castAlias("binary", BinaryType),
*     castAlias("string", StringType),
Bug: Cannot CREATE VIEW despite columns explicitly aliased 
Status: Open | Priority: Major | Resolution: 
Created: 01/Jun/23 17:14
Affects Version/s: 3.4.0

spark.sql("CREATE VIEW v(c) AS SELECT b AS c FROM (SELECT (SELECT 1)) AS T(b)").show()

org.apache.spark.sql.AnalysisException: Not allowed to create a permanent view `spark_catalog`.`default`.`v` without explicitly assigning an alias for expression c.

The problem seems to be the scalar subquery (SELECT 1) not being aliased.
But that shouldn't matter. "AS X should be the backstop. In fact, AS T(b) should have been sufficient.
Not to speak of the fact that the column is named c in the view header.
Bug: Outdated JARs in PySpark package
Status: Open | Priority: Minor | Resolution: 
Created: 23/Jun/23 15:03
Affects Version/s: 3.4.1

The JARs which ship embedded within PySpark's package in PyPi don't seem aligned with the deps specified in Spark's own `pom.xml`.

For example, in Spark's `pom.xml`, `protobuf-java` is set to `3.21.12`:

[https://github.com/apache/spark/blob/6b1ff22dde1ead51cbf370be6e48a802daae58b6/pom.xml#L127]

However, if we look at the JARs embedded within PySpark tarball, the version of `protobuf-java` is `2.5.0` (i.e. `..../site-packages/pyspark/jars/protobuf-java-2.5.0.jar`). Same seems to apply to all other dependencies.

This introduces a set of CVEs which are fixed on upstream Spark, but are still present in PySpark (e.g. `CVE-2022-3509`, `CVE-2021-22569`, ` CVE-2015-5237` and a few others). As well as potentially introduce a source of conflict whenever there's a breaking change on these deps.
Bug: Setting the topKSortFallbackThreshold value may lead to inaccurate results
Status: Open | Priority: Minor | Resolution: 
Created: 29/Jun/23 05:13
Affects Version/s: 2.4.0

 
{code:java}
set spark.sql.execution.topKSortFallbackThreshold=10000;
SELECT min(id) FROM ( SELECT id FROM range(999999999) ORDER BY id LIMIT 10000) a; {code}
 

If GlobalLimitExec is not the final operator and has a sort operator, shuffle read does not guarantee the order, which leads to the limit read data that may be random.

TakeOrderedAndProjectExec has ordering, so there is no such problem.

 

!topKSortFallbackThreshold.png!
{code:java}
set spark.sql.execution.topKSortFallbackThreshold=10000;
select min(id) from (select  id  from range(999999999) order by id desc limit 10000) a; {code}
!topKSortFallbackThresholdDesc.png!
Bug: Remove need to preinstall the buf compiler
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Jun/23 09:05
Affects Version/s: 3.4.0

In order to lower the barrier of entry even further for this project we can remove need to have {{buf}} preinstalled and just use {{go run}}

This also ensures that the tool chain remains consistent and there is less works on my machine issues to be had.
Bug: JdbcUtils hardcodes some SQL statements
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jun/23 09:09
Affects Version/s: 2.2.0

I am currently investigating an integration with the [Neo4j JBDC driver|https://github.com/neo4j-contrib/neo4j-jdbc] and a Spark-based cloud vendor SDK.

 

This SDK relies on Spark's {{JdbcUtils}} to run queries and insert data.

While {{JdbcUtils}} partly delegates to \{{org.apache.spark.sql.jdbc.JdbcDialect}} for some queries, some others are hardcoded to SQL, see:
 * {{org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#dropTable}}
 * {{org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils#getInsertStatement}}

 

This works fine for relational databases but breaks for NOSQL stores that do not support SQL translation (like Neo4j).

Is there a plan to augment the {{JdbcDialect}} surface so that it is also responsible for these currently-hardcoded queries?
Bug: Change handling of iterable objects for on field in joins
Status: Open | Priority: Minor | Resolution: 
Created: 21/Jun/23 21:23
Affects Version/s: 3.5.0

The {{on}} field complained when I passed it a Tuple. That's because it saw that it checked for {{list}} exactly, and so wrapped it into a list like {{{}[on]{}}}, leading to immediate failure. This was surprising -- typically, tuple and list should be interchangeable, and typically tuple is the more readily accepted type. I have proposed a change that moves towards the principle of least surprise for this situation.
Bug: Unable to deserialize broadcasted map statuses when executor decommissioned
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 10/Jun/23 09:24
Affects Version/s: 3.1.0

during examination of graceful executor decommission at high rate of interruptions, jobs occasionally abort on the exception(which isn't reproducd when decommission is disabled):
{code:java}
org.apache.spark.shuffle.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle xx: java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_yyy_pieceZ of broadcast_yyy{code}
This exceptions are reproduced from reducer tasks while multiple decommission are in progress or finished which most of the time causes the entire stage to fail due to its highly reproduced.

This seems to be related to `updateMapOuput` broadcast invalidation which destroys and creates those broadcast variables on each shuffle migration([reference|https://github.com/apache/spark/blob/v3.1.0-rc1/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L134]).

*Example sequence of events from driver:*
{code:java}
2023-06-03 18:41:17:798 INFO TorrentBroadcast:61 Destroying Broadcast(171) (from updateMapOutput at BlockManagerMasterEndpoint.scala:639)
...
2023-06-03 18:41:17:811 INFO BlockManagerInfo:61 Removed broadcast_171_piece0 on xxx_host_xxx:7079 in memory (size: 4.0 MiB, free: 10.4 GiB)
...
2023-06-03 18:41:17:841 WARN TaskSetManager:73 Lost task 6.0 in stage 124.2 (TID xxxxx) (x.x.x.x executor xxxx): FetchFailed(null, shuffleId=xx, mapIndex=-1, mapId=-1, reduceId=-1, message=
org.apache.spark.shuffle.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle xx: java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_171_piece0 of broadcast_171{{}}{code}
Bug: Support propagation of the log level to the executors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jun/23 17:47
Affects Version/s: 3.5.0

Currently, the *sc.setLogLevel()* method only sets the log level on the Spark driver, failing to reflect the desired log level on the executors. This inconsistency can lead to difficulties in debugging and monitoring Spark applications, as log messages from the executors may not align with the expected log level set on the user code.
Bug: assign clear error class names for some logic that directly uses exceptions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jun/23 11:04
Affects Version/s: 3.5.0

include:
 * ALL_PARTITION_COLUMNS_NOT_ALLOWED
 * INVALID_HIVE_COLUMN_NAME
 * SPECIFY_BUCKETING_IS_NOT_ALLOWED
 * SPECIFY_PARTITION_IS_NOT_ALLOWED
 * UNSUPPORTED_ADD_FILE.DIRECTORY
 * UNSUPPORTED_ADD_FILE.LOCAL_DIRECTORY
Bug: dropDuplicates prevents correct metadata caching
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jun/23 17:39
Affects Version/s: 3.3.2

{quote}given a df, applying dropDuplicates (dD) AND after writing+reading it to s3 (cache), the metadata is deleted

applying just dD or just cache preserves metadata

however, the issue is fixed by recreating the df after dD

{{spark.createDataFrame(dedupedPositiveUsersDf.rdd, dedupedPositiveUsersDf.schema)}}

this leads to the conclusion that there is some issue with dD (affects the df in some unknown way that prevents metadata caching)

 

Example below:


{{{}val dfWithVector = spark.createDataFrame({}}}{{{}spark.sparkContext.parallelize(denseData),{}}}{{{}StructType(schema){}}}{{{}){}}}
{quote}
{quote}{{println("metadata before:" + dfWithVector.schema("features").metadata)}}
{quote}
{quote}{{var dfDroppedDuplicates = dfWithVector.dropDuplicates("id")}}
{quote}
{quote}{{println("metadata after duplicates dropped:" + droppedDuplicatesWithOverwrittenSchema.schema("features").metadata)}}
{quote}
{quote}{{dfDroppedDuplicates.write.mode("overwrite").parquet(outputLocation)}}
{quote}
{quote}{{val dfRead = spark.read.parquet(outputLocation)}}
{quote}
{quote}{{{}println("metadata after caching: " + dfRead.schema("features").metadata){}}}{{{{}}{}}}
{quote}
Bug: Improve for PlanGenerationTestSuite & ProtoToParsedPlanTestSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jun/23 11:22
Affects Version/s: 3.5.0

Improve for PlanGenerationTestSuite & ProtoToParsedPlanTestSuite, include:
- When generating `GOLDEN` files, we should first delete the corresponding directories and generate new ones to avoid submitting some redundant files during the review process. eg:
When we write a test named `make_timestamp_ltz` for the overloaded method, and during the review process, the reviewer wishes to add more tests for the method. The name of this method has changed during the next submission process, such as `make_timestamp_ltz without timezone`.At this point, if the `queries/function_make_timestamp_ltz.json`, `queries/function_make_timestamp_ltz.proto.bin` and `explain-results/function_make_timestamp_ltz.explain` files of `function_make_timestamp_ltz` are already in the commit, and there are many of these files, we generally do not notice the above problem, which leads to the incorrect submission of `queries/function_make_timestamp_ltz.json`, `queries/function_make_timestamp_ltz.proto.bin` and `explain-results/function_make_timestamp_ltz.explain` files without any impact on UT. These files are redundant.

- Clear and update some redundant files submitted incorrectly
Bug: Update helper methods to create SQLMetric with initial value
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jun/23 14:19
Affects Version/s: 3.4.0

The helper methods in [SQLMetric.scala|https://github.com/apache/spark/blob/7107742a381cde2e6de9425e3e436282a8c0d27c/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala#L38] all use a fixed default value `-1`. Callers may want the metric to start with a different initial value.