av9ash/GitBugs huggingface dataset:
Bug: jdbc connect to duckdb with error Unrecognized configuration property "path"
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/24 07:03
Affects Version/s: 3.4.0

Link issue: _[https://github.com/duckdb/duckdb/issues/11651_]
 # reproduce python code 

```python

from pyspark.sql import SparkSession

if {_}{{_}}name{{_}}{_} == '{_}{{_}}main{{_}}{_}':
    spark = SparkSession.builder \
        .appName("Example Application") \
        .config("spark.master", "local") \
        .config("spark.jars.packages",
                "io.delta:delta-core_2.12:2.4.0,org.xerial:sqlite-jdbc:3.45.2.0,org.duckdb:duckdb_jdbc:0.9.2") \
        .getOrCreate()

    spark.sql(
        f"""
            create table default.movies  
            using jdbc
            options (url "jdbc:duckdb:database/duckdb.db" , driver "org.duckdb.DuckDBDriver" , dbtable "duckdb.main.test");
            """
    )

    spark.sql("select * from default.movies").show()

    spark.stop()

```

2. error log

```

16:28:57    Runtime Error in model movies (models/sources/movies.sql)
  An error occurred while calling o40.sql.
  : java.sql.SQLException: Invalid Input Error: Unrecognized configuration property "path"
        at org.duckdb.DuckDBNative.duckdb_jdbc_startup(Native Method)
        at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:48)
        at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:41)
        at org.apache.spark.sql.execution.datasources.jdbc.connection.BasicConnectionProvider.getConnection(BasicConnectionProvider.scala:49)
    ```

3.  relative spark code

[https://github.com/apache/spark/blob/e6b7950f553cff5adc02b8b5195e79cffff3c97c/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala#L64]

Why do we need to replicate the `path` into the JDBC connection?
Bug: Performance problem with QuantileSummaries
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/24 13:20
Affects Version/s: 3.5.1

SPARK-29336 caused a severe performance regression.
In practice a partial_aggregate with several approx_percentile calls ran less than hour and the final aggrergation after exchange would have taken over a week.
Simple percentile ran about the same time in the first part and the final aggregate ran very quickly.

I made a benchmark, and it reveals that the merge operation is very-very slow: https://github.com/tanelk/spark/commit/3b16f429a77b10003572295f42361fbfb2f3c63e
From my experiments it looks like it is n^2 with the number of partitions (number of partial aggregations to merge).
When I reverted the changes made in this PR, then the "Only insert" and "Insert & merge" were very similar.

The cause seems to be, that compressImmut does not reduce the number samples allmost at all after merges and just keeps iterating over an evergrowing list.
I was not able to figure out how to fix the issue without just reverting the PR.

I also added a benchmark with KllDoublesSketch from the apache datasketches project and it worked even better than this class before this PR.
Only downside was that it is not-deterministic.
Bug: Fix problematic test in TPC-DS Collations test when ANSI flag is set
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/24 09:57
Affects Version/s: 4.0.0

"Build / ANSI (master, Hadoop 3, JDK 17, Scala 2.13)" CI is broken by TPC-DS collations test. Error:
{code:java}
[info] - q35-v2.7 *** FAILED *** (2 seconds, 695 milliseconds)
3489[info]   java.lang.Exception: Expected "[null	f	d	0	1	0.0	0	0	2	1	2.0	2	2	2	1	2.0	2	2
3490
...
null	m	m	4	1	4.0	4	4	1	1	1.0	1	1	3	1	3.0	3	3]", but got "[org.apache.spark.sparkexception
3589[info] {
3590[info]   "errorclass" : "_legacy_error_temp_2250",
3591[info]   "messageparameters" : {
3592[info]     "analyzetblmsg" : " or analyze these tables through: analyze table `spark_catalog`.`tpcds_utf8`.`customer_demographics` compute statistics;.",
3593[info]     "autobroadcastjointhreshold" : "spark.sql.autobroadcastjointhreshold",
3594[info]     "drivermemory" : "spark.driver.memory"
3595[info]   }
3596[info] }]"
3597[info] Error using configs:
3598[info]   at org.apache.spark.sql.TPCDSCollationQueryTestSuite.$anonfun$runQuery$1(TPCDSCollationQueryTestSuite.scala:228)
3599
... {code}
Bug: [State API v2] Merge TimeoutMode and TTLMode into TimeMode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/24 16:23
Affects Version/s: 4.0.0

Currently, users need to specify the notion of time (ProcessingTime/EventTime) for timers and ttl separately. This allows users to use a single parameter.

We do not expect users to use mix/match EventTime/ProcessingTime for timers and ttl in a single query because it makes hard to reason about the time semantics (when will timer be fired?, when will the state be evicted? etc.). Its simpler to stick to one notion of time throughout timers and ttl.
Bug: Improve appName and getOrCreate usage for Spark Connect
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/24 21:40
Affects Version/s: 4.0.0

 

In Spark Connect
{code:java}
spark = SparkSession.builder.appName("...").getOrCreate(){code}
 

raises error

 
{code:java}
[CANNOT_CONFIGURE_SPARK_CONNECT_MASTER] Spark Connect server and Spark master cannot be configured together: Spark master [...], Spark Connect [...]{code}
 

We should ban the usage of appName in Spark Connect
Bug: JSON parsing fails with "java.lang.ClassCastException: org.apache.spark.sql.catalyst.util.ArrayBasedMapData cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData" when spark.sql.json.enablePartialResults is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 05:32
Affects Version/s: 3.5.1

When reading the following JSON \{"a":[{"key":{"b":0}}]}: 
{code:java}
val df = spark.read.schema("a array<map<string, struct<b boolean>>>").json(path){code}
Spark throws exception: 
{code:java}
Cause: java.lang.ClassCastException: class org.apache.spark.sql.catalyst.util.ArrayBasedMapData cannot be cast to class org.apache.spark.sql.catalyst.util.ArrayData (org.apache.spark.sql.catalyst.util.ArrayBasedMapData and org.apache.spark.sql.catalyst.util.ArrayData are in unnamed module of loader 'app')
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray(rows.scala:53)
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getArray$(rows.scala:53)
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getArray(rows.scala:172)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1$$anon$2.getNext(FileScanRDD.scala:605)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.$anonfun$prepareNextFile$1(FileScanRDD.scala:884)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) {code}
 

The same happens for map: \{"a":{"key":[{"b":0}]}} when array and map types are swapped.
{code:java}
val df = spark.read.schema("a map<string, array<struct<b boolean>>>").json(path) {code}
 

This is a corner case that https://issues.apache.org/jira/browse/SPARK-44940 missed.
Bug: Revert mapping ( star ) to named_struct ( star )
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/24 19:13
Affects Version/s: 4.0.0

Turning star within parens into named_struct ( star) as opposed to ignoring the parens turns out to be more risky than anticipated. Given that this was done solely for consistency with ( c1, c2...) it's best to not go there at all.
Bug: After executing the reboot command on the host where the Driver node is located, the spark streaming ends in a SUCCESSED state
Status: Open | Priority: Major | Resolution: 
Created: 10/Apr/24 08:05
Affects Version/s: 2.4.8

While running Spark streaming applications on YARN in cluster mode, reboot/shutdown of the node hosting AM causes the application to terminate SparkContext and mark it as SUCCEEDED。

The reboot/shutdown command will send an graceful stop signal of "kill -15" to all processes on this machine. From the Spark Streaming code, this end signal will make the Spark Streaming application think it has ended normally. The log is as follows:

!image-2024-04-10-16-03-29-393.png!

But in most cases, the reboot/shutdown command may occur due to misoperation, other services needing to be restarted, or the operating system itself needing to be restarted. Is it appropriate for Spark to report such a "SUCCEEDED" status?

Especially now, many scheduling systems decide whether to restart the Spark task based on its status, such as the "FAILED" status. Spark streaming reports such a "SUCCEEDED" status, making it difficult for the scheduling system to handle.

Moreover, Spark streaming runs for a long time, and reporting the "SUCCEEDED" status is also very ambiguous.
Bug: Make DISTINCT(*) a synonym for DISTINCT * 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/24 20:11
Affects Version/s: 4.0.0

We overshot when we removed the rule that ignored parens around star to instead produce a named structs.
It turns out that there is a substantial set of users abusing thsi property do do DISTINCT(*) to mean DISTINCT *.
Instead of rolling back the entire improvement, let's special case DISTINCT ( *)
Bug: .sql() does not recognize watermark defined upstream
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 03/Apr/24 17:01
Affects Version/s: 3.5.1

I have a data pipeline set up in such a way that it reads data from a Kafka source, does some transformation on the data using pyspark, then writes the output into a sink (Kafka, Redis, etc).

 

My entire pipeline in written in SQL, so I wish to use the .sql() method to execute SQL on my streaming source directly.

 

However, I'm running into the issue where my watermark is not being recognized by the downstream query via the .sql() method.

 

```
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:49:36) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyspark
>>> print(pyspark.__version__)
3.5.1
>>> from pyspark.sql import SparkSession
>>>
>>> session = SparkSession.builder \
...     .config("spark.jars.packages", "org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.1")\
...     .getOrCreate()
>>> from pyspark.sql.functions import col, from_json
>>> from pyspark.sql.types import StructField, StructType, TimestampType, LongType, DoubleType, IntegerType
>>> schema = StructType(
...     [
...         StructField('createTime', TimestampType(), True),
...         StructField('orderId', LongType(), True),
...         StructField('payAmount', DoubleType(), True),
...         StructField('payPlatform', IntegerType(), True),
...         StructField('provinceId', IntegerType(), True),
...     ])
>>>
>>> streaming_df = session.readStream\
...     .format("kafka")\
...     .option("kafka.bootstrap.servers", "localhost:9092")\
...     .option("subscribe", "payment_msg")\
...     .option("startingOffsets","earliest")\
...     .load()\
...     .select(from_json(col("value").cast("string"), schema).alias("parsed_value"))\
...     .select("parsed_value.*")\
...     .withWatermark("createTime", "10 seconds")
>>>
>>> streaming_df.createOrReplaceTempView("streaming_df")
>>> session.sql("""
... SELECT
...     window.start, window.end, provinceId, sum(payAmount) as totalPayAmount
...     FROM streaming_df
...     GROUP BY provinceId, window('createTime', '1 hour', '30 minutes')
...     ORDER BY window.start
... """)\
...   .writeStream\
...   .format("kafka") \
...   .option("checkpointLocation", "checkpoint") \
...   .option("kafka.bootstrap.servers", "localhost:9092") \
...   .option("topic", "sink") \
...   .start()
```
 
This throws exception
```
pyspark.errors.exceptions.captured.AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark; line 6 pos 4;
```
Bug: Promote `--wait` option to all services
Status: Open | Priority: Major | Resolution: 
Created: 09/Apr/24 08:21
Affects Version/s: 4.0.0

SPARK-47040 add `--wait` support to `start-connect-server.sh`

./sbin/start-connect-server.sh [--wait] [options]

 

In [https://github.com/apache/spark/pull/45852,] we discussed and reached a consensus to promote `–wait` to all service for consistency
Bug: Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/24 23:52
Affects Version/s: 4.0.0

https://issues.apache.org/jira/browse/SPARK-47199 didn't fix the flakiness in the pyspark.sql.dataframe.DataFrame.writeStream doctest : the problem is not that we are colliding on the test but, rather, that the test is starting a background thread to write to a directory then deleting that directory from the main test thread, something which is inherently race prone.

The fix is simple: stop the streaming query in the doctest itself, similar to other streaming doctest examples.
Bug: Data Source names unavailable when using Delta Share and Kafka SQL
Status: Open | Priority: Major | Resolution: 
Created: 08/Apr/24 16:25
Affects Version/s: 3.5.1

I have a simple Spark application that is reading from a csv file via Delta Share and writing the contents to Kafka. When both the Delta Share Kafka SQL libraries are included in the project, Spark is unable to load them by their format short names.

If I use one of them without the other, everything works fine. When both are included, then I get this root exception: ClassNotFoundException: deltaSharing.DefaultSource.

If I specify the source class names (
io.delta.sharing.spark.DeltaSharingDataSource, org.apache.spark.sql.kafka010.KafkaSourceProvider) instead of the short names, it works correctly.
Bug: Use column ordinals instead of prefix ordering columns in the range scan encoder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/24 00:02
Affects Version/s: 4.0.0

Currently, the State V2 implementations do projections in their state managers, and then provide some prefix (ordering) columns to the RocksDBStateEncoder. However, we can avoid doing extra projection by just reading the ordinals we need, in the order we need, in the state encoder.
Bug: Add support for negative byte types in RocksDB range scan encoder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Apr/24 22:54
Affects Version/s: 4.0.0

SPARK-47372 introduced the ability for the RocksDB state provider to big-endian encode values so that they could be range scanned. However, signed support for Byte types [was missed|https://github.com/apache/spark/blob/1efbf43160aa4e36710a4668f05fe61534f49648/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala#L326], even though Scala Bytes are [signed|https://www.scala-lang.org/api/2.13.5/scala/Byte.html].
Bug: Dataframe.collect should accept duplicated column names
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/24 00:48
Affects Version/s: 4.0.0

{code:java}
+---+---+---+---+
|  i|  j|  i|  j|
+---+---+---+---+
|  1|  a|  1|  a|
+---+---+---+---+ {code}
 

collect fails with

 
{code:java}
[info]   org.apache.spark.sql.AnalysisException: [AMBIGUOUS_COLUMN_OR_FIELD] Column or field `i` is ambiguous and has 2 matches. SQLSTATE: 42702
[info]   at org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError(CompilationErrors.scala:28)
[info]   at org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError$(CompilationErrors.scala:23)
[info]   at org.apache.spark.sql.errors.CompilationErrors$.ambiguousColumnOrFieldError(CompilationErrors.scala:54)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1(ArrowDeserializer.scala:460)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1$adapted(ArrowDeserializer.scala:454)
[info]   at scala.collection.immutable.List.foreach(List.scala:334)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.createFieldLookup(ArrowDeserializer.scala:454)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:328)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:86)
[info]   at org.apache.spark.sql.connect.client.arrow.ArrowDeserializingIterator.<init>(ArrowDeserializer.scala:542) {code}
Bug: Change default of spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 17:53
Affects Version/s: 4.0.0

spark.sql.legacy.timeParserPolicy was introduced in Spark 3.0 and has been set to EXCEPTION.
Changing it from EXCEPTION for SPark 4.0 to CORRECTED will reduce errors and reflects a prudent timeframe.
Bug: Spark Transformation with Multi Case filter can improve efficiency
Status: Open | Priority: Major | Resolution: 
Created: 05/Apr/24 17:58
Affects Version/s: 4.0.0

In Feature Engineering we need to process the input data to create feature and feature vectors which are required to train the model. For which we need to do multiple spark transformations (etc:map, filter etc) the spark has very good optimization for multiple transformations due to its lazy execution. It combines multiple transformations into fewer transformations which helps to optimize the overall execution time.

I found that we can still improve the execution time in the case of filters.

 
{code:java}
val rddfilter0 = personRdd.filter(t => t.age>5 && t.age<=12)
val rddfilter1 = personRdd.filter(t => t.age>12 && t.age<=18)
val rddfilter2 = personRdd.filter(t => t.age>18 && t.age<=25)
val rddfilter3 = personRdd.filter(t => t.age>25 && t.age<=35)
val rddfilter4 = personRdd.filter(t => t.age>35 && t.age<=65) {code}
*Sample Run Results:*

Records :50,000,000

5 filter Execution Time: 24854 milli sec

5 filter with Map Execution Time: 5212 milli sec

We can very well improve multiple X times and reduce significant memory footprint for a complex DAG of Spark Transformation.

Sample illustration can be found here

[https://docs.google.com/document/d/1gdWR2TwbCfiuRF51EHA1zRnD9ES_neIvIsgEvizrjuo/edit?usp=sharing]

Need support of such transformation in Spark Core so that more complex transformation can be supported. Some illustration is provided in above document.
Bug: Spark Connect plugins are unable to create and process plans
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 11:11
Affects Version/s: 4.0.0

Spark Connect plugins are unable to create datasets/dataframes and are unable to process relations and expressions in their planner plugins. The issue is caused by the interfaces for this in the SparkSession and the SparkConnectPlanner expect a protobuf.Any, which ends up being shaded and therefore inaccessible by the plugin.
Bug: Ability for Spark to execute "embarassingly parallel" operations
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/24 18:03
Affects Version/s: 3.5.1

Consider a large operation. It applies over distinct values in a column, and each distinct value is a universe unto itself; but running the whole batch is certain to OOM.

This could be something like many geographies, item classes that are only comparable to their own type, or similar such things. Perhaps they cartesian join on themselves and so subsets are tractable but not the whole thing, but that's going to be dependent on use case.

Since the whole dataset is large, a join condition is intractable. Today, we might for-loop over the distinct values, and union the results that are persisted. However, that for-loop is sequential for no good reason at all.

 

I propose an API, something like `pyspark.sql.DataFrame.applyOver`, that takes three arguments:
 # A function that accepts `pyspark.sql.DataFrame` (namely, this one) and returns a `pyspark.sql.DataFrame`
 # A column or list of columns over which distinct values are found
 # The schema returned in (1)

This method would apply (1) to each filter of (2) *in parallel over available executors,* evaluate each result, and return the union of those results. Even a thoroughly naive implementation that persists the dataframes to shared storage and the resulting object is explicitly the union of those files would reduce overall execution time massively, to as little as the single longest subset time rather than the sum of all of them.
Bug: Fix the 2b+ rows in a single rowgroup for row_index in Parquet reader
Status: Open | Priority: Major | Resolution: 
Created: 04/Apr/24 15:27
Affects Version/s: 3.5.0

Parquet reader in Spark has a bug where a file containing 2b+ rows in a single rowgroup causes it to run out of the `Integer` range. This prevents Delta Parquet readers from exposing the row_index field as a metadata field.

It would be great to have this fix so that we can use 2b+ rows in a single rowgroup and also to safely allow row_index field to be used in the Delta Parquet readers for any functionalities that might depend on it.

Link to the comment in the code: https://github.com/delta-io/delta/blob/e3a481bd6c42a4f91686377d78ec9d9c934e27ee/spark/src/main/scala/org/apache/spark/sql/delta/DeltaParquetFileFormat.scala#L200
Bug: Do not log the gRPC exception to stderr in PySpark
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 08:18
Affects Version/s: 3.5.0

Currently if there's any gRPC exception, instead of just handling it, the PySpark's gRPC error handler is going to print it out to the stderr, not allowing the user to cleanly ignore the exception by using try/except control flow statement.
Bug: Fix formatting of error messages with treeNode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/24 20:22
Affects Version/s: 4.0.0

Example: We are concatenating the plan without any separation: `in this locationFilter (dept_id#652`. We should add a colon and space or newline in between:
{code:java}
[UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.ACCESSING_OUTER_QUERY_COLUMN_IS_NOT_ALLOWED] Unsupported subquery expression: Accessing outer query column is not allowed in this locationFilter (dept_id#652 = outer(dept_id#646))
+- SubqueryAlias dept
   +- View (`DEPT`, [dept_id#652, dept_name#653, state#654])
      +- Project [cast(dept_id#655 as int) AS dept_id#652, cast(dept_name#656 as string) AS dept_name#653, cast(state#657 as string) AS state#654]
         +- Project [dept_id#655, dept_name#656, state#657]
            +- SubqueryAlias DEPT
               +- LocalRelation [dept_id#655, dept_name#656, state#657]
. SQLSTATE: 0A000 {code}
Bug: Support Hive tables as a streaming source and sink
Status: Open | Priority: Major | Resolution: 
Created: 03/Apr/24 15:08
Affects Version/s: 3.3.2

People have data stored in Hive tables. Currently these tables do not support Spark streaming, so customers do not have a good way to natively stream this data in Spark. The current solutions involve an intermediary to track which data has been read and periodically execute batch jobs. This use case should be supported by Spark's in-built streaming mechanism.

 

From doing some research, Hive supports streaming [https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest+V2] but Spark does not support streaming on tables in Hive format. I don't think it makes sense to start copying Hive server-side code into Spark, but we could copy the relevant logic and wrap it in the DataSourceV2 APIs to enable this feature. To not break backwards compatibility, we would probably want to gate this behind a new Spark property.