av9ash/GitBugs huggingface dataset:
Bug: Distinguish the streaming nodes from the text representation of logical plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/24 07:33
Affects Version/s: 4.0.0

We had a hard time to figure out whether the nodes are streaming or not, when we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 .

Plan text for logical plan does not show the property of isStreaming, hence we had to speculate the value based on other context. In addition, even though the type of leaf node is explicitly meant to be streaming which enables us to track down the isStreaming for certain subtree, the plan could be very long and it’s a non-trivial effort to trace down to the leaf nodes. Also, if the leaf nodes are skipped on the representation due to the size, there is no way to get the information of isStreaming.

We propose to introduce a marker of the representation for streaming, which will be shown in the text logical plan. There is no concept of "isStreaming" in physical plan, so the change only needs to happen in logical plan.
Bug: Event driven block manager decommissioner
Status: Open | Priority: Major | Resolution: 
Created: 16/Jun/24 12:35
Affects Version/s: 3.1.3

The current blockmanager decommisssioner uses the periodic threads to refresh blocks/ peers, monitor progress. It can be low-effeicent this way. For example, in the worst case, it takes 30s (by default) at most for an executor to exit itself even if all the blocks have been migrated. The cause is that the migration status is checked every 30s (by default).

So this ticket proposes the blockmanager decommissioner to leverage the events driven way to improve its effeciency.
Bug: 50% cpu time spent on SQLConf.getConf in ALTER ADD COLUMN query
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/24 19:50
Affects Version/s: 3.5.1

SQLConf.caseSensitiveAnalsysis is evaluated for all (existing and added) columns in an ALTER ADD COLUMN query. For a query on a table with 10k columns and adding 1 new column, 50% of cpu time is spent in SQLConf.getConf (profile in attachments)
Bug: Enable Spark SIMD JSON
Status: Open | Priority: Major | Resolution: 
Created: 13/Jun/24 09:47
Affects Version/s: 4.0.0

This is used to improve the performance of get_json_object and json_tuple.
https://github.com/simdjson/simdjson-java/
https://www.i-programmer.info/news/80-java/16553-the-lightning-fast-json-parser-library-for-java.html
Bug: Support more types of expressions in VALUES lists
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 12/Jun/24 17:22
Affects Version/s: 4.0.0

Currently we return the `INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE` error class   [1] if SQL VALUES lists contain non-literal expressions such as CURRENT_DATE(). In this Jira we propose to relax that restriction to allow more types of expressions.

[1] https://github.com/apache/spark/blob/b5e1b7988031044d3cbdb277668b775c08db1a74/common/utils/src/main/resources/error/error-conditions.json#L2247
Bug: Upgrade `google-java-format` to 1.22.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jun/24 16:02
Affects Version/s: kubernetes-operator-0.1.0

This issue aims to upgrade `google-java-format` plugin of Spark Kubernetes Operator repository to bring the latest bug fixes like the following. The latest version is recommended.
{code}
java.lang.Exception: google-java-format 1.17.0 is currently being used, but outdated.
google-java-format 1.19.2 is the recommended version, which may have fixed this problem.
google-java-format 1.19.2 requires JVM 11+.
{code}
Bug: Writing to JDBC Temporary View Failed
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/24 07:15
Affects Version/s: 3.4.0

When creating a JDBC temporary view, *ApplyCharTypePadding* would add a Project before LogicalRelation if CHAR/VARCHAR column exists and Spark would save it as a view plan. Then if we try to write this view, Spark would put this view plan to *InsertintoStatement* in *ResolveRelations* which would fall {*}PrewriteCheck{*}.

Adding the following code to *JDBCTableCatalogSuite* would meet this problem.
{code:java}
test("test writing temporary jdbc view") {
    withConnection { conn =>
      conn.prepareStatement("""CREATE TABLE "test"."to_drop" (id CHAR)""").executeUpdate()
    }
    sql(
      s"""
        CREATE TEMPORARY TABLE jdbcTable
        USING jdbc
        OPTIONS (
          url='$url',
          dbtable='"test"."to_drop"');""")
    sql("INSERT INTO jdbcTable values(1),(2)")
    sql("select * from test.to_drop").show()
    withConnection { conn =>
      conn.prepareStatement("""DROP TABLE "test"."to_drop"""").executeUpdate()
    }
  } {code}
 

Then we would get the following error.
{code:java}
[UNSUPPORTED_INSERT.RDD_BASED] Can't insert into the target. An RDD-based table is not allowed. SQLSTATE: 42809;
'InsertIntoStatement Project [staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, ID#0, 1, true, false, true) AS ID#1], false, false, false
+- LocalRelation [col1#3] {code}
Bug: Incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/24 16:17
Affects Version/s: 4.0.0

Following sequence of queries produces UNSUPPORTED_GROUPING_EXPRESSION error:
{code:java}
create table t1(a int, b int) using parquet;
select grouping(a), dummy from t1 group by a with rollup; {code}
However, the appropriate error should point the user to the invalid dummy column name.
Bug: Reduce memory pressure of empty TreeNode BitSets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/24 20:36
Affects Version/s: 4.0.0

By default, the TreeNode `treePatternBits` and `ineffectiveRules` variables are initialized to a `BitSet`, which takes up memory even when empty. For plans with a high number of TreeNodes, this adds unnecessary memory pressure, especially for large expression trees. Reduce the memory pressure by lazily initializing this variable.

For `treePatternBits` we will observe no benefit because these are always populated. However, for `ineffectiveRules` it is not necessary to initialize this for *every* node. We should make this variable lazy, being careful to not initialize even when `get` is called.

Related to SPARK-48047.
Bug: Repartition, sort and partitionBy not working together
Status: Open | Priority: Major | Resolution: 
Created: 04/Jun/24 09:27
Affects Version/s: 3.5.0

Hi, we are having some problems writing sorted csv’s using Spark 3.5.1.

Example data:
[Parquet 1M records from flights|https://www.tablab.app/datasets/sample/parquet?datatable-source=demo-flights-1m]

 

Example code:
{code:scala}
val df = spark.read.parquet("Flights 1m.parquet").withColumn("partition_col", lit("2024")).localCheckpoint
df.repartition(1).sort("FL_DATE", "DISTANCE").write.mode("overwrite").partitionBy("partition_col").csv("repartition_order") {code}
 

Running previous example using Spark 3.3.4 writes a single file ordered by FL_DATE and DISTANCE fields inside the folder partition_col=2024.

On the other hand when using Spark 3.5.1 it returns same number of files as cores in the executors (4 in my case using 2 executors with 2 cores each) and rows are not sorted inside.
We can see that after repartition(1) and before the sort Spark adds another stage repartition(200) because of the default shuffle partitions value, and then AQE coalesce small partitions.

 

Spark 3.3.4 plan:
{code:java}
== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand (8)
+- AdaptiveSparkPlan (7)
   +- == Final Plan ==
      * Sort (4)
      +- ShuffleQueryStage (3), Statistics(sizeInBytes=76.3 MiB, rowCount=1.00E+6)
         +- Exchange (2)
            +- * Scan ExistingRDD (1)
   +- == Initial Plan ==
      Sort (6)
      +- Exchange (5)
         +- Scan ExistingRDD (1) {code}
 

Spark 3.5.1 plan:
{code:java}
== Physical Plan ==
AdaptiveSparkPlan (15)
+- == Final Plan ==
   Execute InsertIntoHadoopFsRelationCommand (9)
   +- WriteFiles (8)
      +- * Sort (7)
         +- AQEShuffleRead (6)
            +- ShuffleQueryStage (5), Statistics(sizeInBytes=76.3 MiB, rowCount=1.00E+6)
               +- Exchange (4)
                  +- ShuffleQueryStage (3), Statistics(sizeInBytes=76.3 MiB, rowCount=1.00E+6)
                     +- Exchange (2)
                        +- * Scan ExistingRDD (1)
+- == Initial Plan ==
   Execute InsertIntoHadoopFsRelationCommand (14)
   +- WriteFiles (13)
      +- Sort (12)
         +- Exchange (11)
            +- Exchange (10)
               +- Scan ExistingRDD (1) {code}
Bug: PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#main()
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 04/Jun/24 04:27
Affects Version/s: 3.3.2

PythonWorkerFactory does not print the error stream in case the daemon fails before executing the main daemon.py#mainI(). It throws SparkException like below but does not print the error stream which has the failure why pyspark.daemon failed to start.
{code:java}
org.apache.spark.SparkException: 
2024-05-07T16:04:53.169524256Z stderr F Bad data in pyspark.daemon's standard output. Invalid port number:
2024-05-07T16:04:53.169530703Z stderr F   1097887852 (0x4170706c){code}
The error stream is being [read|#L303]] after throwing SparkException. It has to be captured during the exception as well.

 

*Simple Repro:*

1. Run a sample pyspark job by setting wrong
spark.python.daemon.module like pyspark.wrongdaemon instead of default one pyspark.daemon.
 
2. The forked  python process will fail with {*}"{*}{*}/opt/python/3.9.2/bin/python: No module named pyspark.wrongdaemon"{*} but it is not captured by PythonWorkerFactory.
Bug: Loosen `correlated scalar subqueries must be aggregated` error by doing runtime check for scalar subqueries output rowcount
Status: Open | Priority: Major | Resolution: 
Created: 02/Jun/24 21:42
Affects Version/s: 4.0.0

Currently if a scalar subquery’s result isn’t aggregated or limit 1, we throw an error {{{}Correlated scalar subqueries must be aggregated{}}}.

This check is often too restrictive, there are many cases where it should actually be runnable even though we don’t know it - e.g. unique keys or functional dependencies might ensure that there's only one row.

To handle these cases, it’s better to do the check at runtime instead of statically. This could be implemented as a special aggregate operator that throws exception on >=2 rows input, a “single join” operator that throws an exception when >= 2 rows match, or something similar.

There are also cases where we were incorrectly allowing queries before that returned wrong results, and should have been rejected as invalid (e.g. SPARK-48503, SPARK-18504). Doing the check at runtime would help avoid those bugs.

Current workarounds: Users can add an aggregate like {{any_value()}} or {{first()}} to the output of the subquery, or users can add {{limit 1}}
Bug: Refactor SparkConnect Service to extracted error handling functions to trait
Status: Open | Priority: Minor | Resolution: 
Created: 20/Jun/24 14:25
Affects Version/s: 3.4.0

Since SparkConnect gRPC server can have multiple services (addService function on NettyServerBuilder) and these functions can be reused across services, specially when we would like to extend sparkconnect with various services.

We can extract error handling functions from SparkConnectService to a trait, that will increase code reusability. By doing this we can reuse these functions across multiple service implementations. Since we can add multiple Bindable service handlers to SparkConnect gRPC server it will be easy to use such common functions to handle errors and exceptions.
Bug: Catch missing window specification error early
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jun/24 18:44
Affects Version/s: 4.0.0

Before, aggregate queries containing a window function without a window specification (e.g. `PARTITION BY`) would return a non-descriptive internal error message: 

`org.apache.spark.sql.catalyst.analysis.UnresolvedException: [INTERNAL_ERROR] Invalid call to exprId on unresolved object SQLSTATE: XX000`

This PR catches the user error early and returns a more accurate description of the issue:

`Window specification <windowName> is not defined in the WINDOW clause.`
Bug: Reduce number of exception creations for default expression analysis
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Jun/24 12:16
Affects Version/s: 3.5.2

When resolving of default value being done in method _resolveColumnDefaultInAssignmentValue,_ exception that should be thrown in case default value is not resolvable is created in any case (positive or negative outcome), but exception creation can be expensive (because reading error-classes file to get exception message) we should avoid it and create exception only when it is needed.
Bug: Fix incorrect conf settings of ignoreCorruptFiles related tests case in ParquetQuerySuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Jun/24 11:26
Affects Version/s: 4.0.0

The code as follows:
{code:java}
withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key -> sqlConf) {        
  withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key -> "false") { 
}{code}
he inner withSQLConf (SQLConf.IGNORE_CORRUPT_FILES.key -> "false") will overwrite the outer configuration, making it impossible to test the situation where sqlConf is true.
Bug: The document is out of date and needs to be updated
Status: Open | Priority: Minor | Resolution: 
Created: 19/Jun/24 01:28
Affects Version/s: 3.5.1

I am looking for a data source implementation for Spark SQL 3.5.1 that can accept MQTT and REST interface, through Google search, I found the latest version is two years ago, and Java implementation is very few, I found that the custom data source has v1 and v2 and unbounded table, I am confused, which implementation should I use for 3.5.1, how to implement, Can you update the documentation or help me? thank you.
Bug: Narrow down default regex of sql options redaction.
Status: Resolved | Priority: Minor | Resolution: Won't Fix
Created: 17/Jun/24 06:55
Affects Version/s: 4.0.0

The current *`spark.sql.redaction.options.regex`* configuration is too loose, resulting in masking of some insensitive options.

 

Like *['tag.creation-period'='hourly'|https://paimon.apache.org/docs/master/maintenance/manage-tags/]* option in Paimon,

Original option:
{code:java}
'tag.creation-period' = 'hourly' {code}
Redacted option:
{code:java}
'tag.creation-period' = '*********(redacted)' {code}
Bug: spark-sql does not support using Spark Connect
Status: Open | Priority: Minor | Resolution: 
Created: 04/Jun/24 08:59
Affects Version/s: 3.5.1

Similar to spark-shell (for Scala) specifying a Spark Connect option results in a "master URL must be set in your configuration" error.

 

Sample execution:

{{SPARK_REMOTE=sc://localhost spark-sql}}

 

Another attempt at setting the same value:

{{spark-sql --remote localhost:50051}}
Bug: Improve Performance of CodeFormatter with Java StringBuilder
Status: Open | Priority: Trivial | Resolution: 
Created: 07/Jun/24 20:21
Affects Version/s: 4.0.0

- Update CodeFormatter to use Java's `StringBuilder` directly instead of Scala's
- The Scala `StringBuilder` is a very thin API that wraps Java's.
- All callsites that need to change just trivially copy what Scala was delegating to Java. For example `clear()` simply calls `setLength(0)` under the hood.
- The reason for this change is that it substantially improves the performance of the CodeFormatter.
- From some basic profiling, in a ~100s suite, code formatting took ~2.7 seconds of CPU time. Post change it takes about 800ms.
- My hypothesis is that Java StringBuilder is much more JIT friendly. Scala's StringBuilder has many layers as it implements a significant portion of the Scala mutable collection API. It's also likely the case that the JVM has special JIT handling for Java StringBuilder
Bug: Scheduling Across Applications in k8s mode 
Status: Resolved | Priority: Trivial | Resolution: Information Provided
Created: 20/Jun/24 13:28
Affects Version/s: 3.5.1

I have been trying autoscaling in Kubernetes for Spark Jobs,When first job is triggered based on load workers pods are scaling which is fine but When second job is submitted its not getting allocating any resources as First Job is consuming all the resources.

Second job is in Waiting State until First Job is finished.I have gone through documentation to set max cores in standalone mode which is not a ideal solution as we are planning autoscaling based on load and Jobs submitted.

Is there any solution for this or any alternatives ?
Bug: Do not test SET command in Python UDTF test
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 27/Jun/24 04:28
Affects Version/s: 4.0.0

{code}
        # We do not use `self.sql_conf` here to test the SQL SET command
        # instead of using PySpark's `spark.conf.set`.
        old_value = self.spark.conf.get("spark.sql.execution.pythonUDTF.arrow.enabled")
        self.spark.sql("SET spark.sql.execution.pythonUDTF.arrow.enabled=False")
        self.assertEqual(udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_TABLE_UDF)
        self.spark.sql("SET spark.sql.execution.pythonUDTF.arrow.enabled=True")
        self.assertEqual(
            udtf(TestUDTF, returnType="x: int").evalType, PythonEvalType.SQL_ARROW_TABLE_UDF
        )
        self.spark.conf.set("spark.sql.execution.pythonUDTF.arrow.enabled", old_value)
{code}

we don;t need to test SET command there
Bug: Add default java opt for YARN AM
Status: Open | Priority: Trivial | Resolution: 
Created: 08/Jun/24 07:48
Affects Version/s: 3.5.1

The options spark.driver.defaultJavaOptions / spark.executor.defaultJavaOptions were added in version 3.0.0. But there is no option for YARN AM's defualt java option in client mode. Can you add a corresponding default option for YARN AM ?
Bug: Allow passing custom sink to StreamTest::testStream
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 04/Jun/24 16:28
Affects Version/s: 4.0.0

The testing helpers for streaming don't allow providing a custom sink, this is limiting in (at least) two ways:
 * A sink can't be reused across multiple calls to `testStream`, e.g. when canceling and resuming streaming
 * A custom sink implementation other than `MemorySink` can't be provided. A use case here is for example to test the Delta streaming sink by wrapping it in a MemorySink interface and passing it to the test framework.
Bug: Incorrect result for window frames defined by a range clause on large decimals 
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 31/Dec/22 01:06
Affects Version/s: 3.4.0

Context https://github.com/NVIDIA/spark-rapids/issues/7429#issuecomment-1368040686

The following windowing query on a simple two-row input should produce two non-empty windows as a result

{code}
from pprint import pprint
data = [
  ('9223372036854775807', '11342371013783243717493546650944543.47'),
  ('9223372036854775807', '999999999999999999999999999999999999.99')
]
df1 = spark.createDataFrame(data, 'a STRING, b STRING')
df2 = df1.select(df1.a.cast('LONG'), df1.b.cast('DECIMAL(38,2)'))
df2.createOrReplaceTempView('test_table')
df = sql('''
  SELECT 
    COUNT(1) OVER (
      PARTITION BY a 
      ORDER BY b ASC 
      RANGE BETWEEN 10.2345 PRECEDING AND 6.7890 FOLLOWING
    ) AS CNT_1 
  FROM 
    test_table
  ''')
res = df.collect()
df.explain(True)
pprint(res)
{code}

Spark 3.4.0-SNAPSHOT output:
{code}
[Row(CNT_1=1), Row(CNT_1=0)]
{code}

Spark 3.3.1 output as expected:
{code}
Row(CNT_1=1), Row(CNT_1=1)]
{code}
Bug: Document the new feature "Async Progress Tracking" to Structured Streaming guide doc
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 20/Dec/22 07:48
Affects Version/s: 3.4.0

Given that we merged the new SPIP feature SPARK-39591, we have to document the new feature to the Structured Streaming guide doc so that end users can refer to the doc and start experimenting the feature.
Bug: PyTorch Distributor
Status: Reopened | Priority: Critical | Resolution: 
Created: 20/Dec/22 00:14
Affects Version/s: 3.4.0

This is a project to make it easier for PySpark users to distribute PyTorch code using PySpark. The corresponding [Design Document|https://docs.google.com/document/d/1QPO1Ly8WteL6aIPvVcR7Xne9qVtJiB3fdrRn7NwBcpA/edit?usp=sharing] can give more context. This was a project determined by the Databricks ML Training Team; please reach out to [~gurwls223] (Spark-side) or [~erithwik] for more context.
Bug: Reading V2 datasource masks underlying error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/22 01:08
Affects Version/s: 3.3.0

In Spark 3.3, 
 # DataSourceV2Utils, the loadV2Source calls: {*}(CatalogV2Util.loadTable(catalog, ident, timeTravel).get{*}, Some(catalog), Some(ident)).
 # CatalogV2Util.scala, when it tries to *loadTable(x,x,x)* and it fails with any of these exceptions NoSuchTableException, NoSuchDatabaseException, NoSuchNamespaceException, it would return None
 # Coming back to DataSourceV2Utils, None was previously returned and calling None.get results in a cryptic error technically "correct", but the *original exceptions NoSuchTableException, NoSuchDatabaseException, NoSuchNamespaceException are thrown away.*

 

*Ask:*

Retain the original error and propagate this to the user. Prior to Spark 3.3, the *original error* was shown and this seems like a design flaw.

 

*Sample user facing error:*

None.get
java.util.NoSuchElementException: None.get
    at scala.None$.get(Option.scala:529)
    at scala.None$.get(Option.scala:527)
    at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
    at scala.Option.flatMap(Option.scala:271)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)

 

*DataSourceV2Utils.scala - CatalogV2Util.loadTable(x,x,x).get*
[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala#L137]

*CatalogV2Util.scala - Option(catalog.asTableCatalog.loadTable(ident))*

{*}{{*}}[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L341]

*CatalogV2Util.scala - catching the exceptions and return None*
[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L344]
Bug: Task rerun on decommissioned executor can be avoided if shuffle data has migrated
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/22 15:13
Affects Version/s: 3.1.3

Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.