av9ash/GitBugs huggingface dataset:
Bug: Add back Matomo analytics to release docs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Sep/23 19:30
Affects Version/s: 4.0.0

We had previously removed Google Analytics from the website and release docs, per ASF policy: https://github.com/apache/spark/pull/36310

We just restored analytics using the ASF-hosted Matomo service on the website:
https://github.com/apache/spark-website/commit/a1548627b48a62c2e51870d1488ca3e09397bd30

This change would put the same new tracking code back into the release docs.
Bug: download krb5.conf from remote storage in spark-submit on k8s
Status: Open | Priority: Minor | Resolution: 
Created: 15/Sep/23 03:17
Affects Version/s: 3.4.1

krb5.conf currently only supports the local file format. Tenants would like to save this file on their own servers and download it during the spark-submit phase for better implementation of multi-tenant scenarios. The proposed solution is to use the *downloadFile*  function[1], similar to the configuration of *spark.kubernetes.driver/executor.podTemplateFile*

 

[1]https://github.com/apache/spark/blob/822f58f0d26b7d760469151a65eaf9ee863a07a1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/PodTemplateConfigMapStep.scala#L82C24-L82C24
Bug: InMemoryTableScanExec simpleStringWithNodeId adds columnar info
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/23 02:27
Affects Version/s: 4.0.0

InMemoryTableScanExec supports both row-based and columnar input and output which is based on the cache serialzier. It would be more friendly for user if we can provide the columnar info to show whether it is columnar in/out.
Bug: Non-nullable schema is not effective in DF from JSON
Status: Open | Priority: Minor | Resolution: 
Created: 21/Sep/23 13:41
Affects Version/s: 3.3.3

In Spark 3.3 and 3.4, when creating a DF with schema with non-nullable fields, the created DF ends up having schema with nullable fields.

 

 
{code:java}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types.{BooleanType, StructField, StructType}
object Foo extends App {
  val spark: SparkSession = SparkSession.builder()
    .appName("foo")
    .master("local[*]")
    .config("spark.driver.host", "127.0.0.1")
    .getOrCreate()
  val schema = StructType(Array(StructField("a", BooleanType, nullable = false)))
  import spark.implicits._
  val df = spark.read.schema(schema).json(Seq(
    """{"a":null}""",
    """{"a":true}""",
    """{"a":false}""",
  ).toDS)
  df.collect()
    .map(_.toString())
    .foreach(println(_))
  schema.printTreeString()
  df.schema.printTreeString()
}
 
{code}
 

 

Produces:

 
{code:java}
[null]
[true]
[false]
root
 |-- a: boolean (nullable = false)
root
 |-- a: boolean (nullable = true)
{code}
Bug: HistoryServer should set headers Content-Security-Policy (CSP)
Status: Open | Priority: Minor | Resolution: 
Created: 20/Sep/23 08:35
Affects Version/s: 3.5.0

Hi,

I provide spark history server UI to different users within my company.

The security team scan all URLs with this script : [https://github.com/santoru/shcheck] 
On the SHS site everything went smoothly except for Content-Security-Policy (CSP).
I tried with the policy: "default-src self" but it is not sufficient to get things working.
There is a massive amount of nonce/sha-256 to be included to make things work.

Even if I may find some dirty hack on nginx side (there is a reverse proxy) to avoid using unsafe-inline, it will not be Ok since it will report the security issue from spark to nginx which is worse.

It may be more appropriate to tweak jetty configuration to have everything set in the origin server.
Bug: ExecutorPodsLifecycleManager delete a pod multi times.
Status: Open | Priority: Minor | Resolution: 
Created: 16/Sep/23 01:20
Affects Version/s: 3.2.0

 Because `ExecutorPodsLifecycleManager`.`removedExecutorsCache` is not thread safe, will cause a pod deleted many times when  `ExecutorPodsLifecycleManager`.`onNewSnapshots` called by multi threads.
Bug: ProgressReport should include an accurate effective shuffle partition number
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Sep/23 23:17
Affects Version/s: 3.4.2

Currently, there is a numShufflePartitions "metric" reported in 
StateOperatorProgress part of the progress report. However, the number is reported by aggregating executors so in the case of task retry or speculative executor, the metric is higher than number of shuffle partitions for the query plan. Number of shuffle partitions can be useful for reporting purpose so having a metric is helpful.
Bug: SortMergeExec with Outer using join forgets sort information
Status: Open | Priority: Minor | Resolution: 
Created: 07/Sep/23 11:18
Affects Version/s: 3.4.1

When performing a 'using' join with a sort hint in a full outer, the ResolveNaturalAndUsingJoin will kick in and build a new join with Equality conditions and a Projection like this:
{quote}val joinedCols = joinPairs.map \{ case (l, r) => Alias(Coalesce(Seq(l, r)), l.name)() }
{quote}
There's nothing wrong with this per se, but, SortMergeJoinExec has it's output ordering for a full outer join as empty, even though these join pairs in their final coalesced form actually are ordered.

This means that code like this:
{quote}frames.reduceLeft(case (l, r) => l.join(r.hint("merge"), usingColumns = Seq("a", "b"), joinType = "outer"))
{quote}
Given a non empty list of frames, will not 'stream' without a shuffle step, as each join forgets its sort order.

Ideally this whole operation wouldn't require any shuffles if all the frames are grouped and sorted by the keys.

(Forgive the parens instead of brackets the code snippet please, Jira was inferring macros)
Bug: Remove misleading comment in parquetSchemaConverter
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 26/Sep/23 06:40
Affects Version/s: 3.5.0

I'm debugging a parquet issue and reading spark code as references. Happened to find a misleading comment which remains in the latest version as well.
{code:java}
Types
  .buildGroup(repetition).as(LogicalTypeAnnotation.listType())
  .addField(Types
    .buildGroup(REPEATED)
    // "array" is the name chosen by parquet-hive (1.7.0 and prior version)
    .addField(convertField(StructField("array", elementType, nullable)))
    .named("bag"))
  .named(field.name) {code}
the comment above is misleading since Hive always uses "array_element" as the name.

It is imported by this PR [https://github.com/apache/spark/pull/14399] and relates to this issue https://issues.apache.org/jira/browse/SPARK-16777

Furthermore, the parquet-hive module has been removed from the parquet-mr project https://issues.apache.org/jira/browse/PARQUET-1676 

I suggest removing this piece of comment and will submit a PR later.
Bug: Window-function support
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 28/Nov/22 07:45
Affects Version/s: 3.4.0

For compatibility, we need support for expressing window functions. Window functions are different from regular unresolved expressions as they need a window spec and are generally treated more like aggregate functions.

Part of this task is to identify if we can fully express the logic of window functions using unresolved functions with expression arguments that represent the window spec.

Only once this validation is done, we should consider adding a new plan operator / expression type.
Bug: Can insert NULL into hive table table with NOT NULL column
Status: Open | Priority: Critical | Resolution: 
Created: 10/Nov/22 22:13
Affects Version/s: 3.4.0

spark-sql> CREATE TABLE tttd(c1 int not null);
22/11/10 14:04:28 WARN ResolveSessionCatalog: A Hive serde table will be created as there is no table provider specified. You can set spark.sql.legacy.createHiveTableByDefault to false so that native data source table will be created instead.
22/11/10 14:04:28 WARN HiveMetaStore: Location: file:/Users/serge.rielau/spark/spark-warehouse/tttd specified for non-external table:tttd
Time taken: 0.078 seconds
spark-sql> INSERT INTO tttd VALUES(null);
Time taken: 0.36 seconds
spark-sql> SELECT * FROM tttd;
NULL
Time taken: 0.074 seconds, Fetched 1 row(s)
spark-sql> 
Does hive not support NOT NULL? That's fine, but then we should fail on CREATE TABLE
Bug: `hive-thriftserver` module compilation deadlock
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Nov/22 03:34
Affects Version/s: 3.4.0

[https://pipelines.actions.githubusercontent.com/serviceHosts/03398d36-4378-4d47-a936-fba0a5e8ccb9/_apis/pipelines/1/runs/194220/signedlogcontent/20?urlExpires=2022-11-09T03%3A31%3A51.5657953Z&urlSigningMethod=HMACV1&urlSignature=Jnn4uML8U79K6MF%2F%2BRUrrUbaOqxsqMA0DL0%2BQtrlBpM%3D]

 

I have seen it when compiling with Maven locally,  but I haven't investigated
Bug: Nondeterministic expressions have unstable values if they are children of CodegenFallback expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 11:25
Affects Version/s: 3.1.2

h2. Expectation

For a given row, Nondeterministic expressions are expected to have stable values.
{code:scala}
import org.apache.spark.sql.functions._
val df = sparkContext.parallelize(1 to 5).toDF("x")
val v1 = rand().*(lit(10000)).cast(IntegerType)
df.select(v1, v1).collect{code}
Returns a set like this:
|8777|8777|
|1357|1357|
|3435|3435|
|9204|9204|
|3870|3870|

where both columns always have the same value, but what that value is changes from row to row. This is different from the following:
{code:scala}
df.select(rand(), rand()).collect{code}
In this case, because the rand() calls are distinct, the values in both columns should be different.
h2. Problem

This expectation does not appear to be stable in the event that any subsequent expression is a CodegenFallback. This program:
{code:scala}
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._

val sparkSession = SparkSession.builder().getOrCreate()
val df = sparkSession.sparkContext.parallelize(1 to 5).toDF("x")
val v1 = rand().*(lit(10000)).cast(IntegerType)
val v2 = to_csv(struct(v1.as("a"))) // to_csv is CodegenFallback
df.select(v1, v1, v2, v2).collect {code}
produces output like this:
|8159|8159|8159|{color:#ff0000}2028{color}|
|8320|8320|8320|{color:#ff0000}1640{color}|
|7937|7937|7937|{color:#ff0000}769{color}|
|436|436|436|{color:#ff0000}8924{color}|
|8924|8924|2827|{color:#ff0000}2731{color}|

Not sure why the first call via the CodegenFallback path should be correct while subsequent calls aren't.
h2. Workaround

If the Nondeterministic expression is moved to a separate, earlier select() call, so the CodegenFallback instead only refers to a column reference, then the problem seems to go away. But this workaround may not be reliable if optimization is ever able to restructure adjacent select()s.
Bug: Improve output partitioning and ordering with AQE cache 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 10:47
Affects Version/s: 3.4.0

The cached plan in InMemoryRelation can be AdaptiveSparkPlanExec, however AdaptiveSparkPlanExec deos not specify its output partitioning and ordering. It causes unnecessary shuffle and local sort for downstream action.
Bug: Add config flag to control before of JSON partial results parsing in SPARK-40646
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/22 05:06
Affects Version/s: 3.4.0

This is a follow-up for https://issues.apache.org/jira/browse/SPARK-40646.

 

It was observed in internal benchmarks that the JSON partial results parsing can be 30% slower compared to parsing without the patch. I could not find a regression and the Apache Spark JSON benchmark results are very similar with and without SPARK-40646.

However, I would still like to add a config flag to enable/disable the feature in the case the regression is observed in users' queries.

Benchmark results are attached below.
Bug: Parameterized SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/22 10:14
Affects Version/s: 3.4.0

Enhance the Spark SQL API with support for parameterized SQL statements to improve security and reusability. Application developers will be able to write SQL with parameter markers whose values will be passed separately from the SQL code and interpreted as literals. This will help prevent SQL injection attacks for applications that generate SQL based on a user’s selections, which is often done via a user interface.
Bug: Support GENERATED ALWAYS AS syntax in create/replace table to create a generated column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/22 04:48
Affects Version/s: 3.4.0

Support GENERATED ALWAYS AS syntax for defining generated columns in create table and replace table.

For example,
{code:java}
CREATE TABLE default.example (
    time TIMESTAMP,
    date DATE GENERATED ALWAYS AS (CAST(time AS DATE))
)
{code}
 

This syntax is SQL standard and will enable defining generated columns in Spark SQL for data sources that support it.
Bug: High-order function: array_prepend
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/22 08:02
Affects Version/s: 3.4.0

refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_prepend.html



1, about the data type validation:
In Snowflake’s array_append, array_prepend and array_insert functions, the element data type does not need to match the data type of the existing elements in the array.

While in Spark, we want to leverage the same data type validation as array_remove.


2, about the NULL handling
Currently, SparkSQL, SnowSQL and PostgreSQL deal with NULL values in different ways.
Existing functions array_contains, array_position and array_remove in SparkSQL handle NULL in this way, if the input array or/and element is NULL, returns NULL. However, this behavior should be broken.

We should implement the NULL handling in array_prepend in this way:
2.1, if the array is NULL, returns NULL;
2.2 if the array is not NULL, the element is NULL, append the NULL value into the array
Bug: Push down filter through window when partitionSpec is empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Nov/22 12:01
Affects Version/s: 3.5.0

Sometimes, filter compares the rank-like window functions with number.

{code:java}
SELECT *, ROW_NUMBER() OVER(ORDER BY a) AS rn FROM Tab1 WHERE rn <= 5
{code}

We can create a Limit(5) and push down it as the child of Window.

{code:java}
SELECT *, ROW_NUMBER() OVER(ORDER BY a) AS rn FROM (SELECT * FROM Tab1 ORDER BY a LIMIT 5) t
{code}
Bug: Set executorEnv OMP_NUM_THREADS to be spark.task.cpus by default for spark executor JVM processes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/22 03:16
Affects Version/s: 3.3.1

Set executorEnv OMP_NUM_THREADS to be spark.task.cpus by default for spark executor JVM processes, this is for limiting the thread number for OpenBLAS routine to the number of cores assigned to this executor because some spark ML algorithms calls OpenBlAS via netlib-java
Bug: Follow-up on JDK-8180450
Status: Open | Priority: Major | Resolution: 
Created: 29/Nov/22 18:36
Affects Version/s: 3.2.2

Per [https://twitter.com/forked_franz/status/1597468851968831489]

We should follow-up on: [https://bugs.openjdk.org/browse/JDK-8180450]

There are two concrete tasks here:
 # Upgrade to Netty 4.1.84.
 # (Optional) Write a benchmark that exercises this code path. Anchoring this in the build will be a bit of challenge though.
 # Check if there are other places where this bug manifests itself.
Bug: Keep built-in file _metadata column nullable value consistent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 03:59
Affects Version/s: 3.3.0

In FileSourceStrategy, we add an Alias node to wrap the file metadata fields (e.g. file_name, file_size) in a NamedStruct ([here|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L279]). But `CreateNamedStruct` has an override `nullable` value `false` ([here|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala#L443]), which is different from the `_metadata` struct `nullable` value `true` ([here|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala#L467]). 

 

We should keep the nullable value the same, otherwise, the downstream optimization rules might use the nullability here and cause unexpected behaviors.
Bug: Better Spark UI scalability and Driver stability for large applications
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Nov/22 22:13
Affects Version/s: 3.4.0

After SPARK-18085, the Spark history server(SHS) becomes more scalable for processing large applications by supporting a persistent KV-store(LevelDB/RocksDB) as the storage layer.

As for the live Spark UI, all the data is still stored in memory, which can bring memory pressures to the Spark driver for large applications.

For better Spark UI scalability and Driver stability, I propose to
 * {*}Support storing all the UI data in a persistent KV store{*}. RocksDB/LevelDB provides low memory overhead. Their write/read performance is fast enough to serve the write/read workload for live UI. SHS can leverage the persistent KV store to fasten its startup.

 * *Support a new Protobuf serializer for all the UI data.* The new serializer is supposed to be faster, according to benchmarks. It will be the default serializer for the persistent KV store of live UI. As for event logs, it is optional. The current serializer for UI data is JSON. When writing persistent KV-store, there is GZip compression. Since there is compression support in RocksDB/LevelDB, the new serializer won’t compress the output before writing to the persistent KV store. Here is a benchmark of writing/reading 100,000 SQLExecutionUIData to/from RocksDB:

 
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|

I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the live UI.

SPIP: [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]

SPIP vote: https://lists.apache.org/thread/lom4zcob6237q6nnj46jylkzwmmsxvgj
Bug: Anti-join must not be pushed below aggregation with ambiguous predicates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Nov/22 15:20
Affects Version/s: 3.0.3

The following query should return a single row as all values for {{id}} except for the largest will be eliminated by the anti-join:

{code}
val ids = Seq(1, 2, 3).toDF("id").distinct()
val result = ids.withColumn("id", $"id" + 1).join(ids, Seq("id"), "left_anti").collect()
assert(result.length == 1)
{code}

Without the {{distinct()}}, the assertion is true. With {{distinct()}}, the assertion should still hold but is false.

Rule {{PushDownLeftSemiAntiJoin}} pushes the {{Join}} below the left {{Aggregate}} with join condition {{(id#750 + 1) = id#750}}, which can never be true.

{code}
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownLeftSemiAntiJoin ===
!Join LeftAnti, (id#752 = id#750)                  'Aggregate [id#750], [(id#750 + 1) AS id#752]
!:- Aggregate [id#750], [(id#750 + 1) AS id#752]   +- 'Join LeftAnti, ((id#750 + 1) = id#750)
!:  +- LocalRelation [id#750]                         :- LocalRelation [id#750]
!+- Aggregate [id#750], [id#750]                      +- Aggregate [id#750], [id#750]
!   +- LocalRelation [id#750]                            +- LocalRelation [id#750]
{code}

The optimizer then rightly removes the left-anti join altogether, returning the left child only.

Rule {{PushDownLeftSemiAntiJoin}} should not push down predicates that reference left *and* right child.
Bug: High-order function: array_insert
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/22 08:03
Affects Version/s: 3.4.0

refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_insert.html

1, about the data type validation:
In Snowflake’s array_append, array_prepend and array_insert functions, the element data type does not need to match the data type of the existing elements in the array.

While in Spark, we want to leverage the same data type validation as array_remove.

2, about the NULL handling
Currently, SparkSQL, SnowSQL and PostgreSQL deal with NULL values in different ways.
Existing functions array_contains, array_position and array_remove in SparkSQL handle NULL in this way, if the input array or/and element is NULL, returns NULL. However, this behavior should be broken.

We should implement the NULL handling in array_insert in this way:
2.1, if the array is NULL, returns NULL;
2.2 if the array is not NULL, the element is NULL, append the NULL value into the array
Bug: Regression in IntegralDivide returning null instead of 0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Nov/22 02:57
Affects Version/s: 3.4.0

There seems to be a regression in Spark 3.4 Integral Divide

 
{code:java}
scala> val df = Seq("0.5944910","0.3314242").toDF("a")
df: org.apache.spark.sql.DataFrame = [a: string]

scala> df.selectExpr("cast(a as decimal(7,7)) div 100").show
+---------------------------------+
|(CAST(a AS DECIMAL(7,7)) div 100)|
+---------------------------------+
|                             null|
|                             null|
+---------------------------------+
{code}
 

While in Spark 3.3.0
{code:java}
scala> val df = Seq("0.5944910","0.3314242").toDF("a")
df: org.apache.spark.sql.DataFrame = [a: string]

scala> df.selectExpr("cast(a as decimal(7,7)) div 100").show
+---------------------------------+
|(CAST(a AS DECIMAL(7,7)) div 100)|
+---------------------------------+
|                                0|
|                                0|
+---------------------------------+
{code}
Bug: High-order function: array_append
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/22 08:01
Affects Version/s: 3.4.0

refer to https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_append.html


1, about the data type validation:
In Snowflake’s array_append, array_prepend and array_insert functions, the element data type does not need to match the data type of the existing elements in the array.

While in Spark, we want to leverage the same data type validation as array_remove.


2, about the NULL handling
Currently, SparkSQL, SnowSQL and PostgreSQL deal with NULL values in different ways.
Existing functions array_contains, array_position and array_remove in SparkSQL handle NULL in this way, if the input array or/and element is NULL, returns NULL. However, this behavior should be broken.

We should implement the NULL handling in array_append in this way:
2.1, if the array is NULL, returns NULL;
2.2 if the array is not NULL, the element is NULL, append the NULL value into the array