av9ash/GitBugs huggingface dataset:
Bug: JoinWith on non-inner joins adds an unnecessary shuffle
Status: Open | Priority: Major | Resolution: 
Created: 13/Feb/24 22:04
Affects Version/s: 3.5.0

Assume the following code:
{code:java}
final case class AAA(i: Int, s: String)

val ds1 = spark.createDataset(Seq(
  AAA(1, "s")
)).repartition($"i")

val ds2 = spark.createDataset(Seq(
  AAA(2, "sd")
)).repartition($"i")

val value = ds1.as("a")
  .joinWith(ds2.as("b"), $"a.i" === $"b.i", "left")

val plan = value.queryExecution.executedPlan
println(plan) {code}
This code creates the following plan:
{code:java}
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [_1#30.i], [_2#31.i], LeftOuter
   :- Sort [_1#30.i ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(_1#30.i, 200), ENSURE_REQUIREMENTS, [id=#29]
   :     +- Project [named_struct(i, i#2, s, s#3) AS _1#30]
   :        +- Exchange hashpartitioning(i#2, 200), REPARTITION_BY_COL, [id=#21]
   :           +- LocalTableScan [i#2, s#3]
   +- Sort [_2#31.i ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(_2#31.i, 200), ENSURE_REQUIREMENTS, [id=#30]
         +- Project [named_struct(i, i#12, s, s#13) AS _2#31]
            +- Exchange hashpartitioning(i#12, 200), REPARTITION_BY_COL, [id=#24]
               +- LocalTableScan [i#12, s#13] {code}
Note how the plan has 2 `Exchange` steps for each side. even though the datasets were already partitioned correctly.

This of course is an unnecessary shuffle and should be skipped
Bug: PushDownUtils uses FileScanBuilder instead of SupportsPushDownCatalystFilters trait
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/24 12:36
Affects Version/s: 3.5.0

It could use an existing more generic interface looking like it was created for that reason, but uses a narrower type forcing you to extend
FileScanBuilder when implementing a ScanBuilder.
Bug: Unexpected nulls when casting decimal values in specific cases
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 22/Feb/24 17:49
Affects Version/s: 3.4.1

In specific cases, casting decimal values can result in `null` values where no overflow exists.

The cases appear very specific, and I don't have the depth of knowledge to generalize this issue, so here is a simple spark-shell reproduction:

*Setup:*

{code:scala}
scala> val ds = 0.to(23386).map(x => if (x > 13878) ("A", x) else ("B", x)).toDS
ds: org.apache.spark.sql.Dataset[(String, Int)] = [_1: string, _2: int]

scala> ds.createOrReplaceTempView("t")
{code}
 
*Spark 3.2.1 behaviour (correct):*

{code:scala}
scala> spark.sql("select CAST(SUM(1.00000000000000) AS DECIMAL(28,14)) as ct FROM t GROUP BY `_1` ORDER BY ct ASC").show()
+--------------------+
|                  ct|
+--------------------+
| 9508.00000000000000|
|13879.00000000000000|
+--------------------+
{code}

*Spark 3.4.1 / Spark 3.5.0 behaviour:*

{code:scala}
scala> spark.sql("select CAST(SUM(1.00000000000000) AS DECIMAL(28,14)) as ct FROM t GROUP BY `_1` ORDER BY ct ASC").show()
+-------------------+
|                 ct|
+-------------------+
|               null|
|9508.00000000000000|
+-------------------+
{code}

This is fairly delicate:
 - removing the {{ORDER BY}} clause produces the correct result
 - removing the {{CAST}} produces the correct result
 - changing the number of 0s in the argument to {{SUM}} produces the correct result
 - setting {{spark.ansi.enabled}} to {{true}} produces the correct result (and does not throw an error)

Also, removing the {{ORDER BY}}, but writing {{ds}} to a parquet will also result in the unexpected nulls.

Please let me know if you need additional information.

We are also interested in understanding whether setting {{spark.ansi.enabled}} can be considered a reliable workaround to this issue prior to a fix being released, if possible.

Text files that include {{explain()}} output attached.
Bug: Show metrics of the physical plan of RDDScanExec's internal RDD in the history server
Status: Open | Priority: Major | Resolution: 
Created: 09/Feb/24 17:41
Affects Version/s: 3.4.0

The RDDScanExec wraps an internal RDD (as below). In our environment, we find that this RDD is usually produced by some very large physical plans which contain quite a few physical nodes. Those nodes may have various metrics which are very useful for us to know what the execution looks like and any room for optimization, etc.

 
{code:java}
case class RDDScanExec(
    output: Seq[Attribute],
    rdd: RDD[InternalRow],     <-- this field
    name: String, {code}
 

However, the physical plan and the metrics are invisible from the SQL DAG in the Spark History Server. As it is an "existing RDD", the physical plan may be found from some previous SQL. The metrics are not visible from that previous SQL either. This is because the "definition" of these metrics are reported along with the SparkListenerSQLExecutionStart event of the "previous SQL" (where the physical plan of the RDDScanExec.rdd is in), but the metric values are reported from the SparkListenerTaskEnd event of the tasks which are attached to the SQL with RDDScanExec.

!ScanExistingRDD.jpg|width=336,height=296!

 

Do we consider showing the physical plan and metrics of the RDDScanExec.rdd (the "Scan Existing RDD" node in the above DAG). For example, it may be shown as a "leg" (similar to but not the same as a child) in the DAG, or something else that may show the physical plan and metrics?
Bug: Deflake "interrupt tag" at SparkSessionE2ESuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/24 01:44
Affects Version/s: 4.0.0

{code}
- interrupt tag *** FAILED ***
  The code passed to eventually never returned normally. Attempted 30 times over 20.037421464999998 seconds. Last failure message: ListBuffer("2beba4ac-a994-45f5-bd46-fca3e43fb5ef") had length 1 instead of expected length 2 Interrupted operations: ListBuffer(2beba4ac-a994-45f5-bd46-fca3e43fb5ef).. (SparkSessionE2ESuite.scala:216)
{code}

https://github.com/apache/spark/actions/runs/7959951623/job/21727929211
Bug: XML: Ignore commented Row Tags in XML tokenizer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/24 23:43
Affects Version/s: 4.0.0

The following returns rows that was within comments:

{{}}
{code:java}
// BUG: rowTag in comment -- incorrectly processed display(spark.read.xml(write("""<root> <ROW><a>1</a></ROW> <!--<ROW><a>2</a></ROW>--> </root>"""))){code}
{{}}

This has been reported before:[!https://github.com/fluidicon.png!How to Ignore XML comments like this · Issue #208 · databricks/spark-xml|https://github.com/databricks/spark-xml/issues/208]
{{}}
Bug: Kubernetes: support csi driver for volume type
Status: Open | Priority: Major | Resolution: 
Created: 08/Feb/24 16:42
Affects Version/s: 3.5.0

Today Spark supports the following types of Kubernetes [volumes|https://kubernetes.io/docs/concepts/storage/volumes/]: hostPath, emptyDir, nfs and persistentVolumeClaim.

In our case, Kubernetes cluster is multi-tenant and we cannot make cluster-wide changes when deploying our application to the Kubernetes cluster. Our application requires static shared file system. So, we cannot use hostPath (don't have control of hosting VMs) and persistentVolumeClaim (requires cluster-wide change when deploying PV). Our security department does not allow nfs. 

What would help in our case, is the use of csi driver (taken from here: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/deploy/example/e2e_usage.md#option3-inline-volume):
{code:java}
kind: Pod
apiVersion: v1
metadata:
  name: nginx-azurefile-inline-volume
spec:
  nodeSelector:
    "kubernetes.io/os": linux
  containers:
    - image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
      name: nginx-azurefile
      command:
        - "/bin/bash"
        - "-c"
        - set -euo pipefail; while true; do echo $(date) >> /mnt/azurefile/outfile; sleep 1; done
      volumeMounts:
        - name: persistent-storage
          mountPath: "/mnt/azurefile"
          readOnly: false
  volumes:
    - name: persistent-storage
      csi:
        driver: file.csi.azure.com
        volumeAttributes:
          shareName: EXISTING_SHARE_NAME  # required
          secretName: azure-secret  # required
          mountOptions: "dir_mode=0777,file_mode=0777,cache=strict,actimeo=30,nosharesock"  # optional {code}
Bug: Pushdown Verification in Optimizer.scala should support changed data types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/24 19:07
Affects Version/s: 3.3.0

When pushing a filter down in a union the data type may not match exactly if the filter was constructed using the child dataframe reference. This is because the unions output is updated with a structype merge of union which can turn non-nullable to nullable. These are still the same column despite the different nullability so the filter should be safe to push down. As it currently stands we get an exception.
Bug: contains, startswith, endswith (binary & lowercase collation only)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/24 10:18
Affects Version/s: 4.0.0

Refactored built-in string functions to enable collation support for: {_}contains{_}, {_}startsWith{_}, {_}endsWith{_}. Spark SQL users should now be able to use COLLATE within arguments for built-in string functions: CONTAINS, STARTSWITH, ENDSWITH in Spark SQL queries. Note: CONTAINS implementation for non-binary collations is a separate subtask (SPARK-47248).
Bug: Disable bucketing on collated collumns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Feb/24 11:47
Affects Version/s: 4.0.0

*What changes were proposed in this pull request?*
Disable bucketing on columns that are non default collated.

*Why are the changes needed?*
With current implementation bucketIds are generated from a string value where each unique string guarantees unique id, but when collation is turned on, this is not the case.
Bug: Unable to create PySpark dataframe containing Variant columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/24 01:14
Affects Version/s: 3.5.0

Trying to create a dataframe containing a variant type results in:

AssertionError: Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': "Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters:

{'error': 'variant'}

"}
Bug: Fix incorrect error class in create_data_source.py
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/24 20:59
Affects Version/s: 4.0.0

This is part of the effort of SPARK-44076 that enable python user to develop data source in python and make python accessible to wider python community.

Error class "PYTHON_DATA_SOURCE_CREATE_ERROR" and "PYTHON_DATA_SOURCE_METHOD_NOT_IMPLEMENTED" used in create_data_source.py doesn't exist and will throw error class not found error which make error message confusing to customer. Need to add the corresponding error class to error-class.py or use existing error class.
Bug: Subquery rewrite inside an aggregation makes an aggregation invalid
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/24 04:48
Affects Version/s: 3.5.0

When an in/exists-subquery appears inside an aggregate expression within a top-level GROUP BY, it gets rewritten and a new `exists` variable is introduced. However, this variable is incorrectly handled in aggregation. For example, consider the following query:

```
SELECT
CASE
WHEN t1.id IN (SELECT id FROM t2) THEN 10
ELSE -10
END AS v1
FROM t1
GROUP BY t1.id;
```
 
Executing it leads to the following error:
```
java.lang.IllegalArgumentException: Cannot find column index for attribute 'exists#844' in: Map()
```
Bug: EXECUTE IMMEDIATE USING does not recognize session variable names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/24 18:54
Affects Version/s: 4.0.0

{noformat}
DECLARE parm = 'Hello';

EXECUTE IMMEDIATE 'SELECT :parm' USING parm;
[ALL_PARAMETERS_MUST_BE_NAMED] Using name parameterized queries requires all parameters to be named. Parameters missing names: "parm". SQLSTATE: 07001

EXECUTE IMMEDIATE 'SELECT :parm' USING parm AS parm;
Hello

{noformat}
variables are like column references, they act as their own aliases and thus should not be required to be named to associate with a named parameter with the same name.

Note that unlike for pySpark this should be case insensitive (haven't verified).
Bug: Is it possible to dynamically add backend service to ingress with Kubernetes?
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 28/Feb/24 01:18
Affects Version/s: 4.0.0

spark on k8s runs multiple spark apps at the same time. proxy/[sparkappid] path, forwarding to different sparkapp ui console based on sparkappid. spark apps are dynamically added and decreased. ingress Dynamically adds spark svc.

[sparkappid]_svc == spark svc name

[https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-ingress-guide-nginx-example.html]

[~Qin Yao]
Bug: Add COLLATION_ENABLED config flag
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Feb/24 12:02
Affects Version/s: 4.0.0

*What changes were proposed in this pull request?*
This PR adds COLLATION_ENABLED config to `SQLConf` and introduces new error class `COLLATION_SUPPORT_NOT_ENABLED` to appropriately report error on usage of feature under development. 

*Why are the changes needed?*
We want to make collations configurable on this flag. These changes disable usage of `collate` and `collation` functions, along with any `COLLATE` syntax when the flag is set to false. By default, the flag is set to false.
Bug: Cached SQL plan do not display final AQE plan in explain string
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/24 01:05
Affects Version/s: 3.4.2

AQE plan is expected to display final plan after execution. This is not true for cached SQL plan: it will show the initial plan instead. This behavior change is introduced in [https://github.com/apache/spark/pull/40812] it tried to fix the concurrency issue with cached plan. 

*In short, the plan used to executed and the plan used to explain is not the same instance, thus causing the inconsistency.*

 

I don't have a clear idea how yet
 * maybe we just a coarse granularity lock in explain?
 * make innerChildren a function: clone the initial plan, every time checked for whether the original AQE plan is finalized (making the final flag atomic first, of course), if no return the cloned initial plan, if it's finalized, clone the final plan and return that one. But still this won't be able to reflect the AQE plan in real time, in a concurrent situation, but at least we have initial version and final version.

 

A simple repro:
{code:java}
d1 = spark.range(1000).withColumn("key", expr("id % 100")).groupBy("key").agg({"key": "count"})
cached_d2 = d1.cache()
df = cached_d2.filter("key > 10")
df.collect() {code}
{code:java}
>>> df.explain()
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(1) Filter (isnotnull(key#4L) AND (key#4L > 10))
   +- TableCacheQueryStage 0
      +- InMemoryTableScan [key#4L, count(key)#10L], [isnotnull(key#4L), (key#4L > 10)]
            +- InMemoryRelation [key#4L, count(key)#10L], StorageLevel(disk, memory, deserialized, 1 replicas)
                  +- AdaptiveSparkPlan isFinalPlan=false
                     +- HashAggregate(keys=[key#4L], functions=[count(key#4L)])
                        +- Exchange hashpartitioning(key#4L, 200), ENSURE_REQUIREMENTS, [plan_id=24]
                           +- HashAggregate(keys=[key#4L], functions=[partial_count(key#4L)])
                              +- Project [(id#2L % 100) AS key#4L]
                                 +- Range (0, 1000, step=1, splits=10)
+- == Initial Plan ==
   Filter (isnotnull(key#4L) AND (key#4L > 10))
   +- InMemoryTableScan [key#4L, count(key)#10L], [isnotnull(key#4L), (key#4L > 10)]
         +- InMemoryRelation [key#4L, count(key)#10L], StorageLevel(disk, memory, deserialized, 1 replicas)
               +- AdaptiveSparkPlan isFinalPlan=false
                  +- HashAggregate(keys=[key#4L], functions=[count(key#4L)])
                     +- Exchange hashpartitioning(key#4L, 200), ENSURE_REQUIREMENTS, [plan_id=24]
                        +- HashAggregate(keys=[key#4L], functions=[partial_count(key#4L)])
                           +- Project [(id#2L % 100) AS key#4L]
                              +- Range (0, 1000, step=1, splits=10){code}
Bug: Fix ignored PySpark Connect string collation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/24 14:17
Affects Version/s: 4.0.0

When using Connect with PySpark, string collation silently gets dropped:
{code:java}
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.sql("select 'abc' collate 'UNICODE'")
DataFrame[collate(abc): string]
>>> from pyspark.sql.types import StructType, StringType, StructField
>>> spark.createDataFrame([], StructType([StructField('id', StringType(2))]))
DataFrame[id: string]
{code}
Instead of "string" type in dataframe, we should be seeing "string COLLATE 'UNICODE'".
Bug: Spark-UI stages and other tabs not accessible in standalone mode when reverse-proxy is enabled
Status: Open | Priority: Major | Resolution: 
Created: 29/Feb/24 21:27
Affects Version/s: 3.4.0

When enabled reverse proxy in master and worker configOptions. We're not able to access different tabs available in spark UI e.g.(stages, environment, storage etc.)

We're deploying spark through bitnami helm chart : [https://github.com/bitnami/charts/tree/main/bitnami/spark]

Name and Version

bitnami/spark - 6.0.0

What steps will reproduce the bug?

Kubernetes Version: 1.25
Spark: 3.4.2
Helm chart: 6.0.0

Steps to reproduce:
After installing the chart Spark Cluster(Master and worker) UI is available at:

!Screenshot 2024-02-09 at 5.04.49 PM.png|width=609,height=235!
[https://spark.staging.abc.com/]

We are able to access running application by click on applicationID under Running Applications link:

!Screenshot 2024-02-09 at 5.05.48 PM.png|width=620,height=239!

We can access spark UI by clicking Application Detail UI:

We are taken to jobs tab when we click on Application Detail UI

!Screenshot 2024-02-09 at 5.09.28 PM.png|width=617,height=177!
URL looks like: [https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]

When we click any of the tab from spark UI e.g. stages or environment etc, it takes us back to spark cluster UI page
We noticed that endpoint changes to 

!Screenshot 2024-02-09 at 5.13.01 PM.png|width=617,height=191!
[https://spark.staging.abc.com/stages/] 
instead of 
[https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]

 

Are you using any custom parameters or values?

Configurations set in values.yaml
```
master:
  configOptions:
    -Dspark.ui.reverseProxy=true
    -Dspark.ui.reverseProxyUrl=[https://spark.staging.abc.com|https://spark.staging.abc.com/]

worker:
  configOptions:
    -Dspark.ui.reverseProxy=true
    -Dspark.ui.reverseProxyUrl=[https://spark.staging.abc.com|https://spark.staging.abc.com/]

service:
  type: ClusterIP
  ports:
    http: 8080
    https: 443
    cluster: 7077

ingress:

  enabled: true
  pathType: ImplementationSpecific
  apiVersion: ""
  hostname: spark.staging.abc.com
  ingressClassName: "staging"
  path: /
```

 

What is the expected behavior?

Expected behaviour is that when I click on stages tab, instead of taking me to 
[https://spark.staging.abc.com/stages/] 
it should take me to following URL:
[https://spark.staging.abc.com/proxy/app-20240208103209-0030/stages/]

What do you see instead?

current behaviour is it takes me to URL: [https://spark.staging.abc.com/stages/] , which shows spark cluster UI with master and worker details
Bug: spark.files are not copied to Spark Executors in client session mode
Status: Open | Priority: Major | Resolution: 
Created: 29/Feb/24 10:56
Affects Version/s: 2.4.5

Hi folks! And thank you for you brilliant project.

I'm using PySpark for a couple of years and last week I have encountered with a problem that I don't know how to solve.

 

{{I'm using Spark session in client mode (with .master("yarn")) and I have a file (JKS certificate) that I have to copy to all executor node of my Spark cluster (it's needed for another library using Spark).}}

{{I have read some docs and have find out that I can use SparkSession.builder.config("spark.files", "/path/to/my/precious.file") or SparkContext.addFile("/path/to/my/precious.file") – and after that call SparkFiles.get("precious.file") – and it will return a path that to copied file that could be used on executors.}}

Alas, SparkFiles.get("precious.file") gives me a path like

{{'/tmp/spark-d64b1d73-3745-4362-adb1-586a6228e7a5/userFiles-2f5a6735-f297-489b-b6e1-6ff9b0c3a214/precious.file'}}

such directory with this file appears on my driver node – but does not appear on executor nodes.

How can It be fixed? Isnt't here a bug in Spark?

P.S. I've tried putting the file on HDFS but it gives the same result: it appears on driver node and not on executors.

P.P.S. Here's some code with which I was trying to copy files to executors:

 

{{from pyspark.sql import SparkSession}}
{{from pyspark import SparkFiles}}

{{spark = SparkSession.builder.enableHiveSupport()}}
{{.config("spark.files", "hdfs:///tmp/dummy.txt")}}
{{.master("yarn").getOrCreate()}}
{{file_system = FileSystem(spark)}}

{{# I have uploaded a file "dummy.txt" to HDFS (to hdfs:///tmp/dummy.txt)}}
{{# I want this file from HDFS to be copied to all executors.}}
{{# spark.sparkContext.addFile("hdfs:///tmp/dummy.txt")}}

{{# Let's look where it was uploaded...}}
{{path_on_exectors = SparkFiles.get("dummy.txt")}}
{{print(path_on_exectors)}}

{{# > '/tmp/spark-6e61ed03-6866-4863-adb6-b7345836caf3/userFiles-28d28f93-e4df-44c3-a1e5-7411d02d4cf5/dummy.txt'}}
{{# This path was created on my driver node but DOES NOT EXIST on my executors.}}
{{# What can I do to fix it?}}

{{# (Quitting programming, moving to Somalia and becoming a pirate is not an option yet).}}
Bug: log4j race condition during shutdown
Status: Open | Priority: Major | Resolution: 
Created: 29/Feb/24 00:46
Affects Version/s: 3.3.0

There is a race condition during shutdown which can result in a few different errors:
 * ERROR Attempted to append to non-started appender
 *  ERROR Unable to write to stream

Since I've only seen it during stop() triggered within a shutdown hook I believe this is caused by the parallel execution of shutdown hooks (see [https://stackoverflow.com/questions/17400136/how-to-log-within-shutdown-hooks-with-log4j2] )
Bug: Fix Pyspark collated string conversion error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/24 16:06
Affects Version/s: 4.0.0

When running Pyspark shell in non-Spark Connect mode, query "SELECT 'abc' COLLATE 'UCS_BASIC_LCASE'" produces the following error:
{code:java}
AssertionError: Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': "Undefined error message parameter for error class: CANNOT_PARSE_DATATYPE. Parameters: {'error': 'string(UCS_BASIC_LCASE)'}"}
{code}
Bug: Convert _LEGACY_ERROR_TEMP_0035 (unsupported hive feature)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/24 16:29
Affects Version/s: 4.0.0

Old:



> GRANT ROLE;

_LEGACY_ERROR_TEMP_0035

Operation not allowed: grant role. (line 1, pos 0)

 

New: 

error class: HIVE_OPERATION_NOT_SUPPORTED

The Hive operation <operation> is not supported. (line 1, pos 0)
 
sqlstate: 0A000