av9ash/GitBugs huggingface dataset:
Bug: Multiple calls to GET_JSON_OBJECT with the same JSON str should parse it just one time
Status: Open | Priority: Major | Resolution: 
Created: 01/Apr/24 16:30
Affects Version/s: 3.5.2

For a query like the following:

{{SELECT}}
{{  GET_JSON_OBJECT(json_col, '$.a.b'),}}
{{  GET_JSON_OBJECT(json_col, '$.a.c')}}
{{FROM t}}

SparkSQL would generate a plan that parse the json_col twice.

Ideally, SparkSQL should only parse the `json_col` once.  The optimizer should find out the common JSON parsing, and modify the plan to parse the JSON once, get the result out, and flatten it back.

An alternative way to support this is the ":" notation (JSON Path) as in other systems where the query optimizer will automatically share a single JSON parsing.
Bug: Generate stable and uniquely-named classes in the Catalyst code generator
Status: Open | Priority: Minor | Resolution: 
Created: 09/Apr/24 09:23
Affects Version/s: 3.5.1

Code generated by Catalyst can not be executed on [GraalVM Native Image|https://www.graalvm.org/] for the following two reasons:
 # The generated classes for Scala UDFs are not stable so they can not be [pre-defined in a native image|https://www.graalvm.org/latest/reference-manual/native-image/metadata/ExperimentalAgentOptions/#support-for-predefined-classes]. The instability comes from the address of hidden functions ({{{}/0x<address>{}}}) that [end up in the generated code|https://github.com/apache/spark/blob/v3.5.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala#L1173].  These addresses are completely random and therefore provide little value to the end user. 
 # The name of the generated class is always the same ({{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass{}}}) which makes it impossible to [pre-define multiple such classes in a native image|https://github.com/oracle/graal/blob/vm-ce-23.1.2/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ClassPredefinitionFeature.java#L153].  We can fix this by generating the class name in the following format: {{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass$<codeHash>{}}}.

It would be useful enable Spark workloads on GraalVM Native Image. With Native Image, one can generate low-memory Spark binaries for their frequent workloads. We would also like to make all Spark benchmarks in the [renaissance benchmark suite|https://renaissance.dev/] run as native images.
Bug: Text Datasource supports zstd compression codec
Status: Open | Priority: Minor | Resolution: 
Created: 12/Apr/24 07:23
Affects Version/s: 3.5.0




{code:java}
drop table if exists tmp_compressed_datasource;CREATE TABLE tmp_compressed_datasource (key STRING) USING TEXT OPTIONS (compression 'zstd');INSERT INTO TABLE tmp_compressed_datasource values ('a'),('b'),('c');select * from tmp_compressed_datasource ;
{code}



{code:java}
java.lang.IllegalArgumentException: Codec [zstd] is not available. Known codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.
    at org.apache.spark.sql.catalyst.util.CompressionCodecs$.getCodecClassName(CompressionCodecs.scala:53)
    at org.apache.spark.sql.execution.datasources.text.TextOptions.$anonfun$compressionCodec$1(TextOptions.scala:38)
    at scala.Option.map(Option.scala:230)
    at org.apache.spark.sql.execution.datasources.text.TextOptions.<init>(TextOptions.scala:38)
    at org.apache.spark.sql.execution.datasources.text.TextOptions.<init>(TextOptions.scala:33)
    at org.apache.spark.sql.execution.datasources.text.TextFileFormat.prepareWrite(TextFileFormat.scala:72)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:123)
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:190)

{code}
Bug: When creating a table through ThriftServer, set the table owner to the connected user
Status: Open | Priority: Minor | Resolution: 
Created: 02/Apr/24 06:43
Affects Version/s: 3.4.2

1. Start spark ThriftServer with user user_a.
2. Use beeline to connect to ThriftServer, and set username to user_b.
3. Create a table through beeline.
4. Use "desc formatted table" to view the table. The owner of the table is user_a, not user_b.

 
Need to change "org.apache.spark.sql.hive.client.HiveClientImpl#userName" from val to def
Bug: Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Apr/24 07:06
Affects Version/s: 4.0.0

1.User Story:
Our data analysts and data scientists use Jupyter notebooks provisioned on Kubernetes (k8s) with limited CPU/memory resources to run Spark-shell/pyspark in the terminal via Yarn Client mode. However, Yarn Client mode consumes significant local memory if the job is heavy, and the total resource pool of k8s for notebooks is limited. To leverage the abundant resources of our Hadoop cluster for scalability purposes, we aim to utilize SparkConnect. This allows the driver on Yarn with SparkConnectService started and uses SparkConnect client to connect to the remote driver.

To provide a seamless experience with one command startup for both server and client, we've wrapped the following processes in one script:

1) Start a local coordinator server (implemented by us, not in this PR) with a specified port.
2) Start SparkConnectServer by spark-submit via Yarn Cluster mode with user-input Spark configurations and the local coordinator server's address and port. Append an additional listener class in the configuration for SparkConnectService callback with the actual address and port on Yarn to the coordinator server.
3) Wait for the coordinator server to receive the address callback from the SparkConnectService on Yarn and export the real address.
4) Start the client (pyspark --remote) with the remote address.

Finally, a remote SparkConnect Server is started on Yarn with a local SparkConnect client connected. Users no longer need to start the server beforehand and connect to the remote server after they manually explore the address on Yarn.

2.Problem statement of this change:
1) The specified port for the SparkConnectService GRPC server might be occupied on the node of the Hadoop Cluster. To increase the success rate of startup, it needs to retry on conflicts rather than fail directly.
2) Because the final binding port could be uncertain based on #1 and the remote address is unpredictable on Yarn, we need to retrieve the address and port programmatically and inject it automatically on the start of `pyspark --remote`. The SparkConnectService needs to communicate its location back to the launcher side.
Bug: Avoid orNull in TypedConfigBuilder and OptionalConfigEntry
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Apr/24 02:07
Affects Version/s: 4.0.0

Configuration values/keys cannot be nulls. We should fix:

{code}
diff --git a/core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala b/core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala
index 1f19e9444d38..d06535722625 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala
@@ -94,7 +94,7 @@ private[spark] class TypedConfigBuilder[T](
   import ConfigHelpers._

   def this(parent: ConfigBuilder, converter: String => T) = {
-    this(parent, converter, Option(_).map(_.toString).orNull)
+    this(parent, converter, { v: T => v.toString })
   }

   /** Apply a transformation to the user-provided values of the config entry. */
{code}
Bug: Pandas API groupby with multi-agg-relabel ignores as_index=False
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Apr/24 13:45
Affects Version/s: 3.5.1

A Pandas API DataFrame groupby with as_index=False and a multilevel relabeling, such as
{code:java}
from pyspark import pandas as ps
ps.DataFrame({"a": [0, 0], "b": [0, 1]}).groupby("a", as_index=False).agg(b_max=("b", "max")){code}
fails to include group keys in the resulting DataFrame. This diverges from expected behavior as well as from the behavior of native Pandas, e.g.

*actual*
{code:java}
   b_max
0      1 {code}
*expected*
{code:java}
   a  b_max
0  0      1 {code}
 

A possible fix is to prepend groupby key columns to {{*order*}} and {{*columns*}} before filtering here:  [https://github.com/apache/spark/blob/master/python/pyspark/pandas/groupby.py#L327-L328]
Bug: Specifications for Apache Spark hadoop Avro append operation
Status: Open | Priority: Minor | Resolution: 
Created: 26/Apr/24 12:59
Affects Version/s: 3.4.3

Consider a path /a/b/c 

Assume, I write the avro to folder using apache spark.

After it is written, Assume I try to append dataset to this to folder. 

I want to see the specification of what happens in case of append.

After doing PoC, I found out that when dataet which is appended is having same schema as of existing data, data gets just appended. But I want to see clear docs of what happens exactly in case of append.

I am attaching my testing java code.
Bug: pandas-on-spark DataFrame.concat will not join a Pandas dataframe and raises a misleading error
Status: Open | Priority: Minor | Resolution: 
Created: 25/Apr/24 18:40
Affects Version/s: 3.4.3

The `concat` method has a strict type check, that raises a misleading error:

!image-2024-04-25-11-33-29-208.png!
Note that the type raised is of `objs`, rather than `obj`, so a list of various objects will say that it cannot concatenate objects of type list, rather than the failed internal types.

 

Additionally, this strictly checks for pandas-on-spark Series and DataFrames; since both objects will happily convert a naive Pandas object, something like

 

objs = [DataFrame(x) if isinstance(x, pd.Dataframe) else Series(x) if isinstance(x, pd.Series) else x for x in objs] 

would trivially make this work in those cases and prevent a different strange error reporting that a dataframe wasn't valid in a dataframe concatenation.
Bug: Pandas-on-Spark incompletely implements DataFrame.drop
Status: Open | Priority: Minor | Resolution: 
Created: 25/Apr/24 18:27
Affects Version/s: 3.4.3

For Pandas v1.0+, `drop` supports the `errors` kwarg:

[https://pandas.pydata.org/pandas-docs/version/1.0/reference/api/pandas.DataFrame.drop.html]

 

Pandas-on-Spark does not implement it. This is especially glaring since the pyspark drop is a no-op on absent columns, behaving like `errors='ignore'`, so _extra_ work needed to be done to implement the raise behaviour.
Bug: Pandas-on-Spark incompletely implements merge methods
Status: Open | Priority: Minor | Resolution: 
Created: 25/Apr/24 18:21
Affects Version/s: 3.4.3

For Pandas >= 1.2 ( [https://pandas.pydata.org/pandas-docs/version/1.2/reference/api/pandas.DataFrame.merge.html] ) (current = 2.2) how implements method "cross". which is absent.

 

This breaks API compatibility.
Bug: Support retrieving the real SparkConnectService GRPC address and port programmatically when running on Yarn
Status: Open | Priority: Minor | Resolution: 
Created: 23/Apr/24 07:02
Affects Version/s: 4.0.0

1. {*}User Story{*}:
Our data analysts and data scientists use Jupyter notebooks provisioned on Kubernetes (k8s) with limited CPU/memory resources to run Spark-shell/pyspark in the terminal via Yarn Client mode.
However, Yarn Client mode consumes significant local memory if the job is heavy, and the total resource pool of k8s for notebooks is limited.
To leverage the abundant resources of our Hadoop cluster for scalability purposes, we aim to utilize SparkConnect.
This allows the driver on Yarn with SparkConnectService started and uses SparkConnect client to connect to the remote driver.

To provide a seamless experience with one command startup for both server and client, we've wrapped the following processes in one script:

1). Start a local coordinator server (implemented by us internally, not in this PR) in the host of jupyter notebook.
2). Start SparkConnectServer by spark-submit via Yarn Cluster mode with user-input Spark configurations and the local coordinator server's address and port.
    Append an additional listener class in the configuration for SparkConnectService callback with the actual address and port on Yarn to the coordinator server.
3). Wait for the coordinator server to receive the address callback from the SparkConnectService on Yarn and export the real address.
4). Start the client (pyspark --remote $callback_address) with the remote address.

2. {*}Problem statement of this change{*}:
1). The specified port for the SparkConnectService GRPC server might be occupied on the node of the Hadoop Cluster.
    To increase the success rate of startup, it needs to retry on conflicts rather than fail directly.
2). Because the final binding port could be uncertain based on #1 when retry and the remote address is unpredictable on Yarn,
    we need to retrieve the address and port programmatically and inject it automatically on the start of `pyspark --remote`.
    To get the address of SparkConnectService on Yarn programmatically, The SparkConnectService needs to communicate its location back to the launcher side.
Bug: StageFailed event should attach the exception chain
Status: Open | Priority: Minor | Resolution: 
Created: 18/Apr/24 08:49
Affects Version/s: 3.4.0

As part of SPARK-39195, task is marked as failed but the exception chain was not sent, ultimately the cause becomes `null` in SparkException. It is not convenient to find the root cause from the detailed message.

{code}
  /**
   * Called by the OutputCommitCoordinator to cancel stage due to data duplication may happen.
   */
  private[scheduler] def stageFailed(stageId: Int, reason: String): Unit = {
    eventProcessLoop.post(StageFailed(stageId, reason, None))
  }
{code}
Bug: credulousTrustStoreManagers should return empty array for accepted issuers
Status: Open | Priority: Minor | Resolution: 
Created: 19/Apr/24 16:21
Affects Version/s: 4.0.0

{{TrustManager}} in  {{SSLFactory.credulousTrustStoreManagers}} currently returns {{null}}, but should be returning an empty array instead - as per javadoc and expectation.
Bug: Accounting the impact of failures in spark jobs
Status: Open | Priority: Minor | Resolution: 
Created: 19/Apr/24 11:16
Affects Version/s: 3.5.1

Hello,
 
In my organization, we have an accounting system for spark jobs that uses the task execution time to determine how much time a spark job uses the executors for and we use it as a way to segregate cost. We sum all the task times per job and apply proportions. Our clusters follow a 1 task per core model & this works well.
 
A job goes through several failures during its run, due to executor failure, node failure ( spot interruptions ), and spark retries tasks & sometimes entire stages.
 
We now want to account for this failure and determine what % of a job's total task time is due to these retries. Basically, if a job with failures & retries has a total task time of X, there is a X' representing the goodput of this job – i.e. a hypothetical run of the job with 0 failures & retries. In this case, ( X-X' ) / X quantifies the cost of failures.
 
This form of accounting requires tracking execution history of each task i.e. tasks that compute the same logical partition of some RDD. This was quite easy with AQE disabled as stage ids never changed, but with AQE enabled that's no longer the case. 
 
Do you have any suggestions on how I can use the Spark event system?
Bug: Document push-based shuffle metrics
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Apr/24 06:58
Affects Version/s: 3.4.2

This is to add documentation for the metrics related to push-based shuffle. It's a follow up documentation ticket from: https://issues.apache.org/jira/browse/SPARK-36620

Related to this, note also: https://issues.apache.org/jira/browse/SPARK-42203
Bug: Explore further improving filter push downs with multiple aliased distinct components
Status: Open | Priority: Minor | Resolution: 
Created: 16/Apr/24 23:31
Affects Version/s: 4.0.0

There could be a future benefit if stayUp has multiple elements in it to break up the projection further and try and filter more aggressively, but that also might not be worth it. This is a potential follow up to SPARK-47672
Bug: Official Spark Docker Container images are available from DockerHub in 2 locations
Status: Open | Priority: Minor | Resolution: 
Created: 12/Apr/24 14:57
Affects Version/s: 3.4.1

The downloads section of the project's website [1] provides a link [2] to an official set of Docker containers hosted on DockerHub [3]. However, most of these containers are quite outdated now and it appears as if there is now a new location where the lateset offical containers are maintained and distributed on DockerHub [4]. It would be nice to fix/update the documentation to guide users to the new / recommendation location of the official containers maintained or endorsed by the project and/or community.

[1] [https://spark.apache.org/downloads.html]

[2] [https://hub.docker.com/r/apache/spark-py/tags]

[3] [https://hub.docker.com/r/apache/spark]

[4] [https://hub.docker.com/_/spark]
Bug: Add scala style check for invalid MDC usage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Apr/24 18:22
Affects Version/s: 4.0.0

To avoid invalid MDC usage s"Task ${MDC(TASK_ID, taskId)} failed", which should be log"Task ${MDC(TASK_ID, taskId)} failed. This makes development and PR review easier.
Bug: Special handling of JSON type for MySQL Connector/J 5.x
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Apr/24 08:07
Affects Version/s: 3.4.0

MySQL JDBC driver `mysql-connector-java-5.1.49.jar` converts JSON type into Types.CHAR with a precision of Int.Max.
When receiving CHAR with Int.Max precision, Spark executor will throw an error of `java.lang.OutOfMemoryError: Requested array size exceeds VM limit `.

For {{mysql-connector-java-5.1.49.jar}} json sqlType is {{Char}} and precision is {{{}Int.Max{}}}.
For {{mysql-connector-java-8.0.16.jar}} json sqlType is {{LONGVARCHAR}} and precision is {{{}Int.Max{}}}.

Spark use {{mysql-connector-java-8.0.16.jar}} is right.
{code:java}
private def getCatalystType(
      sqlType: Int,
      typeName: String,
      precision: Int,
      scale: Int,
      signed: Boolean,
      isTimestampNTZ: Boolean): DataType = sqlType match{     
    ...     
    case java.sql.Types.LONGNVARCHAR => StringType    
    ...   
}
{code}
 

If compatibility with 5.1.49 is not required, the current code is sufficient.
Bug: Current doc section handling-late-data-and-watermarking is misleading after the support for multiple stateful operators
Status: Open | Priority: Minor | Resolution: 
Created: 02/Apr/24 19:04
Affects Version/s: 3.4.0

[This section|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#handling-late-data-and-watermarking] of the doc which explains the nuances of handling late records using beautiful diagrams has gone out of context and a bit misleading after *multiple stateful operators'* support was introduced with https://issues.apache.org/jira/browse/SPARK-40925 

!image-2024-04-02-15-04-34-287.png!

 

Previously watermark is applied for batch N to filter out inputs in batch N. With support for multiple stateful operators, the watermark is applied for batch N-1 instead. The doc section above should reflect this new behavior to avoid confusion.
Bug: Ensure trailing slashes in `HistoryServer` URL redirections
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Apr/24 07:13
Affects Version/s: 4.0.0

*Summary:*
The current implementation of URL redirection in Spark's history web UI does not consistently add trailing slashes to URLs when constructing redirection targets. This inconsistency leads to additional HTTP redirects by Jetty, which increases the load time and reduces the efficiency of the Spark UI.

*Problem Description:*
When constructing redirect URLs, particularly in scenarios where an attempt ID needs to be appended, the system does not ensure that the base URL ends with a slash. This omission results in the generated URL being redirected by Jetty to add a trailing slash, thus causing an unnecessary additional HTTP redirect.

For example, when the `shouldAppendAttemptId` flag is true, the URL is formed without a trailing slash before the attempt ID is appended, leading to two redirects: one by our logic to add the attempt ID, and another by Jetty to correct the missing slash. 

!image-2024-04-22-15-14-13-468.png!

*Proposed Solution:*

[https://github.com/apache/spark/blob/2d0b56c3eac611e743c41d16ea8e439bc8a504e4/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala#L118]

Ensure that all redirect URLs uniformly end with a trailing slash regardless of whether an attempt ID is appended. This can be achieved by modifying the URL construction logic as follows:

```scala
val redirect = if (shouldAppendAttemptId)

{ req.getRequestURI.stripSuffix("/") + "/" + attemptId.get + "/" }

else

{ req.getRequestURI.stripSuffix("/") + "/" }

```
Bug: Star expansion after outer join asymmetrically includes joining key
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Aug/22 09:32
Affects Version/s: 3.2.0

When star expansion is used on left side of a join, the result will include joining key, while on the right side of join it doesn't. I would expect the behaviour to be symmetric (either include on both sides or on neither). 

Example:
{code:python}
from pyspark.sql import SparkSession
import pyspark.sql.functions as f

spark = SparkSession.builder.getOrCreate()

df_left = spark.range(5).withColumn('val', f.lit('left'))
df_right = spark.range(3, 7).withColumn('val', f.lit('right'))

df_merged = (
    df_left
    .alias('left')
    .join(df_right.alias('right'), on='id', how='full_outer')
    .withColumn('left_all', f.struct('left.*'))
    .withColumn('right_all', f.struct('right.*'))
)

df_merged.show()
{code}
result:
{code:java}
+---+----+-----+------------+---------+
| id| val|  val|    left_all|right_all|
+---+----+-----+------------+---------+
|  0|left| null|   {0, left}|   {null}|
|  1|left| null|   {1, left}|   {null}|
|  2|left| null|   {2, left}|   {null}|
|  3|left|right|   {3, left}|  {right}|
|  4|left|right|   {4, left}|  {right}|
|  5|null|right|{null, null}|  {right}|
|  6|null|right|{null, null}|  {right}|
+---+----+-----+------------+---------+
{code}
This behaviour started with release 3.2.0. Previously the key was not included on either side. 
Result from Spark 3.1.3
{code:java}
+---+----+-----+--------+---------+
| id| val|  val|left_all|right_all|
+---+----+-----+--------+---------+
|  0|left| null|  {left}|   {null}|
|  6|null|right|  {null}|  {right}|
|  5|null|right|  {null}|  {right}|
|  1|left| null|  {left}|   {null}|
|  3|left|right|  {left}|  {right}|
|  2|left| null|  {left}|   {null}|
|  4|left|right|  {left}|  {right}|
+---+----+-----+--------+---------+ {code}
I have a gut feeling this is related to these issues:
https://issues.apache.org/jira/browse/SPARK-39376
https://issues.apache.org/jira/browse/SPARK-34527
https://issues.apache.org/jira/browse/SPARK-38603
Bug: Instead of using the scalar tasksSuccessful, use the successful array to calculate whether the task is completed
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 03/Aug/22 07:07
Affects Version/s: 2.4.3

When counting the number of successful tasks in the stage of spark, spark uses the indicator of `tasksSuccessful`, but in fact, the success or failure of tasks is based on the array of `successful`. Through the log I added, it is found that the number of failed tasks counted by `tasksSuccessful` is inconsistent with the number of failures stored in the array of `successful`. We should take `successful` as the standard.