av9ash/GitBugs huggingface dataset:
Bug: Passdown fileLength when reading parquet footer to avoid call HDFS namenode in executor side
Status: Open | Priority: Minor | Resolution: 
Created: 18/Dec/24 06:27
Affects Version/s: 4.0.0

When reading parquet footers, we need to know the file length, by request hdfs namenode in executor side for each task. 

But in fact, the file length is already known in driver side from `PartitionedFile`
Bug: Clearing dataframes/views persisted in isolated Connect session
Status: Open | Priority: Minor | Resolution: 
Created: 13/Dec/24 11:55
Affects Version/s: 3.5.3

With Spark Connect, `sparkSession.catalog.clearCache` clears all dataframes that have been persisted in the Spark Connect server, including those persisted by different isolated sessions.

Similarly, views createdby `DataFrame.createOrReplaceTempView` are not removed after an isolated session is terminated, even though the documentation might be interpreted as though it should:

> The lifetime of this temporary table is tied to the [{{SparkSession}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.html#pyspark.sql.SparkSession] that was used to create this [{{DataFrame}}|https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html#pyspark.sql.DataFrame].

It'd be very useful to be able to do the following:

- When calling `clearCache`, controlling whether to clear dataframes persisted in the current isolated session only, or all dataframes.
- Configuring the Spark Connect server to clean up all persisted dataframes/views from an isolated session when it is terminated.
Bug: Fix compression option being overwritten in df.write.parquet in SparkConnect Python
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Dec/24 17:56
Affects Version/s: 3.4.0

There is a small bug in Spark Connect's {{{}DataFrameWriter{}}}.

 
df.write.option("compression", "gzip").parquet(path)
When this code is used, the specified compression option "gzip" gets overwritten by None. This happens because {{parquet()}} function has a default {{compression=None}} parameter which is used to set the compression option directly
with {{{}self.option("compression", compression){}}}.
The Spark Connect server then receives a request without a specified compression option and uses "snappy" compression by default instead.
The fix is to use the {{{}self._set_opts(compression=compression){}}}. This method of setting options is used by most other {{DataFrameWriter}} APIs.
Bug: Unable to run custom PDF Data Source on Databricks
Status: Open | Priority: Minor | Resolution: 
Created: 03/Dec/24 08:33
Affects Version/s: 3.5.0

Experienced error when running custom PDF DataSource on Databricks Runtime: 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)

PDF DataSource works fine on the community version of the Spark 3.5.0, but fail on Databricks:

Log:
Py4JJavaError: An error occurred while calling o428.showString.
: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.PartitionedFile.(Lorg/apache/spark/sql/catalyst/InternalRow;Lorg/apache/spark/paths/SparkPath;JJ[Ljava/lang/String;JJLscala/collection/immutable/Map;)V
at com.stabrise.sparkpdf.datasources.PdfPartitionedFileUtil$.$anonfun$splitFiles$1(PdfPartitionedFileUtil.scala:32)
at com.stabrise.sparkpdf.datasources.PdfPartitionedFileUtil$.$anonfun$splitFiles$1$adapted(PdfPartitionedFileUtil.scala:28)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)

[Source Code of the PDF DataSource|https://github.com/StabRise/spark-pdf/tree/spark_3_5]

[Source code of PdfPartitionedFileUtil.scala |https://github.com/StabRise/spark-pdf/blob/spark_3_5/src/main/scala/datasources/PdfPartitionedFileUtil.scala#L32]

PDF DataSource jar file:  [https://github.com/StabRise/spark-pdf/releases/download/0.1.12_spark_3_5/spark-pdf-0.1.12.jar]

Notebook with full example and traceback: [https://github.com/StabRise/spark-pdf/blob/spark_3_5/examples/PdfDataSourceDatabricks.ipynb|https://github.com/StabRise/spark-pdf/blob/spark_3_5/examples/PdfDataSourceDatabricks.ipynb]
Bug: Update cross-spawn JS module to surpress a warning in lint
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Dec/24 17:38
Affects Version/s: 4.0.0

Recently, ./dev/lint-js shows a warning like as follows.

{code}
1 high severity vulnerability

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
{code}

To inspect more, I executed npm audit and the result is:
{code}
cross-spawn  7.0.0 - 7.0.4
Severity: high
Regular Expression Denial of Service (ReDoS) in cross-spawn - https://github.com/advisories/GHSA-3xgq-45jj-v275
fix available via `npm audit fix`
node_modules/cross-spawn

1 high severity vulnerability
{code}
Bug:  Quoted Empty Fields in CSV Not Interpreted as Null When escape is Set to ""
Status: Open | Priority: Minor | Resolution: 
Created: 05/Dec/24 16:56
Affects Version/s: 3.4.1

When reading a CSV file with quoted empty fields using Spark, the fields are not interpreted as {{null}} consistently across all columns. Some fields retain a single double-quote character ({{{}"{}}}), leading to inconsistent behavior. This occurs when the {{escape}} option is explicitly set to an empty string ({{{}""{}}}). Using the default escape character ({{{}\{}}}) avoids the issue.

Steps to Reproduce:
Create a CSV file with the following content:
id;name;amount;comment
1;John;100.50;""
2;Alice;200.75;"Approved"
3;Bob;300.00;""
4;Eve;;"Pending"

Use the following Spark code to read the file:

spark.read
    .option("header", "true")
    .option("sep", ";")
    .option("encoding", "UTF-8")
    .option("nullValue", "")
    .option("quote", '"')
    .option("escape", "")  // Explicitly set escape to an empty string
    .csv("path/to/csv")
h3. *Expected Behavior:*

The DataFrame should interpret quoted empty fields ({{{}""{}}}) as {{null}} values consistently across all columns:

+---+------+--------+---------+
|id | name | amount | comment |
+---+------+--------+---------+
| 1 | John | 100.50 | null    |
| 2 | Alice| 200.75 | Approved|
| 3 | Bob  | 300.00 | null    |
| 4 | Eve  | null   | Pending |
+---+------+--------+---------+

 
h3. *Observed Behavior:*

Quoted empty fields in the {{comment}} column are retained as a single double-quote ({{{}"{}}}) instead of being interpreted as {{{}null{}}}, causing inconsistencies:

+---+------+--------+---------+
|id | name | amount | comment |
+---+------+--------+---------+
| 1 | John | 100.50 | "       |
| 2 | Alice| 200.75 | Approved|
| 3 | Bob  | 300.00 | "       |
| 4 | Eve  | null   | Pending |
+---+------+--------+---------+

 
h3. *Analysis:*
 * Quoted empty fields in the {{comment}} column are incorrectly read as {{{}"{}}}.
 * The issue occurs only when the {{escape}} option is set to an empty string ({{{}""{}}}).
 * Defaulting the {{escape}} option to {{\}} resolves the problem.
Bug: `No enum constant java.sql.JDBCType.varchar` error writing array values
Status: Open | Priority: Minor | Resolution: 
Created: 04/Dec/24 07:42
Affects Version/s: 3.1.3

Hi, the code which tries to write a value of type Array[TYPE] converts the type name of the inner type to lower case since the feature wad added:

* [https://github.com/apache/spark/pull/14323/files#diff-c3859e97335ead4b131263565c987d877bea0af3adbd6c5bf2d3716768d2e083R217]

 

The conversion to lower case has been removed in master afaics:
 * [https://github.com/apache/spark/blob/0e45e217996248a093cfcb892407bb523f885c08/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L707]
 * [https://github.com/apache/spark/blob/0e45e217996248a093cfcb892407bb523f885c08/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L714]

 

So i assume the conversin to lower case is not needed (i wonder why it was added)

 

That conversion to lower case makes jdbc drivers (in this case snowfalke jdbc driver) fails in connection.createArrayOf code when trying to get a value of JDBCType enum using that type name:

 

```

java.lang.IllegalArgumentException: No enum constant java.sql.JDBCType.varchar
    at java.lang.Enum.valueOf(Enum.java:238)
    at java.sql.JDBCType.valueOf(JDBCType.java:34)
    at net.snowflake.client.jdbc.SnowflakeConnectionV1.createArrayOf(SnowflakeConnectionV1.java:725)
```

So i would remove the call toLowerCase in prior versions to master, at least 3.5.

 

Many thanks in advance!
Bug: simplify shell scripts using mvn help:evaluate
Status: Open | Priority: Trivial | Resolution: 
Created: 19/Dec/24 19:14
Affects Version/s: 3.5.3

mvn script could be cleaned a little, for example in script ./dev/make-distribution.sh, to use "-q -DforceStdout=true" instead of "mvn ... | grep -v INFO | grep -v WARNING"

a typical example

{noformat}
$ mvn help:evaluate -Dexpression=project.version -q -DforceStdout=true
4.0.0-SNAPSHOT
{noformat}

So using 

{noformat}
VERSION=$("$MVN" help:evaluate -q -DforceStdout -Dexpression=project.version $@)
{noformat}

instead of 

{noformat}
VERSION=$("$MVN" help:evaluate -Dexpression=project.version $@ \
    | grep -v "INFO"\
    | grep -v "WARNING"\
    | tail -n 1)
{noformat}

This pattern occurs 6 times in scripts 
{noformat}
$ grep -R -- "grep -v \"INFO\""
dev/make-distribution.sh:    | grep -v "INFO"\
dev/make-distribution.sh:    | grep -v "INFO"\
dev/make-distribution.sh:    | grep -v "INFO"\
dev/make-distribution.sh:    | grep -v "INFO"\
resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh:    | grep -v "INFO"\
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh:    | grep -v "INFO"\
{noformat}

The syntax "-q -DforceStdout" was already used in few other places, so minor fix for more consistency and cleaner.
Bug: Extend spark 3.5.1 to support hadoop-client-api 3.4.0, hadoop-client-runtime-3.4.0
Status: Closed | Priority: Blocker | Resolution: Invalid
Created: 08/Apr/24 16:20
Affects Version/s: 3.5.1

We have some HIGH CVEs which are coming from hadoop-client-runtime 3.3.4 and hence we need to address those

 

com.fasterxml.jackson.core:jackson-databind              causing    *CVE-2022-42003* and *CVE-2022-42004*

(org.apache.hadoop_hadoop-client-runtime-3.3.4.jar)

 

 

com.google.protobuf:protobuf-java      

(org.apache.hadoop_hadoop-client-runtime-3.3.4.jar)  causing *CVE-2021-22569,* *CVE-2021-22570,* *CVE-2022-3509* and *CVE-2022-3510*

 

net.minidev:json-smart                                                         causing *CVE-2021-31684,* *CVE-2023-1370*

(org.apache.hadoop_hadoop-client-runtime-3.3.4.jar)  

 

 

org.apache.avro:avro 

(org.apache.hadoop_hadoop-client-runtime-3.3.4.jar)        causing *CVE-2023-39410*    

 

 

org.apache.commons:commons-compress         causing *CVE-2024-25710, CVE-2024-26308* 

(org.apache.hadoop_hadoop-client-runtime-3.3.4.jar) 

 

 

Most of these have gone in hadoop client  runtime 3.4.0

 

Is there a plan to support hadoop 3.4.0 ?
Bug: Spark Session is not created for spark-2.1.1 when log4j is migrated from 1.* to 2.17.1 to fix the security issue
Status: Closed | Priority: Blocker | Resolution: Workaround
Created: 10/Apr/24 08:49
Affects Version/s: 2.1.1

Spark Download Link: [https://archive.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-hadoop2.7.tgz]

log4j Migration: 

[https://logging.apache.org/log4j/2.x/manual/migration.html]

The spark creation is not happening .

Below is the exception:

*{color:#4c9aff}1.exception.png (Attached){color}*

When  analyzed further found that the class file “RendererMap“ is not at all present in the bridge jar.

*log4j-1.2.17* 

*{color:#4c9aff}2.log4j1.png (Attached){color}*

 ** *log4j-1.2-api-2.17.1.jar (Bridge jar)*

*{color:#4c9aff}3.log4j-2.png (Attached){color}*

Following the migration steps in [https://logging.apache.org/log4j/2.x/manual/migration.html]

We are simply replacing the

Jar: */usr/local/spark/jars/log4j-1.2.17.jar*

With:
 * log4j-1.2-api-2.17.1.jar
 * log4j-api-2.17.1.jar
 * log4j-core-2.17.1.jar

similar to what we had done 2 years back for version spark 2.1.3 version for which we were not downloading the spark zip on fly ,but we already had a downloaded zip on which we applied the patching changes. That seemed to work for 2.1.3 but for an older version where we are downloading the tar.gz now i.e 2.1.1, there seems to be an issue.

Looking for a right resolution in this regard. Kindly let us know if any further information is required.

FYI, Some of the various things tried:
 # Upgrading the bridge ,api ,core jar to 2.17.2 and even to the latest
 # Replaced the two slf4j jars with higher version : ex. slf4j-api-1.7.25.jar slf4j-log4j12-1.7.25.jar
 # Adding a log4j.properties file from the template available.
Bug: State store operation cannot work properly with binary inequality collation
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Apr/24 06:57
Affects Version/s: 4.0.0

Arguably this is a correctness issue, though we haven't released collation feature yet.

collation introduces the concept of binary (in)equality, which means in some collation we no longer be able to just compare the binary format of two UnsafeRows to determine equality.

For example, 'aaa' and 'AAA' can be "semantically" same in case insensitive collation.

State store is basically key-value storage, and the most provider implementations rely on the fact that all the columns in the key schema support binary equality. We need to disallow using binary inequality column in the key schema, before we could support this in majority of state store providers (or high-level of state store.)

Why this is correctness issue? For example, streaming aggregation will produce an output of aggregation which does not care about the semantic equality.

e.g. df.groupBy(strCol).count() 

Although strCol is case insensitive, 'a' and 'A' won't be counted together in streaming aggregation, while they should be.
Bug: Got fetch failed exception when new executor reused same ip address from a previously killed executor
Status: Open | Priority: Critical | Resolution: 
Created: 02/Apr/24 00:40
Affects Version/s: 3.4.0

This is an edge case which caused Spark on Kubernetes getting fetch failed exception when new executor reused same ip address from a previously killed executor.

The new executor checks shuffle block ip address and compares it with its own host address. If the two ip addresses are the same, the new executor will assume the block on its own local disk and try to read it locally. This causes failure since the block is actually on the previously killed executor which happened to have same ip address.
Bug: Spark job relying over Hudi are blocked after one or zero commit
Status: Closed | Priority: Major | Resolution: Incomplete
Created: 13/Apr/24 14:00
Affects Version/s: 3.3.0

Hello, we are facing the fact that some pyspark job that rely on Hudi seems to be blocked, in fact if we go over the spark console we can see the situation in the attachment
we can see that we have 71 completed jobs but those are CDC process that should read from Kafka topic continuously. We verified yet that there are messages queued over the kafka topic. If you kill the application and then restart in some cases the job will act normally and other times the job still remain stacked.

Our deploy condition are the following:
We read INSERT, UPDATE and DELETE operation from a Kafka topic and we replicate them in a target hudi table stored on Hive via a pyspark job running 24/7

 

PYSPARK WRITE
df_source.writeStream.foreachBatch(foreach_batch_write_function)
 {{ FOR EACH BATCH FUNCTION:
#management of delete messages
batchDF_deletes.write.format('hudi') \
.option('hoodie.datasource.write.operation', 'delete') \
.options(**hudiOptions_table) \
.mode('append') \
.save(S3_OUTPUT_PATH)

#management of update and insert messages
batchDF_upserts.write.format('org.apache.hudi') \
.option('hoodie.datasource.write.operation', 'upsert') \
.options(**hudiOptions_table) \
.mode('append') \
.save(S3_OUTPUT_PATH)}}
 
SPARK SUBMIT
spark-submit --master yarn --deploy-mode cluster --num-executors 1 --executor-memory 1G --executor-cores 2 --conf spark.dynamicAllocation.enabled=false --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.hive.convertMetastoreParquet=false --jars /usr/lib/hudi/hudi-spark-bundle.jar <path_to_script>
Bug: Add operational metrics for TWS operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Apr/24 23:48
Affects Version/s: 4.0.0

Add metrics to improve observability for the newly added operator TransformWithState and some changes we've made into RocksDB.
Proposed metrics to add:
 * on the RocksDB StateStore metrics side, we will add the following:

 ** num external col families
 ** num internal col families
 * on the operator side, we will add the following:

 ** number of state vars
 ** count of state vars by type
 ** output mode
 ** timeout mode
 ** registered timers in batch
 ** expired timers in batch
 ** initial state enabled or not
 ** number of state vars removed in batch
Bug: [PYTHON] Avoid shadowing python built-ins in python function variable naming
Status: Open | Priority: Major | Resolution: 
Created: 15/Apr/24 07:25
Affects Version/s: 3.3.4

Given that spark 4.0.0 is upcoming I wonder if we should at least consider renaming certain function variable naming in python. Otherwise, we may need to wait another 4 years to do so.

Example

[https://github.com/apache/spark/blob/e6b7950f553cff5adc02b8b5195e79cffff3c97c/python/pyspark/sql/functions/builtin.py#L12768]

There are 8 uses of `len` and 35 `str` as variable names, both of which are python built-ins. Shadowing `str` is somewhat dangerous in that the following would be nonsensical:
{code:java}
def foo(str: "ColumnOrName", bar: "ColumnOrName"):
    # str is variable now, cannot be used as type
    bar = if lit(bar) if isinstance(bar, str) else bar
{code}
 

Now obviously this would be breaking change for user code if the function is called with kwargs style. If we rename `str` to `src` or `col`, certain old code using kwargs would break:
{code:java}
# breaks:
foo(str="x", bar="y")

# okay:
foo("x", bar="y"){code}
Is this change a possibility for 4.0? Or are we thinking that the kwargs breaking change is too big to make compared to the benefit?
Bug: Avoid double evaluation of non-trivial projected elements from filter pushdown
Status: Open | Priority: Major | Resolution: 
Created: 01/Apr/24 17:49
Affects Version/s: 3.5.1

Repro here [https://gist.github.com/holdenk/0f9660bcbd9e63aaff904f15d3439db1] 
 
You can work around this by setting an expensive UDF to non-deterministic but that's not ideal and won't fix expensive internal operations (like string matching).
 
Instead when we go to bubble up a filter, if we should not move a filter up above a projection of what we are filtering on.
 
https://issues.apache.org/jira/browse/SPARK-40045 partially fixed some of this by (roughly) ordering filter expressions by cost so that we're not evaluating more than ~2x (e.g. in old behavior bubbled up filter could become the first elem of the filter and then the cheap null checks would go away and we'd have expensive compute on everything not just filtered data), but we should "trust" the users projection + later use of that projection to indicate that a UDF is expensive and we should only evaluate it once inside of the projection and filter after.
Bug: Shuffle service endpoint is not removed from the locations list when RDD block is removed form a node.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/24 03:33
Affects Version/s: 3.5.1

If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both executor end point and the external shuffle end points for a RDD block. When the RDD is migrated, the location info is updated to add the end point corresponds to new location and the old end point is removed. But currently, only the executor end point is removed. The shuffle service end point is not removed. This cause failure during RDD read if the shuffle service end point is chosen due to task locality.
Bug: Skip deleting pod from k8s if the pod does not exists
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/24 09:02
Affects Version/s: 4.0.0

in ExecutorPodsLifecycleManager#removeExecutorFromK8s method. Get pod before deleting it, we can skip deleting if pod is already deleted so that we do not send too many requests to api server.
Bug: ColumnVectors with dictionaries and nulls are not read/copied correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/24 01:41
Affects Version/s: 3.4.3

{{ColumnVectors}} have APIs like {{getInts}}, {{getFloats}} and so on. Those return a primitive array with the contents of the vector. When the ColumnVector has a dictionary, the values are decoded with the dictionary before filling in the primitive array.

However, {{ColumnVectors}} can have nulls, and for those {{null}} entries, the dictionary id is irrelevant, and can also be invalid. The dictionary should not be used for the {{null}} entries of the vector. Sometimes, this can cause an {{ArrayIndexOutOfBoundsException}} .

In addition to the possible Exception, copying a {{ColumnarArray}} is not correct. A {{ColumnarArray}} contains a {{ColumnVector}} so it can contain {{null}} values. However, the {{copy()}} for primitive types does not take into account the null-ness of the entries, and blindly copies all the primitive values. That means the null entries get lost.
Bug: add SortOrder for window function which has no orderSpec
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/24 09:27
Affects Version/s: 4.0.0

I am doing Hive SQL to switch to Spark SQL.

 

In Hive SQL

 

hive> explain select *,row_number() over (partition by day) rn from testdb.zeropart_db;

OK
Explain

 

In Spark SQL

spark-sql> explain select *,row_number() over (partition by age ) rn  from testdb.zeropart_db;

plan

== Physical Plan ==

org.apache.spark.sql.AnalysisException: Window function row_number() requires window to be ordered, please add ORDER BY clause. For example SELECT row_number()(value_expr) OVER (PARTITION BY window_partition ORDER BY window_ordering) from table

Time taken: 0.172 seconds, Fetched 1 row(s)

 

For better compatibility with migration. For better compatibility with migration, new parameters are added to ensure compatibility with the same behavior as Hive SQL
Bug: Support UDAF in Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/24 11:04
Affects Version/s: 4.0.0

Currently Spark Connect supports only UDFs. We need to add support for UDAFs, specifically `Aggregator[INT, BUF, OUT]`.

The user-facing API should not change, which includes Aggregator methods and the `spark.udf.register("agg", udaf(agg))` API.
Bug: Cleanup shuffle dependencies for Spark Connect SQL executions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/24 12:51
Affects Version/s: 4.0.0

Shuffle dependencies are created by shuffle map stages, which consists of files on disks and the corresponding references in Spark JVM heap memory. Currently Spark cleanup unused  shuffle dependencies through JVM GCs, and periodic GCs are triggered once every 30 minutes (see ContextCleaner). However, we still found cases in which the size of the shuffle data files are too large, which makes shuffle data migration slow.

 

We do have chances to cleanup shuffle dependencies, especially for SQL queries created by Spark Connect, since we do have better control of the DataFrame instances there. Even if DataFrame instances are reused in the client side, on the server side the instances are still recreated. 

 

We might also provide the option to 1. cleanup eagerly after each query executions, or 2. only mark the shuffle executions and do not migrate them at node decommissions.