av9ash/GitBugs huggingface dataset:
Bug: TableChange.updateColumnComment does not remove comment entry
Status: Open | Priority: Major | Resolution: 
Created: 04/May/24 21:52
Affects Version/s: 3.4.1

The function {{TableChange.updateColumnComment}} can be used to add a comment to a column, change it, but not remove the comment entry from the column's metadata.

Moreover, trying to update the column with value {{null}} risks raising a NPE if the function {{TableChange.UpdateColumnComment.equals}} is called.

Examples in Scala:
{code:scala}
TableChange.updateColumnComment(Array("foo"), "foo comment") // adds or changes comment => OK
TableChange.updateColumnComment(Array("foo"), "") // adds or changes comment to empty string, instead of removing comment from metadata => KO
TableChange.updateColumnComment(Array("foo"), null) // adds or changes comment to null, instead of removing comment from metadata => KO
TableChange.updateColumnComment(Array("foo"), "foo comment") == TableChange.updateColumnComment(Array("foo"), null) // raises an NPE => KO
{code}

Solution would be to accept empty string or null as values that remove the {{"comment"}} entry from the metadata, or create a new case {{TableChange.RemoveColumnComment}}.
Bug: BitwiseCount / bit_count generated code for boolean inputs fails to compile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/24 05:46
Affects Version/s: 3.0.0

If the `BitwiseCount` / `bit_count` expresison is applied to a boolean type column then then it will trigger codegen fallback to interpreted because the generated code contains invalid Java syntax, triggering errors like

{code}
 java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41, Column 11: Failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 41, Column 11: Unexpected token "if" in primary
{code}

This problem was masked because the QueryTest framework may not be fully exercising codegen paths (e.g. if constant folding occurs).
Bug: Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/May/24 03:23
Affects Version/s: 4.0.0

- https://docs.python.org/3/whatsnew/3.11.html#summary-release-highlights

bq. Python 3.11 is between 10-60% faster than Python 3.10. On average, we measured a 1.25x speedup on the standard benchmark suite.
Bug: Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 21:59
Affects Version/s: 4.0.0

`Java 21 on MacOS 14` is the fastest Maven test and covers both Java 17 and Apple Silicon use case.

 !Screenshot 2024-05-02 at 14.59.14.png|width=100%!
Bug: Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/24 07:29
Affects Version/s: 4.0.0

{code:java}
sql("select ntile(99.9) OVER (order by id) from range(10)"){code}

results in

{code}
 java.lang.ClassCastException: class org.apache.spark.sql.types.Decimal cannot be cast to class java.lang.Integer (org.apache.spark.sql.types.Decimal is in unnamed module of loader 'app'; java.lang.Integer is in module java.base of loader 'bootstrap')
  at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:99)
  at org.apache.spark.sql.catalyst.expressions.NTile.checkInputDataTypes(windowExpressions.scala:877)
  at org.apache.spark.sql.catalyst.expressions.Expression.resolved$lzycompute(Expression.scala:267)
  at org.apache.spark.sql.catalyst.expressions.Expression.resolved(Expression.scala:267)
  at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$childrenResolved$1(Expression.scala:279)
  at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$childrenResolved$1$adapted(Expression.scala:279)
  at scala.collection.IterableOnceOps.forall(IterableOnce.scala:633)
  at scala.collection.IterableOnceOps.forall$(IterableOnce.scala:630)
  at scala.collection.AbstractIterable.forall(Iterable.scala:935)
  at org.apache.spark.sql.catalyst.expressions.Expression.childrenResolved(Expression.scala:279)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveFunctions$$anonfun$apply$22$$anonfun$applyOrElse$157.applyOrElse(Analyzer.scala:2243) 
{code}

instead of the intended user-facing error message. This is a minor bug that was introduced in a previous error class refactoring PR.
Bug: Fix Variant default columns for more complex default variants
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/24 01:28
Affects Version/s: 4.0.0

Default columns are stored as structfield metadata (string -> string) map. This means the literal values are stored as strings.

However, the string representation of a variant is the JSONified string. Thus, we need to wrap the string with `parse_json` to correctly use the default values
Bug: Enforce type checking in from_avro and to_avro functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/24 20:13
Affects Version/s: 4.0.0

Because types are not enforced in Python, users can pass in all sort of arguments for functions. This could lead to invoking wrong functions in spark connect.

If we perform type checking for arguments and output sensible errors when the type of arguments passed into the functions don’t match, we can give the user a better user experience
Bug: `mypy` should have `--python-executable` parameter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/May/24 05:57
Affects Version/s: 3.3.0

We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not true. We need to use `mypy`'s parameter to make it sure.

https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705

{code}
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
3428
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
root@18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l
1
{code}
Bug: Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored
Status: Open | Priority: Minor | Resolution: 
Created: 02/May/24 12:29
Affects Version/s: 3.4.0

When using an `explode` function, and `transform` function in the same select statement, aliases used inside the transformed column are ignored.

This behavior only happens using the pyspark API and the scala API, but not when using the SQL API

 
{code:java}
from pyspark.sql import functions as F

# Create the df
df = spark.createDataFrame([
    {"id": 1, "array1": ['a', 'b'], 'array2': [2,3,4]}
]){code}
Good case, where all aliases are used

 
{code:java}
df.select(
    F.transform(
        'array2',
        lambda x: F.struct(x.alias("some_alias"), F.col("id").alias("second_alias"))
    ).alias("new_array2")
).printSchema() 

root
 |-- new_array2: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- some_alias: long (nullable = true)
 |    |    |-- second_alias: long (nullable = true){code}
Bad case, when using explode, the alises inside the transformed column is ignored, and  `id` is kept instead of `second_alias`, and `x_17` is used instead of `some_alias`

 

 
{code:java}
df.select(
    F.explode("array1").alias("exploded"),
    F.transform(
        'array2',
        lambda x: F.struct(x.alias("some_alias"), F.col("id").alias("second_alias"))
    ).alias("new_array2")
).printSchema()

root
 |-- exploded: string (nullable = true)
 |-- new_array2: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- x_17: long (nullable = true)
 |    |    |-- id: long (nullable = true) {code}
 

 {code:scala}
import org.apache.spark.sql.functions._
var df2 = df.select(array(lit(1), lit(2), lit(3)).as("my_array"), array(lit(1), lit(2), lit(3)).as("my_array2"))

df2.select(
  explode($"my_array").as("exploded"),
  transform($"my_array2", (x) => struct(x.as("data"))).as("my_struct")
).printSchema
{code}


{noformat}
root
 |-- exploded: integer (nullable = false)
 |-- my_struct: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- x_2: integer (nullable = false)
{noformat}


 

When using the SQL API instead, it works fine
{code:java}
spark.sql(
    """
    select explode(array1) as exploded, transform(array2, x-> struct(x as some_alias, id as second_alias)) as array2 from {df}
    """, df=df
).printSchema()

root
 |-- exploded: string (nullable = true)
 |-- array2: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- some_alias: long (nullable = true)
 |    |    |-- second_alias: long (nullable = true) {code}
 

Workaround: for now, using F.named_struct can be used as a workaround
Bug: Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/May/24 18:15
Affects Version/s: 4.0.0

When a task attemp fails but it is authorized to do task commit, OutputCommitCoordinator will make the stage failed with a reason message which says that task commit success, but actually the driver never knows if a task commit is successful or not. We should update the reason message to make it less confused.

See https://github.com/apache/spark/pull/36564#discussion_r1598660630
Bug: Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/24 20:00
Affects Version/s: 4.0.0

This enhancement adds an option to the Python Datasource Reader to yield Arrow batches directly, significantly boosting performance compared to using tuples or Rows. This implementation leverages the existing work with MapInArrow (see SPARK-46253 ).

Tests with a custom Python Datasource for High Energy Physics (HEP) data using the ROOT format reader showed an 8x speed increase when using Arrow batches over the traditional method of feeding data via tuples.

Additional context:
 * The ROOT data format is widely used in High Energy Physics (HEP) with approximately 1 exabyte of ROOT data currently in existence.
 * You can easily read ROOT data using libraries from the Python ecosystem, notably {{uproot}} and {{{}awkward-array{}}}. These libraries facilitate the reading of ROOT data and its conversion to Arrow among other formats.
 * You can write a simple ROOT data source using the Python datasource API. While this may not be optimal for performance, it is easy to implement and can leverage the mentioned libraries.
 * For better performance, ingest data via Arrow batches rather than row by row, as the latter method is significantly slower (an initial test showed it to be 8 times slower).
 * Arrow is very popular now, and this enhancement can benefit other communities beyond HEP that use Arrow for efficient data processing.

This enhancement will provide substantial performance improvements and make it easier to work with HEP data and other data types using Apache Spark.
Bug: Throw an user-facing error when reading invalid schema from text DataSource
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 31/May/24 12:37
Affects Version/s: 3.4.3

Text DataSource produces table schema with only 1 column, but it is possible to try and create a table with schema having multiple columns.

Currently, when user tries this, we have an assert in the code, which fails and throws internal spark error. We should throw a better user-facing error.
Bug: Update SS Doc of dropDuplicatesWithinWatermark to use the right syntax
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/May/24 00:25
Affects Version/s: 4.0.0

For dropDuplicates, the example on [https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#:~:text=)%20%5C%0A%20%20.-,dropDuplicates,-(%22guid%22] is out of date compared with [https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.dropDuplicates.html]. The argument should be a list.

The discrepancy is also true for dropDuplicatesWithinWatermark.
Bug: Virtualenv section of "Python Package Management" does not work on Windows
Status: Open | Priority: Minor | Resolution: 
Created: 23/May/24 14:12
Affects Version/s: 3.4.3

[https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html]

Has a section about using virtualenv, including venv-pack. venv-pack does not currently work on Windows: [https://github.com/jcrist/venv-pack/issues/6]

Either the docs need adjusting to reflect that, or a different tool needs to be used. I'm far from a python expert, so i have no idea if the latter is even possible.

Would be nice to add a hint, to save people some time (maybe suggesting to use WSL instead?).
Bug: Enable `spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/May/24 00:09
Affects Version/s: 4.0.0

The SPJ feature flag 'spark.sql.sources.v2.bucketing.pushPartValues.enabled' has proven valuable for most use cases.  We should take advantage of 4.0 release and change the value to true.
Bug: TransportClientFactory.createClient may NPE cause FetchFailedException
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/May/24 10:48
Affects Version/s: 4.0.0



{code:java}
org.apache.spark.shuffle.FetchFailedException
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:1180)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:913)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84)
	at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)

Caused by: java.lang.NullPointerException
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:178)
	at org.apache.spark.network.shuffle.ExternalBlockStoreClient.lambda$fetchBlocks$0(ExternalBlockStoreClient.java:128)
	at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:154)
	at org.apache.spark.network.shuffle.RetryingBlockTransferor.start(RetryingBlockTransferor.java:133)
	at org.apache.spark.network.shuffle.ExternalBlockStoreClient.fetchBlocks(ExternalBlockStoreClient.java:139)
{code}
Bug: RoundRobin based coalesce in spark
Status: Resolved | Priority: Minor | Resolution: Abandoned
Created: 13/May/24 11:26
Affects Version/s: 3.5.1

Currently default coalsce does not take partition size into account and simply merges partitions. This often results in non-uniform data distribution. There have been proposal for size based coalesce([https://github.com/apache/spark/pull/27248).|https://github.com/apache/spark/pull/27248),]

I am proposing a custom roundrobin coalesce which will distribute data evenly across partitions within same executor.
Bug: Docstrings of the pyspark DataStream Reader methods are inaccurate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/May/24 17:10
Affects Version/s: 3.4.3

The docstrings of the pyspark DataStream Reader methods {{csv()}} and {{text()}} say that the {{path}} parameter can be a list, but actually when a list is passed an error is raised.
Bug: Update the Oracle docker image version used for test and integration to use Oracle Database 23ai Free
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/May/24 08:24
Affects Version/s: 4.0.0

{color:#0d0d0d}Oracle recently released Oracle Database 23ai Free, specifically version 23.4, as their latest free database version. {color}

{color:#0d0d0d}We should update our testing infrastructure to utilize this free version, using the Docker image available at{color}[ {color:#0d0d0d} {color}|https://github.com/gvenzl/oci-oracle-free] [https://github.com/gvenzl/oci-oracle-free |https://github.com/gvenzl/oci-oracle-free]

{color:#0d0d0d}This repository is known for being a reliable and well-maintained source for Oracle Database images.{color}
Bug: When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved
Status: Open | Priority: Minor | Resolution: 
Created: 02/May/24 22:27
Affects Version/s: 3.3.0

Repro:

```sql

DROP TABLE IF EXISTS local.cte1;
DROP TABLE IF EXISTS local.cte2;
DROP TABLE IF EXISTS local.cte3;
CREATE TABLE local.cte1 (id INT, fname STRING);
CREATE TABLE local.cte2 (id2 INT);
CREATE TABLE local.cte3 (id INT);
WITH test_fake AS (SELECT * FROM local.cte1 WHERE id = 1 AND id2 = 1), test_fake2 AS (SELECT * FROM local.cte2 WHERE id2 = 1) INSERT OVERWRITE TABLE local.cte3 SELECT id2 as id FROM test_fake2;
WITH test_fake AS (SELECT * FROM local.cte1 WHERE id = 1 AND id2 = 1), test_fake2 AS (SELECT * FROM local.cte2 WHERE id2 = 1) SELECT id2 as id FROM test_fake2;

```

 

Here we would expect both of the last two SQL expressions to fail, but instead only the first one does.

 

There are more complicated cases, and in those cases, the invalid CTE is treated as a null table, but this is the simplest repro I've been able to come up with so far.

 

This occurs using both local w/Iceberg catalog or the SparkSession catalog.
Bug: Unify getting data schema without partition columns in FileSourceStrategy
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 16/May/24 12:42
Affects Version/s: 3.5.1

In [FileSourceStrategy,|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L191] the schema of the data excluding partition columns is computed 2 times in a slightly different way:

 
{code:java}
val dataColumnsWithoutPartitionCols = dataColumns.filterNot(partitionSet.contains) {code}
vs 
{code:java}
val readDataColumns = dataColumns
  .filterNot(partitionColumns.contains) {code}
Bug: Analyze 'exists' default expression instead of 'current' default expression in structField to v2 column conversion
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 15/May/24 09:13
Affects Version/s: 4.0.0

org.apache.spark.sql.catalyst.util.ResolveDefaultColumns#analyze method accepts 3 parameter

1) Field to analyze
2) Statement type - String

3) Metadata key - CURRENT_DEFAULT or EXISTS_DEFAULT



Method org.apache.spark.sql.connector.catalog.CatalogV2Util#structFieldToV2Column
pass fieldToAnalyze and EXISTS_DEFAULT as second parameter, so it is not metadata key, instead of that, it is statement type, so bad expression is analyzed.
Bug: Add parenthesis to awaitTermination call
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 28/May/24 18:18
Affects Version/s: 3.4.3

In {{test_stream_reader}} and {{test_stream_writer}} of {*}test_python_streaming_datasource.py{*}, the call {{q.awaitTermination}} does not invoke a function call as intended, but instead returns a python function object. The fix is to change this to {{{}q.awaitTermination(){}}}.
Bug: Prepare Apache Spark 4.0.0
Status: Open | Priority: Critical | Resolution: 
Created: 20/Jun/23 17:45
Affects Version/s: 4.0.0

For now, this issue aims to collect ideas for planning Apache Spark 4.0.0.

We will add more items which will be excluded from Apache Spark 3.5.0 (Feature Freeze: July 16th, 2023).

{code}
Spark 1: 2014.05 (1.0.0) ~ 2016.11 (1.6.3)
Spark 2: 2016.07 (2.0.0) ~ 2021.05 (2.4.8)
Spark 3: 2020.06 (3.0.0) ~ 2026.xx (3.5.x)
Spark 4: 2025.02 (4.0.0, NEW)
- January 15th 2025	Code freeze. Release branch cut.
- February 1st 2025	QA period. Focus on bug fixes, tests, stability and docs. Generally, no new features merged.
- February 15th 2025	Release candidates (RC), voting, etc. until final release passes
{code}
Bug: Unable to Mount ConfigMap in Driver Pod - ConfigMap Creation Issue
Status: Open | Priority: Critical | Resolution: 
Created: 14/Jun/23 05:36
Affects Version/s: 3.3.1

Dear Spark community,

I am facing an issue related to mounting a ConfigMap in the driver pod of my Spark application. Upon investigation, I realized that the problem is caused by the ConfigMap not being created successfully.

*Problem Description:*
When attempting to mount the ConfigMap in the driver pod, I encounter consistent failures and my pod stays in containerCreating state. Upon further investigation, I discovered that the ConfigMap does not exist in the Kubernetes cluster, which results in the driver pod's inability to access the required configuration data.

*Additional Information:*

I would like to highlight that this issue is not a frequent occurrence. It has been observed randomly, affecting the mounting of the ConfigMap in the driver pod only approximately 5% of the time. This intermittent behavior adds complexity to the troubleshooting process, as it is challenging to reproduce consistently.

*Error Message:*

when describing driver pod (kubectl describe pod pod_name)  get the below error.

"ConfigMap '<configmap-name>' not found."

*To Reproduce:*

1. Download spark 3.3.1 from [https://spark.apache.org/downloads.html]

2. create an image with "bin/docker-image-tool.sh"

3. Submit on spark-client via bash command by passing all the details and configurations.

4. Randomly in some of the driver pod we can observe this issue.
Bug: Built-in XML data source support
Status: Open | Priority: Critical | Resolution: 
Created: 30/Jun/23 21:43
Affects Version/s: 4.0.0

XML is a widely used data format. An external spark-xml package ([https://github.com/databricks/spark-xml)] is available to read and write XML data in spark. Making spark-xml built-in will provide a better user experience for Spark SQL and structured streaming. The proposal is to inline code from spark-xml package.

 

Here is the link to [SPIP|https://docs.google.com/document/d/1ZaOBT4-YFtN58UCx2cdFhlsKbie1ugAn-Fgz_Dddz-Q/edit]
Bug: Separate finalizeShuffleMerge Processing to Dedicated Thread Pools
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 07/Jun/23 05:01
Affects Version/s: 3.2.0

In our production environment, _finalizeShuffleMerge_ processing took longer time (p90 is around 20s) than other PRC requests. This is due to _finalizeShuffleMerge_ invoking IO operations like truncate and file open/close.  

More importantly, processing this _finalizeShuffleMerge_ can block other critical lightweight messages like authentications, which can cause authentication timeout as well as fetch failures. Those timeout and fetch failures affect the stability of the Spark job executions.