av9ash/GitBugs huggingface dataset:
Bug: CSV reader reads data inconsistently depending on column position
Status: Open | Priority: Critical | Resolution: 
Created: 02/Feb/24 12:09
Affects Version/s: 3.4.1

Reading the following CSV
{code:java}
"a";"b";"c";"d"
10;100,00;"Some;String";"ok"
20;200,00;"";"still ok"
30;300,00;"also ok";""
40;400,00;"";"" {code}
with these options
{code:java}
spark.read
        .option("header","true")
        .option("sep",";")
        .option("encoding","ISO-8859-1")
        .option("lineSep","\r\n")
        .option("nullValue","")
        .option("quote",'"')
        .option("escape","") {code}
results in the followin inconsistent dataframe

 
||a||b||c||d||
|10|100,00|Some;String|ok|
|20|200,00|<null>|still ok|
|30|300,00|also ok|"|
|40|400,00|<null>|"|

As one can see, the quoted empty fields of the last column are not correctly read as null but instead contain a single double quote. It works for column c.

If I recall correctly, this only happens when the "escape" option is set to an empty string. Not setting it to "" (defaults to "\") seems to not cause this bug.

I observed this on databricks spark runtime 13.2 (think that is spark 3.4.1).
Bug: Inconsistent results with 'sort', 'cache', and AQE.
Status: Open | Priority: Critical | Resolution: 
Created: 06/Feb/24 17:10
Affects Version/s: 3.3.2

 
With AQE enabled, having {color:#4c9aff}sort{color} in the plan changes {color:#4c9aff}sample{color} results after caching.

Moreover, when cached,  {color:#4c9aff}collect{color} returns records as if it's not cached, which is inconsistent with {color:#4c9aff}count{color} and {color:#4c9aff}show{color}.

A script to reproduce:
{code:scala}
import spark.implicits._
val df = (1 to 4).toDF("id").sort("id").sample(0.4, 123)

println("NON CACHED:")

println("  count: " + df.count())
println("  collect: " + df.collect().mkString(" "))
df.show()

println("CACHED:")
df.cache().count()

println("  count: " + df.count())
println("  collect: " + df.collect().mkString(" "))
df.show()

df.unpersist()
{code}
output:
{code:java}
NON CACHED:
  count: 2
  collect: [1] [4]
+---+
| id|
+---+
|  1|
|  4|
+---+

CACHED:
  count: 3
  collect: [1] [4]
+---+
| id|
+---+
|  1|
|  2|
|  3|
+---+
{code}
BTW, disabling AQE [{color:#4c9aff}spark.conf.set("spark.databricks.optimizer.adaptive.enabled", "false"){color}] helps on Databricks clusters, but locally it has no effect, at least on Spark 3.3.2.
Bug: Recover -1 and 0 case for spark.sql.execution.arrow.maxRecordsPerBatch
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 16/Feb/24 01:16
Affects Version/s: 3.4.1

{code}
import pandas as pd
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", 0)
spark.conf.set("spark.sql.execution.arrow.pyspark.fallback.enabled", False)
spark.createDataFrame(pd.DataFrame({'a': [123]})).toPandas()

spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", -1)
spark.createDataFrame(pd.DataFrame({'a': [123]})).toPandas()
{code}

{code}
/.../spark/python/pyspark/sql/pandas/conversion.py:371: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and will not continue because automatic fallback with 'spark.sql.execution.arrow.pyspark.fallback.enabled' has been set to false.
  range() arg 3 must not be zero
  warn(msg)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/session.py", line 1483, in createDataFrame
    return super(SparkSession, self).createDataFrame(  # type: ignore[call-overload]
  File "/.../spark/python/pyspark/sql/pandas/conversion.py", line 351, in createDataFrame
    return self._create_from_pandas_with_arrow(data, schema, timezone)
  File "/.../spark/python/pyspark/sql/pandas/conversion.py", line 633, in _create_from_pandas_with_arrow
    pdf_slices = (pdf.iloc[start : start + step] for start in range(0, len(pdf), step))
ValueError: range() arg 3 must not be zero
{code}

{code}
Empty DataFrame
Columns: [a]
Index: []
{code}
Bug: Spark Container doesn't have spark group or spark user created
Status: Open | Priority: Critical | Resolution: 
Created: 20/Feb/24 18:08
Affects Version/s: 3.4.1

I see that [https://github.com/apache/spark-docker/blob/431aa516ba58985c902bf2d2a07bf0eaa1df6740/3.4.1/scala2.12-java11-ubuntu/Dockerfile#L19] is supposed to have a spark user and spark group created but checking the container, it doesn't have those uid and gid created.  Both should have 185 uid and 185 gid.

I have no name!@spark-hudi:/opt/spark/bin$ cat /etc/group
root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
disk:x:6:
lp:x:7:
mail:x:8:
news:x:9:
uucp:x:10:
man:x:12:
proxy:x:13:
kmem:x:15:
dialout:x:20:
fax:x:21:
voice:x:22:
cdrom:x:24:
floppy:x:25:
tape:x:26:
sudo:x:27:
audio:x:29:
dip:x:30:
www-data:x:33:
backup:x:34:
operator:x:37:
list:x:38:
irc:x:39:
src:x:40:
gnats:x:41:
shadow:x:42:
utmp:x:43:
video:x:44:
sasl:x:45:
plugdev:x:46:
staff:x:50:
games:x:60:
users:x:100:
nogroup:x:65534:
I have no name!@spark-hudi:/opt/spark/bin$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin

I have no name!@spark-hudi:/opt/spark/bin$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
Bug: SPJ : Dynamically rebalance number of buckets when they are not equal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/24 22:56
Affects Version/s: 3.3.0

SPJ: Storage Partition Join works with Iceberg tables when both the tables have same number of buckets. As part of this feature request, we would like spark to gather the number of buckets information from both the tables and dynamically rebalance the number of buckets by coalesce or repartition so that SPJ will work fine. In this case, we would still have to shuffle but would be better than no SPJ.

Use Case : 

Many times we do not have control of the input tables, hence it's not possible to change partitioning scheme on those tables. As a consumer, we would still like them to be used with SPJ when used with other tables and output tables which has different number of buckets.

In these scenario, we would need to read those tables rewrite them with matching number of buckets for the SPJ to work, this extra step could outweigh the benefits of less shuffle via SPJ. Also when there are multiple different tables being joined, each tables need to be rewritten with matching number of buckets. 

If this feature is implemented, SPJ functionality will be more powerful.
Bug: Allow overriding base overhead memory
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/24 10:00
Affects Version/s: 3.5.1

We can already select the desired overhead memory directly via the _'spark.driver/executor.memoryOverhead'_ flags, however, if that flag is not present the overhead memory calculation goes as follows:
{code:java}
overhead_memory = Max(384, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor')

where the 'memoryOverheadFactor' flag defaults to 0.1{code}

There are certain times where being able to override the 384Mb minimum directly can be beneficial. We may have a scenario where a lot of off-heap operations are performed (ex: using package managers/native compression/decompression) where we don't have a need for a large JVM heap but we may still need a signficant amount of memory in the spark node. 

Using the '{_}memoryOverheadFactor{_}' flag may not prove appropriate. Since we may not want the overhead allocation to directly scale with JVM memory, as a cost saving/resource limitation problem.

As such, I propose the addition of a 'spark.driver/executor.minMemoryOverhead' flag, which can be used to override the 384Mib value used in the overhead calculation.

The memory overhead calculation will now be :
{code:java}
min_memory = sparkConf.get('spark.driver/executor.minMemoryOverhead').getOrElse(384)

overhead_memory = Max(min_memory, 'spark.driver/executor.memory' * 'spark.driver/executor.memoryOverheadFactor'){code}

PR: https://github.com/apache/spark/pull/45240
Bug: Reenble AmmoniteTest tests in Maven builds
Status: Open | Priority: Major | Resolution: 
Created: 21/Feb/24 03:02
Affects Version/s: 4.0.0

Disabled in https://github.com/apache/spark/pull/45186 because of TTY issues.

See also https://github.com/apache/spark/pull/43909 and https://github.com/apache/spark/pull/40675
Bug: Spark to support S3 Express One Zone Storage
Status: Open | Priority: Major | Resolution: 
Created: 08/Feb/24 14:45
Affects Version/s: 4.0.0

Hadoop 3.4.0 adds support for AWS S3 Express One Zone Storage.

Most of this is transparent. However, one aspect which can surface as an issue is that these stores report prefixes in a listing when there are pending uploads, *even when there are no files underneath*

This leads to a situation where a listStatus of a path returns a list of file status entries which appears to contain one or more directories -but a listStatus on that path raises a FileNotFoundException: there is nothing there.

HADOOP-18996 handles this in all of hadoop code, including FileInputFormat, 

A filesystem can now be probed for inconsistent directoriy listings through {{fs.hasPathCapability(path, "fs.capability.directory.listing.inconsistent")}}

If true, then treewalking code SHOULD NOT report a failure if, when walking into a subdirectory, a list/getFileStatus on that directory raises a FileNotFoundException.

Although most of this is handled in the hadoop code, but there some places where treewalking is done inside spark These need to be identified and make resilient to failure on the recurse down the tree

* SparkHadoopUtil list methods , 
* especially listLeafStatuses used by OrcFileOperator
org.apache.spark.util.Utils#fetchHcfsFile

{{org.apache.hadoop.fs.FileUtil.maybeIgnoreMissingDirectory()}} can assist here, or the logic can be replicated. Using the hadoop implementation would be better from a maintenance perspective
Bug: Null comparison push down data filter from subquery produces in NPE in Parquet filter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/24 19:09
Affects Version/s: 3.5.0

This issue has been introduced in [https://github.com/apache/spark/pull/41088]  where we convert scalar subqueries to literals and then convert the literals to {{{}org.apache.spark.sql.sources.Filters{}}}. These filters are then pushed down to parquet.

If the literal is a comparison with {{null}} then the parquet filter conversion code throws NPE. 

 

repro code which results in NPE
{code:java}
create table t1(d date) using parquet
create table t2(d date) using parquet
insert into t1 values date'2021-01-01'
insert into t2 values (null)
select * from t1 where 1=1 and d > (select d from t2){code}
[fix PR |https://github.com/apache/spark/pull/45202/files]
Bug: Deprecate `spark.sql.legacy.allowZeroIndexInFormatString`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/24 09:39
Affects Version/s: 4.0.0

The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't allow to use the zero index. Even set it to true, users get the error:

{code:sql}
> select format_string('%0$s', 'Hello');

Illegal format argument index = 0
java.util.IllegalFormatArgumentIndexException: Illegal format argument index = 0
	at java.base/java.util.Formatter$FormatSpecifier.index(Formatter.java:2808)
	at java.base/java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2879)
{code}
Bug: The column pruning is not working as expected for nested struct in an array
Status: Open | Priority: Major | Resolution: 
Created: 29/Feb/24 15:31
Affects Version/s: 3.3.0

See the code snippet below, when explode an array of struct and select one field in the struct,  some unexpected behaviour observed:
 * If the field in the struct is in the select clause, not in the where clause, the column pruning works as expected.
 * If the field in the struct is in the select clause and in the where clause, the column pruning not working.
 * If the field in the struct is not even selected, the column pruning not working

{code:java}
from pyspark.sql import SparkSession
from pyspark.sql.types import IntegerType, StructField, StructType, ArrayType
import random

spark = SparkSession.builder.appName("example").getOrCreate()# Create an RDD with an array of structs, each array having a random size between 5 to 10
rdd = spark.range(1000).rdd.map(lambda x: (x.id + 3, [(x.id + i, x.id - i) for i in range(1, random.randint(5, 11))]))

# Define a new schema

schema = StructType([
    StructField("a", IntegerType(), True),
    StructField("b", ArrayType(StructType([
        StructField("x", IntegerType(), True),
        StructField("y", IntegerType(), True)
    ])), True)
])

# Create a DataFrame with the new schema
df = spark.createDataFrame(rdd, schema=schema)

# Write the DataFrame to a parquet file
df.repartition(1).write.mode('overwrite').parquet('test.parquet')

# Read the parquet file back into a DataFrame
df = spark.read.parquet('test.parquet') 
df.createOrReplaceTempView("df_view")
spark.conf.set('spark.sql.optimizer.nestedSchemaPruning.enabled', 'true')

# case 1, as expected
sql_query = """
SELECT a, EXPLODE(b.x) AS bb
FROM df_view
"""
spark.sql(sql_query).explain()

# ReadSchema: struct<a:int,b:array<struct<x:int>>>

# case 2, as expected
sql_query = """
SELECT a, bb.x 
FROM df_view 
lateral view explode(b) as bb
"""
spark.sql(sql_query).explain()

# ReadSchema: struct<a:int,b:array<struct<x:int>>>

# case 3, bug: should only read b.x
sql_query = """
SELECT a, bb.x 
FROM df_view 
lateral view explode(b) as bb
where bb.x is not null
"""
spark.sql(sql_query).explain()

#ReadSchema: struct<a:int,b:array<struct<x:int,y:int>>>

#case 4, bug? seems no need to read both a and b
sql_query = """
SELECT a
FROM df_view 
lateral view explode(b) as bb
"""
spark.sql(sql_query).explain()

#ReadSchema: struct<a:int,b:array<struct<x:int,y:int>>>{code}
Bug: Allow AQE coalesce final stage in SQL cached plan
Status: Open | Priority: Major | Resolution: 
Created: 07/Feb/24 01:49
Affects Version/s: 4.0.0

[https://github.com/apache/spark/pull/43435] and [https://github.com/apache/spark/pull/43760] are fixing a correctness issue which will be triggered when AQE applied on cached query plan, specifically, when AQE coalescing the final result stage of the cached plan.

 

The current semantic of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}}

([source code|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala#L403-L411]):
 * when true, we enable AQE, but disable coalescing final stage ({*}default{*})

 * when false, we disable AQE

 

But let’s revisit the semantic of this config: actually for caller the only thing that matters is whether we change the output partitioning of the cached plan. And we should only try to apply AQE if possible.  Thus we want to modify the semantic of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}}
 * when true, we enable AQE and allow coalescing final: this might lead to perf regression, because it introduce extra shuffle

 * when false, we enable AQE, but disable coalescing final stage. *(this is actually the `true` semantic of old behavior)*

Also, to keep the default behavior unchanged, we might want to flip the default value of {{spark.sql.optimizer.canChangeCachedPlanOutputPartitioning}} to `false`
Bug: Addition of implicit casting without indeterminate support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/24 13:48
Affects Version/s: 4.0.0

*What changes were proposed in this pull request?*
This PR adds automatic casting and collations resolution as per `PGSQL` behaviour:

1. Collations set on the metadata level are implicit
2. Collations set using the `COLLATE` expression are explicit
3. When there is a combination of expressions of multiple collations the output will be:
 - if there are explicit collations and all of them are equal then that collation will be the output
 - if there are multiple different explicit collations `COLLATION_MISMATCH.EXPLICIT` will be thrown
 - if there are no explicit collations and only a single type of non default collation, that one will be used
 - if there are no explicit collations and multiple non-default implicit ones `COLLATION_MISMATCH.IMPLICIT` will be thrown

*Why are the changes needed?*
We need to be able to compare columns and values with different collations and set a way of explicitly changing the collation we want to use.
Bug: Avoid to materialize AQE ExchangeQueryStageExec on the cancellation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/24 18:00
Affects Version/s: 4.0.0

AQE can materialize both *ShuffleQueryStage* and *BroadcastQueryStage* on the cancellation. This causes unnecessary stage materialization by submitting Shuffle Job and Broadcast Job. Under normal circumstances, if the stage is already non-materialized (a.k.a *ShuffleQueryStage.shuffleFuture* or *{{BroadcastQueryStage.broadcastFuture}}* is not initialized yet), it should just be skipped without materializing it.

Please find sample use-case:
*1- Stage Materialization Steps:*
When stage materialization is failed:
{code:java}
1.1- ShuffleQueryStage1 - is materialized successfully,
1.2- ShuffleQueryStage2 - materialization is failed,
1.3- ShuffleQueryStage3 - Not materialized yet so ShuffleQueryStage3.shuffleFuture is not initialized yet{code}
*2- Stage Cancellation Steps:*
{code:java}
2.1- ShuffleQueryStage1 - is canceled due to already materialized,
2.2- ShuffleQueryStage2 - is earlyFailedStage so currently, it is skipped as default by AQE because it could not be materialized,
2.3- ShuffleQueryStage3 - Problem is here: This stage is not materialized yet but currently, it is also tried to cancel and this stage requires to be materialized first.{code}
Bug: Support enabling migration of shuffle data directly to external storage using config parameter
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/24 03:27
Affects Version/s: 4.0.0

Currently Spark supports migration of shuffle data to peer nodes during node decommissioning. If peer nodes are not accessible, then Spark falls back to external storage. User needs to provide the storage location path. There are scenarios where user may want to migrate to external storage instead of peer nodes. This may be because of unstable  nodes or due to the need of aggressive scale down. So user should be able to configure to migrate the shuffle data directly to external storage if the use case permits.
Bug: De-duplication of Relations in Joins, can result in plan resolution failure
Status: Open | Priority: Major | Resolution: 
Created: 28/Feb/24 22:56
Affects Version/s: 3.5.1

In case of some flavours of  nested joins involving repetition of relation, the projected columns when passed to the DataFrame.select API , as form of df.column , can result in plan resolution failure due to attribute resolution not happening.

A scenario in which this happens is
{noformat}
               
                          Project ( dataframe A.column("col-a") )
                                         |
                                      Join2
                          |                            | 
                       Join1                      DataFrame A      
                          |
         DataFrame A            DataFrame B

{noformat}
In such cases, If it so happens that Join2 - right leg DataFrame A gets re-aliased due to De-Duplication of relations, and if the project uses Column definition obtained from DataFrame A, its exprId will not match the re-aliased Join2 - right Leg- DataFrame A , causing resolution failure.
Bug: Executor memory related initialization should happen after custom DriverPlugin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/24 23:15
Affects Version/s: 4.0.0

Today, almost all executor memory related settings, including {{UnifiedMemoryManager}}, {{ResourceProfileManager}}, etc, are initialized before a custom Driver plugin is initialized. This means it is impossible to overwrite memory related configurations for Spark executors.

This JIRA proposes to change the initialization for the above.
Bug: Fix `spark-common-utils` module to have explicit `jackson-core` and `jackson-annotations`dependency
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/24 14:58
Affects Version/s: 4.0.0

Following scala code depends on `jackson-core` and `jackson-annotations` explicitly.  However, spark-common-utils modules missing related dependency.
{code:java}
~/dev/sources/spark$ grep -R jackson.core ./common/utils/* | grep import
./common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala:import com.fasterxml.jackson.core.`type`.TypeReference
./common/utils/src/main/scala/org/apache/spark/util/JsonUtils.scala:import com.fasterxml.jackson.core.{JsonEncoding, JsonGenerator}
~/dev/sources/spark$ grep -R jackson.annotation ./common/utils/* | grep import
./common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala:import com.fasterxml.jackson.annotation.JsonIgnore
{code}
Bug: Fix `spark-common-utils` module to have explicit `commons-lang3` dependency
Status: Open | Priority: Major | Resolution: 
Created: 14/Feb/24 14:22
Affects Version/s: 4.0.0

Following scala code depends on `commons-lang3` explicitly.  However, the common-utils modules missing related dependency.
{code:java}
~/dev/sources/spark/common/utils$ grep -R lang3 * | grep import
src/main/scala/org/apache/spark/util/MavenUtils.scala:import org.apache.commons.lang3.StringUtils
src/main/scala/org/apache/spark/util/ClosureCleaner.scala:import org.apache.commons.lang3.ClassUtils
src/main/java/org/apache/spark/network/util/JavaUtils.java:import org.apache.commons.lang3.SystemUtils; {code}