av9ash/GitBugs huggingface dataset:
Bug: Ensure Spark Connect scala client CloseableIterator is closed in all cases where exception could be thrown
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/23 14:40
Affects Version/s: 4.0.0

We currently already ensure that CloseableIterator is consumed in all places, and in  ExecutePlanResponseReattachableIterator we ensure that the iterator is closed in case of GRPC error.

We should also ensure that all places in the client that use a CloseableIterator will close it gracefully, also in case of another exception being thrown, including InterruptedException.

Some try \{ } finally \{ iteretor.close() } blocks may be needed for that.
Bug: Raise a ArrowIOError with empty messages
Status: Open | Priority: Major | Resolution: 
Created: 04/Aug/23 09:34
Affects Version/s: 2.4.4

We encountered an problem while using PySpark 2.4.4, where the Python Runner of a certain task threw a pyarrow.lib.ArrowIOError without any message, which is too confusing. And this task was scheduled multiple times, all of these task attempts failed for the same pyarrow.lib.ArrowIOError.
Bug: Spark Connect reattachable Execute when initial ExecutePlan didn't reach server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Aug/23 20:39
Affects Version/s: 3.5.0

If the ExecutePlan never reached the server, a ReattachExecute will fail with INVALID_HANDLE.OPERATION_NOT_FOUND. In that case, we could try to send ExecutePlan again.
Bug: __repr__ broken for Row when the field is empty Row
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 20:05
Affects Version/s: 3.5.0

PySpark {{Row}} raises and exception if the field is empty Row:

{code:python}
>>> repr(Row(Row()))
Traceback (most recent call last):
...
TypeError: not enough arguments for format string
{code}
Bug: pandas-on-Spark Dataframe.between_time fails when timestamp fields are present
Status: Open | Priority: Major | Resolution: 
Created: 02/Aug/23 07:46
Affects Version/s: 3.3.1

I tried to execute the between_time() method of a pandas-on-Spark dataframe that has a DatatimeIndex and contains fields of timestamp types, and hit the following error.

!image.png!
Bug: Remove session-based directory when the isolated session cache is evicted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Aug/23 04:13
Affects Version/s: 3.5.0

SPARK-44078 added the cache for isolated sessions, and SPARK-44348 added the session-based directory for isolation.

 

When the isolated session cache is evicted, we should remove the session-based directory so it doesn't fail when the same session is used, see also https://github.com/apache/spark/pull/41625#discussion_r1251427466
Bug: VolcanoFeatureStep should not create `DefaultVolcanoClient` if not needed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/23 00:13
Affects Version/s: 4.0.0

Since `VolcanoFeatureStep` creates `DefaultVolcanoClient` always, the unit test suite `VolcanoFeatureStepSuite` behaves like an integration test. In other words, it fails when there is no backend K8s clusters.

{code}
$ build/sbt -Pkubernetes -Pvolcano "kubernetes/testOnly *Volcano* -- -z SPARK-36061"
...
[info] VolcanoFeatureStepSuite:
[info] - SPARK-36061: Driver Pod with Volcano PodGroup *** FAILED *** (646 milliseconds)
[info]   org.snakeyaml.engine.v2.exceptions.ScannerException: mapping values are not allowed here
[info]  in reader, line 1, column 94:
[info]      ... well-known/openid-configuration": dial tcp: lookup iam.corp. ...
[info]                                          ^
{code}
Bug: Add configurations for FileStreamSource cached files
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Aug/23 02:32
Affects Version/s: 3.1.0

With https://issues.apache.org/jira/browse/SPARK-30866, caching of listed files was added for structured streaming to reduce cost of relisting from filesystem each batch.  The settings that drive this are currently hardcoded and there is no way to change them.  

 

This impacts some of our workloads where we process large datasets where its unknown how "heavy" some files are, so a single batch can take a long period of time.  When we set maxFilesPerTrigger to 100k files, a subsequent batch using the cached max of 10k files is causing the job to take longer since the cluster is capable of handling the 100k files but is stuck doing 10% of the workload.  The benefit of the caching doesn't outweigh the cost of the performance on the rest of the job.

 

With config settings available for this, we could either absorb some increased driver memory usage for caching the next 100k files, or opt to disable caching entirely and just relist files each batch by setting the cache amount to 0.
Bug: Upgrade Derby to 10.15.1.3+
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Aug/23 03:08
Affects Version/s: 4.0.0

Currently, Derby is fixed at 10.14.2.0 because 10.15.1.3 requires JDK9 at least.

This issue aims to upgrade it after dropping Java 8.
Bug: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'default' not found (state=08S01,code=0)
Status: Open | Priority: Minor | Resolution: 
Created: 30/Aug/23 11:16
Affects Version/s: 3.1.0

There is an alert that fires up when a Spark 3.1 cluster is created using shared metastore with Spark 2.4. The alert says DefaultDatabase does not exist. This is misleading and thus we need to suppress this alert. 
In the class SessionCatalog.scala, the method requireDbExists() is not handling the case when the db = defaultDB. This needs to be added to suppress this misleading alert.
Bug: Include full identifier in Storage tab
Status: Open | Priority: Minor | Resolution: 
Created: 31/Aug/23 22:38
Affects Version/s: 4.0.0

If you have 2 databases with tables with the same name like:  db1.table, db2.table

 
{code:java}
scala> spark.sql("use db1")

scala> spark.sql("cache table table")
   
scala> spark.sql("use db2")

scala> spark.sql("cache table table"){code}
You cannot differentiate it in the Storage tab

!image-2023-08-31-18-38-37-856.png!
Bug: No need to retry parsing event log path again when FileNotFoundException occurs
Status: Open | Priority: Minor | Resolution: 
Created: 29/Aug/23 02:46
Affects Version/s: 3.4.1

I found a lot of retry parsing inprogress event log records in history server log. The application is already done while parsing, so we don't need to retry parsing it again when FileNotFoundException occurs.

 

!image-2023-08-29-10-47-08-027.png!

!image-2023-08-29-10-47-43-567.png!
Bug: In subquery cannot perform partition pruning
Status: Open | Priority: Minor | Resolution: 
Created: 03/Aug/23 09:21
Affects Version/s: 3.0.0

The following SQL cannot perform partition pruning
{code:java}
SELECT * FROM parquet_part WHERE id_type in (SELECT max(id_type) from parquet_part){code}
As can be seen from the execution plan below, the partition pruning of left cannot be performed after the subquery of in is converted into join

!image-2023-08-03-17-22-53-981.png!

The current issue proposes to optimize insubquery. Only when the value of in is greater than a threshold, insubquery will be converted into Join
Bug: Support memory limit configurable
Status: Open | Priority: Minor | Resolution: 
Created: 10/Aug/23 09:20
Affects Version/s: 3.4.1

Currently the memory request and limit are set by summing the values of spark.\{driver,executor}.memory and spark.\{driver,executor}.memoryOverhead. Supporting memory limits configurable can bring some benefits. For example, use unfixed memory to use page cache, reduce disk IO of shuffle read to improve performance.
Bug: SPJ: Include keyGroupedPartitioning in StoragePartitionJoinParams equality check
Status: Open | Priority: Minor | Resolution: 
Created: 03/Aug/23 17:57
Affects Version/s: 3.5.0

Currently {{StoragePartitionJoinParams}} doesn't include {{keyGroupedPartitioning}} in its {{equals}} and {{hashCode}} computation. For completeness, we should include it as well since it is a member of the class.
Bug: Log a warning when inserting columns with the same name by row that don't match up
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Aug/23 03:41
Affects Version/s: 3.4.2

With SPARK-42750 people can now insert by name, but sometimes people forget it. We should log warning when it *looks like* someone forgot it (e.g. insert by column number with all the same names *but* not matching up in row).
Bug: Improve error message for ALTER TABLE ALTER COLUMN on partition columns in non-delta tables
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/23 22:18
Affects Version/s: 3.0.3

 
{code:java}
-- hive table
sql("create table some_table (x int, y int, z int) using parquet PARTITIONED BY (x, y) " +
"location '/Users/someone/runtime/tmp-data/some_table'")
sql("alter table some_table alter column x comment 'some-comment'").collect()
Can't find column `x` given table data columns [`z`].{code}
Improve error message to indicate to users that the command is not supported on partition columns in non-delta tables.
Bug: Considering 'daemon', 'priority' from higher JDKs for ThreadStackTrace class
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 21/Aug/23 11:05
Affects Version/s: 4.0.0

{code:java}
jshell> var t = java.lang.management.ManagementFactory.getThreadMXBean()t ==> com.sun.management.internal.HotSpotThreadImpl@7daf6ecc
jshell> var tt = t.dumpAllThreads(true, true)tt ==> ThreadInfo[10] { "main" prio=5 Id=1 RUNNABLE	at  ... k$NonfairSync@27fa135a
 }
jshell> for (java.lang.management.ThreadInfo t1: tt) {System.out.println(t1.toString());}"main" prio=5 Id=1 RUNNABLE	at java.management@20.0.1/sun.management.ThreadImpl.dumpThreads0(Native Method)	at java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:540)	at java.management@20.0.1/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:527)	at REPL.$JShell$12.do_it$Aux($JShell$12.java:7)	at REPL.$JShell$12.do_it$($JShell$12.java:11)	at java.base@20.0.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)	at java.base@20.0.1/java.lang.invoke.LambdaForm$MH/0x0000007001008c00.invoke(LambdaForm$MH)	at java.base@20.0.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)	...

"Reference Handler" daemon prio=10 Id=8 RUNNABLE	at java.base@20.0.1/java.lang.ref.Reference.waitForReferencePendingList(Native Method)	at java.base@20.0.1/java.lang.ref.Reference.processPendingReferences(Reference.java:246)	at java.base@20.0.1/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:208) {code}
the `daemon prio=10` is not available for ThreadInfo of jdk8
Bug: There should be a gap at the bottom of the HTML
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/23 08:44
Affects Version/s: 4.0.0

Before:

!image-2023-08-28-16-47-11-582.png|width=794,height=392!

 

After:

!image-2023-08-28-16-46-04-705.png|width=744,height=329!
Bug: Refactor ExternalSorter to reduce checks on shouldPartition when calling getPartition
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 29/Aug/23 03:39
Affects Version/s: 4.0.0

{code:java}
  private def getPartition(key: K): Int = {
    if (shouldPartition) partitioner.get.getPartition(key) else 0
  } {code}
 

The {{getPartition}} method checks {{shouldPartition}} every time it is called. However, {{shouldPartition}} should not be able to change after the {{ExternalSorter}} is instantiated. Therefore, it can be refactored to reduce the checks on {{{}shouldPartition{}}}.
Bug: Cache results of simple udfs on executors if same arguments are passed.
Status: Open | Priority: Minor | Resolution: 
Created: 27/Aug/23 13:02
Affects Version/s: 3.4.1

Consider two dataframes :

{{keyword_given = [
["green pstr",],
["greenpstr",],
["wlmrt", ],
["walmart",],
["walmart super",]
]}}

{{variations = [
("type green pstr", "ABC", 100),
("type green pstr","PQR",200),
("type green pstr", "NZSD", 2999),
("wlmrt payment","walmart",200),
("wlmrt solutions", "walmart", 200),
("nppssdwlmrt", "walmart", 2000)
]}}

{{Imagine I have a task to do fuzzy substring matching between keyword and variation[0] using in built levenstein function. It is possible to optimize this futher in the code itself where we extract out the uniques and then do fuzzy matching on the uniques and join back with the original tables. }}

{{But it could be possible as an optimization to cache the results of the already computed udfs till now and do a lookup on each executor separately.}}

Just a thought. Not sure if it makes any sense. This behaviour could be behind a config.
Bug: Filter out static configurations used in local mode
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Aug/23 02:41
Affects Version/s: 4.0.0

If you set a static configuration with `--remote local` mode, it shows a bunch of warnings as below:

{code}
23/08/28 11:39:42 ERROR ErrorUtils: Spark Connect RPC error during: config. UserId: hyukjin.kwon. SessionId: 424674ef-af95-4b12-b10e-86479413f9fd.
org.apache.spark.sql.AnalysisException: Cannot modify the value of a static config: spark.connect.copyFromLocalToFs.allowDestLocal.
	at org.apache.spark.sql.errors.QueryCompilationErrors$.cannotModifyValueOfStaticConfigError(QueryCompilationErrors.scala:3227)
	at org.apache.spark.sql.RuntimeConfig.requireNonStaticConf(RuntimeConfig.scala:162)
	at org.apache.spark.sql.RuntimeConfig.set(RuntimeConfig.scala:42)
	at org.apache.spark.sql.connect.service.SparkConnectConfigHandler.$anonfun$handleSet$1(SparkConnectConfigHandler.scala:67)
	at org.apache.spark.sql.connect.service.SparkConnectConfigHandler.$anonfun$handleSet$1$adapted(SparkConnectConfigHandler.scala:65)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at org.apache.spark.sql.connect.service.SparkConnectConfigHandler.handleSet(SparkConnectConfigHandler.scala:65)
	at org.apache.spark.sql.connect.service.SparkConnectConfigHandler.handle(SparkConnectConfigHandler.scala:40)
	at org.apache.spark.sql.connect.service.SparkConnectService.config(SparkConnectService.scala:120)
	at org.apache.spark.connect.proto.SparkConnectServiceGrpc$MethodHandlers.invoke(SparkConnectServiceGrpc.java:751)
	at org.sparkproject.connect.grpc.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at org.sparkproject.connect.grpc.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:346)
	at org.sparkproject.connect.grpc.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:860)
	at org.sparkproject.connect.grpc.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at org.sparkproject.connect.grpc.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}
Bug: Taking sum of two columns behaves differently from sum aggregation function
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 24/Aug/23 14:22
Affects Version/s: 3.4.1

Taking the sum of two columns behaves differently when there are NULL values than taking the SUM of a column. This is odd and confusing for users

Reproducible example: 
{code:java}
$ from pyspark.sql import SparkSession
$ import pyspark.sql.functions as f
$ spark = SparkSession.builder.getOrCreate()

$ df = spark.createDataFrame([(1, 2), (2, None)], ["foo", "bar"])
$ df.show()
> 
+---+----+
|foo| bar|
+---+----+
|  1|   2|
|  2|null|
+---+----+

$ df.select(f.sum("foo"), f.sum("bar")).show()
>
+--------+--------+
|sum(foo)|sum(bar)|
+--------+--------+
|       3|       2|
+--------+--------+

$ df.select((f.col("foo") + f.col("bar")).alias("sum(foobar)")).show()
> 
+-----------+
|sum(foobar)|
+-----------+
|          3|
|       null|
+-----------+

// I expected to get, but I was surprised to see the result above
+-----------+
|sum(foobar)|
+-----------+
|          3|
|          2|
+-----------+
{code}
Bug: Fix test_assert_approx_equal_decimaltype_custom_rtol_pass when SPARK_ANSI_SQL_MODE=true
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/23 09:01
Affects Version/s: 4.0.0

https://github.com/apache/spark/actions/runs/5873530086/job/15926967006

!image-2023-08-16-17-01-43-819.png!
Bug: PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 23/Aug/23 20:47
Affects Version/s: 3.3.3

When running the query
{code:java}
with cte as (
 select c1, c1, c2, c3 from t where random() > 0
)
select cte.c1, cte2.c1, cte.c2, cte2.c3 from
 (select c1, c2 from cte) cte
 inner join
 (select c1, c3 from cte) cte2
 on cte.c1 = cte2.c1 {code}
 
The query fails with the error
{code:java}
org.apache.spark.scheduler.DAGScheduler: Failed to update accumulator 9523 (Unknown class) for task 1
org.apache.spark.SparkException: attempted to access non-existent accumulator 9523{code}
Further investigation shows that the rule PushdownPredicatesAndPruneColumnsForCTEDef creates an invalid plan when the output of a CTE contains duplicate expression IDs.
Bug: Make Kubernetes[Driver|Executor]Conf.annotations substitute annotations instead of feature steps
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Aug/23 05:57
Affects Version/s: 4.0.0

Move Utils. SubstituteAppNExecIds logic  into KubernetesConf.annotations as the default logic, easy for users to reuse, rather than to rewrite it again at the same logic.
Bug: Python timedelta to DayTimeIntervalType edge cases bug
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Aug/23 16:57
Affects Version/s: 3.4.0

h1. Python Timedelta to PySpark DayTimeIntervalType bug

There is a bug that exists which means certain Python datetime.timedelta objects get converted to a PySpark DayTimeIntervalType column with a different value to that which is stored in the Python timedelta.

A simple illustrative example can be produced with the below code:

 
{code:java}
from datetime import timedelta
from pyspark.sql.types import DayTimeIntervalType, StructField, StructType

spark = ...spark session setup here...

td = timedelta(days=4498031, seconds=16054, microseconds=999981)
df = spark.createDataFrame([(td,)], StructType([StructField(name="timedelta_col", dataType=DayTimeIntervalType(), nullable=False)]))
df.show(truncate=False)

> +------------------------------------------------+
> |timedelta_col                                   | 
> +------------------------------------------------+ 
> |INTERVAL '4498031 04:27:35.999981' DAY TO SECOND| 
> +------------------------------------------------+

print(str(td))

>  '4498031 days, 4:27:34.999981' {code}
In the above example, look at the seconds. The original python timedelta object has 34 seconds, the pyspark DayTimeIntervalType column has 35 seconds.
h1. Fix

This issue arises because the current conversion from python timedelta uses the timedelta function `.total_seconds()` to get the number of seconds, and then adds the microsecond component back in afterwards. Unfortunately the `.total_seconds()` function with some timedeltas (ones with microsecond entries close to 1_000_000 I believe) ends up rounding *up* to the nearest second (probably due to floating point precision), with the microseconds then added on top of that. The effect is that 1 second gets added incorrectly.

The issue can be fixed by doing the processing in a slightly different way. Instead of doing:

 
{code:java}
(math.floor(dt.total_seconds()) * 1000000) + dt.microseconds{code}
 

Instead we construct the timedelta from its components:

 
{code:java}
(((dt.days * 86400) + dt.seconds) * 1_000_000) + dt.microseconds {code}
 
h1. Tests

An illustrative edge case example for timedeltas is the above (which can also be written as `datetime.timedelta(microseconds=388629894454999981)`)

 

A related edge case which is already handled but not tested exists for the situation where there are positive and negative components to the created timedelta object. An entry for this edge case is also included as it is related.
h1. PR

Link to the PR addressing this issue: https://github.com/apache/spark/pull/42541
h1. Keywords to help people searching for this issue:

datetime.timedelta

timedelta

pyspark.sql.types.DayTimeIntervalType

DayTimeIntervalType
Bug: Spark insertInto with location GCS bucket root causes NPE
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 20/Aug/23 11:00
Affects Version/s: 3.3.0

In our Organisation, we are using GCS bucket root location to point to our Hive table. Dataproc's latest 2.1 uses *Spark* *3.3.0* and this needs to be fixed.

Spark Scala code to reproduce this issue
{noformat}
val DF = Seq(("test1", 123)).toDF("name", "num")
DF.write.option("path", "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("schema_name.table_name")


val DF1 = Seq(("test2", 125)).toDF("name", "num")
DF.write.mode(SaveMode.Overwrite).format("orc").insertInto("schema_name.table_name")


java.lang.NullPointerException
  at org.apache.hadoop.fs.Path.<init>(Path.java:141)
  at org.apache.hadoop.fs.Path.<init>(Path.java:120)
  at org.apache.hadoop.fs.Path.suffix(Path.java:441)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.$anonfun$getCustomPartitionLocations$1(InsertIntoHadoopFsRelationCommand.scala:254) {noformat}
Looks like the issue is coming from Hadoop Path. 
{noformat}
scala> import org.apache.hadoop.fs.Path
import org.apache.hadoop.fs.Path
scala> val path: Path = new Path("gs://test_dd123/")
path: org.apache.hadoop.fs.Path = gs://test_dd123/

scala> path.suffix("/num=123")
java.lang.NullPointerException
  at org.apache.hadoop.fs.Path.<init>(Path.java:150)
  at org.apache.hadoop.fs.Path.<init>(Path.java:129)
  at org.apache.hadoop.fs.Path.suffix(Path.java:450){noformat}
Path.suffix throughs NPE when writing into GS buckets root.
Bug: Spark Connect: Cleaner thread not stopped when SparkSession stops
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 08/Aug/23 23:50
Affects Version/s: 3.5.0

Spark Connect scala client SparkSession has a cleaner, which starts a daemon thread to clean up Closeable objects after GC. This daemon thread is never stopped, and every SparkSession creates a new one.

Cleaner implements a stop() function, but no-one ever calls it. Possibly because even after SparkSession.stop(), the cleaner may still be needed when remaining references are GCed... For this reason it seems that the Cleaner should rather be a global singleton than within a session.
Bug: Expose uploadAllArtifactClasses in ArtifactManager to `sql` package
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Aug/23 11:39
Affects Version/s: 3.5.1

Currently, the [uploadAllClassFilesArtifacts|https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/connect/client/ArtifactManager.scala#L144-L146] method is private[client] but this limits the ability of non-client features to use UDFs (which require the class files). Currently, this is not an issue because classfiles are uploaded in all analyze/execute operations. Any new code paths would suffer from CNFE if they are not able to call this method.