av9ash/GitBugs huggingface dataset:
Bug: Upgrade several Maven plugins to the latest versions
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Feb/24 08:30
Affects Version/s: 4.0.0

* {{versions-maven-plugin}} from 2.16.0 to 2.16.2.
 * {{maven-enforcer-plugin}} from 3.3.0 to 3.4.1.
 * {{maven-compiler-plugin}} from 3.11.0 to 3.12.1.
 * {{maven-surefire-plugin}} from 3.1.2 to 3.2.5.
 * {{maven-clean-plugin}} from 3.3.1 to 3.3.2.
 * {{maven-javadoc-plugin}} from 3.5.0 to 3.6.3.
 * {{maven-shade-plugin}} from 3.5.0 to 3.5.1.
 * {{maven-dependency-plugin}} from 3.6.0 to 3.6.1.
 * {{maven-checkstyle-plugin}} from 3.3.0 to 3.3.1.
Bug: Honor .alias() in pivot aggregation
Status: Open | Priority: Minor | Resolution: 
Created: 15/Feb/24 17:28
Affects Version/s: 3.5.0

{{.alias()}} in {{df.groupBy("group_col").pivot("pivot_col").agg(sum("m1").alias(s"${col("pivot_col")}_m1")}} is not being honored when there's only one column `m1` to aggregate. But it's being honoured when there are multiple columns.

When {{alias}} is not provided in aggregate, it's fine to use pivot column values as the new column names. But if alias is given, it should be honored.
Bug: spark.rpc.askTimeout default value is not observed
Status: Open | Priority: Minor | Resolution: 
Created: 14/Feb/24 18:32
Affects Version/s: 3.5.0

I noticed an error in my Driver log:

 

org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply from /XXX.YYY.ZZZ.WWW:35560 in 10 seconds. This timeout is controlled by spark.rpc.askTimeout

 

The error message and indeed the Driver invocation command references a timeout value of 10 seconds.

 

However the Spark documentation implies that the default value for spark.rpc.askTimeout should be 120 seconds (indirectly from {{{}spark.network.timeout{}}})

 

Ref: [https://spark.apache.org/docs/3.5.0/configuration.html#networking]

 

I can see the problem from the code here: [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/Client.scala#L285]
Bug: Refactor refill() method to isExhausted() in NioBufferedFileInputStream
Status: Open | Priority: Minor | Resolution: 
Created: 08/Feb/24 08:05
Affects Version/s: 4.0.0

Currently, in NioBufferedFileInputStream, the refill() method is always invoked in a negated context (!refill()), which can be confusing and counter-intuitive. We can refactor the method so that it's no longer necessary to invert the result of the method call.
Bug: Document the config spark.sql.streaming.minBatchesToRetain
Status: Open | Priority: Minor | Resolution: 
Created: 09/Feb/24 00:17
Affects Version/s: 3.5.0

Add the config spark.sql.streaming.minBatchesToRetain to the [streaming docs|https://spark.apache.org/docs/latest/configuration.html#spark-streaming] page which basically controls the minimum number of batches that must be retained and made recoverable.

This would also help control the lifecycle of the state files held in the checkpoint folder i.e, State files are cleaned up based on the config spark.sql.streaming.minBatchesToRetain
Bug: Support higher dimensional array return values in predict_batch_udf beyond 1 or 2 dimensional arrays
Status: Open | Priority: Minor | Resolution: 
Created: 01/Feb/24 23:20
Affects Version/s: 3.5.0

pyspark.ml.functions.predict_batch_udf does not support return types with more than 2 dimensions:

[https://github.com/apache/spark/pull/37734#discussion_r1016156053]


Many computer vision models return ndarrays with 3 or 4 dimensions. Segmentation returns 3 dimensions: [Category, H, W]and if there is a time dimension, that's the fourth dimension.
Bug: Update usage of deprecated Thread.getId() to Thread.threadId()
Status: Closed | Priority: Trivial | Resolution: Invalid
Created: 29/Feb/24 04:28
Affects Version/s: 3.5.1

Update usage of deprecated Thread.getId() to Thread.threadId().

 

Currently in Spark, there are multiple references still to the deprecated method [Thread.getId()|#getId()]] given that the current version is using Java 21. Java officially requests any type of usage to be switched to the [Thread.threadId()|#threadId()]] method instead.
Bug: Mistake in Docstring for Pyspark's Dataframe.head()
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 22/Feb/24 13:39
Affects Version/s: 4.0.0

The docstring claims that {{head(n)}} would return a {{Row}} (rather than a list of rows) iff n == 1, but that's incorrect.

Type hints, example, and implementation show that the difference between row or list of rows lies in whether n is supplied at all -- if it isn't, {{head()}} returns a {{{}Row{}}}, if it is, even if it is 1, {{head(n)}} returns a list.

 

A suggestion to fix is here: https://github.com/apache/spark/pull/45197
Bug: Add Use Pandas API on Spark section on Pandas Scaling to large datasets page
Status: Open | Priority: Trivial | Resolution: 
Created: 23/Feb/24 20:48
Affects Version/s: 4.0.0

We should make a PR like [DOC: Add Use Modin section on Scaling to large datasets page|https://github.com/pandas-dev/pandas/issues/57585]

We can wait to see if pandas does accepts it.

I hope it will be a good thing and that it can attract more users.
Bug: Union of with non-determinstic expression should be non-deterministic
Status: Open | Priority: Trivial | Resolution: 
Created: 13/Feb/24 02:16
Affects Version/s: 3.5.0

We have special case handling for nullability already where any expression which is unioned with a nullable field becomes nullable, but we should do the same for deterministic.

 

I found this while I was poking around with push downs.

 

I believe the code to be updated would be output in the union case class.
Bug: Change the signature of the hllInvalidLgK query execution error to take an integer as 4th argument
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 02/Feb/24 19:46
Affects Version/s: 4.0.0

The current signature of the {{hllInvalidLgK}} query execution error takes four arguments:
 # The SQL function (a string).
 # The minimum possible {{lgk}} value (an integer).
 # The maximum possible {{lgk}} value (an integer).
 # The actual invalid {{lgk}} value (a string).

There is no meaningful reason for the 4th argument to be a string. This issue is about changing the 4th argument to an integer.
Bug: Disable lineSep option in 'from_csv' and 'schema_of_csv'
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Apr/22 21:55
Affects Version/s: 3.2.0

I don't know how critical this is. I was doing some general testing to understand {{from_csv}} and found that if I happen to have a {{lineSep}} in the input data and I noticed that the next row appears to be corrupted. {{multiLine}} does not appear to fix it. Because this is data corruption I am inclined to mark this as CRITICAL or BLOCKER, but it is an odd corner case so I m not going to set it myself.

{code}
Seq[String]("1,\n2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]())).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|   1,\n2,3,4,5|      {1, null}|
|    6,7,8,9,10|      {null, 8}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+
{code}

{code}
Seq[String]("1,:2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]("lineSep" -> ":"))).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|    1,:2,3,4,5|      {1, null}|
|    6,7,8,9,10|      {null, 8}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+
{code}

{code}
Seq[String]("1,\n2,3,4,5","6,7,8,9,10", "11,12,13,14,15", null).toDF.select(col("value"), from_csv(col("value"), StructType(Seq(StructField("a", LongType), StructField("b", StringType))), Map[String,String]("lineSep" -> ":"))).show()
+--------------+---------------+
|         value|from_csv(value)|
+--------------+---------------+
|   1,\n2,3,4,5|       {1, \n2}|
|    6,7,8,9,10|         {6, 7}|
|11,12,13,14,15|       {11, 12}|
|          null|           null|
+--------------+---------------+
{code}
Bug: When ConfigMap creation fails, Spark driver starts but fails to start executors
Status: Open | Priority: Major | Resolution: 
Created: 05/Apr/22 22:34
Affects Version/s: 3.1.1

When running Spark in Kubernetes client mode, all executors assume that a ConfigMap exactly matching `KubernetesClientUtils.configMapNameExecutor` will exist (see [https://github.com/apache/spark/blob/02a055a42de5597cd42c1c0d4470f0e769571dc3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L98])

If the ConfigMap creation fails, [https://github.com/apache/spark/blob/02a055a42de5597cd42c1c0d4470f0e769571dc3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L80], (due to the Kubernetes control plane being temporarily unavailable or the permissions of the serviceaccount being insufficient to create a ConfigMap), the driver will start fully, then will wait for executors that will forever fail to start due to "MountVolume.SetUp failed for volume \"spark-conf-volume-exec\" : configmap \"spark-exec-...-conf-map\" not found" 

 

Either the driver start-up should fail with an error, or the driver should retry the attempt to create the ConfigMap

--

To reproduce the problem when the Kubernetes control plane is not experiencing issues, start Spark in client mode, but do not give the Kubernetes ServiceAccount permission to create ConfigMap. The driver pod will start successfully, but the executor pods will terminate upon creation, and the driver will not create new executors.
Bug: Let consumers provide their own method for Authentication for The REST Submission Server
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/22 16:28
Affects Version/s: 3.4.0

[Spark documentation|https://spark.apache.org/docs/latest/security.html] states that

??The REST Submission Server and the MesosClusterDispatcher do not support authentication. You should ensure that all network access to the REST API & MesosClusterDispatcher (port 6066 and 7077 respectively by default) are restricted to hosts that are trusted to submit jobs.??

Whilst it is true that we can use network policies to restrict access to our exposed submission endpoint, it would be preferable to at least also allow some primitive form of authentication at a global level, whether this is by some token provided to the runtime environment or is a "system user" using basic authentication of a username/password combination - I am not strictly opinionated and I think either would suffice.

Alternatively, one could implement a custom proxy to provide this authentication check, but upon investigation this option is rejected by the spark master as-is today.

I would imagine that whatever solution is agreed for a first phase, a custom authenticator may be something we want a user to be able to provide so that if an admin needed some more advanced authentication check, such as RBAC et al, it could be facilitated without the need for writing a complete custom proxy layer; although it could be argued just some basic built in layer being available; eg. RestSubmissionBasicAuthenticator could be preferable.
Bug: Override S3 Client in Spark Write/Read calls
Status: Open | Priority: Major | Resolution: 
Created: 20/Apr/22 01:16
Affects Version/s: 3.2.1

Hello,

I have been working to use spark to read and write data to S3. Unfortunately, there are a few S3 headers that I need to add to my spark read/write calls. After much looking, I have not found a way to replace the S3 client that spark uses to make the read/write calls. I also have not found a configuration that allows me to pass in S3 headers. Here is an example of some common S3 request headers ([https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRequestHeaders.html).] Does there already exist functionality to add S3 headers to spark read/write calls or pass in a custom client that would pass these headers on every read/write request? Appreciate the help and feedback

 

Thanks,
Bug: Refactor `GroupBy._reduce_for_stat_function` on accepted data types 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/22 17:34
Affects Version/s: 3.4.0

`Groupby._reduce_for_stat_function` is a common helper function leveraged by multiple statistical functions of GroupBy objects.

It defines parameters `only_numeric` and `bool_as_numeric` to control accepted Spark types.

To be consistent with pandas API, we may also have to introduce `str_as_numeric` for `sum` for example.

Instead of introducing parameters designated for each Spark type, the PR is proposed to introduce a parameter `accepted_spark_types` to specify accepted types of Spark columns to be aggregated.
Bug: Better Data Source V2 operator pushdown framework
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/22 06:43
Affects Version/s: 3.3.0

Currently, Spark supports push down Filters and Aggregates to data source.
However, the Data Source V2 operator pushdown framework has the following shortcomings:

# Only simple filter and aggregate are supported, which makes it impossible to apply in most scenarios
# The incompatibility of SQL syntax makes it impossible to apply in most scenarios
# Aggregate push down does not support multiple partitions of data sources
# Spark's additional aggregate will cause some overhead
# Limit push down is not supported
# Top n push down is not supported
# Aggregate push down does not support group by expressions
# Aggregate push down does not support not use aggregate functions
# Offset push down is not supported
# Paging push down is not supported
# UDF/UDAF push down is not supported
Bug: Pandas API on spark Can`t apply lamda to columns.  
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Apr/22 16:21
Affects Version/s: 3.3.0

When I use a spark master build from 08 November 21 I can use this code to rename columns 

{code:java}
pf05 = pf05.rename(columns=lambda x: re.sub('DOFFIN_ESENDERS:', '', x))
pf05 = pf05.rename(columns=lambda x: re.sub('FORM_SECTION:', '', x))
pf05 = pf05.rename(columns=lambda x: re.sub('F05_2014:', '', x))
{code}

But now after I get this error when I use this code.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 pf05 = pf05.rename(columns=lambda x: re.sub('DOFFIN_ESENDERS:', '', x))
      2 pf05 = pf05.rename(columns=lambda x: re.sub('FORM_SECTION:', '', x))
      3 pf05 = pf05.rename(columns=lambda x: re.sub('F05_2014:', '', x))

File /opt/spark/python/pyspark/pandas/frame.py:10636, in DataFrame.rename(self, mapper, index, columns, axis, inplace, level, errors)
  10632     index_mapper_fn, index_mapper_ret_dtype, index_mapper_ret_stype = gen_mapper_fn(
  10633         index
  10634     )
  10635 if columns:
> 10636     columns_mapper_fn, _, _ = gen_mapper_fn(columns)
  10638 if not index and not columns:
  10639     raise ValueError("Either `index` or `columns` should be provided.")

File /opt/spark/python/pyspark/pandas/frame.py:10603, in DataFrame.rename.<locals>.gen_mapper_fn(mapper)
  10601 elif callable(mapper):
  10602     mapper_callable = cast(Callable, mapper)
> 10603     return_type = cast(ScalarType, infer_return_type(mapper))
  10604     dtype = return_type.dtype
  10605     spark_return_type = return_type.spark_type

File /opt/spark/python/pyspark/pandas/typedef/typehints.py:563, in infer_return_type(f)
    560 tpe = get_type_hints(f).get("return", None)
    562 if tpe is None:
--> 563     raise ValueError("A return value is required for the input function")
    565 if hasattr(tpe, "__origin__") and issubclass(tpe.__origin__, SeriesType):
    566     tpe = tpe.__args__[0]

ValueError: A return value is required for the input function
Bug: When push-based shuffle is enabled, a stage may not complete when retried
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/22 21:28
Affects Version/s: 3.2.0

With push-based shuffle enabled and adaptive merge finalization, there are scenarios where a re-attempt of ShuffleMapStage may not complete. 

With Adaptive Merge Finalization, a stage may be triggered for finalization when it is in the below state:
 # The stage is *not* running ({*}not{*} in the _running_ set of the DAGScheduler) - had failed or canceled or waiting, and
 # The stage has no pending partitions (all the tasks completed at-least once).

For such a stage when the finalization completes, the stage will still not be marked as {_}mergeFinalized{_}. 

The stage of the stage will be: 
 * _stage.shuffleDependency.mergeFinalized = false_
 * _stage.shuffleDependency.getFinalizeTask = finalizeTask_
 * Merged statuses of the state are unregistered

 

When the stage is resubmitted, the newer attempt of the stage will never complete even though its tasks may be completed. This is because the newer attempt of the stage will have {_}shuffleMergeEnabled = true{_}, since with the previous attempt the stage was never marked as {_}mergedFinalized{_}, and the _finalizeTask_ is present (from finalization attempt for previous stage attempt).

 

So, when all the tasks of the newer attempt complete, then these conditions will be true:
 * stage will be running
 * There will be no pending partitions since all the tasks completed

 * _stage.shuffleDependency.shuffleMergeEnabled = true_

 * _stage.shuffleDependency.shuffleMergeFinalized = false_

 * _stage.shuffleDependency.getFinalizeTask_ is not empty

This leads the DAGScheduler to try scheduling finalization and not trigger the completion of the Stage. However because of the last condition it never even schedules the finalization and the stage never completes.
Bug: Pivoting a dataframe after two explodes raises an org.codehaus.commons.compiler.CompileException
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 29/Apr/22 11:39
Affects Version/s: 3.2.0

The following Code raises an exception starting in spark version 3.2.0
{code:java}
import pyspark.sql.functions as sf
from pyspark.sql import Row
data = [
    Row(
        other_value=10,
        ships=[
            Row(
                fields=[
                    Row(name="field1", value=1),
                    Row(name="field2", value=2),
                ]
            ),
            Row(
                fields=[
                    Row(name="field1", value=3),
                    Row(name="field3", value=4),
                ]
            ),
        ],
    )
]

df = spark.createDataFrame(data)
df = df.withColumn("ships", sf.explode("ships")).select(
    sf.col("other_value"), sf.col("ships.*")
)
df = df.withColumn("fields", sf.explode("fields")).select("other_value", "fields.*")
df = df.groupBy("other_value").pivot("name") #ERROR THROWN HERE {code}
The "pivot("name")" is what causes the error to be thrown.

Having even one less explode (by manually transforming the input dataframe to what it would look like after the first explode) causes this error not to appear.
Adding a .cache() before the groupBy and pivot also causes this error to dissapear.

Strangely, this error does not appear when running the above code in Databricks using Databricks Runtime 10.4 which is also using spark version 3.2.1

I have attached the full stacktrace of the error below.
I will try to investigate more after work today and will add anything else I find.

The exception raised is
{noformat}
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 78: A method named "numElements" is not declared in any enclosing class nor any supertype, nor through a static import{noformat}
but seems to be caused by this exception
{noformat}
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.UnsafeArrayData cannot be cast to org.apache.spark.sql.catalyst.InternalRow{noformat}
The following stackoverflow question seems to pertain to a very similar error
[https://stackoverflow.com/questions/70356375/pyspark-pivot-unsafearraydata-cannot-be-cast-to-internalrow]
Bug: FileNotFoundException while reading from Kafka
Status: Open | Priority: Major | Resolution: 
Created: 29/Apr/22 06:29
Affects Version/s: 2.4.8

We are running a stateful structured streaming job which reads from Kafka and writes to HDFS. And we are hitting this exception:

 

 

17/12/08 05:20:12 ERROR FileFormatWriter: Aborting job null. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, hcube1-1n03.eng.hortonworks.com, executor 1): java.lang.IllegalStateException: Error reading delta file /checkpointDir/state/0/0/1.delta of HDFSStateStoreProvider[id = (op=0, part=0), dir = /checkpointDir/state/0/0]: /checkpointDir/state/0/0/1.delta does not exist at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$updateFromDeltaFile(HDFSBackedStateStoreProvider.scala:410) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1$$anonfun$6.apply(HDFSBackedStateStoreProvider.scala:362) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1$$anonfun$6.apply(HDFSBackedStateStoreProvider.scala:359) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1.apply(HDFSBackedStateStoreProvider.scala:359) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1.apply(HDFSBackedStateStoreProvider.scala:358) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider.org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap(HDFSBackedStateStoreProvider.scala:358) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1$$anonfun$6.apply(HDFSBackedStateStoreProvider.scala:360) at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$$anonfun$org$apache$spark$sql$execution$streaming$state$HDFSBackedStateStoreProvider$$loadMap$1$$anonfun$6.apply(HDFSBackedStateStoreProvider.scala:359) at scala.Option.getOrElse(Option.scala:121

 

Of course, the file doesn't exist in HDFS. And in the {{state/0/0}} directory there is no file at all. While we have some files in the commits and offsets folders. I am not sure about the reason of this behavior. It seems to happen on the second time the job is started,

 

 If any more information is needed then I would be happy to provide it. Would also appreciate on some input on how to best resolve this issue?
Bug: Add tests for options from `to_json` and `from_json`.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/22 04:31
Affects Version/s: 3.4.0

There are many supported options for to_json and from_json (https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option), but they are currently not tested.

We should test for options to `sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala`.