av9ash/GitBugs huggingface dataset:
Bug: Spark driver logs are filled with Initializing service data for shuffle service using name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/23 16:53
Affects Version/s: 3.2.0

With SPARK-34828, we added the ability to make the shuffle service name configurable and we added a log [here|https://github.com/apache/spark/blob/8860f69455e5a722626194c4797b4b42cccd4510/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala#L118] that will log the shuffle service name. However, this log is printed in the driver logs whenever there is new executor launched and pollutes the log. 
{code}
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
22/08/03 20:42:07 INFO ExecutorRunnable: Initializing service data for shuffle service using name 'spark_shuffle_311'
{code}
We can just log this once in the driver.
Bug: Create golden file test framework for analysis
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 18:43
Affects Version/s: 3.4.0

Here we track the work to add new golden file test support for the Spark analyzer. Each golden file can contain a list of SQL queries followed by the string representations of their analyzed logical plans.
 
This can be similar to Spark's existing `SQLQueryTestSuite` [1], but stopping after analysis and listing analyzed plans as the results instead of fully executing queries end-to-end. As another example, ZetaSQL has analyzer-based golden file testing like this as well [2].
 
This way, any changes to analysis will show up as test diffs, which are easy to spot in review and also easy to update automatically. This could help the community together maintain the qualify of Apache Spark's query analysis.
 
[1] [https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala] 
[2] [https://github.com/google/zetasql/blob/master/zetasql/analyzer/testdata/limit.test].
Bug: Spark SQL reads unnecessary nested fields
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/23 13:55
Affects Version/s: 3.3.2

When we use high order functions in spark sql query, it would be great if it will be somehow possible to write following example in way that spark will read only necessary nested fields.

Example:
1) Loading data
{code:scala}
val jsonStr = """{
 "items": [
   {"itemId": 1, "itemData": "a"},
   {"itemId": 2, "itemData": "b"}
 ]
}"""
val df = spark.read.json(Seq(jsonStr).toDS)
df.write.format("parquet").mode("overwrite").saveAsTable("persisted")
{code}
2) read query with explain
{code:scala}
val read = spark.table("persisted")
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", true)

read.select(transform($"items", i=>i.getItem("itemId")).as('itemIds)).explain(true)
// ReadSchema: struct<items:array<struct<itemData:string,itemId:bigint>>>
{code}
We use only *itemId* field from structure in array, but read schema contains all fields of structure.
Bug: Unable to CreateDataFrame from Decimal128
Status: Open | Priority: Major | Resolution: 
Created: 20/Mar/23 02:28
Affects Version/s: 3.4.0

To reproduce: 

{code:sh}
>>> from decimal import Decimal
>>> spark.createDataFrame([Decimal(1.123)])
Traceback (most recent call last):
...
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
>>> 
{code}

The version of PyArrow is 10.0.1; the issue doesn't exist in vanilla PySpark with the same PyArrow version.
Bug: Pyspark.pandas.series.str.findall can't handle tuples that are returned by regex
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/23 18:05
Affects Version/s: 3.3.2

When you use the str.findall accessor method on a ps.series and you're passing a regex pattern that will return match groups, it will return a pyarrow data error.

In pandas the result is this:
{code:java}
df.to_pandas()[col].str.findall(regex_pattern, flags=re.IGNORECASE)

returns 

 [("value", , , , )],
 [("value", , , , )],
 [(, , ,"value", )]{code}
 

In pyspark.pandas the result is:
{code:java}
org.apache.spark.api.python.PythonException: 'pyarrow.lib.ArrowTypeError: Expected bytes, got a 'tuple' object'.{code}
 

My temporary workaround is using 
{code:java}
df.apply(lambda x: re.findall(regex_pattern, x, flags=re.IGNORECASE)[0]{code}
Bug: Ambiguous reference for case-insensitive grouping column in groupby. applyInPandas
Status: Open | Priority: Major | Resolution: 
Created: 16/Mar/23 05:46
Affects Version/s: 3.4.0

Ambiguous reference for case-insensitive grouping column, as shown below:


{code:sh}
>>> df = spark.createDataFrame([[1, 1]], ["column", "score"])
>>> def my_pandas_udf(pdf):
...   return pdf.assign(score=0.5)
... 
>>> df.groupby("COLUMN").applyInPandas(my_pandas_udf, schema="column integer, score float").first()
23/03/16 10:18:23 ERROR SparkConnectService: Error during: execute
org.apache.spark.sql.AnalysisException: [AMBIGUOUS_REFERENCE] Reference `column` is ambiguous, could be: [`column`, `column`].
{code}

Relevant change in PySpark legacy API: https://github.com/apache/spark/pull/28777/files#diff-32b043dfe6b906fb2b240e8557f98b03a648ba792ce58d11b631744b34bcea71
Bug: Allow V2 writes to indicate advisory partition size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 07:08
Affects Version/s: 3.5.0

Data sources may request a particular distribution and ordering of data for V2 writes. If AQE is enabled, the default session advisory partition size (64MB) will be used as guidance. Unfortunately, this default value can still lead to small files because the written data can be compressed nicely using columnar file formats. Spark should allow data sources to indicate the advisory shuffle partition size, just like it lets data sources request a particular number of partitions.
Bug: Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
Status: Closed | Priority: Major | Resolution: Won't Do
Created: 17/Mar/23 09:44
Affects Version/s: 3.2.0

{color:#222222}Sometimes when run a SQL job with push based shuffle, exception occurs as below.  It seems that there’s no element in the bitmaps which stores merge chunk meta. {color}


{color:#222222}Is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?{color}
 
{code:java}
Caused by: java.lang.ArithmeticException: / by zero
at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84) {code}
related code:
{code:java}
def createChunkBlockInfosFromMetaResponse(
    shuffleId: Int,
    shuffleMergeId: Int,
    reduceId: Int,
    blockSize: Long,
    bitmaps: Array[RoaringBitmap]): ArrayBuffer[(BlockId, Long, Int)] = {
  val approxChunkSize = blockSize / bitmaps.length
  val blocksToFetch = new ArrayBuffer[(BlockId, Long, Int)]()
  for (i <- bitmaps.indices) {
    val blockChunkId = ShuffleBlockChunkId(shuffleId, shuffleMergeId, reduceId, i)
    chunksMetaMap.put(blockChunkId, bitmaps(i))
    logDebug(s"adding block chunk $blockChunkId of size $approxChunkSize")
    blocksToFetch += ((blockChunkId, approxChunkSize, SHUFFLE_PUSH_MAP_ID))
  }
  blocksToFetch
} {code}
Bug: Implement Dataset.semanticHash
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 19:33
Affects Version/s: 3.4.0

Implement Dataset.semanticHash:
{code:java}
/**
* Returns a `hashCode` of the logical query plan against this [[Dataset]].
*
* @note Unlike the standard `hashCode`, the hash is calculated against the query plan
* simplified by tolerating the cosmetic differences such as attribute names.
* @since 3.4.0
*/
@DeveloperApi
def semanticHash(): Int{code}
This has to be computed on the spark connect server to do this. Please extend the 

AnalyzePlanRequest and AnalyzePlanResponse messages for this.

Also make sure this works in PySpark.
Bug: Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/23 01:00
Affects Version/s: 3.4.0

Add support for setting max_write_buffer_number and write_buffer_size for RocksDB used in streaming

 

We need these settings in order to control memory tuning for RocksDB. We already expose settings for blockCache size. However, these 2 settings are missing. This change proposes to add them.
Bug: when target table format is textfile using `insert into select` will got error
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 15/Mar/23 09:46
Affects Version/s: 3.2.3

*create* *table* test.tex_t1(name string, address string) *ROW* FORMAT DELIMITED FIELDS TERMINATED *BY* ',' STORED *AS* TEXTFILE;


*insert* *into* test.tex_t1 *select* 'a', 'b';

will got alot of message about :
WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (24 of 24) after 5s. fireListenerEvent
org.apache.thrift.transport.TTransportException
 
But the data was actual write to table.
Bug: df.write.format().save() should support calling with no path or table name
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/23 13:35
Affects Version/s: 3.4.0

When calling `session.range(5).write.format("xxx").options().save()` Spark Connect currently throws an assertion error because it expects that either path or tableName are present. According to our current PySpark implementation that is not necessary though.

       
{code:python}
if format is not None:
            self.format(format)
        if path is None:
            self._jwrite.save()
        else:
            self._jwrite.save(path)
{code}
Bug: Customized K8s Scheduler GA
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/23 06:44
Affects Version/s: 3.4.0

Since Apache Spark 3.3.1, we have been supported the customized K8s scheduler via SPARK-36057. This issue aims to announce GA of the customized K8s schedulers in Apache Spark 3.4.0.
Bug: Add support to track FLUSH_WRITE_BYTES for RocksDB state store for streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 21:36
Affects Version/s: 3.4.0

Add support to track FLUSH_WRITE_BYTES for RocksDB state store for streaming

 

Its useful to get this metric for bytes written during flush from RocksDB as part of the DB custom metrics. We propose to add this to the existing metrics that are collected. There is no additional overhead since we are just querying the internal ticker guage, similar to other metrics.
Bug: Spark Connect - Grammatical improvements to Spark Overview and Spark Connect Overview doc pages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/23 00:45
Affects Version/s: 3.4.0

Grammatical improvements, this is a follow-up to this ticket:

Introducing Spark Connect on the main page and adding Spark Connect Overview page
https://issues.apache.org/jira/browse/SPARK-42496
Bug: createDataFrame doesn't work with non-nullable schema.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/23 07:00
Affects Version/s: 3.4.0

spark.createDataFrame won't work with non-nullable schema as below:
{code:java}
from pyspark.sql.types import *
schema_false = StructType([StructField("id", IntegerType(), False)])
spark.createDataFrame([[1]], schema=schema_false)

Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.AnalysisException: [NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's required to be non-nullable.{code}
whereas it works fine with nullable schema:
{code:java}
schema_true = StructType([StructField("id", IntegerType(), True)])
spark.createDataFrame([[1]], schema=schema_true)

DataFrame[id: int]{code}
Bug: Fix `createDataFrame` to work properly with rows and schema
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 03/Mar/23 20:40
Affects Version/s: 3.5.0

The code below is not working properly in Spark Connect:
{code:java}
>>> sdf = spark.range(10)
>>> spark.createDataFrame(sdf.tail(5), sdf.schema) 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 94, in __repr__
    return "DataFrame[%s]" % (", ".join("%s: %s" % c for c in self.dtypes))
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 162, in dtypes
    return [(str(f.name), f.dataType.simpleString()) for f in self.schema.fields]
  File "/.../spark/python/pyspark/sql/connect/dataframe.py", line 1346, in schema
    self._schema = self._session.client.schema(query)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 614, in schema
    proto_schema = self._analyze(method="schema", plan=plan).schema
  File "/.../spark/python/pyspark/sql/connect/client.py", line 755, in _analyze
    self._handle_error(rpc_error)
  File "/.../spark/python/pyspark/sql/connect/client.py", line 894, in _handle_error
    raise convert_exception(info, status.message) from None
pyspark.errors.exceptions.connect.AnalysisException: [NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's required to be non-nullable.{code}
whereas working properly in regular PySpark:
{code:java}
>>> sdf = spark.range(10)
>>> spark.createDataFrame(sdf.tail(5), sdf.schema).show()
+---+
| id|
+---+
|  5|
|  6|
|  7|
|  8|
|  9|
+---+ {code}
Bug: Port the tests for get_json_object from the Apache Hive project
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 08:24
Affects Version/s: 3.5.0

https://github.com/apache/hive/blob/ba0217ff17501fb849d8999e808d37579db7b4f1/ql/src/test/org/apache/hadoop/hive/ql/udf/TestUDFJson.java
Bug: [K8S][Core] When spark submit without --deploy-mode, will face NPE in Kubernetes Case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/23 09:49
Affects Version/s: 3.2.4

According to this PR [https://github.com/apache/spark/pull/37880#issuecomment-1347777890,] when user spark submit without `--deploy-mode XXX` or `–conf spark.submit.deployMode=XXXX`, may face NPE with this code

 
args.deployMode.equals("client")
Bug: Improved AliasAwareOutputExpression works with DSv2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/23 09:16
Affects Version/s: 3.4.0

After SPARK-40086 / SPARK-42049 the following, simple subselect expression containing query:
{noformat}
select (select sum(id) from t1)
{noformat}
fails with:

{noformat}
09:48:57.645 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.NullPointerException
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.batch$lzycompute(BatchScanExec.scala:47)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.batch(BatchScanExec.scala:47)
	at org.apache.spark.sql.execution.datasources.v2.BatchScanExec.hashCode(BatchScanExec.scala:60)
	at scala.runtime.Statics.anyHash(Statics.java:122)
        ...
	at org.apache.spark.sql.catalyst.trees.TreeNode.hashCode(TreeNode.scala:249)
	at scala.runtime.Statics.anyHash(Statics.java:122)
	at scala.collection.mutable.HashTable$HashUtils.elemHashCode(HashTable.scala:416)
	at scala.collection.mutable.HashTable$HashUtils.elemHashCode$(HashTable.scala:416)
	at scala.collection.mutable.HashMap.elemHashCode(HashMap.scala:44)
	at scala.collection.mutable.HashTable.addEntry(HashTable.scala:149)
	at scala.collection.mutable.HashTable.addEntry$(HashTable.scala:148)
	at scala.collection.mutable.HashMap.addEntry(HashMap.scala:44)
	at scala.collection.mutable.HashTable.init(HashTable.scala:110)
	at scala.collection.mutable.HashTable.init$(HashTable.scala:89)
	at scala.collection.mutable.HashMap.init(HashMap.scala:44)
	at scala.collection.mutable.HashMap.readObject(HashMap.scala:195)
        ...
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:87)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:129)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:85)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1520)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
{noformat}
when DSv2 is enabled.
Bug: CAST(x as int) does not generate error with overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/23 16:27
Affects Version/s: 3.2.1

Hi,

When performing the following code:

{{select cast(7.415246799222789E19 as int)}}

according to the documentation, an error is expected as {{7.415246799222789E19 }}is an overflow value for datatype INT.

However, the value 2147483647 is returned. 

The behaviour of the following is correct as it returns NULL:

{{select try_cast(7.415246799222789E19 as int) }}

This results in unexpected behaviour and data corruption.
Bug: YarnAllocator should filter excluded nodes when launching allocated containers
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Mar/23 03:38
Affects Version/s: 3.3.2

In production environment, we hit an issue like this:

If we request 10 containers form nodeA and nodeB, first response from Yarn return 5 contianers from nodeA and nodeB, then nodeA blacklisted, and second response from Yarn maybe return some containers from nodeA and launching containers, but when containers(Executor) setup and send register request to Driver, it will be rejected and this failure will be counted to 
{code:java}
spark.yarn.max.executor.failures {code}
, and will casue app failed.
{code:java}
Max number of executor failures ($maxNumExecutorFailures) reached{code}
Bug: ReusedExchange refers to non-existent node
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Mar/23 22:13
Affects Version/s: 3.4.0

There is an AQE “issue“ where during AQE planning, the Exchange "that's being" reused could be replaced in the plan tree. So, when we print the query plan, the ReusedExchange will refer to an “unknown“ Exchange. An example below:

 
{code:java}
(2775) ReusedExchange [Reuses operator id: unknown]
 Output [3]: [sr_customer_sk#271, sr_store_sk#275, sum#377L]{code}
 

 

Below is an example to demonstrate the root cause:

 
{code:java}
AdaptiveSparkPlan
  |-- SomeNode X (subquery xxx)
      |-- Exchange A
          |-- SomeNode Y
              |-- Exchange B
Subquery:Hosting operator = SomeNode Hosting Expression = xxx dynamicpruning#388
AdaptiveSparkPlan
  |-- SomeNode M
      |-- Exchange C
          |-- SomeNode N
              |-- Exchange D
{code}
 

 

Step 1: Exchange B is materialized and the QueryStage is added to stage cache

Step 2: Exchange D reuses Exchange B

Step 3: Exchange C is materialized and the QueryStage is added to stage cache

Step 4: Exchange A reuses Exchange C

 

Then the final plan looks like:

 
{code:java}
AdaptiveSparkPlan
  |-- SomeNode X (subquery xxx)
      |-- Exchange A -> ReusedExchange (reuses Exchange C)

Subquery:Hosting operator = SomeNode Hosting Expression = xxx dynamicpruning#388
AdaptiveSparkPlan
  |-- SomeNode M
      |-- Exchange C -> PhotonShuffleMapStage ....
          |-- SomeNode N
              |-- Exchange D -> ReusedExchange (reuses Exchange B)
{code}
 

 

As a result, the ReusedExchange (reuses Exchange B) will refer to a non-exist node. This *DOES NOT* affect query execution but will cause the query visualization malfunction in the following ways:
 # The ReusedExchange child subtree will still appear in the Spark UI graph but will contain no node IDs.
 # The ReusedExchange node details in the Explain plan will refer to a UNKNOWN node. Example below.

{code:java}
(2775) ReusedExchange [Reuses operator id: unknown]{code}
 # The child exchange and its subtree may be missing from the Explain text completely. No node details or tree string shown.
Bug: Sparksql temporary file conflict
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/23 07:17
Affects Version/s: 3.3.2

When sparksql inserts overwrite, the name of the temporary file in the middle is not unique. This will cause that when multiple applications write different partition data to the same partition table, it will be possible to delete each other's temporary files between applications, resulting in task failure
Bug: delete uploaded file when job finish
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/23 08:07
Affects Version/s: 3.1.2

On kubernetes, spark use spark.kubernetes.file.upload.path to upload local files to 

Hadoop compatible file system. But spark do not delete those files at all.

In this issue, we add a configuration spark.kubernetes.uploaded.file.delete.on.termination to delete those files by driver when job finishs.