av9ash/GitBugs huggingface dataset:
Bug: Add dataSize field in CoalescedPartitionSpec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 03:56
Affects Version/s: 3.2.0

Currently, all test suite about `CoalescedPartitionSpec` do not check the data size due to it doesn't contains data size field.

We can add data size in `CoalescedPartitionSpec` and then add test case for better coverage.
Bug: [Deploy] Upgrade Apache Curator Dependency to 4.2.0
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Jun/21 15:25
Affects Version/s: 3.1.2

+Abstract :+ as a Spark Cluster Administrator, I want to connect spark masters deployed in HA/mode to Zookeeper over SSL/TLS, so that my network traffic is ciphered between y components. ([https://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper]  )

 

With the release of Hadoop 3.3.1, ZKFC libraries and their dependencies were updated : it is now possible to connect ZKFC to ZooKeeper over TLS.

 

+Note:+ TLS is possible with Zookeeper Server Version >= 3.5.6 ([https://docs.confluent.io/platform/current/installation/versions-interoperability.html#zk] )

 

Spark 3.2.0 aims to support Hadoop 3.3.1 ; this Hadoop release bundles the following shared libraries :
 * curator-client-4.2.0.jar
 * curator-framework-4.2.0.jar
 * curator-recipes-4.2.0.jar
 * zookeeper-3.5.6.jar
 * zookeeper-jute-3.5.6.jar

 

Currently, Spark dependency is set to 2.13.0 for the Currator framework ([https://github.com/apache/spark/blob/master/pom.xml#L127).|https://github.com/apache/spark/blob/master/pom.xml#L127)]

 

It would be great to update "curator-*" dependencies to 4.2.0 in order to be compatible with shared jars of the hadoop stack.

Moreover, it will allow administrators to connect Spark Masters to ZooKeeper over TLS.

 

Some patches will be required, such as :
 * [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkCuratorUtil.scala#L51]

 

I will try to prepare a MR for this.
Bug: Inclusive terminology usage in Spark
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 30/Jun/21 10:16
Affects Version/s: 3.0.3

Terms such as Blacklist/Whitelist and master/slave is used at different places in Spark Code. Do we have any plans to modify this to more inclusive terminology, for eg: Denylist/Allowlist and Leader/Follower?
Bug: [SS] Include more granular metrics for stateful operators in StreamingQueryProgress
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 14:30
Affects Version/s: 3.1.2

Currently the streaming progress is missing a few important stateful operator metrics in {{StateOperatorProgress}}. Each stateful operator consists of multiple steps. Ex: {{flatMapGroupsWithState}} has two major steps: 1) processing the input and 2) timeout processing to remove entries from the state which have expired. The main motivation is to track down the time it took for each individual step (such as timeout processing, watermark processing etc) and how much data is processed to pinpoint the bottlenecks and compare for reasoning why some microbatches are slow compared to others in the same job.

Below are the final metrics common to all stateful operators (the one in _*bold-italic*_ are proposed new). These metrics are in {{StateOperatorProgress}} which is part of {{StreamingQueryProgress}}.
 * _*operatorName*_ - State operator name. Can help us identify any operator specific slowness and state store usage patterns. Ex. "dedupe" (derived using {{StateStoreWriter.shortName}})
 * _numRowsTotal_ - number of rows in the state store across all tasks in a stage where the operator has executed.
 * _numRowsUpdated_ - number of rows added to or update in the store
 * _*allUpdatesTimeMs*_ - time taken to add new rows or update existing state store rows across all tasks in a stage where the operator has executed.
 * _*numRowsRemoved*_ - number of rows deleted from state store as part of the state cleanup mechanism across all tasks in a stage where the operator has executed. This number helps measure the state store deletions and impact on checkpoint commit and other latencies.
 * _*allRemovalsTimeMs*_ - time taken to remove the rows from the state store as part of state (also includes the iterating through the entire state store to find which rows to delete) across all tasks in a stage where the operator has executed. If we see jobs spending significant time here, it may justify a better layout in the state store to read only the required rows than the entire state store that is read currently.
 * _*commitTimeMs*_ - time taken to commit the state store changes to external storage for checkpointing. This is cumulative across all tasks in a stage where this operator has executed.
 * _*numShufflePartitions*_ - number of shuffle partitions this state operator is part of. Currently the metrics like times are aggregated across all tasks in a stage where the operator has executed. Having the number shuffle partitions (corresponds to number of tasks) helps us find the average task contribution to the metric.
 * _*numStateStores*_ - number of state stores in the operator across all tasks in the stage. Some stateful operators have more than one state store (eg. stream-stream join). Tracking this number helps us find correlations between state stores instances and microbatch latency.
Bug: Ability to override Yarn Cluster Submit Class with Configuration
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Jun/21 14:20
Affects Version/s: 3.1.2

Currently, YarnClusterApplication class is used to initiate Yarn Client and submit application to Yarn cluster. In case of a custom variant of Yarn cluster, there may be significant (non-generic) changes required in Yarn client. User should be able to override YarnClusterApplication via config.  Using this configuration, they can add custom implementation of YarnClusterApplication (as a separate jar) which can invoke a custom Yarn client.
Bug: Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 04:46
Affects Version/s: 3.2.0

Since [SPARK-35447](https://issues.apache.org/jira/browse/SPARK-35447), we apply `OptimizeSkewedJoin` before `CoalesceShufflePartitions`. However, There are something different with the order of these two rules.

Let's say if we have a skewed partitions: [0, 128MB, 0, 128MB, 0]:
 # coalesce partitions first then optimize skewed partitions:
 [64MB, 64MB, 64MB, 64MB]

 # optimize skew partition first then coalesce partitions:
 [0, 64MB, 64MB, 0, 64MB, 64MB, 0]

So we can do coalesce in ShufflePartitionsUtil.coalescePartitionsWithSkew with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec if CoalescedPartitionSpec is empty.
Bug: Consider whether we need to broadcast dynamic filtering subquery results for v2 tables
Status: Open | Priority: Major | Resolution: 
Created: 25/Jun/21 18:16
Affects Version/s: 3.2.0

We don't necessarily have to broadcast the results of dynamic filtering subqueries for v2 tables.

See [here|https://github.com/apache/spark/pull/32921#discussion_r653212341].
Bug: Add a utility to convert connector expressions to Catalyst expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 17:40
Affects Version/s: 3.2.0

There are more and more places that require converting a v2 connector expression to an internal Catalyst expression. We need to build a utility method to avoid having the same logic in a lot of places.

See [here|https://github.com/apache/spark/pull/32921#discussion_r653129793].
Bug: Update remoteBlockBytes based on merged block info
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jun/21 16:41
Affects Version/s: 3.2.0

Currently, we calculate the `remoteBlockBytes` based on the original block info list. If the original reducer size is big but the actual reducer size is small due to automatically partition coalescing of AQE, the reducer will take more time to calculate `remoteBlockBytes`. We can reduce this cost via remote requests which contain merged block info lists.
Bug: Converting arrays with RowToColumnConverter triggers assertion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 15:45
Affects Version/s: 3.1.2

When trying to convert a row that contains an array to a ColumnVector with RowToColumnConverter the following error is thrown:
{code:java}
java.lang.AssertionError at org.apache.spark.sql.execution.vectorized.OffHeapColumnVector.putArray(OffHeapColumnVector.java:560) at org.apache.spark.sql.execution.vectorized.WritableColumnVector.appendArray(WritableColumnVector.java:622) at org.apache.spark.sql.execution.RowToColumnConverter$ArrayConverter.append(Columnar.scala:353) at org.apache.spark.sql.execution.RowToColumnConverter$BasicNullableTypeConverter.append(Columnar.scala:241) at org.apache.spark.sql.execution.RowToColumnConverter.convert(Columnar.scala:221)
{code}
Bug: Codegen issue for decimal type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 02:25
Affects Version/s: 3.0.3

How to reproduce this issue:
{code:scala}
spark.sql(
  """
    |CREATE TABLE t1 (
    |  c1 DECIMAL(18,6),
    |  c2 DECIMAL(18,6),
    |  c3 DECIMAL(18,6))
    |USING parquet;
    |""".stripMargin)

spark.sql("SELECT sum(c1 * c3) + sum(c2 * c3) FROM t1").show
{code}


{noformat}
20:23:36.272 ERROR org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 6: Expression "agg_exprIsNull_2_0" is not an rvalue
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56, Column 6: Expression "agg_exprIsNull_2_0" is not an rvalue
	at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:12675)
	at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:7676)
{noformat}
Bug: Introduce new style enforce to not import scala.collection.Seq/IndexedSeq
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 12:05
Affects Version/s: 3.2.0

Due to the changes on Scala 2.13, importing scala.collection.Seq or scala.collection.IndexedSeq would bring weird issue.
(It doesn't bring issue on Scala 2.12 so it's high likely we couldn't indicate the problem till we see complication failure in Scala 2.13.)

Please refer below page to see the details of changes around Seq.
https://docs.scala-lang.org/overviews/core/collections-migration-213.html

It would be nice if we can prevent the case.
Bug: numPartitions does not work when saves the RDD to database
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Jun/21 07:53
Affects Version/s: 3.1.1

When use SQL to insert data in spark to database, suppose the original RDD's partition num is 10, and when i set the numPartitions to 20 in SQL(because i need more parallelism to insert data to database), but it does not work.
Bug: Check multiply/divide of day-time intervals of any fields by numeric
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 09:33
Affects Version/s: 3.2.0

Write tests that checks multiply/divide of the following intervals by numeric:
# INTERVAL DAY
# INTERVAL DAY TO HOUR
# INTERVAL DAY TO MINUTE
# INTERVAL HOUR
# INTERVAL HOUR TO MINUTE
# INTERVAL HOUR TO SECOND
# INTERVAL MINUTE
# INTERVAL MINUTE TO SECOND
# INTERVAL SECOND
Bug: Query performance degradation additional predicate and UDF call for explode
Status: Open | Priority: Major | Resolution: 
Created: 24/Jun/21 17:49
Affects Version/s: 3.1.1

This issue cannot be seen in 3.0.1, but has been introduced since and observed in 3.1.1 and 3.1.2. I have a reproduce for this issue here: [https://github.com/johnbateman/spark-udf-slowdown] just change the sbt file between 3.1.2 and 3.0.1 to observe the difference in performance. It is a rather silly example but it demonstrates the issue.

Physical plan for 3.0.1, it executes on my machine in about 40 seconds.

 
{code:java}
== Physical Plan == Generate explode(fib#3), [id#1L, fib#3], false, [fib2#7] 
+- *(1) Project [id#1L, UDF(cast(id#1L as int)) AS fib#3] 
  +- *(1) Range (1, 500000, step=1, splits=8)
{code}
 

Physical plan for 3.1.2, it executes on my machine in about 4.7 min.

 
{code:java}
== Physical Plan ==
Generate (4)
+- * Project (3)
   +- * Filter (2)
      +- * Range (1)

(1) Range [codegen id : 1]
Output [1]: [id#2L]
Arguments: Range (1, 500000, step=1, splits=Some(8))

(2) Filter [codegen id : 1]
Input [1]: [id#2L]
Condition : ((size(UDF(cast(id#2L as int)), true) > 0) AND isnotnull(UDF(cast(id#2L as int))))

(3) Project [codegen id : 1]
Output [2]: [id#2L, UDF(cast(id#2L as int)) AS fib#4]
Input [1]: [id#2L]

(4) Generate
Input [2]: [id#2L, fib#4]
Arguments: explode(fib#4), [id#2L, fib#4], false, [fib2#11]{code}
 

You can see that there is an additional predicate generated in step 2, I can also confirm that the UDF is now called multiple times instead of once. I am aware that this is to be expected sometimes, but it is a change that has resulted in performance degradation particularly for expensive UDFs. Obviously, there is something specific to this query (ie the explode) that seems to be responsible for this predicate and UDF issue occurring, but I am not sure what that is.

For reference, this is the same issue (https://issues.apache.org/jira/browse/SPARK-35787)
Bug: Queries against wide Avro tables can be slow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 20:41
Affects Version/s: 3.1.1

A query against an Avro table can be quite slow when all are true:
- There are many columns in the Avro file
- The query contains a wide projection
- There are many splits in the input
- Some of the splits are read serially (e.g., less executors than there are tasks)

A write to an Avro table can be quite slow when all are true:
- There are many columns in the new rows
- The operation is creating many files

For example, a single-threaded query against a 6000 column Avro data set with 50K rows and 20 files takes less than a minute with Spark 3.0.1 but over 7 minutes with Spark 3.2.0-SNAPSHOT.

The culprit appears to be this line of code:
https://github.com/apache/spark/blob/3fb044e043a2feab01d79b30c25b93d4fd166b12/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala#L226

For each split, AvroDeserializer will call this function once for each column in the projection, resulting in a potential n^2 lookup per split.

For each file, AvroSerializer will call this function once for each column, resulting in an n^2 lookup per file.
Bug: Casting string to decimal type doesn't work if the sum of the digits is greater than 38
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 09:11
Affects Version/s: 3.1.1

Since Spark 3.1.1, NULL is returned when casting a string with many decimal places to a decimal type. If the sum of the digits before and after the decimal point is less than 39, a value is returned. From 39 digits, however, NULL is returned.
This worked until Spark 3.0.X.

Code to reproduce:

* A string with 2 decimal places in front of the decimal point and 37 decimal places after the decimal point returns null

{code:python}
data = ['28.9259999999999983799625624669715762138']
dfs = spark.createDataFrame(data, StringType())
dfd = dfs.withColumn('value', col('value').cast('decimal(10, 5)'))
dfd.show(truncate=False)
{code}

+-----+
|value|
+-----+
|null |
+-----+
 
* A string with 2 decimal places in front of the decimal point and 36 decimal places after the decimal point returns the number as decimal

{code:python}
data = ['28.925999999999998379962562466971576213']
dfs = spark.createDataFrame(data, StringType())
dfd = dfs.withColumn('value', col('value').cast('decimal(10, 5)'))
dfd.show(truncate=False)
{code}

+--------+
|value   |
+--------+
|28.92600|
+--------+

* A string with 1 decimal place in front of the decimal point and 37 decimal places after the decimal point returns the number as decimal

{code:python}
data = ['2.9259999999999983799625624669715762138']
dfs = spark.createDataFrame(data, StringType())
dfd = dfs.withColumn('value', col('value').cast('decimal(10, 5)'))
dfd.show(truncate=False)
{code}

+-------+
|value  |
+-------+
|2.92600|
+-------+
Bug: Parse any day-time interval types in SQL
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 10:05
Affects Version/s: 3.2.0

Extend the parser rules to be able to parse the following types:
# INTERVAL DAY
# INTERVAL DAY TO HOUR
# INTERVAL DAY TO MINUTE
# INTERVAL HOUR
# INTERVAL HOUR TO MINUTE
# INTERVAL HOUR TO SECOND
# INTERVAL MINUTE
# INTERVAL MINUTE TO SECOND
# INTERVAL SECOND
Bug: Check all day-time interval types in UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 09:45
Affects Version/s: 3.2.0

Check all day-time interval types in UDF:
# INTERVAL DAY
# INTERVAL DAY TO HOUR
# INTERVAL DAY TO MINUTE
# INTERVAL HOUR
# INTERVAL HOUR TO MINUTE
# INTERVAL HOUR TO SECOND
# INTERVAL MINUTE
# INTERVAL MINUTE TO SECOND
# INTERVAL SECOND
Bug: Improve the implementation for DateType +/- DayTimeIntervalType(DAY)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/21 03:41
Affects Version/s: 3.2.0

At now, `DateType +/- DayTimeIntervalType()` will convert the DateType to TimestampType, then TimeAdd. When interval type is  DayTimeIntervalType(DAY), it can use DateAdd instead of TimeAdd.
Bug: Set the list of read columns in the task configuration to reduce reading of ORC data.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/21 05:11
Affects Version/s: 3.0.3

Now, the ORC reader will read all columns of the ORC table when the task configuration does not set the read column list. Therefore, we should set the list of read columns in the task configuration to reduce reading of ORC data.
Bug: QueryExecutionListener does not see any observed metrics fired before persist/cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 08:01
Affects Version/s: 3.2.0

This example properly fires the event
{code}
spark.range(100)
  .observe(
    name = "other_event",
    avg($"id").cast("int").as("avg_val"))
  .collect()
{code}

But when I add persist, then no event is fired or seen (not sure which):
{code}
spark.range(100)
  .observe(
    name = "my_event",
    avg($"id").cast("int").as("avg_val"))
  .persist()
  .collect()
{code}

The listener:
{code}
    val metricMaps = ArrayBuffer.empty[Map[String, Row]]
    val listener = new QueryExecutionListener {
      override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
        metricMaps += qe.observedMetrics
      }

      override def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit = {
        // No-op
      }
    }
    spark.listenerManager.register(listener)
{code}
Bug: Remove await (syncMode) in ChunkFetchRequestHandler
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/21 17:40
Affects Version/s: 2.4.8

SPARK-24355 introduces syncMode to mitigate the issue of sasl timeout by throting the max number of threads for sending responses of chunk fetch requests. But it causes severe performance degradation because the throughput of handling chunk fetch requests is reduced. SPARK-30623 makes the async and sync mode configurable and makes the async mode the default. 

SPARK-30512 uses a dedicated boss event loop to mitigate the sasl timeout issue and we rarely see sasl timeout issues with async mode in our production clusters today. 

Few days ago we accidentally turned on sync mode on one cluster and we observed severe shuffle performance degradation. As a result, We benchmarked the performance comparison between async and sync mode and *we suggest removing sync mode in the code base* as it seems not to provide any benefits today. We would like to share the benchmark result and hear the opinion from the community.

 

benchmark on job's run time (sync mode is 2x - 3x slower):
 YARN cluster setup: 6 nodes, 18 executors, each executor has 1 core and 3 GB memory, each node manager has 1GB heap size.

shuffle stages: 5GB shuffle data (400M key-value records), 1000 map tasks and 1000 reduce tasks.

results: shuffle read 5GB data, async mode takes 2-3 mins and sync mode takes 6 mins.

 

benchmark on metrics of external shuffle service:
 YARN cluster setup: 4 nodes in total. I set 2 nodes as async mode and 2 nodes as sync mode, shuffling 2.5 GB data.

results: in openblockreuqestslatencymillis_ratemean and some other metrics, the nodes in sync mode are 3x - 4x higher than nodes in async mode. I attached some screenshots of the metrics.
Bug: Check all day-time interval types in HiveInspectors tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 09:56
Affects Version/s: 3.2.0

Check all day-time interval types are supported by HiveInspectors:
# INTERVAL DAY
# INTERVAL DAY TO HOUR
# INTERVAL DAY TO MINUTE
# INTERVAL HOUR
# INTERVAL HOUR TO MINUTE
# INTERVAL HOUR TO SECOND
# INTERVAL MINUTE
# INTERVAL MINUTE TO SECOND
# INTERVAL SECOND
Bug: The ANSI flag of Cast should be kept after being copied
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/21 15:38
Affects Version/s: 3.2.0

For Views and UDFs, it is important to show consistent results even the ANSI configuration is different in the running session. This is why many expressions like 'Add'/'Divide' making the ANSI flag part of its case class parameter list.
We should make it consistent for `Cast`
Bug: Variance of 1 record gives NULL in Spark 3.x and NaN in Spark 2.x
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Jun/21 16:56
Affects Version/s: 3.1.1

I have a few testcases in my suite which started failing after moving to Spark 3

And I noticed that the reason was that VARIANCE() function was earlier returning NaN when it was run on 1 record. Now, it gives NULL.

 
{code:java}
export SPARK_HOME=/usr/local/hadoop/spark-2.4.6-bin-hadoop2.7/
python
>>> import pyspark
>>> spark = pyspark.sql.SparkSession.builder.getOrCreate()
>>> spark.sql('SELECT  VARIANCE(1)').show()
+---------------------------+
|var_samp(CAST(1 AS DOUBLE))|
+---------------------------+
|                        NaN|
+---------------------------+{code}
With spark 3:
{code:java}
export SPARK_HOME=/usr/local/hadoop/spark-3.1.1-bin-hadoop2.7/
python
>>> import pyspark
>>> spark = pyspark.sql.SparkSession.builder.getOrCreate()
>>> spark.sql('SELECT VARIANCE(1)').show()
+---------------------------+
|variance(CAST(1 AS DOUBLE))|
+---------------------------+
|                       null|
+---------------------------+
{code}
 

Just thought I'd report it here as I didn't see it in any of the release notes
Bug: Remark the shuffle origin to ENSURE_REQUIREMENTS as far as possible
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Jun/21 07:25
Affects Version/s: 3.2.0

In some queries, we might repartition by some columns with a large partition number manually to make parallelism big enough. However if its output partitioning satisfies some other node (e.g. join/aggregate), this shuffle can not be optimized by AQE due to the shuffle origin.
Bug: Make the conversion to pandas data-type-based for ExtensionDtypes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 21:12
Affects Version/s: 3.2.0

The conversion from/to pandas includes logic for checking ExtensionDtypes data types and behaving accordingly.

That makes code hard to change or maintain.

We want to introduce the Ops class per ExtensionDtypes data type, and then make the conversion from/to pandas data-type-based for ExtensionDtypes.