av9ash/GitBugs huggingface dataset:
Bug: Do not add dynamic partition pruning if there exists static partition pruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 02:30
Affects Version/s: 3.3.0

Dynamic partition pruning add a filter as long as the join condition contains partition columns. But if there exists other condition which contains the static partition pruning, it's unnecessary to add an extra dynamic partition pruning.

For example:
{code:java}
CREATE TABLE t1 (c1 int) USING PARQUET PARTITIONED BY (p1 string);
CREATE TABLE t2 (c2 int) USING PARQUET PARTITIONED BY (p2 string);

SELECT * FROM t1 JOIN t2 ON t1.p1 = t2.p2 and t1.p1 = 'a' AND t2.c2 > 0;
{code}
Bug: SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/22 22:18
Affects Version/s: 3.1.0

*Background*

In [this PR|https://github.com/apache/spark/pull/26508] (SPARK-26260) the SHS stage metric percentiles were updated to only include successful tasks when using disk storage. It did this by making the values for each metric negative when the task is not in a successful state. This approach was chosen to avoid breaking changes to disk storage. See [this comment|https://github.com/apache/spark/pull/26508#issuecomment-554540314] for context.

To get the percentiles, it reads the metric values, starting at 0, in ascending order. This filters out all tasks that are not successful because the values are less than 0. To get the percentile values it scales the percentiles to the list index of successful tasks. For example if there are 200 tasks and you want percentiles [0, 25, 50, 75, 100] the lookup indexes in the task collection are [0, 50, 100, 150, 199].

*Issue*
For metrics 1) shuffle total reads and 2) shuffle total blocks, the above PR incorrectly makes the metric indices positive. This means tasks that are not successful are included in the percentile calculations. The percentile lookup index calculation is still based on the number of successful task so the wrong task metric is returned for a given percentile. This was not caught because the unit test only verified values for one metric, executorRunTime.

*Steps to Reproduce*
_SHS UI_
 # Find a spark application in the SHS that has failed tasks for a stage with shuffle read.
 # Navigate to the stage UI.
 # Look at the max shuffle read size in the summary metrics
 # Sort the tasks by shuffle read size descending. You'll see it doesn't match step 3.

 

!image-2022-02-23-14-19-33-255.png|width=789,height=389!

_API_
 # For the same stage in the above repro steps, make a request to the task summary endpoint (e.g. /api/v1/applications/application_1632281309592_21294517/1/stages/6/0/taskSummary?quantiles=0,0.25,0.5,0.75,1.0)
 # Look at the shuffleReadMetrics.readBytes and shuffleReadMetrics.totalBlocksFetched. You will see -2 for at least some of the lower percentiles and the positive values will also be wrong.
Bug: Use error classes in the execution errors of date/timestamp handling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 18:31
Affects Version/s: 3.3.0

Migrate the following errors in QueryExecutionErrors:
* sparkUpgradeInReadingDatesError
* sparkUpgradeInWritingDatesError
* timeZoneIdNotSpecifiedForTimestampTypeError
* cannotConvertOrcTimestampToTimestampNTZError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryExecutionErrorsSuite.
Bug: Avoid duplicating complex partitioning expressions
Status: Open | Priority: Major | Resolution: 
Created: 22/Feb/22 07:03
Affects Version/s: 3.3.0

Spark will duplicate all non-trivial expressions in Window.partitionBy, that will result in duplicate exchanges and WindowExec nodes.

An example unit test:
{code}
  test("SPARK-38282: Avoid duplicating complex partitioning expressions") {
    val group = functions.col("id") % 2
    val min = functions.min("id").over(Window.partitionBy(group))
    val max = functions.max("id").over(Window.partitionBy(group))

    val df1 = spark.range(1, 4)
      .withColumn("ratio", max / min)

    val df2 = spark.range(1, 4)
      .withColumn("min", min)
      .withColumn("max", max)
      .select(col("id"), (col("max") / col("min")).as("ratio"))

    Seq(df1, df2).foreach { df =>
      checkAnswer(
        df,
        Seq(Row(1L, 3.0), Row(2L, 1.0), Row(3L, 3.0)))

      val windows = collect(df.queryExecution.executedPlan) {
        case w: WindowExec => w
      }
      assert(windows.size == 1)
    }
  }
{code}

The query plan for this (_w0#5L and _w1#6L are duplicates):
{code}
Window [min(id#2L) windowspecdefinition(_w1#6L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we1#8L], [_w1#6L]
   +- *(4) Sort [_w1#6L ASC NULLS FIRST], false, 0
      +- AQEShuffleRead coalesced
         +- ShuffleQueryStage 1
            +- Exchange hashpartitioning(_w1#6L, 5), ENSURE_REQUIREMENTS, [id=#256]
               +- *(3) Project [id#2L, _w1#6L, _we0#7L]
                  +- Window [max(id#2L) windowspecdefinition(_w0#5L, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS _we0#7L], [_w0#5L]
                     +- *(2) Sort [_w0#5L ASC NULLS FIRST], false, 0
                        +- AQEShuffleRead coalesced
                           +- ShuffleQueryStage 0
                              +- Exchange hashpartitioning(_w0#5L, 5), ENSURE_REQUIREMENTS, [id=#203]
                                 +- *(1) Project [id#2L, (id#2L % 2) AS _w0#5L, (id#2L % 2) AS _w1#6L]
                                    +- *(1) Range (1, 4, step=1, splits=2)
{code}
Bug: Use error classes in the parsing errors of windows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 15:53
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* repetitiveWindowDefinitionError
* invalidWindowReferenceError
* cannotResolveWindowReferenceError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Fix flaky text of  HealthTrackerIntegrationSuite
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/22 01:33
Affects Version/s: 3.2.1

{code:java}
[info] HealthTrackerIntegrationSuite:
[info] - If preferred node is bad, without excludeOnFailure job will fail (120 milliseconds)
[info] - With default settings, job can succeed despite multiple bad executors on node (3 seconds, 78 milliseconds)
[info] - Bad node with multiple executors, job will still succeed with the right confs *** FAILED *** (61 milliseconds)
[info]   Map() did not equal Map(0 -> 42, 5 -> 42, 1 -> 42, 6 -> 42, 9 -> 42, 2 -> 42, 7 -> 42, 3 -> 42, 8 -> 42, 4 -> 42) (HealthTrackerIntegrationSuite.scala:94)
[info]   Analysis:
[info]   HashMap(0: -> 42, 1: -> 42, 2: -> 42, 3: -> 42, 4: -> 42, 5: -> 42, 6: -> 42, 7: -> 42, 8: -> 42, 9: -> 42)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)

{code}
Bug: Allow safe up-cast expressions in correlated equality predicates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Feb/22 19:17
Affects Version/s: 3.3.0

Relax the constraint added in SPARK-35080 by allowing safe up-cast expressions in correlated equality predicates. For example: 
{code:java}
cast(x AS STRING) = outer(x) {code}
Bug: Nullability propagation in transformUpWithNewOutput
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 09:14
Affects Version/s: 3.2.0

The nullability of a replaced attribute should be `a.nullable` instead of `b.nullable`:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala#L357

The scenario is a left outer join where the RHS attributes are replaced bottom up.
Bug: Make the output of V1/V2 "desc extended table" consistent
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Feb/22 13:00
Affects Version/s: 3.3.0

The V1 "DESC TABLE EXTENDED" command contains info of "Database" and "Table", which can be used by external tools like DBT.

However, the V2 version contains only one field "name" representing "catalog.database.table". External tools can't recognize it. Also, it is weird to have different command output from the same command.

We should fix it.
Bug: Fail to read parquet files after writing the hidden file metadata in
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/22 10:25
Affects Version/s: 3.2.1

Selecting and then writing df containing hidden file metadata column `_metadata` into a file format like `parquet`, `delta` will still keep the internal `Attribute` metadata information. Then when reading those `parquet`, `delta` files again, it will actually break the code, because it wrongly thinks user data schema named `_metadata` is a hidden file source metadata column.

 

Reproducible code:
{code:java}
// prepare a file source df
df.select("*", "_metadata")
  .write.format("parquet").save(path)
spark.read.format("parquet").load(path)
  .select("*").show(){code}
Bug: Introduce a new config to require all cluster keys on Aggregate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/22 09:01
Affects Version/s: 3.3.0

We still find HashClusteredDistribution be useful for batch query as well. For example, we had a case with lower parallelism than expected due to the fact ClusteredDistribution is used for aggregation which matches with HashPartitioning with sub-key groups (note that the technical parallelism also depends on "cardinality" - picking sub-key groups means having less cardinality).

We propose to introduce a new config to require all cluster keys on Aggregate, leveraging HashClusteredDistribution. That said, we propose to rename back HashClusteredDistribution with retaining NOTE for stateful operator. The distribution should not be still touched anyway due to the requirement of stateful operator, but can be co-used with batch case if needed.
Bug: ANSI mode: avoid potential runtime error in HashJoin.extractKeyExprAt() 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 05:26
Affects Version/s: 3.2.2

SubqueryBroadcastExec retrieves the partition key from the broadcast results based on the type of HashedRelation returned. If the key is packed inside a Long, we extract it through bitwise operations and cast it as Byte/Short/Int if necessary.

The casting here can cause a potential runtime error. We should fix it.
Bug: Support query stage show runtime statistics in formatted explain mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 03:24
Affects Version/s: 3.3.0

The formatted explalin mode is the powerful explain mode to show the details of query plan. In AQE, the query stage know its statistics if has already materialized. So it can help to quick check the conversion of plan, e.g. join selection. 

A simple example:
{code:java}
SELECT * FROM t JOIN t2 ON t.c = t2.c;{code}
 
{code:java}
== Physical Plan ==
AdaptiveSparkPlan (21)
+- == Final Plan ==
   * SortMergeJoin Inner (13)
   :- * Sort (6)
   :  +- AQEShuffleRead (5)
   :     +- ShuffleQueryStage (4), Statistics(sizeInBytes=16.0 B, rowCount=1)
   :        +- Exchange (3)
   :           +- * Filter (2)
   :              +- Scan hive default.t (1)
   +- * Sort (12)
      +- AQEShuffleRead (11)
         +- ShuffleQueryStage (10), Statistics(sizeInBytes=16.0 B, rowCount=1)
            +- Exchange (9)
               +- * Filter (8)
                  +- Scan hive default.t2 (7)
+- == Initial Plan ==
   SortMergeJoin Inner (20)
   :- Sort (16)
   :  +- Exchange (15)
   :     +- Filter (14)
   :        +- Scan hive default.t (1)
   +- Sort (19)
      +- Exchange (18)
         +- Filter (17)
            +- Scan hive default.t2 (7){code}
Bug: Consider to include WriteBatch's memory in the memory usage of RocksDB state store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 11:16
Affects Version/s: 3.2.1

Current RocksDB state store actually use a unlimited {{WriteBatch}} with a DB, the {{WriteBatch}} would not be cleared until the micro-batch data committed, which results that the memoy usage of {{WriteBatch}} could be very huge.

We should consider to add the approximate memory usgae of WriteBatch as the totdal memory usage and also print it separately.
Bug: Adaptive coalesce not working with df persist
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 10/Feb/22 10:02
Affects Version/s: 3.2.1

{code:java}
// code placeholder
val spark = SparkSession.builder().master("local[4]").appName("Test")
                        .config("spark.sql.adaptive.enabled", "true")
                        .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
                        .config("spark.sql.adaptive.advisoryPartitionSizeInBytes", "50m")
                        .config("spark.sql.adaptive.coalescePartitions.minPartitionNum", "1")
                        .config("spark.sql.adaptive.coalescePartitions.initialPartitionNum", "1024")
                        .getOrCreate()
val df = spark.read.csv("<Input File Path>")
val df1 = df.distinct()
df1.persist() // On removing this line. Code works as expected
df1.write.csv("<Output File Path>") {code}
Without df1.persist, df1.write.csv writes 4 partition files of 50 MB each which is expected

[https://i.stack.imgur.com/tDxpV.png]

If I include df1.persist, Spark is writing 200 partitions(adaptive coalesce not working) With persist

[https://i.stack.imgur.com/W13hA.png]
Bug: Absolute file paths specified in create/alter table are treated as relative
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/22 08:25
Affects Version/s: 3.1.1

After https://github.com/apache/spark/pull/28527 we change to create table under the database location when the table location specified is relative. However the criteria to determine if a table location is relative/absolute is URI.isAbsolute, which basically checks if the table location URI has a scheme defined. So table URIs like /table/path are treated as relative and the scheme and authority of the database location URI are used to create the table. For example, when the database location URI is s3a://bucket/db, the table will be created at s3a://bucket/table/path, while it should be created under the file system defined in SessionCatalog.hadoopConf instead.

This also applies to alter table.
Bug: Union's maxRows and maxRowsPerPartition may overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 11:41
Affects Version/s: 3.0.3

{code:java}
scala> val df1 = spark.range(0, Long.MaxValue, 1, 1)
df1: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> val df2 = spark.range(0, 100, 1, 10)
df2: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> val union = df1.union(df2)
union: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> union.queryExecution.logical.maxRowsPerPartition
res19: Option[Long] = Some(-9223372036854775799)

scala> union.queryExecution.logical.maxRows
res20: Option[Long] = Some(-9223372036854775709)
 {code}
Bug: Error Classes AnalysisException is not propagated in FunctionRegistry
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Feb/22 03:43
Affects Version/s: 3.3.0

While trying to migrate invalidLiteralForWindowDurationError to use Error Classes (SPARK-38110), I realized that the error class is not propagating correctly. This is because the AnalysisException is caught and rethrown at [FunctionRegistry:154|https://github.com/apache/spark/blob/43e93b581ea5f7a1ba6cf943e6624f6847ebc3a8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L154] without a case for Error Classes. I will attach a fix for this specific error here to avoid bundling too many changes into one PR for SPARK-38110. I anticipate that similar changes will need to be made where AnalysisException is rethrown as the migration continues.
Bug: Avro Complex Union Type return `member$I`
Status: Open | Priority: Major | Resolution: 
Created: 17/Feb/22 23:28
Affects Version/s: 3.2.1

*Short Description*

When reading complex union types from Avro files, there seems to be some information lost as the name of the record is omitted and {{member$i}} is instead returned.

*Long Description*

+Error+

Given the Avro schema {{{}schema.avsc{}}}, I would expected the schema when reading the avro file using {{read_avro.py}} to be as {{{}expected.txt{}}}. Instead, I get the schema output in {{reality.txt}} where {{RecordOne}} became {{{}member0{}}}, etc.

This causes information lost and makes the DataFrame unusable.

From my understanding this behavior was implemented [here.|https://github.com/databricks/spark-avro/pull/117]

 
{code:java|title=read_avro.py}
df = spark.read.format("avro").load("path/to/my/file.avro")
df.printSchema()
 {code}
{code:java|title=schema.avsc}
 {
 "type": "record",
 "name": "SomeData",
 "namespace": "my.name.space",
 "fields": [
  {
   "name": "ts",
   "type": {
    "type": "long",
    "logicalType": "timestamp-millis"
   }
  },
  {
   "name": "field_id",
   "type": [
    "null",
    "string"
   ],
   "default": null
  },
  {
   "name": "values",
   "type": [
    {
     "type": "record",
     "name": "RecordOne",
     "fields": [
      {
       "name": "field_a",
       "type": "long"
      },
      {
       "name": "field_b",
       "type": {
        "type": "enum",
        "name": "FieldB",
        "symbols": [
            "..."
        ],
       }
      },
      {
       "name": "field_C",
       "type": {
        "type": "array",
        "items": "long"
       }
      }
     ]
    },
    {
     "type": "record",
     "name": "RecordTwo",
     "fields": [
      {
       "name": "field_a",
       "type": "long"
      }
     ]
    }
   ]
  }
 ]
}{code}
{code:java|title=expected.txt}
root
 |-- ts: timestamp (nullable = true)
 |-- field_id: string (nullable = true)
 |-- values: struct (nullable = true)
 |    |-- RecordOne: struct (nullable = true)
 |    |    |-- field_a: long (nullable = true)
 |    |    |-- field_b: string (nullable = true)
 |    |    |-- field_c: array (nullable = true)
 |    |    |    |-- element: long (containsNull = true)
 |    |-- RecordTwo: struct (nullable = true)
 |    |    |-- field_a: long (nullable = true)
{code}
{code:java|title=reality.txt}
root
 |-- ts: timestamp (nullable = true)
 |-- field_id: string (nullable = true)
 |-- values: struct (nullable = true)
 |    |-- member0: struct (nullable = true)
 |    |    |-- field_a: long (nullable = true)
 |    |    |-- field_b: string (nullable = true)
 |    |    |-- field_c: array (nullable = true)
 |    |    |    |-- element: long (containsNull = true)
 |    |-- member1: struct (nullable = true)
 |    |    |-- field_a: long (nullable = true)
{code}
Bug: Fix JsonSuite and ParquetIOSuite under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 14:09
Affects Version/s: 3.3.0

Fix the following test suites with ANSI mode on:
 * JsonLegacyTimeParserSuite

 * JsonV1Suite

 * JsonV2Suite

 * ParquetIOSuite
Bug: Test invalid datetime parsing under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 07:45
Affects Version/s: 3.3.0

We should run datetime-parsing-invalid.sql under ANSI mode in SQLQueryTestSuite for improving test coverage. 

Also, we can simply set ANSI mode as off in DateFunctionsSuite, so that the test suite can pass after we set up a new test job with ANSI on.
Bug: UnresolvedException: Invalid call to dataType on unresolved object caused by GetDateFieldOperations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 02:39
Affects Version/s: 3.2.1

{code:java}
test("GetDateFieldOperations should skip unresolved nodes") {
  withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
    val df = Seq("1644821603").map(i => (i.toInt, i)).toDF("tsInt", "tsStr")
    val df1 = df.select(df("tsStr").cast("timestamp")).as("df1")
    val df2 = df.select(df("tsStr").cast("timestamp")).as("df2")
    df1.join(df2, $"df1.tsStr" === $"df2.tsStr", "left_outer")
    val df3 = df1.join(df2, $"df1.tsStr" === $"df2.tsStr", "left_outer")
      .select($"df1.tsStr".as("timeStr")).as("df3")
    // This throws "UnresolvedException: Invalid call to
    // dataType on unresolved object" instead of "AnalysisException: Column 'df1.timeStr' does not exist."
    df3.join(df1, year($"df1.timeStr") === year($"df3.tsStr"))
  }
} {code}
Bug: Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 05:31
Affects Version/s: 3.3.0

Block lateral subqueries in CheckAnalysis that contain DISTINCT aggregate and correlated non-equality predicates. This can lead to incorrect results as DISTINCT will be rewritten as Aggregate during the optimization phase.

For example
{code:java}
CREATE VIEW t1(c1, c2) AS VALUES (0, 1)
CREATE VIEW t2(c1, c2) AS VALUES (1, 2), (2, 2)
SELECT * FROM t1 JOIN LATERAL (SELECT DISTINCT c2 FROM t2 WHERE c1 > t1.c1)
{code}
The correct results should be (0, 1, 2) but currently, it gives  be[(0, 1, 2), (0, 1, 2)].
Bug: Relax the requirement of data type comparison for keys in stream-stream join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/22 11:22
Affects Version/s: 3.1.2

Currently, stream-stream join checks for the data type compatible between left keys and right keys. It is done as "strict" checking, requiring nullability as same for both sides. This leads to throw assertion error if optimizer turns some columns in one side from nullable to non-nullable but not touching opposite side.

If it is logically correct to relax the nullability check (with deciding proper type on output schema), we should do it to avoid any possible issue from optimization.
Bug: Apply strict nullability of nested column in time window / session window
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Feb/22 07:29
Affects Version/s: 3.2.1

In TimeWindow and SessionWindow, we define dataType of these function expressions as StructType having two nested columns "start" and "end", which is "nullable".

And we replace these expressions in the analyzer via corresponding rules, TimeWindowing for TimeWindow, and SessionWindowing for SessionWindow.

The rules replace the function expressions with Alias, referring CreateNamedStruct. For the value side of CreateNamedStruct, we don't specify anything about nullability, which leads to a risk the value side may be interpreted (or optimized) as non-nullable, which would make inconsistency.

We should make sure the nullability of columns in CreateNamedStruct remains the same with dataType definition on these function expressions.
Bug: Improve vectorized loop for Packed encodings
Status: Open | Priority: Major | Resolution: 
Created: 18/Feb/22 19:14
Affects Version/s: 3.2.1

In [https://github.com/apache/spark/blob/439975590cf4f21c2a548a2ac6231eb234e1a2f3/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java#L239] the vectorized reader is called one row at a time instead of calling multiple rows at a time. This loses the benefit of a vectorized reader and should be improved.
Bug: Change Cast.toString as "cast" instead of "ansi_cast" under ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/22 13:24
Affects Version/s: 3.3.0

Revert [https://github.com/apache/spark/pull/27608] and make Cast.toString as

"cast($child as ${dataType.simpleString})" under ANSI mode.

 

So that we can avoid test failures from TPCDS plan stability test suites