av9ash/GitBugs huggingface dataset:
Bug: Do not collapse projects in CombineUnions if it hasCorrelatedSubquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 02:15
Affects Version/s: 3.3.0

 

SPARK-37915 added CollapseProject in rule CombineUnions, but it shouldn't collapse projects that contain correlated subqueries since haven't been de-correlated (PullupCorrelatedPredicates).

Here is a simple example to reproduce this issue
{code:java}
SELECT (SELECT IF(x, 1, 0)) AS a
FROM (SELECT true) t(x)
UNION 
SELECT 1 AS a {code}
Exception:
{code:java}
java.lang.IllegalStateException: Couldn't find x#4 in [] {code}
Bug: Nested subquery expressions deduplicate relations should be done bottom up
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/22 20:52
Affects Version/s: 3.3.0

When we have nested subquery expressions, there is a chance that deduplicate relations could replace an attributes with a wrong one. This is because the attributes replacement is done by top down than bottom up. This could happen if the subplan gets deduplicate relations first (thus two same relation with different attributes id), then a more complex plan built on top of the subplan (e.g. a UNION of queries with nested subquery expressions) can trigger this wrong attribute replacement error.
Bug: Cleanup unused spark.rpc.numRetries and spark.rpc.retry.wait configs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/May/22 23:34
Affects Version/s: 2.2.0

In SPARK-19450 (Spark 2.2.0), the last usages of the `spark.rpc.numRetries` and `spark.rpc.retry.wait` configs were removed. We should update the documentation and code to reflect this.
Bug: Provide query context in runtime error of cast overflow
Status: Open | Priority: Major | Resolution: 
Created: 15/May/22 07:30
Affects Version/s: 3.3.0

See castingCauseOverflowError and the error class CAST_CAUSES_OVERFLOW
https://github.com/apache/spark/blob/1b37f19876298e995596a30edc322c856ea1bbb4/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala#L97-L101
Bug: Spark SQL 'Like' operator behaves wrongly while filtering on partitioned column after Spark 3.1
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 20/May/22 15:08
Affects Version/s: 3.1.2

It seems like introduction of "like any" in spark 3.1 breaks "like" behaviour when filtering on partitioned column. Here is the example:

1. Create test table:
{code:java}
scala> spark.sql(
     | """
     | CREATE EXTERNAL TABLE tmp(
     |         f1 STRING
     |     )
     |     PARTITIONED BY (dt STRING)
     |     ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
     |     LINES TERMINATED BY '\n'
     |     STORED AS TEXTFILE
     |     LOCATION 's3://vlg-data-us-east-1/tmp/tmp/';
     | """) 
res2: org.apache.spark.sql.DataFrame = []{code}
2. insert something there:
{code:java}
scala> spark.sql(
     | """
     |     insert into table tmp partition(dt="2022051000") values("1")
     | """
     | )
res3: org.apache.spark.sql.DataFrame = [] {code}
3. Do select using 'like':
{code:java}
scala> spark.sql(
     |     """
     |         select * from tmp
     |         where dt like '202205100%'
     |     """
     |     ).show()
+---+---+
| f1| dt|
+---+---+
+---+---+ {code}
4. Do select using 'like any':
{code:java}
scala> spark.sql(
     |     """
     |         select * from tmp
     |         where dt like any ('202205100%')
     |     """
     |     ).show()
22/05/20 14:50:26 WARN HiveConf: HiveConf of name hive.server2.thrift.url does not exist
+---+----------+
| f1|        dt|
+---+----------+
|  1|2022051000|
+---+----------+ {code}
Expectation is that results 3 and 4 are identical, however this is not the case and result #3 is obviously wrong. 

 

*Environment: EMR*
Release label:emr-6.5.0
Hadoop distribution:Amazon 3.2.1
Applications:{*}Spark 3.1.2{*}, Hive 3.1.2, Livy 0.7.1
Bug: Throw an exception w/ an error class for multiple rows from a subquery used as an expression
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/22 17:53
Affects Version/s: 3.4.0

Users can trigger an illegal state exception by the SQL statement:

{code:sql}
> select (select a from (select 1 as a union all select 2 as a) t) as b
{code}

{code:java}
Caused by: java.lang.IllegalStateException: more than one row returned by a subquery used as an expression:
Subquery subquery#242, [id=#100]
+- AdaptiveSparkPlan isFinalPlan=true
   +- == Final Plan ==
      Union
      :- *(1) Project [1 AS a#240]
      :  +- *(1) Scan OneRowRelation[]
      +- *(2) Project [2 AS a#241]
         +- *(2) Scan OneRowRelation[]
   +- == Initial Plan ==
      Union
      :- Project [1 AS a#240]
      :  +- Scan OneRowRelation[]
      +- Project [2 AS a#241]
         +- Scan OneRowRelation[]

	at org.apache.spark.sql.execution.ScalarSubquery.updateResult(subquery.scala:83)
{code}
but such kind of exceptions are not supposed to be visible to users. Need to introduce an error class (or re-use an existing one), and replace the IllegalStateException.
Bug: Decimal precision loss from applying WidenSetOperationTypes before DecimalPrecision 
Status: In Progress | Priority: Major | Resolution: 
Created: 20/May/22 04:08
Affects Version/s: 3.3.1

The result of following SQL is 1.00 while 1.00000000000000000000 is expected
{code:java}
CREATE OR REPLACE TEMPORARY VIEW t3 AS VALUES (decimal(1)) tbl(v)
CREATE OR REPLACE TEMPORARY VIEW t4 AS SELECT CAST(v AS DECIMAL(18, 2)) AS v FROM t3;

SELECT CAST(1 AS DECIMAL(28, 2))
UNION ALL
SELECT v / v FROM t4; {code}
Bug: Support spark.history.fs.update.batchSize
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 16:35
Affects Version/s: 3.4.0

Current Spark History Server suffers when there are a large number of eventlog files under eventLog.dir: when a SHS starts, the initial scan may take a long time, and new eventlog files would not be scanned/parsed until the initial scan completes.

For example, if the initial scan takes 1-2 days(this is not uncommon in large environments), the newly finished spark jobs would not show up in SHS since their eventlog files are not scanned/parsed until the initial scan process finishes. This would result in a 1-2 days SHS malfunctioning since the newly finished spark jobs are most likely to be queried by users.
Bug: Remove the check for TimestampNTZ output in Analyzer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/May/22 10:16
Affects Version/s: 3.3.0

In [#36094|https://github.com/apache/spark/pull/36094], a check for failing TimestampNTZ output is added.

However, the check can cause misleading error message.

In 3.3:

 
{code:java}
> sql( "select date '2018-11-17' > 1").show()
org.apache.spark.sql.AnalysisException: Invalid call to toAttribute on unresolved object;
'Project [unresolvedalias((2018-11-17 > 1), None)]
+- OneRowRelation   
 
  at org.apache.spark.sql.catalyst.analysis.UnresolvedAlias.toAttribute(unresolved.scala:510)
  at org.apache.spark.sql.catalyst.plans.logical.Project.$anonfun$output$1(basicLogicalOperators.scala:70) {code}
In master or 3.2
{code:java}
> sql( "select date '2018-11-17' > 1").show()
org.apache.spark.sql.AnalysisException: cannot resolve '(DATE '2018-11-17' > 1)' due to data type mismatch: differing types in '(DATE '2018-11-17' > 1)' (date and int).; line 1 pos 7;
'Project [unresolvedalias((2018-11-17 > 1), None)]
+- OneRowRelation


  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) {code}
We should just remove the check
Bug: Use double quotes for values of SQL configs/DS options in error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/May/22 17:36
Affects Version/s: 3.3.0

All SQL configs/DS option values should be printed in SQL style in error messages, and wrapped by double quotes. For example, the value true of the config spark.sql.ansi.enabled should be highlighted as "true" to make it more visible in error messages.
Bug: Spark History Main Page App List Filtering
Status: In Progress | Priority: Major | Resolution: 
Created: 19/May/22 04:47
Affects Version/s: 3.4.0

Currently Spark History Server suffers when there are large number of jobs under the eventLog.dir. Loading the SHS main page may be very slow(showing `Loading History Summary ...`).

The issue is because the current SHS returns all jobs in the appList KV Store to the client browser without any filtering. This resulted massive data transfer and slowness for SHS in a large environment. For most users, many of the returned jobs are not in their interests.

Therefore, it could reduce the waiting time if we could add some filtering(such as user or queue) in the SHS main page.
Bug: Lower general ProcfsMetricsGetter error log levels except /proc/ lookup error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 12:01
Affects Version/s: 3.3.0

There are many user complain about the noisy warning messages in ProcessMetricsGetter

{code:java}
22/05/18 16:48:58 WARN ProcfsMetricsGetter: There was a problem with reading the stat file of the process.
java.io.FileNotFoundException: /proc/50371/stat (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at org.apache.spark.executor.ProcfsMetricsGetter.openReader$1(ProcfsMetricsGetter.scala:174)
	at org.apache.spark.executor.ProcfsMetricsGetter.$anonfun$addProcfsMetricsFromOneProcess$1(ProcfsMetricsGetter.scala:176)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2647)
	at org.apache.spark.executor.ProcfsMetricsGetter.addProcfsMetricsFromOneProcess(ProcfsMetricsGetter.scala:176)
	at org.apache.spark.executor.ProcfsMetricsGetter.$anonfun$computeAllMetrics$1(ProcfsMetricsGetter.scala:216)
	at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
	at scala.collection.immutable.Set$Set2.foreach(Set.scala:132)
	at org.apache.spark.executor.ProcfsMetricsGetter.computeAllMetrics(ProcfsMetricsGetter.scala:214)
	at org.apache.spark.metrics.ProcessTreeMetrics$.getMetricValues(ExecutorMetricType.scala:93)
	at org.apache.spark.executor.ExecutorMetrics$.$anonfun$getCurrentMetrics$1(ExecutorMetrics.scala:103)
	at org.apache.spark.executor.ExecutorMetrics$.$anonfun$getCurrentMetrics$1$adapted(ExecutorMetrics.scala:102)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at org.apache.spark.executor.ExecutorMetrics$.getCurrentMetrics(ExecutorMetrics.scala:102)
	at org.apache.spark.SparkContext.reportHeartBeat(SparkContext.scala:2578)
	at org.apache.spark.SparkContext.$anonfun$new$30(SparkContext.scala:587)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2022)
	at org.apache.spark.Heartbeater$$anon$1.run(Heartbeater.scala:46)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
{code}
Bug: Promote Structured Streaming over Spark Streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/22 06:59
Affects Version/s: 3.1.3

We see efforts of community are more focused on Structured Streaming (based on Spark SQL) than Spark Streaming (DStream). We would like to encourage end users to use Structured Streaming than Spark Streaming whenever possible for their workloads.
Bug: Jdbc dialect should decide which function could be pushed down.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/22 09:31
Affects Version/s: 3.4.0

Regardless of whether the functions are ANSI or not, most databases are actually unsure of their support.
So we should add a new API into JdbcDialect so that Jdbc dialect could decide which function could be pushed down.
Bug: Add JvmPauseMonitor to Spark History Server
Status: Open | Priority: Major | Resolution: 
Created: 18/May/22 09:38
Affects Version/s: 2.0.0

It is hard to understand what happens with Spark History Server when long JVM pauses (GC or host level pauses) are causing unresponsiveness. 
Similar to Hadoop's implementation (initially in HADOOP-9618) it would be beneficial to add the [JVMPauseMonitor|https://github.com/apache/hadoop/blob/f4b24c68e76df40d55258fc5391baabfa9ac362d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java] to the HistoryServer (HistoryServerSuite.scala).
This will make GC pauses obvious in logs and lets administrators easily notice it and react in time - adjust configurations with increasing the SHS heap size.
Bug: Prunes the duplicate columns from child of UnaryNode
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 07/May/22 02:23
Affects Version/s: 3.4.0


{code:scala}
sql("create table t1(a int, b int, c int) using parquet")
sql("select a, max(b) from (select a, b, a, b, a from t1) group by a").explain(true)
{code}

Current:
{noformat}
== Optimized Logical Plan ==
Aggregate [a#0], [a#0, max(b#1) AS max(b)#4]
+- Project [a#0, b#1, a#0, b#1, a#0]
   +- Relation default.t1[a#0,b#1,c#2] parquet
{noformat}


Expected:
{noformat}
== Optimized Logical Plan ==
Aggregate [a#0], [a#0, max(b#1) AS max(b)#4]
+- Project [a#0, b#1]
   +- Relation default.t1[a#0,b#1,c#2] parquet
{noformat}
Bug: Remove outer join if all output come from streamed side and buffered side keys exist unique key
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/22 02:57
Affects Version/s: 3.4.0

Improve the optimzation case using the distinct keys framework.

For example:
{code:java}
SELECT t1.* FROM t1 LEFT JOIN (SELECT distinct c1 as c1 FROM t)t2 ON t1.c1 = t2.c1
==>
SELECT t1.* FROM t1 {code}
Bug: The error message is different if disable broadcast join
Status: Open | Priority: Major | Resolution: 
Created: 13/May/22 04:42
Affects Version/s: 3.4.0

How to reproduce this issue:
{code:scala}
   sql(
      """
        |CREATE TABLE store_sales(
        |  ss_item_sk INT,
        |  ss_ext_sales_price DECIMAL(38,4)) USING parquet
      """.stripMargin)
    sql(
      """
        |CREATE TABLE item(
        |  i_item_sk INT,
        |  i_brand_id INT) USING parquet
      """.stripMargin)

    sql(
      """
        |INSERT INTO store_sales VALUES
        |  (1, 9999999999999999999999999999999999.6012),
        |  (1, 9999999999999999999999999999999999.9234),
        |  (1, 9999999999999999999999999999999999.2856),
        |  (2, 6874.6012),
        |  (2, 2828.9223),
        |  (2, 6067.6034),
        |  (2, 6067.6034),
        |  (3, 999999999999999999999999999999999.2812),
        |  (3, 999999999999999999999999999999999.2823)
      """.stripMargin)

    sql(
      """
        |INSERT INTO item VALUES
        |  (1, 7003002),
        |  (1, 7003002),
        |  (2, 10002003),
        |  (2, 10002003),
        |  (2, 10002003),
        |  (3, 10002004),
        |  (3, 10002004),
        |  (3, 10002004),
        |  (3, 10002004)
      """.stripMargin)

Seq(-1, 1000000000L).foreach { broadcastThreshold =>
  withSQLConf(
    SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> broadcastThreshold.toString,
    SQLConf.ANSI_ENABLED.key -> "true") {
    val df = sql(
      """
        |SELECT
        |  item.i_brand_id brand_id,
        |  avg(ss_ext_sales_price) avg_agg
        |FROM store_sales, item
        |WHERE store_sales.ss_item_sk = item.i_item_sk
        |GROUP BY item.i_brand_id
      """.stripMargin)
    val error = intercept[SparkException] {
      df.collect()
    }
    println("Error message: " + error.getMessage)
  }
}
{code}

{noformat}
Error message: org.apache.spark.SparkArithmeticException: [CANNOT_CHANGE_DECIMAL_PRECISION] Decimal(expanded,999999999999999999999999999999999.28175,38,5}) cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to false to bypass this error.
Error message: org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow in sum of decimals. If necessary set spark.sql.ansi.enabled to false (except for ANSI interval type) to bypass this error.
{noformat}
Bug: Provide runtime error query context for Binary Arithmetic when WSCG is off
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/22 14:35
Affects Version/s: 3.3.0

Currently, for most of the cases, the project https://issues.apache.org/jira/browse/SPARK-38615 is able to show where the runtime errors happen within the original query.
However, after trying on production, I found that the following queries won't show where the divide by 0 error happens


{code:java}
create table aggTest(i int, j int, k int, d date) using parquet
insert into aggTest values(1, 2, 0, date'2022-01-01')
select sum(j)/sum(k),percentile(i, 0.9) from aggTest group by d{code}

With `percentile` function in the query, the plan can't execute with whole stage codegen. Thus the child plan of `Project` is serialized to executors for execution, from ProjectExec:


{code:java}
  protected override def doExecute(): RDD[InternalRow] = {
    child.execute().mapPartitionsWithIndexInternal { (index, iter) =>
      val project = UnsafeProjection.create(projectList, child.output)
      project.initialize(index)
      iter.map(project)
    }
  }{code}
Note that the `TreeNode.origin` is not serialized to executors since `TreeNode` doesn't extend the trait `Serializable`, which results in an empty query context on errors. For more details, please read https://issues.apache.org/jira/browse/SPARK-39140

A dummy fix is to make `TreeNode` extend the trait `Serializable`. However, it can be performance regression if the query text is long (every `TreeNode` carries it for serialization). 
A better fix is to introduce a new trait `SupportQueryContext` and materialize the truncated query context for special expressions. This jira targets on binary arithmetic expressions only. I will create follow-ups for the remaining expressions which support runtime error query context.
Bug: Optimize FIRST when used as a single aggregate function
Status: In Progress | Priority: Major | Resolution: 
Created: 12/May/22 23:56
Affects Version/s: 3.3.0

When `FIRST` is a single aggregate function in `Aggregate` we could either rewrite whole query or optimize execution logic. 
 * Plan => `SELECT FIRST(<col>) FROM <table>` => `SELECT <col> FROM <table> LIMIT 1`. Note that setting `ignoreNulls` to `true` should block such rewrite since returns could differ in case all values of <col> are `NULL`
 * Execution => `SELECT FIRST(<col>) FROM <table> GROUP BY <some_col>` => short circuit iteration per key once a value for `FIRST` is set.
Bug: JavaSerializer doesn't serialize the fields of superclass
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 10/May/22 15:10
Affects Version/s: 3.3.0

To reproduce:

 
{code:java}
abstract class AA {
  val ts = System.nanoTime()
}
case class BB(x: Int) extends AA {
}

val input = BB(1)
println("original ts: " + input.ts)

val javaSerializer = new JavaSerializer(new SparkConf())
val javaInstance = javaSerializer.newInstance()
val bytes1 = javaInstance.serialize[BB](input)
val obj1 = javaInstance.deserialize[BB](bytes1)
println("deserialization result from java: " + obj1.ts)

val kryoSerializer = new KryoSerializer(new SparkConf())
val kryoInstance = kryoSerializer.newInstance()
val bytes2 = kryoInstance.serialize[BB](input)
val obj2 = kryoInstance.deserialize[BB](bytes2)
println("deserialization result from kryo: " + obj2.ts) {code}
 

 

The output is

 
{code:java}
original ts: 115014173658666
deserialization result from java: 115014306794333
deserialization result from kryo: 115014173658666{code}
 

We can see that the fields from the superclass AA are not serialized with JavaSerializer. When switching to KryoSerializer, it works.

This caused bugs in the project SPARK-38615: TreeNode.origin with actual information is not serialized to executors when a plan can't be executed with whole-staged-codegen.

It could also lead to bugs in serializing the lambda function within RDD API like 

mapPartitions/mapPartitionsWithIndex/etc.
Bug: Add custom metric of skipped null values for stream join operator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/May/22 02:33
Affects Version/s: 3.4.0

When STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS is enabled, Spark will skip null values in hash-based stream-stream joins. But currently we cannot track if this is happened and how many it is. For our customers, they have an interest to know the number of values skipped for better tracking purpose.
Bug: Correct conditional expression constant folding
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/22 11:50
Affects Version/s: 3.4.0

For a conditional expression, we can not partially fold the constant inside it's children. For example if c1 or c2 is not null, the last branch should be never hit.
{code:java}
SELECT COALESCE(c1, c2, 1/0);
{code}
Besides, for CaseWhen and If, we should mark it as foldable if it's children are foldable. It is safe since the both non-codegen and codegen code path have already respected the evaluation order.
Bug: Fix FsHistoryProvider race condition between update and clean app data
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/May/22 05:22
Affects Version/s: 3.2.1

After SPARK-29043, FsHistoryProvider will list the log info without waiting all `mergeApplicationListing` task finished.

However, the method `cleanAppData` for stale data is not thread safe and it could lead to a scenario when `cleanAppData` delete the entry of 

ApplicationInfoWrapper for an application right after it has been updated by `mergeApplicationListing`.

So there will be cases when the HS Web UI displays `Application not found` for applications whose logs does exist.
Bug: Python UDF does not follow the conditional expression evaluation order
Status: Open | Priority: Major | Resolution: 
Created: 07/May/22 08:11
Affects Version/s: 3.4.0

We will pull out python udf and run them eagerly whatever can be reached or not, so the below query will fail:
{code:java}
SELECT if(true, 1, python_udf(1/0)) {code}