av9ash/GitBugs huggingface dataset:
Bug: Respect NaNvl in EquivalentExpressions for expression elimination
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/22 08:27
Affects Version/s: 3.4.0

For example the query will fail:
{code:java}
set spark.sql.ansi.enabled=true;
set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SELECT nanvl(1, 1/0 + 1/0);  {code}
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 1 times, most recent failure: Lost task 0.0 in stage 4.0 (TID 4) (10.221.98.68 executor driver): org.apache.spark.SparkArithmeticException: divide by zero. To return NULL instead, use 'try_divide'. If necessary set spark.sql.ansi.enabled to false (except for ANSI interval type) to bypass this error.
== SQL(line 1, position 17) ==
select nanvl(1 , 1/0 + 1/0)
                 ^^^    at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:151)
 {code}
We should respect the ordering of conditional expression that always evaluate the predicate branch first, so the query above should not fail.
Bug: Respect Table capability `ACCEPT_ANY_SCHEMA` in default column resolution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/22 09:16
Affects Version/s: 3.4.0

If a V2 table has the capability of [ACCEPT_ANY_SCHEMA|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java#L94], we should skip adding default column values to the insert schema.
Bug: Add Executor Pod inter-pod anti-affinity
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 26/Apr/22 09:13
Affects Version/s: 3.2.1

h3. Why should we need this?

When Spark On Kubernetes is running, Executor Pod clusters occur in certain conditions (uneven resource allocation in Kubernetes, high load On some nodes, low load On some nodes), causing Shuffle data skew. This causes Spark application to fail or performance bottlenecks, such as Shuffle Fetch timeout and connection refuse after connection number.
h3. Why should use this?

The functionality mentioned in this PR was tested on a cluster.

Using three Kubernetes Node(node-1, node-2, node-3).

In the case of sufficient or insufficient cluster resources, it  has the same effect Whether the feature is enabled or not. Kubernetes assigns pods to nodes with low load based on global resources. When only one node has a small load, Kubernetes will schedule all executor pods to this node.
Bug: Hive client should not gather statistic by default.
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Apr/22 10:22
Affects Version/s: 3.1.2

When use `InsertIntoHiveTable`, when insert overwrite partition, it will call
Hive.loadPartition(), in this method, when `hive.stats.autogather` is true(default is true)

 

{code:java}
// Some comments here
public String getFoo()
      if (oldPart == null) {
        newTPart.getTPartition().setParameters(new HashMap<String,String>());
        if (this.getConf().getBoolVar(HiveConf.ConfVars.HIVESTATSAUTOGATHER)) {
          StatsSetupConst.setBasicStatsStateForCreateTable(newTPart.getParameters(),
              StatsSetupConst.TRUE);
        }

public static void setBasicStatsStateForCreateTable(Map<String, String> params, String setting) {
  if (TRUE.equals(setting)) {
    for (String stat : StatsSetupConst.supportedStats) {
      params.put(stat, "0");
    }
  }
  setBasicStatsState(params, setting);
} 

public static final String[] supportedStats = {NUM_FILES,ROW_COUNT,TOTAL_SIZE,RAW_DATA_SIZE};
{code}




Then it set default rowNum as 0, but since spark will update numFiles and rawSize, so rowNum remain 0.

This impact other system like presto's CBO.
Bug: Use double quotes for SQL configs in error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Apr/22 16:54
Affects Version/s: 3.3.0

All SQL configs should be printed in SQL style in error messages, and wrapped by double quotes. For example, the config spark.sql.ansi.enabled should be highlighted as "spark.sql.ansi.enabled" to make it more visible in error messages.
Bug: SparkSQL parse partition value does not support all data types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/22 21:26
Affects Version/s: 3.3.0

When Spark needs to infer schema, it needs to parse string to a type. Not all data types are supported so far in this path. For example, binary is known to not be supported. If a user uses binary column, and if the user does not use a metastore, then SparkSQL could fall back to schema inference thus fail to execute during table scan. This should be a bug as schema inference is supported but some types are missing.

string might be converted to all types except ARRAY, MAP, STRUCT, etc. Also because when converting from a string, small scale type won't be identified if there is a larger scale type. For example, short and long 

Based on Spark SQL data types: https://spark.apache.org/docs/latest/sql-ref-datatypes.html, we can support the following types:

BINARY
BOOLEAN

And there are two types that I am not sure if SparkSQL is supporting:
YearMonthIntervalType
DayTimeIntervalType
Bug: Typo in error messages of decimal overflow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/22 19:50
Affects Version/s: 3.2.1

   org.apache.spark.SparkArithmeticException 

   Decimal(expanded,10000000000000000000000000000000000000.1,39,1}) cannot be represented as Decimal(38, 1). If necessary set spark.sql.ansi.enabled to false to bypass this error.
 

As shown in {{decimalArithmeticOperations.sql.out}}

Notice the extra {{}}} before ‘cannot’
Bug: Skipping allocating vector for repetition & definition levels when possible
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/22 19:49
Affects Version/s: 3.3.0

Currently the vectorized Parquet reader will allocate vectors for repetition and definition levels in all cases. However in certain cases (e.g., when reading primitive types) this is not necessary and should be avoided.
Bug: Keep rowCount after hive table partition pruning if table only have hive statistics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/22 14:37
Affects Version/s: 3.2.1

If the partitioned table only has hive generated statistics, and the statistics are stored in partition properties. HiveTableRelation cannot obtain rowCount because the table statistics do not exist. We can generate rowCount based on the statistics of these pruned partitions
Bug: Support ALTER TABLE ... DROP [IF EXISTS] COLUMN .. syntax
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/22 19:07
Affects Version/s: 3.2.0

Currently `ALTER TABLE ... DROP COLUMN(s) ...` syntax will always throw error if the column doesn't exist. We would like to provide an (IF EXISTS) syntax to provide better user experience for downstream handlers (such as Delta) that support it, and make consistent with some other DMLs such as `DROP TABLE (IF EXISTS)`
Bug: Add `getInputSignature` and `getOutputSignature` APIs for spark ML models/transformers
Status: Open | Priority: Major | Resolution: 
Created: 28/Apr/22 14:22
Affects Version/s: 3.4.0

Add `getInputSignature` and `getOutputSignature` APIs for spark ML models/transformers:

The `getInputSignature` API return a list of column name and type which represent the ML model / transformer's input columns.

The `getOutputSignature` API return a list of column column name and type which represent the ML model / transformer's output columns.

These 2 APIs are useful to in third-party library such as mlflow, which requires the information of input / output signature of the Model.
Bug: Support Alter Table/Partition Concatenate command
Status: Open | Priority: Major | Resolution: 
Created: 27/Apr/22 06:33
Affects Version/s: 3.3.0

Hi, folks, 

In Hive, we can use following command to merge small files, however, there is not a corresponding command to do that in Spark SQL. 

I believe it's useful and it's not enough only using AQE.  Is anyone working on this to merge small files? If not, I want to create a PR to implement it

 
{code:java}
ALTER TABLE table_name [PARTITION (partition_key = 'partition_value' [, ...])] CONCATENATE;{code}
 

[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterTable/PartitionConcatenate]
Bug: Make thriftserver and sparksql-cli support in-memory catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/22 11:06
Affects Version/s: 3.4.0

# with in-memory, we can start multiple instances in the same folder without the limitation of derby.
 # spark now is multi catalog architecture, the built-in catalog is not always necessary, in-memory is much lightweight if it is unused
Bug: Parser changes to enforce `()` for creating table without any columns
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 25/Apr/22 21:55
Affects Version/s: 3.4.0

We would like to enforce the `()` for `CREATE TABLE` queries to explicit indicate a table without any columns will be created.

E.g. `CREATE TABLE table () USING DELTA`.

Existing behavior of CTAS and CREATE external table at location are not affected.
Bug: `assert_true` fails unconditionnaly after `left_outer` joins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/22 08:27
Affects Version/s: 3.1.1

When `assert_true` is used after a `left_outer` join the assert exception is raised even though all the rows meet the condition. Using an `inner` join does not expose this issue.

 
{code:java}
from pyspark.sql import SparkSession
from pyspark.sql import functions as sf

session = SparkSession.builder.getOrCreate()

entries = session.createDataFrame(
    [
        ("a", 1),
        ("b", 2),
        ("c", 3),
    ],
    ["id", "outcome_id"],
)

outcomes = session.createDataFrame(
    [
        (1, 12),
        (2, 34),
        (3, 32),
    ],
    ["outcome_id", "outcome_value"],
)

# Inner join works as expected
(
    entries.join(outcomes, on="outcome_id", how="inner")
    .withColumn("valid", sf.assert_true(sf.col("outcome_value") > 10))
    .filter(sf.col("valid").isNull())
    .show()
)

# Left join fails with «'('outcome_value > 10)' is not true!» even though it is the case
(
    entries.join(outcomes, on="outcome_id", how="left_outer")
    .withColumn("valid", sf.assert_true(sf.col("outcome_value") > 10))
    .filter(sf.col("valid").isNull())
    .show()
){code}
Reproduced on `pyspark` versions: `3.2.1`, `3.2.0`, `3.1.2` and `3.1.1`. I am not sure if "native" Spark exposes this issue as well or not, I don't have the knowledge/setup to test that.
Bug: Output SQL statements in upper case in error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Apr/22 10:52
Affects Version/s: 3.3.0

Output SQL statements in upper case ONLY in error messages. For example, the error message:
{code:sql}
The operation "DESC PARTITION" is not allowed
{code}
should be replaced by:
{code:sql}
The operation DESC PARTITION is not allowed
{code}
Bug: Support sub-error-class for UNSUPPORTED_FEATURE et al
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/22 15:13
Affects Version/s: 3.3.0

Some error classes like "unsupported feature" or "unsupported operation" require a lot of variation n free text which cannot be parameterized easily.

There are two choices:
 * Add a distinct error-class for each situation, which mean a lot of error classes, some of which with a short lifespan
 * Add sub-error classes which can be used to pick the right message for each situation.

 

sub-error-classes need only be locally unique.
Bug: NaN != NaN in pivot
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/22 20:39
Affects Version/s: 3.2.0

I know this is an odd corner case, but NaN != NaN in pivot which is inconsistent with other places in Spark.

 

{code}

scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.groupBy("value").count.show()
+-----+-----+                                                                   
|value|count|
+-----+-----+
|  NaN|    3|
|  1.0|    3|
+-----+-----+


scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.groupBy("value").pivot("value").count.show()
+-----+----+----+
|value| 1.0| NaN|
+-----+----+----+
|  NaN|null|null|
|  1.0|   3|null|
+-----+----+----+

{code}

 

It looks like the issue is that in PivotFirst if the pivotColumn is an AtomicType a HashMap is used, but for other types a TreeMap is used with an interpretedOrdering. If we made DoubleType and FloatType use the TreeMap then the equality checks would be correct. But I am not able to really test it because if I try to pivot on an array or struct I get analysis exceptions.

 

{code}
scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.selectExpr("value", "struct(value) as ar_value").groupBy("value").pivot("ar_value").count.show()
java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema [1.0]
  at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:182)
  at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:101)

{code}

 
{code}
 scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.selectExpr("value", "array(value) as ar_value").groupBy("value").pivot("ar_value").count.show()
org.apache.spark.sql.AnalysisException: Invalid pivot value '[1.0]': value data type array<double> does not match pivot column data type array<double>
  at org.apache.spark.sql.errors.QueryCompilationErrors$.pivotValDataTypeMismatchError(QueryCompilationErrors.scala:85)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$10.$anonfun$applyOrElse$21(Analyzer.scala:762)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
{code}
Bug: Wrap SQL statements by double quotes in error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Apr/22 15:45
Affects Version/s: 3.3.0

Wrap SQL statements by double quotes "" in error messages. For example, the error message:

{code:sql}
The operation DESC PARTITION is not allowed
{code}

should be replaced by:

{code:sql}
The operation "DESC PARTITION" is not allowed
{code}
Bug: Change Java8 datetime support to configurable 
Status: Open | Priority: Major | Resolution: 
Created: 26/Apr/22 03:25
Affects Version/s: 3.2.1

*Background:*

In the Spark 3.1.0 there is an improvement introduced to enable Java 8 datetime API by default.

It stops user set this configuration, *spark.sql.datetime.java8API.enabled,* in command line or configuration file when using Spark SQL shell or Spark Thrift Server.

The only way to set it is in the SQL Session using SET command like:
{code:java}
spark-sql> SET spark.sql.datetime.java8API.enabled=false {code}
There are a few issues related to this improvement 
 * [https://github.com/apache/iceberg/issues/2530]
 * [https://github.com/delta-io/delta/issues/760]

There is a solution in 3.2.0 for it by having *LocalDateConverter* uses *DateConverter* and in the *DateConverter* it handles both *LocalDate* and *Date* types.

 

*Discussion:*

I think we should give back the ability to user to set this option in configuration file and command line. Both of the improvement defeats the reason why we have a java8API option configurable.

 

Please advise

Thanks
Bug: make AppHistoryServerPlugin public api for developer
Status: Open | Priority: Major | Resolution: 
Created: 23/Apr/22 09:43
Affects Version/s: 2.3.0

For history server, there is an interface called {{{}AppHistoryServerPlugin{}}}, which is loaded based on SPI. However it is accessible for spark package, so, can we change it as a public interface? With that, developer can extend application history.
Bug: Fix schema pruning with correlated subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/22 04:35
Affects Version/s: 3.2.0

Schema pruning fails for some queries with correlated subqueries.


{noformat}
sql(
  s"""SELECT name FROM contacts c
     |WHERE
     | EXISTS (SELECT 1 FROM ids i WHERE i.value = c.id)
     | AND
     | EXISTS (SELECT 1 FROM first_names n WHERE c.name.first = n.value)
     |""".stripMargin)
{noformat}

{noformat}
[info]   org.apache.spark.SparkException: Failed to merge fields 'value' and 'value'. Failed to merge incompatible data types int and string
[info]   at org.apache.spark.sql.errors.QueryExecutionErrors$.failedMergingFieldsError(QueryExecutionErrors.scala:936)
{noformat}
Bug: Call to MetricsSystem#getServletHandlers() may take place before MetricsSystem becomes running
Status: Open | Priority: Major | Resolution: 
Created: 22/Apr/22 16:18
Affects Version/s: 3.1.2

Sometimes the following exception is observed:
{code}
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
    at scala.Predef$.require(Predef.scala:281)
    at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:92)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:597)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
{code}
It seems somehow the MetricsSystem was stopped in between start() and getServletHandlers() calls.
SparkContext should check the MetricsSystem becomes running before calling getServletHandlers()
Bug: Prepend error class tag to error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/22 16:33
Affects Version/s: 3.4.0

Prepend error messages by error class tags by extending the following method in SparkThrowableHelper:

{code:scala}
  def getMessage(errorClass: String, messageParameters: Array[String]): String = {
    val errorInfo = errorClassToInfoMap.getOrElse(errorClass,
      throw new IllegalArgumentException(s"Cannot find error class '$errorClass'"))
    String.format(errorInfo.messageFormat.replaceAll("<[a-zA-Z0-9_-]+>", "%s"),
      messageParameters: _*)
  }
{code}
Bug: Improve error messages for cast failures in ANSI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/22 04:35
Affects Version/s: 3.3.0

Improve several error messages for cast failures in ANSI.
h2. Cast to numeric types
{code:java}
java.lang.NumberFormatException: invalid input syntax for type numeric: 1.0. To return NULL instead, use 'try_cast'. ...{code}
This is confusing as 1.0 is numeric to an average user. Need to mention the specific target type (integer in this case) and put 1.0 in single quotes. If we can mention this is a cast from string to an integer that’s even better.

*Proposed change*
{code:java}
Invalid `int` literal: '1.0'. To return NULL instead, use 'try_cast'.{code}
h2. Cast to date types
{code:java}
java.time.DateTimeException: Cannot cast 2021-09- 2 to DateType.{code}
Can align with the above change.

*Proposed change*
{code:java}
Invalid `date` literal: '2021-09- 2'. To return NULL instead, use 'try_cast'.{code}
Bug: TaskLocation.apply throw NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Apr/22 13:19
Affects Version/s: 3.2.1

 
{code:java}
//Caused by: java.lang.NullPointerException
    at scala.collection.immutable.StringLike$class.stripPrefix(StringLike.scala:155)
    at scala.collection.immutable.StringOps.stripPrefix(StringOps.scala:29)
    at org.apache.spark.scheduler.TaskLocation$.apply(TaskLocation.scala:71)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal {code}
Bug: Datasource v2 support report unique keys
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Apr/22 09:23
Affects Version/s: 3.4.0

Datasource v2 can be used to connect to some databases who support [*unique key*|https://en.wikipedia.org/wiki/Unique_key].

Spark catalyst optimizer support do further optimization through unique keys. So it can improve the performance if the Scan reports its unique keys to Spark.
Bug: Remove outer join if aggregate functions are duplicate agnostic on streamed side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/22 07:05
Affects Version/s: 3.4.0

If aggregate child is outer join, and the aggregate references are all coming from the streamed side and the aggregate functions are all duplicate agnostic, we can remve the outer join.

For example:
{code:java}
SELECT t1.c1, min(t1.c2) FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1 GROUP BY t1.c1
==>
SELECT t1.c1, min(t1.c2) FROM t1 GROUP BY t1.c1
{code}
Bug: Spark  Executors  status always is KILLED
Status: Open | Priority: Major | Resolution: 
Created: 18/Apr/22 08:50
Affects Version/s: 3.1.2

In  standalone deploy mode, try run spark org.apache.spark.examples.SparkPi or other spark program ,the ui always show Executors  status is killed

spark worker error log :

22/04/18 17:08:27 INFO Worker: Asked to kill executor app-20220418170822-0039/0
22/04/18 17:08:27 INFO ExecutorRunner: Runner thread for executor app-20220418170822-0039/0 interrupted
22/04/18 17:08:27 INFO ExecutorRunner: Killing process!
22/04/18 17:08:27 DEBUG SizeBasedRollingPolicy: 55 + 18896 > 1073741824
22/04/18 17:08:27 DEBUG RollingFileAppender: Closed file /opt/spark/work/app-20220418170822-0039/0/stderr
22/04/18 17:08:27 DEBUG RollingFileAppender: Closed file /opt/spark/work/app-20220418170822-0039/0/stdout
22/04/18 17:08:27 INFO ExecutorRunner: exitCode:Some(143)
22/04/18 17:08:27 INFO Worker: Executor app-20220418170822-0039/0 finished with state KILLED exitStatus 143

 

haved patch  [https://github.com/apache/spark/pull/12012]

run SparkPi commad:

/opt/app/applications/bd-spark/bin/run-example  -{-}class org.apache.spark.examples.SparkPi{-}  -master spark://10.205.90.120:7077,10.205.90.131:7077 --deploy-mode cluster --driver-java-options "-Dlog4j.configuration=[file:/opt/app/applications/bd-spark/conf/log4j.properties|file://opt/app/applications/bd-spark/conf/log4j.properties]" --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=[file:/opt/app/applications/bd-spark/conf/log4j.properties|file://opt/app/applications/bd-spark/conf/log4j.properties]"