av9ash/GitBugs huggingface dataset:
Bug: Use SparkSubmitTestUtils to core and use it in SparkSubmitSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Sep/21 04:27
Affects Version/s: 3.0.0

{{SparkSubmitSuite}} test failures can be painful to debug: currently, failed tests result in messages like
{code:java}
org.scalatest.exceptions.TestFailedException: Process returned with exit code 1. See the log4j logs for more detail.
{code}
which require the developer to hunt in test logs in order to identify the root cause of the test failure.

In contrast, {{HiveSparkSubmitSuite}}'s output packs lots more useful information into its test failure messages:
{code:java}
[info] - temporary Hive UDF: define a UDF and use it *** FAILED *** (4 seconds, 135 milliseconds)
[info]   spark-submit returned with exit code 101.
[info]   Command line: '/Users/joshrosen/oss-spark/bin/spark-submit' '--class' 'wrongClassName' '--name' 'TemporaryHiveUDFTest' '--master' 'local-cluster[2,1,1024]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--driver-java-options' '-Dderby.system.durability=test' '--jars' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-32d0a47c-33eb-4488-866a-9994b3727b5b/testJar-1631766233448.jar,file:/Users/joshrosen/oss-spark/target/tmp/spark-e9e32588-83fa-43bd-a60f-9538fd30ab9a/testJar-1631766233601.jar' 'file:/Users/joshrosen/oss-spark/target/tmp/spark-daef4628-d953-41e3-b935-a726a7796418/testJar-1631766232701.jar' 'SparkSubmitClassA' 'SparkSubmitClassB'
[info]
[info]   2021-09-15 21:23:54.752 - stderr> SLF4J: Class path contains multiple SLF4J bindings.
[info]   2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/oss-spark/assembly/target/scala-2.12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info]   2021-09-15 21:23:54.752 - stderr> SLF4J: Found binding in [jar:file:/Users/joshrosen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
[info]   2021-09-15 21:23:54.752 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
[info]   2021-09-15 21:23:54.753 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[info]   2021-09-15 21:23:55.623 - stderr> Error: Failed to load class wrongClassName. (SparkSubmitTestUtils.scala:97)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[...]
{code}

This is done via code in {{SparkSubmitTestUtils}}, which is currently located in the {{sql/hive}} module. I propose a refactoring to move this into the {{core}} module so that it can be used by both {{SparkSubmitSuite}} and {{HiveSparkSubmitSuite}}, enriching the former with nicer error outputs.
Bug: ExtractEquiJoinKeys should return the original predicates on join keys 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Sep/21 18:54
Affects Version/s: 3.1.2

The join condition returned from ExtractEquiJoinKeys does not correspond to the equi-join on the extracted left and right keys. A call site can be risky if it is not aware of that.

The pattern extractor should extract the rest of original join condition as well.
Bug: Spark uncaught exception handler is using logError
Status: Open | Priority: Minor | Resolution: 
Created: 14/Sep/21 15:14
Affects Version/s: 3.0.0

Spark is [setting up an handler|https://github.com/apache/spark/blob/v3.0.1/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L915] to catch any uncaught exception.

This [handler itself catch any subsequent exception|https://github.com/apache/spark/blob/v3.0.1/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala#L64] that can happen while reporting the initially uncaught exception.

Issue is that if the subsequent exception is due to a log4j issue, as the {{catch}} there is also using {{logError}}, it will loop, not display any exception in the logs, and exiting with code 51.

e.g. We got an log4j issue when using logstash JSON logging :

{noformat}
at net.logstash.log4j.JSONEventLayoutV1.format(JSONEventLayoutV1.java:137)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
at org.apache.spark.internal.Logging.logError(Logging.scala:94)
at org.apache.spark.internal.Logging.logError$(Logging.scala:93)
at org.apache.spark.util.SparkUncaughtExceptionHandler.logError(SparkUncaughtExceptionHandler.scala:28)
at org.apache.spark.util.SparkUncaughtExceptionHandler.uncaughtException(SparkUncaughtExceptionHandler.scala:37)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
{noformat}

*Suggested fix:*

Directly using {{println}} and {{printStackTrace}} as safe fallback in this {{catch}}.
Bug: Published 2.13 POM lists `scala-parallel-collections` only in `scala-2.13` profile
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Sep/21 07:14
Affects Version/s: 3.3.0

As [reported on dev@spark.apache.org|https://lists.apache.org/thread.html/r84cff66217de438f1389899e6d6891b573780159cd45463acf3657aa%40%3Cdev.spark.apache.org%3E], the published POMs when building with Scala 2.13 have the scala-parallel-collections dependency only in the scala-2.13 profile of the pom.

This can cause issues for developers that write unit tests for their applications.
Bug: Use java.util.Objects API instead of Guava API
Status: In Progress | Priority: Minor | Resolution: 
Created: 14/Sep/21 06:07
Affects Version/s: 3.3.0

Java 8 provides the java.util.Objects, we can use it to replace some guava API usages with the same semantics.

 
 * Preconditions.checkNotNull -> j.u.Objects.requireNonNull
 * Objects.hashCode -> j.u.Objects.hash
 * Objects.equal -> j.u.Objects.equals
 *
Bug: Upgrade Netty from 4.1.63 to 4.1.68
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Sep/21 14:11
Affects Version/s: 3.3.0

Recently Netty 4.1.68 was released, which includes official M1 Mac support.
https://github.com/netty/netty/pull/11666

4.1.65 also includes a critical bug fix which Spark might be affected.
https://github.com/netty/netty/issues/11209
Bug: Ensure HiveThriftServer2Suites to stop Thrift JDBC server on exit
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Sep/21 19:17
Affects Version/s: 3.0.3

HiveThriftServer2Suites stops Thrift JDBC server via afterAll method.
But, if they are killed by signal (e.g. Ctrl-C), Thrift JDBC server will be remain.
{code}
$ jps
2792969 SparkSubmit
{code}
Bug: Allow expand 'quotedRegexColumnNames' in all expressions when it’s expanded to one named expression
Status: In Progress | Priority: Minor | Resolution: 
Created: 08/Sep/21 22:51
Affects Version/s: 3.3.0

With
{code:java}
spark.sql.parser.quotedRegexColumnNames=true{code}
regular expressions are not allowed in expressions like
{code:java}
SELECT `col_.*`/exp FROM (SELECT 3 AS col_a, 1 as exp){code}
 I propose to improve this behavior and allow this regular expression when it expands to only one named expression. It’s the same behavior in Hive.

*Example 1:* 
{code:java}
SELECT `col_.*`/exp FROM (SELECT 3 AS col_a, 1 as exp){code}
 col_.* expands to col_a: OK

*Example 2:*
{code:java}
SELECT `col_.*`/col_b FROM (SELECT 3 AS col_a, 1 as col_b){code}
col_.* expands to (col_a, col_b): Fail like now

 

Related with SPARK-36488
Bug: Support ScriptTransformation for timestamp_ntz
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Sep/21 13:04
Affects Version/s: 3.3.0

In the current master, ScriptTransformation doesn't work with timestamp_ntz values.
{code}
spark.sql("SELECT transform(col1) USING 'cat' AS (col1 timestamp_ntz) FROM VALUES timestamp_ntz'2021-09-06 20:19:13' t").show(false)
21/09/06 22:03:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: SparkScriptTransformation without serde does not support TimestampNTZType$ as output data type
	at org.apache.spark.sql.errors.QueryExecutionErrors$.outputDataTypeUnsupportedByNodeWithoutSerdeError(QueryExecutionErrors.scala:1740)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec.$anonfun$outputFieldWriters$1(BaseScriptTransformationExec.scala:245)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec.org$apache$spark$sql$execution$BaseScriptTransformationExec$$outputFieldWriters(BaseScriptTransformationExec.scala:194)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec.org$apache$spark$sql$execution$BaseScriptTransformationExec$$outputFieldWriters$(BaseScriptTransformationExec.scala:194)
	at org.apache.spark.sql.execution.SparkScriptTransformationExec.org$apache$spark$sql$execution$BaseScriptTransformationExec$$outputFieldWriters$lzycompute(SparkScriptTransformationExec.scala:38)
	at org.apache.spark.sql.execution.SparkScriptTransformationExec.org$apache$spark$sql$execution$BaseScriptTransformationExec$$outputFieldWriters(SparkScriptTransformationExec.scala:38)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec$$anon$1.$anonfun$processRowWithoutSerde$1(BaseScriptTransformationExec.scala:121)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec$$anon$1.next(BaseScriptTransformationExec.scala:162)
	at org.apache.spark.sql.execution.BaseScriptTransformationExec$$anon$1.next(BaseScriptTransformationExec.scala:113)
{code}
Bug: Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Sep/21 16:45
Affects Version/s: 3.1.2

Right now, by default sql.legacy.setCommandRejectsSparkCoreConfs is set as true in Spark 3.* versions int order to avoid changing Spark Confs. But from the error message we get confused if we can not modify/change Spark conf in Spark 3.* or not.

Current Error Message :
{code:java}
Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.driver.host
 at org.apache.spark.sql.RuntimeConfig.requireNonStaticConf(RuntimeConfig.scala:156)
 at org.apache.spark.sql.RuntimeConfig.set(RuntimeConfig.scala:40){code}
 

So adding little more information( how to modify Spark Conf), in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is 'true', will be helpful to avoid confusions.
Bug: Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Sep/21 03:00
Affects Version/s: 3.2.0

spark.sql.execution.topKSortFallbackThreshold now is an internal config hidden from users Integer.MAX_VALUE - 15 as its default. In many real-world cases, if the K is very big,  there would be performance issues.
Bug: Expose executionId to QueryExecutionListener
Status: In Progress | Priority: Minor | Resolution: 
Created: 03/Sep/21 01:57
Affects Version/s: 3.1.2

Now in [QueryExecutionListener|https://github.com/apache/spark/blob/v3.2.0-rc2/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L38] we have exposed API to get the query execution information:

def onSuccess(funcName: String, qe: QueryExecution, durationNs: Long): Unit

def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit

 

But we can not get a clear information that which query is this. In Spark SQL, I think that executionId is the direct identifier of a query execution. So I think it make sense to expose executionId to the QueryExecutionListener, so that people can easily find the exact query in UI or history server to track more information of the query execution. And there is no easy way we can find the relevant executionId from a QueryExecution object.
Bug: Rename error classes with _ERROR suffix
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Sep/21 20:55
Affects Version/s: 3.2.1

Clean up error classes with the redundant _ERROR suffix to reduce clutter, such as [CONCURRENT_QUERY_ERROR|https://github.com/apache/spark/blob/f78d8394dcf19891141e353ea3b6a76020faf844/core/src/main/resources/error/error-classes.json#L6].
Bug: Fix wrong assert messages
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 07/Sep/21 17:30
Affects Version/s: 2.4.8

{code:scala}
require(numCols == mat.numCols, "The number of rows of the matrices in this sequence, " + "don't match!")
{code}
Shall the error message be "The number of columns..."?

This issue also appears in the open source spark:
 [https://github.com/apache/spark/blob/master/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala#L1266]
Bug: The  create table as select statement verifies the valid column name
Status: Open | Priority: Trivial | Resolution: 
Created: 19/Sep/21 09:25
Affects Version/s: 3.1.0

If the column name output by select is not a valid column name, the prompt is not very clear, it is recommended to add a column name check
{code:java}
create table tdl_demo_dd as select 1+1{code}
!image-2021-09-19-17-25-02-239.png!
Bug: Decommissioning executors get killed before transferring their data because of the hardcoded timeout of 60 secs
Status: Open | Priority: Trivial | Resolution: 
Created: 28/Sep/21 02:41
Affects Version/s: 3.1.1

During the graceful decommissioning phase, executors need to transfer all of their shuffle and cache data to the peer executors. However, they get killed before transferring all the data because of the hardcoded timeout value of 60 secs in the decommissioning script. As a result of executors dying prematurely, the spark tasks on other executors fail which causes application failures, and it is hard to debug those failures. To fix the issue, we ended up writing a custom script with a different timeout and rebuilt the spark image but we would prefer an easier solution that does not require rebuilding the image.
Bug: Document change for Spark sql jdbc
Status: In Progress | Priority: Trivial | Resolution: 
Created: 19/Sep/21 18:19
Affects Version/s: 3.0.0

Reading using Spark SQL jdbc DataSource  does not maintain nullable type and changes "non nullable" columns to "nullable".

 

For example:

mysql> CREATE TABLE Persons(Id int NOT NULL, FirstName varchar(255), LastName varchar(255), Age int);
Query OK, 0 rows affected (0.04 sec)

mysql> show tables;
+-------------------+
| Tables_in_test_db |
+-------------------+
| Persons |
+-------------------+
1 row in set (0.00 sec)

mysql> desc Persons;
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| Id | int | NO | | NULL | |
| FirstName | varchar(255) | YES | | NULL | |
| LastName | varchar(255) | YES | | NULL | |
| Age | int | YES | | NULL | |
+-----------+--------------+------+-----+---------+-------+

 

 

{color:#cc7832}val {color}df = spark.read.format({color:#6a8759}"jdbc"{color})
 .option({color:#6a8759}"database"{color}{color:#cc7832},{color}{color:#6a8759}"Test_DB"{color})
 .option({color:#6a8759}"user"{color}{color:#cc7832}, {color}{color:#6a8759}"root"{color})
 .option({color:#6a8759}"password"{color}{color:#cc7832}, {color}{color:#6a8759}""{color})
 .option({color:#6a8759}"driver"{color}{color:#cc7832}, {color}{color:#6a8759}"com.mysql.cj.jdbc.Driver"{color})
 .option({color:#6a8759}"url"{color}{color:#cc7832}, {color}{color:#6a8759}"jdbc:mysql://localhost:3306/Test_DB"{color})
 .option({color:#6a8759}"query"{color}{color:#cc7832}, {color}{color:#6a8759}"(select * from Persons)"{color})
 .load()
df.printSchema()

 

*output:*

 

root
 |-- Id: integer (nullable = true)
 |-- FirstName: string (nullable = true)
 |-- LastName: string (nullable = true)
 |-- Age: integer (nullable = true)

 

 

So we need to add a note, in Documentation[1], "All columns are automatically converted to be nullable for compatibility reasons."

 Ref:

[1 ][https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html#jdbc-to-other-databases]
Bug: Fix lambda variable name issues in nested DataFrame functions in Python APIs
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/May/21 04:11
Affects Version/s: 3.1.1

Python side also has the same issue as SPARK-34794

{code}
from pyspark.sql.functions import *
df = sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
df.select(
    transform(
        "numbers",
        lambda n: transform("letters", lambda l: struct(n.alias("n"), l.alias("l")))
    )
).show()
{code}

{code}
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|transform(numbers, lambdafunction(transform(letters, lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l), namedlambdavariable())), namedlambdavariable()))|
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                                                                [[{a, a}, {b, b},...|
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
{code}
Bug: Fix lambda variable name issues in nested DataFrame functions in R APIs
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/May/21 04:03
Affects Version/s: 3.1.1

R's higher order functions also have the same problem with SPARK-34794:

{code}
df <- sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as letters")
collect(select(
  df,
  array_transform("numbers", function(number) {
    array_transform("letters", function(latter) {
      struct(alias(number, "n"), alias(latter, "l"))
    })
  })
))
{code}

{code}
  transform(numbers, lambdafunction(transform(letters, lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l), namedlambdavariable())), namedlambdavariable()))
1                                                                                                                                 a, a, b, b, c, c, a, a, b, b, c, c, a, a, b, b, c, c
{code}
Bug: Support subexpression elimination for non-common branches of conditional expressions
Status: In Progress | Priority: Major | Resolution: 
Created: 29/May/21 21:34
Affects Version/s: 3.1.1

https://issues.apache.org/jira/browse/SPARK-33337 added support for pulling subexpressions out of branches of conditional expressions for expressions present in all branches. We should be able to take this a step further and pull out subexpressions for any branch, as long as that expression will definitely be evaluated at least once.

Consider a common data validation example:
{code:java}
from pyspark.sql.functions import *
df = spark.createDataFrame([['word'], ['1234']])
col = regexp_replace('_1', r'\d', '')
df = df.withColumn('numbers_removed', when(length(col) > 0, col)){code}
We only want to keep the value if it's non-empty with numbers removed, otherwise we want it to be null. 

Because we have no otherwise value, `col` is not a candidate for subexpression elimination (you can see two regular expression replacements in the codegen). But whenever the length is greater than 0, we will have to execute the regular expression replacement twice. Since we know we will always calculate `col` at least once, it makes sense to consider that as a subexpression since we might need it again in the branch value. So we can update the logic from:

Create a subexpression if an expression will always be evaluated at least twice

To:

Create a subexpression if an expression will always be evaluated at least once AND will either always or conditionally be evaluated at least twice.

The trade off is potentially another subexpression function call (for split subexpressions) if the second evaluation doesn't happen, but this seems like it would be worth it for when it is evaluated the second time.
Bug: Use expressions to filter Hive partitions at client side
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/21 15:33
Affects Version/s: 3.1.1

When we have a table with a lot of partitions and there is no way to filter it on the MetaStore Server, we will get all the partition details and filter it on the client side. This is slow and puts a lot of pressure on the MetaStore Server.
We can first pull all the partition names, filter by expressions, and then obtain detailed information about the corresponding partitions from the MetaStore Server.
Bug: Can not insert into hive bucket table if create table with upper case schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/21 07:39
Affects Version/s: 3.0.0

 

 

create table TEST1(
 V1 BIGINT,
 S1 INT)
 partitioned by (PK BIGINT)
 clustered by (V1)
 sorted by (S1)
 into 200 buckets
 STORED AS PARQUET;

 

insert into test1
 select
 * from values(1,1,1);

 

 

org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns V1 is not part of the table columns ([FieldSchema(name:v1, type:bigint, comment:null), FieldSchema(name:s1, type:int, comment:null)]
org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Bucket columns V1 is not part of the table columns ([FieldSchema(name:v1, type:bigint, comment:null), FieldSchema(name:s1, type:int, comment:null)]
Bug: [SQL] Window operations with over Int.MaxValue + 1 rows can silently drop rows
Status: Open | Priority: Major | Resolution: 
Created: 29/May/21 15:53
Affects Version/s: 3.0.2

I think this impacts a lot more versions of Spark, but I don't know for sure because it takes a long time to test. As a part of doing corner case validation testing for spark rapids I found that if a window function has more than {{Int.MaxValue + 1}} rows the result is silently truncated to that many rows. I have only tested this on 3.0.2 with {{row_number}}, but I suspect it will impact others as well. This is a really rare corner case, but because it is silent data corruption I personally think it is quite serious.
{code:scala}
import org.apache.spark.sql.expressions.Window

val windowSpec = Window.partitionBy("a").orderBy("b")

val df = spark.range(Int.MaxValue.toLong + 100).selectExpr(s"1 as a", "id as b")

spark.time(df.select(col("a"), col("b"), row_number().over(windowSpec).alias("rn")).orderBy(desc("a"), desc("b")).select((col("rn") < 0).alias("dir")).groupBy("dir").count.show(20))

+-----+----------+                                                              
|  dir|     count|
+-----+----------+
|false|2147483647|
| true|         1|
+-----+----------+

Time taken: 1139089 ms

Int.MaxValue.toLong + 100
res15: Long = 2147483747

2147483647L + 1
res16: Long = 2147483648
{code}
I had to make sure that I ran the above with at least 64GiB of heap for the executor (I did it in local mode and it worked, but took forever to run)