av9ash/GitBugs huggingface dataset:
Bug: Single GPU is allocated multiple times
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 28/Jun/20 18:50
Affects Version/s: 3.0.0

I am running Spark in a {{local-cluster[2,1,1024]}} with one GPU per worker, task and executor, and two GPUs provided through a GPU discovery script. The same GPU is allocated to both executors.

Discovery script output:
{code:java}
{"name": "gpu", "addresses": ["0", "1"]}
{code}
Spark local cluster setup through {{spark-shell}}:
{code:java}
./spark-3.0.0-bin-hadoop2.7/bin/spark-shell --master "local-cluster[2,1,1024]" --conf spark.worker.resource.gpu.discoveryScript=/tmp/gpu.json --conf spark.worker.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1 --conf spark.executor.resource.gpu.amount=1
{code}
Executor page of this cluster:
 !screenshot-2.png!

You can see that both executors have the same GPU allocated: {{[1]}}

Code run in the Spark shell:
{code:java}
scala> import org.apache.spark.TaskContext
import org.apache.spark.TaskContext

scala> def fn(it: Iterator[java.lang.Long]): Iterator[(String, (String, Array[String]))] = { TaskContext.get().resources().mapValues(v => (v.name, v.addresses)).iterator }
fn: (it: Iterator[Long])Iterator[(String, (String, Array[String]))]

scala> spark.range(0,2,1,2).mapPartitions(fn).collect
res0: Array[(String, (String, Array[String]))] = Array((gpu,(gpu,Array(1))), (gpu,(gpu,Array(1))))
{code}
The result shows that each task got GPU {{1}}. The executor page shows that each task has been run on different executors (see above screenshot).

The expected behaviour would have been to have GPU {{0}} assigned to one executor and GPU {{1}} to the other executor. Consequently, each partition / task should then see a different GPU.

With Spark 3.0.0-preview2 the allocation was as expected (identical code and Spark shell setup):
{code:java}
res0: Array[(String, (String, Array[String]))] = Array((gpu,(gpu,Array(0))), (gpu,(gpu,Array(1))))
{code}
!screenshot-3.png!

Happy to contribute a patch if this is an accepted bug.
Bug: Failed to register SIGPWR handler on MacOS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 01:58
Affects Version/s: 3.1.0

 
{code:java}
20/06/09 22:54:54 WARN SignalUtils: Failed to register SIGPWR handler - disabling decommission feature.
java.lang.IllegalArgumentException: Unknown signal: PWR
	at sun.misc.Signal.<init>(Signal.java:143)
	at org.apache.spark.util.SignalUtils$.$anonfun$register$1(SignalUtils.scala:83)
	at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
	at org.apache.spark.util.SignalUtils$.register(SignalUtils.scala:81)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.onStart(CoarseGrainedExecutorBackend.scala:86)
	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:120)
	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:203)
	at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
	at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
	at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}
Seem like MacOS is *POSIX* compliant. But SIGPWR is not specified in the *POSIX* specification. See [https://en.wikipedia.org/wiki/Signal_(IPC)#SIGPWR]
Bug: -0.0 vs 0.0 is inconsistent
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/20 20:02
Affects Version/s: 3.0.0

This is related to SPARK-26021 where some things were fixed but there is still a lot that is not consistent.

When parsing SQL {{-0.0}} is turned into {{0.0}}. This can produce quick results that appear to be correct but are totally inconsistent for the same operators.
{code:java}
scala> import spark.implicits._
import spark.implicits._

scala> spark.sql("SELECT 0.0 = -0.0").collect
res0: Array[org.apache.spark.sql.Row] = Array([true])

scala> Seq((0.0, -0.0)).toDF("a", "b").selectExpr("a = b").collect
res1: Array[org.apache.spark.sql.Row] = Array([false])
{code}
This also shows up in sorts
{code:java}
scala> Seq((0.0, -100.0), (-0.0, 100.0), (0.0, 100.0), (-0.0, -100.0)).toDF("a", "b").orderBy("a", "b").collect
res2: Array[org.apache.spark.sql.Row] = Array([-0.0,-100.0], [-0.0,100.0], [0.0,-100.0], [0.0,100.0])
{code}
But not for a equi-join or for an aggregate
{code:java}
scala> Seq((0.0, -0.0)).toDF("a", "b").join(Seq((-0.0, 0.0)).toDF("r_a", "r_b"), $"a" === $"r_a").collect
res3: Array[org.apache.spark.sql.Row] = Array([0.0,-0.0,-0.0,0.0])

scala> Seq((0.0, 1.0), (-0.0, 1.0)).toDF("a", "b").groupBy("a").count.collect
res6: Array[org.apache.spark.sql.Row] = Array([0.0,2])
{code}
This can lead to some very odd results. Like an equi-join with a filter that logically should do nothing, but ends up filtering the result to nothing.
{code:java}
scala> Seq((0.0, -0.0)).toDF("a", "b").join(Seq((-0.0, 0.0)).toDF("r_a", "r_b"), $"a" === $"r_a" && $"a" <= $"r_a").collect
res8: Array[org.apache.spark.sql.Row] = Array()

scala> Seq((0.0, -0.0)).toDF("a", "b").join(Seq((-0.0, 0.0)).toDF("r_a", "r_b"), $"a" === $"r_a").collect
res9: Array[org.apache.spark.sql.Row] = Array([0.0,-0.0,-0.0,0.0])
{code}
Hive never normalizes -0.0 to 0.0 so this results in non-ieee complaint behavior everywhere, but at least it is consistently odd.

MySQL, Oracle, Postgres, and SQLite all appear to normalize the {{-0.0}} to {{0.0}}.

The root cause of this appears to be that the java implementation of {{Double.compare}} and {{Float.compare}} for open JDK places {{-0.0}} < {{0.0}}.

This is not documented in the java docs but it is clearly documented in the code, so it is not a "bug" that java is going to fix.

[https://github.com/openjdk/jdk/blob/a0a0539b0d3f9b6809c9759e697bfafd7b138ec1/src/java.base/share/classes/java/lang/Double.java#L1022-L1035]

It is also consistent with what is in the java docs for {{Double.equals}}
 [https://docs.oracle.com/javase/8/docs/api/java/lang/Double.html#equals-java.lang.Object-]

To be clear I am filing this mostly to document the current state rather than to think it needs to be fixed ASAP. It is a rare corner case, but ended up being really frustrating for me to debug what was happening.
Bug: Keep TimestampType when taking an average of a Timestamp
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jun/20 09:48
Affects Version/s: 3.0.0

Currently, when you take an average of a Timestamp, you'll end up with a Double, representing the seconds since epoch. This is because of old Hive behavior. I strongly believe that it is better to return a Timestamp.

root@8c4241b617ec:/# psql postgres postgres
psql (12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.

postgres=# CREATE TABLE timestamp_demo (ts TIMESTAMP);
CREATE TABLE
postgres=# INSERT INTO timestamp_demo VALUES('2019-01-01 18:22:11');
INSERT 0 1
postgres=# INSERT INTO timestamp_demo VALUES('2018-01-01 18:22:11');
INSERT 0 1
postgres=# INSERT INTO timestamp_demo VALUES('2017-01-01 18:22:11');
INSERT 0 1
postgres=# SELECT AVG(ts) FROM timestamp_demo;
ERROR: function avg(timestamp without time zone) does not exist
LINE 1: SELECT AVG(ts) FROM timestamp_demo;

 


root@bab43a5731e8:/# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.20 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE TABLE timestamp_demo (ts TIMESTAMP);
Query OK, 0 rows affected (0.05 sec)

mysql> INSERT INTO timestamp_demo VALUES('2019-01-01 18:22:11');
Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO timestamp_demo VALUES('2018-01-01 18:22:11');
Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO timestamp_demo VALUES('2017-01-01 18:22:11');
Query OK, 1 row affected (0.01 sec)

mysql> SELECT AVG(ts) FROM timestamp_demo;
+---------------------+
| AVG(ts) |
+---------------------+
| 20180101182211.0000 |
+---------------------+
1 row in set (0.00 sec)
Bug: spark error while select nest data
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Jun/20 13:12
Affects Version/s: 2.4.4

nest-data.json
{code:java}
{"a": [{"b": [{"c": [1,2]}]}]}
{"a": [{"b": [{"c": [1]}, {"c": [2]}]}]}{code}
{code:java}
val df: DataFrame = spark.read.json(testFile("nest-data.json"))
df.createTempView("nest_table")
sql("select a.b.c from nest_table").show()

{code}
{color:#ff0000}org.apache.spark.sql.AnalysisException: cannot resolve 'nest_table.`a`.`b`['c']' due to data type mismatch: argument 2 requires integral type, however, ''c'' is of string type.; line 1 pos 7;{color}
 {color:#ff0000}'Project [a#6.b[c] AS c#8|#6.b[c] AS c#8]{color}
 {color:#ff0000}+- SubqueryAlias `nest_table`{color}
 {color:#ff0000} +- Relation[a#6|#6] json{color}

{color:#172b4d}Analyse the causes, a.b Expression dataType match extractor for c, but a.b extractor is GetArrayStructFields, ArrayType(ArrayType()) match {color}GetArrayItem, extraction ("c") treat as an ordinal.

org.apache.spark.sql.catalyst.expressions.ExtractValue
{code:java}
def apply(
      child: Expression,
      extraction: Expression,
      resolver: Resolver): Expression = {    
   (child.dataType, extraction) match {
      case (StructType(fields), NonNullLiteral(v, StringType)) =>
        val fieldName = v.toString
        val ordinal = findField(fields, fieldName, resolver)
        GetStructField(child, ordinal, Some(fieldName))      
      case (ArrayType(StructType(fields), containsNull), NonNullLiteral(v, StringType)) =>
        val fieldName = v.toString
        val ordinal = findField(fields, fieldName, resolver)
        GetArrayStructFields(child, fields(ordinal).copy(name = fieldName),
          ordinal, fields.length, containsNull)      
      case (_: ArrayType, _) => GetArrayItem(child, extraction)      
      case (MapType(kt, _, _), _) => GetMapValue(child, extraction)      
      case (otherType, _) =>
        val errorMsg = otherType match {
          case StructType(_) =>
            s"Field name should be String Literal, but it's $extraction"
          case other =>
            s"Can't extract value from $child: need struct type but got ${other.catalogString}"
        }
        throw new AnalysisException(errorMsg)
    }
  }{code}
Bug: Spark native temporary table
Status: Open | Priority: Major | Resolution: 
Created: 23/Jun/20 02:28
Affects Version/s: 3.1.0

Many databases and data warehouse SQL engines support temporary tables. A temporary table, as its named implied, is a short-lived table that its life will be only for current session.

In Spark, there is no temporary table. the DDL “CREATE TEMPORARY TABLE AS SELECT” will create a temporary view. A temporary view is totally different with a temporary table. 

A temporary view is just a VIEW. It doesn’t materialize data in storage. So it has below shortage:
 # View will not give improved performance. Materialize intermediate data in temporary tables for a complex query will accurate queries, especially in an ETL pipeline.
 # View which calls other views can cause severe performance issues. Even, executing a very complex view may fail in Spark. 
 # Temporary view has no database namespace. In some complex ETL pipelines or data warehouse applications, without database prefix is not convenient. It needs some tables which only used in current session.

 

More details are described in [Design Docs|https://docs.google.com/document/d/1RS4Q3VbxlZ_Yy0fdWgTJ-k0QxFd1dToCqpLAYvIJ34U/edit?usp=sharing]
Bug: UserDefinedType.equal() does not have symmetry 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/20 12:57
Affects Version/s: 2.1.1

ExampleSubTypeUDT.userClass is a subclass of ExampleBaseTypeUDT.userClass
val udt1 = new ExampleBaseTypeUDT
val udt2 = new ExampleSubTypeUDT
println(udt1 == udt2) // true
println(udt2 == udt1) // false
Bug: Improve heurestic for selecting nodes for scale down to take into account graceful decommission cost
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Jun/20 23:35
Affects Version/s: 3.1.0

Once SPARK-31198 is in we should see if we can come up with a better graceful decommissioning aware heuristic to use for selecting nodes to scale down.
Bug: Use unique ConfigMap name for executor pod template
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/20 03:12
Affects Version/s: 3.0.1

THE BUG:

The bug is reproducible by spark-submit two different apps (app1 and app2) with different executor pod templates (e.g., different labels) to K8s sequentially,  with app2 launching while app1 is still in the middle of ramping up all its executor pods. The unwanted result is that some launched executor pods of app1 end up having app2's executor pod template applied to them.

The root cause appears to be that app1's podspec-configmap got overwritten by app2 during the overlapping launching periods because both apps use the same ConfigMap (name). This causes some app1's executor pods being ramped up after app2 is launched to be inadvertently launched with the app2's pod template. The issue can be seen as follows:

First, after submitting app1, you get these configmaps:
{code:java}
NAMESPACE    NAME                                       DATA    AGE
default      app1-1111111111111111-driver-conf-map      1       9m46s
default      podspec-configmap                          1       12m{code}
Then submit app2 while app1 is still ramping up its executors. The podspec-confimap is modified by app2.
{code:java}
NAMESPACE    NAME                                       DATA    AGE
default      app1-1111111111111111-driver-conf-map      1       11m43s
default      app2-2222222222222222-driver-conf-map      1       10s
default      podspec-configmap                          1       13m57s{code}
 

PROPOSED SOLUTION:

Properly prefix the podspec-configmap for each submitted app, ideally the same way as the driver configmap:
{code:java}
NAMESPACE    NAME                                       DATA    AGE
default      app1-1111111111111111-driver-conf-map      1       11m43s
default      app1-1111111111111111-podspec-configmap    1       13m57s
default      app2-2222222222222222-driver-conf-map      1       10s 
default      app2-2222222222222222-podspec-configmap    1       3m{code}
Bug: Create remote shuffle service reference implementation
Status: Open | Priority: Major | Resolution: 
Created: 07/Jun/20 06:53
Affects Version/s: 3.1.0

People in [Spark Scalability & Reliability Sync Meeting |https://docs.google.com/document/d/1T3y25dOaKWVO0pWd838GeiTeI3DUQJtwy6MKYPLuleg]have discussed a lot about remote (disaggregated) shuffle service, and plan to do a reference implementation to help demonstrate some basic design and pave the way for a future production grade remote  shuffle service.

 

There are already two pull requests to enhance Spark shuffle metadata API to make it easy/possible to implement remote shuffle service ([PR 28616|https://github.com/apache/spark/pull/28616], [PR 28618|https://github.com/apache/spark/pull/28618]). Creating a remote shuffle service reference implementation will help to validate those shuffle metadata API.
Bug: DAG-viz is not rendered correctly with pagination.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/20 04:12
Affects Version/s: 2.4.4

Because DAG-viz for a job fetches link urls for each stage from the stage table, rendering can fail with pagination.

You can reproduce this issue with the following operation.
{code:java}
 sc.parallelize(1 to 10).map(value => (value ,value)).repartition(1).repartition(1).repartition(1).reduceByKey(_ + _).collect{code}
And then, visit the corresponding job page.
There are 5 stages so show <5 stages in the paged table.
Bug: Improve sorting performance for Spark SQL rank window function​
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Jun/20 21:12
Affects Version/s: 3.0.0

Spark SQL rank window function needs to sort the data in each window partition, and it relies on the execution operator[ |https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsqlhelsinki.visualstudio.com%2Foss%2F_git%2Fspark%3Fpath%3D%252Fsql%252Fcore%252Fsrc%252Fmain%252Fscala%252Forg%252Fapache%252Fspark%252Fsql%252Fexecution%252FSortExec.scala%26version%3DGBsql-2.4%26line%3D37%26lineEnd%3D38%26lineStartColumn%3D1%26lineEndColumn%3D1%26lineStyle%3Dplain&data=02%7C01%7Czixu%40microsoft.com%7Cdc51f9940fc64981c8bd08d7f05ef7c0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637242163078452885&sdata=HGPm4TbMeJLp9wS0YZmIyqyE4%2BS4Ylw7lebFztX8PWc%3D&reserved=0] [*_SortExec_* |https://sqlhelsinki.visualstudio.com/oss/_git/spark?path=%2Fsql%2Fcore%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2FSortExec.scala&version=GBsql-2.4&line=37&lineEnd=43&lineStartColumn=1&lineEndColumn=1&lineStyle=plain]to do the sort. During sorting, the window partition key is also put at the front of the sort order and thus it brings unnecessary comparisons on the partition key. Instead, we can group the rows by partition key first, and inside each group we sort the rows without comparing the partition key.​ 

 

The Jira https://issues.apache.org/jira/browse/SPARK-32947 is a follow-up effort of this improvement.
Bug: Support UTF8String literals
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 21/Jun/20 07:00
Affects Version/s: 3.1.0

How to reproduce this issue:
{code:scala}
val min = spark.sql("select min(concat('a', id)) from range(10)").queryExecution.executedPlan.executeCollect().head
assert(Literal(min.get(0, StringType)) === Literal("a0"))
{code}
{noformat}
Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a0
java.lang.RuntimeException: Unsupported literal type class org.apache.spark.unsafe.types.UTF8String a0
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:89)
{noformat}
Bug: Fix UnsafeRow set overflowed decimal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jun/20 23:59
Affects Version/s: 2.4.6

There is a bug that writing an overflowed decimal into UnsafeRow is fine but reading it out will throw ArithmeticException. This exception is thrown when calling {{getDecimal}} in UnsafeRow with input decimal's precision greater than the input precision. Setting the value of the overflowed decimal to null when writing into UnsafeRow should fix this issue.
Bug: Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 15:56
Affects Version/s: 2.4.6

A customer's cluster has a node that goes down while a Spark application is running. (They are running Spark on YARN with the external shuffle service enabled.) An executor is lost (apparently the only one running on the node). This executor lost event is handled in the DAGScheduler, which removes the executor from its BlockManagerMaster. At this point, there is no unregistering of shuffle files for the executor or the node. Soon after, tasks trying to fetch shuffle files output by that executor fail with FetchFailed (because the node is down, there is no NodeManager available to serve shuffle files). By right, such fetch failures should cause the shuffle files for the executor to be unregistered, but they do not.

Due to task failure, the stage is re-attempted. Tasks continue to fail due to fetch failure form the lost executor's shuffle output. This time, since the failed epoch for the executor is higher, the executor is removed again (this doesn't really do anything, the executor was already removed when it was lost) and this time the shuffle output is unregistered.

So it takes two stage attempts instead of one to clear the shuffle output. We get 4 attempts by default. The customer was unlucky and two nodes went down during the stage, i.e., the same problem happened twice. So they used up 4 stage attempts and the stage failed and thus the job.
Bug: Introduce UnsafeRow format validation for streaming state store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/20 18:22
Affects Version/s: 3.1.0

Currently, Structured Streaming directly puts the UnsafeRow into StateStore without any schema validation. It's a dangerous behavior when users reusing the checkpoint file during migration. Any changes or bug fix related to the aggregate function may cause random exceptions, even the wrong answer, e.g SPARK-28067.

Here we introduce an UnsafeRow format validation for the state store.
Bug: Unify Classification Training Summary
Status: Open | Priority: Major | Resolution: 
Created: 30/Jun/20 20:33
Affects Version/s: 3.1.0

add classification model summary and training summary to each of the classification algorithms. The classification model summary basically gives a summary for all the classification algorithms evaluation metrics such as accuracy/precision/recall. The training summary describes information about model training iterations and the objective function (scaled loss + regularization) at each iteration. These are very useful information for users.
Bug: Generated code in 'concat_ws' fails to compile when splitting method is in effect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/20 06:40
Affects Version/s: 2.3.4

https://github.com/apache/spark/blob/a0187cd6b59a6b6bb2cadc6711bb663d4d35a844/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L88-L195

There're three parts of generated code in concat_ws (codes, varargCounts, varargBuilds) and all parts try to split method by itself, while `varargCounts` and `varargBuilds` refer on the generated code in `codes`, hence the overall generated code fails to compile if any of part succeeds to split.
Bug: Remote inheritable thread local variables after spark context is stopped
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 17/Jun/20 11:58
Affects Version/s: 3.0.0

There is still a reference(eg. `java.lang.ApplicationShutdownHooks` -> `org.apache.hive.common.util.ShutdownHookManager$1` -> `SparkContext`) to `sparkContext` after the spark context is stopped, so the JVM cannot collect that.

If we start and stop sparkSession several times, there will be a OOM exception.

 

!https://img-blog.csdnimg.cn/2020051111561087.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dhbmt1bmRl,size_16,color_FFFFFF,t_70!
Bug: Remove the incomplete code path on aggregation for continuous mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/20 07:33
Affects Version/s: 3.1.0

During work I realized we left some incomplete code parts on supporting aggregation for continuous mode. (shuffle & coalesce)
 
The work had been occurred around first half of 2018 and stopped, and no work has been done for around 2 years, so I don't expect anyone is working on this.
 
The functionality is undocumented (as the work was only done partially) and continuous mode is experimental so I don't feel risks to get rid of the part.