av9ash/GitBugs huggingface dataset:
Bug: Spark web ui kill application but the thread still exists background
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/20 02:49
Affects Version/s: 2.4.3

When I kill an application on the Web UI (which I submit with standalone-client mode), it seems to be killed already; But when I use 'jps' command I can still see the application running background. This is my demo code to reappear this problem.
{code:java}
// code placeholder
val rdd = sparkSession.sparkContext.parallelize(List(1, 2, 3, 4)).repartition(4)

//section 1: 
//do something for a long time
val rdd2 = rdd.map(x => { 
  for (i <- 1 to 300) {
    for (j <- 1 to 999999999) {
    }

    if (i % 10 == 0) {
      println(i + " rdd map process running!")
    }

  }

  x * 2
})
rdd2.take(10).foreach(println(_))

//section 2: 
//do something for a long time in driver
for (i <- 1 to 500) {
  for (j <- 1 to 999999999) {
    }

  if (i % 10 == 0) {
    println(i + " main process running!")
  }
}

{code}
And, 
 # If I kill the application on web ui when section 1 rdd.map process, it can be stopped clearly;
 # If I kill the application on web ui when section 2 do something in the driver, and then it stll running background.

So, is it a bug of spark and how to solve it ?
Bug: Throw more specific exceptions when relations/namespaces are not found
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Dec/20 22:33
Affects Version/s: 3.2.0

Currently, when relations/namespaces are not found, a general `AnalysisException` is thrown. This can be improved by throwing more specific exceptions such as `NoSuchNamespaceException` and `NoSuchTableException`.
Bug: Normalize expressions passes to DS V2 from OverwriteByExpression
Status: Open | Priority: Major | Resolution: 
Created: 21/Dec/20 15:28
Affects Version/s: 3.2.0

We need to normalize expressions passed to data source v2 implementations from nodes like {{OverwriteByExpression}} per discussion [here|https://github.com/apache/spark/pull/30806#discussion_r545374031].
Bug: EXPLAIN FORMATTED doesn't show the plan for subqueries if AQE is enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Dec/20 17:25
Affects Version/s: 3.0.0

When AQE is enabled, EXPLAIN FORMATTED doesn't show the plan for subqueries.

{code}
spark.conf.set("spark.sql.adaptive.enabled", "true")
val df = spark.range(1, 100)
df.createTempView("df")
spark.sql("SELECT (SELECT min(id) AS v FROM df)").explain("FORMATTED")

== Physical Plan ==
AdaptiveSparkPlan (3)
+- Project (2)
 +- Scan OneRowRelation (1)


(1) Scan OneRowRelation
Output: []
Arguments: ParallelCollectionRDD[0] at explain at <console>:24, OneRowRelation, UnknownPartitioning(0)

(2) Project
Output [1]: [Subquery subquery#3, [id=#20] AS scalarsubquery()#5L]
Input: []

(3) AdaptiveSparkPlan
Output [1]: [scalarsubquery()#5L]
Arguments: isFinalPlan=false
{code}
Bug: Jobs disappear intermittently from the SHS under high load
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Dec/20 13:01
Affects Version/s: 3.0.0

Ran into an issue when a particular job was displayed in the SHS and disappeared after some time, but then, in several minutes showed up again.

The issue is caused by SPARK-29043, which is designated to improve the concurrent performance of the History Server. The [change|https://github.com/apache/spark/pull/25797/files#] breaks the ["app deletion" logic|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R563] because of missing proper synchronization for {{processing}} event log entries. Since SHS now [filters out|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R462] all {{processing}} event log entries, such entries do not have a chance to be [updated with the new {{lastProcessed}}|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R472] time and thus any entity that completes processing right after [filtering|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R462] and before [the check for stale entities|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R560] will be identified as stale and will be deleted from the UI until the next {{checkForLogs}} run. This is because [updated {{lastProcessed}} time is used as criteria|https://github.com/apache/spark/pull/25797/files#diff-128a6af0d78f4a6180774faedb335d6168dfc4defff58f5aa3021fc1bd767bc0R557], and event log entries that missed to be updated with a new time, will match that criteria.

The issue can be reproduced by generating a big number of event logs and uploading them to the SHS event log directory on S3. Essentially, around 800(82.6 MB) copies of an event log file were created using [shs-monitor|https://github.com/vladhlinsky/shs-monitor] script. Strange behavior of SHS counting the total number of applications was noticed - at first, the number was increasing as expected, but with the next page refresh, the total number of applications decreased. No errors were logged by SHS.
Bug: Add spark.shuffle.io.sessionTimeout to check a established connnection for being idled or dead
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Dec/20 17:58
Affects Version/s: 3.2.0

We have spark.shuffle.io.connectionTimeout for timeout checking when shuffle clients establish connections with shuffle servers. We also use it to check the established connection is idled or dead too. If we want a connection that can keep alive for a long time, we may increase the blocking time to create a shuffle client if the server is busy at that time.
In practice, we always use connection timeout and session timeout to control these two different phases.
Bug: Unload State Store Provider after configured keep alive time
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 15/Dec/20 01:10
Affects Version/s: 3.2.0

Currently Spark unloads an inactive state store provider in an maintenance task which is run periodically. So it is said one state store provider might be unloaded immediately after it becomes inactive because the maintenance task is asynchronous. It is inefficient as the state store provider could be reused in next batches and we should be able to have more control of unloading behavior.
Bug: Make stateful operations reuse previous state store
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 17/Dec/20 01:47
Affects Version/s: 3.2.0

Currently Spark SS can schedule stateful operations to active state store locations. For some reasons, Spark might not be able to schedule tasks to active state store locations. In the case, we should be able to schedule the tasks to sub-optimal locations.
Bug: ExecutorMonitor should handle SparkListenerExecutorExcluded/SparkListenerNodeExcluded
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Dec/20 05:43
Affects Version/s: 3.2.0

Currently, when dynamic-executor-allocation and exclude-on-failure (but exclude_on_kill_enabled is disabled) are both enabled, the excluded executors are still counted as available executor for running tasks. But that's obviously not true.

We could let the ExecutorMonitor handle SparkListenerExecutorExcluded/SparkListenerNodeExcluded to not count excluded executors.
Bug: Provide preferred locations for stateful operations without reported state store locations
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Dec/20 20:29
Affects Version/s: 3.2.0

Stateful operators in SS provides preferred locations on the previous batches if any. However, if there is no previous batch to follow, Spark possibly schedules stateful tasks in inefficient distribution. As stateful operations probably need to maintain large state stores, it is better we schedule stateful tasks across all executors.
Bug: Cache's storage level is not respected when a table name is altered.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Dec/20 03:43
Affects Version/s: 3.2.0

To repro:
{code:java}
        Seq(1 -> "a").toDF("i", "j").write.parquet(path.getCanonicalPath)
        sql(s"CREATE TABLE old USING parquet LOCATION '${path.toURI}'")
        sql("CACHE TABLE old OPTIONS('storageLevel' 'MEMORY_ONLY')")
        val oldStorageLevel = getStorageLevel("old")

        sql("ALTER TABLE old RENAME TO new")
        val newStorageLevel = getStorageLevel("new")
        assert(oldStorageLevel === newStorageLevel)
{code}
The assert fails:
Expected :StorageLevel(disk, memory, deserialized, 1 replicas)
Actual   :StorageLevel(memory, deserialized, 1 replicas)
Bug: Allow creating orc table when row format separator is defined
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 11/Dec/20 09:46
Affects Version/s: 2.4.2

When creating table like this:
{code:java}
create table test_orc(c1 string) row format delimited fields terminated by '002' stored as orcfile;
{code}
spark throws exception like :
{code:java}
Operation
  not allowed: ROW FORMAT DELIMITED is only compatible with 'textfile', not
  'orcfile'(line 2, pos 0)
{code}
I don’t think we need such strict rules, we can support it.
Bug: Allow typesafe join for LeftSemi and LeftAnti
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Dec/20 17:35
Affects Version/s: 3.0.1

With [SPARK-21333|https://issues.apache.org/jira/browse/SPARK-21333] change, LeftSemi and LeftAnti no longer has a typesafe join API. It makes sense to not support LeftSemi and LeftAnti as part of joinWith as it returns tuples which includes values from both the datasets which is not possible in the above joins. Neverthless, it would be nice to have a separate join API or in the existing API to support LeftSemi and LeftAnti which returns Dataset.
Bug: Prune unnecessary output partitioning when the attribute is not part of output.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/20 14:09
Affects Version/s: 3.0.1

Consider the query:
{noformat}
val planned = sql(
          """
            | SELECT t1.id as t1id
            | FROM t1, t2
            | WHERE t1.id = t2.id
          """.stripMargin).queryExecution.executedPlan

println(planned.outputPartitioning)
{noformat}
The output of this will be:

{noformat}
res10: org.apache.spark.sql.catalyst.plans.physical.Partitioning = (hashpartitioning(it1d#6L, 200) or hashpartitioning(t2id#7L, 200)) 
{noformat}

 

This query will have top level Project node which will just project t1.id. So outputPartitioning of this project node should be:

hashpartitioning(it1d#6L, 200)

 

cc - [~maropu] [~cloud_fan]
Bug: Unify v1 and v2 ALTER TABLE .. DROP PARTITION tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Dec/20 17:04
Affects Version/s: 3.1.0

Extract ALTER TABLE .. DROP PARTITION tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Sort output of V2 SHOW PARTITIONS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 16:15
Affects Version/s: 3.2.0

V1 SHOW PARTITIONS command sorts its results. Both V1 implementations in-memory and Hive catalog (according to Hive docs [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowPartitions)] perform sorting. V2 should have the same behavior.
Bug: Fix Invalid value for HourOfAmPm when testing on JDK 14
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Dec/20 00:54
Affects Version/s: 3.2.0

- parsing hour with various patterns *** FAILED *** java.time.format.DateTimeParseException: Text '2009-12-12 12 am' could not be parsed: Invalid value for HourOfAmPm (valid values 0 - 11): 12
Bug: Do not push down partition filters to ParquetScan for DataSourceV2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Dec/20 01:31
Affects Version/s: 3.2.0


{noformat}
- Spark vectorized reader - with partition data column - select a single complex field array and its parent struct array *** FAILED ***
- Non-vectorized reader - with partition data column - select a single complex field array and its parent struct array *** FAILED ***
- Spark vectorized reader - with partition data column - select a single complex field from a map entry and its parent map entry *** FAILED ***
- Non-vectorized reader - with partition data column - select a single complex field from a map entry and its parent map entry *** FAILED ***
- Spark vectorized reader - with partition data column - partial schema intersection - select missing subfield *** FAILED ***
- Non-vectorized reader - with partition data column - partial schema intersection - select missing subfield *** FAILED ***
- Spark vectorized reader - with partition data column - no unnecessary schema pruning *** FAILED ***
- Non-vectorized reader - with partition data column - no unnecessary schema pruning *** FAILED ***
- Spark vectorized reader - with partition data column - empty schema intersection *** FAILED ***
- Non-vectorized reader - with partition data column - empty schema intersection *** FAILED ***
- Spark vectorized reader - with partition data column - select a single complex field and in where clause *** FAILED ***
- Non-vectorized reader - with partition data column - select a single complex field and in where clause *** FAILED ***
- Spark vectorized reader - with partition data column - select nullable complex field and having is not null predicate *** FAILED ***
- Non-vectorized reader - with partition data column - select nullable complex field and having is not null predicate *** FAILED ***
{noformat}
These test will fail, this is because Parquet will return empty results for non-existent column since PARQUET-1765.
Bug: [k8s] Support fetching driver and executor pod templates from HCFS
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/20 18:35
Affects Version/s: 3.0.1

Currently Spark 3 on Kubernetes supports loading driver and executor pod templates from Local file system: [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L87.] However, this is not very convenient as user needs to either bake the pod templates into client pod image or manually mounting the file as configMap. It would be nice if Spark supports loading pod templates from Hadoop Compatible File Systems (such as S3A), so that user can directly update the pod template files in S3 without changing the underline Kubernetes job definition (eg. Updating Docker image or updating ConfigMap)
Bug: Bump commons-codec to latest version. 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Dec/20 18:46
Affects Version/s: 2.4.7

Currently Spark pulls in commons-codec version 2.10 which was released 6 years ago. Some Open Source scans have found a possible encoding/decoding concern related to versions prior to 2.13:

[https://github.com/apache/commons-codec/commit/48b615756d1d770091ea3322eefc08011ee8b113]

Upgrade to the latest version of commons-codec in order to include this fix.
Bug: Upgrade snappy-java to 1.1.8.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Dec/20 20:10
Affects Version/s: 2.4.7

Minor version upgrade that includes:
 * Fixed an initialization issue when using a recent Mac OS X version #265
 * Support Apple Silicon (M1, Mac-aarch64)
 * Fixed the pure-java Snappy fallback logic when no native library for your platform is found.
Bug: Bump Jackson to 2.10.5 and databind to 2.10.5.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 23:24
Affects Version/s: 2.4.7

Jackson reported a vulnerability under CVE-2020-25649. The version pulled in Spark currently is 2.10.0. Upgrading to 2.10.5.1 will resolve problem.
Bug: Require fully specified partition identifier in partitionExists()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 07:54
Affects Version/s: 3.2.0

Currently, partitionExists() from SupportsPartitionManagement accept any partition identifier even which is not fully specified. This ticket aim to add a check for the length of partition schema and partition identifier, and require exact matching. So, we should prohibit not fully specified IDs.
Bug: Support partition filters in ALTER TABLE DROP PARTITION
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 07/Dec/20 12:53
Affects Version/s: 2.4.2

User 'mgaido91' has created a pull request for this issue:
[https://github.com/apache/spark/pull/20999]

but i found some problems when using it, i tried to rewrite it.
Bug: Use an Informer+Lister API in the ExecutorPodWatcher
Status: Open | Priority: Major | Resolution: 
Created: 10/Dec/20 11:38
Affects Version/s: 3.2.0

Kubernetes backend uses Fabric8 client and a [watch|https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala#L42] to monitor the K8s Api server for pod changes. Every watcher keeps a websocket connection open and has no caching mechanism at that part. Caching at the Spark K8s resource manager exists in other areas where we are hitting the Api Server for Pod CRUD ops like [here|https://github.com/apache/spark/blob/b8ccd755244d3cd8a81a9f4a1eafa2a4e48759d2/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala#L49].


In an env where a lot of connections are kept due to large scale jobs this could be problematic and impose a lot of load against the API server. A lot of long running jobs should not create pod changes eg. Streaming jobs to justify a continuous watching mechanism.

Latest Frabric8 client versions have implemented a SharedInformer API+Lister, an example can be found [here|https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/InformerExample.java#L37].
This new API follows the implementation of the official java K8s client and the go counterpart and it is backed up by a caching mechanism which is re-synced after a configurable period to avoid hitting the API server all the time. There is also a lister that keeps track of current status of resources. Using such a mechanism is common place when implementing a K8s controller.
The suggestion is to update to v4.13.0 the client (has all updates in wrt that API) and use the informer+lister API where applicable. 
I think the lister could also replace part of the snapshotting/notification mechanism.

/cc [~dongjoon] [~eje] [~holden] WDYTH?
Bug: view shouldn't use current catalog and namespace to lookup function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 15:00
Affects Version/s: 3.0.1

Reproduce steps:
spark.sql("CREATE FUNCTION udf_plus AS 'udf.UdfPlus10' USING JAR '/home/linhong.liu/spark-udf_2.12-0.1.0-SNAPSHOT.jar'")

spark.sql("create view v1 as select udf_plus(1)")
spark.sql("select * from v1").show() // output 11

spark.sql("CREATE TEMPORARY FUNCTION udf_plus AS 'udf.UdfPlus20' USING JAR '/home/linhong.liu/spark-udf_2.12-0.1.0-SNAPSHOT.jar'")

spark.sql("select * from v1").show() // throw exception

org.apache.spark.sql.AnalysisException: Attribute with name 'default.udf_plus(1)' is not found in '(udf_plus(1))';;
Project [default.udf_plus(1)#60]
+- SubqueryAlias spark_catalog.default.v1
   +- View (`default`.`v1`, [default.udf_plus(1)#60])
      +- Project [HiveSimpleUDF#udf.UdfPlus20(1) AS udf_plus(1)#61]
         +- OneRowRelation
Bug: Support to use Hive build-in functions by configuration
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Dec/20 11:53
Affects Version/s: 2.4.3

Hive and Spark SQL engines have many differences in built-in functions.The differences between several functions are shown below：
||*build-in functions*||SQL|| result of Hive SQL ||result of Spark SQL||
|unix_timestamp|{{select}} {{unix_timestamp(concat(}}{{'2020-06-01'}}{{,  }}{{' 24:00:00'}}{{));}}|1591027200| NULL|
|to_date|{{select}} {{to_date(}}{{'0000-00-00'}}{{);}}|0002-11-30| NULL|
|datediff|{{select }}{{datediff(}}{{CURRENT_DATE}}{{, }}{{'0000-00-00'}}{{);}}|737986| NULL|
|collect_set|{{select}}{{c1}}{{,c2}}{{,concat_ws(}}{{'##'}}{{, collect_set(c3)) c3_set }}{{from }}{{bigdata_offline.test_collect_set }}{{group }}{{by }}{{c1, c2;}}
 {{bigdata_offline.test_collect_set contains data:}}
 {{(1, 1, }}{{'1'}}{{)，}}{{(1, 1, }}{{'2'}}{{)}}{{,}}
 {{(1, 1, }}{{'3'}}{{)}}{{,}}{{(1, 1, }}{{'4'}}{{)}}{{,}}
 {{(1, 1, }}{{'5'}}{{)}}|{{c1  c2  c3_set}}
 {{1   1   2##3##4##5##1}}|{{c1  c2      c3_set}}
 {{1   1   3##1##2##5##4}}|

There is no conclusion on which engine is  more accurate. Users prefer to be able to make choices according to their real production environment.

I think we should do some improvement for this.

 

Hive version is 1.2.1
Bug: Shuffle Index use Guava cache OOM, Yarn NodeManage GC Alarm
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Dec/20 09:33
Affects Version/s: 3.2.0

h2. CDH6.3 Yarn nodemanger frequently GC, and then the dump file is generated due to memory overflow

!https://upload-images.jianshu.io/upload_images/18249296-24acecfcc46dc744.png?imageMogr2/auto-orient/strip|imageView2/2/w/607/format/webp!
  
h2. Use the Memory Analyzer Tool to locate the shuffle index module

 

Using guava to cache the memory limit, there is no restriction on the cache key, resulting in a lot of path information in the memory. If the size of shuffleindexinformation in the cache is very small, the number of keys will be very large, and eventually lead to memory overflow. I think there is a defect here, and the capacity of key should be added to the statistics of 100MB

 

!https://upload-images.jianshu.io/upload_images/18249296-ed0cfee76b6f6bf2.png?imageMogr2/auto-orient/strip|imageView2/2/w/630/format/webp!

According to the MAT, the ExternalShuffleBlockHandler uses guava's local cache and takes up 82.88% of the heap memory

 

 

!https://upload-images.jianshu.io/upload_images/18249296-43ec91771f3c68b7.png?imageMogr2/auto-orient/strip|imageView2/2/w/760/format/webp!

!https://upload-images.jianshu.io/upload_images/18249296-f85e27a501605260.png?imageMogr2/auto-orient/strip|imageView2/2/w/1147/format/webp!

Through the analysis, it is found that there are a lot of shuffle index path information in the memory, which takes up more than 400 MB of memory, and the number is very large. This path is the key of shuffleindex cache in the external shufflebock resolver. After looking at the source code, we know that there may be some defects in the cache management, because the limited 100MB does not include the key statistics

  !https://upload-images.jianshu.io/upload_images/18249296-87118ce13744c2ca.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp!
Bug: Error about cores being limiting resource confusing
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/20 20:26
Affects Version/s: 3.0.1

If you misconfigure your resources and cores is not the limiting resource we print the following error:

20/12/08 20:23:37 ERROR Main: Failed to initialize Spark session.
java.lang.IllegalArgumentException: The number of slots on an executor has to be limited by the number of cores, otherwise you waste resources and dynamic allocation doesn't work properly. Your configuration has core/task cpu slots = 8 and gpu = 1. Please adjust your configuration so that all resources require same number of executor slots.

I recieved reports that this was confusing to users. Specifically the sentence "Your configuration has core/task cpu slots = 8 and gpu = 1" so I think we can improve that message.

Note this only affects 3.0.0 and 3.0.1, 3.1.0 changed this functionality.

To reproduce just run spark 3.0.1 with something like:

$SPARK_HOME/bin/spark-shell --master yarn --executor-cores 8 --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1