av9ash/GitBugs huggingface dataset:
Bug: Directly push IN predicates to the Hive Metastore
Status: Open | Priority: Major | Resolution: 
Created: 24/Nov/20 16:18
Affects Version/s: 3.1.0

Hive 2.0 support directly push IN predicates to the Hive Metastore. Plase see https://issues.apache.org/jira/browse/HIVE-11726 for more detail.

We should use this api to improve performance.
Bug: Resolver of V2 partition specs doesn't handle __HIVE_DEFAULT_PARTITION__
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Nov/20 09:25
Affects Version/s: 3.1.0

The partition value '__HIVE_DEFAULT_PARTITION__' should be handled as null - the same as DSv1 does.

For example in DSv1:

{code:java}
spark-sql> CREATE TABLE tbl11 (id int, part0 string) USING parquet PARTITIONED BY (part0);
spark-sql> ALTER TABLE tbl11 ADD PARTITION (part0 = '__HIVE_DEFAULT_PARTITION__');
spark-sql> INSERT INTO tbl11 PARTITION (part0='__HIVE_DEFAULT_PARTITION__') SELECT 1;
spark-sql> SELECT * FROM tbl11;
1	NULL
{code}
Bug: Universal type conversion of V2 partition values
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/20 19:49
Affects Version/s: 3.1.0

Support other types while resolving partition specs in

https://github.com/apache/spark/blob/23e9920b3910e4f05269853429c7f18888cdc7b5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolvePartitionSpec.scala#L72
Bug: running spark application in kerbernetes,bug the application log shows yarn authentications 
Status: Open | Priority: Major | Resolution: 
Created: 19/Nov/20 06:41
Affects Version/s: 3.0.0

My spark application accessing kerberized HDFS is running in kubernetes cluster， but the application log shows: "Setting spark.hadoop.yarn.resourcemanager.principal to tester(which is one of my kerberos principals, yet I uses the other principal joan to read HDFS files)":

... 

+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
 + exec /usr/bin/tini -s – /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.1.61 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class WordCount local:///opt/spark/jars/WordCount-1.0-SNAPSHOT.jar
 *Setting spark.hadoop.yarn.resourcemanager.principal to tester*

...

20/11/19 04:31:28 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1041285450_1, ugi=*tester@JOANTEST* (auth:KERBEROS)]] with renewer tester
 20/11/19 04:31:37 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 60 for tester on ha-hdfs:nameservice1
 20/11/19 04:31:37 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1041285450_1, ugi=*tester@JOANTEST* (auth:KERBEROS)]] with renewer tester@JOANTEST
 20/11/19 04:31:37 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 61 for *tester* on ha-hdfs:nameservice1
 20/11/19 04:31:37 INFO HadoopFSDelegationTokenProvider: Renewal interval is 86400073 for token HDFS_DELEGATION_TOKEN

...

 20/11/19 04:31:51 INFO UserGroupInformation: *Login successful for user joan using keytab file /opt/hadoop/conf/joan.keytab*

...

 

I don't know why yarn authentication is needed here?And why use the principal tester for autherization? Anyone can help? Thanks !

The log and my spark project is attached blow for reference.
Bug: Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/20 23:01
Affects Version/s: 3.0.1

`sql/core` module seems to be broken. I will file a new Jira.
{code:java}
$ build/sbt "sql/testOnly *.DataSourceV2SQLSuite" -Pscala-2.13
...
[info] - SPARK-31255: Project a metadata column *** FAILED *** (96 milliseconds)
[info] - SPARK-31255: Projects data column when metadata column has the same name *** FAILED *** (77 milliseconds){code}
Bug: Respect case sensitivity in resolving partition specs V2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/20 19:55
Affects Version/s: 3.1.0

DSv1 DDL commands respect the SQL config spark.sql.caseSensitive, for example
{code:java}
spark-sql> CREATE TABLE tbl1 (id bigint, data string) USING parquet PARTITIONED BY (id);
spark-sql> ALTER TABLE tbl1 ADD PARTITION (ID=1);
spark-sql> SHOW PARTITIONS tbl1;
id=1
{code}
but the same ALTER TABLE command fails on DSv2.
Bug: Interpreted subexpression elimination
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/20 22:03
Affects Version/s: 3.1.0

Currently we only do subexpression elimination for codegen. For some reasons, we may need to run interpreted expression valuation. For example, codegen fails to compile and fallback to interpreted mode. It is commonly seen for complex schema from expressions that is possibly caused by the query optimizer too, e.g. SPARK-32945.

We should also support subexpression elimination for interpreted evaluation. That could reduce performance difference when Spark fallbacks from codegen to interpreted expression evaluation.
Bug: t 130038ERROR ContextCleaner:91 - Error cleaning broadcast
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 20/Nov/20 22:09
Affects Version/s: 2.3.3

I am using spark 2.3.3  with 16 workers with 30 cores each. I facing similar exception. This issue is occurring once a while and spark streaming process do not handle any request after this. We need to restart the streaming process.

 

Error cleaning broadcast 130038ERROR ContextCleaner:91 - Error cleaning broadcast 130038
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [240 seconds]. This timeout is controlled by spark.network.timeout
 at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
 at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
 at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
 at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
 at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
 at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:92)
 at org.apache.spark.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:76)
 at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:148)
 at org.apache.spark.broadcast.TorrentBroadcast$.unpersist(TorrentBroadcast.scala:321)
 at org.apache.spark.broadcast.TorrentBroadcastFactory.unbroadcast(TorrentBroadcastFactory.scala:45)
 at org.apache.spark.broadcast.BroadcastManager.unbroadcast(BroadcastManager.scala:66)
 at org.apache.spark.ContextCleaner.doCleanupBroadcast(ContextCleaner.scala:238)
 at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:194)
 at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$an
Bug: Fix insert into `InMemoryPartitionTable`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 19:01
Affects Version/s: 3.1.0

Currently, INSERT INTO a partitioned table in V2 in-memory catalog doesn't create partitions. The example below demonstrates the issue:

{code:scala}
  test("insert into partitioned table") {
    val t = "testpart.ns1.ns2.tbl"
    withTable(t) {
      spark.sql(
        s"""
           |CREATE TABLE $t (id bigint, name string, data string)
           |USING foo
           |PARTITIONED BY (id, name)""".stripMargin)
      spark.sql(s"INSERT INTO $t PARTITION(id = 1, name = 'Max') SELECT 'abc'")

      val partTable = catalog("testpart").asTableCatalog
        .loadTable(Identifier.of(Array("ns1", "ns2"), "tbl")).asInstanceOf[InMemoryPartitionTable]
      assert(partTable.partitionExists(InternalRow.fromSeq(Seq(1, UTF8String.fromString("Max")))))
    }
  }
{code}

The partitionExists() function return false for the partitions that must be created.
Bug: Align DSv2 commands to DSv1 implementation
Status: Open | Priority: Major | Resolution: 
Created: 09/Nov/20 10:02
Affects Version/s: 3.1.0

The purpose of this umbrella ticket is:
# Implement missing features of datasource v1 commands in DSv2
# Align behavior of DSv2 commands to the current implementation of DSv1 commands as much as possible.
Bug: DSv2: REFRESH TABLE should invalidate caches
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/20 18:26
Affects Version/s: 3.0.1

Currently, in DSv2 {{RefreshTableExec}}, we only invalidate metadata cache but not all the caches that referencing the table to be refreshed. This may cause correctness issue if these caches go stale and get queried later.

Note that since we don't support caching a v2 table yet, we can't recache the table itself at the moment.
Bug: Support field "EPOCH" in datetime function extract/date_part
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Nov/20 11:12
Affects Version/s: 3.1.0

Support filed EPOCH in the function `extract` and `date_part`.
For example:
SELECT EXTRACT(EPOCH FROM TIMESTAMP'2001-02-16 20:38:40-08');
SELECT DATE_PART('EPOCH', TIMESTAMP'2001-02-16 20:38:40-08');

This is useful for getting the number of seconds since `1970-01-01 00:00:00-00`
Bug: Scala 2.13 build with SBT failure
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 19/Nov/20 17:40
Affects Version/s: 3.1.0

Github actions sometimes fail with:
{code}
[error] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1417:41: type mismatch;
[error]  found   : scala.collection.mutable.Buffer[org.apache.spark.unsafe.types.UTF8String]
[error]  required: Seq[org.apache.spark.unsafe.types.UTF8String]
[error]                 case null => LikeAll(e, patterns)
[error]                                         ^
[error] /home/runner/work/spark/spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1418:41: type mismatch;
[error]  found   : scala.collection.mutable.Buffer[org.apache.spark.unsafe.types.UTF8String]
[error]  required: Seq[org.apache.spark.unsafe.types.UTF8String]
[error]                 case _ => NotLikeAll(e, patterns)
[error]                                         ^
{code}
Bug: Normalize sameOrderExpressions in SortOrder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 15:40
Affects Version/s: 2.4.7

When a SortMergeJoin is followed by a Project with aliases, the outputOrdering is not propogated properly and in some cases, it leads to unrequired Sort operation:

 

 
{noformat}
spark.range(10).repartition($"id").createTempView("t1")
spark.range(20).repartition($"id").createTempView("t2")
spark.range(30).repartition($"id").createTempView("t3")

val planned = sql(
   """
     |SELECT t2id, t3.id as t3id
     |FROM (
     |    SELECT t1.id as t1id, t2.id as t2id
     |    FROM t1, t2
     |    WHERE t1.id = t2.id
     |) t12, t3
     |WHERE t1id = t3.id
   """.stripMargin).queryExecution.executedPlan



*(8) Project [t2id#1059L, id#1004L AS t3id#1060L]
+- *(8) SortMergeJoin [t2id#1059L], [id#1004L], Inner
   :- *(5) Sort [t2id#1059L ASC NULLS FIRST ], false, 0  <-----------------------
   :  +- *(5) Project [id#1000L AS t2id#1059L]
   :     +- *(5) SortMergeJoin [id#996L], [id#1000L], Inner
   :        :- *(2) Sort [id#996L ASC NULLS FIRST ], false, 0
   :        :  +- Exchange hashpartitioning(id#996L, 5), true, [id=#1426]
   :        :     +- *(1) Range (0, 10, step=1, splits=2)
   :        +- *(4) Sort [id#1000L ASC NULLS FIRST ], false, 0
   :           +- Exchange hashpartitioning(id#1000L, 5), true, [id=#1432]
   :              +- *(3) Range (0, 20, step=1, splits=2)
   +- *(7) Sort [id#1004L ASC NULLS FIRST ], false, 0
      +- Exchange hashpartitioning(id#1004L, 5), true, [id=#1443]
         +- *(6) Range (0, 30, step=1, splits=2)

{noformat}
The above marked Sort node could have been avoided.
Bug: Better heuristics to compute number of shuffle mergers required for a ShuffleMapStage
Status: Open | Priority: Major | Resolution: 
Created: 18/Nov/20 18:16
Affects Version/s: 3.1.0

Currently we have a naive way of computing the number of shuffle mergers required for a ShuffleMapStage by considering the resulting number of partitions. This needs to be further brainstormed and enhanced.
Bug: Allow overwrite a path that is also being read under dynamic partition overwrite
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Nov/20 09:34
Affects Version/s: 3.0.0

Currently, Insert overwrite cannot overwrite a path that is also being read under dynamic partition overwrite. 
In the class DataSourceAnalysis, DDLUtils.verifyNotReadPath is called to determine whether it can be overwrite, but in the InsertIntoHadoopFsRelationCommand.dynamicPartitionOverwrite method, we need to consider the scenario where staticPartitions.size equals partitionColumns.length, which should also be allowed to overwrite path this is also being read.

Consider the following statement:

CREATE TABLE insertTable(i int, part1 int, part2 int) USING PARQUET PARTITIONED BY (part1, part2);

INSERT OVERWRITE TABLE insertTable PARTITION(part1=1, part2=1) SELECT i + 1 FROM insertTable;
Bug: RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 17/Nov/20 06:08
Affects Version/s: 3.1.0

{{RDD.takeOrdered}} API sorts elements in each partition and puts the sorted elements into a `BoundedPriorityQueue`.  So actually in the result RDD each partition has only one priority queue. Then the API calls {{RDD.reduce}} API to reduce the elements. But as mentioned before the RDD has only one queue at each partition, it doesn't make sense to call reduce to reduce elements (here the element is queue).

We should either simplify {{RDD.reduce}} call in {{RDD.takeOrdered}} or replace it with {{treeReduce}} which can actually do partially reducing for this case.
Bug: HistoryServer stuck "Loading history summary" with Spark 3.0.1 and hadoop 2.10.1
Status: Open | Priority: Major | Resolution: 
Created: 17/Nov/20 19:12
Affects Version/s: 3.0.0

History Server starts correctly and shows "No completed applications found!" message when /tmp/spark-events directory is empty. As soon as there is an event log in the folder generated by a Spark streaming app (haven't tested other apps) the History Server gets stuck showing the "Loading history summary..." popup.

Spark version: 3.0.1 and 3.0.0. (Spark 2.4.5 works as expected (running application shown in History Server))

Hadoop version: 2.10.0 and 2.10.1

Java version: openjdk 1.8.0_272

OS: CentOS 7.7

Command to reproduce:

 
{code:java}
spark-submit --class MyClass --master yarn --deploy-mode cluster --jars /usr/local/hadoop/share/hadoop/libs/*.jar myclass.jar{code}
spark-defaults.conf:

 

 
{code:java}
spark.eventLog.dir=/tmp/spark-events
spark.eventLog.enabled=true
spark.eventLog.rolling.enabled=true
spark.ui.enabled=true
spark.yarn.stagingDir=hdfs://10.0.1.4:54310/
{code}
 

Stack trace generated with jstack of the HistoryServer process is attached
Bug: The link address of ‘’this page’’ in docs/pyspark-migration-guide.md is incorrect
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Nov/20 03:42
Affects Version/s: 3.0.1

 
 
  The link address of ‘’this page’’ in +docs/pyspark-migration-guide.md+ is incorrect, and it will show ""Not Found The requested URL was not found on this server.""
[[https://github.com/apache/spark/blob/master/docs/pyspark-migration-guide.md]|[https://github.com/apache/spark/blob/master/docs/pyspark-migration-guide.md]]
Bug: Normalize output partitioning and sortorder with respect to aliases to avoid unneeded exchange/sort nodes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 15:04
Affects Version/s: 2.4.7

Spark introduces unneeded exchanges if there is a Project after Inner join. Example:

 
{noformat}
spark.range(10).repartition($"id").createTempView("t1")
spark.range(20).repartition($"id").createTempView("t2")
spark.range(30).repartition($"id").createTempView("t3")

val planned = sql(
   """
     |SELECT t2id, t3.id as t3id
     |FROM (
     |    SELECT t1.id as t1id, t2.id as t2id
     |    FROM t1, t2
     |    WHERE t1.id = t2.id
     |) t12, t3
     |WHERE t1id = t3.id
   """.stripMargin).queryExecution.executedPlan


*(9) Project [t2id#1034L, id#1004L AS t3id#1035L]
+- *(9) SortMergeJoin [t1id#1033L], [id#1004L], Inner
   :- *(6) Sort [t1id#1033L ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(t1id#1033L, 5), true, [id=#1343] <---------------
   :     +- *(5) Project [id#996L AS t1id#1033L, id#1000L AS t2id#1034L]
   :        +- *(5) SortMergeJoin [id#996L], [id#1000L], Inner
   :           :- *(2) Sort [id#996L ASC NULLS FIRST], false, 0
   :           :  +- Exchange hashpartitioning(id#996L, 5), true, [id=#1329]
   :           :     +- *(1) Range (0, 10, step=1, splits=2)
   :           +- *(4) Sort [id#1000L ASC NULLS FIRST], false, 0
   :              +- Exchange hashpartitioning(id#1000L, 5), true, [id=#1335]
   :                 +- *(3) Range (0, 20, step=1, splits=2)
   +- *(8) Sort [id#1004L ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(id#1004L, 5), true, [id=#1349]
         +- *(7) Range (0, 30, step=1, splits=2){noformat}
The marked exchange in the above plan can be removed.
Bug: ResourceProfile use Int for memory in ExecutorResourcesOrDefaults
Status: Open | Priority: Major | Resolution: 
Created: 16/Nov/20 19:25
Affects Version/s: 3.1.0

A followup from SPARK-33288, since memory is in MB we can just store as Int rather then Long in ExecutorResourcesOrDefaults.

 

See https://github.com/apache/spark/pull/30375#issuecomment-728270233
Bug: Throw `NoSuchDatabaseException` for not existing namespace in DSv2 SHOW TABLES
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 10:16
Affects Version/s: 3.1.0

Current implementation of DSv2 SHOW TABLES return an empty result for not existing database/namespace. This implementation should be aligned to DSv1 which throws the `NoSuchDatabaseException` exception.
Bug: Support subexpression elimination in branches of conditional expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Nov/20 06:14
Affects Version/s: 3.1.0

Currently we skip subexpression elimination in branches of conditional expressions including {{If}}, {{CaseWhen}}, and {{Coalesce}}. Actually we can do subexpression elimination for such branches if the subexpression is common across all branches.
Bug: Unexpected behavior when using SET commands before a query in SparkSession.sql
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Nov/20 08:27
Affects Version/s: 2.4.7

SparkSession.sql converts a string value to a DataFrame, and the string value should be one single SQL statement ending up w/ or w/o one or more semicolons. e.g.


{code:sql}
scala> spark.sql(" select 2").show
+---+
|  2|
+---+
|  2|
+---+


scala> spark.sql(" select 2;").show
+---+
|  2|
+---+
|  2|
+---+

scala> spark.sql(" select 2;;;;").show
+---+
|  2|
+---+
|  2|
+---+
{code}


If you put 2 or more statements in, it fails in the parser e.g.  


{code:java}
scala> spark.sql(" select 2; select 1;").show
org.apache.spark.sql.catalyst.parser.ParseException:
extraneous input 'select' expecting {<EOF>, ';'}(line 1, pos 11)

== SQL ==
 select 2; select 1;
-----------^^^

  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:263)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:130)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:51)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:81)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:610)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:610)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:769)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:607)
  ... 47 elided

{code}

As a very generic user scenario,  they want to change some settings before they execute
the queries. They may pass a string value like `set spark.sql.abc=2; select 1;` into this API, which creates a confusing gap between the actual effect and the user's expectations.

The user may want the query to be executed with spark.sql.abc=2, but Spark actually treats the whole part of `2; select 1;` as the value of the property 'spark.sql.abc',
 e.g.

{code:java}
scala> spark.sql("set spark.sql.abc=2; select 1;").show
+-------------+------------+
|          key|       value|
+-------------+------------+
|spark.sql.abc|2; select 1;|
+-------------+------------+
{code}
Bug: "date_trunc" expression returns incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 23:42
Affects Version/s: 3.0.0

`date_trunc` SQL expression returns incorrect results for {{minute}} formatting string.

Context: The {{minute}} formatting string should truncate the timestamps such that the seconds is set to ZERO.

Repro (run the following commands in spark-shell):
{quote}
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
spark.sql("SELECT date_trunc('minute', '1769-10-17 17:10:02')").show()
{quote}

Spark currently incorrectly returns 
{quote}
1769-10-17 17:10:02
{quote}
against the expected return value of 
{quote}
1769-10-17 17:10:00
{quote}
This happens as {{truncTimestamp}} in package {{org.apache.spark.sql.catalyst.util.DateTimeUtils}} incorrectly assumes that time zone offsets can never have the granularity of a second and thus does not account for time zone adjustment when truncating the timestamp to {{minute}}. 
This assumption is currently used when truncating the timestamps to {{microsecond, millisecond, second, or minute}}.
Bug: Jobs launched in same second have duplicate MapReduce JobIDs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Nov/20 20:01
Affects Version/s: 2.4.8

Spark uses the current timestamp to generate a MapReduce JobID.
If > 1 job attempt is generated in the same second, these can clash

Committers which expect this to be unique can conflict with the other jobs

* S3A staging committer (cluster FS staging dir and local task output dir)
* Any committer which supports parallel jobs writing to the same destination
  directory and requires unique names for the attempts
* Code which uses the jobID as part of its algorithm to generate unique filenames

Note: {{HadoopMapReduceCommitProtocol.getFilename()}} doesn't use this JobID for
uniqueness, it uses task attempt ID and stage ID. It probably deserves its own
audit.
Bug: [CORE] Remove unnecessary python path from spark home
Status: Closed | Priority: Major | Resolution: Invalid
Created: 06/Nov/20 19:24
Affects Version/s: 3.0.1

Currently, spark-submit will upload pyspark.zip and py4j-0.10.9-src.zip into staging folder, and both files will be added into PYTHONPATH. So it's unnecessary to add duplicate files in current spark home folder on local machine. 
Output of `sys.path` as below:

'D:\\data\\yarnnm\\local\\usercache\\z\\appcache\\application_1603546638930_150736\\container_e1148_1603546638930_150736_01_000002\\pyspark.zip',
'D:\\data\\yarnnm\\local\\usercache\\z\\appcache\\application_1603546638930_150736\\container_e1148_1603546638930_150736_01_000002\\py4j-0.10.7-src.zip',
'D:\\data\\spark.latest\\python\\lib\\pyspark.zip',
'D:\\data\\spark.latest\\python\\lib\\py4j-0.10.7-src.zip',