av9ash/GitBugs huggingface dataset:
Bug: JDK 11 compilation failure due to StackOverflowError
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/May/21 11:47
Affects Version/s: 3.2.0

See https://github.com/apache/spark/runs/2554067779

{code}
java.lang.StackOverflowError
scala.reflect.internal.Trees$UnderConstructionTransformer.transform(Trees.scala:1741)
scala.reflect.internal.Trees$UnderConstructionTransformer.transform$(Trees.scala:1740)
scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:289)
scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:477)
scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:330)
scala.reflect.api.Trees$Transformer.$anonfun$transformStats$1(Trees.scala:2597)
scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:2595)
scala.reflect.internal.Trees.itransform(Trees.scala:1404)
scala.reflect.internal.Trees.itransform$(Trees.scala:1374)
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:28)
scala.reflect.api.Trees$Transformer.transform(Trees.scala:2563)
scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:51)
scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.scala$reflect$internal$Trees$UnderConstructionTransformer$$super$transform(ExplicitOuter.scala:212)
scala.reflect.internal.Trees$UnderConstructionTransformer.transform(Trees.scala:1745)
scala.reflect.internal.Trees$UnderConstructionTransformer.transform$(Trees.scala:1740)
scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:289)
scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:477)
scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:330)
scala.reflect.internal.Trees.itransform(Trees.scala:1383)
{code}
Bug: Spark-SQL test fails on IBM Z for certain config combinations.
Status: Open | Priority: Major | Resolution: 
Created: 25/May/21 16:24
Affects Version/s: 3.1.1

Some queries of SQL related test cases: in-joins.sql, in-order-by.sql, not-in-group-by.sql and SubquerySuite.scala are failing with specific configuration combinations on IBM Z(s390x).

For example: 

sql("select * from l where a = 6 and a not in (select c from r where c is not null)") query from SubquerySuite.scala fails for following config combinations:
|enableNAAJ|enableAQE|enableCodegen|
|TRUE|FALSE|FALSE|
|TRUE|TRUE|FALSE|

The above combination is also causing 2 other queries in in-joins.sql and in-order-by.sql failing.

Another query: 

SELECT Count(*)
 FROM (SELECT *
 FROM t2
 WHERE t2a NOT IN (SELECT t3a
 FROM t3
 WHERE t3h != t2h)) t2
 WHERE t2b NOT IN (SELECT Min(t2b)
 FROM t2
 WHERE t2b = t2b
 GROUP BY t2c);

from not-in-group-by.sql is failing for following combinations:
|enableAQE|enableCodegen|
|FALSE|TRUE|
|FALSE|FALSE|

 

These Test cases are not failing for 3.0.1 release and I believe might have been introduced with [SPARK-32290|https://issues.apache.org/jira/browse/SPARK-32290] . 

There is another strange behaviour observed, if expected output is 1,3 , I am getting 1, 3, 9. If I update the Golden file to expect 1, 3, 9, the output will be 1, 3.
Bug: Spark history Server to S3 doesn't show incomplete applications
Status: Open | Priority: Major | Resolution: 
Created: 17/May/21 21:00
Affects Version/s: 2.4.5

Jupyter Notebook sparkmagic with Spark(2.4.5)  client mode running on Kubernetes.  I am redirecting the spark event logs to a S3 with the following configuration:
  
 spark.eventLog.enabled = true
 spark.history.ui.port = 18080
 spark.eventLog.dir = s3://livy-spark-log/spark-history/
 spark.history.fs.logDirectory = s3://livy-spark-log/spark-history/
 spark.history.fs.update.interval = 5s

spark.eventLog.buffer.kb = 1k

 

spark.streaming.driver.writeAheadLog.closeFileAfterWrite = true
 spark.streaming.receiver.writeAheadLog.closeFileAfterWrite = true

 
  
 Once my application is completed, I can see it shows up on the spark history server. However, running applications doesn't show up on "incomplete applications". I have also checked the log, whenever my application end, I can see this message:
  
 {{21/05/17 06:14:18 INFO k8s.KubernetesClusterSchedulerBackend: Shutting down all executors}}
 {{21/05/17 06:14:18 INFO k8s.KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down}}
 {{21/05/17 06:14:18 WARN k8s.ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)}}
 *{{21/05/17 06:14:18 INFO s3n.MultipartUploadOutputStream: close closed:false s3://livy-spark-log/spark-history/spark-48c3141875fe4c67b5708400134ea3d6.inprogress}}*
 *{{21/05/17 06:14:19 INFO s3n.S3NativeFileSystem: rename s3://livy-spark-log/spark-history/spark-48c3141875fe4c67b5708400134ea3d6.inprogress s3://livy-spark-log/spark-history/spark-48c3141875fe4c67b5708400134ea3d6}}*
 {{21/05/17 06:14:19 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!}}
 {{21/05/17 06:14:19 INFO memory.MemoryStore: MemoryStore cleared}}
 {{21/05/17 06:14:19 INFO storage.BlockManager: BlockManager stopped}}
  
  
 {{I am not able to see any xx.inprogress file on S3 though. Anyone had this problem before? Otherwise, I would take it as a bug.}}
Bug: Filters out the TPC-DS queries that can cause flaky test results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/May/21 13:56
Affects Version/s: 3.0.2

This ticket aims at filtering out TPCDS v1.4 q6 and q75 in `TPCDSQueryTestSuite`.

I saw`TPCDSQueryTestSuite` failed nondeterministically because output row orders were different with those in the golden files. For example, the failure in the GA job, https://github.com/linhongliu-db/spark/runs/2507928605?check_suite_focus=true, happened because the `tpcds/q6.sql` query output rows were only sorted by `cnt`:

https://github.com/apache/spark/blob/a0c76a8755a148e2bd774edcda12fe20f2f38c75/sql/core/src/test/resources/tpcds/q6.sql#L20
Actually, `tpcds/q6.sql` and `tpcds-v2.7.0/q6.sql` are almost the same and the only difference is that `tpcds-v2.7.0/q6.sql` sorts both `cnt` and `a.ca_state`:
https://github.com/apache/spark/blob/a0c76a8755a148e2bd774edcda12fe20f2f38c75/sql/core/src/test/resources/tpcds-v2.7.0/q6.sql#L22
So, I think it's okay just to test `tpcds-v2.7.0/q6.sql` in this case (q75 has the same issue).
Bug: Critical Vulnerabilities: jackson-databind 2.4.0 shipped with htrace-core4-4.1.0-incubating.jar
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 25/May/21 14:22
Affects Version/s: 3.0.2

Vulnerabilities scanner is highlighting following CRITICAL vulnerabilities in {{spark-3.0.2-bin-hadoop3.2}} coming from obsolete {{jackson-databind}} {{2.4.0}} :
 * [CVE-2018-7489|https://nvd.nist.gov/vuln/detail/CVE-2018-7489]
 * [CVE-2018-14718|https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2018-14718]

This package is shipped within {{jars/htrace-core4-4.1.0-incubating.jar}}
Bug: Eagerly execute commands in QueryExecution instead of caller sides
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/21 02:59
Affects Version/s: 3.2.0

Currently, Spark doesn't support LeafRunnableCommand as sub query.
Because the LeafRunnableCommand always output GenericInternalRow and some node(e.g. SortExec, AdaptiveExecutionExec, WholeCodegenExec) will convert GenericInternalRow to UnsafeRow. So will causes error as follows:

{code:java}
java.lang.ClassCastException
org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow
{code}
Bug: Completely fix the broadcast timeout issue in AQE
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/May/21 14:25
Affects Version/s: 3.0.0

SPARK-33933 report a issue that in AQE, when the resources is limited, broadcast timeout could happened. 

[#31269|https://github.com/apache/spark/pull/31269] gives a partial fix by reorder newStages by class type to make sure BroadcastQueryState precede others when calling materialized(). However, it only guarantee that the order of task to be scheduled in normal circumstances, but, the guarantee is not strict since the submit of broadcast job and shuffle map job are in different thread.

So we need a completely fix to avoid the edge case triggering broadcast timeout.
Bug: Not Coalesce shuffle partitions when cache table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/May/21 04:53
Affects Version/s: 3.0.1

*How to reproduce the problem*

_linux shell command to prepare data:_
 for i in $(seq 200000);do echo "$(($i+100000)),name$i,$(($i*10))";done > data.text

_sql to reproduce the problem:_
 * create table data_table(id int, str string, num int) row format delimited fields terminated by ',';
 * load data local inpath '/path/to/data.text' into table data_table;
 * CACHE TABLE test_cache_table AS
 SELECT str
 FROM
 (SELECT id,str FROM data_table
 )group by str;

Finally you will see a stage with 200 tasks and not coalesce shuffle partitions, the problem will waste resource when data size is small.
Bug: Create more robust link for Data Source Options
Status: In Progress | Priority: Major | Resolution: 
Created: 21/May/21 04:22
Affects Version/s: 3.2.0

Now the link for the Data Source Options using /latest/, but it possibly be broken when we cut branch-3.2

For example, [Data Source Option for Avro|https://spark.apache.org/docs/latest/sql-data-sources-avro.html#data-source-option]

It should point 3.2 document only in branch-3.2, so it's better to use a relative link instead of /latest/.
Bug: Adapt uses of JDK 17 Internal APIs
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 28/May/21 15:33
Affects Version/s: 3.2.0

I tried to run a Spark pipeline using the most recent 3.2.0-SNAPSHOT with Spark 2.12.4 on Java 17 and I found this exception:
{code:java}
java.lang.ExceptionInInitializerError
 at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit> (ByteArrayMethods.java:54)
 at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149)
 at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654)
 at org.apache.spark.SparkConf.contains (SparkConf.scala:455)
...
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @110df513
 at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:357)
 at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:297)
 at java.lang.reflect.Constructor.checkCanSetAccessible (Constructor.java:188)
 at java.lang.reflect.Constructor.setAccessible (Constructor.java:181)
 at org.apache.spark.unsafe.Platform.<clinit> (Platform.java:56)
 at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit> (ByteArrayMethods.java:54)
 at org.apache.spark.internal.config.package$.<clinit> (package.scala:1149)
 at org.apache.spark.SparkConf$.<clinit> (SparkConf.scala:654)
 at org.apache.spark.SparkConf.contains (SparkConf.scala:455)}}
{code}
It seems that Java 17 will be more strict about uses of JDK Internals [https://openjdk.java.net/jeps/403]
Bug: Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 21:20
Affects Version/s: 3.0.2

https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API {{get_all_functions}} which is only supported in Hive 1.3.0/2.0.0 and up. This is called when creating a new {{Hive}} object:
{code}
  private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException {
    conf = c;
    if (doRegisterAllFns) {
      registerAllFunctionsOnce();
    }
  }
{code}

{{registerAllFunctionsOnce}} will reload all the permanent functions by calling the {{get_all_functions}} API from the megastore. In Spark, we always pass {{doRegisterAllFns}} as true, and this will cause failure:
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions'
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
	... 96 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions'
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833)
{code}

It looks like Spark doesn't really need to call {{registerAllFunctionsOnce}} since it loads the Hive permanent function directly from HMS API. The Hive {{FunctionRegistry}} is only used for loading Hive built-in functions.
Bug: Make config maxShuffledHashJoinLocalMapThreshold fallback to advisoryPartitionSizeInBytes
Status: In Progress | Priority: Major | Resolution: 
Created: 27/May/21 04:28
Affects Version/s: 3.2.0

Make config `spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold` fallback to `spark.sql.adaptive.advisoryPartitionSizeInBytes` so that we can convert SMJ to SHJ in AQE by default.
Bug: Error when reading dictionary-encoded Parquet int column when read schema is bigint
Status: Open | Priority: Major | Resolution: 
Created: 20/May/21 18:06
Affects Version/s: 3.2.0

When reading a dictionary-encoded integer column from a Parquet file, and users specify read schema to be bigint, Spark currently will fail with the following exception:
{code}
java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary
	at org.apache.parquet.column.Dictionary.decodeToLong(Dictionary.java:49)
	at org.apache.spark.sql.execution.datasources.parquet.ParquetDictionary.decodeToLong(ParquetDictionary.java:50)
	at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.getLong(OnHeapColumnVector.java:364)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:344)
{code}

To reproduce:
{code}
    val data = (0 to 10).flatMap(n => Seq.fill(10)(n)).map(i => (i, i.toString))
    withParquetFile(data) { path =>
      val readSchema = StructType(Seq(StructField("_1", LongType)))
      spark.read.schema(readSchema).parquet(path).first()
    }
{code}
Bug: Critical Vulnerabilities: nimbusds_nimbus-jose-jwt 4.41.1 shipped
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 25/May/21 14:29
Affects Version/s: 3.0.2

Vulnerabilities scanner is highlighting following CRITICAL vulnerabilities in {{spark-3.0.2-bin-hadoop3.2}} coming from obsolete {{nimbus-jose-jwt}} {{4.41.1}} :

*  [CVE-2019-17195|https://nvd.nist.gov/vuln/detail/CVE-2019-17195]

This package is shipped within {{jars/nimbus-jose-jwt-4.41.1.jar}}
Bug: UnwrapCastInBinaryComparison support In/InSet predicate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/May/21 02:36
Affects Version/s: 3.2.0

It will not pushdown filters for In/InSet predicates:
{code:scala}
spark.range(50).selectExpr("cast(id as int) as id").write.mode("overwrite").parquet("/tmp/parquet/t1")
spark.read.parquet("/tmp/parquet/t1").where("id in (1L, 2L, 4L)").explain
spark.read.parquet("/tmp/parquet/t1").where("id = 1L or id = 2L or id = 4L").explain
{code}


{noformat}
== Physical Plan ==
*(1) Filter cast(id#5 as bigint) IN (1,2,4)
+- *(1) ColumnarToRow
   +- FileScan parquet [id#5] Batched: true, DataFilters: [cast(id#5 as bigint) IN (1,2,4)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/parquet/t1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>


== Physical Plan ==
*(1) Filter (((id#7 = 1) OR (id#7 = 2)) OR (id#7 = 4))
+- *(1) ColumnarToRow
   +- FileScan parquet [id#7] Batched: true, DataFilters: [(((id#7 = 1) OR (id#7 = 2)) OR (id#7 = 4))], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/parquet/t1], PartitionFilters: [], PushedFilters: [Or(Or(EqualTo(id,1),EqualTo(id,2)),EqualTo(id,4))], ReadSchema: struct<id:int>
{noformat}
Bug: Casting special strings to DATE/TIMESTAMP returns inconsistent results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/21 19:55
Affects Version/s: 3.2.0

When casting the special values "now", "today", "tomorrow", and "yesterday" to DATE/TIMESTAMP, Spark may return inconsistent results.

Looks like Spark runs the expression on each executor, on every row independently. So the results could differ across executors if they have different system time, and across rows because of the resolution of "now".

https://github.com/databricks/runtime/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L876
Bug: Make CatalogString contains nullable information
Status: In Progress | Priority: Major | Resolution: 
Created: 02/May/21 09:39
Affects Version/s: 3.1.1

When trying to cast ArrayType/MapType/StructType from containing NULL to not contain NULL, we will get a confusing failure message.
 Here's a example:
{code:java}
// new UT in CastSuite
test("array casting") {
 val arrayContainsNull = Literal.create(Seq("contains", "null", null),
 ArrayType(StringType, containsNull = true))
 val ret = cast(arrayContainsNull, ArrayType(StringType, containsNull = false))
 assert(ret.resolved === false)
 // scalastyle:off println
 println(ret.typeCheckFailureMessage)
 // scalastyle:on println
}
{code}
Failure massage:
{code:java}
cannot cast array<string> to array<string>{code}
From the above error message, it is not clear why the error is reported, because the data type before and after is the same.
 Therefore, I think we need to specify nullable info for ArrayType/MapType?/StructType in the Catalog String to identify the reason for Cast Failures.
Bug: Introduce new Option in Kafka source to specify minimum number of records to read per trigger
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/May/21 11:22
Affects Version/s: 3.1.1

Kafka source currently provides options to set the maximum number of offsets to read per trigger.

I will like to introduce a new option to specify the minimum number of offsets to read per trigger i.e. *minOffsetsPerTrigger*.

This new option will allow skipping trigger/batch when the number of records available in Kafka is low. This is a very useful feature in cases where we have a sudden burst of data at certain intervals in a day and data volume is low for the rest of the day. Tunning such jobs is difficult as decreasing trigger processing time increasing the number of batches and hence cluster resource usage and adds to small file issues. Increasing trigger processing time adds consumer lag. This will save cluster resources and also help solve small file issues as it is running lesser batches.
 Along with this, I would like to introduce '*maxTriggerDelay*' option which will help to avoid cases of infinite delay in scheduling trigger and the trigger will happen irrespective of records available if the maxTriggerDelay time exceeds the last trigger. It would be an optional parameter with a default value of 15 mins. _This option will be only applicable if minOffsetsPerTrigger is set._

*minOffsetsPerTrigger* option would be optional of course, but once specified it would take precedence over *maxOffestsPerTrigger* which will be honored only after *minOffsetsPerTrigger* is satisfied.
Bug: When addMergerLocation exceed the maxRetainedMergerLocations , we should remove the merger based on merged shuffle data size.
Status: Open | Priority: Major | Resolution: 
Created: 17/May/21 14:36
Affects Version/s: 3.2.0

Now When addMergerLocation exceed the maxRetainedMergerLocations , we just remove the oldest merger, but we'd better remove the merger based on merged shuffle data size. 

We should remove mergers with the largest amount of merged shuffle data, so that the remaining mergers have potentially more disk space to store new merged shuffle data
Bug: Handling new attempt has started error message in BlockPushErrorHandler in client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/21 01:02
Affects Version/s: 3.1.0

In SPARK-33350, a new type of error message is introduced in BlockPushErrorHandler which indicates the PushblockStream message is received after a new application attempt has started. This error message should be correctly handled in client without retrying the block push.
Bug: Investigate the failure of "PVs with local storage" integration test on Docker driver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/May/21 03:55
Affects Version/s: 3.2.0

With https://issues.apache.org/jira/browse/SPARK-34738 integration tests are migrated to docker but "PVs with local storage" was failing so we created a separate test tag in https://github.com/apache/spark/pull/31829 called "persistentVolume" test tag which not used by the dev-run-integration-tests.sh so this way that tests is skipped.

Here we should revert "persistentVolume" and investigate the error.
Bug: partition result is incorrect when insert into partition table with int datatype partition column
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/May/21 02:00
Affects Version/s: 3.0.0

when inserting into partitioned table with int datatype partition column, if partition column value is starting with 0, like 001, get wrong partition result

 

*How to reproduce the problem:*

CREATE TABLE partitiontb04 (id INT, c_string STRING) STORED AS orc; 
 insert into table partitiontb04 values (10001,'test1');
 CREATE TABLE orc_part03(id INT, c_string STRING) partitioned by (p_int int) STORED AS orc;
 insert into table orc_part03 partition (p_int=001) select * from partitiontb04 where id < 10006;
 show partitions orc_part03;

expect result:

p_int=001

 

actural result:

p_int=1
Bug: Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/21 00:52
Affects Version/s: 3.1.0

In the current implementation of RemoteBlockPushResolver, two ConcurrentHashmap are used to store #1 applicationId -> mergedShuffleLocalDirPath #2 applicationId+attemptId+shuffleID -> mergedShuffleParitionInfo. As there are four types of messages: ExecutorRegister, PushBlocks, FinalizeShuffleMerge and ApplicationRemove, will trigger different types of operations within these two hashmaps, it is required to maintain strong consistency about the informations stored in these two hashmaps. Otherwise, either there will be data corruption/correctness issues or memory leak in shuffle server. 

We should come up with systematic way to resolve this, other than spot fixing the potential issues.