av9ash/GitBugs huggingface dataset:
Bug: Remove `Spark Packaging` jobs from Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 18:31
Affects Version/s: 2.4.8

SPARK-33675 migrated snapshot publishing to `GitHub Action`.

Now, we can remove the jobs from Jenkins.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/
Bug: Avoid generating small files for INSERT INTO TABLE from  VALUES
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/20 08:27
Affects Version/s: 3.1.0

How to reproduce this issue:

{code:sql}
create table t1(id int) stored as textfile;
insert into table t1 values (1), (2), (3), (4), (5), (6), (7), (8);
{code}

It will generate these files:

{noformat}
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00000-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00001-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00002-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00003-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00004-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00005-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00006-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
-rwxr-xr-x 1 root root 2 Nov 30 23:07 part-00007-76a5ddf9-10df-41f8-ac19-8186449d958d-c000
{noformat}
Bug: Spark sql drop non-existent table will not report an error after failure
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Dec/20 03:22
Affects Version/s: 2.4.0

In the migration of spark sql to hive sql, in order to reduce user changes, spark sql will report an error when executing a table where drop does not exist, but hive sql will not. In order to ensure that the grammar can be executed normally, no error is reported when changing to a table that does not exist in drop.
Bug: Unable to load RandomForestClassificationModel trained in Spark 2.x
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/Dec/20 16:38
Affects Version/s: 3.0.1

When attempting to load a RandomForestClassificationModel that was trained in Spark 2.x using Spark 3.x, an exception is raised:

{code:python}
...
    RandomForestClassificationModel.load('/path/to/my/model')
  File "/usr/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 330, in load
  File "/usr/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 291, in load
  File "/usr/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 280, in load
  File "/usr/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/usr/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 134, in deco
  File "<string>", line 3, in raise_from
pyspark.sql.utils.AnalysisException: No such struct field rawCount in id, prediction, impurity, impurityStats, gain, leftChild, rightChild, split;
{code}

There seems to be a schema incompatibility between the trained model data saved by Spark 2.x and the expected data for a model trained in Spark 3.x

If this issue is not resolved, users will be forced to retrain any existing random forest models they trained in Spark 2.x using Spark 3.x before they can upgrade
Bug: Suggest using datetime conversion functions for invalid ANSI casting
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Dec/20 04:15
Affects Version/s: 3.1.0

In ANSI mode, explicit cast between DateTime types and Numeric types is not allowed.
As of now, we have introduced new functions UNIX_SECONDS/UNIX_MILLIS/UNIX_MICROS/UNIX_DATE/DATE_FROM_UNIX_DATE, we can show suggestions to users so that they can complete these type conversions precisely and easily in ANSI mode.
Bug: Make spark.archives working in Kubernates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Dec/20 04:50
Affects Version/s: 3.1.0

{{--archive}} submit option and {{spark.archives}} configuration were added at SPARK-33530.
Looks like there is a bug. I found this while working on adding an IT test. Please refer the PR for more details.
Bug: Task not started after filtering
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 01/Dec/20 10:35
Affects Version/s: 2.4.7

Hello i have problem with big memory used ~2000gb hdfs stack. With 300gb memory used task starting and complete, but we need use unlimited stack. Please help

 

!VlwWJ.png|width=644,height=150!

 

!mgg1s.png|width=651,height=182!

 
 This my code:
  {{var filteredRDD = sparkContext.emptyRDD[String]
 for (path<- pathBuffer)

{ val someRDD = sparkContext.textFile(path) if (isValidRDD(someRDD)) filteredRDD = filteredRDD.++(someRDD.filter(row =>\{...}

) 
 }
 hiveService.insertRDD(filteredRDD.repartition(10), outTable)}}

 

been other way. When i got StackOverflowError after many iteration spark

 
  {{java.lang.StackOverflowError
 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2303)
 at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2596)
 at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2606)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1319)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
 at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345)
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)}}
 \{{}}
 \{{}}
 How i must build my code with repartitional and persist\coalesce for to nodes not crashes?

I tried to rebuild the program in different ways, transferring repartitioning and saving in memory / disk inside the loop, installed a large number of partitions - 200.

The program either hangs on the “repartition” stage or crashes into error code 143 (outOfMemory), throwing a stackOverflowError in a strange way.
Bug: to_date doesn't behave as documented
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/Dec/20 14:24
Affects Version/s: 3.0.1

I'm trying to use to_date on a string formatted as "10/31/20".
Expected output is "2020-10-31".
Actual output is "0020-01-31".

The [documentation|https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html] suggests 2020 or 20 as input for "y".

Example below. Expected behaviour is included in the udf.

{code:scala}
import java.sql.Date

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.{to_date, udf}

object ToDate {
  val toDate = udf((date: String) => {
    val split = date.split("/")
    val month = "%02d".format(split(0).toInt)
    val day = "%02d".format(split(1).toInt)
    val year = split(2).toInt + 2000

    Date.valueOf(s"${year}-${month}-${day}")
  })

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[2]").getOrCreate()
    spark.sparkContext.setLogLevel("ERROR")
    import spark.implicits._

    Seq("1/1/20", "10/31/20")
      .toDF("raw")
      .withColumn("to_date", to_date($"raw", "m/d/y"))
      .withColumn("udf", toDate($"raw"))
      .show
  }
}

{code}
Bug: ALTER TABLE .. DROP PARTITION doesn't refresh cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Dec/20 10:28
Affects Version/s: 3.0.1

Here is the example to reproduce the issue:
{code:sql}
spark-sql> CREATE TABLE tbl1 (col0 int, part0 int) USING parquet PARTITIONED BY (part0);
spark-sql> INSERT INTO tbl1 PARTITION (part0=0) SELECT 0;
spark-sql> INSERT INTO tbl1 PARTITION (part0=1) SELECT 1;
spark-sql> CACHE TABLE tbl1;
spark-sql> SELECT * FROM tbl1;
0	0
1	1
spark-sql> ALTER TABLE tbl1 DROP PARTITION (part0=0);
spark-sql> SELECT * FROM tbl1;
0	0
1	1
{code}
Bug: Support user-defined script command wrapper for more use case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/20 14:47
Affects Version/s: 3.2.0

In Hive script transform, we can use `USING xxx.py` but in Spark we will got error 
{code:java}
Job aborted due to stage failure: Task 17 in stage 530.0 failed 4 times, most recent failure: Lost task 17.3 in stage 530.0 (TID 38639, host, executor 339): org.apache.spark.SparkException: Subprocess exited with status 127. Error: /bin/bash: xxx.py: can't find the command
{code}
Bug: Full support of V2 table creation in Structured Streaming writer path
Status: Open | Priority: Major | Resolution: 
Created: 03/Dec/20 03:35
Affects Version/s: 3.2.0

Currently, we want to add support of creating if not exists in DataStreamWriter.toTable API. Since the file format in streaming doesn't support DSv2 for now, the current implementation mainly focuses on V1 support. We need more work to do for the full support of V2 table creation.
Bug: Join queries across different datasources (hive and hive)
Status: Open | Priority: Major | Resolution: 
Created: 01/Dec/20 06:41
Affects Version/s: 3.0.1

spark3.0 datasource v2 has supported join queries across different datasource like mysql and hive, it's very convenient to handle data from different datasource. But in our scene，sometimes we need handle Hive data of different clusters，so l want to kown if there is a plan to support multi hive datadource in later spark version
Bug: Cannot connect to spark hive after session timeout
Status: Open | Priority: Major | Resolution: 
Created: 31/Dec/20 01:54
Affects Version/s: 2.3.1

Test with hive setting:
* hive.server2.idle.session.timeout=60000
* hive.server2.session.check.interval=10000
* hive.server2.thrift.max.worker.threads=2
* hive.server2.thrift.min.worker.threads=1

1. Connect with user test1 and hold the connection.
2. Wait  1 minute, Connect with user test2
3. Seen from web ui  http://localhost:4040/sqlserver/， wait session for  test1 finished 
4. Connect with user test3, connect was refused 
  Task has been rejected by ExecutorService 10 times till timedout, reason: java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.TThreadPoolServer$WorkerProcess@71b1ea28 rejected from java.util.concurrent.ThreadPoolExecutor@716d9872[Running, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 2]
Seems the session was not removed, while UI show finished time for session
Bug: ALTER TABLE .. DROP PARTITION doesn't invalidate the cache
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Dec/20 09:38
Affects Version/s: 2.4.7

1. Create a partitioned table:
{code:scala}
scala> sql("CREATE TABLE tbl2 (id int, part int) USING parquet PARTITIONED BY (part)")
res31: org.apache.spark.sql.DataFrame = []
{code}
2. Create partitions:
{code:scala}
scala> sql("INSERT INTO tbl2 PARTITION (part=0) SELECT 0")
res32: org.apache.spark.sql.DataFrame = []

scala> sql("INSERT INTO tbl2 PARTITION (part=1) SELECT 1")
res33: org.apache.spark.sql.DataFrame = []
{code}
3. Create dataframes from the table:
{code:scala}
scala> val df1 = spark.table("tbl2")
df1: org.apache.spark.sql.DataFrame = [id: int, part: int]

scala> val df2 = spark.table("tbl2")
df2: org.apache.spark.sql.DataFrame = [id: int, part: int]
{code}
4. Cache df2 and fill in the cache:
{code:scala}
scala> df2.cache
res34: df2.type = [id: int, part: int]

scala> df1.show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+


scala> df2.show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+
{code}
5. Drop a partition:
{code:scala}
scala> sql("ALTER TABLE tbl2 DROP PARTITION(part=0)")
res37: org.apache.spark.sql.DataFrame = []

scala> df1.show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+


scala> df2.show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+
{code}

The same without caching:
{code:scala}
scala> sql("CREATE TABLE tbl3 (id int, part int) USING parquet PARTITIONED BY (part)")
res40: org.apache.spark.sql.DataFrame = []

scala> sql("INSERT INTO tbl3 PARTITION (part=0) SELECT 0")
res41: org.apache.spark.sql.DataFrame = []

scala> sql("INSERT INTO tbl3 PARTITION (part=1) SELECT 1")
res42: org.apache.spark.sql.DataFrame = []

scala> val df3 = spark.table("tbl3")
df3: org.apache.spark.sql.DataFrame = [id: int, part: int]

scala> df3.show(false)
+---+----+
|id |part|
+---+----+
|0  |0   |
|1  |1   |
+---+----+


scala> sql("ALTER TABLE tbl3 DROP PARTITION(part=0)")
res44: org.apache.spark.sql.DataFrame = []

scala> df3.show(false)
+---+----+
|id |part|
+---+----+
|1  |1   |
+---+----+
{code}
Bug: Spark SQL no serde row format field delimit default is '\u0001'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Dec/20 05:11
Affects Version/s: 3.2.0

For same sql
{code:java}
SELECT TRANSFORM(a, b, c, null)
ROW FORMAT DELIMITED
USING 'cat' 
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '&'
FROM (select 1 as a, 2 as b, 3  as c) t

{code}

!image-2020-12-29-13-11-31-336.png!

 

!image-2020-12-29-13-11-45-734.png!
Bug: java.time.Instant and java.time.LocalDate not handled in org.apache.spark.sql.jdbc.JdbcDialect#compileValue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Dec/20 12:45
Affects Version/s: 3.0.1

When using the new java time API (spark.sql.datetime.java8API.enabled=true) LocalDate and Instant aren't handled in org.apache.spark.sql.jdbc.JdbcDialect#compileValue so exceptions are thrown when they are used in filters since a filter condition would be translated to something like this: "valid_from" > 2020-12-21T11:40:24.413681Z.

To reproduce you can write a simple filter like where dataset is backed by a DB table (in my case PostgreSQL): 

dataset.filter(current_timestamp().gt(col(VALID_FROM)))

The error and stacktrace:

Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "T11"Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "T11"  Position: 285 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(JDBCRDD.scala:304) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349) at org.apache.spark.rdd.RDD.iterator(RDD.scala:313) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Bug: Simplify conditional in predicate
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 20/Dec/20 15:54
Affects Version/s: 3.2.0

The use case is:
{noformat}
spark.sql("create table t1 using parquet as select id as a, id as b from range(10)")
spark.sql("select * from t1 where CASE WHEN a > 2 THEN b + 10 END > 5").explain()
{noformat}
Before this pr:
{noformat}
== Physical Plan ==
*(1) Filter CASE WHEN (a#3L > 2) THEN ((b#4L + 10) > 5) END
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[a#3L,b#4L] Batched: true, DataFilters: [CASE WHEN (a#3L > 2) THEN ((b#4L + 10) > 5) END], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark.sql.DataF..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>

{noformat}
After this pr:
{noformat}
== Physical Plan ==
*(1) Filter (((isnotnull(a#3L) AND isnotnull(b#4L)) AND (a#3L > 2)) AND ((b#4L + 10) > 5))
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[a#3L,b#4L] Batched: true, DataFilters: [isnotnull(a#3L), isnotnull(b#4L), (a#3L > 2), ((b#4L + 10) > 5)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark.sql.DataF..., PartitionFilters: [], PushedFilters: [IsNotNull(a), IsNotNull(b), GreaterThan(a,2)], ReadSchema: struct<a:bigint,b:bigint>

{noformat}
Bug: Extend Dynamic Partition Pruning Support to DataSources
Status: Open | Priority: Major | Resolution: 
Created: 11/Dec/20 17:08
Affects Version/s: 3.0.1

The implementation of Dynamic Partition Pruning  (DPP) in Spark is [specific|https://github.com/apache/spark/blob/fb2e3af4b5d92398d57e61b766466cc7efd9d7cb/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala#L59-L64] to HadoopFSRelation. As a result, DPP is not triggered for queries that use data sources. 

The DataSource v2 readers can expose the partition metadata. Can we use this metadata and extend DPP to work on data sources as well?

Would appreciate thoughts or corner cases we need to handle.
Bug: Add a vectorized BLAS implementation 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Dec/20 20:26
Affects Version/s: 3.2.0

Introduce a VectorizedBLAS class that implements hardware-accelerated BLAS operations. This feature is hidden behind the "vectorized" profile that you can enable by passing "-Pvectorized" to sbt or maven.

The Vector API has been introduced in JDK 16. Following a discussion on the mailing list, this API is introduced transparently and needs to be enabled explicitly.
Bug: Adaptive shuffle merge finalization for push-based shuffle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Dec/20 05:16
Affects Version/s: 3.1.0

SPARK-32920 implements a simple approach for shuffle merge finalization, which transitions from shuffle map stage to reduce stage when push-based shuffle is enabled.

This simple approach basically waits for a static period of time after all map tasks are finished before initiating shuffle merge finalization. This approach is not very ideal to handle jobs with varying size of shuffles. For a small shuffle, we want the merge finalization to happen as early and as quickly as possible. For a large shuffle, we might want to wait for longer time to achieve a better merge ratio. A static configuration for the entire job cannot adapt to such varying needs.

This raises the need for adaptive shuffle merge finalization, where the amount of time to wait before merge finalization is adaptive to the size of the shuffle. We have implemented an effective adaptive shuffle merge finalization mechanism, which introduces 2 more config parameters: spark.shuffle.push.minShuffleSizeToWait and spark.shuffle.push.minPushRatio. Together with spark.shuffle.push.finalize.time, the adaptive shuffle merge finalization works in the following way:
 # Whenever a ShuffleBlockPusher finishes pushing all the shuffle data generated by a mapper, it notifies the Spark driver about this.
 # When the Spark driver receives notification of a completed shuffle push, it updates state maintained in the corresponding ShuffleDependency.
 # If the ratio of completed pushes (# completed pushes / # map tasks) exceeds minPushRatio, the driver would then immediately schedule shuffle merge finalization.
 # If the driver receives notification that all map tasks have finished first, it would then gather the size of the shuffle from MapOutputStatistics. If the total shuffle size is smaller than minSizeToWait, the driver would ignore the pushed shuffle partition and treat the shuffle as a regular shuffle and start schedule the reduce stage. It would also asynchronously schedule shuffle merge finalization immediately, but ignores all the responses.
 # If the total shuffle size is larger than minSizeToWait, the driver would schedule shuffle merge finalization after waiting for a period of time of finalize.time. If during this wait time the driver receives enough push completion notification to reach minPushRatio, the driver would then reschedule the shuffle merge finalization for immediate execution.

In addition to the above, per SPARK-36530, we should also check if no block gets pushed because all blocks are larger than spark.shuffle.push.maxBlockSizeToPush. If so, we should also skip shuffle merge finalization. The information about whether any blocks from a mapper get pushed can be included in the new RPC between Spark executor/driver to notify driver about push completion.
Bug: Improve caching of MergeStatus on the executor side to save memory
Status: In Progress | Priority: Major | Resolution: 
Created: 14/Dec/20 18:57
Affects Version/s: 3.1.0

In MapOutputTrackerWorker, it would cache the retrieved MapStatus or MergeStatus array for a given shuffle received from the driver in memory so that all tasks doing shuffle fetch for that shuffle can reuse the cached metadata.

However, different from MapStatus array, where each task would need to access every single instance in the array, each task would only need one or just a few MergeStatus objects from the MergeStatus array depending on which shuffle partitions the task is processing.

For large shuffles with 10s or 100s of thousands of shuffle partitions, caching the entire deserialized and decompressed MergeStatus array on the executor side, while perhaps only 0.1% of them are going to be used by the tasks running in this executor is a huge waste of memory.

We could improve this by caching the serialized and compressed bytes for MergeStatus array instead and only cache the needed deserialized MergeStatus object on the executor side. In addition to saving memory, it also helps with reducing GC pressure on executor side.
Bug: Allow insert overwrite same table with static partition if using dynamic partition overwrite mode
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Dec/20 06:50
Affects Version/s: 3.0.1

{code:sql}
CREATE TABLE insertTable(i int, part int) USING PARQUET
PARTITIONED BY (part);

INSERT INTO TABLE insertTable PARTITION(part=1) SELECT 1;

SET spark.sql.sources.partitionOverwriteMode=DYNAMIC;

INSERT OVERWRITE TABLE insertTable PARTITION(part=1)
SELECT i + 1 FROM insertTable WHERE part = 1;
{code}

The SQL will fail because Spark complains  "Cannot overwrite a path that is also being read from."