av9ash/GitBugs huggingface dataset:
Bug: Planner error when aggregating multiple distinct DECIMAL columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 06:34
Affects Version/s: 3.0.0

Running different DISTINCT decimal aggregations causes a query planner error:
{code:java}
java.lang.RuntimeException: You hit a query analyzer bug. Please report your query to Spark user mailing list.
at scala.sys.package$.error(package.scala:30)
	at org.apache.spark.sql.execution.SparkStrategies$Aggregation$.apply(SparkStrategies.scala:473)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$1(QueryPlanner.scala:67)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:97)
	at org.apache.spark.sql.execution.SparkStrategies.plan(SparkStrategies.scala:74)
	at org.apache.spark.sql.catalyst.planning.QueryPlanner.$anonfun$plan$3(QueryPlanner.scala:82)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:162)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:162)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
{code}
example failing query
{code:java}
import org.apache.spark.util.Utils

// Changing decimal(9, 0) to decimal(8, 0) fixes the problem. Root cause seems to have to do with
// UnscaledValue being used in one of the expressions but not the other.
val df = spark.range(0, 50000, 1, 1).selectExpr(
      "id",
      "cast(id as decimal(9, 0)) as ss_ext_list_price")
val cacheDir = Utils.createTempDir().getCanonicalPath
df.write.parquet(cacheDir)

spark.read.parquet(cacheDir).createOrReplaceTempView("test_table")

spark.sql("""
select
avg(distinct ss_ext_list_price), sum(distinct ss_ext_list_price)
from test_table""").explain
{code}
Bug: AQE local shuffle reader breaks repartitioning for dynamic partition overwrite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 04:02
Affects Version/s: 3.0.0

With AQE, local shuffle reader breaks users' repartitioning for dynamic partition overwrite as in the following case.
{code:java}
test("repartition with local reader") {
  withSQLConf(SQLConf.PARTITION_OVERWRITE_MODE.key -> PartitionOverwriteMode.DYNAMIC.toString,
    SQLConf.SHUFFLE_PARTITIONS.key -> "5",
    SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
    withTable("t") {
      val data = for (
        i <- 1 to 10;
        j <- 1 to 3
      ) yield (i, j)
      data.toDF("a", "b")
        .repartition($"b")
        .write
        .partitionBy("b")
        .mode("overwrite")
        .saveAsTable("t")
      assert(spark.read.table("t").inputFiles.length == 3)
    }
  }
}{code}
-Coalescing shuffle partitions could also break it.-
Bug: Add numRows to metric of BroadcastExchangeExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 16:08
Affects Version/s: 3.1.0

{{numRows}} can be used here: https://github.com/apache/spark/blob/d6a68e0b67ff7de58073c176dd097070e88ac831/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala#L55-L156
Bug: The constrains may grow exponentially in sql optimizer 'InferFiltersFromConstraints', which leads to driver oom
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 28/Sep/20 07:34
Affects Version/s: 3.0.1

 

 

 Consider the case below:
{code:java}
Seq((1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)).toDF("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t").write.saveAsTable("test") 
val df = spark.table("test") 
val df2 = df.filter("a+b+c+d+e+f+g+h+i+j+k+l+m+n+o+p+q+r+s+t > 100") 
val df3 = df2.select('a as 'a1, 'b as 'b1, 'c as 'c1, 'd as 'd1, 'e as 'e1, 'f as 'f1, 'g as 'g1, 'h as 'h1, 'i as 'i1, 'j as 'j1, 'k as 'k1, 'l as 'l1, 'm as 'm1, 'n as 'n1, 'o as 'o1, 'p as 'p1, 'q as 'q1, 'r as 'r1, 's as 's1, 't as 't1) 
val df4 = df3.join(df2, df3("a1") === df2("a")) 
df4.explain(true)
{code}
If you run the this in spark shell, it will got stuck at "df4.explain(true)". The reason is in sql optimizer 'InferFiltersFromConstraints', it will try to infer all the constrains from the plan. And the plan has a constrain contains about 20 columns, each column has an alias. It will try to replace the column with alias, and at the same time keep the origin constrain, that will lead to the constrains grow exponentially. And make driver oom in the end.

The related code:
{code:java}
  /**
   * Generates all valid constraints including an set of aliased constraints by replacing the
   * original constraint expressions with the corresponding alias
   */
  protected def getAllValidConstraints(projectList: Seq[NamedExpression]): Set[Expression] = {
    var allConstraints = child.constraints.asInstanceOf[Set[Expression]]
    projectList.foreach {
      case a @ Alias(l: Literal, _) =>
        allConstraints += EqualNullSafe(a.toAttribute, l)
      case a @ Alias(e, _) =>
        // For every alias in `projectList`, replace the reference in constraints by its attribute.
        allConstraints ++= allConstraints.map(_ transform {
          case expr: Expression if expr.semanticEquals(e) =>
            a.toAttribute
        })
        allConstraints += EqualNullSafe(e, a.toAttribute)
      case _ => // Don't change.
    }    
    allConstraints
  }
{code}
Bug: UnsafeExternalSorter may cause a SparkOutOfMemoryError to be thrown while spilling
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 09:47
Affects Version/s: 2.4.7

Consider the following sequence of events:
 # {{UnsafeExternalSorter}} runs out of space in its pointer array and attempts to allocate a large array to replace the current one.
 # {{TaskMemoryManager}} tries to allocate the memory backing the large array using {{MemoryManager}}, but {{MemoryManager}} is only willing to return most but not all of the memory requested.
 # {{TaskMemoryManager}} asks {{UnsafeExternalSorter}} to spill, which causes {{UnsafeExternalSorter}} to spill the current run to disk, to free its record pages and to reset its {{UnsafeInMemorySorter}}.
 # {{UnsafeInMemorySorter}} frees its pointer array, and tries to allocate a new small pointer array.
 # {{TaskMemoryManager}} tries to allocate the memory backing the small array using {{MemoryManager}}, but {{MemoryManager}} is unwilling to give it any memory, as the {{TaskMemoryManager}} is still holding on to the memory it got for the large array.
 # {{TaskMemoryManager}} again asks {{UnsafeExternalSorter}} to spill, but this time there is nothing to spill.
 # {{UnsafeInMemorySorter}} receives less memory than it requested, and causes a {{SparkOutOfMemoryError}} to be thrown, which causes the current task to fail.

A simple way to fix this is to avoid allocating a new array in {{UnsafeInMemorySorter.reset()}} and to do this on-demand instead.
Bug: Updates the obsoleted entries of attribute mapping in QueryPlan#transformUpWithNewOutput
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 12:20
Affects Version/s: 3.0.1

This ticket aims at fixing corner-case bugs in the `QueryPlan#transformUpWithNewOutput` that is used to propagate updated `ExprId`s in a bottom-up way. Let's say we have a rule to simply assign new `ExprId`s in a projection list like this;
{code}
case class TestRule extends Rule[LogicalPlan] {
  override def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithNewOutput {
    case p @ Project(projList, _) =>
      val newPlan = p.copy(projectList = projList.map { _.transform {
        // Assigns a new `ExprId` for references
        case a: AttributeReference => Alias(a, a.name)()
      }}.asInstanceOf[Seq[NamedExpression]])

      val attrMapping = p.output.zip(newPlan.output)
      newPlan -> attrMapping
  }
}
{code}
Then, this rule is applied into a plan below;
{code}
(3) Project [a#5, b#6]
+- (2) Project [a#5, b#6]
   +- (1) Project [a#5, b#6]
      +- LocalRelation <empty>, [a#5, b#6]
{code}
In the first transformation, the rule assigns new `ExprId`s in `(1) Project` (e.g., a#5 AS a#7, b#6 AS b#8). In the second transformation, the rule corrects the input references of `(2) Project`  first by using attribute mapping given from `(1) Project` (a#5->a#7 and b#6->b#8) and then assigns new `ExprId`s (e.g., a#7 AS a#9, b#8 AS b#10). But, in the third transformation, the rule fails because it tries to correct the references of `(3) Project` by using incorrect attribute mapping (a#7->a#9 and b#8->b#10) even though the correct one is a#5->a#9 and b#6->b#10.
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (Oracle dialect)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 12:11
Affects Version/s: 3.1.0

Override the default SQL strings for:
- ALTER TABLE ADD COLUMN
- ALTER TABLE UPDATE COLUMN TYPE
- ALTER TABLE UPDATE COLUMN NULLABILITY

in the following Oracle JDBC dialect according to official documentation.

Write Oracle integration tests for JDBC.
Bug: Optimize to_json + named_struct + from_json expression chain
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/20 23:53
Affects Version/s: 3.1.0

Due to query optimization, it is possibly to generate inefficient json expression chain like to_json(named_struct(col1, from_json(json).col1, col2, from_json(json).col2), ....). We could optimize it.
Bug: empty string should be consistent for schema name in SparkGetSchemasOperation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/20 08:48
Affects Version/s: 3.0.1

When the schema name is empty string, it is considered as ".*" and can match all databases in the catalog.
But when it can not match the global temp view as it is not converted to ".*"
Bug: Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 22:06
Affects Version/s: 3.1.0

Apache Spark 3.0 switches its Hive execution version from 1.2 to 2.3, but we still provide the unofficial forked Hive 1.2 version from our distribution. This issue aims to remove it from Apache Spark 3.1.0.
{code}
spark-3.0.1-bin-hadoop2.7-hive1.2.tgz
spark-3.0.1-bin-hadoop2.7-hive1.2.tgz.asc
spark-3.0.1-bin-hadoop2.7-hive1.2.tgz.sha512 
{code}
Bug: Add timestamp_seconds to SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/20 06:05
Affects Version/s: 3.1.0

Since Spark 3.1 we disallow casting numeric to timestamp by default (SPARK-31710) and Scala and Python API provide {{TIMESTAMP_SECONDS}}, which can be used instead.

It should be added to SparkR as well.
Bug: Remove knownManagers hardcoded list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/20 17:17
Affects Version/s: 2.4.7

Spark has a hardcode list to contain known shuffle managers, which has two values now. It does not contain user's custom shuffle manager which is set through Spark config "spark.shuffle.manager".

 

We hit issue when set "spark.shuffle.manager" with our own shuffle manager plugin (Uber Remote Shuffle Service implementation, [https://github.com/uber/RemoteShuffleService]). Other users will hit same issue when they implement their own shuffle manager.

 

Need to add "spark.shuffle.manager" config value to the known managers list as well.

 

The know managers list is in code:

common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
{quote}private final List<String> knownManagers = Arrays.asList(
   "org.apache.spark.shuffle.sort.SortShuffleManager",
   "org.apache.spark.shuffle.unsafe.UnsafeShuffleManager");
{quote}
Bug: Handle Option.empty v1.ExecutorSummary#peakMemoryMetrics
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/20 17:05
Affects Version/s: 3.0.1

When {{peakMemoryMetrics}} in {{ExecutorSummary}} is {{Option.empty}}, then the {{ExecutorMetricsJsonSerializer#serialize}} method does not execute the {{jsonGenerator.writeObject}} method. This causes the json to be generated with {{peakMemoryMetrics}} key added to the serialized string, but no corresponding value.
This causes an error to be thrown when it is the next key {{attributes}} turn to be added to the json:
{{com.fasterxml.jackson.core.JsonGenerationException: Can not write a field name, expecting a value.}}

{{}}

{{}}
Bug: Fix CodeGen fallback issue of UDFSuite in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 13:40
Affects Version/s: 3.1.0

After SPARK-32851 set `CODEGEN_FACTORY_MODE` to `CODEGEN_ONLY` in SharedSparkSessionBase of sparkConf to construction SparkSession in Test,

The test suite `SPARK-32459: UDF should not fail on WrappedArray` in s.sql.UDFSuite exposed a codegen fallback issue in Scala 2.13 as follow:
{code:java}
- SPARK-32459: UDF should not fail on WrappedArray *** FAILED ***
Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 99: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 47, Column 99: No applicable constructor/method found for zero actual parameters; candidates are: "public scala.collection.mutable.Builder scala.collection.mutable.ArraySeq$.newBuilder(java.lang.Object)", "public scala.collection.mutable.Builder scala.collection.mutable.ArraySeq$.newBuilder(scala.reflect.ClassTag)", "public abstract scala.collection.mutable.Builder scala.collection.EvidenceIterableFactory.newBuilder(java.lang.Object)"

{code}
Bug: pyspark reading csv files with utf_16le encoding
Status: Reopened | Priority: Major | Resolution: 
Created: 22/Sep/20 15:49
Affects Version/s: 2.4.7

If you have a file encoded in utf_16le or utf_16be and try to use spark.read.csv("<file_name>", encoding="utf_16le") the dataframe isn't rendered properly

if you use python decoding like:

prdd = spark_session._sc.binaryFiles(path_url).values().flatMap(lambda x : x.decode("utf_16le").splitlines())

and then do spark.read.csv(prdd), then it works.
Bug: Better error messages when PySpark Java Gateway Crashes
Status: Open | Priority: Major | Resolution: 
Created: 30/Sep/20 21:54
Affects Version/s: 2.4.7

Currently the startup works by opening the Gateway process and waiting until the the process has written the conn_info_file. Once the conn_file is written it proceeds to attempt to connect to the port.

This connection can succeed and the process can start normally, but if the gateway process dies or is killed the error that the user ends up getting is a confusing "connection_failed" style error like

{code}
Traceback (most recent call last):
  File "/usr/lib/spark-packages/spark2.4.4/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque
{code}

Since we have a handle on the py4j process, we should probably check whether it has terminated before surfacing any exceptions like this. 

CC [~holden]
Bug: partition length is wrong after merge partition segments in BypassMergeSortShuffleWriter
Status: In Progress | Priority: Major | Resolution: 
Created: 29/Sep/20 09:15
Affects Version/s: 2.4.6

The segment file may be empty even after writer committing caused by disk/kernel in heavy load cluster. Compare segment length with copied length to guard it.

A data file might be empty even after  DiskBlockObjectWriter committing it in BypassMergeSortShuffleWriter, returned wrong lengths in writePartitionedFile used by MapStatus, and then cause data lost.

This is related to disk/kernel but we can avoid it in spark without any performance loss. We can compare partitionWriterSegments[i].length with the length[i] after Utils.copyStream.

I added some logs and reproduce the issue,

The log when this issue happened
{code:java}
20/09/28 00:42:44 INFO sort.BypassMergeSortShuffleWriter: partitionWriterSegments[0]: (name=temp_shuffle_38244ef5-8e97-4428-97b8-feffc16fc9f7, offset=0, length=1462) 
20/09/28 00:42:46 INFO sort.BypassMergeSortShuffleWriter: File length: 0 
20/09/28 00:42:46 INFO sort.BypassMergeSortShuffleWriter: Copied stream length: 0{code}
 

The peer log when this issue didn't happen
{code:java}
20/09/28 10:11:45 INFO sort.BypassMergeSortShuffleWriter: partitionWriterSegments[0]: (name=temp_shuffle_f6937469-39fd-4576-b40e-69f4276cc8e4, offset=0, length=1462)
20/09/28 10:11:45 INFO sort.BypassMergeSortShuffleWriter: File length: 1462
20/09/28 10:11:45 INFO sort.BypassMergeSortShuffleWriter: Copied stream length: 1462
{code}
Bug: Should  throw SparkException if broadcast large table
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 30/Sep/20 00:06
Affects Version/s: 3.1.0

Spark 3.1:
{noformat}
20/09/29 17:04:40 WARN TaskMemoryManager: Failed to allocate a page (16777216 bytes), try again.
20/09/29 17:04:45 WARN TaskMemoryManager: Failed to allocate a page (16777216 bytes), try again.
{noformat}


Spark 3.0.1:
{noformat}
Caused by: org.apache.spark.SparkException: Cannot broadcast the table that is larger than 8GB: 10 GB
	at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.$anonfun$relationFuture$1(BroadcastExchangeExec.scala:145)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:182)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
{noformat}
Bug: Support top-N sort for Spark SQL window function
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/20 04:57
Affects Version/s: 3.0.1

In Spark SQL, there are two types of sort execution, *_SortExec_* and *_TakeOrderedAndProjectExec_* . *_SortExec_* is a general sorting execution and it does not support top-N sort. ​*_TakeOrderedAndProjectExec_* is the execution for top-N sort in Spark. Spark SQL rank window function needs to sort the data locally and it relies on the execution plan *_SortExec_* to sort the data in each physical data partition. When the filter of the window rank (e.g. rank <= 100) is specified in a user's query, the filter can actually be pushed down to the SortExec and then we let SortExec operates top-N sort. Right now SortExec does not support top-N sort and we need to extend the capability of SortExec to support top-N sort. 

 
This Jira has dependency on another existing Jira
https://issues.apache.org/jira/browse/SPARK-32096
Bug: Fix compute statistics issue
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/20 15:35
Affects Version/s: 3.0.0

The statistics may be incorrect if it contains {{0}}.
https://github.com/apache/spark/blob/98be8953c75c026c1cb432cc8f66dd312feed0c6/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/SizeInBytesOnlyStatsPlanVisitor.scala#L54-L58
Bug: Compute the current date only once
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/20 08:20
Affects Version/s: 3.0.2

According to the doc for current_date(), it must compute the current date at the start of query evaluation: http://spark.apache.org/docs/latest/api/sql/#current_date but it can compute it multiple times: https://github.com/apache/spark/blob/0df8dd60733066076967f0525210bbdb5e12415a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala#L85
Bug: TreeNode.nodeName should not throw malformed class name error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Sep/20 21:09
Affects Version/s: 2.4.0

Similar to SPARK-32238, the use of {{java.lang.Class.getSimpleName}} in {{TreeNode.nodeName}} is problematic because Scala classes may trigger {{java.lang.InternalError: Malformed class name}}.

This happens more often when using nested classes in Scala (or declaring classes in Scala REPL which implies class nesting).

Note that on newer versions of JDK the underlying malformed class name no longer reproduces, so it's less of an issue there. But on JDK8u this problem still exists so we still have to fix it.
Bug: Pass all `mllib` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 10:05
Affects Version/s: 3.1.0

There are 51 Scala test and 3 java test Failed of `mllib` module, the failed case as follow:

*Java:*
 * org.apache.spark.mllib.fpm.JavaPrefixSpanSuite ( 1 FAILED)
 * org.apache.spark.ml.feature.JavaVectorIndexerSuite ( 1 FAILED)
 * org.apache.spark.ml.feature.JavaWord2VecSuite ( 1 FAILED)

*Scala:*
 * MatrixFactorizationModelSuite ( 1 FAILED)
 * LDASuite ( 1 FAILED)
 * MLTestSuite ( 1 FAILED)
 * PrefixSpanSuite ( 1 FAILED)
 * BucketedRandomProjectionLSHSuite ( 3 FAILED)
 * Word2VecSuite ( 3 FAILED)
 * Word2VecSuite ( 5 FAILED)
 * MinHashLSHSuite ( 3 FAILED)
 * DecisionTreeSuite ( 1 FAILED)
 * FPGrowthSuite ( 2 FAILED)
 * NaiveBayesSuite ( 2 FAILED)
 * NGramSuite ( 4 FAILED)
 * RFormulaSuite ( 4 FAILED)
 * GradientBoostedTreesSuite ( 1 FAILED)
 * StopWordsRemoverSuite ( 10 FAILED)
 * RandomForestSuite ( 1 FAILED)
 * PrefixSpanSuite ( 4 FAILED)
 * StringIndexerSuite ( 2 FAILED)
 * IDFSuite ( 1 FAILED)
 * RandomForestRegressorSuite ( 1 FAILED)
Bug: Add DataStreamReader.table API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Sep/20 06:21
Affects Version/s: 3.1.0

This ticket aims to add a new `table` API in DataStreamReader, which is similar to the table API in DataFrameReader. Users can directly use this API to get a Streaming DataFrame on a table. Below is a simple example:

Application 1 for initializing and starting the streaming job:
{code:java}
val path = "/home/yuanjian.li/runtime/to_be_deleted"
val tblName = "my_table"

// Write some data to `my_table`
spark.range(3).write.format("parquet").option("path", path).saveAsTable(tblName)

// Read the table as a streaming source, write result to destination directory
val table = spark.readStream.table(tblName)
table.writeStream.format("parquet").option("checkpointLocation", "/home/yuanjian.li/runtime/to_be_deleted_ck").start("/home/yuanjian.li/runtime/to_be_deleted_2")
{code}
Application 2 for appending new data:
{code:java}
// Append new data into the path
spark.range(5).write.format("parquet").option("path", "/home/yuanjian.li/runtime/to_be_deleted").mode("append").save(){code}
Check result:
{code:java}
// The desitination directory should contains all written data
spark.read.parquet("/home/yuanjian.li/runtime/to_be_deleted_2").show()
{code}