av9ash/GitBugs huggingface dataset:
Bug: Collapse two adjacent windows with semantically-same partition/order
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/23 02:39
Affects Version/s: 3.2.3

Extend the CollapseWindow rule to collapse Window nodes with semantically-same partition/order but the qualifiers are different.

 
{code:java}
select a, b, c, row_number() over (partition by a order by b) as d from
( select a, b, rank() over (partition by a order by b) as c from t1) t2
== Optimized Logical Plan ==
before
Window [row_number() windowspecdefinition(a#11, b#12 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS d#26], [a#11], [b#12 ASC NULLS FIRST]
+- Window [rank(b#12) windowspecdefinition(a#11, b#12 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS c#25], [a#11], [b#12 ASC NULLS FIRST]
   +- InMemoryRelation [a#11, b#12], StorageLevel(disk, memory, deserialized, 1 replicas)
         +- *(1) Project [_1#6 AS a#11, _2#7 AS b#12]
            +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#6, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#7]
               +- *(1) MapElements org.apache.spark.sql.DataFrameSuite$$Lambda$1517/1628848368@3a479fda, obj#5: scala.Tuple2
                  +- *(1) DeserializeToObject staticinvoke(class java.lang.Long, ObjectType(class java.lang.Long), valueOf, id#0L, true, false, true), obj#4: java.lang.Long
                     +- *(1) Range (0, 10, step=1, splits=2)

after
Window [rank(b#12) windowspecdefinition(a#11, b#12 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS c#25, row_number() windowspecdefinition(a#11, b#12 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS d#26], [a#11], [b#12 ASC NULLS FIRST]
+- InMemoryRelation [a#11, b#12], StorageLevel(disk, memory, deserialized, 1 replicas)
      +- *(1) Project [_1#6 AS a#11, _2#7 AS b#12]
         +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1 AS _1#6, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2 AS _2#7]
            +- *(1) MapElements org.apache.spark.sql.DataFrameSuite$$Lambda$1518/1928028672@4d7a64ca, obj#5: scala.Tuple2
               +- *(1) DeserializeToObject staticinvoke(class java.lang.Long, ObjectType(class java.lang.Long), valueOf, id#0L, true, false, true), obj#4: java.lang.Long
                  +- *(1) Range (0, 10, step=1, splits=2){code}
Bug: Provide more details in TaskEndReason s for tasks killed by TaskScheduler.cancelTasks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 11:48
Affects Version/s: 3.5.0

Currently tasks killed by `TaskScheduler.cancelTasks` will have a `TaskEndReason` "TaskKilled (Stage cancelled)". We should do better at differentiating reasons for stage cancellations (e.g. user-initiated or caused by task failures in the stage).
Bug: [PROTOBUF] Recursive field handling is incompatible with delta
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Feb/23 20:53
Affects Version/s: 3.4.0

Protobuf deserializer (`from_protobuf()` function()) optionally supports recursive fields by limiting the depth to certain level. See example below. It assigns a 'NullType' for such a field when allowed depth is reached. 

It causes a few issues. E.g. a repeated field as in the following example results in a Array field with 'NullType'. Delta does not support null type in a complex type.

Actually `Array[NullType]` is not really useful anyway.

How about this fix: Drop the recursive field when the limit reached rather than using a NullType. 

The example below makes it clear:

Consider a recursive Protobuf:

 
{code:python}
message TreeNode {
  string value = 1;
  repeated TreeNode children = 2;
}
{code}
Allow depth of 2: 

 
{code:python}
   df.select(
    'proto',
     messageName = 'TreeNode',
     options = { ... "recursive.fields.max.depth" : "2" }
  ).printSchema()
{code}
Schema looks like this:
{noformat}
root
|– from_protobuf(proto): struct (nullable = true)|
| |– value: string (nullable = true)|
| |– children: array (nullable = false)|
| | |– element: struct (containsNull = false)|
| | | |– value: string (nullable = true)|
| | | |– children: array (nullable = false)|
| | | | |– element: struct (containsNull = false)|
| | | | | |– value: string (nullable = true)|
| | | | | |– children: array (nullable = false). [ === Proposed fix: Drop this field === ]|
| | | | | | |– element: void (containsNull = false) [ === NOTICE 'void' HERE === ] 
{noformat}
When we try to write this to a delta table, we get an error:
{noformat}
AnalysisException: Found nested NullType in column from_protobuf(proto).children which is of ArrayType. Delta doesn't support writing NullType in complex types.
{noformat}
 
We could just drop the field 'element' when recursion depth is reached. It is simpler and does not need to deal with NullType. We are ignoring the value anyway. There is no use in keeping the field.

Another issue is setting for 'recursive.fields.max.depth': It is not enforced correctly. '0' does not make sense.
Bug: DataFrame.drop should handle duplicated columns properly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Feb/23 06:19
Affects Version/s: 3.4.0


{code:java}
>>> df.join(df2, df.name == df2.name, 'inner').show()
+---+----+------+----+
|age|name|height|name|
+---+----+------+----+
| 16| Bob|    85| Bob|
| 14| Tom|    80| Tom|
+---+----+------+----+

>>> df.join(df2, df.name == df2.name, 'inner').drop('name').show()
+---+------+
|age|height|
+---+------+
| 16|    85|
| 14|    80|
+---+------+

{code}
Bug: [YARN] OMP_NUM_THREADS not set to number of executor cores by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/23 07:28
Affects Version/s: 3.3.2

Run this PySpark script with `spark.executor.cores=1`
{code:python}
import os
from pyspark.sql import SparkSession
from pyspark.sql.functions import udf

spark = SparkSession.builder.getOrCreate()

var_name = 'OMP_NUM_THREADS'

def get_env_var():
  return os.getenv(var_name)

udf_get_env_var = udf(get_env_var)
spark.range(1).toDF("id").withColumn(f"env_{var_name}", udf_get_env_var()).show(truncate=False)
{code}
Output with release `3.3.2`:
{noformat}
+---+-----------------------+
|id |env_OMP_NUM_THREADS    |
+---+-----------------------+
|0  |null                   |
+---+-----------------------+
{noformat}
Output with release `3.3.0`:
{noformat}
+---+-----------------------+
|id |env_OMP_NUM_THREADS    |
+---+-----------------------+
|0  |1                      |
+---+-----------------------+
{noformat}
Bug: Add error class INVALID_TEMP_OBJ_REFERENCE
Status: Resolved | Priority: Major | Resolution: Resolved
Created: 03/Feb/23 23:11
Affects Version/s: 3.4.0

Add the new error class INVALID_TEMP_OBJ_REFERENCE and move the following error classes to use the new one:
 * _LEGACY_ERROR_TEMP_1283
 * _LEGACY_ERROR_TEMP_1284
Bug: spark can not read lastest view sql when run `create or replace view` by hive
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 27/Feb/23 05:30
Affects Version/s: 3.3.2

1. Spark would save view schema as tabel param. 
2. Spark will make tabel param as output schema when select the view .
3. Hive will not update tabel param when runing `create or replace view` to update the view.

!image-2023-02-27-13-31-20-420.png!

So when hive and spark are mixed and update the view, spark may ignore some col.

To reproduce this issue:

1. running in spark
```
create table test_spark (id string);
create view test_spark_view as select id from test_spark;
```

2. running in hive

```
create or replace view test_spark_view as select id , "test" as new_id from test_spark;

```

3. We can see spark will ignore `test_spark_view#new_id` when select test_spark_view using spark. But hive can read it.

I'm not sure if this is a feature of spark.
Bug: collapse two adjacent windows with the equivalent partition/order expression
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Feb/23 02:18
Affects Version/s: 3.2.3

Extend the CollapseWindow rule to collapse Window nodes with the equivalent partition/order expressions

{code:java}
Seq((1, 1), (2, 2)).toDF("a", "b")
      .withColumn("max_b", expr("max(b) OVER (PARTITION BY abs(a))"))
      .withColumn("min_b", expr("min(b) OVER (PARTITION BY abs(a))"))

== Optimized Logical Plan ==
before
Project [a#7, b#8, max_b#11, min_b#17]
+- Window [min(b#8) windowspecdefinition(_w0#19, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#19]
   +- Project [a#7, b#8, max_b#11, abs(a#7) AS _w0#19]
      +- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11], [_w0#13]
         +- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
            +- LocalRelation [_1#2, _2#3]
after
Project [a#7, b#8, max_b#11, min_b#17]
+- Window [max(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS max_b#11, min(b#8) windowspecdefinition(_w0#13, specifiedwindowframe(RowFrame, unboundedpreceding$(), unboundedfollowing$())) AS min_b#17], [_w0#13]
   +- Project [_1#2 AS a#7, _2#3 AS b#8, abs(_1#2) AS _w0#13]
      +- LocalRelation [_1#2, _2#3]
{code}
Bug: Implement Dataset.version and Dataset.time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 18:47
Affects Version/s: 3.4.0

Implement Dataset.version and Dataset.time
{code:java}
/**
 * The version of Spark on which this application is running.
 *
 * @since 2.0.0
 */
def version: String = SPARK_VERSION

/**
 * Executes some code block and prints to stdout the time taken to execute the block. This is
 * available in Scala only and is used primarily for interactive testing and debugging.
 *
 * @since 2.1.0
 */
def time[T](f: => T): T = {
  val start = System.nanoTime()
  val ret = f
  val end = System.nanoTime()
  // scalastyle:off println
  println(s"Time taken: ${NANOSECONDS.toMillis(end - start)} ms")
  // scalastyle:on println
  ret
} {code}
Bug: Persisted RDD blocks can be inconsistent if the RDD computation is indeterminate
Status: Open | Priority: Major | Resolution: 
Created: 26/Feb/23 03:24
Affects Version/s: 3.3.2

When a rdd includes indeterminate operations, the rdd results can be different each time we recompute it.
And when we cache such a rdd, we may have multiple rdd block replicas having different data. Here is an example:
1. Task A generated the rdd block rdd_1_1 on executor E1;
2. Task B on executor E2 tried to fetch remote rdd_1_1 from E1 but failed, then it will compute and cache another block on E2; 
If the results on E1 and E2 are differnet, we'll have 2 blocks for the same rdd partition with different data.

The behavior will be unexpcted for such cases.
Bug: Add 2nd groupBy method to Dataset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/23 02:35
Affects Version/s: 3.4.0

Dataset is missing a groupBy method:
{code:java}
/**
 * Groups the Dataset using the specified columns, so that we can run aggregation on them.
 * See [[RelationalGroupedDataset]] for all the available aggregate functions.
 *
 * This is a variant of groupBy that can only group by existing columns using column names
 * (i.e. cannot construct expressions).
 *
 * {{{
 *   // Compute the average for all numeric columns grouped by department.
 *   ds.groupBy("department").avg()
 *
 *   // Compute the max age and average salary, grouped by department and gender.
 *   ds.groupBy($"department", $"gender").agg(Map(
 *     "salary" -> "avg",
 *     "age" -> "max"
 *   ))
 * }}}
 * @group untypedrel
 * @since 3.4.0
 */
@scala.annotation.varargs
def groupBy(col1: String, cols: String*): RelationalGroupedDataset {code}
Bug: Add TempView APIs to Dataset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 18:33
Affects Version/s: 3.4.0

Add all temp view APIs to connect's Dataset.
{code:java}
/**
* Registers this Dataset as a temporary table using the given name. The lifetime of this
* temporary table is tied to the [[SparkSession]] that was used to create this Dataset.
*
* @group basic
* @since 1.6.0
*/
@deprecated("Use createOrReplaceTempView(viewName) instead.", "2.0.0")
def registerTempTable(tableName: String): Unit

/**
* Creates a local temporary view using the given name. The lifetime of this
* temporary view is tied to the [[SparkSession]] that was used to create this Dataset.
*
* Local temporary view is session-scoped. Its lifetime is the lifetime of the session that
* created it, i.e. it will be automatically dropped when the session terminates. It's not
* tied to any databases, i.e. we can't use `db1.view1` to reference a local temporary view.
*
* @throws AnalysisException if the view name is invalid or already exists
*
* @group basic
* @since 2.0.0
*/
@throws[AnalysisException]
def createTempView(viewName: String): Unit

/**
* Creates a local temporary view using the given name. The lifetime of this
* temporary view is tied to the [[SparkSession]] that was used to create this Dataset.
*
* @group basic
* @since 2.0.0
*/
def createOrReplaceTempView(viewName: String): Unit

/**
* Creates a global temporary view using the given name. The lifetime of this
* temporary view is tied to this Spark application.
*
* Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
* i.e. it will be automatically dropped when the application terminates. It's tied to a system
* preserved database `global_temp`, and we must use the qualified name to refer a global temp
* view, e.g. `SELECT * FROM global_temp.view1`.
*
* @throws AnalysisException if the view name is invalid or already exists
*
* @group basic
* @since 2.1.0
*/
@throws[AnalysisException]
def createGlobalTempView(viewName: String): Unit

/**
* Creates or replaces a global temporary view using the given name. The lifetime of this
* temporary view is tied to this Spark application.
*
* Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
* i.e. it will be automatically dropped when the application terminates. It's tied to a system
* preserved database `global_temp`, and we must use the qualified name to refer a global temp
* view, e.g. `SELECT * FROM global_temp.view1`.
*
* @group basic
* @since 2.2.0
*/
def createOrReplaceGlobalTempView(viewName: String): Unit {code}
Bug: Track state store provider load time and log warning if it exceeds a threshold
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/23 20:04
Affects Version/s: 3.4.1

Track state store provider load time and log warning if it exceeds a threshold

 

In some cases, we see that the filesystem initialization might take time for the first time that we create the provider and initialize it. This change will log the time taken if it exceeds a certain threshold
Bug: Fix internal error for valid CASE WHEN expression with CAST when inserting into a table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 23:00
Affects Version/s: 3.3.1

```

spark-sql> create or replace table es570639t1 as select x FROM values (1), (2), (3) as tab(x);
spark-sql> create or replace table es570639t2 (x Decimal(9, 0));
spark-sql> insert into es570639t2 select 0 - (case when x = 1 then 1 else x end) from es570639t1 where x = 1;

```

hits the following internal error
org.apache.spark.SparkException: [INTERNAL_ERROR] Child is not Cast or ExpressionProxy of Cast
 

Stack trace:
org.apache.spark.SparkException: [INTERNAL_ERROR] Child is not Cast or ExpressionProxy of Cast at org.apache.spark.SparkException$.internalError(SparkException.scala:78) at org.apache.spark.SparkException$.internalError(SparkException.scala:82) at org.apache.spark.sql.catalyst.expressions.CheckOverflowInTableInsert.checkChild(Cast.scala:2693) at org.apache.spark.sql.catalyst.expressions.CheckOverflowInTableInsert.withNewChildInternal(Cast.scala:2697) at org.apache.spark.sql.catalyst.expressions.CheckOverflowInTableInsert.withNewChildInternal(Cast.scala:2683) at org.apache.spark.sql.catalyst.trees.UnaryLike.$anonfun$mapChildren$5(TreeNode.scala:1315) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:106) at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1314) at org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1309) at org.apache.spark.sql.catalyst.expressions.UnaryExpression.mapChildren(Expression.scala:636) at org.apache.spark.sql.catalyst.trees.TreeNode.transformUpWithPruning(TreeNode.scala:570) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUpWithPruning$1(TreeNode.scala:570)
 

This internal error comes from `CheckOverflowInTableInsert``checkChild`, where we covered only `Cast` expr and `ExpressionProxy` expr, but not the `CaseWhen` expr.
Bug: Allow multiple event time columns in single DataFrame
Status: Open | Priority: Major | Resolution: 
Created: 24/Feb/23 05:20
Affects Version/s: 3.5.0

This is a follow-up task for SPARK-42376.

I've observed that all stateful operators deals with event time column via finding the "first" event time column and picking it up. This was safe before supporting multiple stateful operators, because the only way a single DataFrame contains more than one event time column is stream-stream join, and then we didn't support adding another stateful operator. It's safer to assume there is only one event time column.

With the support of multiple stateful operators, the assumption no longer holds true. Users can add another stateful operators after stream-stream join, which breaks the assumption. Even for the case of (stream-stream join)-stream join, any side of stream-stream join can have multiple event time columns.

Since it's non-trivial to reason about the right behavior for all stateful operators, we just disallow DataFrame to have more than one event time columns in SPARK-42376. (The check will be performed in stateful operator against input DataFrame.)

This ticket tracks the effort to reason about the right behavior for all stateful operators, and support more than one event time columns in the input DataFrame for stateful operators.
Bug: Make spark connect supporting canceling job group
Status: Open | Priority: Major | Resolution: 
Created: 17/Feb/23 05:24
Affects Version/s: 3.4.0

PySpark ML crossValidator relies on this:

proposal:
when spark client sending request, can we make it send with a "job_group_id", and in server side, for each request that with the same (user_id, job_group_id), handling request in a fixed thread
Bug: Getting Started: Live Notebook for Spark Connect
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/23 09:43
Affects Version/s: 3.4.0

It would be great to have Live Notebook for Spark Connect in [Getting Started|https://spark.apache.org/docs/latest/api/python/getting_started/index.html] section to help users quick start on Spark Connect.
Bug: Add more test case(s) for longer chain of multiple stateful operators
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/23 20:40
Affects Version/s: 3.5.0

This is a follow-up task for SPARK-42376.

We added various test cases for both SPARK-40925 and SPARK-42376 but these test cases are more likely chaining two stateful operators. It would be really nice to have another test case(s) dealing with longer chain of stateful operators to increase test coverage.
Bug: The HA support of Spark Thrift Server
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/23 07:57
Affects Version/s: 3.3.2

When there are many Spark SQL connects on Spark Thrift Server.if the Thrift Server is down ,the connect cannot get service .So we need to consider the High Availability support for Spark Thrift Server . 
We want to import the pattern of HiveServer HA to provide ThriftServer HA.  we need to get HA support as the 'addServerInstanceToZooKeeper' method of HiveServer2. 
Use zookeeper to choose the active thrift server when you connect the Spark Thrift Server.
Bug: Fix DataFrameWriterV2 to find the default source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/23 21:42
Affects Version/s: 3.4.0

{code:python}
df.writeTo("test_table").create()
{code}

throws:

{noformat}
pyspark.errors.exceptions.connect.SparkConnectGrpcException: (org.apache.spark.SparkClassNotFoundException) [DATA_SOURCE_NOT_FOUND] Failed to find the data source: . Please find packages at `https://spark.apache.org/third-party-projects.html`.
{noformat}
Bug: Use available ephemeral port for Spark Connect server in testing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/23 06:16
Affects Version/s: 3.4.0

Currently Spark Connect tests cannot run in parallel, and requires to set the parallelism as 1

{code}
python/run-tests --module pyspark-connect --parallelism 1
{code}

The main reason is because of the port being used is hardcorded as the default 15002. We should better search available port, and use it
Bug: Non-captured session time zone in view creation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/23 17:05
Affects Version/s: 3.3.3

The session time zone config is captured only when it is set explicitly but if it is not the view is instantiated with the current settings. That's might confuse users since query results depends on explicit SQL config settings while creating a view.

The example below portraits the issue:
{code:java}
val viewName = "v1_capture_test"
withView(viewName) {
  assert(get.sessionLocalTimeZone === "America/Los_Angeles")
  createView(viewName,
    """select hour(ts) as H from (
      |  select cast('2022-01-01T00:00:00.000 America/Los_Angeles' as timestamp) as ts
      |)""".stripMargin, Seq("H"))
  withDefaultTimeZone(java.time.ZoneId.of("UTC-09:00")) {
    withSQLConf(SESSION_LOCAL_TIMEZONE.key -> "UTC-10:00") {
      sql(s"select H from $viewName").show(false)
    }
  }
} {code}
It is expected to output:
{code:java}
+---+
|H  |
+---+
|0  |
+---+ {code}
but actual output is:
{code:java}
+---+
|H  |
+---+
|8  |
+---+ {code}
Bug: Upgrade numpy and pandas in the release Dockerfile
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/23 01:55
Affects Version/s: 3.4.0

Otherwise, errors are raised as shown below when building release docs.


{code}
ImportError: Warning: Latest version of pandas (1.5.3) is required to generate the documentation; however, your version was 1.1.5

ImportError: this version of pandas is incompatible with numpy < 1.20.3
your numpy version is 1.19.4.
Please upgrade numpy to >= 1.20.3 to use this pandas version
{code}
Bug: Log `exitCode` when `SparkContext.stop` starts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Feb/23 10:15
Affects Version/s: 3.4.0

{code}
23/02/06 02:12:55 INFO SparkContext: SparkContext is stopping with exitCode 0.
{code}

{code}
Pi is roughly 3.147080
23/02/06 02:12:55 INFO SparkContext: SparkContext is stopping with exitCode 0.
...
23/02/06 02:12:55 INFO AbstractConnector: Stopped Spark@1cb72b8{HTTP/1.1, (http/1.1)}{localhost:4040}
23/02/06 02:12:55 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
23/02/06 02:12:55 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/02/06 02:12:55 INFO MemoryStore: MemoryStore cleared
23/02/06 02:12:55 INFO BlockManager: BlockManager stopped
23/02/06 02:12:55 INFO BlockManagerMaster: BlockManagerMaster stopped
23/02/06 02:12:55 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/02/06 02:12:55 INFO SparkContext: Successfully stopped SparkContext
23/02/06 02:12:56 INFO ShutdownHookManager: Shutdown hook called
{code}