av9ash/GitBugs huggingface dataset:
Bug: [SQL] AQE does not support columnar execution for the final query stage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jun/21 15:28
Affects Version/s: 3.0.3

In AdaptiveSparkPlanExec, a query is broken down into stages and these stages are executed until the entire query has been executed. These stages can be row-based or columnar. However, the final stage, produced by the private getFinalPhysicalPlan method is always assumed to be row-based. The only way to execute the final stage is by calling the various doExecute methods on AdaptiveSparkPlanExec, and doExecuteColumnar is not implemented. The supportsColumnar method also always returns false.

In the RAPIDS Accelerator for Apache Spark, we currently call the private getFinalPhysicalPlan method using reflection and then determine if that plan is columnar or not, and then call the appropriate doExecute method, bypassing the doExecute methods on AdaptiveSparkPlanExec. We would like a supported mechanism for executing a columnar AQE plan so that we do not need to use reflection.
Bug: Consolidate logic between AvroSerializer/AvroDeserializer for schema mismatch handling and error messages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/21 21:11
Affects Version/s: 3.1.2

While working on [PR #31490|https://github.com/apache/spark/pull/31490] for SPARK-34365, we discussed that there is room for improvement in how schema mismatch errors are reported ([comment1|https://github.com/apache/spark/pull/31490#discussion_r659970793], [comment2|https://github.com/apache/spark/pull/31490#issuecomment-869866848]). We can also consolidate more of the logic between AvroSerializer and AvroDeserializer to avoid some duplication of error handling and consolidate how these error messages are generated.

This will essentially be taking the [logic from the initial proposal from PR #31490|https://github.com/apache/spark/pull/31490/commits/83a922fdff08528e59233f67930ac78bfb3fa178], but applied separately from the current set of proposed changes to cut down on PR size.
Bug: Disable push-based shuffle until the feature is complete
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 28/Jun/21 17:40
Affects Version/s: 3.1.0

Push-based shuffle is partially merged in apache master but some of the tasks are still incomplete. Since 3.2 is going to cut soon, we will not be able to get the pending tasks reviewed and merged. Few of the pending tasks make protocol changes to the push-based shuffle protocols, so we would like to prevent users from enabling push-based shuffle both on the client and the server until push-based shuffle implementation is complete. 
We can prevent push-based shuffle to be used by throwing {{UnsupportedOperationException}} (or something like that) both on the client and the server when the user tries to enable it.
Bug: Add metrics about coalesced partitions to CustomShuffleReader in AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/21 23:02
Affects Version/s: 3.0.0

{{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number of skewed partition splits".
 It would be useful to also report "number of partitions to coalesce" and "number of coalesced partitions" and include this in string rendering of the SparkPlan node so that it looks like this
{code:java}
(12) CustomShuffleReader
Input [2]: [a#23, b#24]
Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
{code}
For accurate description see https://github.com/apache/spark/pull/32776#issue-661274727
Bug: Extend SPARK-SQL to support REPEATABLE TABLESAMPLE clause. 
Status: Open | Priority: Major | Resolution: 
Created: 29/Jun/21 16:20
Affects Version/s: 3.1.2

Extend Spark-SQL so as to support the `REPEATABLE TABLESAMPLE` clause per SQL-standard:
{code:sql}
SELECT * FROM `foo` TABLESAMPLE (50.0 PERCENT) REPEATABLE (1234) {code}
Certain applications need to return repeatable/deterministic results when applying TABLESAMPLE clause.
Bug: Support DATE/TIMESTAMP literals across the full range
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/21 02:44
Affects Version/s: 3.2.0

DATE/TIMESTAMP literals support years 0000 to 9999.
However, internally we support a range that is much larger.
I can add or subtract large intervals from a date/timestamp and the system will happily process and display large negative and positive dates.

Since we obviously cannot put this genie back into the bottle the only thing we can do is allow matching DATE/TIMESTAMP literals.
Bug: Support adding TimestampWithoutTZ with Interval types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 04:07
Affects Version/s: 3.3.0

Supprot the following operations:
* TimestampWithoutTZ + Calendar interval
* TimestampWithoutTZ + Year-Month interval
* TimestampWithoutTZ + Daytime interval
Bug: Support extracting date fields from timestamp without time zone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 14:17
Affects Version/s: 3.3.0

Support extracting date fields from timestamp without time zone, which includes:
* year
* month
* day
* year of week
* week
* day of week 
* quarter
* day of month
* day of year
Bug: Support subtraction among Date/Timestamp/TimestampWithoutTZ
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Jun/21 13:02
Affects Version/s: 3.3.0

Support the following operations:
* TimestampWithoutTZ - Date
* Date - TimestampWithoutTZ
* TimestampWithoutTZ - Timestamp
* Timestamp - TimestampWithoutTZ
* TimestampWithoutTZ - TimestampWithoutTZ
Bug: New SQL function: to_timestamp_ntz
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jun/21 07:14
Affects Version/s: 3.3.0

Implement new SQL function: to_timestamp_ntz. It is similar to the built-in function to_timestamp, except that the result type is TimestampWithoutTZType.
The naming is from snowflake: https://docs.snowflake.com/en/sql-reference/functions/to_timestamp.html
Bug: Improve the error message of to_timestamp_ntz with invalid format pattern
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jun/21 09:34
Affects Version/s: 3.3.0

As discussed in https://github.com/apache/spark/pull/32995/files#r655148980, there is an error message saying
"You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyy-MM-dd GGGGG' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0"

This is not true for function to_timestamp_ntz, which only uses the Iso8601TimestampFormatter and added since Spark 3.2. We should improve it.
Bug: Support type conversion between timestamp and timestamp without time zone type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/21 02:47
Affects Version/s: 3.3.0

1. Extend the Cast expression and support TimestampType in casting to TimestampWithoutTZType.
2. There was a mistake in casting TimestampWithoutTZType as TimestampType in https://github.com/apache/spark/pull/32864. The target value should be `sourceValue - timeZoneOffset` instead of being the same value.
Bug: Support extracting hour/minute/second from timestamp without time zone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jun/21 15:21
Affects Version/s: 3.3.0

Support extracting hour/minute/second fields from timestamp without time zone values. In details, the following syntax is supported:
1. extract [hour | minute | second] from timestampWithoutTZ
2. date_part('[hour | minute | second]', timestampWithoutTZ)
3. hour(timestampWithoutTZ)
4. minute(timestampWithoutTZ)
5. second(timestampWithoutTZ)
Bug: Deprecate the `num_files` argument
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jun/21 04:06
Affects Version/s: 3.2.0

We should deprecate the num_files argument in [DataFrame.to_csv |https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_csv.html]and [DataFrame.to_json|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.to_json.html].

Because the behavior of num_files is not actually specify the number of files, but it specifies the number of partition.

So we should encourage users to use [DataFrame.spark.repartition|https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.DataFrame.spark.repartition.html] instead in the warning message.
Bug: Parse year-month interval literals to tightest types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/21 16:38
Affects Version/s: 3.2.0

Modify AstBuilder.visitInterval to parse year-month interval literals to tightest types. For example:
INTERVAL '10' YEAR should be parsed as YearMonthIntervalType(YEAR, YEAR) but not as YearMonthIntervalType(YEAR, MONTH).
Bug: Parse unit list interval literals as year-month/day-time interval types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/21 20:57
Affects Version/s: 3.2.0

Currently, unit list interval literals like `interval '1' year '2' months` or `interval '1' day` or `interval '2' hours` are parsed as `CalendarIntervalType`.
Such fields should be parsed as `YearMonthIntervalType` or `DayTimeIntervalType`.
Bug: Clean up evaluates subexpressions and add more flexibility to evaluate particular subexpressoin
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/21 19:39
Affects Version/s: 3.2.0

Currently `subexpressionEliminationForWholeStageCodegen` return` the gen-ed code of subexpressions. The caller simply puts the code into its code block. We need more flexible evaluation here. For example, for Filter operator's subexpression evaluation, we may need to evaluate particular subexpression for one predicate. Current approach cannot satisfy the requirement.
Bug: Introduce "prefix match scan" feature on state store
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jun/21 06:31
Affects Version/s: 3.2.0

We are proposing the new functionality in Structured Streaming’s state store, “prefix match scan”, to group the keys into logical groups, and scan the keys in the same group efficiently.

This change is to enable us to construct the “logically list-like” data structure on top of the current state store, which only provides the operations for key-value. There’s an iterator in the state store to scan through the key space, but only supports full scan. Prefix match scan would open the possibility to scan on a specific range of the keys efficiently.

This helps to simplify the state and also logic to the streaming session window, and also achieve similar performance (or even higher) than the approach we proposed before.
Bug: Refactor SparkError.scala to SparkThrowable.java
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 20:17
Affects Version/s: 3.2.0

Following up from SPARK-34920:

Error has a special meaning in Java; SparkError should encompass all Throwables. It'd be more correct to rename SparkError to SparkThrowable.

In addition, some Throwables come from Java, so to maximize usability, we should migrate the base trait from Scala to Java.
Bug: Kubernetes: Allow different namespaces for Driver and Executors
Status: Open | Priority: Major | Resolution: 
Created: 09/Jun/21 17:28
Affects Version/s: 3.1.2

Currently the Kubernetes namespace of driver AND executors always needs to be identical - specified with {{spark.kubernetes.namespace}}.
 Especially when developing in Kubernetes, this is a harsh restriction: In a _notebook_ Namespace I want to run spark in driver in client mode for interaction, however executors should get their own dedicated resources in a dedicated namespace with different policies and different lifecycles.

This could be solved by adding the following two additional configuration options:
{code:java}
 spark.kubernetes.executor.pod.namespace
 spark.kubernetes.driver.pod.namespace
{code}
Bug: Make unary and comparison operators data-type-based
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 21:54
Affects Version/s: 3.2.0

We have been refactoring basic operators to be data-type-based for readability, flexibility, and extensibility.

Unary and comparison operators are still not data-type-based yet. We should fill the gaps.
 * Unary operators: {{__neg__}}, {{__abs__}}, {{__invert__}},
 * Comparison operators: {{>}}, {{>=}}, {{<}}, {{<=}}, {{==}}, {{!=}}
Bug: Remove order by if the maximum number of rows less than or equal to 1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jun/21 13:49
Affects Version/s: 3.2.0

{code:scala}
spark.sql("select count(*) from range(1, 10, 2, 2) order by 1 limit 10").explain("cost")
{code}

Current optimized logical plan:
{noformat}
== Optimized Logical Plan ==
Sort [count(1)#2L ASC NULLS FIRST], true, Statistics(sizeInBytes=16.0 B)
+- Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
   +- Project, Statistics(sizeInBytes=20.0 B)
      +- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
{noformat}

Expected optimized logical plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Project, Statistics(sizeInBytes=20.0 B)
   +- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
{noformat}
Bug: Fix decimal overflow issues for Average
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jun/21 17:27
Affects Version/s: 3.0.0

Fix decimal overflow issues for decimal average in ANSI mode. Linked to SPARK-32018 and SPARK-28067, which address decimal sum.

Repro:

 
{code:java}
import org.apache.spark.sql.functions._
spark.conf.set("spark.sql.ansi.enabled", true)

val df = Seq(
 (BigDecimal("10000000000000000000"), 1),
 (BigDecimal("10000000000000000000"), 1),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2)).toDF("decNum", "intNum")
val df2 = df.withColumnRenamed("decNum", "decNum2").join(df, "intNum").agg(mean("decNum"))
df2.show(40,false)
{code}
 

Should throw an exception (as sum overflows), but instead returns:

 
{code:java}
+-----------+
|avg(decNum)|
+-----------+
|null       |
+-----------+{code}
Bug: Support user defined initial state with flatMapGroupsWithState in Structured Streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jun/21 15:37
Affects Version/s: 3.1.2

Structured Streaming supports arbitrary stateful processing using mapGroupsWithState and flatMapGroupWithState operators. The state is created by processing the data that comes in with every batch. This API improvement will allow users to specify an initial state which is applied at the time of executing the first batch.

 
h2. Proposed new APIs (Scala)

 

 

  def mapGroupsWithState[S: Encoder, U: Encoder](

      timeoutConf: GroupStateTimeout,

      initialState: Dataset[(K, S)])(

      func: (K, Iterator[V], GroupState[S]) => U): Dataset[U] 

 

  def flatMapGroupsWithState[S: Encoder, U: Encoder](

      outputMode: OutputMode,

      timeoutConf: GroupStateTimeout,

      initialState: Dataset[(K, S)])(

      func: (K, Iterator[V], GroupState[S]) => Iterator[U])

 
h2.    Proposed new APIs (Java)

  

    def mapGroupsWithState[S, U](

      func: MapGroupsWithStateFunction[K, V, S, U],

      stateEncoder: Encoder[S],

      outputEncoder: Encoder[U],

      timeoutConf: GroupStateTimeout,

      initialState: Dataset[(K, S)]): Dataset[U]





    def flatMapGroupsWithState[S, U](

      func: FlatMapGroupsWithStateFunction[K, V, S, U],

      outputMode: OutputMode,

      stateEncoder: Encoder[S],

      outputEncoder: Encoder[U],

      timeoutConf: GroupStateTimeout,

      initialState: Dataset[(K, S)]): Dataset[U]

 

   
h2. Example Usage

   

    val initialState: Dataset[(String, RunningCount)] = Seq(

      ("a", new RunningCount(1)),

 ("b", new RunningCount(1))

    ).toDS()

 

    val inputData = MemoryStream[String]

    val result =

      inputData.toDS()

        .groupByKey(x => x)

        .mapGroupsWithState(initialState, timeoutConf)(stateFunc)
Bug: Increase the heap and stack size for Maven build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jun/21 11:42
Affects Version/s: 3.2.0

The jenkins jobs are unstable due to the stackoverflow errors: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-3.2-jdk-11/
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/2274/


We should increase memory configuration for Maven build.
Stack size: 64MB => 128MB
Initial heap size: 1024MB => 2048MB
Maximum heap size: 1024MB => 2048MB

The SBT builds are ok so let's keep the current configuration.
Bug: Support dynamic filtering for v2 tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/21 20:27
Affects Version/s: 3.2.0

We need to support dynamic filtering for v2 tables.

Design doc:
https://docs.google.com/document/d/1RfFn2e9o_1uHJ8jFGsSakp-BZMizX1uRrJSybMe2a6M
Bug: unionByName should support nested struct also
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/21 14:24
Affects Version/s: 3.1.1

It would be cool if `unionByName` supports also nested struct. I don't kwon if it's the expected behaviour already or not so I am not sure if its a bug or an improvement proposal. 
{code:java}
case class Struct1(c1: Int, c2: Int)
case class Struct2(c2: Int, c1: Int)
val ds1 = Seq((1, Struct1(1,2))).toDS
val ds2 = Seq((1, Struct2(1,2))).toDS
ds1.unionByName(ds2.as[(Int,Struct1)]) {code}
gives 
{code:java}
org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<c2:int,c1:int> <> struct<c1:int,c2:int> at the second column of the second table; 'Union false, false :- LocalRelation [_1#38, _2#39] +- LocalRelation _1#45, _2#46
{code}
The code documentation of the function `unionByName` says `Note that allowMissingColumns supports nested column in struct types` but doesn't say if the function itself supports the nested column ordering or not.
Bug: Avoid using auto generated alias when creating view
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 03:03
Affects Version/s: 3.2.0

If the user creates a view in 2.4 and reads it in 3.2, there will be an incompatible schema issue. the root cause is that we changed the alias auto generation rule after 2.4. To avoid this happening again, we should let the user explicitly specifying the column names
Bug: Prompt recreating the View when there is a schema incompatible change
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/21 02:31
Affects Version/s: 3.2.0

Prompt recreating the View when there is a schema incompatible change. Something like:

"there is an incompatible schema change and the column couldn't be resolved. Please consider to recreate the view to fix this: CREATE OR REPLACE VIEW v AS xxx"
Bug: Resolve star expressions in subquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/21 23:19
Affects Version/s: 3.2.0

Currently, Spark does not resolve star expressions in subqueries correctly. It can only resolve the star expressions using the inner query attributes. For example:

{{CREATE VIEW t(a) AS VALUES (1), (2);}}

{{SELECT * FROM t WHERE a in (SELECT t.*)}}

{{SELECT * FROM t, LATERAL (SELECT t.*)}}

{{org.apache.spark.sql.AnalysisException: cannot resolve 't.*' given input columns '';}}

Instead, we should try to resolve star expressions in subquery first using the inner attributes and then using the outer query attributes.
Bug: Remove repartition if the child maximum number of rows less than or equal to 1
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 27/Jun/21 01:01
Affects Version/s: 3.2.0

{code:scala}
spark.sql("select count(*) from range(1, 10, 2, 2)").repartition(10).explain("cost")
{code}

Current optimized logical plan:
{noformat}
== Optimized Logical Plan ==
Repartition 10, true, Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
   +- Project, Statistics(sizeInBytes=20.0 B)
      +- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
{noformat}

Expected optimized logical plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Project, Statistics(sizeInBytes=20.0 B)
   +- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
{noformat}
Bug: working with spring boot, spark context will stopped while application is started.
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Jun/21 07:11
Affects Version/s: 3.1.2

Spark 3.1.2 with springboot, on client mode, the spark context while stopped while the application is started.

 

 
{code:java}
//代码占位符
@Bean
@ConditionalOnMissingBean(SparkSession.class)
public SparkSession sparkSession(SparkConf conf) {
    return SparkSession.builder()
            .enableHiveSupport()
            .config(conf)
            .getOrCreate();
}{code}
 
{quote} 21/06/30 12:03:38 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
 21/06/30 12:03:38 INFO Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
 21/06/30 12:03:38 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000000000(ns)
 21/06/30 12:03:39 INFO WelcomePageHandlerMapping: Adding welcome page template: index
 21/06/30 12:03:40 INFO Http11NioProtocol: Starting ProtocolHandler ["http-nio-9000"]
 21/06/30 12:03:40 INFO TomcatWebServer: Tomcat started on port(s): 9000 (http) with context path ''
 21/06/30 12:03:40 INFO SpringApplication: Started application in 525.411 seconds (JVM running for 529.958)
 21/06/30 12:03:40 INFO AbstractConnector: Stopped Spark@3e1d19ea\{HTTP/1.1, (http/1.1)}
 Unknown macro: \{0.0.0.0}
 21/06/30 12:03:40 INFO SparkUI: Stopped Spark web UI at 
 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Interrupting monitor thread
 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: Shutting down all executors
 21/06/30 12:03:40 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
 21/06/30 12:03:40 INFO YarnClientSchedulerBackend: YARN client scheduler backend Stopped
 21/06/30 12:03:40 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
 21/06/30 12:03:40 INFO MemoryStore: MemoryStore cleared
 21/06/30 12:03:40 INFO BlockManager: BlockManager stopped
 21/06/30 12:03:40 INFO BlockManagerMaster: BlockManagerMaster stopped
 21/06/30 12:03:40 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
 21/06/30 12:03:40 INFO SparkContext: Successfully stopped SparkContext
 21/06/30 12:03:40 INFO [/]: Initializing Spring DispatcherServlet 'dispatcherServlet'
 21/06/30 12:03:40 INFO DispatcherServlet: Initializing Servlet 'dispatcherServlet'
 21/06/30 12:03:40 INFO DispatcherServlet: Completed initialization in 1 ms
{quote}
Bug: Deprecate Python 3.6 support
Status: Resolved | Priority: Major | Resolution: Done
Created: 29/Jun/21 17:00
Affects Version/s: 3.2.0

According to [https://endoflife.date/python], Python 3.6 will be EOL on 23 Dec, 2021.

We should prepare for the deprecation of Python 3.6 support in Spark in advance.