av9ash/GitBugs huggingface dataset:
Bug: Interval types are not truncated to the expected endField when creating a DataFrame via Duration
Status: Open | Priority: Major | Resolution: 
Created: 17/Mar/22 01:18
Affects Version/s: 3.3.0

*Problem:*

ANSI interval types are store as long internally.

The long value are not truncated to the expected endField when creating a DataFrame via Duration.

 

*Reproduce:*

Create a "day to day" interval, the seconds are not truncated, see below code.

The internal long is not {*}86400 * 1000000{*}, but it's ({*}86400 + 1)  * 1000000{*}{*}{{*}}

 
{code:java}
  test("my test") {
    val data = Seq(Row(Duration.ofDays(1).plusSeconds(1)))
    val schema = StructType(Array(
      StructField("t", DayTimeIntervalType(DayTimeIntervalType.DAY, DayTimeIntervalType.DAY))
    ))
    val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
    df.show()
  } {code}
 

 

After debug, the {{endField}} is always {{SECOND}} in {{{}durationToMicros{}}}, see below:

 
{code:java}
  // IntervalUtils class

  def durationToMicros(duration: Duration): Long = {
    durationToMicros(duration, DT.SECOND)   // always SECOND
  }

  def durationToMicros(duration: Duration, endField: Byte)

{code}
Seems should use different endField which could be [DAY, HOUR, MINUTE, SECOND]

Or Spark can throw an exception to avoid truncating.
Bug: Requesting Restful API can cause NullPointerException
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 17/Mar/22 02:38
Affects Version/s: 3.1.0

When requesting Restful API  \{baseURL}/api/v1/applications/$appId/sql/$executionId which is introduced by this PR [https://github.com/apache/spark/pull/28208,] it can cause NullPointerException. The root cause is, when calling method doUpdate() of `LiveExecutionData`, `metricsValues` can be null. Then, when statement `printableMetrics(graph.allNodes, exec.metricValues)` is executed, it will throw  NullPointerException.
Bug: display the number of empty partitions on spark ui
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Mar/22 02:51
Affects Version/s: 3.1.2

When demoting join from broadcast-hash to smj, i think it is necessary to display the number of empty partitions on spark ui.

Otherwise, users might wonder why SMJ is used when joining a small table. Displaying the number of empty partitions is useful for users to understand changes to the execution plan.

Before updated the ui:

!image-2022-03-16-10-56-46-446.png!

After updated the ui, display the number of empty partitions:

!image-2022-03-16-11-07-39-182.png!
Bug: Fix Volcano weight to be positive integer and use cpu capability instead
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 13:43
Affects Version/s: 3.3.0

As description from [https://volcano.sh/en/docs/queue/]

- weight is a soft constraint.

- capability is a hard constraint.

We better to use capability to make thing simple to avoid being influenced by other queues
Bug: replace deprecated ChiSqSelector with UnivariateFeatureSelector
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 14/Mar/22 11:37
Affects Version/s: 3.1.2

UnivariateFeatureSelector was added and ChiSqSelector was labeled as deprecated in  

SPARK-34080

So we need replace deprecated ChiSqSelector with UnivariateFeatureSelector.
Bug: Sending Available Resources in Yarn Cluster Information to Spark Driver
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Mar/22 12:18
Affects Version/s: 3.2.1

Yarn Cluster Manager Provides information on available resources (VCores / Memory) in the Cluster via AM-RM heartbeat. This task is for creating infra to Send Available VCores and Memory information from Yarn AM-RM heartbeat response to Spark Driver.
Bug: SessionWindowStateStoreRestoreExec should provide numRowsDroppedByWatermark metric
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Mar/22 23:35
Affects Version/s: 3.3.0

SessionWindowStateStoreRestoreExec filters out outdated inputs by watermark but it doesn't provide numRowsDroppedByWatermark. It makes users confused about stream query result.
Bug: Add the datetimeUnit enum to the grammar and use it in TIMESTAMPADD/DIFF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/22 11:49
Affects Version/s: 3.3.0

Add the following enum:

{code}
datetimeUnit
    : YEAR | QUARTER | MONTH
    | WEEK | DAY | DAYOFYEAR
    | HOUR | MINUTE | SECOND | MILLISECOND | MICROSECOND
    ;
{code}
and use it in the grammar rules for TIMESTAMPADD and TIMESTAMPDIFF.
Bug: Failure on referring to the corrupt record from CSV
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 12:41
Affects Version/s: 3.3.0

The file below has a invalid value in a field:
{code:java}
0,2013-111_11 12:13:14
1,1983-08-04 {code}
where the timestamp 2013-111_11 12:13:14 is incorrect.

The query fails when it refers to the corrupt record column:
{code:java}
spark.read.format("csv")
 .option("header", "true")
 .schema(schema)
 .load("csv_corrupt_record.csv")
 .filter($"_corrupt_record".isNotNull) {code}
with the exception:
{code:java}
org.apache.spark.sql.AnalysisException: 
Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column
(named _corrupt_record by default). For example:
spark.read.schema(schema).csv(file).filter($"_corrupt_record".isNotNull).count()
and spark.read.schema(schema).csv(file).select("_corrupt_record").show().
Instead, you can cache or save the parsed results and then send the same query.
For example, val df = spark.read.schema(schema).csv(file).cache() and then
df.filter($"_corrupt_record".isNotNull).count().
      
    at org.apache.spark.sql.errors.QueryCompilationErrors$.queryFromRawFilesIncludeCorruptRecordColumnError(QueryCompilationErrors.scala:2047)
    at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat.buildReader(CSVFileFormat.scala:116) {code}
Bug: Update histogram_numeric (x, y) result type to make x == input type 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/22 18:35
Affects Version/s: 3.3.0

Currently the `histogram_numeric` function always returns a struct of (x, y) double-precision floating-point pairs. However, the function may accept various numeric input types including integers, intervals, etc., in which case users must generally explicitly cast the double-precision result back to the original input type. In this improvement, we will update the output type to propagate the input type to the output.

Note that usage analysis for this function does not appear to yield much traffic currently, so it seems OK to just change the output type directly.
Bug: Support specify initial partition number for rebalance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/22 04:22
Affects Version/s: 3.3.0

Rebalance partitions resolve the skew issue during shuffle dataset. It always returns an indeterminate partition number so at the beginning we do not pass partition as parameter.

 

However, we find the initial partition number can affect the data compression ratio. So it would be better to make the partition number isolation.

 

Note that, it only affects the initial partition number at map side during shuffle.
Bug: AQE throw exception should respect SparkFatalException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Mar/22 04:55
Affects Version/s: 3.3.0

BroadcastExchangeExec will wrap fatal exception inside SparkFatalException and unwarp it before throw.

AQE should also respect SparkFatalException  and throw original error.
{code:java}
Caused by: org.apache.spark.util.SparkFatalException
	at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.$anonfun$relationFuture$1(BroadcastExchangeExec.scala:168)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withThreadLocalCaptured$1(SQLExecution.scala:191)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748) {code}
Bug: Disable to_timestamp('366', 'DD') test case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/22 08:34
Affects Version/s: 3.3.0

Currently, Daily Java 11 and 17 build are broken.
- https://github.com/apache/spark/runs/5511239176?check_suite_focus=true
- https://github.com/apache/spark/actions/runs/1969736117

**Java 8**
{code}
$ bin/spark-shell --conf spark.sql.ansi.enabled=true
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/03/12 00:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://172.16.0.31:4040
Spark context available as 'sc' (master = local[*], app id = local-1647075572229).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0-SNAPSHOT
      /_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_322)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("select to_timestamp('366', 'DD')").show
java.time.format.DateTimeParseException: Text '366' could not be parsed, unparsed text found at index 2. If necessary set spark.sql.ansi.enabled to false to bypass this error.
{code}

**Java 11+**
{code}
$ bin/spark-shell --conf spark.sql.ansi.enabled=true
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/03/12 01:00:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://172.16.0.31:4040
Spark context available as 'sc' (master = local[*], app id = local-1647075607932).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.0-SNAPSHOT
      /_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.12)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("select to_timestamp('366', 'DD')").show
java.time.DateTimeException: Invalid date 'DayOfYear 366' as '1970' is not a leap year. If necessary set spark.sql.ansi.enabled to false to bypass this error.
{code}
Bug: Distribution with hadoop-provided is missing log4j2
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 10/Mar/22 12:40
Affects Version/s: 3.3.0

Currently building spark 3.3.0-SNAPSHOT using `./dev/make-distribution.sh --tgz --name hadoop-provided-test -Phadoop-provided -Pyarn` script will build a package that does not included log4j2. Trying to run spark-submit with the latest hadoop release 3.3.2 and this build will result in


{code:java}
$ spark-submit run-example org.apache.spark.examples.SparkPi
Error: Unable to initialize main class org.apache.spark.deploy.SparkSubmit
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter
{code}


Since log4j2 is not found. So I believe the maven build settings needs to be tweaked so that log4j2 is included in the spark distribution.
Bug: Substitute Java overflow exception from TIMESTAMPADD by Spark exception
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Mar/22 11:01
Affects Version/s: 3.3.0

Currently, Spark SQL can throw Java exceptions from the timestampadd()/date_add()/dateadd() functions, for instance:

{code:java}
spark-sql> select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03');
22/03/09 14:47:15 ERROR SparkSQLDriver: Failed in [select timestampadd(YEAR, 1000000, timestamp'2022-03-09 01:02:03')]
java.lang.ArithmeticException: long overflow
	at java.lang.Math.multiplyExact(Math.java:892) ~[?:1.8.0_292]
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.instantToMicros(DateTimeUtils.scala:505) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAddMonths(DateTimeUtils.scala:724) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.timestampAdd(DateTimeUtils.scala:1197) ~[spark-catalyst_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
{code}

That might confuse non-Scala/Java users. Need to wrap such kind of exception by Spark's exception using an error class.
Bug: Aggregate.groupOnly support foldable expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Mar/22 02:36
Affects Version/s: 3.3.0

{code:scala}
    spark.range(200L).selectExpr("id AS a").createTempView("t1")
    spark.range(300L).selectExpr("id AS b").createTempView("t2")
    spark.sql("SELECT DISTINCT a AS newAlias, true FROM t1 LEFT JOIN t2 ON a = b").explain(true)
{code}

Expected optimized plan:
{noformat}
== Optimized Logical Plan ==
Aggregate [newAlias#8L], [newAlias#8L, true#9L]
+- Project [id#0L AS newAlias#8L, true#9L]
   +- Range (0, 200, step=1, splits=Some(2))
{noformat}
Bug: Support add StreamingListener by conf
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Mar/22 06:34
Affects Version/s: 3.2.1

Currently, if user want to add an customized StreamingListener to StreamingContext, we need to add this listener in customized code.
```
streamingContext.addStreamingListener()
```

we should support use conf to add this
Bug: Partition Data Type Prevents Filtering Sporadically
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/22 19:00
Affects Version/s: 3.2.0

A pipeline (an airflow DAG) that has been running successfully in +production+ for 72+ hours has started failing with the same error on two different queries with the only difference being the table. We believe the root of the error is 


{quote}Caused by: MetaException(message:Filtering is supported only on partition keys of type string){quote}
 

We've seen this error resolve itself on task retry attempts, but the latest occurrence of this task was not resolved on retry attempts, and all proceeding airflow DAGs failed. The queries that trigger this error are 
{quote}select * from db.cleansed_layer_table  where (`dataset`='20220305185000_4d' AND `date_partition`=CAST('2022-03-05' as DATE)):


select * from db.raw_layer_table  where (`date_partition`=CAST('2022-03-05' as DATE) AND `dataset`='20220305185000_4d')
{quote}
 

The date_partition field was a DATE type when this error started occurring. The task writes and queries the raw layer before the cleansed layer is written & queried.

 

The first task failure was caused by the cleansed layer query, and the proceeding ones all failed on the raw layer query. The inconsistent behavior of the pipeline is of highest concern; there were 35 successful DAG runs in Airflow of this pipeline.

 

The error suggests
{quote}{{You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem}}
{quote}
which resulted in too large of a performance hit to keep. 

 

We've changed the field to a STRING in our +development+ environment, and have had 78 consecutive successful __ task runs. We've paused that test for now, in favor of filtering only on dataset for now which we just started running.

 

Is our assessment that we will experience higher reliability by changing the data type of date_partition to STRING reasonable?
Bug: Improve K8s Integration Tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 19:40
Affects Version/s: 3.3.0

This JIRA aims to improve K8s integration tests for the following recent activity.
 - Support Java 17 / Native Apple Silicon
 - Cloud K8s testing (including Graviton instances and K8s 1.22 GA on EKS on March 15th)
 - New K8s model (K8s clients)
 - New custom schedulers and statefulset
Bug: Dynamic serialization of Java datetime objects to micros/days
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/22 19:36
Affects Version/s: 3.3.0

Make serializers to micros/days more tolerant to input Java objects, and accept:
- for timestamps: java.sql.Timestamp and java.time.Instant
- for days: java.sql.Date and java.time.LocalDate

This should make Spark SQL more reliable to user's and datasource inputs.
Bug: ANSI Cast: loosen the limitation of casting non-null complex types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/22 15:44
Affects Version/s: 3.2.2

When ANSI mode is off, `ArrayType(DoubleType, containsNull = false)` can't cast as `ArrayType(IntegerType, containsNull = false)` since there can be overflow thus result in null results and breaks the non-null constraint.

 

When ANSI mode is on, currently Spark SQL has the same behavior. However, this is not correct since the non-null constraint won't be break. Spark SQL can just execute the cast and throw runtime error on overflow, just like casting DoubleType as IntegerType.

 

This applies to MapType and StructType as well.
Bug: `from` and `to` is swapped in the StateSchemaCompatibilityChecker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/22 06:06
Affects Version/s: 3.1.2

During the work in SPARK-38204, I figured out the parameter is swapped which led to test failure on new test (I disabled the schema check for now in the PR of SPARK-38204).

That allows nullable column to be stored into non-nullable column, which should be prohibited. This is less likely making runtime problem since state schema is conceptual one and row can be stored even not respecting the state schema.

The worse problem is happening in opposite way, that disallows non-nullable column to be stored into nullable column, which should be allowed. Spark fails the query for this case.

We should fix this to allow the case properly.
Bug: Improve error messages of 'mismatched input' cases from ANTLR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/22 05:34
Affects Version/s: 3.3.0

Please view the parent task description for the general idea: https://issues.apache.org/jira/browse/SPARK-38384
h1. Mismatched Input
h2. Case 1

Before
{code:java}
ParseException: 
mismatched input 'sel' expecting {'(', 'APPLY', 'CONVERT', 'COPY', 'OPTIMIZE', 'RESTORE', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'SYNC', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
sel 1
^^^ {code}
After
{code:java}
ParseException: 
syntax error at or near 'sel'(line 1, pos 0)

== SQL ==
sel 1
^^^ {code}
Changes:
 # Adjust the words, from ‘mismatched input {}’ to more readable one, ‘syntax error at or near {}.’. This also aligns with the PostgreSQL error messages. 
 # Remove the expecting full list.

h2. Case 2

Before
{code:java}
ParseException: 
mismatched input '<EOF>' expecting {'(', 'CONVERT', 'COPY', 'OPTIMIZE', 'RESTORE', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==

^^^ {code}
After
{code:java}
ParseException: 
syntax error, unexpected empty SQL statement(line 1, pos 0)

== SQL ==

^^^{code}
Changes:
 # For empty query, output a specific error message ‘syntax error, unexpected empty SQL statement’

h2. Case 3

Before
{code:java}
ParseException: 
mismatched input '<EOF>' expecting {'APPLY', 'CALLED', 'CHANGES', 'CLONE', 'COLLECT', 'CONTAINS', 'CONVERT', 'COPY', 'COPY_OPTIONS', 'CREDENTIAL', 'CREDENTIALS', 'DEEP', 'DEFINER', 'DELTA', 'DETERMINISTIC', 'ENCRYPTION', 'EXPECT', 'FAIL', 'FILES',… (omit long message) 'TRIM', 'TRUE', 'TRUNCATE', 'TRY_CAST', 'TYPE', 'UNARCHIVE', 'UNBOUNDED', 'UNCACHE', 'UNION', 'UNIQUE', 'UNKNOWN', 'UNLOCK', 'UNSET', 'UPDATE', 'USE', 'USER', 'USING', 'VALUES', 'VERSION', 'VIEW', 'VIEWS', 'WHEN', 'WHERE', 'WINDOW', 'WITH', 'WITHIN', 'YEAR', 'ZONE', IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 1, pos 11)

== SQL ==
select 1  (
-----------^^^ {code}
After
{code:java}
ParseException: 
syntax error at or near end of input(line 1, pos 11)

== SQL ==
select 1  (
-----------^^^{code}
Changes:
 # For the faulty token <EOF>, substitute it to a readable string ‘end of input’.
Bug: Using Available Resources in Yarn Cluster Information in Spark Dynamic Allocation
Status: Open | Priority: Major | Resolution: 
Created: 08/Mar/22 12:14
Affects Version/s: 3.2.1

Yarn Cluster Manager Provides information on available resources (VCores / Memory) in the Cluster via AM-RM heartbeat. In autoscaling, usually, latency to start executor containers on nodes that are immediately available in the cluster is comparatively lesser as compared to latency in adding new nodes to the cluster. Dynamic Allocation can leverage this information of the number of resources immediately available and latency in adding new nodes in deciding the number of executors to be requested from Yarn RM.

This improvement can be built in two parts:
 #  Infra to Send Available VCores and Memory information from Yarn AM-RM heartbeat response to ExecutorAllocationClient.
 # Leveraging Available VCores and Memory information in ExecutorAllocationManger to decide the number of executors to be requested from Yarn RM.

I'll create PRs for both the tasks one by one.
Bug: Improve perfermance of ShufflePartitionsUtil createSkewPartitionSpecs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/22 12:25
Affects Version/s: 3.3.0

If shuffle is skewed with tens of thousands of map partitions and reduce partitions in AQE, the method `ShufflePartitionsUtil#createSkewPartitionSpecs` will be very slow. More unfortunately, it is running at driver side.
Bug: Automatically calculate the upper and lower bounds of partitions when no specified partition related params
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Mar/22 08:31
Affects Version/s: 3.1.2

when access rdbms, such as mysql, if partitionColumn, lowerBound, upperBound, numPartitions are not specified, by default only one partition to scan database is working. 

It makes load data from database slow and makes it difficult for users to configure multiple parameters to improve parallelism.
Bug: Supports customization of the entire  optimizer and planner.
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Mar/22 11:07
Affects Version/s: 3.1.1

In some scenarios, users need to add some optimization rules after all existing optimization rules are executed. However, the current sparksession extension cannot meet the requirements.

PR:https://github.com/apache/spark/pull/35721
Bug: Fix the permissions for GitHub workflows
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Mar/22 16:02
Affects Version/s: 3.2.1

# The workflow files don't have permission restricted. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions
 # In some of the workflows, the actions aren't pinned by SHA. https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies