av9ash/GitBugs huggingface dataset:
Bug: Restore F403 checks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/22 11:50
Affects Version/s: 3.3.0

Adding global exclude on F403 (SPARK-37836) seemed to affect F401 unused import checks.

See [https://github.com/apache/spark/pull/35199#discussion_r784526473] and related CI runs.
Bug: [SQL] Some classes are move from org.codehaus.janino:janino to org.codehaus.janino:common-compiler after version 3.1.x 
Status: Open | Priority: Major | Resolution: 
Created: 12/Jan/22 03:49
Affects Version/s: 2.4.5

Here is the code:

 

[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L32]

 

ByteArrayClassLoader and InternalCompilerException are moved to org.codehaus.janino:common-compiler

 

[https://github.com/janino-compiler/janino/blob/3.1.6/commons-compiler/src/main/java/org/codehaus/commons/compiler/util/reflect/ByteArrayClassLoader.java]

 

[https://github.com/janino-compiler/janino/blob/3.1.6/commons-compiler/src/main/java/org/codehaus/commons/compiler/InternalCompilerException.java]

 

The last working version of janino is 3.0.16 but it is out of date.

Can we make change to this and upgrade to new version of janino and common-compiler?
Bug: Show test report in GitHub Actions builds from PRs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/22 12:51
Affects Version/s: 3.3.0

Currently, the test report like https://github.com/apache/spark/runs/4788468586 is not directly shown in workflow runs in the link provided in PRs, e.g.) https://github.com/yaooqinn/spark/actions/runs/1687326379

We should make the test report visible.
Bug: Allow pandas_udf to take type annotations with future annotations enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 00:57
Affects Version/s: 3.3.0

When using {{{}from __future__ import annotations{}}}, the type hints will be all strings, then pandas UDF type inference won't work as follows:
{code:python}
>>> from __future__ import annotations
>>> from typing import Union
>>> import pandas as pd
>>> from pyspark.sql.functions import pandas_udf
>>> @pandas_udf("long")
... def plus_one(v: Union[pd.Series, pd.DataFrame]) -> pd.Series:
...     return v + 1

Traceback (most recent call last):
...
NotImplementedError: Unsupported signature: (v: 'Union[pd.Series, pd.DataFrame]') -> 'pd.Series'.
{code}
Bug: PySpark shell sets log level to INFO by default 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/22 01:34
Affects Version/s: 3.3.0

{code}
./bin/pyspark
{code}

{code}
Python 3.9.5 (default, May 18 2021, 12:31:01)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
22/01/13 10:28:15 INFO HiveConf: Found configuration file null
22/01/13 10:28:15 INFO SparkContext: Running Spark version 3.3.0-SNAPSHOT
...
>>> spark.range(10)
22/01/13 10:31:48 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
...
{code}
Bug: Update pytest-mypy-plugins to 1.9.3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 16:09
Affects Version/s: 3.3.0

Currently we install from specific revision

https://github.com/apache/spark/blob/3faced8b2ab922c2a3f25bdf393aa0511d87ccc8/.github/workflows/build_and_test.yml#L481

but a new version (1.9.3) was released today and we can update to PyPI.
Bug: security scan issue with htrace-core4-4.1.0-incubating
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 28/Jan/22 18:55
Affects Version/s: 3.2.0

Hi,

running into security scan issue with docker image built on spark-3.2.0-bin-hadoop3.2, is there a way to resolve 

 

most issues related to https://issues.apache.org/jira/browse/HDFS-15333 

attaching the CVE report
Bug: spark thrift server RDD ID overflow lead sql execute failed
Status: Open | Priority: Major | Resolution: 
Created: 06/Jan/22 02:55
Affects Version/s: 3.2.0

this problem will happen in long run spark application，such as thrift server；

as only one SparkContext instance in thrift server driver size，so if the concurrency of sql request is large or the sql is too complicate（this will create a lot of rdd）, the rdd will be generate too fast , the rdd id （SparkContext.scala#nextRddId:[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala] ）will be consume fast, after a few months the nextRddId will overflow。the newRddId will be negative number，but the rdd's block id need to be positive, so this will lead a exception"Failed to parse rdd_-2123452330_2 into block ID"（rdd block id formate“val RDD = "rdd_([0-9]{+})_([0-9]{+})".r”：[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockId.scala]），so can not exchange data during sql execution, and lead sql execute failed

if rddId overflow , when rdd.MapPartition execute , error will occur, the error is occur on driver side, when driver deserialize block id from "block message" inputstream

when executor invoke rdd.MapPartition, it will call block manager to report block status,  the the block id is negative，when the message send back to driver , the driver regex will failed match and throw an exception

 

how to fix the problem???

SparkContext.scala

 
{code:java}
...
...
 private val nextShuffleId = new AtomicInteger(0)
  private[spark] def newShuffleId(): Int = nextShuffleId.getAndIncrement()
  private var nextRddId = new AtomicInteger(0) // change happen
  /** Register a new RDD, returning its RDD ID */  
// change happen
private[spark] def newRddId(): Int = {
  var id = nextRddId.getAndIncrement()
  if (id > 0) {
    return id
  }
  this.synchronized {
    id = nextRddId.getAndIncrement()
   if (id < 0) {       
      nextRddId = new AtomicInteger(0)
      id = nextRddId.getAndIncrement()
    }
  }
  id
}
...
...{code}
Bug: Enable more flake8 rules for PEP 8 compliance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/22 05:31
Affects Version/s: 3.3.0

Most of disabled linter rules here:

https://github.com/apache/spark/blob/master/dev/tox.ini#L19-L31

should be enabled to comply PEP8.
Bug: RuntimeError: Java gateway process exited before sending its port number
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 30/Jan/22 17:12
Affects Version/s: 3.2.1

I am getting 

RuntimeError: Java gateway process exited before sending its port number when running python tests in  Docker.  

 

I am using *spark_home* as /usr/lib/python3.9/site-packages/pyspark

 

sc = SparkContext() File "/usr/lib/python3.9/site-packages/pyspark/context.py", line 144, in __init__ SparkContext._ensure_initialized(self, gateway=gateway, conf=conf) File

"/usr/lib/python3.9/site-packages/pyspark/context.py", line 339, in _ensure_initialized

SparkContext._gateway = gateway or launch_gateway(conf)

File "/usr/lib/python3.9/site-packages/pyspark/java_gateway.py", line 136, in launch_gateway raise RuntimeError("Java gateway process exited before sending its port number")

RuntimeError: Java gateway process exited before sending its port number

 

I can provide additional details. Any Help is appreciated
Bug: Update atexit function to avoid issues with late binding
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jan/22 15:25
Affects Version/s: 3.2.0

When {{PYSPARK_DRIVER_PYTHON=$(which ipython) bin/pyspark}} is executed with Python >= 3.8, function registered wiht atexit seems to be executed in different scope than in Python 3.7.

It result in {{NameError: name 'sc' is not defined}} on exit:

{code:python}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.3.0-SNAPSHOT
      /_/

Using Python version 3.8.12 (default, Oct 12 2021 21:57:06)
Spark context Web UI available at http://192.168.0.198:4040
Spark context available as 'sc' (master = local[*], app id = local-1643555855409).
SparkSession available as 'spark'.

In [1]:                                                                                                                                                                                 
Do you really want to exit ([y]/n)? y
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/path/to/spark/python/pyspark/shell.py", line 49, in <lambda>
    atexit.register(lambda: sc.stop())
NameError: name 'sc' is not defined
{code}


This could be easily fixed by capturing `sc` instance

{code:none}
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index f0c487877a..4164e3ab0c 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -46,7 +46,7 @@ except Exception:
 
 sc = spark.sparkContext
 sql = spark.sql
-atexit.register(lambda: sc.stop())
+atexit.register((lambda sc: lambda: sc.stop())(sc))
 
 # for compatibility
 sqlContext = spark._wrapped
{code}
Bug: Writing a spark dataframe to Azure Sql Server is causing duplicate records intermittently
Status: Open | Priority: Major | Resolution: 
Created: 28/Jan/22 15:07
Affects Version/s: 3.1.0

We are using JDBC option to insert transformed data in a spark DataFrame to a table in Azure SQL Server. Below is the code snippet we are using for this insert. However, we noticed on few occasions that some records are being duplicated in the destination table. This is happening for large tables. e.g. if a DataFrame has 600K records, after inserting data into the table, we get around 620K records.  we still want to understand why that's happening.
 {{DataToLoad.write.jdbc(url = jdbcUrl, table = targetTable, mode = "overwrite", properties = jdbcConnectionProperties)}}
 
Only reason we could think of is that while inserts are happening in distributed fashion, if one of the executors fail in between, they are being re-tried and could be inserting duplicate records. This could be totally meaningless but just to see if that could be an issue.{{{}{}}}
Bug: Pyspark SparkContext.AddFile() does not respect spark.files.overwrite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/22 19:45
Affects Version/s: 3.1.1

I am currently running apache spark 3.1.1. on kubernetes.

When I try to re-add a file that has already been added I see that the updated file is not actually loaded into the cluster. I see the following warning when calling the addFile() function.
{code:java}
22/01/18 19:05:50 WARN SparkContext: The path http://15.4.12.12:80/demo_data.csv has been added already. Overwriting of added paths is not supported in the current version. {code}
When I display the dataframe that was loaded I see that the old data is loaded. If I log into the worker pods and delete the file, the same results or observed.

My SparkConf has the following configurations
{code:java}
('spark.master', 'k8s://https://15.4.7.11:6443')
('spark.app.name', 'spark-jupyter-mlib')
('spark.submit.deploy.mode', 'cluster')
('spark.kubernetes.container.image', 'tschneider/apache-spark-k8:v7')
('spark.kubernetes.namespace', 'spark')
('spark.kubernetes.pyspark.pythonVersion', '3')
('spark.kubernetes.authenticate.driver.serviceAccountName', 'spark-sa')
('spark.kubernetes.authenticate.serviceAccountName', 'spark-sa')
('spark.executor.instances', '3')
('spark.executor.cores', '2')
('spark.executor.memory', '4096m')
('spark.executor.memoryOverhead', '1024m')
('spark.driver.memory', '1024m')
('spark.driver.host', '15.4.12.12')
('spark.files.overwrite', 'true')
('spark.files.useFetchCache', 'false') {code}
According to the documentation for 3.1.1. The spark.files.overwrite parameter should in fact load the updated files. The documentation can be found here: [https://spark.apache.org/docs/3.1.1/configuration.html]

The only workaround is to use a python function to manually delete and re-download the file. Calling addFile still shows the warning in this case. My code for the delete and redownload is as follows:
{code:java}
def os_remove(file_path):
    import socket
    hostname = socket.gethostname()    action = None
    import os
    if os.path.exists(file_path):
        action = "delete"
        os.remove(file_path)
        
    return (hostname, action)worker_file_path = u"file:///{0}".format(csv_file_name)


worker_count = int(spark_session.conf.get('spark.executor.instances'))
rdd = sc.parallelize(range(worker_count)).map(lambda var: os_remove(worker_file_path))
rdd.collect()


def download_updated_file(file_url):
    import urllib.parse as parse
    file_name = os.path.basename(parse.urlparse(csv_file_url).path)
    local_file_path = "/{0}".format(file_name)
    
    import urllib.request as urllib
    urllib.urlretrieve(file_url, local_file_path)
    

rdd = sc.parallelize(range(worker_count)).map(lambda var: download_updated_file(csv_file_url))
rdd.collect(){code}
I believe this is either a bug or a documentation mistake. Perhaps the configuration parameter has a misleading description?
Bug: Log4j remediation on spark binary files
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 28/Jan/22 14:50
Affects Version/s: 3.2.0

Hi Team , We were checking for log4j upgrade in Open source spark version to avoid the recent vulnerability in the spark binary . Do we have any new release which is planned to upgrade the log4j  from 1.2.17 to  2.17.1.Any sooner response is appreciated ?
Bug: DataFilter pushed down with PartitionFilter
Status: In Progress | Priority: Major | Resolution: 
Created: 27/Jan/22 04:10
Affects Version/s: 3.4.0

At present, the Filter is divided into DataFilter and PartitionFilter when it is pushed down, but when the Filter removes the PartitionFilter, it means that all Partitions will scan all DataFilter conditions, which may cause full data scan.

Here is a example.

before
{code:java}
== Physical Plan ==
*(1) Filter (((a#40L < 10) AND (c#42 = 0)) OR (((a#40L >= 10) AND (c#42 >= 1)) AND (c#42 < 3)))
+- *(1) ColumnarToRow
   +- BatchScan[a#40L, b#41L, c#42, d#43] ParquetScan DataFilters: [(((a#40L < 10) AND (c#42 = 0)) OR (((a#40L >= 10) AND (c#42 >= 1)) AND (c#42 < 3)))], Format: parquet, Location: InMemoryFileIndex(1 paths)[path, PartitionFilters: [((c#42 = 0) OR ((c#42 >= 1) AND (c#42 < 3)))], PushedAggregation: [], PushedFilters: [Or(LessThan(a,10),GreaterThanOrEqual(a,10))], PushedGroupBy: [], ReadSchema: struct<a:bigint,b:bigint>, PushedFilters: [Or(LessThan(a,10),GreaterThanOrEqual(a,10))], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: []
{code}
after
{code:java}
== Physical Plan ==
*(1) Filter (((a#40L < 10) AND (c#42 = 0)) OR (((a#40L >= 10) AND (c#42 >= 1)) AND (c#42 < 3)))
+- *(1) ColumnarToRow
   +- BatchScan[a#40L, b#41L, c#42, d#43] ParquetScan DataFilters: [(((a#40L < 10) AND (c#42 = 0)) OR (((a#40L >= 10) AND (c#42 >= 1)) AND (c#42 < 3)))], Format: parquet, Location: InMemoryFileIndex(1 paths)[path, PartitionFilters: [((c#42 = 0) OR ((c#42 >= 1) AND (c#42 < 3)))], PushedAggregation: [], PushedFilters: [Or(And(LessThan(a,10),EqualTo(c,0)),And(And(GreaterThanOrEqual(a,10),GreaterThanOrEqual(c,1)),Le..., PushedGroupBy: [], ReadSchema: struct<a:bigint,b:bigint>, PushedFilters: [Or(And(LessThan(a,10),EqualTo(c,0)),And(And(GreaterThanOrEqual(a,10),GreaterThanOrEqual(c,1)),LessThan(c,3)))], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: [] {code}
Bug: Enable binary compatibility check for APIs in Catalyst, KVStore and Avro modules
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/22 03:07
Affects Version/s: 3.3.0

{code}
./dev/mima
{code}

We don't currently run binary compatibility check in below modules:

{code}
[info] spark-parent: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-network-common: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-tags: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-unsafe: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-network-shuffle: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-kvstore: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-tools: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-token-provider-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-streaming-kafka-0-10-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-catalyst: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-repl: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-avro: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-sql-kafka-0-10: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-hive: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-assembly: mimaPreviousArtifacts not set, not analyzing binary compatibility
[info] spark-examples: mimaPreviousArtifacts not set, not analyzing binary compatibility
{code}

However, there are some APIs under these modules. For example, https://github.com/apache/spark/blob/master/external/avro/src/main/scala/org/apache/spark/sql/avro/functions.scala for Avro,  https://github.com/apache/spark/tree/master/common/kvstore/src/main/java/org/apache/spark/util/kvstore for KVStore (to be API), and https://github.com/apache/spark/tree/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector for Catalyst
Bug: TPCDS 1TB q72 fails when spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly is false
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/22 09:11
Affects Version/s: 3.2.0

TPCDS 1TB q72 fails in 3.2 Spark when spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly is false. We have been running with this config in 3.1 as well and it worked fine in that version. This used to add a subquery dpp in q72.

Relevant stack trace
{code:java}
rror: java.lang.ClassCastException: org.apache.spark.sql.catalyst.plans.logical.Project cannot be cast to org.apache.spark.sql.execution.SparkPlan  at scala.collection.immutable.List.map(List.scala:293)  at org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:75)  at org.apache.spark.sql.execution.SparkPlanInfo$.$anonfun$fromSparkPlan$3(SparkPlanInfo.scala:75)  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
....
....
at org.apache.spark.sql.execution.SparkPlanInfo$.fromSparkPlan(SparkPlanInfo.scala:75)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.onUpdatePlan(AdaptiveSparkPlanExec.scala:708)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$2(AdaptiveSparkPlanExec.scala:239)  at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.java:23)  at scala.Option.foreach(Option.scala:407)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.$anonfun$getFinalPhysicalPlan$1(AdaptiveSparkPlanExec.scala:239)  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.getFinalPhysicalPlan(AdaptiveSparkPlanExec.scala:226)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.withFinalPlanUpdate(AdaptiveSparkPlanExec.scala:365)  at org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanExec.executeCollect(AdaptiveSparkPlanExec.scala:338) {code}
Bug: Mark org.apache.spark.sql.hive.execution as slow tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/22 03:33
Affects Version/s: 3.3.0

Related to SPARK-33171 and SPARK-32884. We should rebalance both:

"hive -slow tests": https://github.com/apache/spark/runs/4755996153?check_suite_focus=true
"hive - other tests": https://github.com/apache/spark/runs/4755996212?check_suite_focus=true
Bug: Enable black formatter in dev Python scripts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/22 06:18
Affects Version/s: 3.3.0

black formatter is only enabled for python/pyspark to minimize side-effects e.g., reformating auto generated or thrid-party Python scripts.

This JIRA aims to enable black formatter in dev directory where there's no generated Python scripts to exclude.
Bug: Add submitTime for Spark Application
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/22 06:45
Affects Version/s: 3.3.0

Spark Driver may take a long time to startup when running with cluster mode, if YARN temporarily does not have enough resources, or HDFS's performance is poor due to high pressure.

After`submitTime` is added, we can detect such situation by comparing it with the `spark.app.startTime`(already exists) .
Bug: Use error classes in the execution errors related to partitions
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jan/22 17:48
Affects Version/s: 3.3.0

Migrate the following errors in QueryExecutionErrors:
* unableToDeletePartitionPathError
* unableToCreatePartitionPathError
* unableToRenamePartitionPathError
* notADatasourceRDDPartitionError
* cannotClearPartitionDirectoryError
* failedToCastValueToDataTypeForPartitionColumnError
* unsupportedPartitionTransformError
* cannotCreateJDBCTableWithPartitionsError
* requestedPartitionsMismatchTablePartitionsError
* dynamicPartitionKeyNotAmongWrittenPartitionPathsError
* cannotRemovePartitionDirError
* alterTableWithDropPartitionAndPurgeUnsupportedError
* invalidPartitionFilterError
* getPartitionMetadataByFilterError
* illegalLocationClauseForViewPartitionError
* partitionColumnNotFoundInSchemaError
* cannotAddMultiPartitionsOnNonatomicPartitionTableError
* cannotDropMultiPartitionsOnNonatomicPartitionTableError
* truncateMultiPartitionUnsupportedError
* dynamicPartitionOverwriteUnsupportedByTableError
* writePartitionExceedConfigSizeWhenDynamicPartitionError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryExecutionErrorsSuite.
Bug: Use error classes in the execution errors of arithmetic ops
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/22 17:39
Affects Version/s: 3.3.0

Migrate the following errors in QueryExecutionErrors:
* overflowInSumOfDecimalError
* overflowInIntegralDivideError
* arithmeticOverflowError
* unaryMinusCauseOverflowError
* binaryArithmeticCauseOverflowError
* unscaledValueTooLargeForPrecisionError
* decimalPrecisionExceedsMaxPrecisionError
* outOfDecimalTypeRangeError
* integerOverflowError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryExecutionErrorsSuite.
Bug: Use error classes in the compilation errors of partitions
Status: Open | Priority: Major | Resolution: 
Created: 17/Jan/22 17:16
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* unsupportedIfNotExistsError
* nonPartitionColError
* missingStaticPartitionColumn
* alterV2TableSetLocationWithPartitionNotSupportedError
* invalidPartitionSpecError
* partitionNotSpecifyLocationUriError
* describeDoesNotSupportPartitionForV2TablesError
* tableDoesNotSupportPartitionManagementError
* tableDoesNotSupportAtomicPartitionManagementError
* alterTableRecoverPartitionsNotSupportedForV2TablesError
* partitionColumnNotSpecifiedError
* invalidPartitionColumnError
* multiplePartitionColumnValuesSpecifiedError
* cannotUseDataTypeForPartitionColumnError
* cannotUseAllColumnsForPartitionColumnsError
* partitionColumnNotFoundInSchemaError
* mismatchedTablePartitionColumnError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Add yunikorn feature step
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 04/Jan/22 17:22
Affects Version/s: 3.2.0

Based on the design doc https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg. This task is to add YuniKornFeatureStep in order to integrate with YuniKorn.