av9ash/GitBugs huggingface dataset:
Bug: ml.recommendation.ALS doctests failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Feb/22 12:26
Affects Version/s: 3.3.0

In my dev setups, ml.recommendation:ALS test consistently converges to value lower than expected and fails with:

{code:python}
File "/path/to/spark/python/pyspark/ml/recommendation.py", line 322, in __main__.ALS
Failed example:
    predictions[0]
Expected:
    Row(user=0, item=2, newPrediction=0.69291...)
Got:
    Row(user=0, item=2, newPrediction=0.6929099559783936)
{code}

In can correct for that, but it creates some noise, so if anyone else experiences this, we could drop  a digit from the results

{code}
diff --git a/python/pyspark/ml/recommendation.py b/python/pyspark/ml/recommendation.py
index f0628fb922..b8e2a6097d 100644
--- a/python/pyspark/ml/recommendation.py
+++ b/python/pyspark/ml/recommendation.py
@@ -320,7 +320,7 @@ class ALS(JavaEstimator, _ALSParams, JavaMLWritable, JavaMLReadable):
     >>> test = spark.createDataFrame([(0, 2), (1, 0), (2, 0)], ["user", "item"])
     >>> predictions = sorted(model.transform(test).collect(), key=lambda r: r[0])
     >>> predictions[0]
-    Row(user=0, item=2, newPrediction=0.69291...)
+    Row(user=0, item=2, newPrediction=0.6929...)
     >>> predictions[1]
     Row(user=1, item=0, newPrediction=3.47356...)
     >>> predictions[2]

{code}
Bug: Support `na_filter` for pyspark.pandas.read_csv
Status: In Progress | Priority: Major | Resolution: 
Created: 23/Feb/22 01:31
Affects Version/s: 3.3.0

pandas support `na_filter` parameter for `read_csv` function. (https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html)

We also want to support this to follow the behavior of pandas.
Bug: when clean data hope to spilt one dataframe or dataset  to two dataframe
Status: Open | Priority: Major | Resolution: 
Created: 09/Feb/22 09:07
Affects Version/s: 3.2.1

when I am  processing  data clean, I meet such scene.

one coloumn need judge by empy or null condition.

so I do it right now similar code as following:

df1= dataframe.filter("coloumn=null")

df2= dataframe.filter("coloumn=!null")

and then write df1 and df2 into hdfs parquet file.

but when i have thousand condition. every job need more stage.

I hope dataframe can filter by one condition once and not twice. and that can generate two dataframe.
Bug: SparkR documentation build fails in CI
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 10/Feb/22 11:46
Affects Version/s: 3.3.0

SparkR documentation job in GitHub Actions seems to be broken now as below (https://github.com/apache/spark/runs/5138914521?check_suite_focus=true):

{code}
Writing 'reference/head.html'
Reading 'man/hint.Rd'
Writing 'reference/hint.html'
Reading 'man/histogram.Rd'
Error in .f(.x[[i]], ...) : Failed to parse Rd in histogram.Rd
ℹ ReadItem: unknown type 73, perhaps written by later version of R
Caused by error in `readRDS()`:
! ReadItem: unknown type 73, perhaps written by later version of R

Error: <callr_status_error: callr subprocess failed: Failed to parse Rd in histogram.Rd
ℹ ReadItem: unknown type 73, perhaps written by later version of R
Caused by error in `readRDS()`:
! ReadItem: unknown type 73, perhaps written by later version of R>
-->
<callr_remote_error in .f(.x[[i]], ...):
 Failed to parse Rd in histogram.Rd
ℹ ReadItem: unknown type 73, perhaps written by later version of R
Caused by error in `readRDS()`:
! ReadItem: unknown type 73, perhaps written by later version of R>
 in process 13417 

 Stack trace:

 Process 13346:
 1. pkgdown::build_site("..")
 2. pkgdown:::build_site_external(pkg = pkg, examples = examples,  ...
 3. callr::r(function(..., crayon_enabled, crayon_colors, pkgdown_internet) { ...
 4. callr:::get_result(output = out, options)
 5. throw(newerr, parent = remerr[[2]])

 x callr subprocess failed: Failed to parse Rd in histogram.Rd
ℹ ReadItem: unknown type 73, perhaps written by later version of R
Caused by error in `readRDS()`:
! ReadItem: unknown type 73, perhaps written by later version of R 

 Process 13417:
 17. (function (..., crayon_enabled, crayon_colors, pkgdown_internet)  ...
 18. pkgdown::build_site(...)
 19. pkgdown:::build_site_local(pkg = pkg, examples = examples, run_do ...
 20. pkgdown:::build_reference(pkg, lazy = lazy, examples = examples,  ...
 21. purrr::map(topics, build_reference_topic, pkg = pkg, lazy = lazy,  ...
{code}
Bug: Preserve the error class of `AnalysisException` while constructing of function builder
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Feb/22 11:30
Affects Version/s: 3.3.0

When the cause exception is `AnalysisException` at
https://github.com/apache/spark/blob/9c02dd4035c9412ca03e5a5f4721ee223953c004/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L132, Spark loses info about the error class. Need to preserve the info.

The example below demonstrates the issue:

{code:scala}
scala> try { sql("select format_string('%0$s', 'Hello')") } catch { case e: org.apache.spark.sql.AnalysisException => println(e.getErrorClass) }
null
{code}
Bug: Use error classes in the compilation errors of python/pandas UDFs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 16:32
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* pandasUDFAggregateNotSupportedInPivotError
* groupAggPandasUDFUnsupportedByStreamingAggError
* cannotUseMixtureOfAggFunctionAndGroupAggPandasUDFError
* usePythonUDFInJoinConditionUnsupportedError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Change mktemp() to mkstemp()
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 21:16
Affects Version/s: 3.3.0

In the file pandasutils.py on line 262 yield tempfile.mktemp(dir=tmp)

The mktemp() is [deprecated and is not secure|https://docs.python.org/3/library/tempfile.html#deprecated-functions-and-variables]
Bug: Update minimum numpy version to 1.15
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Feb/22 15:44
Affects Version/s: 3.3.0

Currently, we use set numpy version in {{extras_require}} to be {{>=1.7}}.

However, 1.7 has been released almost 9 years ago and since then some methods that we use have been deprecated in favor of new additions and  anew API ({{numpy.typing}}, that is of some interest to us, has been added.

We should update minimum version requirement to one of the following

- {{>=1.9.0}} ‒ this is minimum reasonable bound, that will allow us to replace deprecated {{tostring}} calls with {{tobytes}}.
- {{>=1.15.0}} (released 2018-07-23) ‒ this is reasonable bound to match our minimum supported pandas version.
- {{>=1.20.0}} (released 2021-01-30) ‒ to fully utilize numpy typing.

The last one might be somewhat controversial, but 1.15 shouldn't require much discussion.
Bug: High I/O wait when Spark Structured Streaming checkpoint changed to EFS
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 25/Feb/22 13:45
Affects Version/s: 3.2.1

I'm currently running spark structured streaming application written in python(pyspark) where my source is kafka topic and sink i mongodb. I changed my checkpoint to Amazon EFS, which is distributed on all spark workers and after that I got increased I/o wait, averaging 8%

 

!Screenshot from 2022-02-25 14-16-11.png!

Currently I have 6000 messages coming to kafka every second, and I get every once in a while a WARN message:
{quote}22/02/25 13:12:31 WARN HDFSBackedStateStoreProvider: Error cleaning up files for HDFSStateStoreProvider[id = (op=0,part=90),dir = file:/mnt/efs_max_io/spark/state/0/90] java.lang.NumberFormatException: For input string: ""
{quote}
I'm not quite sure if that message has anything to do with high I/O wait and is this behavior expected, or something to be concerned about?
Bug: Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/22 19:24
Affects Version/s: 3.2.1

Currently we are setting shuffleMergeAllowed to false before prepareShuffleServicesForShuffleMapStage if the shuffle dependency is already finalized. Ideally it is better to do it right after shuffle dependency finalization for a determinate stage. cc [~mridulm80]
Bug: SQL CLI AM should keep same exitcode with client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 07:16
Affects Version/s: 3.2.1

Currently for SQL CLI, we all use  shutdown hook to stop SC

{code:java}
    // Clean up after we exit
    ShutdownHookManager.addShutdownHook { () => SparkSQLEnv.stop() }

{code}
This cause Yarn AM always success even client exit with code not 0.
Bug: Kryo DataWritingSparkTaskResult registration error
Status: Open | Priority: Major | Resolution: 
Created: 02/Feb/22 09:07
Affects Version/s: 3.1.2

Spark 3.1.2, Scala 2.12

I'm registering classes with _sparkConf.registerKryoClasses(Array( ..._ method. Inside there are spark structured streaming code. Following settings are added as well:

sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
sparkConf.set("spark.kryo.registrationRequired", "true")

Unfortunately, during execution following error is thrown:

Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult
Note: To register this class use: kryo.register(org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult.class); 

As far as I can see in [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala] 

class org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTaskResult is private (private[v2] case class DataWritingSparkTaskResult) therefore not available to register.

 

!image-2022-02-02-10-09-14-858.png!
Bug: Parquet: enable matching schema columns by field id
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Feb/22 00:39
Affects Version/s: 3.3.0

Field Id is a native field in the Parquet schema ([https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L398])

After this PR, when the requested schema has field IDs, Parquet readers will first use the field ID to determine which Parquet columns to read, before falling back to using column names as before. It enables matching columns by field id for supported DWs like iceberg and Delta.

This PR supports:
 * vectorized reader
 * Parquet-mr reader
Bug: Supporting custom commitProtocolClass when using saveAsNewAPIHadoopDataset
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Feb/22 15:37
Affects Version/s: 3.2.0

There is no way to apply spark-hadoop-cloud's commitProtocolClass when using saveAsNewAPIHadoopDataset that are to avoid object storage's problem.

[https://spark.apache.org/docs/latest/cloud-integration.html]

 

It is needed to support custom commitProtocolClass class when using saveAsNewAPIHadoopDataset by an option. For example,
{code:java}
spark.hadoop.mapreduce.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol{code}
Bug: PoissonSampler may output more rows than MaxRows
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Feb/22 08:03
Affects Version/s: 3.0.3

{code:java}
scala> val df = spark.range(0, 1000)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> df.count
res0: Long = 1000

scala> df.sample(true, 0.999999, 10).count
res1: Long = 1004
 {code}
Bug: Make memory overhead factor configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/22 14:03
Affects Version/s: 3.4.0

Currently if the memory overhead is not provided for a Yarn job, it defaults to 10% of the respective driver/executor memory. This 10% is hard-coded and the only way to increase memory overhead is to set the exact memory overhead. We have seen more than 10% memory being used, and it would be helpful to be able to set the default overhead factor so that the overhead doesn't need to be pre-calculated for any driver/executor memory size.
Bug: Replace the error classes related to invalid parameters by `INVALID_PARAMETER_VALUE`
Status: Open | Priority: Major | Resolution: 
Created: 06/Feb/22 09:35
Affects Version/s: 3.3.0

Re-use INVALID_PARAMETER_VALUE instead of more specific error classes:
* ILLEGAL_SUBSTRING
* INVALID_ARRAY_INDEX
* INVALID_ARRAY_INDEX_IN_ELEMENT_AT
* INVALID_FRACTION_OF_SECOND
* INVALID_INPUT_SYNTAX_FOR_NUMERIC_TYPE
* MAP_KEY_DOES_NOT_EXIST
* MAP_KEY_DOES_NOT_EXIST_IN_ELEMENT_AT
Bug: Upgrade Jetty version from 9.4.40 to 9.4.44
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 09/Feb/22 03:28
Affects Version/s: 2.4.8

Upgrade Jetty version to 9.4.44 in Spark 2.4.8 and 3.0.3 in order to fix vulnerability CVE-2021-28169, CVE-2021-34429, PRISMA-2021-0182
Bug: Reactor framework so as JDBC dialect could compile expression by self way
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Feb/22 01:47
Affects Version/s: 3.3.0

https://issues.apache.org/jira/browse/SPARK-37960 provides a new framework to represent catalyst expressions in DS V2 APIs.
Because the framework translate all catalyst expressions to a unified SQL string and cannot keep compatibility between different JDBC database, the framework works not good.

This PR reactor the framework so as JDBC dialect could compile expression by self way.
First, The framework translate catalyst expressions to DS V2 expression.
Second, The JDBC dialect could compile DS V2 expression to different SQL syntax.
Bug: Use error classes in the compilation errors of UDF/UDAF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 16:34
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* noHandlerForUDAFError
* unexpectedEvalTypesForUDFsError
* usingUntypedScalaUDFError
* udfClassDoesNotImplementAnyUDFInterfaceError
* udfClassNotAllowedToImplementMultiUDFInterfacesError
* udfClassWithTooManyTypeArgumentsError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.
Bug: Use error classes in the compilation errors of windows
Status: Open | Priority: Major | Resolution: 
Created: 04/Feb/22 16:43
Affects Version/s: 3.3.0

Migrate the following errors in QueryCompilationErrors:
* windowSpecificationNotDefinedError
* windowAggregateFunctionWithFilterNotSupportedError
* windowFunctionInsideAggregateFunctionNotAllowedError
* expressionWithoutWindowExpressionError
* expressionWithMultiWindowExpressionsError
* windowFunctionNotAllowedError
* cannotSpecifyWindowFrameError
* windowFrameNotMatchRequiredFrameError
* windowFunctionWithWindowFrameNotOrderedError
* multiTimeWindowExpressionsNotSupportedError
* sessionWindowGapDurationDataTypeError
* invalidLiteralForWindowDurationError
* emptyWindowExpressionError
* foundDifferentWindowFunctionTypeError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryCompilationErrorsSuite.

*Feel free to split this to sub-tasks.*
Bug: (flat)MapGroupsWithState can timeout groups which just received inputs in the same microbatch
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/22 17:51
Affects Version/s: 3.2.1

We have identified an issue where the RocksDB state store iterator will not pick up store updates made after its creation. As a result of this, the _timeoutProcessorIter_ in

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala]

will not pick up state changes made during _newDataProcessorIter_ input processing. The user observed behavior is that a group state may receive input records and also be called with timeout in the same micro batch. This contradics the public documentation for GroupState -

[https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/streaming/GroupState.html]
 * The timeout is reset every time the function is called on a group, that is, when the group has new data, or the group has timed out. So the user has to set the timeout duration every time the function is called, otherwise, there will not be any timeout set.
Bug: Introduce SQL function ARRAY_SIZE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/22 06:31
Affects Version/s: 3.3.0

Counting elements within an array is a common use case. ARRAY_SIZE ensures the input to be an array and then returns the size.

Other DBRMS like Snowflake supports that as well: https://docs.snowflake.com/en/sql-reference/functions/array_size.html. Implementing that improves compatibility with DBMS and makes migration easier.
Bug: Add the `DATEADD()` alias for `TIMESTAMPADD()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 19:30
Affects Version/s: 3.3.0

Introduce the dateadd() function, which takes three arguments: unit, number of units and datetime expression, i.e.,
{code:sql}
dateadd(unit, number_of_units, expr)
{code}
The function can be an alias to timestampadd().
Bug: Add the TIMESTAMPDIFF() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Feb/22 08:40
Affects Version/s: 3.3.0

Two functions that are part of the ODBC api and implemented virtually by ALL other databases are missing in Spark SQL.

TIMESTAMPADD() and TIMESTAMPDIFF()

See
* [Time, Date, and Interval Functions|https://docs.microsoft.com/en-us/sql/odbc/reference/appendixes/time-date-and-interval-functions?view=sql-server-ver15]
* [TIMESTAMPDIFF function (ODBC compatible)|https://docs.faircom.com/doc/sqlref/33477.htm]

The TIMESTAMPDIFF() function is also VERY convenient and would be a great addition because getting the difference between two timestamps in seconds in spark today requires converting to unixtimestamp and back.

{code:sql}
SELECT TIMESTAMPDIFF(SECOND, '2021-12-30 01:02:00':timestamp,  '2021-12-30 01:00:00':timestamp)
>120
-- Current Support
SELECT unix_timestamp('2021-12-30 01:00:00':timestamp) - unix_timestamp('2021-12-30 01:00:00':timestamp)
> 120
{code}
Bug: Add the TIMESTAMPADD() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Feb/22 16:24
Affects Version/s: 3.3.0

The function TIMESTAMPADD() is a part of the ODBC api and implemented virtually by ALL other databases are missing in Spark SQL.

The first argument is an unary interval (HOUR, YEAR, DAY, etc)

{code:sql}
TIMESTAMPADD( SECOND, 2 , timestamp '2021-12-12 12:00:00.00') returns 2021-12-12 12:00:02.00
{code}


The ODBC syntax requires the interval value to have a prefix of sql_tsi_ while plain sql doesn't. See
* [Time, Date, and Interval Functions|https://docs.microsoft.com/en-us/sql/odbc/reference/appendixes/time-date-and-interval-functions?view=sql-server-ver15]
* [TIMESTAMPADD function (ODBC compatible)|https://docs.faircom.com/doc/sqlref/33476.htm]
Bug: DPP cause DataSourceScanExec java.lang.NullPointerException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/22 22:34
Affects Version/s: 3.1.2

In DPP,we trigger NPE,like blow:

Caused by: java.lang.NullPointerException
    at org.apache.spark.sql.execution.DataSourceScanExec.$init$(DataSourceScanExec.scala:57)
    at org.apache.spark.sql.execution.FileSourceScanExec.<init>(DataSourceScanExec.scala:172)

...

    at org.apache.spark.sql.catalyst.expressions.CodeGeneratorWithInterpretedFallback.createObject(CodeGeneratorWithInterpretedFallback.scala:56)
    at org.apache.spark.sql.catalyst.expressions.Predicate$.create(predicates.scala:101)
    at org.apache.spark.sql.execution.FilterExec.$anonfun$doExecute$2(basicPhysicalOperators.scala:246)
    at org.apache.spark.sql.execution.FilterExec.$anonfun$doExecute$2$adapted(basicPhysicalOperators.scala:245)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndexInternal$2(RDD.scala:885)

,the root cause is addExprTree funtion in EquivalentExpressions:

```

def addExprTree(
expr: Expression,
addFunc: Expression => Boolean = addExpr): Unit = {
val skip = expr.isInstanceOf[LeafExpression] ||
// `LambdaVariable` is usually used as a loop variable, which can't be evaluated ahead of the
// loop. So we can't evaluate sub-expressions containing `LambdaVariable` at the beginning.
expr.find(_.isInstanceOf[LambdaVariable]).isDefined ||
// `PlanExpression` wraps query plan. To compare query plans of `PlanExpression` on executor,
// can cause error like NPE.
(expr.isInstanceOf[PlanExpression[_]] && TaskContext.get != null)

if (!skip && !addFunc(expr)) {
childrenToRecurse(expr).foreach(addExprTree(_, addFunc))
commonChildrenToRecurse(expr).filter(_.nonEmpty).foreach(addCommonExprs(_, addFunc))

```

maybe we should change it like this :
```

(expr.find(_.isInstanceOf[PlanExpression[_]]).isDefined && TaskContext.get != null)

```

because, in DPP,the filter expression like this:

DynamicPruningExpression(InSubqueryExec(value, broadcastValues, exprId)

so, we should iterator children, if PlanExpression found, such as  InSubqueryExec, we should skip addExprTree, then NPE will not appears
Bug: Invalid URL in SparkContext.addedJars will constantly fails Executor.run()
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 14/Feb/22 08:41
Affects Version/s: 3.2.1

When an invalid URL is used in SparkContext.addJar(), all subsequent query executions will fail since downloading the jar is in the critical path of Executor.run(), even when the query has noting to do with the jar. 

A simple reproduce of the issue:

{code:java}
sc.addJar("http://invalid/library.jar")
(0 to 1).toDF.count
{code}
Bug: Use error classes in the parsing errors of functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 16:01
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* functionNameUnsupportedError
* showFunctionsUnsupportedError
* showFunctionsInvalidPatternError
* createFuncWithBothIfNotExistsAndReplaceError
* defineTempFuncWithIfNotExistsError
* unsupportedFunctionNameError
* specifyingDBInCreateTempFuncError
* invalidNameForDropTempFunc

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Use error classes in the parsing errors of transform
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Feb/22 15:50
Affects Version/s: 3.3.0

Migrate the following errors in QueryParsingErrors:
* transformNotSupportQuantifierError
* transformWithSerdeUnsupportedError
* tooManyArgumentsForTransformError
* notEnoughArgumentsForTransformError
* invalidTransformArgumentError

onto use error classes. Throw an implementation of SparkThrowable. Also write a test per every error in QueryParsingErrorsSuite.
Bug: Introduce `spark.kubernetes.job`  sheduling related configurations
Status: Closed | Priority: Major | Resolution: Invalid
Created: 08/Feb/22 06:53
Affects Version/s: 3.3.0

spark.kubernetes.job.minCPU:  the minimum cpu resources for running job
spark.kubernetes.job.minMemory:  the minimum memory resources for running job
spark.kubernetes.job.minMember: the minimum number of pods for running job
spark.kubernetes.job.priorityClassName: the priority of the running job
spark.kubernetes.job.queue: the queue to which the running job belongs