av9ash/GitBugs huggingface dataset:
Bug: Adjust exponentiation between Series with missing values and bool literal to follow pandas
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 23:46
Affects Version/s: 3.2.0

Currently, exponentiation between ExtentionDtypes and bools is not consistent with pandas' behavior.

 
{code:java}
 >>> pser = pd.Series([1, 2, np.nan], dtype=float)
 >>> psser = ps.from_pandas(pser)
 >>> pser ** False
 0 1.0
 1 1.0
 2 1.0
 dtype: float64
 >>> psser ** False
 0 1.0
 1 1.0
 2 NaN
 dtype: float64
{code}
We ought to adjust that.
Bug: Define the new exception that mix SparkThrowable for all base exe in QueryExecutionErrors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 03:33
Affects Version/s: 3.2.0

The Throwable should extend [SparkThrowable|https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/SparkThrowable.java]; see [SparkArithmeticException|https://github.com/apache/spark/blob/f90eb6a5db0778fd18b0b544f93eac3103bbf03b/core/src/main/scala/org/apache/spark/SparkException.scala#L75] as an example of how to mix SparkThrowable into a base Exception type.
Bug: Flaky Test: TPC-DS in PR builder
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 20/Jul/21 02:38
Affects Version/s: 3.0.3

{code}
[info] - q1 (9 seconds, 603 milliseconds)
[info] - q2 (5 seconds, 860 milliseconds)
[info] - q3 (1 second, 777 milliseconds)
[info] - q4 (31 seconds, 951 milliseconds)
[info] - q5 (4 seconds, 561 milliseconds)
[info] - q7 (2 seconds, 471 milliseconds)
[info] - q8 (2 seconds, 74 milliseconds)
[info] - q9 (4 seconds, 402 milliseconds)
[info] - q10 (4 seconds, 618 milliseconds)
/home/runner/work/spark/spark/build/sbt-launch-lib.bash: line 77:  1659 Killed                  "$@"
Error: Process completed with exit code 137.
{code}

It dies in the middle: https://github.com/apache/spark/runs/3109502701
Bug: Set the lowerbound of mypy version to 0.910
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/21 02:03
Affects Version/s: 3.2.0

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141519/console

{code}
python/pyspark/mllib/tree.pyi:29: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/tree.pyi:38: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:34: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:42: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:48: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:54: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:76: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:124: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/feature.pyi:165: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:45: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/clustering.pyi:72: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:39: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
python/pyspark/mllib/classification.pyi:52: error: Overloaded function signatures 1 and 2 overlap with incompatible return types
Found 13 errors in 4 files (checked 314 source files)
1
{code}

Jenkins installed mypy at https://issues.apache.org/jira/browse/SPARK-32797 but seems the version installed is not same as GIthub Actions
Bug: [SQL] - log level should be warning if files not found in BasicWriteStatsTracker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/21 04:08
Affects Version/s: 3.1.2

This log should at least be WARN not INFO (in [org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala|https://github.com/apache/spark/pull/33332#diff-2a68131bcf611ecf51956d2ba63e8a9ee6d1d20a7a54eeeb4195a4ba1d365134] )

 

"Expected $numSubmittedFiles files, but only saw $numFiles."

 

in my case (using s3) just yesterday, the file was never created and job didn't fail! so I ended up with silently missing data
Bug: Remove some APIs from documentation.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 07:12
Affects Version/s: 3.2.0

We want to remove some APIs from the documentation such as `(Series|Index).spark.data_type`, `(Series|Index).spark.nullable`, `DataFrame.spark.schema`, `DataFrame.spark.print_schema`, `DataFrame.pandas_on_spark.attach_id_column` and `DataFrame.spark.explain` because we might want to hide them from being API in the future.
Bug: pyspark catalog.tableExists doesn't work for temporary views
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jul/21 09:50
Affects Version/s: 3.1.2

Documentation in Catalog.scala for tableExists specifies

   * Check if the table or view with the specified name exists. This can either be a temporary
   * view or a table/view.

The pyspark version doesn't work correctly for temporary views
Bug: Refactor first set of 20 query execution errors to use error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 23:22
Affects Version/s: 3.2.0

Refactor some exceptions in [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala] to use error classes.

There are currently ~350 exceptions in this file; so this PR only focuses on the second set of 20.
{code:java}
columnChangeUnsupportedError
logicalHintOperatorNotRemovedDuringAnalysisError
cannotEvaluateExpressionError
cannotGenerateCodeForExpressionError
cannotTerminateGeneratorError
castingCauseOverflowError
cannotChangeDecimalPrecisionError
invalidInputSyntaxForNumericError
cannotCastFromNullTypeError
cannotCastError
cannotParseDecimalError
simpleStringWithNodeIdUnsupportedError
evaluateUnevaluableAggregateUnsupportedError
dataTypeUnsupportedError
dataTypeUnsupportedError
failedExecuteUserDefinedFunctionError
divideByZeroError
invalidArrayIndexError
mapKeyNotExistError
rowFromCSVParserNotExpectedError
{code}
For more detail, see the parent ticket SPARK-36094.
Bug: Increase timeout for StreamingLinearRegressionWithTests.test_parameter_convergence
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 02:16
Affects Version/s: 3.0.3

Test is flaky (https://github.com/apache/spark/runs/3109815586):

{code}
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 391, in test_parameter_convergence
    eventually(condition, catch_assertions=True)
  File "/__w/spark/spark/python/pyspark/testing/utils.py", line 91, in eventually
    raise lastValue
  File "/__w/spark/spark/python/pyspark/testing/utils.py", line 82, in eventually
    lastValue = condition()
  File "/__w/spark/spark/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 387, in condition
    self.assertEqual(len(model_weights), len(batches))
AssertionError: 9 != 10
{code}

Should probably increase timeout
Bug: HadoopRDD#getPartitions fetches FileSystem Delegation Token for every partition
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Jul/21 10:11
Affects Version/s: 3.1.2

Spark Job creates a separate JobConf for every RDD (every hive table partition) in HadoopRDD#getPartitions.

{code}
  override def getPartitions: Array[Partition] = {
    val jobConf = getJobConf()
    // add the credentials here as this can be called before SparkContext initialized
    SparkHadoopUtil.get.addCredentials(jobConf)
{code}

Hadoop FileSystem fetches FileSystem Delegation Token and sets into the Credentials which is part of JobConf. On further requests, will reuse the token from the Credentials if already exists.

{code}
       if (serviceName != null) { // fs has token, grab it
      final Text service = new Text(serviceName);
      Token<?> token = credentials.getToken(service);
      if (token == null) {
        token = getDelegationToken(renewer);
        if (token != null) {
          tokens.add(token);
          credentials.addToken(service, token);
        }
      }
    }
{code}

 But since Spark Job creates a new JobConf (which will have a new Credentials) for every hive table partition, the token is not reused and gets fetched for every partition. This is slowing down the query as each delegation token has to go through KDC and SSL handshake on Secure Clusters.

*Improvement:*

Spark can add the credentials from previous JobConf into the new JobConf to reuse the FileSystem Delegation Token similar to how the User Credentials are added into JobConf after construction.

{code}
     val jobConf = getJobConf()
    // add the credentials here as this can be called before SparkContext initialized
    SparkHadoopUtil.get.addCredentials(jobConf)
{code}



*Repro*

{code}
beeline>
create table parttable (key char(1), value int) partitioned by (p int);
insert into table parttable partition(p=100) values ('d', 1), ('e', 2), ('f', 3);
insert into table parttable partition(p=200) values ('d', 1), ('e', 2), ('f', 3);
insert into table parttable partition(p=300) values ('d', 1), ('e', 2), ('f', 3);

spark-sql>
select value, count(*) from parttable group by value
{code}
Bug: Support subqueries with correlated non-equality predicates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jul/21 04:34
Affects Version/s: 3.2.0

The new decorrelation framework is able to support subqueries with non-equality predicates. For example:

SELECT * FROM t1 WHERE c1 = (SELECT SUM(c1) FROM t2 WHERE t1.c2 > t2.c2)

The restrictions in CheckAnlysis can be removed.
Bug: Push down join condition evaluation
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Jul/21 15:06
Affects Version/s: 3.3.0

{code:scala}
    val numRows = 1024 * 1024 * 15
    spark.sql(s"CREATE TABLE t1 using parquet AS select id as a, id as b from range(${numRows}L)")
    spark.sql(s"CREATE TABLE t2 using parquet AS select id as a, id as b from range(${numRows}L)")
    val benchmark = new Benchmark("Benchmark push down join condition evaluation", numRows, minNumIters = 5)

    Seq(false, true).foreach { pushDownEnabled =>
      val name = s"Join Condition Evaluation ${if (pushDownEnabled) s"(Pushdown)" else ""}"
      benchmark.addCase(name) { _ =>
        withSQLConf("spark.sql.pushDownJoinConditionEvaluationevaluation" -> s"$pushDownEnabled") {
          spark.sql("SELECT t1.* FROM t1 JOIN t2 ON translate(t1.a, '123', 'abc') = translate(t2.a, '123', 'abc')").write.format("noop").mode("Overwrite").save()
        }
      }
    }
    benchmark.run()
{code}


{noformat}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark push down join condition evaluation:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-----------------------------------------------------------------------------------------------------------------------------
Join Condition Evaluation                              32459          34521        1465          0.5        2063.7       1.0X
Join Condition Evaluation (Pushdown)                   19483          20350         812          0.8        1238.7       1.7X
{noformat}
Bug: Add log files rolling policy for driver running in cluster mode with spark standalone cluster
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Jul/21 04:06
Affects Version/s: 3.1.2

For a long running driver in cluster mode, there is no rolling policy, the log (stdout/stderr) may accupy lots of space, user needs an external tool to clean the old logs, it's not user friendly. 

For executor, following 5 configurations is used to control the log file rolling policy:
{code:java}
spark.executor.logs.rolling.maxRetainedFiles
spark.executor.logs.rolling.enableCompression
spark.executor.logs.rolling.maxSize
spark.executor.logs.rolling.strategy
spark.executor.logs.rolling.time.interval
{code}

For driver running in cluster mode:
1. reuse the executor settings
2. similar to executor: add following configurations (only works for stderr/stdout for driver in cluster mode)
{code:java}
spark.driver.logs.rolling.maxRetainedFiles
spark.driver.logs.rolling.enableCompression
spark.driver.logs.rolling.maxSize
spark.driver.logs.rolling.strategy
spark.driver.logs.rolling.time.interval
{code}

#2 seems better, do you agree?
Bug: Group exception messages in core/storage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jul/21 18:16
Affects Version/s: 3.3.0

'core/src/main/scala/org/apache/spark/storage'
|| Filename                          ||   Count ||
| BlockId.scala                     |       1 |
| BlockInfoManager.scala            |       2 |
| BlockManager.scala                |       9 |
| BlockManagerDecommissioner.scala  |       1 |
| BlockManagerMaster.scala          |       2 |
| DiskBlockManager.scala            |       1 |
| DiskBlockObjectWriter.scala       |       1 |
| ShuffleBlockFetcherIterator.scala |       4 |
Bug: InputFormat of PartitionDesc is not respected
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jul/21 14:29
Affects Version/s: 3.1.2

A hive partition can have different PartitionDesc from TableDesc for describing Serde/InputFormatClass/OutputFormatClass, for a hive partitioned table, we shall respect this information in PartitionDesc first.
Bug: Group SQL component error messages in Spark error class JSON file
Status: Open | Priority: Major | Resolution: 
Created: 12/Jul/21 17:32
Affects Version/s: 3.2.0

To improve auditing, reduce duplication, and improve quality of error messages thrown from Spark, we should group them in a single JSON file (as discussed in the [mailing list|http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Add-error-IDs-td31126.html] and introduced in [SPARK-34920|#diff-d41e24da75af19647fadd76ad0b63ecb22b08c0004b07091e4603a30ec0fe013]).
 In this file, the error messages should be labeled according to a consistent error class and with a SQLSTATE.

We will start with the SQL component first.
 As a starting point, we can build off the exception grouping done in SPARK-33539. In total, there are ~1000 error messages to group split across three files (QueryCompilationErrors, QueryExecutionErrors, and QueryParsingErrors). In this ticket, each of these files is split into chunks of ~20 errors for refactoring.

Here is an example PR that groups a few error messages in the QueryCompilationErrors class: [PR 33309|https://github.com/apache/spark/pull/33309].

[Guidelines|https://github.com/apache/spark/blob/master/core/src/main/resources/error/README.md]:
 - Error classes should be unique and sorted in alphabetical order.
 - Error classes should be unified as much as possible to improve auditing. If error messages are similar, group them into a single error class and add parameters to the error message.
 - SQLSTATE should match the ANSI/ISO standard, without introducing new classes or subclasses. See the error [guidelines|https://github.com/apache/spark/blob/master/core/src/main/resources/error/README.md]; if none of them match, the SQLSTATE field should be empty.
 - The Throwable should extend [SparkThrowable|https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/SparkThrowable.java]; see [SparkArithmeticException|https://github.com/apache/spark/blob/f90eb6a5db0778fd18b0b544f93eac3103bbf03b/core/src/main/scala/org/apache/spark/SparkException.scala#L75] as an example of how to mix SparkThrowable into a base Exception type.

We will improve error message quality as a follow-up.
Bug: Support backing off dynamic allocation increases if resources are "stuck"
Status: Closed | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 17:51
Affects Version/s: 3.2.0

In a over-subscribed environment we may enter a situation where our requests for more pods are not going to be fulfilled. Adding more requests for more pods is not going to help and may slow down the scheduler. We should detect this situation and hold off on increasing pod requests until the scheduler allocates more pods to us. We have a limited version of this in the Kube scheduler it's self but it would be better to plumb this all the way through to the DA logic.
Bug: Support raw string literal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Jul/21 16:18
Affects Version/s: 3.3.0

In the current master, sometimes it's too confusable to represent JSON and regex in a string literal if they contain backslash.
For example, in JSON, \ needs to be escaped like as follows.
{code}
{"a": "\\"}
{code}
But, if the JSON above is represented in a string literal, further two \ are needed because string literal also requires \ to be escaped.
{code}
SELECT from_json('{"a": "\\\\"}', 'a string')
{"a":"\"}
{code}
To make such case simpler, it's great if Spark supports raw string literal.
Bug: Introduce pending pod limit for Spark on K8s
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 11:40
Affects Version/s: 3.0.3

Introduce a new configuration to limit the number of pending PODs for Spark on K8S as the K8S scheduler could be overloaded with requests which slows down the resource allocations (especially in case of dynamic allocation).
Bug: Cannot run leveldb related UTs on Mac OS of M1 architecture
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 05/Jul/21 11:59
Affects Version/s: 3.3.0

When run leveldb related UTs on Mac OS of M1 architecture, there are some test failed as follows:
{code:java}
[INFO] Running org.apache.spark.util.kvstore.LevelDBSuite
[ERROR] Tests run: 10, Failures: 0, Errors: 10, Skipped: 0, Time elapsed: 0.18 s <<< FAILURE! - in org.apache.spark.util.kvstore.LevelDBSuite
[ERROR] org.apache.spark.util.kvstore.LevelDBSuite.testMultipleTypesWriteReadDelete  Time elapsed: 0.146 s  <<< ERROR!
java.lang.UnsatisfiedLinkError: 
Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /Users/yangjie01/SourceCode/git/spark-mine-12/common/kvstore/target/tmp/libleveldbjni-64-1-7259526109351494242.8: dlopen(/Users/yangjie01/SourceCode/git/spark-mine-12/common/kvstore/target/tmp/libleveldbjni-64-1-7259526109351494242.8, 1): no suitable image found.  Did find:
	/Users/yangjie01/SourceCode/git/spark-mine-12/common/kvstore/target/tmp/libleveldbjni-64-1-7259526109351494242.8: no matching architecture in universal wrapper
	/Users/yangjie01/SourceCode/git/spark-mine-12/common/kvstore/target/tmp/libleveldbjni-64-1-7259526109351494242.8: no matching architecture in universal wrapper]
	at org.apache.spark.util.kvstore.LevelDBSuite.setup(LevelDBSuite.java:55)


[ERROR] org.apache.spark.util.kvstore.LevelDBSuite.testObjectWriteReadDelete  Time elapsed: 0 s  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class org.fusesource.leveldbjni.JniDBFactory
	at org.apache.spark.util.kvstore.LevelDBSuite.setup(LevelDBSuite.java:55)

....

[ERROR] Tests run: 105, Failures: 0, Errors: 48, Skipped: 0{code}
There seems to be a lack of JNI support
Bug: Basic speculation metrics at stage level
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/21 04:34
Affects Version/s: 3.1.2

Currently there are no speculation metrics available either at application level or at stage level. With in our platform, we have added speculation metrics at stage level as a summary similarly to the stage level metrics tracking numTotalSpeculated, numCompleted (successful), numFailed, numKilled etc. This enables us to effectively understand speculative execution feature at an application level and helps in further tuning the speculation configs.

cc [~ron8hu]
Bug: HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/21 20:48
Affects Version/s: 3.2.0

At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use {{getAllPartitionsOf}} when {{hive.metastore.try.direct.sql}} is enabled in the remote HMS. However, in certain cases the remote HMS will fallback to use ORM (which only support string type for partition columns) to query the underlying RDBMS even if this config is set to true, and Spark will not be able to recover from the error and will just fail the query. 

For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and Spark was not able to pushdown filter for {{date}} column.
Bug: Support TimestampNTZ type in Orc file source
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/21 11:09
Affects Version/s: 3.3.0

As per https://orc.apache.org/docs/types.html, Orc supports both TIMESTAMP_NTZ and TIMESTAMP_LTZ (Spark's current default timestamp type):

* A TIMESTAMP => TIMESTAMP_LTZ
* Timestamp with local time zone => TIMESTAMP_NTZ
In Spark 3.1 or prior, Spark only considered TIMESTAMP.
Since 3.2, with the support of timestamp without time zone type:
* Orc writer follows the definition and uses "Timestamp with local time zone" on writing TIMESTAMP_NTZ.
* Orc reader converts the "Timestamp with local time zone" to TIMESTAMP_NTZ.
Bug: Support creation and operations of ps.Series/Index with Decimal('NaN')
Status: Resolved | Priority: Major | Resolution: Done
Created: 02/Jul/21 22:25
Affects Version/s: 3.2.0

The creation and operations of ps.Series/Index with Decimal('NaN') doesn't work as expected.

That might be due to the underlying PySpark limit.

Please refer to sub-tasks for issues detected.
Bug: TPCDSQueryTestSuite should run with different config set
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/21 07:41
Affects Version/s: 3.3.0

In current github actions we run TPCDSQueryTestSuite for tpcds benchmark. But it's only tested under default configurations. Since we have added the `spark.sql.join.forceApplyShuffledHashJoin` config. Now we can test all 3 join strategies in TPCDS to improve the coverage.
Bug: EquivalentExpressions fixes and improvements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/21 11:55
Affects Version/s: 3.2.0

Currently `EquivalentExpressions` has 2 issues:
- identifying common expressions in conditional expressions is not correct in all cases
- transparently canonicalized expressions (like `PromotePrecision`) are considered common subexpressions