av9ash/GitBugs huggingface dataset:
Bug: SQL parser should use active SQLConf
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/20 11:43
Affects Version/s: 3.0.1

In ANSI mode, schema string parsing should fail if the schema uses ANSI reserved keyword as attribute name:
{code:scala}
spark.conf.set("spark.sql.ansi.enabled", "true")
spark.sql("""select from_json('{"time":"26/10/2015"}', 'time Timestamp', map('timestampFormat', 'dd/MM/yyyy'));""").show


output:

Cannot parse the data type: 
no viable alternative at input 'time'(line 1, pos 0)

== SQL ==
time Timestamp
^^^
{code}

But this query may accidentally succeed in certain cases cause the DataType parser sticks to the configs of the first created session in the current thread:

{code:scala}
DataType.fromDDL("time Timestamp")
val newSpark = spark.newSession()
newSpark.conf.set("spark.sql.ansi.enabled", "true")
newSpark.sql("""select from_json('{"time":"26/10/2015"}', 'time Timestamp', map('timestampFormat', 'dd/MM/yyyy'));""").show


output:

+--------------------------------+
|from_json({"time":"26/10/2015"})|
+--------------------------------+
|            {2015-10-26 00:00...|
+--------------------------------+
{code}
Bug: Add support for V1Table in stream writer table API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/20 08:44
Affects Version/s: 3.0.0

After SPARK-32896, we have table API for stream writer but only support DataSource v2 tables. Here we add the following enhancements:
 * Create non-existing tables by default
 * Support both managed and external V1Tables
Bug: Support namespaces in JDBC v2 Table Catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Nov/20 10:31
Affects Version/s: 3.1.0

When I extend JDBCTableCatalogSuite by org.apache.spark.sql.execution.command.v2.ShowTablesSuite, for instance:
{code:scala}
import org.apache.spark.sql.execution.command.v2.ShowTablesSuite

class JDBCTableCatalogSuite extends ShowTablesSuite {
  override def version: String = "JDBC V2"
  override def catalog: String = "h2"
...
{code}
some tests from JDBCTableCatalogSuite fail with:
{code}
[info] - SHOW TABLES JDBC V2: show an existing table *** FAILED *** (2 seconds, 502 milliseconds)
[info]   org.apache.spark.sql.AnalysisException: Cannot use catalog h2: does not support namespaces;
[info]   at org.apache.spark.sql.connector.catalog.CatalogV2Implicits$CatalogHelper.asNamespaceCatalog(CatalogV2Implicits.scala:83)
[info]   at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs$$anonfun$apply$1.applyOrElse(ResolveCatalogs.scala:208)
[info]   at org.apache.spark.sql.catalyst.analysis.ResolveCatalogs$$anonfun$apply$1.applyOrElse(ResolveCatalogs.scala:34)
{code}
Bug: make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/evaluator
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Nov/20 16:01
Affects Version/s: 3.1.0

Currently, pyspark support third-party library to define python backend estimator/evaluator, i.e., estimator that inherit `Estimator` instead of `JavaEstimator`, and only can be used in pyspark.

CrossValidator and TrainValidateSplit support tuning these python backend estimator,
but cannot support saving/load, becase CrossValidator and TrainValidateSplit writer implementation is use JavaMLWriter, which require to convert nested estimator and evaluator into java instance.

OneVsRest saving/load now only support java backend classifier due to similar issue.
Bug: Pyspark ML Validator params in estimatorParamMaps may be lost after saving and reloading
Status: In Progress | Priority: Major | Resolution: 
Created: 30/Nov/20 01:09
Affects Version/s: 3.0.0

Two typical cases to reproduce it:
(1)
{code:python}
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression()
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

paramGrid = ParamGridBuilder() \
    .addGrid(hashingTF.numFeatures, [10, 100]) \
    .addGrid(lr.maxIter, [100, 200]) \
    .build()
tvs = TrainValidationSplit(estimator=pipeline,
                           estimatorParamMaps=paramGrid,
                           evaluator=MulticlassClassificationEvaluator())

tvs.save(tvsPath)
loadedTvs = TrainValidationSplit.load(tvsPath)

{code}

Then we can check `loadedTvs.getEstimatorParamMaps()`, the tuning params `hashingTF.numFeatures` and `lr.maxIter` are lost.


(2)
{code:python}
lr = LogisticRegression()
ova = OneVsRest(classifier=lr)
grid = ParamGridBuilder().addGrid(lr.maxIter, [100, 200]).build()
evaluator = MulticlassClassificationEvaluator()
tvs = TrainValidationSplit(estimator=ova, estimatorParamMaps=grid, evaluator=evaluator)

tvs.save(tvsPath)
loadedTvs = TrainValidationSplit.load(tvsPath)

{code}
Then we can check `loadedTvs.getEstimatorParamMaps()`, the tuning params`lr.maxIter` are lost.


Both CrossValidator and TrainValidationSplit in Pyspark has this issue.
Bug: IllegalArgumentException when applying RemoveRedundantSorts before EnsureRequirements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Nov/20 01:00
Affects Version/s: 2.4.8

`RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception:

{{IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.}}

We should switch the order between these two rules to satisfy the requirement when instantiating `PartitioningCollection`.
Bug: Decode Query parameters of the redirect URL for reverse proxy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 14:57
Affects Version/s: 3.0.0

When running Spark with reverse proxy enabled, the query parameter of the request URL can be encoded twice:  one from the browser and another one from the reverse proxy(e.g. Nginx).  

In Spark's stage page, the URL of "/taskTable" contains query parameter order[0][dir].  After encoding twice,  the query parameter becomes `order%255B0%255D%255Bdir%255D` and it will be decoded as `order%5B0%5D%5Bdir%5D` instead of  `order[0][dir]`.  As a result, there will be NullPointerException from https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala#L176

Other than that, the other parameter may not work as expected after encoded twice.

We should decode the query parameters and fix the problem
Bug: Handle DELETE/UPDATE/MERGE in PullupCorrelatedPredicates rule
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Nov/20 12:44
Affects Version/s: 3.1.0

{{PullupCorrelatedPredicates}} should handle DELETE/UPDATE/MERGE commands. Right now, the rule works with filters and unary nodes only. As a result, correlated predicates in DELETE/UPDATE/MERGE are not rewritten.
Bug: Refactor SortOrder class to allow multiple childrens
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 13:10
Affects Version/s: 3.0.1

Currently the SortOrder is a UnaryExpression with only one child. It contains a field "sameOrderExpression" that needs some special handling as done in [https://github.com/apache/spark/pull/30302] .

 

One of the suggestion in [https://github.com/apache/spark/pull/30302#discussion_r526104333] is to make sameOrderExpression as children of SortOrder.
Bug: Datetime building should fail if the year, month, ..., second combination is invalid
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 12:10
Affects Version/s: 3.1.0

Datetime building should fail if the year, month, ..., second combination is invalid, when ANSI mode is enabled. This patch should update MakeDate, MakeTimestamp and MakeInterval.
Bug: NPE when there is no watermark metrics
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/Nov/20 07:06
Affects Version/s: 3.1.0

We parse the process timestamp at https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala#L153, but will throw NPE when there is no event time metrics.
Bug: Spark schedules on updating delegation token with 0 interval under some token provider implementation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Nov/20 04:15
Affects Version/s: 3.0.1

We got a report from customer that under specific circumstance Spark schedules on updating delegation token with 0 interval, ended up with flooding log message & massive requests on token handler side.

After investigation, the problem was they have two delegation token identifiers which one of token identifier (IDBS3ATokenIdentifier) has the value of "issue date" to be 0, whereas another token identifier (DelegationTokenIdentifier) has correct value. 

Both are providing the expire time correctly via Token.renew(), and Spark assumes issue date is "correct", hence calculating the token expire period as (the result of Token.renew() - "issue date").

{code}
20/10/13 06:34:19 INFO security.HadoopFSDelegationTokenProvider: Renewal interval is 1603175657000 for token S3ADelegationToken/IDBroker
20/10/13 06:34:19 INFO security.HadoopFSDelegationTokenProvider: Renewal interval is 86400048 for token HDFS_DELEGATION_TOKEN
{code}

It's safe at least here because Spark picks "minimal" value. The thing is, to calculate the next renewal timestamp, Spark tries to add the renewal interval with issue date for every token, and pick minimum value, hence "86400048" is picked as the next renewal timestamp.

This is "earlier" than now, hence interval to schedule goes to be negative (as we apply subtract with now), and Spark applies safeguard to pick the greater between 0 and interval, hence 0 is picked up, and schedule updating token infinitely. (Schedule is one-time, but the calculation will always lead to the negative, so that's effectively immediate schedule.)

We should construct the better consideration of "safe guard", instead of just guarding the schedule interval doesn't go to negative.
Bug: Partition spec in SHOW TABLE EXTENDED doesn't respect `spark.sql.caseSensitive`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/20 20:27
Affects Version/s: 2.4.7

For example:
{code:sql}
spark-sql> CREATE TABLE tbl1 (price int, qty int, year int, month int)
         > USING parquet
         > partitioned by (year, month);
spark-sql> INSERT INTO tbl1 PARTITION(year = 2015, month = 1) SELECT 1, 1;
spark-sql> SHOW TABLE EXTENDED LIKE 'tbl1' PARTITION(YEAR = 2015, Month = 1);
Error in query: Partition spec is invalid. The spec (YEAR, Month) must match the partition spec (year, month) defined in table '`default`.`tbl1`';
{code}
The spark.sql.caseSensitive flag is false by default, so, the partition spec is valid.
Bug: Create a V2 SHOW PARTITIONS execution node
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Nov/20 17:51
Affects Version/s: 3.1.0

There is the V1 SHOW PARTITIONS implementation:
https://github.com/apache/spark/blob/7e99fcd64efa425f3c985df4fe957a3be274a49a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L975
The ticket aims to add V2 implementation with similar behavior.
Bug: Delay in batch processing when creating Spark metadata compact file
Status: Open | Priority: Major | Resolution: 
Created: 30/Nov/20 12:04
Affects Version/s: 2.4.3

Hi,

We are using Spark structured streaming to consume data from a Kafka cluster, transform it, and write it in an S3 Objectstore Service. Currently our Spark metadata compact files have 4,046,119 lines (1.2GB), and when a compact file gets written there is a delay with the batch processing - the batches are processing 1.8-2 minutes, when it usually takes 10-15 seconds. This delay in the processing generates lag in Kafka, which is a significant problem for us. Is this a known issue and if it is, are there any plans for fixing it in the future?
Bug: Executors blank page behind proxy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Nov/20 16:46
Affects Version/s: 3.0.1

When accessing the Web UI behind a proxy (e.g. a Kubernetes ingress), executors page is blank.

In {{/core/src/main/resources/org/apache/spark/ui/static/utils.js}}, we  should avoid the use of location.origin when constructing URLs for internal API calls within the JavaScript.
 Instead, we should use {{apiRoot}} global variable.

On one hand, it would allow to build relative URLs. On the other hand, {{apiRoot}} reflects the Spark property {{spark.ui.proxyBase}} which can be set to change the root path of the Web UI.

If {{spark.ui.proxyBase}} is actually set, original URLs become incorrect, and we end up with an executors blank page.
 I encounter this bug when accessing the Web UI behind a proxy (in my case a Kubernetes Ingress).

 

See also [https://github.com/jupyterhub/jupyter-server-proxy/issues/57#issuecomment-699163115]
Bug: DSv2: Use callback instead of passing Spark session and v2 relation for refreshing cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 22:27
Affects Version/s: 3.1.0

As discussed [https://github.com/apache/spark/pull/30429], it's better to not pass Spark session and DataSourceV2Relation through Spark plans. Instead we can use a callback which makes the interface cleaner.
Bug: Kill the executor on nested fatal errors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/20 18:18
Affects Version/s: 3.0.1

Currently we kill the executor when hitting a fatal error. However, if the fatal error is wrapped by another exception, such as
- java.util.concurrent.ExecutionException, com.google.common.util.concurrent.UncheckedExecutionException, com.google.common.util.concurrent.ExecutionError when using Guava cache and java thread pool.
- SparkException thrown from this line: https://github.com/apache/spark/blob/cf98a761de677c733f3c33230e1c63ddb785d5c5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala#L231

We will still keep the executor running. Fatal errors are usually unrecoverable (such as OutOfMemoryError), some components may be in a broken state when hitting a fatal error. Hence, it's better to detect the nested fatal error as well and kill the executor. Then we can rely on Spark's fault tolerance to recover.
Bug: [SQL] Reduce shuffle task number when calling CollectLimitExec#executeToIterator
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Nov/20 12:01
Affects Version/s: 2.4.0

Currently, when invoking CollectLimitExec#executeToIterator, a single-partition ShuffledRowRDD containg all parent partitions is created. Spark will compute all these partitions to get the result.
But in most cases, computing the first few partitions is enought to get the result, which takes much less time.

When running a SparkThriftServer and spark.sql.thriftServer.incrementalCollect is enabled, too many shuffle tasks will lead to a significant performance issue for SQLs terminated with LIMIT.

A possible improvement may be as follows:
 # Create a ShuffledRowRDD containing the first parent partition.
 # Collect rows of this ShuffledRowRDD to driver
 # If collected rows is less than limit number, then create the next ShuffledRowRDD with serveral following parent partitions. The number of parent partitions is calculated the same way as SparkPlan#executeTake.
 # Repeat 2~3 until total collected rows reaches limit number or all parent partitions have been computed.
Bug: resolveDependencyPaths should use classifier attribute of artifact
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Nov/20 04:27
Affects Version/s: 3.1.0

`resolveDependencyPaths` now takes artifact type to decide to add "-tests" postfix. However, the path pattern of ivy in `resolveMavenCoordinates` is "[organization]_[artifact]-[revision](-[classifier]).[ext]". We should use classifier instead of type to construct file path.
Bug: Set the proper version of gssapi plugin automatically for MariaDBKrbIntegrationSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 09:18
Affects Version/s: 3.1.0

For MariaDBKrbIntegrationSuite, the version of mariadb-plugin-gssapi-server is currently set to 10.5.5 in mariadb_docker_entrypoint.sh but it's no longer available in the official apt repository and MariaDBKrbIntegrationSuite doesn't pass for now.
It seems that only the most recent three versions are available and they are 10.5.6, 10.5.7 and 10.5.8 for now.
Further, the release cycle of MariaDB seems to be very rapid (1 ~ 2 months) so I don't think it's a good idea to set to an specific version for mariadb-plugin-gssapi-server.
Bug: Query on large dataset with foreachPartitionAsync performance needs to improve
Status: Open | Priority: Major | Resolution: 
Created: 28/Nov/20 08:29
Affects Version/s: 2.4.4

Repro steps:

Load 300GB of data from JSON file into a table.

Note in this table field ID with reasonably well sized sets, identified by ID, some 50,000 rows a set.

Issue query against this table returning DataFrame instance.

Issue df.rdd.foreachPartitionAsync styled row harvesting.

Place a logging line into first lambda expression, iterating over partitions as a first line.

Let's say it will read "Line #1 ( some timestamp with milliseconds )"

Place a logging line into nested lambda expression, reading rows, such, that it would run only when accessing first row.

Let's say it will read "Line #2 ( some timestamp with milliseconds )"

Once query completed take time difference in milliseconds between time noted in logging records from line #1 and line #2 above.

It would be fairly reasonable to assume that the time difference should be as close to 0 as possible. In reality the difference is more then 1 second, usually more than 2.

This really hurts query performance.
Bug: install coverage for pypy3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Nov/20 04:38
Affects Version/s: 3.0.0

from:

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-sbt-hadoop-2.7-hive-1.2/1002/console

 
Coverage is not installed in Python executable 'pypy3' but 'COVERAGE_PROCESS_START' environment variable is set, exiting.
Bug: Do not use custom shuffle reader for repartition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 05:04
Affects Version/s: 3.0.1

We should have a more thorough fix for all sorts of custom shuffle readers when the original query has a repartition shuffle, based on the discussions on the initial PR: [https://github.com/apache/spark/pull/29797].
Bug: distinguish table properties and options in V2 catalog API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Nov/20 14:55
Affects Version/s: 3.1.0

Right now, user-specified table properties and options are merged and passed to `TableCatalog.createTable` as table properties, because v2 catalog does not have the table options concept.

We need to distinguish table properties and options if we want to use v2 catalog API to implement hive catalog. We can add an `option.` prefix to table options when merging it into table properties. For backward compatibility, we can also merge the original table options without the `option.` prefix into table properties as well.
Bug: Improve error message of ANSI explicit cast
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Nov/20 03:36
Affects Version/s: 3.1.0

After https://github.com/apache/spark/pull/30260, there are some type conversions disallowed under ANSI mode.
To make it more user-friendly, we should tell users what they can do if they have to use the casting.
Bug: List partition by names from V2 tables that support partition management
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Nov/20 14:53
Affects Version/s: 3.1.0

Currently, the SupportsPartitionManagement interface exposes only the listPartitionIdentifiers() method which does not allow to list partition by names. So, it is hard to implement:
{code:java}
SHOW PARTITIONS table PARTITION(month=2)
{code}
from the table like:
{code:java}
CREATE TABLE $table (price int, qty int, year int, month int)
USING parquet
partitioned by (year, month)
{code}
because listPartitionIdentifiers() requires to specify value for *year* .
Bug: Peak Execution Memory not display on Spark Executor UI intuitively
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Nov/20 02:16
Affects Version/s: 3.0.0

Now, Peak Execution Memory can only be obtained through restAPI and cannot be displayed on Spark Executor UI intuitively, although spark users tune spark executor memory are dependent on the metrics. Therefore, it is very important to display the peak memory usage on the spark UI.
Bug: Recover hive-service-rpc to built-in Hive version when we upgrade built-in Hive to 3.1.2
Status: Open | Priority: Major | Resolution: 
Created: 25/Nov/20 04:59
Affects Version/s: 3.1.0

Recover hive-service-rpc to built-in Hive version when we upgrade built-in Hive to 3.1.2. Please see 
https://github.com/apache/spark/pull/30478#discussion_r529179587 for more details.
Bug: DSv2: Append/Overwrite/ReplaceTable should invalidate cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Nov/20 16:58
Affects Version/s: 3.1.0

Unlike in DSv1, currently in DSv2 we don't invalidate table caches for operations such as append, overwrite table by expr/partition, replace table, etc. We should fix these so that the behavior is consistent between v1 and v2.
Bug: export LANG to en_US.UTF-8 in jenkins test script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Nov/20 14:12
Affects Version/s: 3.1.0

 
{code:java}
 org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V1 get binary type
 org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V2 get binary type
 org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V3 get binary type
 org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V4 get binary type
 org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.HIVE_CLI_SERVICE_PROTOCOL_V5 get binary type{code}
 

failed Jenkins tests  and passed GitHub Actions. The error message as follows:

 

 
{code:java}
Error Messageorg.scalatest.exceptions.TestFailedException: "[?](" did not equal "[�]("Stacktracesbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: "[?](" did not equal "[�]("
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
	at org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.$anonfun$new$26(SparkThriftServerProtocolVersionsSuite.scala:302)
	at org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.$anonfun$new$26$adapted(SparkThriftServerProtocolVersionsSuite.scala:300)
	at org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.testExecuteStatementWithProtocolVersion(SparkThriftServerProtocolVersionsSuite.scala:68)
	at org.apache.spark.sql.hive.thriftserver.SparkThriftServerProtocolVersionsSuite.$anonfun$new$24(SparkThriftServerProtocolVersionsSuite.scala:300)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
{code}
 

seems that the "LANG" of some  build machines is not "en_US.UTF-8"