av9ash/GitBugs huggingface dataset:
Bug: Implement built-in LIKE ANY and LIKE ALL UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Oct/20 12:09
Affects Version/s: 3.1.0

We already support LIKE ANY / SOME / ALL syntax, but it will throw {{StackOverflowError}} if there are many elements(more than 14378 elements). We should implement built-in LIKE ANY and LIKE ALL UDF to fix this issue.

{noformat}
java.lang.StackOverflowError
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:184)
	at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:47)
	at scala.collection.generic.GenericCompanion.apply(GenericCompanion.scala:53)
	at org.apache.spark.sql.catalyst.expressions.BinaryExpression.children(Expression.scala:549)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:175)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:175)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:175)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreach(TreeNode.scala:175)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1(TreeNode.scala:175)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreach$1$adapted(TreeNode.scala:175)
	at scala.collection.immutable.List.foreach(List.scala:392)
{noformat}
Bug: REFRESH TABLE should invalidate cache even though the table itself may not be cached
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/20 19:21
Affects Version/s: 2.4.7

For the following example:
{code}
CREATE TABLE t ...;
CREATE VIEW t1 AS SELECT * FROM t;
REFRESH TABLE t
{code}

If t is cached, t1 will be invalidated. However if t is not cached as above, the REFRESH command won't invalidate view t1. This could lead to incorrect result if the view is used later.
Bug: DSv2: DROP TABLE command should also invalidate cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/20 01:19
Affects Version/s: 3.0.1

Different from DSv1, {{DROP TABLE}} command in DSv2 currently only drops the table but doesn't invalidate all caches referencing the table. We should make the behavior consistent between v1 and v2.
Bug: Bug in optimizer rule EliminateSorts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/20 17:54
Affects Version/s: 2.4.0

Currently, the rule {{EliminateSorts}} removes a global sort node if its child plan already satisfies the required sort order without checking if the child plan's ordering is local or global. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. 
{code:java}
Sort(orders, global = True, ...)
  Sort(orders, global = False, ...){code}
Bug: SortExec produces incorrect results if sortOrder is a Stream
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 11:00
Affects Version/s: 3.0.0

The following query produces incorrect results. The query has two essential features: (1) it contains a string aggregate, resulting in a {{SortExec}} node, and (2) it contains a duplicate grouping key, causing {{RemoveRepetitionFromGroupExpressions}} to produce a sort order stored as a Stream.

SELECT bigint_col_1, bigint_col_9, MAX(CAST(bigint_col_1 AS string))
FROM table_4
GROUP BY bigint_col_1, bigint_col_9, bigint_col_9

When the sort order is stored as a {{Stream}}, the line {{ordering.map(_.child.genCode(ctx))}} in {{GenerateOrdering#createOrderKeys()}} produces unpredictable side effects to {{ctx}}. This is because {{genCode(ctx)}} modifies {{ctx}}. When {{ordering}} is a {{Stream}}, the modifications will not happen immediately as intended, but will instead occur lazily when the returned {{Stream}} is used later.

Similar bugs have occurred at least three times in the past: https://issues.apache.org/jira/browse/SPARK-24500, https://issues.apache.org/jira/browse/SPARK-25767, https://issues.apache.org/jira/browse/SPARK-26680.

The fix is to check if {{ordering}} is a {{Stream}} and force the modifications to happen immediately if so.
Bug: Respect ANSI standard and change FIRST_VALUE as pure window function.
Status: Resolved | Priority: Major | Resolution: Later
Created: 20/Oct/20 03:46
Affects Version/s: 3.1.0

FIRST_VALUE is an ANSI window function.
But the current implement of FIRST_VALUE in Spark is an aggregate function.
Although FIRST_VALUE could be used as window function.
But it is an aggregate function in fact and will lead to users to mistakenly think that FIRST_VALUE is an aggregate function that can be used.
Users can use FIRST now.
Bug: mismatched input since Spark 3.0
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Oct/20 09:33
Affects Version/s: 3.0.0

Spark 2.4:
{noformat}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_221)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("SELECT 1 UNION SELECT 1 UNION ALL SELECT 1").show
+---+
|  1|
+---+
|  1|
|  1|
+---+
{noformat}


Spark 3.x:
{noformat}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.1)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("SELECT 1 UNION SELECT 1 UNION ALL SELECT 1").show
org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'SELECT' expecting {<EOF>, ';'}(line 1, pos 15)

== SQL ==
SELECT 1 UNION SELECT 1 UNION ALL SELECT 1
---------------^^^

  at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:263)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:130)
  at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:51)
  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:81)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:610)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:610)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:769)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:607)
  ... 47 elided
{noformat}
Bug: Add a check on the number of returned partitions in the HiveShim#getPartitionsByFilter method
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Oct/20 03:56
Affects Version/s: 3.0.2

In the method Shim#getPartitionsByFilter, when filter is empty or when the hive table has a large number of partitions, calling getAllPartitionsMethod or getPartitionsByFilterMethod will results in Driver OOM.

I think we need add a check on the number of returned partitions by calling Hive#getNumPartitionsByFilter, and add SQLConf spark.sql.hive.metastorePartitionLimit, default value is 100_000
Bug: Unify the code paths for spark.table and spark.read.table
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 10:42
Affects Version/s: 3.0.0

* The code paths of `spark.table` and `spark.read.table` should be the same. This behavior is broke in SPARK-32592 since we need to respect options in `spark.read.table` API.
 * Add comments for `{{spark.table`}} to emphasize it also support streaming temp view reading
Bug: FileOutputWriter jobs have duplicate JobIDs if launched in same second
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/20 13:39
Affects Version/s: 2.4.7

The Hadoop S3A staging committer has problems with >1 spark sql query being launched simultaneously, as it uses the jobID for its path in the clusterFS to pass the commit information from tasks to job committer. 

If two queries are launched in the same second, they conflict and the output of job 1 includes that of all job2 files written so far; job 2 will fail with FNFE.

Proposed:
job conf to set {{"spark.sql.sources.writeJobUUID"}} to the value of {{WriteJobDescription.uuid}}

That was the property name which used to serve this purpose; any committers already written which use this property will pick it up without needing any changes.
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MsSqlServer dialect)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 11:12
Affects Version/s: 3.1.0

Override the default SQL strings for:
ALTER TABLE RENAME COLUMN
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following MsSQLServer JDBC dialect according to official documentation.
Write MsSqlServer integration tests for JDBC.
Bug: HiveExternalCatalogVersionsSuite shouldn't use or delete hard-coded /tmp directory 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 21:47
Affects Version/s: 3.0.1

In SPARK-22356, the {{sparkTestingDir}} used by {{HiveExternalCatalogVersionsSuite}} became hard-coded to enable re-use of the downloaded Spark tarball between test executions:
{code}
  // For local test, you can set `sparkTestingDir` to a static value like `/tmp/test-spark`, to
  // avoid downloading Spark of different versions in each run.
  private val sparkTestingDir = new File("/tmp/test-spark")
{code}
However this doesn't work, since it gets deleted every time:
{code}
  override def afterAll(): Unit = {
    try {
      Utils.deleteRecursively(wareHousePath)
      Utils.deleteRecursively(tmpDataDir)
      Utils.deleteRecursively(sparkTestingDir)
    } finally {
      super.afterAll()
    }
  }
{code}

It's bad that we're hard-coding to a {{/tmp}} directory, as in some cases this is not the proper place to store temporary files. We're not currently making any good use of it.
Bug: In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/20 08:14
Affects Version/s: 3.0.0

Spark Version: spark3.0.1

problem：In the Storage UI page, clicking any field to sort the table will cause the header content to be lost，secreen gif  is shown in the attachment.
Bug: Unify schema parsing in from_json/from_csv across all APIs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Oct/20 07:54
Affects Version/s: 3.1.0

Currently, from_json() has extra capability in Scala API. It accepts schema in JSON format but other API (SQL, Python, R) lacks the feature. The ticket aims to unify all APIs, and support schemas in JSON format everywhere.
Bug: approxSimilarityJoin in Structured Stream causes state to explode in size
Status: Open | Priority: Major | Resolution: 
Created: 03/Oct/20 13:32
Affects Version/s: 3.0.0

I'm writing a PySpark application that joins a static and streaming dataframe together using the approxSimilarityJoin function from the ML package. Because of the high volume of data, we need to apply a watermark to make sure a minimal amount of state is preserved. However, the [approxSimilarityJoin scala code contains a `distinct` action|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala#L289]  right after it joins the two datasets together. This call results in a state being created to account for late arriving data. 

Watermarks created in the PySpark code are being ignored and still lead to the state accumulating in size. 

My expectation is that the watermarking is lost in between the communication from Python to Scala. 

I've created [this Stackoverflow question|https://stackoverflow.com/questions/64157104/stream-static-join-without-aggregation-still-results-in-accumulating-spark-state] earlier this week, but after more investigation this really seem like a bug rather than a user error.
Bug: Make Literal ArrayBasedMapData string representation disambiguous
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Oct/20 22:57
Affects Version/s: 2.0.2

Literal ArrayBasedMapData has inconsistent string representation from `LogicalPlan` to `Optimized Logical Plan/Physical Plan`. Also, the representation at `Optimized Logical Plan` and `Physical Plan` is ambiguous like `keys: [key1], values: [value1] AS c#0`.

{code}
scala> spark.version
res0: String = 2.4.7

scala> sql("SELECT map('key1', 'value1') c").explain(true)
== Parsed Logical Plan ==
'Project ['map(key1, value1) AS c#0]
+- OneRowRelation

== Analyzed Logical Plan ==
c: map<string,string>
Project [map(key1, value1) AS c#0]
+- OneRowRelation

== Optimized Logical Plan ==
Project [keys: [key1], values: [value1] AS c#0]
+- OneRowRelation

== Physical Plan ==
*(1) Project [keys: [key1], values: [value1] AS c#0]
+- Scan OneRowRelation[]
{code}
Bug: Fix bugs for casting data from/to PythonUserDefinedType
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/20 12:22
Affects Version/s: 2.4.8

This PR intends to fix bus for casting data from/to PythonUserDefinedType. A sequence of queries to reproduce this issue is as follows;

{code} 
>>> from pyspark.sql import Row
>>> from pyspark.sql.functions import col
>>> from pyspark.sql.types import *
>>> from pyspark.testing.sqlutils import *
>>> 
>>> row = Row(point=ExamplePoint(1.0, 2.0))
>>> df = spark.createDataFrame([row])
>>> df.select(col("point").cast(PythonOnlyUDT()))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/maropu/Repositories/spark/spark-master/python/pyspark/sql/dataframe.py", line 1402, in select
    jdf = self._jdf.select(self._jcols(*cols))
  File "/Users/maropu/Repositories/spark/spark-master/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1305, in __call__
  File "/Users/maropu/Repositories/spark/spark-master/python/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/Users/maropu/Repositories/spark/spark-master/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o44.select.
: java.lang.NullPointerException
	at org.apache.spark.sql.types.UserDefinedType.acceptsType(UserDefinedType.scala:84)
	at org.apache.spark.sql.catalyst.expressions.Cast$.canCast(Cast.scala:96)
	at org.apache.spark.sql.catalyst.expressions.CastBase.checkInputDataTypes(Cast.scala:267)
	at org.apache.spark.sql.catalyst.expressions.CastBase.resolved$lzycompute(Cast.scala:290)
	at org.apache.spark.sql.catalyst.expressions.CastBase.resolved(Cast.scala:290)}}
{code} 

A root cause of this issue is that, since {{PythonUserDefinedType#userClassis}} always null, {{isAssignableFrom}} in {{UserDefinedType#acceptsType}} throws a null exception. To fix it, this PR defines {{acceptsType}} in {{PythonUserDefinedType}} and filters out the null case in {{UserDefinedType#acceptsType}}.
Bug: Fail fast when fails to instantiate configured v2 session catalog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 05:23
Affects Version/s: 3.0.0

Now Spark fails back to use "default catalog" when Spark fails to instantiate configured v2 session catalog.

While the error log message says nothing about why the instantiation has been failing and the error log message pollutes the log file (as it's logged every time when resolving the catalog), it should be considered as "incorrect" behavior as end users are intended to set the custom catalog and Spark sometimes ignores it, which is against the intention.

We should simply fail in the case so that end users indicate the failure earlier and try to fix the issue.
Bug: Add total duration, read duration, and write duration as task level metrics
Status: Open | Priority: Major | Resolution: 
Created: 28/Oct/20 02:34
Affects Version/s: 3.0.1

Sometimes we need to identify performance bottlenecks, for example, how long it took to read from data store, how long it took to write into another data store.

It would be great if we can have total duration, read duration, and write duration as task level metrics.

Currently it seems that both `InputMetrics` and `OutputMetrics` do not have duration related metrics.

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/InputMetrics.scala#L42-L58]

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/OutputMetrics.scala#L41-L56]

 

On the other hand, other metrics such as `ShuffleWriteMetrics` has write time. We might need similar metrics for input/output.

[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/ShuffleWriteMetrics.scala]
Bug: Add a dedicated page for SQL-on-file in SQL documents
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Oct/20 01:04
Affects Version/s: 3.0.2

This ticket intends to add a dedicated page for SQL-on-file in SQL documents.
This comes from the comment: [https://github.com/apache/spark/pull/30095/files#r508965149]
Bug: Don't uncache data when replacing an existing view having the same plan
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/20 13:18
Affects Version/s: 2.4.8

SPARK-30494's updated the `CreateViewCommand` code to implicitly drop cache when replacing an existing view. But, this change drops cache even when replacing a view having the same logical plan. A sequence of queries to reproduce this as follows;
{code}
scala> val df = spark.range(1).selectExpr("id a", "id b")
scala> df.cache()
scala> df.explain()
== Physical Plan ==
*(1) ColumnarToRow
+- InMemoryTableScan [a#2L, b#3L]
 +- InMemoryRelation [a#2L, b#3L], StorageLevel(disk, memory, deserialized, 1 replicas)
 +- *(1) Project [id#0L AS a#2L, id#0L AS b#3L]
 +- *(1) Range (0, 1, step=1, splits=4)


scala> df.createOrReplaceTempView("t")
scala> sql("select * from t").explain()
== Physical Plan ==
*(1) ColumnarToRow
+- InMemoryTableScan [a#2L, b#3L]
 +- InMemoryRelation [a#2L, b#3L], StorageLevel(disk, memory, deserialized, 1 replicas)
 +- *(1) Project [id#0L AS a#2L, id#0L AS b#3L]
 +- *(1) Range (0, 1, step=1, splits=4)


// If one re-runs the same query `df.createOrReplaceTempView("t")`, the cache's swept away
scala> df.createOrReplaceTempView("t")
scala> sql("select * from t").explain()
== Physical Plan ==
*(1) Project [id#0L AS a#2L, id#0L AS b#3L]
+- *(1) Range (0, 1, step=1, splits=4)
{code}
Bug: Make podCreationTimeout configurable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/20 18:36
Affects Version/s: 3.0.0

Execution Monitor & Pod Allocator have differing views of the world which can lead to pod trashing.

The executor monitor can be notified of an executor coming up before a snapshot is delivered to the PodAllocator. This can cause the executor monitor to believe it needs to delete a pod, and the pod allocator to believe that it needs to create a new pod. This happens if the podCreationTimeout is too low for the cluster. Currently podCreationTimeout can only be configured by increasing the batch delay but that has additional consequences leading to slower spin up.
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: update type and nullability of columns (PostgreSQL dialect)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 23:50
Affects Version/s: 3.1.0

Override the default SQL strings for:

ALTER TABLE UPDATE COLUMN TYPE
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following PostgreSQL JDBC dialect according to official documentation.

Write PostgreSQL integration tests for JDBC.
Bug: Speed up event log download by skipping UI rebuild
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Oct/20 01:12
Affects Version/s: 2.4.7

Right now, when we want to download the event logs from the spark history server(SHS), SHS will need to parse entire the event log to rebuild UI, and this is just for view permission checks. UI rebuilding is a time-consuming and memory-intensive task, especially for large logs. However, this process is unnecessary for event log download.

This patch enables SHS to check UI view permissions of a given app/attempt for a given user, without rebuilding the UI. This is achieved by adding a method "checkUIViewPermissions(appId: String, attemptId: Option[String], user: String): Boolean" to many layers of history server components.

With this patch, UI rebuild can be skipped when downloading event logs from the history server. Thus the time of downloading a GB scale event log can be reduced from several minutes to several seconds, and the memory consumption of UI rebuilding can be avoided.
Bug: Use default Hadoop profile by removing explicit `-Phadoop-2.7` from K8s IT Jenkins job
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 00:56
Affects Version/s: 3.1.0

Since Apache Spark 3.1.0, the default Hadoop version is 3.1.0.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/configure
Bug: Dynamic pruning for data column
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Oct/20 05:45
Affects Version/s: 3.1.0

SPARK-11150 add new feature  to support dynamic partition pruning. This feature has some limitations:
1. Only support pruning on partition column.
2. This feature is rarely used in production because the proportion of join on the partition column is relatively low.

This ticket aim to address support dynamic data(column) pruning.
Bug: `scheduleAtFixedRate` change to `schedulerWithFixedDelay` to avoid repeated unecessary scheduling for a short time
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Oct/20 08:05
Affects Version/s: 3.1.0

For some schedule behavior, we will `scheduleAtFixedRate` to schedule task or heartbeat RPC etc...
If thread blocked by a big fullGc or other reason cause delay, will schedule task repeatedly. For some behavior it's not necessary. We can use `scheduleWithFixedDelay` to avoid it.
Bug: Support ALTER TABLE in JDBC v2 Table Catalog: add, update type and nullability of columns (MySQL dialect)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Oct/20 10:37
Affects Version/s: 3.1.0

Override the default SQL strings for:
ALTER TABLE UPDATE COLUMN TYPE
ALTER TABLE UPDATE COLUMN NULLABILITY
in the following MySQL JDBC dialect according to official documentation.
Write MySQL integration tests for JDBC.