av9ash/GitBugs huggingface dataset:
Bug: De-duplicated common columns cannot be resolved from USING/NATURAL JOIN
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Feb/21 19:29
Affects Version/s: 2.0.0

USING/NATURAL JOINS today have unexpectedly asymmetric behavior when resolving the duplicated common columns. For example, the left key columns can be resolved from a USING INNER JOIN, but the right key columns cannot. This is due to the Analyzer's [rewrite|https://github.com/apache/spark/blob/999d3b89b6df14a5ccb94ffc2ffadb82964e9f7d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3397] of NATURAL/USING JOINs, which uses Project to remove the duplicated common columns.
Bug: `first` function is sorting the dataset while sometimes it is used to get "any value"
Status: Resolved | Priority: Minor | Resolution: Not A Bug
Created: 18/Feb/21 15:44
Affects Version/s: 3.0.0

When one wants to groupBy and take any value (not necessarily the first), one usually uses [first|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L485] aggregation function.

Unfortunately, this method uses a `SortAggregate` for some data types, which is not always necessary and might impact performances. Is this the desired behavior?

 

 
{code:java}
Current behavior:
 val df = Seq((0, "value")).toDF("key", "value")
df.groupBy("key").agg(first("value")).explain()
 /*
 == Physical Plan ==
 SortAggregate(key=key#342, functions=first(value#343, false))
 +- *(2) Sort key#342 ASC NULLS FIRST, false, 0
    +- Exchange hashpartitioning(key#342, 200)
       +- SortAggregate(key=key#342, functions=partial_first(value#343, false))
          +- *(1) Sort key#342 ASC NULLS FIRST, false, 0
             +- LocalTableScan key#342, value#343
 */
{code}
 

My understanding of the source code does not allow me to fully understand why this is the current behavior.

The solution might be to implement a new aggregate function. But the code would be highly similar to the first one. And if I don't fully understand why this [createAggregate|https://github.com/apache/spark/blob/3a299aa6480ac22501512cd0310d31a441d7dfdc/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala#L45] method falls back to SortAggregate.
Bug: Improve error message in default spark session catalog
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Feb/21 20:42
Affects Version/s: 3.2.0

See [https://github.com/apache/spark/pull/31541/files]

We should make sure that "unsupported function name 'default.ns1.ns2.fun'" contains information about why the function name is unsupported.
Bug: .foreachPartition command hangs when ran inside Python package but works when ran from Python file outside the package on EMR
Status: Open | Priority: Minor | Resolution: 
Created: 23/Feb/21 17:12
Affects Version/s: 3.0.0

I'm running on EMR Pyspark 3.0.0. with project structure below, process.py is what controls the flow of the application and calls code inside the _file_processor_ package. The command hangs when the .foreachPartition code that is located inside _s3_repo.py_ is called by _process.py_. When the same .foreachPartition code is moved from _s3_repo.py_ and placed inside the _process.py_ it runs just fine.
{code:java}
process.py
file_processor
  config        
    spark.py
  repository        
    s3_repo.py
  structure        
    table_creator.py

{code}
*process.py*
{code:java}
from file_processor.structure import table_creator
from file_processor.repository import s3_repo

def process():
    table_creator.create_table()
    s3_repo.save_to_s3()

if __name__ == '__main__':
    process()
{code}
*spark.py*
{code:java}
from pyspark.sql import SparkSession
spark_session = SparkSession.builder.appName("Test").getOrCreate()
{code}
*s3_repo.py* 
{code:java}
from file_processor.config.spark import spark_session

def save_to_s3():
    spark_session.sql('SELECT * FROM rawFileData').toJSON().foreachPartition(_save_to_s3)

def _save_to_s3(iterator):   
    for record in iterator:
        print(record)
{code}
 *table_creator.py*
{code:java}
from file_processor.config.spark import spark_session
from pyspark.sql import Row

def create_table():
    file_contents = [
        {'line_num': 1, 'contents': 'line 1'},
        {'line_num': 2, 'contents': 'line 2'},
        {'line_num': 3, 'contents': 'line 3'}        
    ]
    spark_session.createDataFrame(Row(**row) for row in file_contents).cache().createOrReplaceTempView("rawFileData")
{code}
Bug: Hive Metastore support filter by not-in
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/21 12:03
Affects Version/s: 3.2.0

`NOT IN` is a useful condition to prune partition, it would be better to support it. Technically, we can convert `c not in(x,y)` to `c != x and c != y`, then push it to metastore.
Bug: Eliminate LEFT ANTI join to empty relation in AQE
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/21 02:49
Affects Version/s: 3.2.0

I discovered from review discussion - [https://github.com/apache/spark/pull/31630#discussion_r581774000] , that we can eliminate LEFT ANTI join (with no join condition) to empty relation, if the right side is known to be non-empty. So with AQE, this is doable similar to [https://github.com/apache/spark/pull/29484] .
Bug: Fixed an issue where data could not be cleaned up when unregisterShuffle
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Feb/21 13:04
Affects Version/s: 3.0.0

While we use the old shuffle fetch protocol, we use partitionId as mapId in the ShuffleBlockId construction,but we use `context.taskAttemptId()` as mapId that it is cached in `taskIdMapsForShuffle` when we `getWriter[K, V]`.

where data could not be cleaned up when unregisterShuffle ,because we remove a shuffle's metadata from the `taskIdMapsForShuffle`'s mapIds, the mapId is `context.taskAttemptId()` instead of partitionId.
Bug: Resolve metadata output from DataFrame
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 26/Feb/21 20:20
Affects Version/s: 3.1.0

Today, we can't resolve a metadata column from a Spark dataframe from Scala with {{df.col("metadataColName")}}. This is because metadata output is only used during {{resolveChildren}} (used during SQL resolution); it should likely also be used by {{resolve}} (used during Scala resolution).
Bug: add a java implementation for the simple writable data source
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Feb/21 16:23
Affects Version/s: 3.0.1

This is a followup of https://github.com/apache/spark/pull/19269

In #19269 , there is only a scala implementation of simple writable data source in `DataSourceV2Suite`.

This PR adds a java implementation of it.
Bug: fix the remaining problems in SPARK-34432
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 22/Feb/21 13:47
Affects Version/s: 3.0.1

the remaining problems :

1.  we don't need to implement SessionConfigSupport in simple writable table data source tests. remove it from both the Scala and Java versions.
2. change the schema of `SimpleWritableDataSource`, to match `TestingV2Source`
Bug: Documentation generation check fails in GitHub 
Status: Closed | Priority: Minor | Resolution: Duplicate
Created: 28/Feb/21 20:04
Affects Version/s: 3.0.2

The run documentation check has been failing for multiple PRs in GitHub because the following [commit|https://github.com/apache/spark/commit/397b843890db974a0534394b1907d33d62c2b888#diff-02f10976f2f83e219445cd4860a2999f45dc254b173f1ffefb323d8fe1d1c817R100-R108]  included a reference to the following file (model_selection_random_hyperparameters_example.py).

However this file is unavailable in the examples folder in the spark project. This is causing the (Linters, licenses, dependencies and documentation generation) check to fail in the documentation build in Github actions. So, this has to be removed to let the build run successfully.
Bug: Refactor the unified v1 and v2 command tests
Status: Open | Priority: Minor | Resolution: 
Created: 16/Feb/21 10:38
Affects Version/s: 3.2.0

The ticket aims to gather potential improvements for the unified tests.
 1. Remove SharedSparkSession from *ParserSuite
 2. Rename tests like AlterTableAddPartitionSuite -> AddPartitionsSuite
 3. Add JIRA ID SPARK-33829 to "SPARK-33786: Cache's storage level should be respected when a table name is altered"
 4. Reset default namespace in ShowTablesSuiteBase."change current catalog and namespace with USE statements" using spark.sessionState.catalogManager.reset()
Bug: The getMetricsSnapshot method of the PrometheusServlet class has a wrong value
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Feb/21 03:42
Affects Version/s: 3.0.1

The mean value of timersLabels in the PrometheusServlet class is wrong, You can look at line 105 of this class: [L105.|https://github.com/apache/spark/blob/37fe8c6d3cd1c5aae3a61fb9b32ca4595267f1bb/core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala#L105]
{code:java}
// code placeholder
sb.append(s"${prefix}Mean$timersLabels ${snapshot.getMax}\n"){code}
Bug: Fix NPE if InSet contains null value during getPartitionsByFilter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Feb/21 04:40
Affects Version/s: 3.1.2

Spark will convert InSet to `>= and <=` if it's values size over `spark.sql.hive.metastorePartitionPruningInSetThreshold` during pruning partition . At this case, if values contain a null, we will get such exception 

 
{code:java}
java.lang.NullPointerException
 at org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:1389)
 at org.apache.spark.unsafe.types.UTF8String.compareTo(UTF8String.java:50)
 at scala.math.LowPriorityOrderingImplicits$$anon$3.compare(Ordering.scala:153)
 at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
 at java.util.TimSort.sort(TimSort.java:220)
 at java.util.Arrays.sort(Arrays.java:1438)
 at scala.collection.SeqLike.sorted(SeqLike.scala:659)
 at scala.collection.SeqLike.sorted$(SeqLike.scala:647)
 at scala.collection.AbstractSeq.sorted(Seq.scala:45)
 at org.apache.spark.sql.hive.client.Shim_v0_13.convert$1(HiveShim.scala:772)
 at org.apache.spark.sql.hive.client.Shim_v0_13.$anonfun$convertFilters$4(HiveShim.scala:826)
 at scala.collection.immutable.Stream.flatMap(Stream.scala:489)
 at org.apache.spark.sql.hive.client.Shim_v0_13.convertFilters(HiveShim.scala:826)
 at org.apache.spark.sql.hive.client.Shim_v0_13.getPartitionsByFilter(HiveShim.scala:848)
 at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getPartitionsByFilter$1(HiveClientImpl.scala:750)
{code}
Bug: ExecutorPodsAllocator use exponential backoff strategy when request executor pod failed
Status: Open | Priority: Minor | Resolution: 
Created: 24/Feb/21 10:27
Affects Version/s: 3.0.1

# create a resouce quota `kubectl create quota test --hard=cpu=20,memory=60G`
 # submit an application request more than quota `spark-submit --executor-cores 5 --executor-memory 10G --num-executors 10 <spark-pi.py>`
 # seems `ExecutorPodsAllocator: Going to request 5 executors from Kubernetes.` print every second

`spark.kubernetes.allocation.batch.delay` default is 1s, which good enough when allocation succeeded, but exponential backoff maybe an better choice when alloction failed.
Bug: ExecutorPodsAllocator fails to identify some excess requests during downscaling
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Feb/21 20:18
Affects Version/s: 3.0.0

During downscaling there are two kinds of POD requests which can lead to POD deletion as it is identified as excess requests by the dynamic allocation:
 * timed out newly created POD requests (which even haven't reached the Pod Pending state, yet) 
 * timed out pending pod requests

The current implementation fails to delete a timed out pending pod requests when there are not enough timed out newly created POD requests to delete but there are some non-timed out ones.
Bug: Map JDBC RowID to StringType rather than LongType
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Feb/21 18:17
Affects Version/s: 3.2.0

In the current implementation, JDBC RowID type is mapped to LongType except for OracleDialect, but there is no guarantee to be able to convert RowID to long.
java.sql.RowId declares toString and the specification of java.sql.RowId says

_all methods on the RowId interface must be fully implemented if the JDBC driver supports the data type_
(https://docs.oracle.com/javase/8/docs/api/java/sql/RowId.html)

So, we should prefer StringType to LongType.
Bug: Module launcher build failed with profile hadoop-3.2 activated
Status: Closed | Priority: Minor | Resolution: Fixed
Created: 20/Feb/21 07:10
Affects Version/s: 3.0.1

Build spark 3.0.1 with profile hadoop-3.2 activated
{code:java}
build/mvn -pl :spark-launcher_2.12 package -DskipTests -Phadoop-3.2 -Phive -Phive-thriftserver -Pkubernetes
{code}
When building the spark-launcher module it complains that lacking the common-lang dependency:


{code:java}
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ spark-launcher_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /Users/lichuanliang/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.10__52.0-1.3.1_20191012T045515.jar
[INFO] Compiling 20 Java sources to /Users/lichuanliang/IdeaProjects/qtt-spark-3.0/launcher/target/scala-2.12/classes ...
[ERROR] [Error] /Users/lichuanliang/IdeaProjects/qtt-spark-3.0/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:20: package org.apache.commons.lang does not exist
[ERROR] [Error] /Users/lichuanliang/IdeaProjects/qtt-spark-3.0/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:226: cannot find symbol
  symbol:   variable StringUtils
  location: class org.apache.spark.launcher.SparkSubmitCommandBuilder
[ERROR] [Error] /Users/lichuanliang/IdeaProjects/qtt-spark-3.0/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:227: cannot find symbol
  symbol:   variable StringUtils
  location: class org.apache.spark.launcher.SparkSubmitCommandBuilder
[ERROR] [Error] /Users/lichuanliang/IdeaProjects/qtt-spark-3.0/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:232: cannot find symbol
  symbol:   variable StringUtils
  location: class org.apache.spark.launcher.SparkSubmitCommandBuilder
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

{code}
Bug: map_from_arrays() throws UnsupportedOperationException with array of ColumnarMap
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 18/Feb/21 00:30
Affects Version/s: 3.2.0

An example to reproduce this error: 
{code:scala}
sql("select map(1, 2) as m_a, map(3, 4) as m_b").write.saveAsTable("t")
sql("select map_from_arrays(array(1, 2), array(m_a, m_b)) from t")
{code}
Exception trace:
{code:java}
java.lang.UnsupportedOperationException
	at org.apache.spark.sql.vectorized.ColumnarMap.copy(ColumnarMap.java:51)
	at org.apache.spark.sql.vectorized.ColumnarMap.copy(ColumnarMap.java:25)
	at org.apache.spark.sql.catalyst.InternalRow$.copyValue(InternalRow.scala:121)
	at org.apache.spark.sql.catalyst.util.GenericArrayData.copy(GenericArrayData.scala:54){code}
This is due to ColumnarMap's copy method not implemented.
Bug: SQL configs from the legacy namespace must be internal
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Feb/21 17:05
Affects Version/s: 3.2.0

Assumed that legacy SQL configs shouldn't be set by users in common cases. The purpose of the configs is to allow switching to old behavior in corner cases. So, the configs can be marked as internals. The ticket aims to inspect existing SQL configs in SQLConf and add internal() call to config entry builders.
Bug: ExecutorPodsLifecycleManager fails to remove executors in Kubernetes, SPARK 3.0.1
Status: Open | Priority: Minor | Resolution: 
Created: 17/Feb/21 12:59
Affects Version/s: 3.0.1

Happens when driver fails to register the reason behind deletion, e.g:
{code:java}
2021-02-17 12:07:56,953 DEBUG KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:61 - Asked to remove executor 1 with reason The executor with id 1 was deleted by a user or the framework.
{code}
 ExecutorPodsLifecycleManager fails to remove missing executor and gets stuck in this loop:  
{code:java}
2021-02-17 12:13:39,023 DEBUG ExecutorPodsLifecycleManager:61 - Removed executors with ids 3 from Spark that were either found to be deleted or non-existent in the cluster.
2021-02-17 12:15:09,042 DEBUG ExecutorPodsLifecycleManager:61 - The executor with ID 3 was not found in the cluster but we didn't get a reason why. Marking the executor as failed. The executor may have been deleted but the driver missed the deletion event.
{code}
 

Steps to reproduce: 
 # Deploy spark cluster into  Kubernetes
 # Delete an executor pod though kubectl

 

Could be linked / duplicate of  SPARK-28488
Bug: Python Driver is not correctly detected using presigned URLs
Status: In Progress | Priority: Minor | Resolution: 
Created: 15/Feb/21 01:51
Affects Version/s: 3.0.0

In AWS one can generate so-called presigned URLs. spark-submit accepts URLs for the driver program, e.g. {{http://my-web-server/driver.py}}. Now a presigned URL has a query fragment {{http://my-web-server/driver.py?signature}}.

Now the check for whether the given URL is a python driver simply checks whether it ends in {{.py}} – which the presigned URL does not, as it ends in {{signature}}.

The relevant check is in {{SparkSubmit.scala}}, Line 1051 (commit tagged {{v3.0.1}}):

[https://github.com/apache/spark/blob/v3.0.1/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L1051] 

Here is a more realistic example URL:

{{https://bucket-name.s3.us-east-1.amazonaws.com/driver.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATBNPKWPCNUMWMLUR%2F20210214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210214T062047Z&X-Amz-Expires=172800&X-Amz-SignedHeaders=host&X-Amz-Signature=49ef39b6bb7090001af9312692788892551916a6ac0ff6c961ce52efb9acc235}}

A fix could be to parse the the given path as a {{java.net.URI}} and look for the pathname to end in {{.py}} (as opposed to the whole thing).

To circumvent this issue I am currently appending a fragment to the query which makes it end in {{.py}}, i.e. {{http://my-web-server/driver.py?signature#.py}} which does work.
Bug: Support avroSchemaUrl in addition to avroSchema
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/21 10:29
Affects Version/s: 3.2.0

We have a use case in which we read a huge table in Avro format. About 30k columns.

using the default Hive reader - `AvroGenericRecordReader` it is just hangs forever. after 4 hours not even one task has finished.

We tried instead to use `spark.read.format("com.databricks.spark.avro").load(..)` but we failed on:

```

org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema

..

at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtils.scala:85)
 at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtils.scala:67)
 at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:421)
 at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
 at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
 ... 53 elided

```

 

because files schema contain duplicate column names (when considering case-insensitive).

So we wanted to provide a user schema with non-duplicated fields, but the schema is huge. a few MBs. it is not practical to provide it in json format.

 

So we patched spark-avro to be able to get also `avroSchemaUrl` in addition to `avroSchema` and it worked perfectly.
Bug: Allow FileTable.fileIndex to be reused for custom partition schema in DataSourceV2 read path
Status: Open | Priority: Minor | Resolution: 
Created: 11/Feb/21 15:54
Affects Version/s: 3.0.1

It is currently possible to provide custom partition schema in DataSourceV2 read path with custom implementations of PartitionAwareFileIndex/PartitionSpec and by overriding fileIndex in a subclass of FileTable. Since fileIndex is lazy val it's not possible to reuse it from the subclass however (i.e. super.fileIndex).

[https://github.com/apache/spark/blob/e0053853c90d39ef6de9d59fb933525e20bae1fa/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileTable.scala#L44-L61]

Duplicating this code in the subclass is possible but somewhat hacky e.g. DataSource globbing function is private API. I was wondering if this logic can be refactored into something like this:
{code:java}
def createFileIndex(): PartitionAwareFileIndex = {
  ...[current fileIndex logic]...
}

lazy val fileIndex: PartitionAwareFileIndex = createFileIndex(){code}
This would allow reusing fileIndex logic downstream by wrapping it up with custom implementations.

(Note that this proposed change considers custom partition schema in read path only. Write path is out of the scope of this change.)
Bug: Throw exception if non-streaming Deduplicate is not replaced by aggregate
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/21 21:47
Affects Version/s: 3.2.0

We replace some operations in the query optimizer. For them we throw some exceptions accordingly in query planner if these logical nodes are not replaced. But `Deduplicate` is missing and it opens a possible hole. For code consistency and to prevent possible unexpected query planning error, we should add similar exception case to query planner.
Bug: Move PartitionTransforms from java to scala directory
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 10/Feb/21 20:23
Affects Version/s: 3.2.0

{{PartitionTransforms}} is currently under {{sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions}}. It should be under {{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions}}
Bug: Add a new build-in function delegate
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Feb/21 12:06
Affects Version/s: 3.2.0

Delegate is just like a big box that can include some other expressions. It will execute all of children and return the last child result as its result.

 

The origin idea is from debug. Debug SQL is hard since SQL is always quite long and complex. This new function can help debug with inject some help functions, e.g., `println, sleep, raise_error`.

 

Two usage examples:
{code:java}
-- raw sql
INSERT INTO TABLE t1
SELECT coalesce(c1, c2) as c FROM t2
 
-- print the column data
INSERT INTO TABLE t1
SELECT delegate(
java_method('scala.Console', 'println', concat('c1: ', c1, ', c2: ', c2)),
coalesce(c1, c2)
) as c FROM t2{code}
 
{code:java}
-- raw sql
SELECT if(spark_partition_id() = 1, c1, raise_error('test error')) FROM t2
 
-- add a sleep time before throw error
SELECT if(spark_partition_id() = 1, c1, delegate(
java_method('java.lang.Thread', 'sleep', 3000l),
raise_error('test error')
)) FROM t2{code}