av9ash/GitBugs huggingface dataset:
Bug: Resolve using child metadata attributes as fallback
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/21 23:58
Affects Version/s: 3.1.0

Today, child expressions may be resolved based on "real" or metadata output attributes. If a "real" attribute and a metadata attribute have the same name, we should prefer the real one during resolution. This resolution fails today, although the user may not be aware of the metadata.
Bug: SQLConf sqlConfEntries map has a global lock, should not lock on get
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Feb/21 22:36
Affects Version/s: 2.4.7

SQLConf object has a sqlConfEntries map, which has a global lock (since it implements a Collections.synchronizedMap).
 Every operation (like get or set) blocks the full object.
 Concurrent threads may wait on lock.

An example is the DatatType.sameType method, that queries SQLConf entries map:
{code:scala}
if (SQLConf.get.caseSensitiveAnalysis) 
 ...
{code}
If this data type check is run in a custom piece of code on an executor with multiple cores (eg: 40), then a lot of time will be lost waiting on the lock.
 An easy fix is to use the a ConcurrentHashMap that does not lock on read SQLConf.get): " ... retrieval operations do not entail locking ..."

NOTE: originally discovered by Benson Hon <benson.h@taboola.com> at Taboola while debugging strange slowdowns in production code
Bug: Cannot join datasets of SHOW TABLES
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/21 12:34
Affects Version/s: 3.2.0

The example portraits the issue:

{code:scala}
scala> sql("CREATE NAMESPACE ns1")
res8: org.apache.spark.sql.DataFrame = []

scala> sql("CREATE NAMESPACE ns2")
res9: org.apache.spark.sql.DataFrame = []

scala> sql("CREATE TABLE ns1.tbl1 (c INT)")
res10: org.apache.spark.sql.DataFrame = []

scala> sql("CREATE TABLE ns2.tbl2 (c INT)")
res11: org.apache.spark.sql.DataFrame = []

scala> val show1 = sql("SHOW TABLES IN ns1")
show1: org.apache.spark.sql.DataFrame = [namespace: string, tableName: string ... 1 more field]

scala> val show2 = sql("SHOW TABLES IN ns2")
show2: org.apache.spark.sql.DataFrame = [namespace: string, tableName: string ... 1 more field]

scala> show1.show
+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
|      ns1|     tbl1|      false|
+---------+---------+-----------+


scala> show2.show
+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
|      ns2|     tbl2|      false|
+---------+---------+-----------+


scala> show1.join(show2).where(show1("tableName") =!= show2("tableName")).show
org.apache.spark.sql.AnalysisException: Column tableName#17 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.
  at org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:157)
{code}
Bug: Vincenty distance spark function
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Feb/21 12:44
Affects Version/s: 3.2.0

Add new spark function to compute the distance between two geo-points using vincenty distance formula
Vincenty uses the oblate spheroid whereas haversine uses unit sphere, this will give roughly
22m better accuracy (in worst case) than haversine
https://en.wikipedia.org/wiki/Vincenty%27s_formulae
Bug: Cannot drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Feb/21 17:06
Affects Version/s: 3.2.0

Dropping a column from a dataset of v2 `DESCRIBE TABLE` fails with:
{code:java}
Resolved attribute(s) col_name#102,data_type#103 missing from col_name#29,data_type#30,comment#31 in operator !Project [col_name#102, data_type#103]. Attribute(s) with the same name appear in the operation: col_name,data_type. Please check if the right attribute(s) are used.;
!Project [col_name#102, data_type#103]
+- LocalRelation [col_name#29, data_type#30, comment#31]{code}
The code below demonstrates the issue:
{code:java}
val tbl = s"${catalogAndNamespace}tbl"
withTable(tbl) {
  sql(s"CREATE TABLE $tbl (c0 INT) USING $v2Format")
  val description = sql(s"DESCRIBE TABLE $tbl")
  val noComment = description.drop("comment")
}
{code}
Bug: ADD JAR with ivy coordinates should be compatible with Hive transitive behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Feb/21 09:34
Affects Version/s: 3.2.0

SPARK-33084 added the ability to use ivy coordinates with `SparkContext.addJar`. [PR #29966|https://github.com/apache/spark/pull/29966] claims to mimic Hive behavior although I found a few cases where it doesn't

1) The default value of the {{transitive}} parameter is false, both in case of parameter not being specified in coordinate or parameter value being invalid. The Hive behavior is {{transitive}} is [true if not specified|https://github.com/apache/hive/blob/cb2ac3dcc6af276c6f64ee00f034f082fe75222b/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L169] in the coordinate and [false for invalid values|https://github.com/apache/hive/blob/cb2ac3dcc6af276c6f64ee00f034f082fe75222b/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L124]. Also, regardless of Hive, I think a default of {{true}} for the transitive parameter also matches [ivy's own defaults|https://ant.apache.org/ivy/history/2.5.0/ivyfile/dependency.html#_attributes].

2) The parameter value for {{transitive}} parameter is regarded as case-sensitive [based on the understanding|https://github.com/apache/spark/pull/29966#discussion_r547752259] that Hive behavior is case-sensitive. However, this is not correct, Hive [treats the parameter value case-insensitively|https://github.com/apache/hive/blob/cb2ac3dcc6af276c6f64ee00f034f082fe75222b/ql/src/java/org/apache/hadoop/hive/ql/util/DependencyResolver.java#L122].
Bug: Provide a more convenient way to deprecate/remove/alternate configs
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Feb/21 15:48
Affects Version/s: 3.2.0

For example,

 
{code:java}
val SOME_CONFIG =
  buildConf("spark.xxx.yyy.zzz")
    .doc(".....")")
    .version("2.3.0")
    .deprecated("3.0", "because...")
    .booleanConf
    .createWithDefault(false)
{code}
Bug: Respect case sensitivity in V1 ALTER TABLE .. SET LOCATION
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Feb/21 15:18
Affects Version/s: 2.4.7

SHOW PARTITIONS is case sensitive, and doesn't respect the SQL config *spark.sql.caseSensitive* which is false by default, for instance:
{code:sql}
spark-sql> CREATE TABLE tbl (id INT, part INT) PARTITIONED BY (part);
spark-sql> INSERT INTO tbl PARTITION (part=0) SELECT 0;
spark-sql> SHOW TABLE EXTENDED LIKE 'tbl' PARTITION (part=0);
Location: file:/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0
spark-sql> ALTER TABLE tbl ADD PARTITION (part=1);
spark-sql> SELECT * FROM tbl;
0	0
{code}
Create new partition folder in the file system:
{code}
$ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa
{code}
Set new location for the partition part=1:
{code:sql}
spark-sql> ALTER TABLE tbl PARTITION (part=1) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/aaa';
spark-sql> SELECT * FROM tbl;
0	0
0	1
spark-sql> ALTER TABLE tbl ADD PARTITION (PART=2);
spark-sql> SELECT * FROM tbl;
0	0
0	1
{code}
Set location for a partition in the upper case:
{code}
$ cp -r /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/part=0 /Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb
{code}
{code:sql}
spark-sql> ALTER TABLE tbl PARTITION (PART=2) SET LOCATION '/Users/maximgekk/proj/set-location-case-sense/spark-warehouse/tbl/bbb';
Error in query: Partition spec is invalid. The spec (PART) must match the partition spec (part) defined in table '`default`.`tbl`'
{code}
Bug: ExternalCatalog listPartitions and listPartitionsByFilter calls should also restore metadata
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Feb/21 09:12
Affects Version/s: 3.0.2

ExternalCatalog call getPartition restores partition-level stats from Hive table metadata. However, listPartitions and listPartitionsByFilter calls do not restore these partition stats, which leads to discrepancies between returned CatalogPartition between these API calls.
Bug: zstd-jni lead to read less shuffle data
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Feb/21 09:37
Affects Version/s: 2.4.0

h2. BackGround

I find a rare case which lead some partitions read less data when use zstd;
h2. Detail

I saved normal shuffle data and wrong shuffle data and found the wrong shuffle data was the head part of the normal shuffle data, and I found zstd-jni tag 1.3.3-2 has the problems which can read  a head part of whole data and normal exit.

The ZstdInputStream in zstd-jni(tag 1.3.3-2) maybe return 0 after a read function call, this doesn't meet the standard of InputStream, InputStream will not return 0 unless len is 0; Spark will use a BufferedInputStream wrapped to ZstdInputStream, when ZstdInputStream read call return 0, BufferedInputStream will consider the 0 as the end of read and exit, this can lead data loss.

zstd-jni issues:

[https://github.com/luben/zstd-jni/issues/159]

zstd-jni commits:
https://github.com/luben/zstd-jni/commit/7eec5581b8ccb0d98350ad5ba422337eebbbe70e

zstd-jni has fixed this problems in tag 1.4.4-3, the code as follows:

!image-2021-02-25-17-50-49-427.png|width=544,height=232!

So, I think it's nessary to upgrade zstd-jni version to 1.4.4-3 in spark2.4 for spark2.4 has a wide use in production.

 

The BufferedInputStream's code as follows:

!image-2021-02-25-17-51-49-998.png|width=539,height=310!
Bug: View result are not consistent after a modification inside a struct of the table
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Feb/21 21:09
Affects Version/s: 3.0.2

After [https://github.com/apache/spark/pull/31368] work to simplify hive view resolution
I found a bug because Hive allow you to change the order inside a struct

1) You create a table in hive with a struct:
 CREATE table test_struct (id int, sub STRUCT <a :INT, b:STRING>);
2) You insert data into it :
INSERT INTO TABLE test_struct select 1, named_struct("a",1,"b","v1");
3) Create a view on top of it :
CREATE view test_view_struct as select id, sub from test_view_struct
4) Change the table struct reodoring the struct
ALTER TABLE test_struct CHANGE COLUMN sub sub STRUCT < b:STRING,a :INT>;
5) Spark can not anymore query the view because struct in spark it's based on the position not on the name of the column.
If the changement it's castable you can even have a silent failed
Bug: Rename `AlterTableRecoverPartitionsCommand` to `RepairTableCommand`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Feb/21 09:01
Affects Version/s: 3.2.0

`AlterTableRecoverPartitionsCommand` is the execution node for the command `ALTER TABLE .. RECOVER PARTITIONS` which cannot drop/sync partitions. Since `ALTER TABLE .. RECOVER PARTITIONS` is a case of `MSCK REPAIR TABLE`, and `ALTER TABLE .. RECOVER PARTITIONS` does not support any options, it makes sense to rename `AlterTableRecoverPartitionsCommand` to `RepairTableCommand`.
Bug: JDK-8194653:  Deadlock involving FileSystems.getDefault and System.loadLibrary call
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 24/Feb/21 15:15
Affects Version/s: 2.4.7

h2. Instruction

This will cause deadlock and hangs concurrent tasks forever on the same executor. for example,
 
In the Spark UI stage tab, you may find some of the tasks hang for hours and all others complete without delay.
 
!screenshot-2.png! 

Also, you may find that these hanging tasks belong to the same executors.
Usually, in this case, you will also get nothing helpful from the executor log.

If you print the executor jstack or you check the ThreadDump via SparkUI executor tab and you find some task thread blocked like below, you are very likely to hit the JDK-8194653 issue.
!screenshot-1.png! 

h2. Solutions

Here are some options to circumvent this problem:

1. For the cluster managers side, you can update the JDK version according to https://bugs.openjdk.java.net/browse/JDK-8194653
2. If you are not able to update the JDK version for the cluster entirely, you can use `spark.executorEnv.JAVA_HOME` to specify a suitable JRE for your apps
2. Also, turn on `spark.speculation` may let spark automatically re-run the hanging tasks and bypass the problem
Bug: Spark 3.0.1 encounter parquet PageHerder IO issue
Status: Open | Priority: Major | Resolution: 
Created: 24/Feb/21 06:22
Affects Version/s: 3.0.1

{code:java}
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader: Required field 'uncompressed_page_size' was not found in serialized data! Struct: org.apache.parquet.format.PageHeader$PageHeaderStandardScheme@42a9002d
	at org.apache.parquet.format.Util.read(Util.java:216)
	at org.apache.parquet.format.Util.readPageHeader(Util.java:65)
	at org.apache.parquet.hadoop.ParquetFileReader$WorkaroundChunk.readPageHeader(ParquetFileReader.java:1064)
	at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:950)
	at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:807)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:313)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:268)
	at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:171)
	at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:173)
	at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
	at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:491)
	at 
{code}
Bug: Rename `map` to `mapAttr` in Catalyst DSL
Status: In Progress | Priority: Major | Resolution: 
Created: 20/Feb/21 15:01
Affects Version/s: 3.2.0

With the Catalyst DSL (dsl/package.scala), we have two ways to represent attributes.

1. Symbol literals (`'` syntax)
2. `$""` syntax which is defined in `sql/catalyst` module using string context.

But they have problems.

Regarding symbol literals, the scala community deprecates the symbol literals in Scala 2.13. We could alternatively use `Symbol` constructor but what is worse, Scala will completely remove `Symbol` in the future (https://scalacenter.github.io/scala-3-migration-guide/docs/incompatibilities/dropped-features.html).

{code}
Although scala.Symbol is useful for migration, beware that it is deprecated and that it will be removed from the scala-library. You are recommended, as a second step, to replace them with plain string literals "xwy" or a dedicated class.
{code}

Regarding `$""` syntax, this has two problems.
The first problem is that the syntax conflicts with another `$""` syntax defined in `sql/core` module.
You can easily see the problem with the Spark Shell.

{code}
import org.apache.spark.sql.catalyst.dsl.expressions._
val attr1 = $"attr1"

       error: type mismatch;
        found   : StringContext
        required: ?{def $: ?}
       Note that implicit conversions are not applicable because they are ambiguous:
        both method StringToColumn in class SQLImplicits of type (sc: StringContext): spark.implicits.StringToColumn
        and method StringToAttributeConversionHelper in trait ExpressionConversions of type (sc: StringContext): org.apache.spark.sql.catalyst.dsl.expressions.StringToAttributeConversionHelper
        are possible conversion functions from StringContext to ?{def $: ?}
{code}

The second problem is that we can't write like `$"attr".map(StringType, StringType)`, though we can write `'attr.map(StringType, StringType)`.
This seems to be a bug of the Scala compiler and will be fixed in neither `2.12` nor `2.13` (https://github.com/scala/scala/pull/7396).

Actually, I'm working on replacing all the symbol literals with `$""` syntax in SPARK-34443 and I found this problem in the following test code.

* EncoderResolutionSuite.scala
* ComplexTypeSuite.scala
* ObjectExpressionsSuite.scala
* NestedColumnAliasingSuite.scala
* ReplaceNullWithFalseInPredicateSuite.scala
* SimplifyCastsSuite.scala
* SimplifyConditionalSuite.scala

{code}
[error] /home/kou/work/oss/spark-scala-2.13/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/EncoderResolutionSuite.scala:212:28: too many arguments (found 2, expected 1) for method map: (f: org.apache.spark.sql.catalyst.expressions.Expression => A): Seq[A]
[error]       $"a".map(StringType, StringType)).foreach { attr =>
{code}

So, it's better to have another way to represent attributes with DSL.
Bug: OptimizeMetadataOnlyQuery should only apply for deterministic filters
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 10/Feb/21 07:25
Affects Version/s: 3.0.1

Similar to FileSourcePartitionPruning, OptimizeMetadataOnlyQuery should only apply for deterministic filters. If filters are non-deterministic, they have to be evaluated against partitions separately.
Bug: Kubernetes Spark Driver Pod Name Length Limitation
Status: Open | Priority: Major | Resolution: 
Created: 24/Feb/21 02:15
Affects Version/s: 3.0.0

Hi,

We are using Spark in Airflow with the k8s-master. Airflow is attaching to our spark-driver pod a unique id utilizing the k8s-subdomain convention '.'

This creates rather long pod-names. 

We noticed an issue with pod names in total (pod name + airflow attached uuid) exceeding 63 chars. Usually pod names can be up to 253 chars long. However Spark seems to have an issue with driver pod names which are longer than 63 characters.

In our case the driver pod name is exactly 65 chars long, but Spark is omitting the last 2 chars in its error message. I assume internally Spark is loosing those two characters. Reducing our Driver Pod Name to just 63 charts fixed the issue.

Here you can see the actual pod name (row 1) and the pod name from the Spark Error log (row 2)
{code:java}
ab-aaaaaa-bbbbbbbb-cccccc-dddddd.3s092032c69f4639adff835a826e0120
ab-aaaaaa-bbbbbbbb-cccccc-dddddd.3s092032c69f4639adff835a826e01{code}
{code:java}
[2021-02-20 00:30:06,289] {pod_launcher.py:136} INFO - Exception in thread "main" org.apache.spark.SparkException: No pod was found named Some(ab-aaaaaa-bbbbbbbb-cccccc-dddddd.3s092032c69f4639adff835a826e01) in the cluster in the namespace airflow-ns (this was supposed to be the driver pod.).{code}
Bug: Support of DELETE/UPDATE operation for Database (MySQL, Postgres etc.) level
Status: Open | Priority: Major | Resolution: 
Created: 23/Feb/21 03:12
Affects Version/s: 3.0.2

Spark 3.0+ SQL started supporting Delete operation. But still there is no way to downpush "Delete" operation to database (like MySQL, Postgres etc.)

In documents there is a mention of "deleteWhere(..)" but not sure whether it cant be used with JDBC data source. If yes, please help with some example.
Bug: Duplicate referenceNames are given for ambiguousReferences
Status: In Progress | Priority: Major | Resolution: 
Created: 19/Feb/21 22:17
Affects Version/s: 3.0.0

When running test with Spark extension that converts custom function to json path expression, I saw the following in test output:
{code}
2021-02-19 21:57:24,550 (Time-limited test) [INFO - org.yb.loadtest.TestSpark3Jsonb.testJsonb(TestSpark3Jsonb.java:102)] plan is == Physical Plan ==
org.apache.spark.sql.AnalysisException: Reference 'phone->'key'->1->'m'->2->>'b'' is ambiguous, could be: mycatalog.test.person.phone->'key'->1->'m'->2->>'b', mycatalog.test.person.phone->'key'->1->'m'->2->>'b'.; line 1 pos 8
{code}
Please note the candidates following 'could be' are the same.
Here is the physical plan for a working query where phone is a jsonb column:
{code}
TakeOrderedAndProject(limit=2, orderBy=[id#6 ASC NULLS FIRST], output=[id#6,address#7,key#0])
+- *(1) Project [id#6, address#7, phone->'key'->1->'m'->2->'b'#12 AS key#0]
   +- BatchScan[id#6, address#7, phone->'key'->1->'m'->2->'b'#12] Cassandra Scan: test.person
 - Cassandra Filters: [[phone->'key'->1->'m'->2->>'b' >= ?, 100]]
 - Requested Columns: [id,address,phone->'key'->1->'m'->2->'b']
{code}
The difference for the failed query is that it tries to use {code}phone->'key'->1->'m'->2->>'b'{code} in the projection (which works as part of filter).
Bug: Unify v1 and v2 ALTER TABLE .. RENAME tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Feb/21 08:34
Affects Version/s: 3.2.0

Extract ALTER TABLE .. RENAME tests to the common place to run them for V1 and v2 datasources. Some tests can be places to V1 and V2 specific test suites.
Bug: Fix v2 ALTER TABLE .. RENAME TO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/21 19:18
Affects Version/s: 3.2.0

The v2 `ALTER TABLE .. RENAME TO` command should rename a table in-place instead of moving it to the "root" namespace:

{code:scala}
sql("ALTER TABLE ns1.ns2.ns3.src_tbl RENAME TO dst_tbl")
sql(s"SHOW TABLES IN $catalog").show(false)

+---------+---------+-----------+
|namespace|tableName|isTemporary|
+---------+---------+-----------+
|         |dst_tbl  |false      |
+---------+---------+-----------+
{code}
Bug: Document DataStreamReader/Writer table APIs in Structured Streaming Programming Guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Feb/21 08:37
Affects Version/s: 3.1.1

We added APIs to enable read/write with tables in SPARK-32885, SPARK-32896 and SPARK-33836.

We need to update the Structured Streaming Programming Guide with the changes above.
Bug: Rename alter table exec nodes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Feb/21 17:33
Affects Version/s: 3.2.0

To be consistent with other exec nodes, rename:
* AlterTableAddPartitionExec -> AddPartitionExec
* AlterTableRenamePartitionExec -> RenamePartitionExec 
* AlterTableDropPartitionExec -> DropPartitionExec