av9ash/GitBugs huggingface dataset:
Bug: ConcurrentAppendException while updating delta lake table
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 23/Oct/20 20:35
Affects Version/s: 3.0.0

I have two Spark Streaming job run concurrently. 
 * Stream join Job: join in Kafka Stream with another stream from Amazon SQS. The result will be appended into delta lake table A
 * Upsert job: read from one Delta lake table B and update table A when there are matching IDs. 

Environment:
 * Databricks cloud run time 7.2, Spark 3.0.0

 

The stream join job works fine but the Upsert job kept failing. 

 

Stack trace:

com.databricks.sql.transaction.tahoe.ConcurrentAppendException: Files were added to partition [dt=2020-xx-yy, request_hour=2020-xx-yy 23:00:00] by a concurrent update.

Please try the operation again. Conflicting commit: {"timestamp":1603477588946,"userId":"xxxxx","operation":"OPTIMIZE","operationParameters":

{"predicate":[],"zOrderBy":[],"batchId":0,"auto":true}

,"job":\{"jobId":"xxxxx","jobName":"Streaming join xxx","runId":"xxx","jobOwnerId":"xxxx","triggerType":"manual"},"notebook":\{"notebookId":"xxxx"},"clusterId":"xxxxxx","readVersion":2222222222,"isolationLevel":"SnapshotIsolation","isBlindAppend":false,"operationMetrics":\{"numRemovedFiles":"444444","numRemovedBytes":"64455537","p25FileSize":"63341646","minFileSize":"63341646","numAddedFiles":"1","maxFileSize":"63341646","p75FileSize":"63341646","p50FileSize":"63341646","numAddedBytes":"63341646"}}

 

Table A have these setting: 

- 'delta.isolationLevel' = 'WriteSerializable'

- spark.databricks.delta.optimizeWrite.enable = True

- spark.databricks.delta.autoCompact.enabled = True

 

Other settings:

spark.databricks.io.cache.compression.enabled true

stateStore = rocksdb

spark.sql.adaptive.enabled true

spark.sql.adaptive.skewJoin.enabled true

 

I already set IsolationLevel to WriteSerializable to handle ConcurrentAppendingException as described in [https://docs.databricks.com/delta/optimizations/isolation-level.html]  and [https://docs.databricks.com/delta/concurrency-control.html] 

 

However the error says "SnapshotIsolation". I didn't have any Optimize operation running on the target table. 

 

What did I miss?
Bug: Improve examples and scenarios in docstrings 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Oct/20 00:47
Affects Version/s: 3.1.0

Currently, PySpark documentation does not have a lot of examples and scenarios. See also https://github.com/apache/spark/pull/30149#issuecomment-716490037.

We should add/improve examples especially in the commonly used APIs. For example, {{Column}}, {{DataFrame}}. {{RDD}}, {{SparkContext}}, etc.

This umbrella JIRA targets to improve them in commonly used APIs.

NOTE that we'll have to convert the docstrings into numpydoc style first in a separate PR (at SPARK-32085), and then add examples. In this way, we can manage migration to numpydoc and example improvement here separately (e.g., reverting numpydoc migration only).
Bug: Enhance ExecutorPlugin API to include methods for task start and end events
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/20 18:49
Affects Version/s: 3.1.0

On [SPARK-24918|https://issues.apache.org/jira/browse/SPARK-24918]'s [SIPP|https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/view#|https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit#], it was raised to potentially add methods to ExecutorPlugin interface on task start and end:

{quote}The basic interface can just be a marker trait, as that allows a plugin to monitor general characteristics of the JVM (eg. monitor memory or take thread dumps).   Optionally, we could include methods for task start and end events.   This would allow more control on monitoring – eg., you could start polling thread dumps only if there was a task from a particular stage that had been taking too long. But anything task related is a bit trickier to decide the right api. Should the task end event also get the failure reason? Should those events get called in the same thread as the task runner, or in another thread?
{quote}

The ask is to add exactly that. I've put up a draft PR [in our fork of spark|https://github.com/palantir/spark/pull/713] and I'm happy to push it upstream. Also happy to receive comments on what's the right interface to expose - not opinionated on that front, tried to expose the simplest interface for now.

The main reason for this ask is to propagate tracing information from the driver to the executors ([SPARK-21962|https://issues.apache.org/jira/browse/SPARK-21962] has some context). On [HADOOP-15566|https://issues.apache.org/jira/browse/HADOOP-15566] I see we're discussing how to add tracing to the Apache ecosystem, but my problem is slightly different: I want to use this interface to propagate tracing information to my framework of choice. If the Hadoop issue gets solved we'll have a framework to communicate tracing information inside the Apache ecosystem, but it's highly unlikely that all Spark users will use the same common framework. Therefore we should still provide plugin interfaces where the tracing information can be propagated appropriately.

To give more color, in our case the tracing information is [stored in a thread local|https://github.com/palantir/tracing-java/blob/4.9.0/tracing/src/main/java/com/palantir/tracing/Tracer.java#L61], therefore it needs to be set in the same thread which is executing the task. [*]

While our framework is specific, I imagine such an interface could be useful in general. Happy to hear your thoughts about it.

[*] Something I did not mention was how to propagate the tracing information from the driver to the executors. For that I intend to use 1. the driver's localProperties, which 2. will be eventually propagated to the executors' TaskContext, which 3. I'll be able to access from the methods above.
Bug: Handling nullability for complex types is broken during resolution of V2 write command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/20 22:04
Affects Version/s: 3.0.0

I figured out Spark 3.x cannot write to complex type with nullable if matching column type in DataFrame is non-nullable.

For example, 
{code:java}
case class StructData(a: String, b: Int)

case class Data(col_b: Boolean, col_i: Int, col_l: Long, col_f: Float, col_d: Double, col_s: String, col_fi: Array[Byte], col_bi: Array[Byte], col_de: Double, col_st: StructData, col_li: Seq[String], col_ma: Map[Int, String]){code}
`col_st.b` would be non-nullable in DataFrame, which should not matter when we insert from DataFrame to the table which has `col_st.b` as nullable. (non-nullable to nullable should be possible)

This looks to be broken in V2 write command.
Bug: Spark Shuffle Index Cache calculates memory usage wrong
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 13:10
Affects Version/s: 2.4.0

SPARK-21501 changed the spark shuffle index service to be based on memory instead of the number of files.

Unfortunately, there's a problem with the calculation which is based on size information provided by `ShuffleIndexInformation`.

It is based purely on the file size of the cached file on disk.

We're running in OOMs with very small index files (byte size ~16 bytes) but the overhead of the ShuffleIndexInformation around this is much larger (e.g. 184 bytes, see screenshot). We need to take this into account and should probably add a fixed overhead of somewhere between 152 and 180 bytes according to my tests. I'm not 100% sure what the correct number is and it'll also depend on the architecture etc. so we can't be exact anyway.

If we do that we can maybe get rid of the size field in ShuffleIndexInformation to save a few more bytes per entry.

In effect this means that for small files we use up about 70-100 times as much memory as we intend to. Our NodeManagers OOM with 4GB and more of indexShuffleCache.
Bug: Add built-in UDF - GETBIT 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Oct/20 13:48
Affects Version/s: 3.1.0

Teradata, Impala, Snowflake and Yellowbrick support this function:

https://docs.teradata.com/reader/kmuOwjp1zEYg98JsB8fu_A/PK1oV1b2jqvG~ohRnOro9w
https://docs.cloudera.com/runtime/7.2.0/impala-sql-reference/topics/impala-bit-functions.html#bit_functions__getbit
https://docs.snowflake.com/en/sql-reference/functions/getbit.html
https://www.yellowbrick.com/docs/2.2/ybd_sqlref/getbit.html
Bug: Replace origin GROUPING SETS with new expression
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 31/Oct/20 03:47
Affects Version/s: 3.1.0

Replace current GroupingSets just using expression

{code}
case class GroupingSets(
    selectedGroupByExprs: Seq[Seq[Expression]],
    groupByExprs: Seq[Expression],
    child: LogicalPlan,
    aggregations: Seq[NamedExpression]) extends UnaryNode {

  override def output: Seq[Attribute] = aggregations.map(_.toAttribute)

  // Needs to be unresolved before its translated to Aggregate + Expand because output attributes
  // will change in analysis.
  override lazy val resolved: Boolean = false
}
{code}
Bug: Need to support partial CUBE/ROLLUP/GROUPING SETS and mixed case
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Oct/20 13:34
Affects Version/s: 3.1.0

How to reproduce this issue:

{code:sql}
create table test_cube using parquet as select id as a, id as b, id as c from range(10);
select a, b, c, count(*) from test_cube group by 1, cube(2, 3);
{code}



{noformat}
spark-sql> select a, b, c, count(*) from test_cube group by 1, cube(2, 3);
20/10/23 06:31:51 ERROR SparkSQLDriver: Failed in [select a, b, c, count(*) from test_cube group by 1, cube(2, 3)]
java.lang.UnsupportedOperationException
	at org.apache.spark.sql.catalyst.expressions.GroupingSet.dataType(grouping.scala:35)
	at org.apache.spark.sql.catalyst.expressions.GroupingSet.dataType$(grouping.scala:35)
	at org.apache.spark.sql.catalyst.expressions.Cube.dataType(grouping.scala:60)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkValidGroupingExprs$1(CheckAnalysis.scala:268)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$12(CheckAnalysis.scala:284)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$12$adapted(CheckAnalysis.scala:284)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:284)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:92)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:177)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:92)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:89)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:130)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:156)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:153)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:133)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
	at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:133)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
{noformat}
Bug: Explicit or implicit casts involving partition columns can sometimes result in a MetaException.
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 09/Oct/20 03:06
Affects Version/s: 3.1.0

The following cases throw
{{MetaException(message:Filtering is supported only on partition keys of type string)}}
{noformat}
sql("create table test (a int) partitioned by (b int) stored as parquet")
sql("insert into test values (1, 1), (1, 2), (2, 2)")

// These throw MetaExceptions
sql("select * from test where b in ('2')").show(false)
sql("select * from test where cast(b as string) = '2'").show(false)
sql("select * from test where cast(b as string) in ('2')").show(false)
sql("select * from test where cast(b as string) in (2)").show(false)
sql("select cast(b as string) as b from test where b in ('2')").show(false)
sql("select cast(b as string) as b from test").filter("b = '2'").show(false) // [1]
sql("select cast(b as string) as b from test").filter("b in (2)").show(false) // [2]
sql("select cast(b as string) as b from test").filter("b in ('2')").show(false)
sql("select * from test where cast(b as string) > '1'").show(false)
sql("select cast(b as string) b from test").filter("b > '1'").show(false) // [3]

// [1] but not sql("select cast(b as string) as b from test where b = '2'").show(false)
// [2] but not sql("select cast(b as string) as b from test where b in (2)").show(false)
// [3] but not sql("select cast(b as string) b from test where b > '1'").show(false)
{noformat}
The message ("Filtering is supported only on partition keys of type string") is misleading. Filter *is* supported on integer columns, for example.
Bug: Add metadata in MapStatus to support custom shuffle manager
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Oct/20 05:40
Affects Version/s: 3.0.1

Current MapStatus class is tightly bound with local (sort merge) shuffle which uses BlockManagerId to store the shuffle data location. It could not support other custom shuffle manager implementation.

We could add "metadata" to MapStatus and allow different shuffle manager implementation to store information related to them.
Bug: TimezoneID is needed when there cast from Date to String
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Oct/20 01:46
Affects Version/s: 3.0.0

A simple way to reproduce this is 

{code}
spark-shell --conf spark.sql.legacy.typeCoercion.datetimeToString.enabled

scala> sql("""

select a.d1 from
 (select to_date(concat('2000-01-0', id)) as d1 from range(1, 2)) a
 join
 (select concat('2000-01-0', id) as d2 from range(1, 2)) b
 on a.d1 = b.d2

""").show
{code}

 

it will throw

{code}

java.util.NoSuchElementException: None.get
 at scala.None$.get(Option.scala:529)
 at scala.None$.get(Option.scala:527)
 at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:56)
 at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:56)
 at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId$lzycompute(Cast.scala:253)
 at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId(Cast.scala:253)
 at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter$lzycompute(Cast.scala:287)
 at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter(Cast.scala:287)

{code}
Bug: Make database versions up-to-date for integration tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Oct/20 05:46
Affects Version/s: 3.1.0

This ticket aims at updating database versions below for integration tests;

 - ibmcom/db2:11.5.0.0a => ibmcom/db2:11.5.4.0 in DB2[Krb]IntegrationSuite
 - mysql:5.7.28 => mysql:5.7.31 in MySQLIntegrationSuite
 - postgres:12.0 => postgres:13.0 in Postgres[Krb]IntegrationSuite
Bug: Set the rebasing mode for parquet INT96 type to `EXCEPTION` by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 20:06
Affects Version/s: 3.1.0

The ticket aims to set the following SQL configs:
- spark.sql.legacy.parquet.int96RebaseModeInWrite
- spark.sql.legacy.parquet.int96RebaseModeInRead
to EXCEPTION by default.

The reason is let users to decide should Spark modify loaded/saved timestamps instead of silently shifting timestamps while rebasing.
Bug: protect setActiveSession and clearActiveSession
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 14/Oct/20 02:01
Affects Version/s: 3.1.0

This PR is a sub-task of [SPARK-33138](https://issues.apache.org/jira/browse/SPARK-33138). In order to make SQLConf.get reliable and stable, we need to make sure user can't pollute the SQLConf and SparkSession Context via calling setActiveSession and clearActiveSession.

Change of the PR:

* add legacy config spark.sql.legacy.allowModifyActiveSession to fallback to old behavior if user do need to call these two API.
* by default, if user call these two API, it will throw exception
* add extra two internal and private API setActiveSessionInternal and clearActiveSessionInternal for current internal usage
* change all internal reference to new internal API exception for SQLContext.setActive and SQLContext.clearActive
Bug: Reduce the number of tasks launched after bucket pruning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Oct/20 13:20
Affects Version/s: 3.1.0

We only need to read 1 bucket, but it still launch 200 tasks.
{code:sql}
create table test_bucket using parquet clustered by (ID) sorted by (ID) into 200 buckets AS (SELECT id FROM range(1000) cluster by id)
spark-sql> explain select * from test_bucket where id = 4;
== Physical Plan ==
*(1) Project [id#7L]
+- *(1) Filter (isnotnull(id#7L) AND (id#7L = 4))
   +- *(1) ColumnarToRow
      +- FileScan parquet default.test_bucket[id#7L] Batched: true, DataFilters: [isnotnull(id#7L), (id#7L = 4)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.1-bin-hadoop3.2/spark-warehouse/test_bucket], PartitionFilters: [], PushedFilters: [IsNotNull(id), EqualTo(id,4)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 1 out of 200
{code}
Bug: Improve semantics for REFRESH TABLE
Status: Open | Priority: Major | Resolution: 
Created: 31/Oct/20 19:11
Affects Version/s: 3.0.1

Currently, the semantics for {{REFRESH TABLE t}} is not well defined for view (let's say {{view}}) that reference the table {{t}}:

1. If {{view}} is cached, the behavior is not well-defined. Should Spark invalidate the cache (current behavior) or recache it?
2. If {{view}} is a temporary view, currently refreshing {{t}} does not refresh {{view}} since it will just reuse the logical plan defined in the session catalog. This could lead query failures (although with a helpful error message) or to incorrect results depending on the refresh behavior.

I think we should clear define and document the behavior here, so that users won't get confused.
Bug: Adds a group tag in all the expressions for built-in functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/20 23:27
Affects Version/s: 3.2.0

We've added the script to automatically generate documents for built-in functions in SPARK-31429. To generate docs, we need to add a `group` tag in `ExpressionDescription`. Currently, a part of built-in functions has the tag, so we need to finish adding it to all the builtin-functions for better documentations.

 This ticket comes from the talk in https://github.com/apache/spark/pull/28224#issuecomment-707195753
Bug: Fix Flaky Test "SPARK-33088: executor failed tasks trigger plugin calls"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Oct/20 00:36
Affects Version/s: 3.1.0

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/lastCompletedBuild/testReport/org.apache.spark.internal.plugin/PluginContainerSuite/SPARK_33088__executor_failed_tasks_trigger_plugin_calls/

{code}
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 1 did not equal 2
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
	at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
	at org.apache.spark.internal.plugin.PluginContainerSuite.$anonfun$new$8(PluginContainerSuite.scala:161)
{code}
Bug: Remove two Hive 1.2-related Jenkins jobs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/20 22:37
Affects Version/s: 3.1.0

SPARK-20202 removed `hive-1.2` profile at Apache Spark 3.1.0 and excluded the following two from `Spark QA Dashboard`. We need to remove it.
- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-1.2/

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-hive-1.2/
Bug: YARN: Print direct links to driver logs alongside application report in cluster mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Oct/20 22:56
Affects Version/s: 3.0.1

Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}} will print out the application report into the logs, to be easily viewed by users. For example:
{code}
INFO yarn.Client: 
 	 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
 	 diagnostics: N/A
 	 ApplicationMaster host: X.X.X.X
 	 ApplicationMaster RPC port: 0
 	 queue: default
 	 start time: 1602782566027
 	 final status: UNDEFINED
 	 tracking URL: http://hostname:8888/proxy/application_<id>/
 	 user: xkrogen
{code}
Typically, the tracking URL can be used to find the logs of the ApplicationMaster/driver while the application is running. Later, the Spark History Server can be used to track this information down, using the stdout/stderr links on the Executors page.

However, in the situation when the driver crashed _before_ writing out a history file, the SHS may not be aware of this application, and thus does not contain links to the driver logs. When this situation arises, it can be difficult for users to debug further, since they can't easily find their driver logs.

It is possible to reach the logs by using the {{yarn logs}} commands, but the average Spark user isn't aware of this and shouldn't have to be.

I propose adding, alongside the application report, some additional lines like:
{code}
         Driver Logs (stdout): http://hostname:8042/node/containerlogs/container_<id>/xkrogen/stdout?start=-4096
         Driver Logs (stderr): http://hostname:8042/node/containerlogs/container_<id>/xkrogen/stderr?start=-4096
{code}
With this information available, users can quickly jump to their driver logs, even if it crashed before the SHS became aware of the application. This has the additional benefit of providing a quick way to access driver logs, which often contain useful information, in a single click (instead of navigating through the Spark UI).