av9ash/GitBugs huggingface dataset:
Bug: DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and support checkAnswer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 03:50
Affects Version/s: 3.1.0

*First*
DAGSchedulerSuite provides completeNextStageWithFetchFailure to make next stage occurs fetch failure.
But many test case uses complete directly as follows:

{code:java}
    complete(taskSets(2), Seq(
      (FetchFailed(BlockManagerId("hostA-exec2", "hostA", 12345),
        firstShuffleId, 0L, 0, 0, "ignored"),
        null)
    ))
{code}

We need to improve completeNextStageWithFetchFailure and reuse it.

*Second*
DAGSchedulerSuite also check the results show below:

{code:java}
complete(taskSets(0), Seq((Success, 42)))
assert(results === Map(0 -> 42))
{code}
We can extract it as a generic method of `checkAnswer`.
Bug: DAGSchedulerSuite: For the pattern of complete + assert, extract the general method
Status: Closed | Priority: Major | Resolution: Duplicate
Created: 28/May/20 03:32
Affects Version/s: 3.1.0

DAGSchedulerSuite exists many test case(e.g. run trivial job/run trivial job w/ dependency/cache location preferences w/ dependency) contains the pattern of complete + assert, such as:

{code:java}
  test("run trivial job") {
    submit(new MyRDD(sc, 1, Nil), Array(0))
    complete(taskSets(0), List((Success, 42)))
    assert(results === Map(0 -> 42))
    assertDataStructuresEmpty()
  }
{code}
We should extract the general method like checkAnswer(job, expected)
Bug: Return map output metadata from shuffle writers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/20 20:05
Affects Version/s: 3.0.0

Part of the overall sub-push for shuffle metadata management as proposed in [this document|https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit].

The shuffle plugin system needs a means for storing custom metadata for shuffle readers to eventually use. This task tracks the API changes required for the map output writer to return some such metadata back to the driver.
Bug: spark serde property path overwrites table property location
Status: In Progress | Priority: Major | Resolution: 
Created: 18/May/20 16:04
Affects Version/s: 2.3.1

This is an issue that have caused us so many data errors. 

1) using spark ( with hive context enabled )

{code}
df = spark.createDataFrame([{"a": "x", "b": "y", "c": "3"}])
df.write.format("orc").option("compression", "ZLIB").mode("overwrite").saveAsTable('test_spark');
{code}

 

2) from hive 

{code}
alter table test_spark rename to test_spark2
{code}

 

3)from spark-sql from command line ( note : not pyspark or spark-shell )  

{code}
select * from test_spark2
{code}

 

will give output 

{code}
NULL NULL NULL
Time taken: 0.334 seconds, Fetched 1 row(s)
{code}

 

This will throw NULL because , pyspark write API will add a serde property called path into the hive metastore. when hive renames the table , it do not understand this serde and hence keep it as it is. Now when spark-sql tries to read it , it will honor the serde property first and then tries to read from the non-existent hdfs location. If it had given an error , then also it would have been fine , but throwing out NULL will cause applications to fail pretty bad. Spark claims to support hive tables , hence it should respect hive metastore location property rather than spark serde property when trying to read a table. This cannot be classified as a expected behaviour.
Bug: CrossValidator supports user-supplied folds 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/20 18:07
Affects Version/s: 3.1.0

As a user, I can specify how CrossValidator should create folds by specifying a foldCol, which should be integer type with range [0, numFolds). If foldCol is specified, Spark won't do random k-fold split. This is useful if there are custom logics to create folds, e.g., random split by users instead of random splits of events.

This is similar to SPARK-16206, which is for the RDD-based APIs.
Bug: DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 09:34
Affects Version/s: 3.1.0

DAGSchedulerSuite provides completeShuffleMapStageSuccessfully to make ShuffleMapStage successfully.
But many test case uses complete directly as follows:
complete(taskSets(0), Seq((Success, makeMapStatus("hostA", 1))))
We need to improve completeShuffleMapStageSuccessfully and reuse it.
Bug: Make Prometheus output consistent with DropWizard 4.1 result
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/May/20 04:21
Affects Version/s: 3.0.0

SPARK-29032 adds Prometheus support.

After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes difference in output labels and number of keys.

 

This issue aims to fix this inconsistency in Spark.
Bug: Thriftserver collecting timestamp not using spark.sql.session.timeZone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 19:08
Affects Version/s: 3.0.0

If JDBC client is in TimeZone PST, and sets spark.sql.session.timeZone to PST, and sends a query "SELECT timestamp '2020-05-20 12:00:00'", and the JVM timezone of the Spark cluster is e.g. CET, then
- the timestamp literal in the query is interpreted as 12:00:00 PST, i.e. 21:00:00 CET
- but currently when it's returned, the timestamps are collected from the query with a collect() in https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala#L299, and then in the end Timestamps are turned into strings using a t.toString() in https://github.com/apache/spark/blob/master/sql/hive-thriftserver/v2.3/src/main/java/org/apache/hive/service/cli/ColumnValue.java#L138 This will use the Spark cluster TimeZone. That results in "21:00:00" returned to the JDBC application.
Bug: Stats computed for HiveTable not used
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 31/May/20 17:53
Affects Version/s: 3.0.0

When fixing SPAK-31850, I made the following observation

As part of DetermineTableStats rule, we compute the stats for a HiveTableRelation.

In most case(the flag for converting Parquet and Orc table is enabled by default), RelationConversion rule converts the HiveTableRelation to LogicalRelation.

The stats computed as part of Hive Table relation does not get used.

The stats compute should be avoided if the Hive Tbale is going to be converted to data source table
Bug: Create date formatter only once in HiveResult
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/20 20:09
Affects Version/s: 3.0.0

Currently, dateFormatter in HiveResult is created per every date value to be sure that catches zone id but this is not necessary because date formatter doesn't depend on any time zone. Time zone is needed only for parsing. So, dateFormatter can be created only once. The ticket aims to change *def dateFormatter* to *val dateFormatter*.
Bug: Grouping sets with having clause returns the wrong result
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/May/20 16:24
Affects Version/s: 2.0.2

Grouping sets with having clause returns the wrong result when the condition of having contained conflicting naming. See the below example:
{code:java}
select sum(a) as b FROM VALUES (1, 10), (2, 20) AS T(a, b) group by GROUPING SETS ((b), (a, b)) having b > 10{code}
The `b` in `having b > 10` should be resolved as `T.b` not `sum(a)`, so the right result should be
{code:java}
+---+
|  b|
+---+
|  2|
|  2|
+---+{code}
instead of an empty result.

The root cause is similar to SPARK-31519, it's caused by we parsed HAVING as Filter(..., Agg(...)) and resolved these two parts in different rules. The CUBE and ROLLUP have the same issue.

Other systems worked as expected, I checked PostgreSQL 9.6 and MS SQL Server 2017.

 

For Apache Spark 2.0.2 ~ 2.3.4, the following query is tested.
{code:java}
spark-sql> select sum(a) as b from t group by b grouping sets(b) having b > 10;
Time taken: 0.194 seconds

hive> select sum(a) as b from t group by b grouping sets(b) having b > 10;
2
Time taken: 1.605 seconds, Fetched: 1 row(s) {code}
Bug: Shift to the new highest locality level if there is when recomputeLocality
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/May/20 14:53
Affects Version/s: 3.1.0

In the new version of delay scheduling, if a task set is submitted before any executors are added to TaskScheduler, the task set will always schedule tasks at ANY locality level. We should improve this corner case.
Bug: Spark to support the auto cancelation of delegation token when an Application completes
Status: Open | Priority: Major | Resolution: 
Created: 25/May/20 08:14
Affects Version/s: 2.4.5

*Context* :
YARN application provides a client API [setCancelTokensWhenComplete|http://hadoop.apache.org/docs/r2.8.0/hadoop-yarn/hadoop-yarn-api/apidocs/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setCancelTokensWhenComplete(boolean)] to manage the  delegation token(DT) lifecycle. By default, YARN [cancels the DT|https://github.com/apache/hadoop/blob/8f78aeb2500011e568929b585ed5b0987355f88d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto#L513] when App finishes. However, the user can override this NOT to cancel the DT after the App completes. In some instances, this is required to lessen the HDFS/KMS memory footprints by reducing the outstanding DTs.
MR and TEZ already allow that through client config  such  as _mapreduce.job.complete.cancel.delegation.tokens_ and _tez.cancel.delegation.tokens.on.completion_ respectively_._

*Proposal* :
Currently, Spark doesn't support it. However, we may need to manage the lifecycle of DT outside YARN/Spark framework.
Bug: Use `FastDateFormat` as the legacy fractional formatter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/May/20 21:21
Affects Version/s: 3.1.0

By default {{HiveResult}}.{{hiveResultString}} retrieves timestamps values as instances of {{java.sql.Timestamp}}, and uses the legacy parser {{SimpleDateFormat}} to convert the timestamps to strings. After the fix [#28024|https://github.com/apache/spark/pull/28024], the fractional formatter and its companion - legacy formatter {{SimpleDateFormat}} are created per every value. By switching from {{LegacySimpleTimestampFormatter}} to {{LegacyFastTimestampFormatter}}, we can utilize the internal cache of {{FastDateFormat}}, and avoid parsing the default pattern {{yyyy-MM-dd HH:mm:ss}}.
Bug: NotNullSafe to get complementary set
Status: Open | Priority: Major | Resolution: 
Created: 30/May/20 16:00
Affects Version/s: 2.3.0

If we have a filter expression to get subset of rows, and then we want the complementary set, Not(expression) cannot work, since Not is NullIntolerent, if expression.eval(row) is null, filter predicate is false too for Not(expression), "row" will not appear in both subset and complementary set.

So, maybe we need a NotNullSafe implementation to get the complementary set which will result in true if the expression.eval(row) is null.
Bug: Thriftserver not setting active SparkSession, SQLConf.get not getting session configs correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 21:31
Affects Version/s: 3.0.0

Thriftserver is not setting the active SparkSession.
Because of that, configuration obtained with SQLConf.get is not the session configuration.
This makes many configs set by "set" in the session not work correctly.
Bug: Redefining struct inside array incorrectly wraps child fields in array
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 20/May/20 21:23
Affects Version/s: 2.4.5

It seems that redefining a {{struct}} for the purpose of removing a sub-field, when that {{struct}} is itself inside an {{array}}, results in the remaining (non-removed) {{struct}} fields themselves being incorrectly wrapped in an array.

For more context, see [this|https://stackoverflow.com/a/46084983/375670] StackOverflow answer and discussion thread.  I have debugged this code and distilled it down to what I believe represents a bug in Spark itself.

Consider the following {{spark-shell}} session (version 2.4.5):

{code}
// use a nested JSON structure that contains a struct inside an array
val jsonData = """{
  "foo": "bar",
  "top": {
    "child1": 5,
    "child2": [
      {
        "child2First": "one",
        "child2Second": 2
      }
    ]
  }
}"""

// read into a DataFrame
val df = spark.read.option("multiline", "true").json(Seq(jsonData).toDS())

// create a new definition for "top", which will remove the "top.child2.child2First" column

val newTop = struct(df("top").getField("child1").alias("child1"), array(struct(df("top").getField("child2").getField("child2Second").alias("child2Second"))).alias("child2"))

// show the schema before and after swapping out the struct definition
df.schema.toDDL
// `foo` STRING,`top` STRUCT<`child1`: BIGINT, `child2`: ARRAY<STRUCT<`child2First`: STRING, `child2Second`: BIGINT>>>
df.withColumn("top", newTop).schema.toDDL
// `foo` STRING,`top` STRUCT<`child1`: BIGINT, `child2`: ARRAY<STRUCT<`child2Second`: ARRAY<BIGINT>>>>
{code}

Notice in this case that the new definition for {{top.child2.child2Second}} is an {{ARRAY<BIGINT>}}.  This is incorrect; it should simply be {{BIGINT}}.  There is nothing in the definition of the {{newTop}} {{struct}} that should have caused the type to become wrapped in an array like this.
Bug: Check reading date/timestamp from Avro files w/ and w/o Spark version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 08:55
Affects Version/s: 3.0.0

Extend the test "SPARK-31183: compatibility with Spark 2.4 in reading dates/timestamps" in AvroSuite and add checks for files:
# With Spark version in meta info, and
# Without Spark version
Bug: Handle locality wait reset better when executors added
Status: Open | Priority: Major | Resolution: 
Created: 28/May/20 14:26
Affects Version/s: 3.1.0

We found a corner case with the new locality algorithm where we were not affectively resetting the locality when you started without executors.  See [https://github.com/apache/spark/pull/28656/]

In that fix we reset the locality when a new executor is added and it changes the valid locality levels.  This could reset it when you don't want to. for instance if the task set has been around for quite a while and you have other executors it could put tasks on. Resetting it back to the most local locality level could cause delays in the tasks getting scheduled that have been waiting for more then the locality wait. 

This goes back to the new algorithm tries to use heuristic on this but its not perfect.  Ideally I think we would still track per executor but that is a lot more change.

Investigate to see if we can handle this better.
Bug: Check reading date/timestamp from Parquet: plain, dictionary enc, Spark version
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/May/20 14:42
Affects Version/s: 3.0.0

Extend the test "SPARK-31159: compatibility with Spark 2.4 in reading dates/timestamps" in ParquetIOSuite and add checks for:
# Dictionary encoding
# Without Spark version in meta info
Bug: DetermineTableStats rules computes stats multiple time for same table
Status: In Progress | Priority: Major | Resolution: 
Created: 28/May/20 05:54
Affects Version/s: 3.0.0

DetermineTableStats rules computes stats multiple time for same table.

There are few rules which invoke

org.apache.spark.sql.catalyst.analysis.Analyzer#executeSameContext
 * The above method cause the logical plan to go through the analysis phase multiple times
 * The above method is invoked as a part of rule(s) which is part of a batch which run till *Fixed point* which implies that that the analysis phase can run multiple time
 * example: [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2138 |https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L2138]
 * This is cause of concern especially in *DetermineTableStats* rule, where the computation of stats can be expensive for a large table(And also it happens multiple time due to the fixed point nature of batch from which analysis is triggered)

 

Repro steps
{code:java}
 
spark.sql("create table c(id INT, name STRING) STORED AS PARQUET")
val df = spark.sql("select count(id) id from c group by name order by id " )
df.queryExecution.analyzed
{code}
  Note: 
 * There is no log line in DetermineTableStats to indicate that stats compute happened. Need to add a log line or use a debugger
 * The above can be repro-ed with first query on a created table.
Bug: The streaming output mode validator didn't treat union well
Status: Open | Priority: Major | Resolution: 
Created: 27/May/20 14:57
Affects Version/s: 3.0.0

{{UnsupportedOperationChecker.checkForStreaming}} didn't treat the union operator well (it only assumes a linear query plan). To temporarily fix the broken semantics before we have a complete improvement plan for the output mode, the validator should do the following.

1. Allow multiple aggregations from different branches of a union operator.
2. For complete output mode, disallow a union on streams with and without aggregations.
Bug: Refactor JoinSelection
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 09:56
Affects Version/s: 3.0.0

This PR extracts the logic for selecting the planned join type out of the `JoinSelection` rule and moves it to `JoinSelectionHelper` in Catalyst. This change both cleans up the code in `JoinSelection` and allows the logic to be in one place and be used from other rules that need to make decision based on the join type before the planning time.
Bug:  DataSourceV2 unexpected behavior with partition data distribution
Status: Open | Priority: Major | Resolution: 
Created: 15/May/20 08:57
Affects Version/s: 2.4.0

Hi team,
  
 We are using DataSourceV2.
  
 We have a queston regarding using interface org.apache.spark.sql.sources.v2.writer.DataWriter<T>
  
 We have faced with following unexpected behavior.
 When we use a repartion on dataframe we expect that for each partion Spark will create new instance of DataWriter interface and sends the repartition data to appropriate instances but sometimes we observe that Spark sends the data from different partitions to the same instance of DataWriter interface.
 It behavior sometimes occures on Yarn cluster.
  
 If we run Spark job as Local run Spark really creates a new instance of DataWriter interface for each partiion after repartion and publishes the repartion data to appropriate instances.
  

Possible there is a Spark limit a number of  DataWriter instances?


 Can you explain it is a bug or expected behavior?
Bug: Check for partition existence for Insert overwrite if not exists queries on Hive Serde Table before computation
Status: In Progress | Priority: Major | Resolution: 
Created: 27/May/20 02:44
Affects Version/s: 2.1.3

If T is a Hive table, Query: INSERT OVERWRITE table T partition(p='existing') IF NOT EXISTS select ... ; executes job/computation on Spark and then avoids loading partitions. It should avoid doing the wasteful computation and exit early.

For Datasource table, it does avoid the computation and exits early (due to work done in SPARK-20831).
Bug: Cost too much resources when read orc hive table for infer schema
Status: Open | Priority: Major | Resolution: 
Created: 26/May/20 09:23
Affects Version/s: 2.4.3

When read a hive orc partitioned table without spark schema properties , spark read all partitions and all files for infer schema. 

Other settings: native orc mode ; _convertMetastoreOrc = true._

 

And I think it can improved by pass  *_partitionFilters_* to *_fileIndex.listFiles_*.
{code:java}
// code placeholder
// org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:238
val inferredSchema = fileFormat
  .inferSchema(
    sparkSession,
    options,
    fileIndex.listFiles(Nil, Nil).flatMap(_.files))
  .map(mergeWithMetastoreSchema(relation.tableMeta.dataSchema, _))

{code}
Bug: Flaky JavaBeanDeserializationSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 06:43
Affects Version/s: 3.1.0

The test suite JavaBeanDeserializationSuite sometimes fails with:
{code}
sbt.ForkMain$ForkError: java.lang.AssertionError: expected:<[JavaBeanDeserializationSuite.RecordSpark22000[shortField=0,intField=0,longField=0,floatField=0.0,doubleField=0.0,stringField=0,booleanField=true,timestampField=2020-05-25 12:39:16.999,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=1,intField=1,longField=1,floatField=1.0,doubleField=1.0,stringField=1,booleanField=false,timestampField=2020-05-25 12:39:17.0,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=2,intField=2,longField=2,floatField=2.0,doubleField=2.0,stringField=2,booleanField=true,timestampField=2020-05-25 12:39:17.0,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=3,intField=3,longField=3,floatField=3.0,doubleField=3.0,stringField=3,booleanField=false,timestampField=2020-05-25 12:39:17.0,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=4,intField=4,longField=4,floatField=4.0,doubleField=4.0,stringField=4,booleanField=true,timestampField=2020-05-25 12:39:17.0,nullIntField=<null>]]> but was:<[JavaBeanDeserializationSuite.RecordSpark22000[shortField=0,intField=0,longField=0,floatField=0.0,doubleField=0.0,stringField=0,booleanField=true,timestampField=2020-05-25 12:39:16.999,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=1,intField=1,longField=1,floatField=1.0,doubleField=1.0,stringField=1,booleanField=false,timestampField=2020-05-25 12:39:17,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=2,intField=2,longField=2,floatField=2.0,doubleField=2.0,stringField=2,booleanField=true,timestampField=2020-05-25 12:39:17,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=3,intField=3,longField=3,floatField=3.0,doubleField=3.0,stringField=3,booleanField=false,timestampField=2020-05-25 12:39:17,nullIntField=<null>], JavaBeanDeserializationSuite.RecordSpark22000[shortField=4,intField=4,longField=4,floatField=4.0,doubleField=4.0,stringField=4,booleanField=true,timestampField=2020-05-25 12:39:17,nullIntField=<null>]]>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:144)
	at test.org.apache.spark.sql.JavaBeanDeserializationSuite.testSpark22000(JavaBeanDeserializationSuite.java:165)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}

See https://github.com/apache/spark/pull/28630#issuecomment-633695723
Bug: Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/May/20 09:09
Affects Version/s: 3.0.0

Five continuous pattern characters with 'G/M/L/E/u/Q/q' means Narrow-Text Style in java.time.DateTimeFormatterBuilder which output the leading single letter of the value, e.g. `December` would be `D`,  while in Spark 2.4 they means Full-Text Style.