av9ash/GitBugs huggingface dataset:
Bug: Connot insert overwite different partition with same table
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Sep/20 03:54
Affects Version/s: 3.0.0

When:
{code:java}
CREATE TABLE tmp.spark3_snap (
id string
)
PARTITIONED BY (dt string)
STORED AS ORC
;

insert overwrite table tmp.spark3_snap partition(dt='2020-09-09')
select 10;
insert overwrite table tmp.spark3_snap partition(dt='2020-09-10')
select 1;

insert overwrite table tmp.spark3_snap partition(dt='2020-09-10')
select id from tmp.spark3_snap where dt='2020-09-09';
{code}
and it will be get a error: "Cannot overwrite a path that is also being read from"

related: https://issues.apache.org/jira/browse/SPARK-24194

This work on spark 2.4.3 and do not work on spark 3.0.0
Bug: Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:41
Affects Version/s: 3.1.0

With push-based shuffle, we are currently decoupling map task executions from the shuffle block push process. Thus, when all map tasks finish, we might want to wait for some small extra time to allow more shuffle blocks to get pushed and merged. This requires some extra coordination in the Spark driver when it transitions from a shuffle map stage to the corresponding reduce stage.
Bug: Lower memory usage in toPandas with Arrow self_destruct
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Sep/20 13:18
Affects Version/s: 3.0.1

As described on the mailing list:
 [http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Reducing-memory-usage-of-toPandas-with-Arrow-quot-self-destruct-quot-option-td30149.html] [https://lists.apache.org/thread.html/r581d7c82ada1c2ac3f0584615785cc60cf5ac231e1f29737d3a6569f%40%3Cdev.spark.apache.org%3E]

toPandas() can as much as double memory usage as both Arrow and Pandas retain a copy of a dataframe in memory during the conversion. Arrow >= 0.16 offers a self_destruct mode that avoids this with some caveats.
Bug: Add support to properly handle different type of stage retries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:56
Affects Version/s: 3.1.0

In SPARK-23243 and SPARK-25341, the concept of an INDETERMINATE stage was introduced, which would be handled differently if retried.

Since these was added to address a data correctness issue, we should also add support for these in push-based shuffle, so that we would be able to rollback the merged shuffle partitions of a shuffle map stage if it's an INDETERMINATE stage.
Bug: Make hive metastore convert config  can be changed session level
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 08/Sep/20 07:28
Affects Version/s: 3.1.0

There are some cases that we have history hive  table, in origin, it use textfile serde, later  it's change to orc serde, since during convert, it will use table level serde to scan all partition, if some old partition file format is not same , will failed to convert , we need to close metastoer convert and restart spark program.

It's bad for adhoc engine such as long running spark thrift server. so I think it's necessary to make these two config can be changed for use in session level
Bug: Avoid re-compute expensive expression
Status: Resolved | Priority: Major | Resolution: Implemented
Created: 19/Sep/20 05:32
Affects Version/s: 3.1.0

{code:java}
  test("Pushdown demo") {
    withTable("t") {
      withTempDir { loc =>
        sql(
          s"""CREATE TABLE t(c1 INT, s STRING) PARTITIONED BY(P1 STRING)
             | LOCATION '${loc.getAbsolutePath}'
             |""".stripMargin)
        sql(
          """
            |SELECT c1,
            |case
            |  when get_json_object(s,'$.a')=1 then "a"
            |  when get_json_object(s,'$.a')=2 then "b"
            |end as s_type
            |FROM t
            |WHERE get_json_object(s,'$.a') in (1, 2)
          """.stripMargin).explain(true)
         }
    }
}
will got plan as 


== Physical Plan ==
*(1) Project [c1#1, CASE WHEN (cast(get_json_object(s#2, $.a) as int) = 1) THEN a WHEN (cast(get_json_object(s#2, $.a) as int) = 2) THEN b END AS s_type#0]
+- *(1) Filter get_json_object(s#2, $.a) IN (1,2)
   +- Scan hive default.t [c1#1, s#2], HiveTableRelation `default`.`t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#1, s#2], [P1#3], Statistics(sizeInBytes=8.0 EiB)

we can see that  get_json_object(s#2, $.a) will be computed tree times
Always there are expensive expressions are re-computed many times in such grammar。

{code}
Bug: Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:36
Affects Version/s: 3.1.0

In the beginning of a shuffle map stage, driver needs to select external shuffle services as the mergers of the shuffle partitions for the corresponding shuffle.

We currently leverage the immediate available information about current and past executor location information for this selection purpose. Ideally, this would be behind a pluggable interface so that we can potentially leverage information tracked outside of a Spark application for better load balancing or for a disaggregate deployment environment.
Bug: Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:51
Affects Version/s: 3.1.0

With the extended MapOutputTracker, the reducers can now get the task input data from the merged shuffle partitions for more efficient shuffle data fetch. The reducers should also be able to fallback to fetching the original unmarked blocks if it encounters failures when fetching the merged shuffle partitions.
Bug: Improve in filter pushdown for ParquetFilters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/20 16:49
Affects Version/s: 3.1.0

Support push down `GreaterThanOrEqual` minimum value and `LessThanOrEqual` maximum value  when its values exceeds `spark.sql.parquet.pushdown.inFilterThreshold`. For example:

```sql
SELECT * FROM t WHERE id IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15)
```

We will push down `id >= 1 and id <= 15`.
Bug: UnsafeExternalSorter.SpillableIterator cannot spill when there are NULLs in the input and radix sorting is used.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 09:19
Affects Version/s: 2.4.7

In order to determine whether {{UnsafeExternalSorter.SpillableIterator}} has spilled already it checks whether {{upstream}} is an instance of {{UnsafeInMemorySorter.SortedIterator}}. When radix sorting is used (added by SPARK-14851) and there are NULLs in the input however, upstream will be an instance of {{UnsafeExternalSorter.ChainedIterator}} instead, but should still be spilled.
Bug: Support submit application with user-defined cluster manager
Status: In Progress | Priority: Major | Resolution: 
Created: 16/Sep/20 07:44
Affects Version/s: 3.0.1

We have supported users to define the customed cluster manager with `ExternalClusterManager` trait. However, we can not submit the application with `SparkSubmit`. This patch adds the support to submit applications with user-defined cluster manager.

 

Add design doc: https://docs.google.com/document/d/1-Sn4Zh-l0SCqH7DQ0esdukS68ptSolK4lStj7MZUqJo/edit?usp=sharing
Bug: Enhance UTF8String.trim
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/20 13:34
Affects Version/s: 3.1.0

It sounds like {{UTF8String.trim}} is not implemented well. We may need to look at how {{java.lang.String.trim}} is implemented.

Please see comment:
 [https://github.com/apache/spark/blob/7eb76d698836a251065753117e22285dd1a8aa8f/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L674-L675]

[https://github.com/apache/spark/pull/29731#discussion_r487709672]
Bug: Add a GitHub Actions job to run WebUI tests with Chrome
Status: In Progress | Priority: Major | Resolution: 
Created: 21/Sep/20 19:48
Affects Version/s: 3.1.0

ChromeUISeleniumSuite and ChromeUIHistoryServerSuite are added to test WebUI with better JavaScript support but they are disabled on the CI environment.

Now that we use GitHub Actions, let's add a job to run them.
Bug: Non-deterministic expressions should not be reordered inside AND and OR
Status: Open | Priority: Major | Resolution: 
Created: 17/Sep/20 20:03
Affects Version/s: 3.1.0

Using the splitDisjunctivePredicates and splitConjunctivePredicates helper methods can change the number of times a non-deterministic expression is executed. This can cause correctness issues on the client side.

An existing test in the FilterPushdownSuite seems to exhibit this problem
{code}
test("generate: non-deterministic predicate referenced no generated column") {
    val originalQuery = {
      testRelationWithArrayType
        .generate(Explode('c_arr), alias = Some("arr"))
        .where(('b >= 5) && ('a + Rand(10).as("rnd") > 6) && ('col > 6))
    }
    val optimized = Optimize.execute(originalQuery.analyze)
    val correctAnswer = {
      testRelationWithArrayType
        .where('b >= 5)
        .generate(Explode('c_arr), alias = Some("arr"))
        .where('a + Rand(10).as("rnd") > 6 && 'col > 6)
        .analyze
    }

    comparePlans(optimized, correctAnswer)
  }
{code}

In the optimized plan, the deterministic filter is moved ahead of the non-deterministic one:
{code}
Filter ((6 < none#0) AND (cast(6 as double) < (rand(10) + cast(none#0 as double))))
{code}
Bug: Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:48
Affects Version/s: 3.1.0

Similar to MapStatus, which tracks the metadata about each map task's shuffle output, we also need to track the metadata about each merged shuffle partition with push-based shuffle. We currently term this as MergeStatus.

Since MergeStatus tracks metadata from the perspective of reducer tasks, it's not efficient to break up the metadata tracked in a MergeStatus and spread it across multiple MapStatus.
Bug: [k8s] Spark client dependency support ignores non-local paths
Status: Open | Priority: Major | Resolution: 
Created: 02/Sep/20 00:16
Affects Version/s: 3.0.0

According to the logic of this line: [https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala#L161,] Spark filters out all paths which are not local (ie. no scheme or [file://|file:///] scheme). It may cause non-local dependencies not loaded by Driver.

For example, when starting a Spark job with spark.jars=*local*:///local/path/1.jar,*s3*://s3/path/2.jar,*file*:///local/path/3.jar, it seems like this logic will upload *file*:///local/path/3.jar to s3, and reset spark.jars to only s3://transformed/path/3.jar, while completely ignoring local:///local/path/1.jar and s3:///s3/path/2.jar.

We need to fix this logic such that Spark uploads local files to S3, and transforms the paths while keeping all other paths as they are.
Bug: Standalone mode blacklist executors page UI marks driver as blacklisted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/20 18:51
Affects Version/s: 3.0.0

I am running a spark shell on a 1 node standalone cluster.  I noticed that the executors page ui was marking the driver as blacklisted for the stage that is running.  Attached a screen shot.

Also, in my case one of the executors died and it doesn't seem like the schedule rpicked up the new one.  It doesn't show up on the stages page and just shows it as active but none of the tasks ran there.

 

You can reproduce this by starting a master and slave on a single node, then launch a shell like where you will get multiple executors (in this case I got 3)

$SPARK_HOME/bin/spark-shell --master spark://yourhost:7077 --executor-cores 4 --conf spark.blacklist.enabled=true

 

From shell run:
{code:java}
import org.apache.spark.TaskContext
val rdd = sc.makeRDD(1 to 1000, 5).mapPartitions { it =>
 val context = TaskContext.get()
 if (context.attemptNumber() < 2) {
 throw new Exception("test attempt num")
 }
 it
}{code}
Bug: Support column list in INSERT statement
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 15:00
Affects Version/s: 3.1.0

INSERT currently does not support named column lists.  

{{INSERT INTO <table> (col1, col2,…) VALUES( 'val1', 'val2', … )}}

Note, we assume the column list contains all the column names. Issue an exception if the list is not complete. The column order could be different from the column order defined in the table definition.
Bug: Optimize Skewed BroadcastNestedLoopJoin with AE
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Sep/20 09:49
Affects Version/s: 3.2.0

For BroadcastNestedLoopJoin, we will broadcast boradcast-side child to all executor and use stream side partition's data traversal broadcast-side data one-by-one. 

We have meet some case that stream side data skew and all success task wait for skewed partition to finish.

We know that the execution time increases exponentially with the amount of partition's data.

If skewd with 100x,  skewed partition's data will execute 100x than non-skewed part.

It is a bottleneck， with AE, we can avoid this by  split skewed part's data  to make it more balanced.
Bug: Add DataStreamWriter.toTable API
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/20 02:44
Affects Version/s: 3.1.0

For now, there's no way to write to the table (especially catalog table) even the table is capable to handle streaming write.

We can add DataStreamWriter. -table- toTable API to let end users specify table as provider, and let streaming query write into the table. That is just to specify the table, and the overall usage of DataStreamWriter isn't changed.
Bug: Full outer stream-stream join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Sep/20 07:52
Affects Version/s: 3.1.0

Current stream-stream join supports inner, left outer and right outer join ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166] ). With current design of stream-stream join (which marks whether the row is matched or not in state store), it would be very easy to support full outer join as well.

 

Full outer stream-stream join will work as followed:

(1).for left side input row, check if there's a match on right side state store. If there's a match, output all matched rows. Put the row in left side state store.

(2).for right side input row, check if there's a match on left side state store. If there's a match, output all matched rows and update left side rows state with "matched" field to set to true. Put the right side row in right side state store.

(3).for left side row needs to be evicted from state store, output the row if "matched" field is false.

(4).for right side row needs to be evicted from state store, output the row if "matched" field is false.
Bug: Pass all `sql/core` module UTs in Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/20 03:38
Affects Version/s: 3.1.0

Now there are  319 TESTS FAILED based on commit `f5360e761ef161f7e04526b59a4baf53f1cf8cd5`
{code:java}
Run completed in 1 hour, 20 minutes, 25 seconds.
Total number of tests run: 8485
Suites: completed 357, aborted 0
Tests: succeeded 8166, failed 319, canceled 1, ignored 52, pending 0
*** 319 TESTS FAILED ***
{code}
 

There are 293 failures associated with TPCDS_XXX_PlanStabilitySuite and TPCDS_XXX_PlanStabilityWithStatsSuite:
 * TPCDSV2_7_PlanStabilitySuite(33 FAILED)
 * TPCDSV1_4_PlanStabilityWithStatsSuite(94 FAILED)
 * TPCDSModifiedPlanStabilityWithStatsSuite(21 FAILED)
 * TPCDSV1_4_PlanStabilitySuite(92 FAILED)
 * TPCDSModifiedPlanStabilitySuite(21 FAILED)
 * TPCDSV2_7_PlanStabilityWithStatsSuite(32 FAILED)

 

Other 26 FAILED cases as follow:
 * StreamingAggregationSuite
 ** count distinct - state format version 1 
 ** count distinct - state format version 2 
 * GeneratorFunctionSuite
 ** explode and other columns
 ** explode_outer and other columns
 * UDFSuite
 ** SPARK-26308: udf with complex types of decimal
 ** SPARK-32459: UDF should not fail on WrappedArray
 * SQLQueryTestSuite
 ** decimalArithmeticOperations.sql
 ** postgreSQL/aggregates_part2.sql
 ** ansi/decimalArithmeticOperations.sql 
 ** udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF
 ** udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF
 * WholeStageCodegenSuite
 ** SPARK-26680: Stream in groupBy does not cause StackOverflowError
 * DataFrameSuite:
 ** explode
 ** SPARK-28067: Aggregate sum should not return wrong results for decimal overflow
 ** Star Expansion - ds.explode should fail with a meaningful message if it takes a star
 * DataStreamReaderWriterSuite
 ** SPARK-18510: use user specified types for partition columns in file sources
 * OrcV1QuerySuite\OrcV2QuerySuite
 ** Simple selection form ORC table * 2
 * ExpressionsSchemaSuite
 ** Check schemas for expression examples
 * DataFrameStatSuite
 ** SPARK-28818: Respect original column nullability in `freqItems`
 * JsonV1Suite\JsonV2Suite\JsonLegacyTimeParserSuite
 ** SPARK-4228 DataFrame to JSON * 3
 ** backward compatibility * 3
Bug: RPC implementation to support control plane coordination for push-based shuffle
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 17:27
Affects Version/s: 3.1.0

RPCs to facilitate coordination of shuffle map/reduce stages. Notifications to external shuffle services to finalize shuffle block merge for a given shuffle are carried through this RPC. It also respond back the metadata about a merged shuffle partition back to the caller.
Bug: Prohibit binary comparisons chain
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Sep/20 10:11
Affects Version/s: 3.1.0

{code:scala}
spark.range(10).selectExpr("id as a", "id as b", "id as c").createTempView("t1")
spark.range(10).selectExpr("id as a", "id as b", "id as c").createTempView("t2")
spark.sql("select * from t1 join t2 on t1.a = t2.a = t1.b = t2.b").explain()
{code}

{noformat}
== Physical Plan ==
BroadcastNestedLoopJoin BuildRight, Inner, (cast((cast((a#2704L = a#2712L) as bigint) = b#2705L) as bigint) = b#2713L)
:- *(1) Project [id#2702L AS a#2704L, id#2702L AS b#2705L, id#2702L AS c#2706L]
:  +- *(1) Range (0, 10, step=1, splits=2)
+- BroadcastExchange IdentityBroadcastMode, [id=#207]
   +- *(2) Project [id#2710L AS a#2712L, id#2710L AS b#2713L, id#2710L AS c#2714L]
      +- *(2) Range (0, 10, step=1, splits=2)
{noformat}


{noformat}
postgres=# create table t1(a int, b int, c int);
CREATE TABLE
postgres=# select * from t1 where a = b =c;
ERROR:  syntax error at or near "="
LINE 1: select * from t1 where a = b =c;
                                     ^
{noformat}
Bug: adaptively blockify instances
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Sep/20 07:21
Affects Version/s: 3.1.0

According to the performance test in https://issues.apache.org/jira/browse/SPARK-31783, the performance gain is mainly related to the nnz of block.

So it is reasonable to control the size of block.

 

I had some offline discuss with [~weichenxu123], then we think following changes are worthy：

1, infer an appropriate blockSize (MB) based on numFeatures and nnz by default;

2, impls should use a relative small memory footprint when processing one block, and should not use a large pre-allocated buffer, so we need to revert gmm;

3, use new blockify strategy in LinearSVC/LoR/LiR/AFT;
Bug: Improve the performance for NTH_VALUE and Reactor the OffsetWindowFunction
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Sep/20 07:44
Affects Version/s: 3.1.0

Spark SQL support some window function like  NTH_VALUE
If we specify window frame like


{code:java}
UNBOUNDED PRECEDING AND CURRENT ROW
{code}

or

{code:java}
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
{code}


We can elimate some calculations.
For example: if we execute the SQL show below:

{code:java}
SELECT NTH_VALUE(col,
         2) OVER(ORDER BY rank UNBOUNDED PRECEDING
        AND CURRENT ROW)
FROM tab;
{code}
The output for row number greater than 1, return the fixed value. otherwise, return null. So we just calculate the value once and notice whether the row number less than 2.
UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING is simpler.
Bug: Division by zero on divide-like operations returns incorrect result
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/20 03:54
Affects Version/s: 3.1.0

Spark SQL:
{code:sql}
spark-sql> SELECT 1/0;
NULL
Time taken: 3.002 seconds, Fetched 1 row(s)
{code}
PostgreSQL:
{code:sql}
postgres=# SELECT 1/0;
ERROR:  division by zero
{code}
Bug: Wildcards not working in get_json_object
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 03/Sep/20 11:22
Affects Version/s: 2.4.4

It seems that wildcards (star) are not supported when using {{get_json_object}}:
{code:java}
spark.sql("""select get_json_object('{"k":{"value":"abc"}}', '$.*.value') as j""").show()
{code}
This results in {{null}} while it should return 'abc'. It works if I replace * with 'k'.

  !image-2020-09-03-13-22-38-569.png!
Bug: Incorrect number of dynamic part metric
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Sep/20 16:05
Affects Version/s: 3.0.0

How to reproduce this issue:
{code:sql}
create table dynamic_partition(i bigint, part bigint) using parquet partitioned by (part);
insert overwrite table dynamic_partition partition(part) select id, id % 50 as part  from range(10000);
{code}
The number of dynamic part should be 50, but it is 800 on web UI.
Bug: Planner error when aggregating multiple distinct Constant columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/20 05:09
Affects Version/s: 3.0.0

SELECT COUNT(DISTINCT 2), COUNT(DISTINCT 2, 3) will trigger this bug.

The problematic code is:

 
{code:java}
val distinctAggGroups = aggExpressions.filter(_.isDistinct).groupBy { e =>
  val unfoldableChildren = e.aggregateFunction.children.filter(!_.foldable).toSet
  if (unfoldableChildren.nonEmpty) {
    // Only expand the unfoldable children
     unfoldableChildren
  } else {
    // If aggregateFunction's children are all foldable
    // we must expand at least one of the children (here we take the first child),
    // or If we don't, we will get the wrong result, for example:
    // count(distinct 1) will be explained to count(1) after the rewrite function.
    // Generally, the distinct aggregateFunction should not run
    // foldable TypeCheck for the first child.
    e.aggregateFunction.children.take(1).toSet
  }
}
{code}