av9ash/GitBugs huggingface dataset:
Bug: Streaming-batch intersects are incorrectly allowed through UnsupportedOperationsChecker
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Apr/21 17:25
Affects Version/s: 3.0.0

The UnsupportedOperationChecker currently rejects streaming intersects only if both sides are streaming, but they don't work if even one side is streaming. The following simple test, for example, fails with a cryptic None.get error because the state store can't plan itself properly.
{code:java}
  test("intersect") {
    val input = MemoryStream[Long]
    val df = input.toDS().intersect(spark.range(10).as[Long])
    testStream(df) (
      AddData(input, 1L),
      CheckAnswer(1)
    )
  }
{code}
Bug: Have a way for other people to retrigger the build in GitHub Actions
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/21 00:15
Affects Version/s: 3.2.0

We should ask contributors to retrigger the tests because the builds run in their fork, and currently committers or other people cannot retrigger out of the box.
 Note that the retriggering has to happen in forked repository. This cannot be done from main repository.

One possible way is to create a workflow *that only runs in forked repository*:
 1. Regularly (15 mins?) get a list of PRs opened from the forked repository or possibly author? (see [https://github.com/apache/spark/blob/master/.github/workflows/update_build_status.yml#L36-L47])
 2. Iterate the PRs:
 ㅤ2.1. Get the latest workflow run ([https://github.com/apache/spark/blob/master/.github/workflows/notify_test_workflow.yml#L41-L59])
 ㅤ2.2. Iterates the comments in the PR (see [https://docs.github.com/en/rest/guides/working-with-comments#pull-request-comments] for Javascript and [https://docs.github.com/en/rest/reference/issues#list-issue-comments] for REST API). Issue number is PR number.
 ㅤㅤ2.2.1. check if there is a comment such as "GitHub Actions: retrigger please" _after the latest workflow run_. Last update time is available when you get workflow run, see also [https://docs.github.com/en/rest/reference/actions#get-a-workflow-run]
 ㅤㅤㅤ2.2.1.1. If there is, retrigger the workflow run, see also [https://docs.github.com/en/rest/reference/actions#create-a-workflow-dispatch-event]
 ㅤㅤㅤ2.2.1.2. If not, skip.
Bug: Guide users to sync branch and enable GitHub Actions in their forked repository
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Apr/21 03:02
Affects Version/s: 3.2.0

If developers don't enable Github Actions in their fork, the Pr builds cannot run. We should guide them to enable.

Also, the branch should be synced to the latest master branch.

We could leverage Action Required status in GitHub check: https://docs.github.com/en/rest/guides/getting-started-with-the-checks-api
Bug: Add some guides for authors to retrigger the workflow run
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/21 01:07
Affects Version/s: 3.2.0

Currently only authors can retrigger the GitHub Actions build in their PRs. We should explicitly guide them, at [https://github.com/apache/spark/blob/master/.github/workflows/notify_test_workflow.yml#L110], something like:
 If the tests fail for reasons unrelated to the change, please retrigger the workflow run in your forked repository.
 If related, please investigate, fix and push new changes to fix the test failure.

This guides can be removed once SPARK-35157 is done.
Bug: False active executor in UI that caused by BlockManager reregistration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/21 17:58
Affects Version/s: 3.1.1

*Note:* This is a follow-up on SPARK-34949, even after the heartbeat fix, driver reports dead executors as alive.

*Problem:*

I was testing Dynamic Allocation on K8s with about 300 executors. While doing so, when the executors were torn down due to "spark.dynamicAllocation.executorIdleTimeout", I noticed all the executor pods being removed from K8s, however, under the "Executors" tab in SparkUI, I could see some executors listed as alive. 

[spark.sparkContext.statusTracker.getExecutorInfos.length|https://github.com/apache/spark/blob/65da9287bc5112564836a555cd2967fc6b05856f/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala#L100] also returned a value greater than 1. 

 

*Cause:*
 * "CoarseGrainedSchedulerBackend" issues async "StopExecutor" on executorEndpoint
 * "CoarseGrainedSchedulerBackend" removes that executor from Driver's internal data structures and publishes "SparkListenerExecutorRemoved" on the "listenerBus".
 * Executor has still not processed "StopExecutor" from the Driver
 * Driver receives heartbeat from the Executor, since it cannot find the "executorId" in its data structures, it responds with "HeartbeatResponse(reregisterBlockManager = true)"
 * "BlockManager" on the Executor reregisters with the "BlockManagerMaster" and "SparkListenerBlockManagerAdded" is published on the "listenerBus"
 * Executor starts processing the "StopExecutor" and exits
 * "AppStatusListener" picks the "SparkListenerBlockManagerAdded" event and updates "AppStatusStore"
 * "statusTracker.getExecutorInfos" refers "AppStatusStore" to get the list of executors which returns the dead executor as alive.

 

*Proposed Solution:*

Maintain a Cache of recently removed executors on Driver. During the registration in BlockManagerMasterEndpoint if the BlockManager belongs to a recently removed executor, return None indicating the registration is ignored since the executor will be shutting down soon.

On BlockManagerHeartbeat, if the BlockManager belongs to a recently removed executor, return true indicating the driver knows about it, thereby preventing reregisteration.
Bug: Avoid shuffle if join type do not match
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 06/Apr/21 12:50
Affects Version/s: 3.2.0

How to reproduce this issue:
{code:scala}
    spark.sql("set spark.sql.autoBroadcastJoinThreshold=-1")
    spark.sql("CREATE TABLE t1 using parquet clustered by (id) into 200 buckets AS SELECT cast(id as bigint) FROM range(1000)")
    spark.sql("CREATE TABLE t2 using parquet clustered by (id) into 200 buckets AS SELECT cast(id as int) FROM range(500)")
    spark.sql("select * from t1 join t2 on (t1.id = t2.id)").explain
{code}

Current:
{noformat}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [id#14L], [cast(id#15 as bigint)], Inner
   :- Sort [id#14L ASC NULLS FIRST], false, 0
   :  +- Filter isnotnull(id#14L)
   :     +- FileScan parquet default.t1[id#14L] Batched: true, DataFilters: [isnotnull(id#14L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 200 out of 200
   +- Sort [cast(id#15 as bigint) ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(cast(id#15 as bigint), 200), ENSURE_REQUIREMENTS, [id=#58]
         +- Filter isnotnull(id#15)
            +- FileScan parquet default.t2[id#15] Batched: true, DataFilters: [isnotnull(id#15)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:int>
{noformat}

Expected:
{noformat}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [id#14L], [cast(id#15 as bigint)], Inner
   :- Sort [id#14L ASC NULLS FIRST], false, 0
   :  +- Filter isnotnull(id#14L)
   :     +- FileScan parquet default.t1[id#14L] Batched: true, DataFilters: [isnotnull(id#14L)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 200 out of 200
   +- Sort [cast(id#15 as bigint) ASC NULLS FIRST], false, 0
      +- Filter isnotnull(id#15)
         +- FileScan parquet default.t2[id#15] Batched: true, DataFilters: [isnotnull(id#15)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/Users/yumwang/opensource/spark/spark-warehouse/org.apache.spark...., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:int>, SelectedBucketsCount: 200 out of 200
{noformat}
Bug: Improve readability of NestingColumnAliasing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/21 17:18
Affects Version/s: 3.1.1

Refactor https://github.com/apache/spark/blob/6c587d262748a2b469a0786c244e2e555f5f5a74/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala#L31 for readability.
Bug: Spark 3.1.1 is slower than 3.0.2 by 4-5 times
Status: Open | Priority: Major | Resolution: 
Created: 14/Apr/21 07:12
Affects Version/s: 3.1.1

Hi,

The following snippet code runs 4-5 times slower when it's used in Apache Spark or PySpark 3.1.1 compare to Apache Spark or PySpark 3.0.2:

 
{code:java}
spark = SparkSession.builder \
        .master("local[*]") \
        .config("spark.driver.memory", "16G") \
        .config("spark.driver.maxResultSize", "0") \
        .config("spark.serializer",
"org.apache.spark.serializer.KryoSerializer") \
        .config("spark.kryoserializer.buffer.max", "2000m") \
        .getOrCreate()

Toys = spark.read \
  .parquet('./toys-cleaned').repartition(12)

# tokenize the text
regexTokenizer = RegexTokenizer(inputCol="reviewText",
outputCol="all_words", pattern="\\W")
toys_with_words = regexTokenizer.transform(Toys)

# remove stop words
remover = StopWordsRemover(inputCol="all_words", outputCol="words")
toys_with_tokens = remover.transform(toys_with_words).drop("all_words")

all_words = toys_with_tokens.select(explode("words").alias("word"))
# group by, sort and limit to 50k
top50k =
all_words.groupBy("word").agg(count("*").alias("total")).sort(col("total").desc()).limit(50000)

top50k.show()
{code}
 

Some debugging on my side revealed that in Spark/PySpark 3.0.2 the 12 partitions are respected in a way that all 12 tasks are being processed altogether. However, in Spark/PySpark 3.1.1 even though we have 12 tasks, 10 of them finish immediately and only 2 are being processed. (I've tried to disable a couple of configs related to something similar, but none of them worked)

Screenshot of spark 3.1.1 task:

!image-2022-03-17-17-18-36-793.png|width=1073,height=652!

!image-2022-03-17-17-19-11-655.png!

 

Screenshot of spark 3.0.2 task:

 

!image-2022-03-17-17-19-34-906.png!

For a longer discussion: [Spark User List |https://lists.apache.org/thread/1hlg9fpxnw8dzx8bd2fvffmk7yozoszf]

 

You can reproduce this big difference of performance between Spark 3.1.1 and Spark 3.0.2 by using the shared code with any dataset that is large enough to take longer than a minute. Not sure if this is related to SQL, any Spark config being enabled in 3.x but not really into action before 3.1.1, or it's about .transform in Spark ML.
Bug: Correlated subqueries with equality predicates can return wrong results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 00:08
Affects Version/s: 3.2.0

Correlated subqueries with aggregate that pass CheckAnalysis (with only correlated equality predicates) can still return wrong results. This is because equality predicates do not guarantee one-to-one mappings between inner and outer attributes, and the semantics of the plan will be changed when the inner attributes are pulled up through an Aggregate, which gives us wrong results. Currently, the decorrelation framework does not support these types of correlated subqueries, and they should be blocked in CheckAnalysis.

Example 1:
{code:sql}
create or replace view t1(c) as values ('a'), ('b')
create or replace view t2(c) as values ('ab'), ('abc'), ('bc')

select c, (select count(*) from t2 where t1.c = substring(t2.c, 1, 1)) from t1
{code}
Correct results: [(a, 2), (b, 1)]
 Spark results:
{code:java}
+---+-----------------+
|c  |scalarsubquery(c)|
+---+-----------------+
|a  |1                |
|a  |1                |
|b  |1                |
+---+-----------------+{code}
Example 2:
{code:sql}
create or replace view t1(a, b) as values (0, 6), (1, 5), (2, 4), (3, 3);
create or replace view t2(c) as values (6);

select c, (select count(*) from t1 where a + b = c) from t2;{code}
Correct results: [(6, 4)]
 Spark results:
{code:java}
+---+-----------------+
|c  |scalarsubquery(c)|
+---+-----------------+
|6  |1                |
|6  |1                |
|6  |1                |
|6  |1                |
+---+-----------------+
{code}
Bug: Throw fetch failure exception when unable to deserialize broadcasted map statuses
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Apr/21 04:26
Affects Version/s: 2.4.7

One customer encountered application error. From the log, it is caused by accessing non-existing broadcasted value. The broadcasted value is map statuses. There is a race-condition.

After map statuses are broadcasted and the executors obtain serialized broadcasted map statuses. If any fetch failure happens after, Spark scheduler invalidates cached map statuses and destroy broadcasted value of the map statuses. Then any executor trying to deserialize serialized broadcasted map statuses and access broadcasted value, {{IOException}} will be thrown. Currently we don't catch it in {{MapOutputTrackerWorker}} and above exception will fail the application.

Normally we should throw a fetch failure exception for such case and let Spark scheduler handle this.
Bug: a general auto merge output small files feature for datasource api
Status: In Progress | Priority: Major | Resolution: 
Created: 25/Apr/21 08:04
Affects Version/s: 3.0.2

in most case, users write data to hive table or hdfs dir with spark sql, since as spark3.0 released, offical didn't encourge to use hive module to read/write hive table, preferred  switching to datasource api from hive strategy rule, so as to centralize io operation with one module.

so given a general auto merge output files ability for datasource api would resolve many users's small files problem in production, and it can bind with datasource write framwork tightly, so that the auto merge course is transparent to users, and it is capable to handle all kinds of writing method, such as writing hdfs dir/non-partitioned hive table/dynamic partition hive table

this is my individual implemetation for the functionality, and it's stable in production environment of my company
Bug: ANSI mode: disallow group by aliases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/21 12:17
Affects Version/s: 3.2.0

As per the ANSI SQL standard secion 7.12 <group by clause>: 
bq. Each <grouping column reference> shall unambiguously reference a column of the table resulting from the <from clause>. A column referenced in a <group by clause> is a grouping column.

By forbidding it, we can avoid ambiguous SQL queries like:
SELECT col + 1 as col FROM t GROUP BY col
Bug: CoarseGrainedExecutorBackend send wrong 'Reason' when executor exits which leading to job failed.
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Apr/21 07:19
Affects Version/s: 2.3.0

when executor lost for some reason(e.g.  Unable to register with external shuffle server)，CoarseGrainedExecutorBackend will send a RemoveExecutor event with 'ExecutorLossReason'.   But this will cause TaskSetManager handle handleFailedTask function with exitCausedByApp=true.   This is not correct
Bug: Support AQE side shuffled hash join formula
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Apr/21 10:09
Affects Version/s: 3.2.0

Use AQE runtime statistics to decide if we can use shuffled hash join instead of sort merge join. Currently, the formula of shuffled hash join selection does not work due to the dymanic shuffle partition number.

 

Add a new config `spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold` to decide if join can be converted to shuffled hash join safely.
Bug: Support AQE side broadcastJoin threshold
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Apr/21 02:41
Affects Version/s: 3.2.0

The main idea here is that make join config isolation between normal planner and aqe planner which shared the same code path.

Actually we don not very trust using the static stat to consider if it can build broadcast hash join. In our experience it's very common that Spark throw broadcast timeout or driver side OOM exception when execute a bit large plan. And due to braodcast join is not reversed which means if we covert join to braodcast hash join at first time, we(AQE) can not optimize it again, so it should make sense to decide if we can do broadcast at aqe side using different sql config.

In order to achieve this we use a specific join hint in advance during AQE framework and then at JoinSelection side it will take and follow the inserted hint.

For now we only support select strategy for equi join, and follow this order
 1. mark join as broadcast hash join if possible
 2. mark join as shuffled hash join if possible

Note that, we don't override join strategy if user specifies a join hint.
Bug: Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Apr/21 05:49
Affects Version/s: 3.1.1

*Problem:*

I was testing Dynamic Allocation on K8s with about 300 executors. While doing so, when the executors were torn down due to "spark.dynamicAllocation.executorIdleTimeout", I noticed all the executor pods being removed from K8s, however, under the "Executors" tab in SparkUI, I could see some executors listed as alive. 

[spark.sparkContext.statusTracker.getExecutorInfos.length|https://github.com/apache/spark/blob/65da9287bc5112564836a555cd2967fc6b05856f/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala#L100] also returned a value greater than 1. 

 

*Cause:*
 * "CoarseGrainedSchedulerBackend" issues RemoveExecutor on a "executorEndpoint" and publishes "SparkListenerExecutorRemoved" on the "listenerBus"
 * "CoarseGrainedExecutorBackend" starts the executor shutdown
 * "HeartbeatReceiver" picks the "SparkListenerExecutorRemoved" event and removes the executor from "executorLastSeen"
 * In the meantime, the executor reports a Heartbeat. Now "HeartbeatReceiver" cannot find the "executorId" in "executorLastSeen" and hence responds with "HeartbeatResponse(reregisterBlockManager = true)"
 * The Executor now calls "env.blockManager.reregister()" and reregisters itself thus creating inconsistency

 

*Proposed Solution:*

The "reportHeartBeat" method is not aware of the fact that Executor is shutting down, it should check "executorShutdown" before reregistering.
Bug: Improve push based shuffle to work with AQE by fetching partial map indexes for a reduce partition
Status: Open | Priority: Major | Resolution: 
Created: 12/Apr/21 19:54
Affects Version/s: 3.1.1

Currently when both Push based shuffle and AQE is enabled and when partial set of map indexes are requested to MapOutputTracker this is delegated the regular shuffle instead of push based shuffle reading map blocks. This is because blocks from mapper in push based shuffle are merged out of order due to which its hard to only get the matching blocks of the reduce partition for the requested start and end map indexes.
Bug: Add join hint build side check
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/21 10:06
Affects Version/s: 3.2.0

Currently we support specify the join implementation with hint, but Spark did not promised it.

For example broadcast outer join and hash outer join we need to check if its build side was supported. And at least we should print some warning log instead of changing to other join implementation silently.
Bug: ExternalBlockHandler metrics have misleading unit in the name
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Apr/21 18:25
Affects Version/s: 3.1.1

Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics:
{code}
    // Time latency for open block request in ms
    private final Timer openBlockRequestLatencyMillis = new Timer();
    // Time latency for executor registration latency in ms
    private final Timer registerExecutorRequestLatencyMillis = new Timer();
    // Time latency for processing fetch merged blocks meta request latency in ms
    private final Timer fetchMergedBlocksMetaLatencyMillis = new Timer();
    // Time latency for processing finalize shuffle merge request latency in ms
    private final Timer finalizeShuffleMergeLatencyMillis = new Timer();
{code}
However these Dropwizard Timers by default use nanoseconds ([documentation|https://metrics.dropwizard.io/3.2.3/getting-started.html#timers]). It's certainly possible to extract milliseconds from them, but it seems misleading to have millis in the name here.

This causes {{YarnShuffleServiceMetrics}} to expose confusingly-named metrics like {{openBlockRequestLatencyMillis_count}} and {{openBlockRequestLatencyMillis_nanos}}. It should be up to the metrics exporter, like {{YarnShuffleServiceMetrics}}, to decide the unit and adjust the name accordingly, so the unit shouldn't be included in the name of the metric itself.
Bug: Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/21 18:19
Affects Version/s: 3.1.1

Today the {{ExternalBlockHandler}} component of ESS exposes some useful metrics, but is lacking around metrics for the rate of block transfers. We have {{blockTransferRateBytes}} to tell us the rate of _bytes_, but no metric to tell us the rate of _blocks_, which is especially relevant when running the ESS on HDDs that are sensitive to random reads. Many small block transfers can have a negative impact on performance, but won't show up as a spike in {{blockTransferRateBytes}} since the sizes are small.

We can also enhance {{YarnShuffleServiceMetrics}} to expose histogram-style metrics from the {{Timer}} instances within {{ExternalBlockHandler}} -- today it is only exposing the count and rate, but not timing information from the {{Snapshot}}.

These two changes can make it easier to monitor the health of the ESS.
Bug: Parse interval literals as ANSI intervals
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Apr/21 19:33
Affects Version/s: 3.2.0

If the SQL config spark.sql.legacy.interval.enabled is disabled (by default), parse interval literals as DayTimeIntervalType if the interval literal doesn't have the month field or YearMonthIntervalType of it doesn't have the day-time fields. In the case of mixed fields, throw an exception.
Bug: Extract a field from ANSI interval
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/21 12:29
Affects Version/s: 3.2.0

Support field extraction from year-month and day-time intervals. For example:

{code:sql}
      > SELECT extract(days FROM INTERVAL '5 0:0:0' day to second);
       5
      > SELECT extract(seconds FROM INTERVAL '10 11:12:30.001001' DAY TO SECOND);
       30.001001
{code}