av9ash/GitBugs huggingface dataset:
Bug: Keep the original tests and codes to avoid potential conflicts in dev in ParquetFilterSuite and ParquetIOSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/20 01:34
Affects Version/s: 3.0.0

There are some unnecessary diff at https://github.com/apache/spark/pull/27728#discussion_r397655390, which could cause potential confusions. It will make more diff when other people match the codes with it.

It might be best to address the comment and make the diff minimized against other branches.
Bug: Bump up the minimum Arrow version as 1.0.0 in SparkR
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/20 02:18
Affects Version/s: 3.1.0

R rather aggressively encourages people to use the latest version, and SparkR vectorization is very experimental that was added from Spark 3.0.

It should be good if we can minimise the maintenance overhead and keep the minimal version up to date.
Bug: Remove heapq3 port from Python 3
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Jul/20 05:29
Affects Version/s: 3.1.0

We have the manual port heapq3 https://github.com/apache/spark/blob/c3be2cd347c42972d9c499b6fd9a6f988f80af12/python/pyspark/heapq3.py to support {{key}} and {{reverse}}.

It should be removed together since Python 2 was dropped.
Bug: Set jobs that runs periodically for other profile combinations
Status: Resolved | Priority: Major | Resolution: Later
Created: 09/Jul/20 15:17
Affects Version/s: 2.4.6

Github Actions supports to scheduled build and job, see https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onschedule

It should be good if we set the build with other profiles and maybe JDK 11 combinations.
Bug: GroupedData Pandas UDF 2Gb limit
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 13/Jul/20 18:27
Affects Version/s: 3.0.0

`spark.sql.execution.arrow.maxRecordsPerBatch` is not respected for GroupedData, the whole group is passed to Pandas UDF at once, which can cause various 2Gb limitations on Arrow side (and in current versions of Arrow, also 2Gb limitation on Netty allocator side) - https://issues.apache.org/jira/browse/ARROW-4890 

Would be great to consider feeding GroupedData into a pandas UDF in batches to solve this issue. 

cc [~hyukjin.kwon]
Bug: Leverage Conda environment at pip packaging test in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/20 01:01
Affects Version/s: 3.1.0

If you take a close look for GitHub Actions log:
{code:java}
 Installing dist into virtual env
Processing ./python/dist/pyspark-3.1.0.dev0.tar.gz
Collecting py4j==0.10.9
 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Using legacy setup.py install for pyspark, since package 'wheel' is not installed.
Installing collected packages: py4j, pyspark
 Running setup.py install for pyspark: started
 Running setup.py install for pyspark: finished with status 'done'
Successfully installed py4j-0.10.9 pyspark-3.1.0.dev0

...

Installing dist into virtual env
Obtaining file:///home/runner/work/spark/spark/python
Collecting py4j==0.10.9
 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB)
Installing collected packages: py4j, pyspark
 Attempting uninstall: py4j
 Found existing installation: py4j 0.10.9
 Uninstalling py4j-0.10.9:
 Successfully uninstalled py4j-0.10.9
 Attempting uninstall: pyspark
 Found existing installation: pyspark 3.1.0.dev0
 Uninstalling pyspark-3.1.0.dev0:
 Successfully uninstalled pyspark-3.1.0.dev0
 Running setup.py develop for pyspark
Successfully installed py4j-0.10.9 pyspark
{code}

It looks not properly using conda as it removes and re-installs again.
Bug: Install pydata_sphinx_theme in Jenkins machines
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/20 12:28
Affects Version/s: 3.0.1

After SPARK-32179, {{pydata_sphinx_theme}} https://pypi.org/project/pydata-sphinx-theme/ is needed as a new Python dependency for PySpark documentation build.
We should install it in Jenkins to test PySpark documentation build in Python 3.
Bug: Flaky pip installation test in Jenkins
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/20 04:48
Affects Version/s: 3.0.0

Currently pip packaging test is flaky in Jenkins:

{code}
Installing collected packages: py4j, pyspark
  Attempting uninstall: py4j
    Found existing installation: py4j 0.10.9
    Uninstalling py4j-0.10.9:
      Successfully uninstalled py4j-0.10.9
  Attempting uninstall: pyspark
    Found existing installation: pyspark 3.1.0.dev0
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 407, in run
    use_user_site=options.use_user_site,
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 64, in install_given_reqs
    auto_confirm=True
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 675, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/tmp/tmp.GX6lHKLHZK/3.6/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 545, in from_dist
    link_pointer, dist.project_name, dist.location)
AssertionError: Egg-link /home/jenkins/workspace/spark-master-test-sbt-hadoop-2.7-hive-2.3/python does not match installed location of pyspark (at /home/jenkins/workspace/SparkPullRequestBuilder@2/python)
Cleaning up temporary directory - /tmp/tmp.GX6lHKLHZK
{code}
Bug: Fix doc link of datetime pattern in 3.0 migration guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/20 08:30
Affects Version/s: 3.0.1

In http://spark.apache.org/docs/latest/sql-migration-guide.html#query-engine, there is a invalid reference for datetime reference "sql-ref-datetime-pattern.md". We should fix it.
Bug: Publish failed and succeeded test reports in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jul/20 02:55
Affects Version/s: 3.1.0

Currently, the readability in the logs are not really good. For example, see https://pipelines.actions.githubusercontent.com/gik0C3if0ep5i8iNpgFlcJRQk9UyifmoD6XvJANMVttkEP5xje/_apis/pipelines/1/runs/564/signedlogcontent/4?urlExpires=2020-07-09T14%3A05%3A52.5110439Z&urlSigningMethod=HMACV1&urlSignature=gMGczJ8vtNPeQFE0GpjMxSS1BGq14RJLXUfjsLnaX7s%3D

Maybe we should have a way to report the results in an easy way to read. For example, Jenkins test report-like feature.
We should maybe also take a look for https://github.com/check-run-reporter/action.
Bug: Development - Debugging
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 08:39
Affects Version/s: 3.1.0

1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspark/profiler.html, 
{code}
>>> sc._conf.set("spark.python.profile", "true")
>>> rdd = sc.parallelize(range(100)).map(str)
>>> rdd.count()
100
>>> sc.show_profiles()
{code}

2. Python Debugger, e.g.) https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html#remote-interpreter \(?\)

3. Monitoring Python Workers: {{top}}, `{{ls}}`, etc. \(?\)

4. PyCharm setup \(?\) (at https://issues.apache.org/jira/browse/SPARK-32189)
Bug: Install Sphinx in Python 3 on Jenkins machines
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jul/20 01:51
Affects Version/s: 3.1.0

Currently Sphinx is only installed in Python 2. We should install it in Python 3 and test it in Jenkins as Python 2, 3.4 and 3.5 were dropped at SPARK-32138.

See also:
https://github.com/apache/spark/pull/28957/files#diff-ccd847a0316575dde31bd89786bbe1f2R176
https://github.com/apache/spark/blob/ec42492b60559a983435a24630d5dc8827cf22d9/dev/lint-python#L176
Bug: Check the Distinct by assuming it as Aggregate for Structured Streaming
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jul/20 09:27
Affects Version/s: 3.0.0

We want to fix 2 things here:

1. Give better error message for Distinct related operations in append mode that doesn't have a watermark

Check the following example: 
{code:java}
val s1 = spark.readStream.format("rate").option("rowsPerSecond", 1).load().createOrReplaceTempView("s1")
val s2 = spark.readStream.format("rate").option("rowsPerSecond", 1).load().createOrReplaceTempView("s2")
val unionRes = spark.sql("select s1.value , s1.timestamp from s1 union select s2.value, s2.timestamp from s2")
unionResult.writeStream.option("checkpointLocation", ${pathA}).start(${pathB}){code}
We'll get the following confusing exception:
{code:java}
java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:529)
	at scala.None$.get(Option.scala:527)
	at org.apache.spark.sql.execution.streaming.StateStoreSaveExec.$anonfun$doExecute$9(statefulOperators.scala:346)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:561)
	at org.apache.spark.sql.execution.streaming.StateStoreWriter.timeTakenMs(statefulOperators.scala:112)
...
{code}
The union clause in SQL has the requirement of deduplication, the parser will generate {{Distinct(Union)}} and the optimizer rule {{ReplaceDistinctWithAggregate}} will change it to {{Aggregate(Union)}}. So the root cause here is the checking logic for Aggregate is missing for Distinct.

Actually it happens for all Distinct related operations in Structured Streaming, e.g
{code:java}
val df = spark.readStream.format("rate").load()
df.createOrReplaceTempView("deduptest")
val distinct = spark.sql("select distinct value from deduptest")
distinct.writeStream.option("checkpointLocation", ${pathA}).start(${pathB}){code}
 

2. Make {{Distinct}} in complete mode runnable.

The distinct in complete mode will throw the exception:

{quote} 

{{Complete output mode not supported when there are no streaming aggregations on streaming DataFrames/Datasets;}}

{quote}
Bug: Query and Batch Id not set for Structured Streaming Jobs in case of ForeachBatch in PySpark
Status: Open | Priority: Major | Resolution: 
Created: 30/Jul/20 15:33
Affects Version/s: 2.4.6

Query Id and Batch Id information is not available for jobs started by structured streaming query when _foreachBatch_ API is used in PySpark.

This happens only with foreachBatch in pyspark. ForeachBatch in scala works fine, and also other structured streaming sinks in pyspark work fine. I am attaching a screenshot of jobs pages.

I think job group is not set properly when _foreachBatch_ is used via pyspark. I have a framework that depends on the _queryId_ and _batchId_ information available in the job properties and so my framework doesn't work for pyspark-foreachBatch use case.
Bug: Install PyPy3 on Jenkins to enable PySpark tests with PyPy
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 11/Jul/20 01:24
Affects Version/s: 3.1.0

Current PyPy installed in Jenkins is too old, which is Python 2 compatible. Python 2 will be dropped at SPARK-32138, and we should now upgrade PyPy to Python 3 compatible PyPy 3.

See also:
https://github.com/apache/spark/pull/28957/files#diff-871d87c62d4e9228a47145a8894b6694R160
https://github.com/apache/spark/blob/ec42492b60559a983435a24630d5dc8827cf22d9/python/run-tests.py#L160
Bug: Upgrade to Zstd 1.4.5-4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jul/20 16:27
Affects Version/s: 3.1.0

This issue aims to upgrade to Zstd 1.4.5-4

ZStd 1.4.5-4 fixes the following. 

- https://github.com/luben/zstd-jni/commit/3d16e515253a0b6941810ab3ee4055201ca9ff54
- https://github.com/luben/zstd-jni/commit/3d51bdcb82c0003f943416cae7aba3f3001b1163
Bug: Decrease parallelism of PySpark tests on Jenkins
Status: Resolved | Priority: Major | Resolution: Later
Created: 10/Jul/20 02:31
Affects Version/s: 2.4.6

PySpark tests became flaky apparently due to the machines being slow. The parallelism of PySpark increased at SPARK-26148 to speed up.

Maybe we should decrease it to make the test stability better. It will only increase around 10 minutes which is rather a small portion compared to 6 ~ 7 hours in total testing time.
Bug: Executors should not be able to create SparkContext.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/20 23:24
Affects Version/s: 2.4.6

Currently executors can create SparkContext, but shouldn't be able to create it.
{code:scala}
sc.range(0, 1).foreach { _ =>
  new SparkContext(new SparkConf().setAppName("test").setMaster("local"))
}
{code}
Bug: Use Hadoop 3 profile in AppVeyor SparkR build
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 14:18
Affects Version/s: 3.1.0

AppVeyor with Hadoop 3 is failed, see

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/33977845

It will be disabled for now at SPARK-32230.

We should investigate and enable it back.
Bug: Test PySpark with Python 3.8 in Github Actions as well
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jul/20 01:31
Affects Version/s: 3.1.0

Currently, only PyPy3 and Python 3.6 are being tested with PySpark. We should test the latest version of Python as well because some optimizations can be only enabled with Python 3.8+. See also https://github.com/apache/spark/pull/29114
Bug: Enable crossPaths back to prevent side effects
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 11:01
Affects Version/s: 3.1.0

After SPARK-32245, crossPaths was disabled in SBT build to run the Junit tests per project properly.
This is correct change since we're not doing the cross build in SBT.

Now, the intermediate classes are placed without Scala version directory in SBT build specifically.
It seems causing side effects that are dependent on that path. See, for example, {{git grep \-r "target/scala\-"}}.

To minimise the side effects, we should enable crossPaths back for now.

This is actually an issue in Jenkins jobs as well. See https://github.com/apache/spark/pull/29205 for the analysis made.
Bug: Don't skip pandas UDF tests in IntegratedUDFTestUtils
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jul/20 04:03
Affects Version/s: 3.1.0

Currently, pandas UDF test cases are being skipped as below:

{code}
[info] - udf/postgreSQL/udf-case.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
[info] - udf/postgreSQL/udf-select_having.sql - Scala UDF (2 seconds, 327 milliseconds)
[info] - udf/postgreSQL/udf-select_having.sql - Regular Python UDF (3 seconds, 656 milliseconds)
[info] - udf/postgreSQL/udf-select_having.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
[info] - udf/postgreSQL/udf-select_implicit.sql - Scala UDF (6 seconds, 769 milliseconds)
[info] - udf/postgreSQL/udf-select_implicit.sql - Regular Python UDF (10 seconds, 487 milliseconds)
[info] - udf/postgreSQL/udf-select_implicit.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
[info] - udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF (119 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF (229 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
[info] - udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF (2 seconds, 376 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF (2 seconds, 449 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
[info] - udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF (3 seconds, 634 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF (5 seconds, 899 milliseconds)
[info] - udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3.6]. !!! IGNORED !!!
{code}

in GitBub Actions. We should test them
Bug: Enable doctests in run-tests.py back
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/20 13:52
Affects Version/s: 2.4.6

In run-tests.py script, we're skipping the tests when {{TEST_ONLY_MODULES}} is set. This is mainly because the doctests fail in Github Actions.

We should test it. Currently it fails as below:

{code}
fatal: ambiguous argument 'fc0a1475ef': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
**********************************************************************
File "./dev/run-tests.py", line 75, in __main__.identify_changed_files_from_git_commits
Failed example:
    [x.name for x in determine_modules_for_files(             identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest __main__.identify_changed_files_from_git_commits[0]>", line 1, in <module>
        [x.name for x in determine_modules_for_files(             identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
      File "./dev/run-tests.py", line 87, in identify_changed_files_from_git_commits
        universal_newlines=True)
      File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
        **kwargs).stdout
      File "/usr/lib/python3.6/subprocess.py", line 438, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['git', 'diff-tree', '--no-commit-id', '--name-only', '-r', 'fc0a1475ef']' returned non-zero exit status 128.
fatal: ambiguous argument '50a0496a43': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
{code}

Looks we should fetch the commit to test in GitHub Actions
Bug: Remove leftover from editable mode installation in PIP test
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/20 10:18
Affects Version/s: 2.4.6

Jenkins fails as below:

{code}
Installing dist into virtual env
Processing ./python/dist/pyspark-3.1.0.dev0.tar.gz
Collecting py4j==0.10.9 (from pyspark==3.1.0.dev0)
  Downloading https://files.pythonhosted.org/packages/9e/b6/6a4fb90cd235dc8e265a6a2067f2a2c99f0d91787f06aca4bcf7c23f3f80/py4j-0.10.9-py2.py3-none-any.whl (198kB)
Installing collected packages: py4j, pyspark
  Found existing installation: py4j 0.10.9
    Uninstalling py4j-0.10.9:
      Successfully uninstalled py4j-0.10.9
  Found existing installation: pyspark 3.1.0.dev0
Exception:
Traceback (most recent call last):
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 50, in install_given_reqs
    auto_confirm=True
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 816, in uninstall
    uninstalled_pathset = UninstallPathSet.from_dist(dist)
  File "/home/anaconda/envs/py36/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 505, in from_dist
    '(at %s)' % (link_pointer, dist.project_name, dist.location)
AssertionError: Egg-link /home/jenkins/workspace/SparkPullRequestBuilder@3/python does not match installed 
{code}

Seems like the editable mode from other installations affect. We should better remove the symbolic link.
Bug: Use Hadoop 2.7 profile in AppVeyor SparkR build 
Status: Closed | Priority: Major | Resolution: Invalid
Created: 08/Jul/20 14:15
Affects Version/s: 3.1.0

Hadoop 3 is used by default as of SPARK-32058 but AppVeyor seems failing:

https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/builds/33977845

We should use Hadoop 2 in AppVeyor build for now.
Bug: Fix testCommandAvailable to use POSIX compatible `command -v`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jul/20 05:30
Affects Version/s: 2.1.3

Currently, we run the command to check the existence. This is dangerous and doesn't work sometimes.
{code}
scala> sys.process.Process("cat").run().exitValue()
res0: Int = 0

scala> sys.process.Process("ls").run().exitValue()
LICENSE
NOTICE
bin
doc
lib
man
res1: Int = 0

scala> sys.process.Process("rm").run().exitValue()
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file
res4: Int = 64
{code}

{code}
scala> sys.process.Process("command -v rm").run().exitValue()
/bin/rm
res5: Int = 0
{code}
Bug: [UI] Add failure summary table in stage page
Status: In Progress | Priority: Major | Resolution: 
Created: 13/Jul/20 02:44
Affects Version/s: 3.0.0

When there're many task failure during one stage, it's hard to find failure pattern such as aggregation task failure by exception type and message. If we have such information, we can easily know which type of exception of failure is the root cause of stage failure.
Bug:  Improve the speculation for the inefficient tasks by the task metrics.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Jul/20 07:17
Affects Version/s: 3.0.0

1) Tasks will be speculated when meet certain conditions no matter they are inefficient or not，this would be a huge waste of cluster resources.
2) In production，the speculation task comes  from an efficient one  will be killed finally，which is unnecessary and will waste of cluster resources.
3) So, we should  evaluate whether the task is inefficient by success tasks metrics firstly, and then decide to speculate it or not. The  inefficient task will be speculated and efficient one will not, it is better for the cluster resources.
Bug: Performance regression on TPCH Q18 in 3.0 - ~20% slower than 2.4
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jul/20 07:04
Affects Version/s: 3.0.0

Hi Spark developers,

Testing with the new Spark 3.0.0 here and found some performance regression on TPCH Q18. Spark 2.4 seems can "reuse" the HashAgg results in two SMJ, while Spark 3.0.0 needs to calculate this results twice. I tried to search the documentation but no turnings found on this. 

Here's the SQL diagram for 2.4. The 2nd exchange on the top right corner was reused.   !2.4.png!

Here's the diagram for 3.0. The 1st exchange on top-right corner was reused. 

!3.0.png!
Bug: Make unionByName null-filling behavior work with struct columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jul/20 22:11
Affects Version/s: 3.1.0

https://issues.apache.org/jira/browse/SPARK-29358 added support for unionByName to work when the two datasets didn't necessarily have the same schema, but it does not work with nested columns like structs.
Bug: Shuffled hash join improvement
Status: Open | Priority: Major | Resolution: 
Created: 27/Jul/20 20:10
Affects Version/s: 3.1.0

Shuffled hash join avoids sort compared to sort merge join. This advantage shows up obviously when joining large table in terms of saving CPU and IO (in case of external sort happens). In latest master trunk, shuffled hash join is disabled by default with config "spark.sql.join.preferSortMergeJoin"=true, with favor of reducing risk of OOM. However shuffled hash join could be improved to a better state (validated in our internal fork). Creating this Jira to track overall progress.