av9ash/GitBugs huggingface dataset:
Bug: Enable streaming-kinesis-asl tests in Github Action CI
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/20 13:39
Affects Version/s: 2.4.6

See https://github.com/HyukjinKwon/spark/pull/4. Kinesis tests depends on external Amazon kinesis service.

We should have a way to run it optionally. Currently, this is not being run in Github Actions.
Bug: Bloom Filter Join
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jul/20 05:58
Affects Version/s: 3.1.0

We can improve the performance of some joins by pre-filtering one side of a join using a Bloom filter and IN predicate generated from the values from the other side of the join.
 For example:[tpcds/q16.sql|https://github.com/apache/spark/blob/a78d6ce376edf2a8836e01f47b9dff5371058d4c/sql/core/src/test/resources/tpcds/q16.sql]. [Before this optimization|https://issues.apache.org/jira/secure/attachment/13007418/q16-default.jpg]. [After this optimization|https://issues.apache.org/jira/secure/attachment/13007416/q16-bloom-filter.jpg].

*Query Performance Benchmarks: TPC-DS Performance Evaluation*
 Our setup for running TPC-DS benchmark was as follows: TPC-DS 5T and Partitioned Parquet table

 
|Query|Default(Seconds)|Enable Bloom Filter Join(Seconds)|
|tpcds q16|84|46|
|tpcds q36|29|21|
|tpcds q57|39|28|
|tpcds q94|42|34|
|tpcds q95|306|288|
Bug: Null Aware Anti Join Optimize Support Multi-Column
Status: Resolved | Priority: Major | Resolution: Later
Created: 30/Jul/20 10:17
Affects Version/s: 3.0.0

In Issue SPARK-32290, we managed to optimize BroadcastNestedLoopJoin into BroadcastHashJoin within the Single-Column NAAJ scenario, by using hash lookup instead of loop join. 

It's simple to just fulfill a "NOT IN" logical when it's a single key, but multi-column not in is much more complicated with all these null aware comparison.

FYI, code logical for single and multi column is defined at

~/sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-single-column.sql

~/sql/core/src/test/resources/sql-tests/inputs/subquery/in-subquery/not-in-unit-tests-multi-column.sql

 

Hence, proposed with a New type HashedRelation, NullAwareHashedRelation. 

For NullAwareHashedRelation
 # it will not skip anyNullColumn key like LongHashedRelation and UnsafeHashedRelation do.
 # while building NullAwareHashedRelation, will put extra keys into the relation, just to make null aware columns comparison in hash lookup style.

the duplication would be 2^numKeys - 1 times, for example, if we are to support NAAJ with 3 column join key, the buildSide would be expanded into (2^3 - 1) times, 7X.

For example, if there is a UnsafeRow key (1,2,3)

In NullAware Mode, it should be expanded into 7 keys with extra C(3,1), C(3,2) combinations, within the combinations, we duplicated these record with null padding as following.

Original record

(1,2,3)

Extra record to be appended into NullAwareHashedRelation

(null, 2, 3) (1, null, 3) (1, 2, null)
 (null, null, 3) (null, 2, null) (1, null, null))

with the expanded data we can extract a common pattern for both single and multi column. allNull refer to a unsafeRow which has all null columns.
 * buildSide is empty input => return all rows
 * allNullColumnKey Exists In buildSide input => reject all rows
 * if streamedSideRow.allNull is true => drop the row
 * if streamedSideRow.allNull is false & findMatch in NullAwareHashedRelation => drop the row
 * if streamedSideRow.allNull is false & notFindMatch in NullAwareHashedRelation => return the row
Bug: User Guide - Monitoring
Status: Resolved | Priority: Major | Resolution: Later
Created: 06/Jul/20 08:35
Affects Version/s: 3.1.0

Monitoring. We should focus on how to monitor PySpark jobs.

- Custom Worker, see also https://github.com/apache/spark/tree/master/python/test_coverage to enable test coverage that include worker sides too.
- Sentry Support \(?\) https://blog.sentry.io/2019/11/12/sentry-for-data-error-monitoring-with-pyspark
- Link back https://spark.apache.org/docs/latest/monitoring.html . 
- ...
Bug: User Guide - PySpark Usage Guide for Pandas with Apache Arrow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 06:44
Affects Version/s: 3.1.0

Port https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html to examples.
See also https://hyukjin-spark.readthedocs.io/en/latest/user_guide/arrow-spark.html
Bug: Make readability better in the test result logs
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 09/Jul/20 14:05
Affects Version/s: 2.4.6

Currently, the readability in the logs are not really good. For example, see https://pipelines.actions.githubusercontent.com/gik0C3if0ep5i8iNpgFlcJRQk9UyifmoD6XvJANMVttkEP5xje/_apis/pipelines/1/runs/564/signedlogcontent/4?urlExpires=2020-07-09T14%3A05%3A52.5110439Z&urlSigningMethod=HMACV1&urlSignature=gMGczJ8vtNPeQFE0GpjMxSS1BGq14RJLXUfjsLnaX7s%3D

We should have a way to easily see the failed test cases.
Bug: Binder Integration
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 05:35
Affects Version/s: 3.1.0

For example,

https://github.com/databricks/koalas

https://mybinder.org/v2/gh/databricks/koalas/master?filepath=docs%2Fsource%2Fgetting_started%2F10min.ipynb
Bug: Getting Started - Quickstart
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 06:42
Affects Version/s: 3.1.0

Example:

https://koalas.readthedocs.io/en/latest/getting_started/10min.html
https://pandas.pydata.org/docs/getting_started/intro_tutorials/index.html
https://pandas.pydata.org/docs/getting_started/10min.html
Bug: Partially pushed partition filters are not explained
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jul/20 16:49
Affects Version/s: 3.1.0

How to reproduce this issue:
{code:scala}
spark.sql(
  s"""
     |CREATE TABLE t(i INT, p STRING)
     |USING parquet
     |PARTITIONED BY (p)""".stripMargin)

spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp")
for (part <- Seq(1, 2, 3, 4)) {
  sql(s"""
         |INSERT OVERWRITE TABLE t PARTITION (p='$part')
         |SELECT col FROM temp""".stripMargin)
}

spark.sql("SELECT * FROM t WHERE  WHERE (p = '1' AND i = 1) OR (p = '2' and i = 2)").explain
{code}

We have pushed down {{p = '1' or p = '2'}} since SPARK-28169, but this pushed down filter not in explain

{noformat}
== Physical Plan ==
*(1) Filter (((p#21 = 1) AND (i#20 = 1)) OR ((p#21 = 2) AND (i#20 = 2)))
+- *(1) ColumnarToRow
   +- FileScan parquet default.t[i#20,p#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32289/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<i:int>

{noformat}
Bug: Installs qpdf package for CRAN check in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 11:58
Affects Version/s: 3.1.0

CRAN check fails as below:

{code}
...
 WARNING
‘qpdf’ is needed for checks on size reduction of PDFs
...
Status: 1 WARNING, 1 NOTE
See
  ‘/home/runner/work/spark/spark/R/SparkR.Rcheck/00check.log’
for details.
{code}

Looks we should install {{qpdf}}.
Bug: Bug in load-spark-env.cmd  with Spark 3.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jul/20 09:55
Affects Version/s: 3.0.0

spark-env.cmd  which is located in conf  is not loaded by load-spark-env.cmd.

 

*How to reproduce:*

1) download spark 3.0.0 without hadoop and extract it

2) put a file conf/spark-env.cmd with the following contents (paths are relative to where my hadoop is - in C:\opt\hadoop\hadoop-3.2.1, you may need to change):

 

SET JAVA_HOME=C:\opt\Java\jdk1.8.0_241
 SET HADOOP_HOME=C:\opt\hadoop\hadoop-3.2.1
 SET HADOOP_CONF_DIR=C:\opt\hadoop\hadoop-3.2.1\conf
 SET SPARK_DIST_CLASSPATH=C:\opt\hadoop\hadoop-3.2.1\etc\hadoop;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\common;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\common\lib*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\common*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\hdfs;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\hdfs\lib*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\hdfs*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\yarn;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\yarn\lib*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\yarn*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\mapreduce\lib*;C:\opt\hadoop\hadoop-3.2.1\share\hadoop\mapreduce*

 

3) go to the bin directory and run pyspark.   You will get an error that log4j can't be found, etc. (reason: the environment was not loaded indeed, it doesn't see where hadoop with all its jars is).

 

*How to fix:*

just take the load-spark-env.cmd  from Spark version 2.4.3, and everything will work.

[UPDATE]:  I attached a fixed version of load-spark-env.cmd  that works fine.

 

*What is the difference?*

I am not a good specialist in Windows batch, but doing a function

:LoadSparkEnv
 if exist "%SPARK_CONF_DIR%\spark-env.cmd" (
  call "%SPARK_CONF_DIR%\spark-env.cmd"
 )

and then calling it (as it was in 2.4.3) helps.
Bug: Add PySpark support for nested timestamps with arrow
Status: In Progress | Priority: Major | Resolution: 
Created: 12/Jul/20 22:42
Affects Version/s: 3.0.0

Currently with arrow optimizations, there is post-processing done in pandas for timestamp columns to localize timezone. This is not done for nested columns with timestamps such as StructType or ArrayType.

Adding support for this is needed for Apache Arrow 1.0.0 upgrade due to use of structs with timestamps in groupedby key over a window.

As a simple first step, timestamps with 1 level nesting could be done first and this will satisfy the immediate need.

NOTE: with Arrow 1.0.0, it might be possible to do the timezone processing with pyarrow.array.cast, which could be easier done than in pandas.
Bug: Development - Setting up PyCharm
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 08:43
Affects Version/s: 3.1.0

PyCharm is probably one of the most widely used IDE for Python development. We should document a standard way so users can easily set up.

Very rough example: https://hyukjin-spark.readthedocs.io/en/latest/development/developer-tools.html#setup-pycharm-with-pyspark but it should be more in details.
Bug: Development - Contribution Guide
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jul/20 08:45
Affects Version/s: 3.1.0

Will be similar with http://spark.apache.org/contributing.html but avoid duplications with more details.

- Style Guide: PEP8 but there are a couple of exceptions such as https://github.com/apache/spark/blob/master/dev/tox.ini#L17.
- Bug Reports: JIRA. Maybe we should just point back the link,  http://spark.apache.org/contributing.html
- Contribution Workflow: e.g.) https://pandas.pydata.org/docs/development/contributing.html
- Documentation Contribution: e.g.) https://pandas.pydata.org/docs/development/contributing.html
- Code Contribution: e.g.) https://pandas.pydata.org/docs/development/contributing.html
Bug: Hide JVM traceback for SparkUpgradeException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/20 01:12
Affects Version/s: 3.1.0

We added {{SparkUpgradeException}} which the JVM traceback is pretty useless. See also https://github.com/apache/spark/pull/28736/files#r449184881

It should better also whitelist and hide JVM traceback.
Bug: Include GitHub Action file as the changes in testing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 10:46
Affects Version/s: 3.1.0

https://github.com/apache/spark/pull/26556 excluded `.github/workflows/master.yml`. So no tests run if the GitHub Actions configuration file is changed.

As of SPARK-32245, we now run the regular tests via the testing script. We should include it.
Bug: Manually install R instead of using setup-r in GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 10:05
Affects Version/s: 3.1.0

Looks like {{setup-r}} fails to install specific R version. See the debugging logs at https://github.com/HyukjinKwon/spark/pull/15:

{code}
ERROR: this R is version 3.4.4, package 'SparkR' requires R >= 3.5
[error] running /home/runner/work/spark/spark/R/install-dev.sh ; received return code 1
##[error]Process completed with exit code 10.
{code}

We should maybe just manually install R.
Bug: Do not install SparkR in test-only mode in testing script
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Jul/20 07:24
Affects Version/s: 3.1.0

Currently GitHub Actions builds are failed as below:

{code}
ERROR: this R is version 3.4.4, package 'SparkR' requires R >= 3.5
[error] running /home/runner/work/spark/spark/R/install-dev.sh ; received return code 1
##[error]Process completed with exit code 10.
{code}

https://github.com/apache/spark/runs/926437963

Looks GitHub Actions has R 3.4.4 installed by default; however, R 3.4 was dropped as of SPARK-32073. When SparkR tests are not needed, GitHub Actions does not install R 3.6 and it caused the test failure.

In fact, SparkR is installed in case of running R linter, see SPARK-8505; however, we don't run the linters in test-only mode at testing script.

We can safely skip it in test-only mode in our testing script.
Bug: java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 29/Jul/20 04:36
Affects Version/s: 3.0.0

When I use the command to compile spark-core_2.12 module, and then use the spark-core_2.12-3.0.0.jar instead of /jars/spark-core_2.12-3.0.0.jar, I will report an error (without making any code changes)

command: ./build/mvn -pl :spark-core_2.12 -Pyarn -Phive -Phive-thriftserver -Dhadoop.version=2.7.4 -DskipTests clean package
version: spark-3.0.0
jdk: 1.8
Bug: spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 29/Jul/20 11:28
Affects Version/s: 2.4.0

hi:

the problem happened to me is that I got a message that error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))] in Spark-shell,  but the same function can be processed in JUnit test
Bug: Error message to show the schema mismatch in gapply with Arrow vectorization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Jul/20 07:39
Affects Version/s: 3.0.0

Currently, the error message is confusing when the output schema type is not matched with the actual R DataFrame in gapply:

{code}
./bin/sparkR --conf spark.sql.execution.arrow.sparkr.enabled=true
{code}

{code}
df <- createDataFrame(list(list(a=1L, b="2")))
count(gapply(df, "a", function(key, group) { group }, structType("a int, b int")))
{code}

{code}
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 43 in stage 2.0 failed 1 times, most recent failure: Lost task 43.0 in stage 2.0 (TID 2, 192.168.35.193, executor driver): java.lang.UnsupportedOperationException
	at org.apache.spark.sql.vectorized.ArrowColumnVector$ArrowVectorAccessor.getInt(ArrowColumnVector.java:212)
	...
{code}

We should probably also document that the type should be matched always.
Bug: Spark SQL throw  Illegal character exception when load certain abnormal path of HDFS 
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jul/20 08:53
Affects Version/s: 2.4.3

In the distributed hdfs storage system，Space and other special character are allowed in the path：
{code:java}
hdfs://ns1/tmp2/hive-staging/hadoop_hive_2020-07-06_17-31-29_139_7042265710400397740-1/-ext-10000/test_table=2020-06-17 18%3A00%3A00/part-00000-84396c4e-ba05-4936-afc7-db46c4251bfa.c000

{code}
When we load data by using
{code:java}
org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
org.apache.spark.sql.execution.datasources.orcOrcFileFormat.scala
org.apache.spark.sql.hive.orc.OrcFileFormat {code}
, exception may throw as below:
{code:java}
Caused by: java.net.URISyntaxException: Illegal character in path at index 136: hdfs://ns1/tmp2/hive-staging/hadoop_hive_2020-07-06_17-31-29_139_7042265710400397740-1/-ext-10000/test_table=2020-06-17 18%3A00%3A00/part-00000-84396c4e-ba05-4936-afc7-db46c4251bfa.c000
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parseHierarchical(URI.java:3105)
at java.net.URI$Parser.parse(URI.java:3053)
at java.net.URI.<init>(URI.java:588)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
anonfun$buildReaderWithPartitionValues$1.apply(ParquetFileFormat.scala:356)atorg.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
anonfunbuildReaderWithPartitionValues1.apply(ParquetFileFormat.scala:352)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.orgapachesparksqlexecutiondatasourcesFileScanRDD
anon
readCurrentFile(FileScanRDD.scala:124)
at org.apache.spark.sql.execution.datasources.FileScanRDD
anon$1.nextIterator(FileScanRDD.scala:177)atorg.apache.spark.sql.execution.datasources.FileScanRDD
anon1.hasNext(FileScanRDD.scala:101)atorg.apache.spark.sql.execution.datasources.FileFormatWriteranonfunorgapachesparksqlexecutiondatasourcesFileFormatWriter
executeTask$3.apply(FileFormatWriter.scala:252)atorg.apache.spark.sql.execution.datasources.FileFormatWriter
anonfunorgapachesparksqlexecutiondatasourcesFileFormatWriterexecuteTask3.apply(FileFormatWriter.scala:250)
at org.apache.spark.util.Utils.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)atorg.apache.spark.sql.execution.datasources.FileFormatWriter.orgapachesparksqlexecutiondatasourcesFileFormatWriter$$executeTask(FileFormatWriter.scala:256)
... 10 more

{code}
 Hdfs  has provided serveral  construct function to build path:

[https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java|https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Path.java#L132]

We could fall back to  construct a path from a String rather than URI.
Bug: saveAsTable deletes all files in path
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 07/Jul/20 21:10
Affects Version/s: 3.0.0

The problem is presented in the following link:

[https://stackoverflow.com/questions/62782637/saveastable-can-delete-all-my-files-in-desktop?noredirect=1#comment111026138_62782637]

Apparently, without no warning, all files is desktop where deleted after writing a file.
There is no warning in Pyspark that the "Path"  parameter makes that problem.
Bug: add mutiple filter in rdd function
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 17/Jul/20 03:48
Affects Version/s: 2.4.6

when i use spark rdd . i often use to read kafka data.And kafka data has lots of kinds data set.

I filter these rdd  by kafka key , then i can use Array[rdd] to fill every topic rdd. 

But at that ,  i use rdd.filter,that  will generate more than one stage.Data will process by many task, that consume too many time. And it is not necessary.

i hope add multiple  filter function not rdd.filter ,that will return Array[RDD] in one stage by dividing all  mixture data  RDD to single data set RDD .

function like Array[RDD]=rdd.multiplefilter(setcondition).
Bug: Upgrade Apache Arrow to 1.0.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jul/20 17:42
Affects Version/s: 3.0.0

Apache Arrow will soon release v1.0.0 which provides backward/forward compatibility guarantees as well as a number of fixes and improvements. This will upgrade the Java artifact and PySpark API. Although PySpark will not need special changes, it might be a good idea to bump up minimum supported version and CI testing.

TBD: list of important improvements and fixes
Bug: TRANSFORM when schema less should keep same with hive
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jul/20 08:47
Affects Version/s: 3.0.0

Hive transform without schema

 
{code:java}
hive> create table t (c0 int, c1 int, c2 int);
hive> INSERT INTO t VALUES (1, 1, 1);
hive> INSERT INTO t VALUES (2, 2, 2);
hive> CREATE VIEW v AS SELECT TRANSFORM(c0, c1, c2) USING 'cat' FROM t;

hive> DESCRIBE v;
key                 	string              	                    
value               	string              	                    

hive> SELECT * FROM v;
1	1	1
2	2	2

hive> SELECT key FROM v;
1
2

hive> SELECT value FROM v;
1	1
2	2{code}
Spark
{code:java}
hive> create table t (c0 int, c1 int, c2 int); 
hive> INSERT INTO t VALUES (1, 1, 1); 
hive> INSERT INTO t VALUES (2, 2, 2); 
hive> CREATE VIEW v AS SELECT TRANSFORM(c0, c1, c2) USING 'cat' FROM t; 

hive> SELECT * FROM v; 
1   11
2   22 {code}
Bug: TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 22/Jul/20 10:18
Affects Version/s: 3.0.0

convert as string to support.

UTD 

[https://github.com/apache/spark/pull/29085/commits/e367c0544298f6639e8898029eab5e29ea1f91ea]

 

https://github.com/apache/spark/pull/29085#issuecomment-660580865
Bug: 使用Structured Streaming窗口统计不能实现topN
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 18/Jul/20 10:32
Affects Version/s: 2.4.0

{quote} 代码如下：

Dataset<Row> result = ds.groupBy(functions.window(ds.col("_c1"), "1 minutes", "1 minutes"), ds.col("_c0")).agg(functions.first("_c2"), functions.last("_c2")).sort("_c1");

例如：_c1是数据时间，窗口统计之后还想根据时间排序，并取出第一条和最后一条，却实现不了! 感觉spark的Structured Streaming实现不了。请问这个需求怎么 实现呢？
{quote}
Bug: how spark collects non-match results after performing broadcast left outer join
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Jul/20 19:02
Affects Version/s: 2.4.0

Does anybody know how spark collects non-match results after performing broadcast hash left outer join?

Suppose we have 4 nodes. 1 driver and 3 executors. We broadcast the left table. After left outer join is performed in each executor, how does spark recognize which records have not been matched, and how does it collect them to the final result?
Bug: Test coverage of HiveScripTransformationExec
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jul/20 02:57
Affects Version/s: 3.0.0

Add more test case for script transformation to make use case more clear. Doing below things
{code:java}
1. Extract common test case (no serde) to BasicScriptTransformationExecSuite
2. Add more test case for no serde mode about supported data type and behavior in `BasicScriptTransformationExecSuite`
3. Add more test case for hive serde mode about supported type and behavior in `HiveScriptTransformationExecSuite`
{code}
Bug: Spark Web UI shows Nan undefined in Shuffle Read Size / Records	
Status: Open | Priority: Major | Resolution: 
Created: 24/Jul/20 23:12
Affects Version/s: 3.0.0

Spark Web UI in Spark3 shows *NaN undefined / 18172517*

in *Shuffle Read Size / Records* row

!image-2020-07-24-23-12-44-500.png!
Bug: Remove SPARK_SCALA_VERSION environment and let load-spark-env scripts detect it in AppVeyor
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 27/Jul/20 03:59
Affects Version/s: 3.1.0

{code}
.\bin\spark-submit2.cmd --driver-java-options "-Dlog4j.configuration=file:///%CD:\=/%/R/log4j.properties" --conf spark.hadoop.fs.defaultFS="file:///" R\pkg\tests\run-all.R
"Presence of build for multiple Scala versions detected ( and )."
"Remove one of them or, set SPARK_SCALA_VERSION= in \spark-env.cmd."
"Visit  for more details about setting environment variables in spark-env.cmd."
"Either clean one of them or, set SPARK_SCALA_VERSION in spark-env.cmd."
Command exited with code 1
{code}

The load-spark-env script fails as below in AppVeyor. It was temporarily explicitly set but we should remove and detect it automatically in this script.

Possibly related to SPARK-26132, SPARK-32227 and SPARK-32434