av9ash/GitBugs huggingface dataset:
Bug: Add benchmarks for nested structs and arrays for different file formats
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Aug/20 19:54
Affects Version/s: 3.0.0

We had found that Spark performance was slow as compared to PIG on some schemas in our pipelines. On investigation, it was found that Spark performance was slow for nested structs and array'd structs and these cases were not being profiled by the current benchmarks. I have some improvements for ORC (SPARK-32532) and Avro (SPARK-32533) file formats which improve the performance in these cases and will be putting up the PRs soon.
Bug: Improve Avro read/write performance on nested structs and array of structs
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Aug/20 20:01
Affects Version/s: 3.0.0

Have some improvements for Avro file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in SPARK-32531 was able to improve performance on branch-3.0 as follows (measurements in seconds):

Read:
Nested Structs: 75 -> 46
Array of Struct: 47 -> 17

Write
Nested Structs: 147 -> 36
Array of Struct: 139 -> 34

Will be putting up the PR soon with the changes.
Bug: Improve ORC read/write performance on nested structs and array of structs
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Aug/20 19:59
Affects Version/s: 3.0.0

Have some improvements for ORC file format to reduce time taken when reading/writing nested/array'd structs. Using benchmarks in SPARK-32531 was able to improve performance on branch-3.0 as follows (measurements in seconds):

Read:
 Nested Structs: 184 -> 44
 Array of Struct: 66 -> 15

Write
 Nested Structs: 543 -> 39
 Array of Struct: 330 -> 37

Will be putting up the PR soon with the changes.
Bug: Support GroupType parquet mapkey field
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 15:10
Affects Version/s: 2.4.6

I have a parquet file, and the MessageType recorded in the file is:
{code:java}
message parquet_schema {
  optional group value (MAP) {
    repeated group key_value {
      required group key {
        optional binary first (UTF8);
        optional binary middle (UTF8);
        optional binary last (UTF8);
      }
      optional binary value (UTF8);
    }
  }
}{code}
 

Use +spark.read.parquet("000.snappy.parquet")+ to read the file. Spark will throw an exception when converting Parquet MessageType to Spark SQL StructType:
{code:java}
AssertionError(Map key type is expected to be a primitive type, but found...)
{code}
 

Use +spark.read.schema("value MAP<STRUCT<first:STRING, middle:STRING, last:STRING>, STRING>").parquet("000.snappy.parquet")+ to read the file, spark returns the correct result .

According to the parquet project document (https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps), the mapKey in the parquet format does not need to be a primitive type.

 
Note: This parquet file is not written by spark, because spark will write additional sparkSchema string information in the parquet file. When Spark reads, it will directly use the additional sparkSchema information in the file instead of converting Parquet MessageType to Spark SQL StructType.


I will submit a PR later.
Bug: Getting jar vulnerability for kryo-shaded for 4.0 version
Status: Open | Priority: Major | Resolution: 
Created: 28/Aug/20 07:55
Affects Version/s: 2.4.0

We are getting jar vulnerability for kryo 4.0. Report suggesting to upgrade to 5.x and 5.x doesn't have backward compatibility. By doing upgrade we might need to change org.apache.spark.serializer.kryoserializer files and corresponding instance file.
Bug: EmptyHashedRelation$; no valid constructor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/20 07:34
Affects Version/s: 3.0.0

Current EmptyHashedRelation is object and it will cause JavaDeserialization Exception as follow

 
{code:java}
// Stack
20/08/26 11:13:30 WARN [task-result-getter-2] TaskSetManager: Lost task 34.0 in stage 57.0 (TID 18076, emr-worker-5.cluster-183257, executor 18): java
.io.InvalidClassException: org.apache.spark.sql.execution.joins.EmptyHashedRelation$; no valid constructor
        at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:169)
        at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:874)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1572)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$unBlockifyObject$4(TorrentBroadcast.scala:328)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:330)
        at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:249)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:223)
        at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
        at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:218)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1343)

{code}
using case object instead to fix serialization issue. And also change EmptyHashedRelation not to extend NullAwareHashedRelation since it's already being used in other non-NAAJ joins.
Bug: Support File Based spark.authenticate secret in Mesos Backend
Status: In Progress | Priority: Major | Resolution: 
Created: 26/Aug/20 14:28
Affects Version/s: 3.0.0

This is a proposal to support different ways of setting RPC based authentication secret, especially for Mesos Backend. Currently, there is a support for passing secret directly with *spark.authenticate.secret*, but we also need to support file-based secret passing.
Bug: Datetime Pattern F not working as expected
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 12:34
Affects Version/s: 3.0.0

h3. Background

From the [documentation|https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html], the pattern F should give a week of the month.
|*Symbol*|*Meaning*|*Presentation*|*Example*|
|F|week-of-month|number(1)|3|
h3. Test Data

Here is my test data, that is a csv file.
{code:java}
date
2020-08-01
2020-08-02
2020-08-03
2020-08-04
2020-08-05
2020-08-06
2020-08-07
2020-08-08
2020-08-09
2020-08-10 {code}
h3. Steps to the bug

I have tested in the scala spark 3.0.0 and pyspark 3.0.0:
{code:java}
// Spark

df.withColumn("date", to_timestamp('date, "yyyy-MM-dd"))
  .withColumn("week", date_format('date, "F")).show

+-------------------+----+
|               date|week|
+-------------------+----+
|2020-08-01 00:00:00|   1|
|2020-08-02 00:00:00|   2|
|2020-08-03 00:00:00|   3|
|2020-08-04 00:00:00|   4|
|2020-08-05 00:00:00|   5|
|2020-08-06 00:00:00|   6|
|2020-08-07 00:00:00|   7|
|2020-08-08 00:00:00|   1|
|2020-08-09 00:00:00|   2|
|2020-08-10 00:00:00|   3|
+-------------------+----+


# pyspark

df.withColumn('date', to_timestamp('date', 'yyyy-MM-dd')) \
  .withColumn('week', date_format('date', 'F')) \
  .show(10, False)

+-------------------+----+
|date               |week|
+-------------------+----+
|2020-08-01 00:00:00|1   |
|2020-08-02 00:00:00|2   |
|2020-08-03 00:00:00|3   |
|2020-08-04 00:00:00|4   |
|2020-08-05 00:00:00|5   |
|2020-08-06 00:00:00|6   |
|2020-08-07 00:00:00|7   |
|2020-08-08 00:00:00|1   |
|2020-08-09 00:00:00|2   |
|2020-08-10 00:00:00|3   |
+-------------------+----+{code}
h3. Expected result

The `week` column is not the week of the month. It is a day of the week as a number.

  !comment.png!

From my calendar, the first day of August should have 1 for the week-of-month and from 2nd to 8th should have 2 and so on.
{code:java}
+-------------------+----+
|date               |week|
+-------------------+----+
|2020-08-01 00:00:00|1   |
|2020-08-02 00:00:00|2   |
|2020-08-03 00:00:00|2   |
|2020-08-04 00:00:00|2   |
|2020-08-05 00:00:00|2   |
|2020-08-06 00:00:00|2   |
|2020-08-07 00:00:00|2   |
|2020-08-08 00:00:00|2   |
|2020-08-09 00:00:00|3   |
|2020-08-10 00:00:00|3   |
+-------------------+----+{code}
Bug: withField + getField on null structs returns incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 20:52
Affects Version/s: 3.1.0

 

There is a bug in the way the optimizer rule in SimplifyExtractValueOps (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ComplexTypes.scala) is currently coded which yields incorrect results in scenarios like the following: 
{code:java}
sql("SELECT CAST(NULL AS struct<a:int,b:int>) struct_col")
.select($"struct_col".withField("d", lit(4)).getField("d").as("d"))

// currently returns this:
+---+
|d  |
+---+
|4  |
+---+

// when in fact it should return this: 
+----+
|d   |
+----+
|null|
+----+
{code}
Bug: ORC predicate pushdown should work with case-insensitive analysis
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 03:33
Affects Version/s: 3.0.0

Currently ORC predicate pushdown doesn't work with case-insensitive analysis, see SPARK-32622 for the test case.

We should make ORC predicate pushdown work with case-insensitive analysis too.
Bug: Spark-32550 affects the performance of some cases
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 24/Aug/20 11:29
Affects Version/s: 3.1.0

I found that [Spark-32550|https://github.com/apache/spark/pull/29366] affected the performance of some cases, the typical cases is "deterministic cardinality estimation" in 

HyperLogLogPlusPlusSuite when rsd is 0.001, we found the code that is significantly slower is

 

[https://github.com/apache/spark/blob/08b951b1cb58cea2c34703e43202fe7c84725c8a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlusSuite.scala#L41]

 
{code:java}
40：def createBuffer(hll: HyperLogLogPlusPlus): InternalRow = {    
41：  val buffer = new SpecificInternalRow(hll.aggBufferAttributes.map(_.dataType))    
42：  hll.initialize(buffer)    
43：  buffer  
44：}
{code}
 

The size of "hll.aggBufferAttributes" in this case is 209716, the results of comparison before and after spark-32550 merged are as follows:
| |After SPARK-32550 create createBuffer|After SPARK-32550 end to end |Before SPARK-32550 create input|Before SPARK-32550 end to end |
|rsd 0.001, n 1000|52715513243|53004810687|195807999|773977677|
|rsd 0.001, n 5000|51881246165|52519358215|13689949|249974855|
|rsd 0.001, n 10000|52234282788|52374639172|14199071|183452846|
|rsd 0.001, n 50000|55503517122|55664035449|15219394|584477125|
|rsd 0.001, n 100000|51862662845|52116774177|19662834|166483678|
|rsd 0.001, n 500000|51619226715|52183189526|178048012|16681330|
|rsd 0.001, n 1000000|54861366981|54976399142|226178708|18826340|
|rsd 0.001, n 5000000|52023602143|52354615149|388173579|15446409|
|rsd 0.001, n 10000000|53008591660|53601392304|533454460|16033032|
Bug: Remove unused DELETE_ACTION in FileStreamSinkLog
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 04:38
Affects Version/s: 3.1.0

DELETE_ACTION in FileStreamSinkLog has been never used from the introduction. 

While there may be the possible usage on the action, given it hasn't been used for years, it makes more sense to remove the action and restore back with introducing actual usage. This is more realistic option as there has been not much efforts on the area, so it might be unlikely addressed.
Bug: Script Transform DELIMIT  value should be formatted
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 15:29
Affects Version/s: 3.0.0

For SQL

 
{code:java}
SELECT TRANSFORM(a, b, c)
  ROW FORMAT DELIMITED
  FIELDS TERMINATED BY ','
  LINES TERMINATED BY '\n'
  NULL DEFINED AS 'null'
  USING 'cat' AS (a, b, c)
  ROW FORMAT DELIMITED
  FIELDS TERMINATED BY ','
  LINES TERMINATED BY '\n'
  NULL DEFINED AS 'NULL'
FROM testData
{code}
The correct 

TOK_TABLEROWFORMATFIELD should be , nut actually  ','

TOK_TABLEROWFORMATLINES should be \n  but actually '\n'
Bug: Use workflow_dispatch to enable manual test triggers
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/20 09:46
Affects Version/s: 3.1.0

This ticket targets at adding a workflow_dispatch entry in the GitHub Action script (build_and_test.yml). This update can enable developers to run the Spark tests for a specific branch on their own local repository, so I think it might help to check if al the tests can pass before opening a new PR.
Bug: Scrip transformation no-serde mode when column less then output length ,  Use null fill
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 11:03
Affects Version/s: 3.0.0

Scrip transform no-serde mode should pad null value to filling column
{code:java}
hive> SELECT TRANSFORM(a, b)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY '|'
    >   LINES TERMINATED BY '\n'
    >   NULL DEFINED AS 'NULL'
    > USING 'cat' as (a string, b string, c string, d string)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY '|'
    >   LINES TERMINATED BY '\n'
    >   NULL DEFINED AS 'NULL'
    > FROM (
    > select 1 as a, 2 as b
    > ) tmp ;
OK
1	2	NULL	NULL
Time taken: 24.626 seconds, Fetched: 1 row(s)

{code}
Bug: TransportClient getting closed when there are outstanding requests to the server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Aug/20 00:47
Affects Version/s: 3.0.0

The implementation of {{removeBlocks}} and {{getHostLocalDirs}} in {{ExternalBlockStoreClient}} closes the client after processing a response in the callback. 

This is a cached client which will be re-used for other responses. There could be other outstanding request to the shuffle service, so it should not be closed after processing a response. 
Seems like this is a bug introduced with SPARK-27651 and SPARK-27677. 

The older methods  {{registerWithShuffleServer}} and {{fetchBlocks}} didn't close the client.

cc [~attilapiros] [~vanzin] [~mridulm80]
Bug: Bad partitioning in spark jdbc method with parameter lowerBound and upperBound
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 17/Aug/20 02:39
Affects Version/s: 3.0.0

When I use the jdbc methed
{code:java}
def jdbc( url: String, table: String, columnName: String, lowerBound: Long, upperBound: Long, numPartitions: Int, connectionProperties: Properties)
{code}
 
  I am confused by the partitions generated by this method,  for  rows of the first partition aren't limited by the lowerBound and the ones of the last partition are not limited by the upperBound. 
  
 For example, I use the method  as follow:
  
{code:java}
val data = spark.read.jdbc(url, table, "id", 2, 5, 3,buildProperties()) .selectExpr("id","appkey","funnel_name")
data.show(100, false)  
{code}
 

The result partitions info is :

 20/08/05 16:58:59 INFO JDBCRelation: Number of partitions: 3, WHERE clauses of these partitions: `id` < 3 or `id` is null, `id` >= 3 AND `id` < 4, `id` >= 4

The returned data is:
||id|| appkey||funnel_name||
|0|yanshi|test001|
|1|yanshi|test002|
|2|yanshi|test003|
|3|xingkong|test_funnel|
|4|xingkong|test_funnel2|
|5|xingkong|test_funnel3|
|6|donews|test_funnel4|
|7|donews|test_funnel|
|8|donews|test_funnel2|
|9|dami|test_funnel3|
|13|dami|test_funnel4|
|15|xiaoai|test_funnel6|

 

Normally, the clause of the first partition should be " `id` >=2 and `id` < 3 "  because the lowerBound is 2, and the clause of the last partition should be " `id` >= 4 and `id` < 5 ",  but the facts are not.
Bug: Spark 3.1 log(NaN) returns null instead of NaN
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/20 20:03
Affects Version/s: 3.1.0

I was testing Spark 3.1.0 and I noticed that if you take the log(NaN) it now returns a null whereas in Spark 3.0 it returned a NaN.  I'm not an expert in this but I thought NaN was correct.

Spark 3.1.0 Example:

>>> df.selectExpr(["value", "log1p(value)"]).show()

+--------------+-----------------+
|        value|      LOG1P(value)|

+--------------+-----------------+
|-3.4028235E38|              null|
|3.4028235E38|88.72283906194683|
|          0.0|               0.0|
|         -0.0|              -0.0|
|          1.0|0.6931471805599453|
|         -1.0|              null|
|          NaN|              null|

+--------------+-----------------+

 

Spark 3.0.0 example:

 

+-------------+------------------+
| value| LOG1P(value)|
+-------------+------------------+
|-3.4028235E38| null|
| 3.4028235E38| 88.72283906194683|
| 0.0| 0.0|
| -0.0| -0.0|
| 1.0|0.6931471805599453|
| -1.0| null|
| NaN| NaN|
+-------------+------------------+

 

Note it also does the same for log1p, log2, log10
Bug: Race condition in MapOutputTracker.getStatistics
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 20/Aug/20 17:04
Affects Version/s: 3.0.0

MapOutputTracker.getStatistics builds an array of partition sizes for a shuffle id and in some cases uses multiple threads running in parallel to update this array. This code is not thread-safe and the output is non-deterministic when there are multiple MapStatus entries for the same partition.

We have unit tests such as the skewed join tests in AdaptiveQueryExecSuite that depend on the output being deterministic, and intermittent failures in these tests led me to track this bug down.

The issue is trivial to fix by using an AtomicLong when building the array of partition sizes.
Bug: Not able to Deserialize Mleap bundle with latest spark configuration
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 18/Aug/20 12:17
Affects Version/s: 3.0.0

Hi,

 

I am using Databricks cluster with configuration as Spark: 3.0.0 and Scala: 2.12 and want to create Machine Learning pipelines using MLeap - latest version: 0.16.1.

In my pipeline, I have created ensemble model. I am to serialize the model to a bundle.

But the issue comes while deserializing it. Below is the error I am facing:
_java.lang.NoSuchMethodError: org.apache.spark.mllib.tree.impurity.ImpurityCalculator$.getCalculator(Ljava/lang/String;[D)Lorg/apache/spark/mllib/tree/impurity/ImpurityCalculator;_
 
I tried using another cluster with the configuration as - spark: 2.4.5, scala: 2.11 and MLeap: 0.16.0. There it is working fine.

Please suggest.
Bug: Script Transformation no-serde `TOK_TABLEROWFORMATLINES` only support `\n`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Aug/20 15:26
Affects Version/s: 3.0.0

Scrip Transform no-serde ({{ROW FORMAT DELIMITED}}) mode {{LINE TERMINNATED BY}}
only support {{\n}}.

Tested in hive :
Hive 1.1
[!https://user-images.githubusercontent.com/46485123/90309510-ce82a180-df1b-11ea-96ab-56e2b3229489.png!|https://user-images.githubusercontent.com/46485123/90309510-ce82a180-df1b-11ea-96ab-56e2b3229489.png]

Hive 2.3.7
[!https://user-images.githubusercontent.com/46485123/90309504-c88cc080-df1b-11ea-853e-8f65e9ed2375.png!|https://user-images.githubusercontent.com/46485123/90309504-c88cc080-df1b-11ea-853e-8f65e9ed2375.png]
Bug: Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 16:45
Affects Version/s: 3.1.0

The decommissioned host/executor should be considered as inactive in TaskSchedulerImpl, so the TaskSetManager won't include them as preferred location since the decommissioned executors will no longer be used to assign any tasks.
Bug: Unify task name in some logs between driver and executor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Aug/20 09:46
Affects Version/s: 3.1.0

The task name (e.g. "{{task 0.0 in stage 1.0 (TID 1)"}}) has already been widely used after SPARK-2317. But some places are still using the {{TID}} only or the name without TID as the task name in logs. It's not convenient when someone needs to track a task for debugging.
Bug: decommission switch configuration should have the highest hierarchy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Aug/20 13:59
Affects Version/s: 3.1.0

Decommission has been supported in Standalone and k8s yet and will be supported in Yarn in the future. Therefore, the switch configuration should have the highest hierarchy rather than belongs to Standalone's Worker(spark.worker.decommission.enabled).
Bug: CoarseGrainedSchedulerBackend.maxNumConcurrentTasks should consider all kinds of resources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Aug/20 05:53
Affects Version/s: 3.0.0

Currently, CoarseGrainedSchedulerBackend.maxNumConcurrentTasks only considers the CPU for the max concurrent tasks. This can cause the application to hang when a barrier stage requires extra custom resources but the cluster doesn't have enough corresponding resources. Because, without the checking for other custom resources in maxNumConcurrentTasks, the barrier stage can be submitted to the TaskSchedulerImpl. But the TaskSchedulerImpl can not launch tasks for the barrier stage due to the insufficient task slots calculated by calculateAvailableSlots(which does check all kinds of resources).
Bug: NAAJ support for ShuffleHashJoin when AQE is on
Status: In Progress | Priority: Major | Resolution: 
Created: 18/Aug/20 02:21
Affects Version/s: 3.0.0

In SPARK-32290, we managed to optimize NAAJ scenario from BNLJ to BHJ, but skipped the checking the BuildSide plan with *spark.sql.autoBroadcastJoinThreshold* parameter, which means in very bad case, BuildSide Plan being too big might cause Driver OOM. So the NAAJ support for  ShuffledHashJoin is important as well.

 

The support of SHJ for NAAJ has some difficulties in NullKey scenario, as for normal HashedRelation and EmtpyHashedRelation, the code logical should be the same when it comes to BHJ and SHJ, but if NullKey exists in global BuildSide data, and only one partition could be built into EmptyHashedRelationWithAllNullKeys, and this partition was not able to do *fast stop* for the entire RDD. So after offline talked with some committer and discussion, decided to support NAAJ for SHJ when AQE is on, because when AQE is on, Shuffle will be pre-executed, and we should be able to know that whether the BuildSide contains NullKey or not before the actual JOIN executed.

 

Basically, In NAAJ SHJ Implementation, we collected information whether BuildSide is Empty or contains NullKey, and keep these information in ShuffleExchangeExec Metrics, and during AQE, we rewritten these two case into LocalTableScan and StreamedSidePlan to Avoid NAAJ, as for the normal relation, we processed it in Distributed style.