av9ash/GitBugs huggingface dataset:
Bug: Performance regression for joins in Spark 3.3 vs Spark 3.2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/22 14:38
Affects Version/s: 3.3.0

When running the TPC-DS benchmarks using a DSv2 datasource in Spark 3.3, a performance regression vs Spark 3.2 was discovered. More specifically, it appears as if {_}EnsureRequirements.ensureDistributionAndOrdering{_}() no longer enforces a minimum number of partitions for a join distribution in some cases. This impacts DSv2 datasources, because if a scan has only a single read partition {_}DataSourceV2ScanExecBase.outputPartitioning{_}() returns a _SinglePartition_ instance. The _SinglePartition_ creates a {_}SinglePartitionShuffleSpec{_}, and {_}SinglePartitionShuffleSpec.canCreatePartitioning{_}() returns true.

Because {_}canCreatePartitioning{_}() returns true in this case, {_}EnsureRequirements.ensureDistributionAndOrdering{_}() won't enforce minimum parallelism and also will favor the single partition when considering the best distribution candidate. Ultimately this results in a single partition being selected for the join distribution, even if the other side of the join is a large table with many partitions. This can seriously impact performance of the join.

Spark 3.2 enforces minimum parallelism differently in {_}ensureDistributionAndOrdering{_}() and thus does not suffer from this issue. It will shuffle both sides of the join to enforce parallelism.

In the TPC-DS benchmark, some queries affected include 14a and 14b. This can also be demonstrated using a simple query, for example:

{{select ics.i_item_sk from catalog_sales cs join item ics on cs.cs_item_sk = ics.i_item_sk}}

...where _item_ is a small table that is read into one partition, and _catalog_sales_ is a large table. These tables are part of the TPC-DS but you can create your own. Also, to demonstrate the issue, you may need to turn off broadcast joins though that is not required for this issue to occur, it happens when running the TPC-DS with broadcast setting at default.

Attached is the plan for this query in Spark 3.2 and in Spark 3.3. The plan shows how in Spark 3.2, the join parallelism of 200 is reached by inserting an exchange after the item table scan. In Spark 3.3, no such exchange is inserted and the join parallelism is 1.
Bug: Check error classes in SparkSqlParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 11:37
Affects Version/s: 3.4.0

Check error classes in SparkSqlParserSuite by using checkError(). For instance, replace

{code:scala}
intercept("SET spark.sql.key value", expectedErrMsg)
{code}
by
{code:scala}
checkError(
      exception = parseException("SET spark.sql.key value"),
      errorClass = "...",
      parameters = Map.empty,
      context = ...)
{code}
at https://github.com/apache/spark/blob/7309e76d8b95e306d6f3d2f611316b748949e9cf/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala#L113
Bug: Check error classes in CreateNamespaceParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Oct/22 11:40
Affects Version/s: 3.4.0

Check error classes in CreateNamespaceParserSuite by using checkError(). For instance, replace

{code:scala}
    intercept(
      "CREATE NAMESPACE a.b.c WITH PROPERTIES('key_without_value', 'key_with_value'='x')",
      "Operation not allowed: Values must be specified for key(s): [key_without_value]")
{code}
by
{code:scala}
checkError(
      exception = parseException( "CREATE NAMESPACE a.b.c WITH PROPERTIES('key_without_value', 'key_with_value'='x')"),
      errorClass = "...",
      parameters = Map.empty,
      context = ...)
{code}
at https://github.com/apache/spark/blob/35d00df9bba7238ad4f409999617fae4d04ddbfd/sql/core/src/test/scala/org/apache/spark/sql/execution/command/CreateNamespaceParserSuite.scala#L75-L77
Bug: Refactor checkCorrelationsInSubquery
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Oct/22 19:12
Affects Version/s: 3.4.0

Currently, the logic in `checkCorrelationsInSubquery` is inefficient and difficult to understand. It uses `foreachUp` to traverse the subquery plan tree, and traverses down an entire subtree of a plan node to check whether it contains any outer references. We can use recursion instead to traverse the plan tree only once.
Bug: Check error classes in ErrorParserSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Oct/22 16:51
Affects Version/s: 3.4.0

Check error classes in ErrorParserSuite by using checkError(). For instance, replace

{code:scala}
intercept("select *\nfrom r\norder by q\ncluster by q", 3, 0, 11,
      "Combination of ORDER BY/SORT BY/DISTRIBUTE BY/CLUSTER BY is not supported",
      "^^^")
{code}
by
{code:scala}
checkError(
      exception = parseException("select *\nfrom r\norder by q\ncluster by q"),
      errorClass = "_LEGACY_ERROR_TEMP_0011",
      parameters = Map.empty,
      context = ExpectedContext(fragment = "order by q\ncluster by q", start = 16, stop = 38))
{code}
at https://github.com/apache/spark/blob/96ed6dc3e38b22b4aefc41e20ba7c953f8f2251e/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ErrorParserSuite.scala#L79-L83
Bug: Infer date type for Json schema inference
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 04/Oct/22 18:18
Affects Version/s: 3.2.1

1. If a column contains only dates, it should be of “date” type in the inferred schema
 * If the date format and the timestamp format are identical (e.g. both are yyyy/mm/dd), entries will default to being interpreted as Date

2. If a column contains dates and timestamps, it should be of “timestamp” type in the inferred schema

 

A similar issue was opened in the past but was reverted due to the lack of strict pattern matching.
Bug: Refactor File Data Source Options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/22 17:42
Affects Version/s: 3.4.0

Currently for each file data source, all options are placed sparsely in the options class and there is no clear list of all options supported. As more and more options are added, the readability gets worse. Thus, we want to refactor those codes so that
 # we can easily get a list of supported options for each data source
 # enforce better practice for adding new options going forwards.
Bug: Issue with spark converting Row to Json using Scala 2.13
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/22 17:42
Affects Version/s: 3.3.0

h2. *Note: This issue can be reproduced only using Scala 2.13*

When I'm trying to convert the Row to a json to publish it, i'm getting this following error

!image-2022-10-07-19-43-42-232.png!

I tried to investigate and I found that the issue is in the matching.

!image-2022-10-07-19-43-51-892.png!

The type `ArraySeq` is not matched in `Row` class.

!image-2022-10-07-19-44-00-332.png!

This is the definition of my field

!image-2022-10-07-19-44-09-972.png!

And an example of it 

 
{code:json}
{
...
Codes: ["Test", "Spark", "Json"]
...
}{code}
The Scala version I'm using is `2.13.9`
Bug: Spark-UI jobs status not updating under high concurrency scenario
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Oct/22 11:51
Affects Version/s: 2.4.5

In a high concurrency scenario, we can see from the spark-ui that there are a lot of jobs running all the time, but only a few sql are actually running

As you can see from the attachment, there are many active jobs on the spark-ui. In fact, I stopped submitting jobs at 30 minutes, and after 14 minutes, there are still many jobs on the spark-ui whose status cannot be updated, and which have actually completed

To solve the problem, I tried to set spark.scheduler.listenerbus.eventqueue.capacity = 600000, but it didn't work
Bug: CTAS should respect TBLPROPERTIES during execution
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Oct/22 10:49
Affects Version/s: 3.1.3

Currently, Spark `CTAS` has inconsistent behavior when we use `STORED AS` and `USING`.

{code}
spark-sql> CREATE TABLE t1 STORED AS PARQUET TBLPROPERTIES (parquet.compression 'zstd') AS SELECT 1;

$ ls spark-warehouse/t1
_SUCCESS
part-00000-c28a99f0-a88f-448d-a60d-e8204bf2f3a7-c000.zstd.parquet
{code}

{code}
spark-sql> CREATE TABLE t2 USING PARQUET TBLPROPERTIES (parquet.compression 'zstd') AS SELECT 1;

$ ls spark-warehouse/t2
_SUCCESS
part-00000-3c5853d0-308d-4571-86c3-3e1a31eacfe6-c000.snappy.parquet
{code}
Bug: Relax the restrictions of broadcast join on bloom filter
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Oct/22 07:57
Affects Version/s: 3.4.0

Currently, if the creation side of bloom filter could be broadcasted, Spark cannot inject a bloom filter or InSunquery filter into the application side.
In fact, although we use broadcast join and improves the performance, compared with the reduction of data volume, it is still worse if the application side have many rows.
We just need the creation side is small enough.
Bug: Support preferring shuffled hash join thought LocalMapThreshold is less than advisory partition size
Status: In Progress | Priority: Major | Resolution: 
Created: 09/Oct/22 07:56
Affects Version/s: 3.4.0

Now DynamicJoinSelection will prefer shuffled hash join only if *advisoryPartitionSize <= maxShuffledHashJoinLocalMapThreshold*
We can also prefer shuffled hash join if the partitions in one side are smaller are smaller than advisoryPartitionSize, and the partitions in the other side are all larger enough(larger than advisoryPartitionSize)
Bug: Confusing partition specs in PartitionsAlreadyExistException
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Oct/22 13:54
Affects Version/s: 3.4.0

The code below demonstrates the issue:
```
> CREATE TABLE t (id bigint, data string) USING _ PARTITIONED BY (id);
> ALTER TABLE t ADD PARTITION (id=2) LOCATION 'loc1';
> ALTER TABLE t ADD PARTITION (id=1) LOCATION 'loc' PARTITION (id=2) LOCATION 'loc1';
The following partitions already exists in table t:2 -> id
```
Bug: Serialization of MapStatuses is somtimes much larger on scala 2.13
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 05/Oct/22 06:36
Affects Version/s: 3.3.0

We have observed a case where the same job run against spark on scala 2.13 fails going out of memory due to the the broadcast for the MapStatuses being huge.

In the logs around the time the job fails it tries to create a broadcast of size 4.8GiB. 
```
2022-09-18 22:46:01,418 INFO memory.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 4.8 GiB, free 12.9 GiB)
```

The same broadcast of the MapStatus for the same job running on 2.12 is 391.5 Mib so 
```
2022-09-18 16:11:58,753 INFO memory.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 391.5 MiB, free 26.4 GiB)
```

in this particular case it seems the broadcast for MapStatuses more than 10 large when using 2.13. This is not something universal for all MapStatus broadcast as we have have many other jobs using Scala 2.13 where the status is ruffly the same size. 

This has been observed on 3.3.0 but I also tested it against 3.3.1-rc2 and build of 3.4.0-SNAPSHOT and both of those also reproduced the issue.
Bug: Update embedded JQuery javascript resources
Status: Open | Priority: Major | Resolution: 
Created: 06/Oct/22 20:52
Affects Version/s: 3.3.0

Spark 3.3 currently ships with embedded versions of JQuery subject to these vulnerability notices:

CVE-2019-11358
CVE-2020-7656
CVE-2020-11023

This issue is not meant to imply a security issue in Spark itself.
Bug: Switch to XORShiftRandom to distribute elements
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Oct/22 01:51
Affects Version/s: 3.4.0


{code:scala}
import java.util.Random
import org.apache.spark.util.random.XORShiftRandom
import scala.util.hashing

def distribution(count: Int, partition: Int) = {
  println((1 to count).map(partitionId => new Random(partitionId).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))

  println((1 to count).map(partitionId => new Random(hashing.byteswap32(partitionId)).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))

  println((1 to count).map(partitionId => new XORShiftRandom(partitionId).nextInt(partition))
    .groupBy(f => f)
    .map(_._2.size).mkString(". "))
}

distribution(200, 4)
{code}


{noformat}
200
50. 60. 46. 44
55. 48. 43. 54
{noformat}
Bug: DAGScheduler should fail job until all related running tasks have been killed
Status: In Progress | Priority: Major | Resolution: 
Created: 04/Oct/22 04:57
Affects Version/s: 3.2.2

*Issue Description*
The staging directory within table location is not removed when {{CTAS}} fails sometimes.
It is a trouble if the new table is a Managed Table when we want to recreate it.

*Root Cause*
SchedulerBackend kills tasks via {{KillTask}} message which is asynchronous, so we may have failed a job but the tasks are still running and create the tmp file. Even if the running tasks will failed and delete the generated file finally, but the temporary directory was left.

*Solution*
Before killing a job, we should make sure that all related running tasks have been killed.

*How to Reproduce*

Step 1: create a source table and insert data to make the file number exceeds 20 on HDFS
{code:sql}
-- create source table
CREATE TABLE IF NOT EXISTS default.test_wechar
(name string)
PARTITIONED BY (grass_date date)
STORED AS PARQUET

-- insert data 24 times
insert into default.test_wechar partition (grass_date='2022-09-03')
select uuid()
lateral view explode(sequence(1,2000)) as temp_view;
{code}
Step 2: create a new path for new table and setQuota to 20
{code:bash}
$hadoop fs -count -q hdfs://Test-DMP01/user/weiqiang.yu/tmp/test_wechar_tmp
          20              19            none             inf            1            0                  0 hdfs://Test-DMP01/user/weiqiang.yu/tmp/test_wechar_tmp
{code}
Step 3: create new table from source table
{code:sql}
create table if not exists default.test_wechar_tmp 
location 'hdfs://Test-DMP01/user/weiqiang.yu/tmp/test_wechar_tmp' 
as select * from default.test_wechar;
{code}
 
Step 4: check the location of new table after the job failed
{code:bash}
$hadoop fs -ls /user/weiqiang.yu/tmp/test_wechar_tmp/*
Found 1 items
drwxrwxr-x   - weiqiang.yu weiqiang.yu          0 2022-10-04 12:56 /user/weiqiang.yu/tmp/test_wechar_tmp/.hive-staging_hive_2022-10-04_12-56-21_545_2745177084386740362-1/-ext-10000
{code}
Bug: Reorder join keys impact performance
Status: Open | Priority: Major | Resolution: 
Created: 01/Oct/22 05:51
Affects Version/s: 3.4.0

{code:scala}
sql("CREATE TABLE t1 (itemid BIGINT, eventType STRING, dt STRING) USING parquet PARTITIONED BY (dt)")
sql("CREATE TABLE t2 (cal_dt DATE, item_id BIGINT) using parquet")

sql("set spark.sql.autoBroadcastJoinThreshold=-1")

sql(
  """
    |SELECT itemid,
    |       eventtype
    |FROM   t1 a
    |       INNER JOIN (SELECT DISTINCT cal_dt,
    |                                   item_id
    |                   FROM   t2) b
    |               ON a.itemid = b.item_id
    |                  AND To_date(a.dt, 'yyyyMMdd') = b.cal_dt
  """.stripMargin).explain()
{code}

The plan:
{noformat}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- Project [itemid#10L, eventtype#11]
   +- SortMergeJoin [cast(gettimestamp(dt#12, yyyyMMdd, TimestampType, Some(America/Los_Angeles), false) as date), itemid#10L], [cal_dt#13, item_id#14L], Inner
      :- Sort [cast(gettimestamp(dt#12, yyyyMMdd, TimestampType, Some(America/Los_Angeles), false) as date) ASC NULLS FIRST, itemid#10L ASC NULLS FIRST], false, 0
      :  +- Exchange hashpartitioning(cast(gettimestamp(dt#12, yyyyMMdd, TimestampType, Some(America/Los_Angeles), false) as date), itemid#10L, 5), ENSURE_REQUIREMENTS, [plan_id=48]
      :     +- Filter isnotnull(itemid#10L)
      :        +- FileScan parquet spark_catalog.default.t1[itemid#10L,eventType#11,dt#12]
      +- Sort [cal_dt#13 ASC NULLS FIRST, item_id#14L ASC NULLS FIRST], false, 0
         +- HashAggregate(keys=[cal_dt#13, item_id#14L], functions=[])
            +- Exchange hashpartitioning(cal_dt#13, item_id#14L, 5), ENSURE_REQUIREMENTS, [plan_id=44]
               +- HashAggregate(keys=[cal_dt#13, item_id#14L], functions=[])
                  +- Filter (isnotnull(item_id#14L) AND isnotnull(cal_dt#13))
                     +- FileScan parquet spark_catalog.default.t2[cal_dt#13,item_id#14L]
{noformat}

Default: 38min.
Do not reorder join keys: 8.1 min
Pull out complex join condition: 3.6 min
Bug: Upgrade Jackson-databind to 2.13.4.1
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Oct/22 05:46
Affects Version/s: 3.4.0

#3590: Add check in primitive value deserializers to avoid deep wrapper array
  nesting wrt `UNWRAP_SINGLE_VALUE_ARRAYS` [CVE-2022-42003]
Bug: Should expose driver service name to config for user features
Status: In Progress | Priority: Minor | Resolution: 
Created: 12/Oct/22 02:24
Affects Version/s: 3.3.0

Current on kubernetes, user's feature step, which build user's kubernetes resource during spark submit spark pod, can't percept some spark resource info, such as spark driver service name.

 

User may want to expose some spark pod info to build their custom resource, such as ingress, etc.

 

We want the way expose now spark driver service name, which is now generated by clock and uuid.
Bug: Creating StructType from Json
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Oct/22 13:52
Affects Version/s: 3.5.0

When create a StructType from a Python dictionary you use [StructType.fromJson|https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py#L792] or in scala [DataType.fromJson|https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala#L158C7-L158C15]

To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.

{code:python}
schema = {
     "name": "name", "type": "string" 
}

StructField.fromJson(schema)
{code}

Python Error:

{code:python}
from pyspark.sql.types import StructField

schema = {
     "name": "c1", "type": "string" 
}

StructField.fromJson(schema)

>>
Traceback (most recent call last):
File "code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "pyspark/sql/types.py", line 583, in fromJson
json["nullable"],
KeyError: 'nullable' 
{code}

Scala Error:
{code:scala}
val schema = """
        |{
        |    "type": "struct",
        |    "fields": [
        |        {
        |            "name": "c1",
        |            "type": "string",
        |            "nullable": false
        |        }
        |    ]
        |}
        |""".stripMargin

DataType.fromJson(schema)

>>
Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
java.lang.IllegalArgumentException: Failed to convert the JSON string '{"name":"c1","type":"string"}' to a field.
at org.apache.spark.sql.types.DataType$.parseStructField(DataType.scala:268)
at org.apache.spark.sql.types.DataType$.$anonfun$parseDataType$1(DataType.scala:225)
{code}
Bug: Support data masking built-in function  'mask_first_n'
Status: In Progress | Priority: Minor | Resolution: 
Created: 07/Oct/22 00:11
Affects Version/s: 3.4.0

Support data masking built-in function  *mask_first_n*

Return a masked version of str with the first n values masked. Upper case letters should be converted to "X", lower case letters should be converted to "x" and numbers should be converted to "n". For example, mask_first_n("1234-5678-8765-4321", 4) results in nnnn-5678-8765-4321.
Bug: Overhead of Exceptions in DeserializationStream 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Oct/22 12:50
Affects Version/s: 3.3.0

The interface of DeserializationStream forces implementation to raise EOFException to indicate that there is no more data. And for the KryoDeserializtionStream it even worse since the kryo library does not raise EOFException we pay for the price of two exceptions for each stream. For large shuffles with lots of small stream this is quite a bit large overhead (seen couple % of cpu time). It also less safe to depend exceptions as it might me raised for different reasons like corrupt data and that currently cause data loss.
Bug: SQL Equivalent for Dataframe overwrite command
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Oct/22 16:59
Affects Version/s: 3.2.2

Proposing syntax
{code:java}
 INSERT INTO tbl REPLACE whereClause identifierList{code}
to the spark SQL, as the equivalent of [dataframe.overwrite()|https://github.com/apache/spark/blob/35d00df9bba7238ad4f409999617fae4d04ddbfd/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala#L163] command. 

 

For Example
{code:java}
 INSERT INTO table1 REPLACE WHERE key = 3 SELECT * FROM table2{code}
will, in an atomic operation, 1) delete rows with key = 3 and 2) insert rows from table2
Bug: Support driver service labels
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Oct/22 03:14
Affects Version/s: 3.4.0

Currently, user can customize driver service annotation using

{{ spark.kubernetes.driver.service.annotation.[AnnotationName]}}

but the label is not configurable. There are use cases of custom label configuration - in our company, some service mesh auth relies on the labels of the service object, so we need the driver service label configurability.
Bug: EXPODE function selects outer column
Status: In Progress | Priority: Minor | Resolution: 
Created: 31/Oct/22 09:02
Affects Version/s: 3.3.0

Im trying to determine if indirectly selecting an outer column is a bug or an intended feature of the EXPLODE function. 

 

If I run the following SQL statement:

```

SELECT

      (SELECT FIRST(name_element_)

        FROM LATERAL VIEW EXPLODE(name) AS name_element_

       *)*

FROM patient

```

 

it fails with:

```

Accessing outer query column is not allowed in:

Generate explode(outer(name#9628))

```

 

However, if I do a "cheeky select" (bolded below), the SQL query is valid and runs:

```

SELECT(

    SELECT FIRST(name_element_)

    FROM (SELECT EXPLODE(name_element_) AS name_element_ 

          \{*}FROM ({*}{*}SELECT{*} *name AS name_element_)*

        **        )

         )

FROM patient

```

From the viewpoint of the EXPLODE function, it seems like the column name_element_ does not come from an outer column. Is this an intended feature or a bug?