av9ash/GitBugs huggingface dataset:
Bug: Use CaseInsensitiveMap for DataStreamReader/Writer options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 16:28
Affects Version/s: 2.0.2

When a user have multiple options like path, paTH, and PATH for the same key path, option/options is non-deterministic because extraOptions is HashMap. This issue aims to use *CaseInsensitiveMap* instead of *HashMap* to fix this bug fundamentally.

{code}
spark.readStream
  .option("paTh", "1")
  .option("PATH", "2")
  .option("Path", "3")
  .option("patH", "4")
  .option("path", "5")
  .load()
...
org.apache.spark.sql.AnalysisException:
Path does not exist: file:/.../1;
{code}
Bug: Add DataStreamWriterV2 API
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Sep/20 12:20
Affects Version/s: 3.1.0

SPARK-28612 introduces DataFrameWriterV2 which is a new API to enable write in batch query against catalog table.

This issue tracks the effort to introduce DataStreamWriterV2, which will be a new API to enable write in streaming query against catalog table.
Bug: Aggregation support aggregate function with multiple foldable expressions.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Sep/20 07:26
Affects Version/s: 3.1.0

Spark SQL exists a bug show below:

{code:java}
spark.sql(
  " SELECT COUNT(DISTINCT 2), COUNT(DISTINCT 2, 3)")
  .show()
+-----------------+--------------------+
|count(DISTINCT 2)|count(DISTINCT 2, 3)|
+-----------------+--------------------+
|                1|                   1|
+-----------------+--------------------+

spark.sql(
  " SELECT COUNT(DISTINCT 2), COUNT(DISTINCT 3, 2)")
  .show()
+-----------------+--------------------+
|count(DISTINCT 2)|count(DISTINCT 3, 2)|
+-----------------+--------------------+
|                1|                   0|
+-----------------+--------------------+
{code}
Bug: Support dynamic partition pruning hint
Status: In Progress | Priority: Major | Resolution: 
Created: 10/Sep/20 08:38
Affects Version/s: 3.1.0

For some joins, one side is very small but can not broadcast, for example:
 * left join but left side very small
 * right join but the right side very small

At this time, {{PlanDynamicPruningFilters}} may not insert dynamic partition pruning. I'd like to add a hint to support users to force insert a dynamic partition pruning.
Bug: Spark SQL aggregate() fails on nested string arrays
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 08:07
Affects Version/s: 3.0.0

The aggregate() function seems to fail if the initial state is array(array( some string )). Seems to work if it's an INT for example.

Example that works with a simple array:

{{select aggregate(split('abcdefgh',''), array(''), (acc, x) -> array( x ) )}}

Example that works with nested array and INTs:

{{select aggregate(sequence(0,9), array(array(0)), (acc, x) -> array(array}}{{( x ) ) }}{{)}}

Example that errors:

{{select aggregate(split('abcdefgh',''), array(array('')), (acc, x) -> array(array}}{{( x ) ) }}{{)}}

Producing the following (shortened) error:

{{data type mismatch: argument 3 requires array<array<string>> type, however ... is of array<array<string>> type.}}
Bug: Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Sep/20 11:56
Affects Version/s: 3.1.0

SupportsStreamingUpdate was added without proper design because the time was running out in releasing Spark 3.0.0. The class has a TODO comment to introduce better API, but nothing has been proposed so far.

Before proposing the design for the new API, it'd be nice to refactor SupportsStreamingUpdate to represent actual meaning of the behavior, as the name of SupportsStreamingUpdate doesn't represent anything.
Bug: CTE support on MSSQL
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 08/Sep/20 17:56
Affects Version/s: 2.4.6

The issue in detail along with stack trace is described over [here|https://github.com/microsoft/mssql-jdbc/issues/1340]. Summary is that WITH CTE clause does not work with MSSQL. This may be because WITH CTE is not supported in FROM clause of MSSQL.
Bug: Fix LibSVM data source loading error on file paths with glob metacharacters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 06:28
Affects Version/s: 2.4.6

SPARK-32810 fixed a long standing bug in a few Spark built-in data sources that fails to read files whose names contain glob metacharacters, such as [, ], \{, }, etc.

CSV and JSON data source on the Spark side were affected. We've also noticed that the LibSVM data source had the same code pattern that leads to the bug, so the fix https://github.com/apache/spark/pull/29659 included a fix for that data source as well, but it did not include a test for the LibSVM data source.

This ticket tracks adding a test case for LibSVM, similar to the ones for CSV/JSON, to verify whether or not the fix works as intended.
Bug: DPP throws error when broadcast side is empty
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Sep/20 07:25
Affects Version/s: 3.1.0

In `SubqueryBroadcastExec.relationFuture`, if the `broadcastRelation` is an `EmptyHashedRelation`, then `broadcastRelation.keys()` will throw `UnsupportedOperationException`.
Bug: Remove redundant shuffle exchanges inserted by EnsureRequirements
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Sep/20 09:25
Affects Version/s: 3.1.0

Redundant repartition operations are removed by CollapseRepartition rule but EnsureRequirements can insert another HashPartitioning or RangePartitioning immediately after the repartition, leading adjacent ShuffleExchanges will be in the physical plan.

 
{code:java}
val ordered = spark.range(1, 100).repartitionByRange(10, $"id".desc).orderBy($"id")
ordered.explain(true)

...

== Physical Plan ==
*(2) Sort [id#0L ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(id#0L ASC NULLS FIRST, 200), true, [id=#15]
   +- Exchange rangepartitioning(id#0L DESC NULLS LAST, 10), false, [id=#14]
      +- *(1) Range (1, 100, step=1, splits=12){code}
{code:java}
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 0)
val left = Seq(1,2,3).toDF.repartition(10, $"value")
val right = Seq(1,2,3).toDF
val joined = left.join(right, left("value") + 1 === right("value")
joined.explain(true)

...

== Physical Plan ==
*(3) SortMergeJoin [(value#7 + 1)], [value#12], Inner
:- *(1) Sort [(value#7 + 1) ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning((value#7 + 1), 200), true, [id=#67]
:     +- Exchange hashpartitioning(value#7, 10), false, [id=#63]
:        +- LocalTableScan [value#7]
+- *(2) Sort [value#12 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(value#12, 200), true, [id=#68]
      +- LocalTableScan [value#12]{code}
Bug: compare of -0.0 < 0.0 return true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/20 10:30
Affects Version/s: 3.0.0

{code:scala}
 val spark: SparkSession = SparkSession
      .builder()
      .master("local")
      .appName("SparkByExamples.com")
      .getOrCreate()
    spark.sparkContext.setLogLevel("ERROR")

    import spark.sqlContext.implicits._

    val df = Seq((-0.0, 0.0)).toDF("neg", "pos")
      .withColumn("comp", col("neg") < col("pos"))
      df.show(false)

======

+----+---+----+
|neg |pos|comp|
+----+---+----+
|-0.0|0.0|true|
+----+---+----+{code}

I think that result should be false.

**Apache Spark 2.4.6 RESULT**
{code}
scala> spark.version
res0: String = 2.4.6

scala> Seq((-0.0, 0.0)).toDF("neg", "pos").withColumn("comp", col("neg") < col("pos")).show
+----+---+-----+
| neg|pos| comp|
+----+---+-----+
|-0.0|0.0|false|
+----+---+-----+
{code}
Bug: Make withField API support nested struct in array
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 04/Sep/20 06:08
Affects Version/s: 3.1.0

Currently {{Column.withField}} only supports {{StructType}}. For nested struct in {{ArrayType}}, it doesn't support. We can support {{ArrayType}} to make the API more general and useful.

It could significantly simplify how we add/replace deeply nested fields.

For example,

{code}
private lazy val arrayType = ArrayType(
    StructType(Seq(
      StructField("a", IntegerType, nullable = false),
      StructField("b", IntegerType, nullable = true),
      StructField("c", IntegerType, nullable = false))),
    containsNull = true)

private lazy val arrayStructArrayLevel1: DataFrame = spark.createDataFrame(
    sparkContext.parallelize(Row(Array(Row(Array(Row(1, null, 3)), null, 3))) :: Nil),
    StructType(
      Seq(StructField("a", ArrayType(
        StructType(Seq(
          StructField("a", arrayType, nullable = false),
          StructField("b", IntegerType, nullable = true),
          StructField("c", IntegerType, nullable = false))),
        containsNull = false)))))
{code}

The data looks like:

{code}
+---------------------------+
|a                          |
+---------------------------+
|[{[{1, null, 3}], null, 3}]|
+---------------------------+
{code}

In order to replace deeply nested b column, like:

{code}
+------------------------+
|a                       |
+------------------------+
|[{[{1, 2, 3}], null, 3}]|
+------------------------+
{code}

Currently by using transform + withField, we probably need:

{code}
arrayStructArrayLevel1.withColumn("a",
      transform($"a", _.withField("a",
        flatten(transform($"a.a", transform(_, _.withField("b", lit(2))))))))
{code}

Using modified withField, we can do it like:

{code}
arrayStructArrayLevel1.withColumn("a", $"a".withField("a.b", lit(2)))
{code}
Bug: Literal integer seems to get confused as column reference
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 06/Sep/20 08:15
Affects Version/s: 3.0.0

When using a literal integer in the group by expression, an error is caused which seems to indicate that the integer was interpreted as a column reference.

I would expect this to succeed, and result in a single grouping with a value of "2", and a count of "1".

Here is an example of a minimal program which reproduces the problem:
{code:java}
import java.util.ArrayList;
import java.util.List;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.functions;

class Scratch {

  public static void main(String[] args) {
    SparkSession spark = SparkSession.builder()
        .master("local[*]")
        .getOrCreate();

    List<Something> somethings = new ArrayList<>();
    Something something = new Something();
    something.setFieldA("foo");
    somethings.add(something);

    Dataset<Row> dataframe = spark.createDataFrame(somethings, Something.class);
    Dataset<Row> result = dataframe.groupBy(functions.lit(2))
        .agg(functions.count("*"));
    result.collectAsList().forEach(row -> System.out.println(row.toString()));
  }

  public static class Something {
    private String fieldA;

    public Something() {
    }

    public String getFieldA() {
      return fieldA;
    }

    public void setFieldA(String fieldA) {
      this.fieldA = fieldA;
    }
  }
}
{code}
Adding .cast("int") to the end of the literal column fixes the problem, but this seems unnecessary, as I have already told the Spark API that I wanted a literal integer.
Bug: Bucket join should work if spark.sql.shuffle.partitions larger than bucket number
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Sep/20 16:37
Affects Version/s: 3.0.0

How to reproduce this issue:
{code:scala}
    spark.range(1000).write.bucketBy(432, "id").saveAsTable("t1")
    spark.range(1000).write.bucketBy(34, "id").saveAsTable("t2")
    sql("set spark.sql.shuffle.partitions=600")
    sql("set spark.sql.autoBroadcastJoinThreshold=-1")
    sql("select * from t1 join t2 on t1.id = t2.id").explain()
{code}

{noformat}
== Physical Plan ==
*(5) SortMergeJoin [id#26L], [id#27L], Inner
:- *(2) Sort [id#26L ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(id#26L, 600), true, [id=#65]
:     +- *(1) Filter isnotnull(id#26L)
:        +- *(1) ColumnarToRow
:           +- FileScan parquet default.t1[id#26L] Batched: true, DataFilters: [isnotnull(id#26L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32444/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 432 out of 432
+- *(4) Sort [id#27L ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(id#27L, 600), true, [id=#74]
      +- *(3) Filter isnotnull(id#27L)
         +- *(3) ColumnarToRow
            +- FileScan parquet default.t2[id#27L] Batched: true, DataFilters: [isnotnull(id#27L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32444/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 34 out of 34

{noformat}

*Expected*:
{noformat}
== Physical Plan ==
*(4) SortMergeJoin [id#26L], [id#27L], Inner
:- *(1) Sort [id#26L ASC NULLS FIRST], false, 0
:  +- *(1) Filter isnotnull(id#26L)
:     +- *(1) ColumnarToRow
:        +- FileScan parquet default.t1[id#26L] Batched: true, DataFilters: [isnotnull(id#26L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32444/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 432 out of 432
+- *(3) Sort [id#27L ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(id#27L, 432), true, [id=#69]
      +- *(2) Filter isnotnull(id#27L)
         +- *(2) ColumnarToRow
            +- FileScan parquet default.t2[id#27L] Batched: true, DataFilters: [isnotnull(id#27L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32444/sql/core/spark-warehouse/org.apache.spark..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>, SelectedBucketsCount: 34 out of 34

{noformat}
Bug: The behavior of listJars and listFiles is not consistent between YARN and other cluster managers
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Sep/20 20:05
Affects Version/s: 3.1.0

Jars/files specified with --jars / --files options are listed by sc.listJars and listFiles except when we run apps on YARN.
If we run apps not on YARN, those files are served by the embedded file server in the driver and listJars/listFiles list the served files.
But with YARN, such files specified by the options are not served by the embedded file server so listJars and listFiles don't list them.
Bug: Support for INET data type
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 01/Sep/20 04:11
Affects Version/s: 2.4.0

PostgreSQL has support for `INET` data type 

[https://www.postgresql.org/docs/9.1/datatype-net-types.html]

We have a few customers that are interested in similar, native support for IP addresses, just like in PostgreSQL.

The issue with storing IP addresses as strings, is that most of the matches (like if an IP address belong to a subnet) in most cases can't take leverage of parquet bloom filters.
Bug: Use keyword-only syntax for keyword_only methods
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Sep/20 07:09
Affects Version/s: 3.1.0

Since 3.0, provides syntax for indicating keyword-only arguments ([PEP 3102|https://www.python.org/dev/peps/pep-3102/]).

It is not a full replacement for our current usage of {{keyword_only}}, but it would allow us to make our expectations explicit:


{code:python}
@keyword_only
def __init__(self, degree=2, inputCol=None, outputCol=None):
{code}

{code:python}
@keyword_only
def __init__(self, *, degree=2, inputCol=None, outputCol=None):
{code}
Bug: Support createDataFrame from an RDD of pd.DataFrames
Status: In Progress | Priority: Minor | Resolution: 
Created: 10/Sep/20 11:56
Affects Version/s: 3.0.1

Add support to createDataFrame from a distributed collection of pandas.DataFrames by converting the RDD of pd.DFs to an RDD of arrow records batches, then directly creating the spark DataFrame from it.

 

Performance is significantly better (vectorized) than creating a spark DF by converting each df to a list of rows, similar to the improvement of SPARK-20791.

 

Initial example & benchmark for older spark versions: [https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5|https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5,]

 

I'm currently working on a PR and will post it soon.

 

Extends the work done in: 

https://issues.apache.org/jira/browse/SPARK-20791 

https://issues.apache.org/jira/browse/SPARK-23030
Bug: Performance regression when selecting from str_to_map
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 24/Sep/20 19:36
Affects Version/s: 3.0.1

When I create a map using str_to_map and select more than a single value, I notice a notable performance regression in 3.0.1 compared to 2.4.7. When selecting a single value, the performance is the same. Plans are identical between versions.

It seems like in 2.x the map from str_to_map is preserved for a given row, but in 3.x it's recalculated for each column. One hint that it might be the case is that when I tried forcing materialisation of said map in 3.x (by a coalesce, don't know if there's a better way), I got the performance roughly to 2.x levels.

Here's a reproducer (the csv in question gets autogenerated by the python code):
{code:java}
$ head regression.csv 
foo
foo=bar&baz=bak&bar=foo
foo=bar&baz=bak&bar=foo
foo=bar&baz=bak&bar=foo
foo=bar&baz=bak&bar=foo
foo=bar&baz=bak&bar=foo
... (10M more rows)
{code}
{code:python}
import time
import os

import pyspark  
from pyspark.sql import SparkSession

import pyspark.sql.functions as f

if __name__ == '__main__':
    print(pyspark.__version__)
    spark = SparkSession.builder.getOrCreate()

    filename = 'regression.csv'
    if not os.path.isfile(filename):
        with open(filename, 'wt') as fw:
            fw.write('foo\n')
            for _ in range(10_000_000):
                fw.write('foo=bar&baz=bak&bar=foo\n')

    df = spark.read.option('header', True).csv(filename)
    t = time.time()
    dd = (df
            .withColumn('my_map', f.expr('str_to_map(foo, "&", "=")'))
            .select(
                f.col('my_map')['foo'],
            )
        )
    dd.write.mode('overwrite').csv('tmp')
    t2 = time.time()
    print('selected one', t2 - t)

    dd = (df
            .withColumn('my_map', f.expr('str_to_map(foo, "&", "=")'))
            # .coalesce(100) # forcing evaluation before selection speeds it up in 3.0.1
            .select(
                f.col('my_map')['foo'],
                f.col('my_map')['bar'],
                f.col('my_map')['baz'],
            )
        )
    dd.explain(True)
    dd.write.mode('overwrite').csv('tmp')
    t3 = time.time()
    print('selected three', t3 - t2)
{code}
Results for 2.4.7 and 3.0.1, both installed from PyPI, Python 3.7, macOS (times are in seconds)
{code:java}
# 3.0.1
# selected one 6.375471830368042                                                  
# selected three 14.847578048706055

# 2.4.7
# selected one 6.679579019546509                                                  
# selected three 6.5622029304504395  
{code}
Bug: ExternalCatalog vs ExternalCatalogWithListener:  backward compatibility problem
Status: Resolved | Priority: Minor | Resolution: Not A Problem
Created: 24/Sep/20 18:36
Affects Version/s: 2.3.2

With Spark 2.3 

spark.sharedState.externalCatalog used to give an object of ExternalCatalog which was actually an instance of HiveExternalCatalog

With Spark 2.4 

spark.sharedState.externalCatalog gives an object of ExternalCatalogWithListener which has HiveExternalCatalog as unwrapped delegate.

 

We use HiveExternalCatalog to get HiveClient to perform Hive Operations.

 

 

This change in method Signature is giving us backward compatibility problems such that code compiled with Spark 2.3 can not work with 2.4 due to the change. This results in Runtime exceptions including ClassCastException, NoSuchMethodError
Bug: Promote the stability annotation to Evolving for MLEvent traits/classes
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Sep/20 04:27
Affects Version/s: 3.1.0

MLEvent, and its implementations are added in Jan 2020 (via SPARK-23674), and there's only one change from SPARK-26818 in Feb 2020.

The feature is released in Spark 3.0.0 having SPARK-26818 as the last change, and haven't changed in Spark 3.0.1.

While we'd better to wait for some minor releases to consider the API as stable, it would worth to promote to Evolving so that we clearly state that we support the API.
Bug: reading a parallized rdd with two identical records results in a zero count df when read via spark.read.csv
Status: Closed | Priority: Minor | Resolution: Fixed
Created: 15/Sep/20 09:43
Affects Version/s: 2.4.5

* Imagine a two-row csv file like so (where the header and first record are duplicate rows):

aaa,bbb

aaa,bbb
 * The following is pyspark code
 * create a parallelized rdd like: {color:#FF0000}prdd = spark.read.text("test.csv").rdd.flatMap(lambda x : x){color}
 * {color:#172b4d}create a df like so: {color:#de350b}mydf = spark.read.csv(prdd, header=True){color}{color}
 * {color:#172b4d}{color:#de350b}df.count(){color:#172b4d} will result in a record count of zero (when it should be 1){color}{color}{color}
Bug: Append toMap to Map#filterKeys if the result of filter is concatenated with another Map for Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Sep/20 19:34
Affects Version/s: 3.1.0

As of Scala 2.13, Map#filterKeys return a MapView, not the original Map type.
This can cause compile error.
{code:java}
/sql/DataFrameReader.scala:279: type mismatch;
[error]  found   : Iterable[(String, String)]
[error]  required: java.util.Map[String,String]
[error] Error occurred in an application involving default arguments.
[error]       val dsOptions = new CaseInsensitiveStringMap(finalOptions.asJava) {code}
Bug: Fix code which causes error when build with sbt and Scala 2.13
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Sep/20 03:17
Affects Version/s: 3.1.0

When we try to build Spark with sbt and Spark 2.13, some piece of code causes error.
{code:java}
[error] [warn] /home/kou/work/oss/spark-scala-2.13/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala:251: method with a single empty parameter list overrides method without any parameter list                                                                
[error] [warn]   override def hasNext(): Boolean = requestOffset < part.untilOffset                                                               
[error] [warn]                                                                                                                                    
[error] [warn] /home/kou/work/oss/spark-scala-2.13/external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala:294: method with a single empty parameter list overrides method without any parameter list                                                                
[error] [warn]   override def hasNext(): Boolean = okNext     {code}
Bug: Expose assert_true in Python/Scala APIs and add error message parameter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Sep/20 18:58
Affects Version/s: 3.1.0

# Add RAISEERROR() (or RAISE_ERROR()) to the API
 # Add Scala/Python/R version of API for ASSERT_TRUE()
 # Add an extra parameter to ASSERT_TRUE() as (cond, message), and for which the `message` parameter is only lazily evaluated when the condition is not true
 # Change the implementation of ASSERT_TRUE() to be rewritten during optimization to IF() instead.