av9ash/GitBugs huggingface dataset:
Bug: Add code-gen support for all join types of sort merge join
Status: Open | Priority: Minor | Resolution: 
Created: 11/Mar/21 02:22
Affects Version/s: 3.2.0

Currently sort merge join only has code-gen for inner join type ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala#L374] ). We added code-gen for other join types internally in our fork and saw obvious CPU performance improvement. Create this Jira to add the code-gen to upstream.
Bug: Support Not(Int) and Not(InSet) propagate null
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 10/Mar/21 11:53
Affects Version/s: 3.2.0

`Not(In)` is a little magic, the semantic of it could be seen like `and(a != b, a != c)`

As we already simplify the `NullIntolerant` expression to null if it's children have null at `NullPropagation`. e.g. `a != null` => null. It's safe to do the same simplify to `Not(In)`/ `Not(InSet)`
Bug: Support Not(Int) and Not(InSet) propagate null
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 10/Mar/21 11:51
Affects Version/s: 3.2.0

`Not(In)` is a little magic, the semantic of it could be seen like `and(a != b, a != c)`

As we already simplify the `NullIntolerant` expression to null if it's children have null at `NullPropagation`. e.g. `a != null` => null. It's safe to do the same simplify to `Not(In)`/ `Not(InSet)`
Bug: Use FunctionIdentifier in UnresolvedTableValuedFunction
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Mar/21 01:31
Affects Version/s: 3.2.0

Currently, UnresolvedTableValuedFunction uses a string as its function name, which is inconsistent with UnresolvedFunction that uses FunctionIdentifier as the function name. We should change UnresolvedTableValuedFunction to also use FunctionIdentifier.
Bug: Provide silhouette score for each sample when using ClusteringEvaluator
Status: Open | Priority: Minor | Resolution: 
Created: 08/Mar/21 13:45
Affects Version/s: 3.1.1

Computing the average silhouette score is already implemented when using ClusteringEvaluator. When looking at the [source code|https://gitlab.com/mark91/SparkClusteringEvaluationMetrics/-/blob/master/src/main/scala/org/apache/spark/ml/evaluation/SquaredEuclideanSilhouetteEvaluator.scala] of ClusteringEvaluator, I think it would be easy to provide not only the average silhouette score but also the silhouette score for each sample, as they are already computed (Line 95-99).
 The silhouette score for each sample can be helpful to generate a silhouette plot for instance as described in [this scikit-learn article|https://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_silhouette_analysis.html]. The resulting feature would be equivalent to the silhouette_samples function implemented in scikit-learn.
Bug: RewritePredicateSubquery Rule must not update Filters without subqueries
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/21 18:58
Affects Version/s: 3.1.1

1. Currently RewritePredicateSubquery rule updates Filter node for queries without any subquery as well. This shouldn't happen. 
2. Also `Filter(conditions.reduce(And), child)` in the rule might create a skewed expression tree even though original expression is  balanced.

 
{noformat}
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery ===
 Project [a#0]                                                        Project [a#0]
!+- Filter (((a#0 > 1) OR (b#1 > 2)) AND ((c#2 > 1) AND (d#3 > 2)))   +- Filter ((((a#0 > 1) OR (b#1 > 2)) AND (c#2 > 1)) AND (d#3 > 2))
    +- LocalRelation <empty>, [a#0, b#1, c#2, d#3]                       +- LocalRelation <empty>, [a#0, b#1, c#2, d#3]{noformat}
Bug: Replaces `ParquetFileReader.readFooter` with `ParquetFileReader.open and getFooter`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 02/Mar/21 13:44
Affects Version/s: 3.2.0

`ParquetFileReader.readFooter `related methods has been identified as `Deprecated`,

and it can be replaced with `ParquetFileReader.open and getFooter`
Bug: Add "Job Id" column in Stage table (WebUI) to go to job page for a given stage
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 07/Mar/21 19:36
Affects Version/s: 3.0.2

In the web ui, under "Stages" all stages are shown , but here is no indication to which Job Id(s) they belong to, requiring more work to find out. A small and helpful ui changes would be to add a new column, called "Job Id(s)", with a list of job ids the stage belongs to, and the link to the job itself. This would help a lot in following up what is going on.
Bug: TypeError in Pyspark Linear Regression docs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 05/Mar/21 17:24
Affects Version/s: 3.1.1

The documentation for Linear Regression in Pyspark includes an example, which contains a TypeError:

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.regression.LinearRegression.html?highlight=linearregression#pyspark.ml.regression.LinearRegression

{code}
>>> abs(model.transform(test1).head().newPrediction - 1.0) < 0.001
True
>>> lr.setParams("vector")
Traceback (most recent call last):
    ...
TypeError: Method setParams forces keyword arguments.
{code}

I'm pretty sure we don't intend this, and it can be resolved by just changing the function call to use keyword args.

(HT Brooke Wenig for flagging this)
Bug: Fix view does not capture disable hint config
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Mar/21 03:38
Affects Version/s: 3.1.1

Spark use origin text sql to store view then capture and store sql config into view metadata.

Capture config will skip some config with some prefix, e.g. `spark.sql.optimizer.` but unfortunately `spark.sql.optimizer.disableHints` is start with `spark.sql.optimizer.`.

 

We need a allow list to help capture the config.
Bug: Improve PlanStabilitySuite to output # of operators information
Status: Open | Priority: Minor | Resolution: 
Created: 03/Mar/21 06:46
Affects Version/s: 3.2.0

Based on discussion from [https://github.com/apache/spark/pull/31708#issuecomment-789444406,] plan check is really good to catch plan difference now per PR. However, it's quite hard to debug given the physical query plan can be very long and verbose for complicated query. It would be good if we can print out the information for important operators, e.g. # of shuffles, sub-queries, reused shuffle, join, aggregation, etc.
Bug: JDBC - Add an option for different stride orders
Status: In Progress | Priority: Trivial | Resolution: 
Created: 30/Mar/21 22:46
Affects Version/s: 3.2.0

Currently, the JDBCRelation columnPartition function orders the strides in ascending order, starting from the lower bound and working its way towards the upper bound.

I'm proposing leaving that as the default, but adding an option (such as strideOrder) in JDBCOptions. Since it will default to the current behavior, this will keep people's current code working as expected. However, people who may have data skew closer to the upper bound might appreciate being able to have the strides in descending order, thus filling up the first partition with the last stride and so forth. Also, people with nondeterministic data skew or sporadic data density might be able to benefit from a random ordering of the strides.

I have the code created to implement this, and it creates a pattern that can be used to add other algorithms that people may want to add (such as counting the rows and ranking each stride, and then ordering from most dense to least). The current two options I have coded is 'descending' and 'random.'

The original idea was to create something closer to Spark's hash partitioner, but for JDBC and pushed down to the database engine for efficiency. However, that would require adding hashing algorithms for each dialect, and the performance from those algorithms may outweigh the benefit. The method I'm proposing in this ticket avoids those complexities while still giving some of the benefit (in the case of random ordering).
Bug: Some `spark-submit`  commands used to run benchmarks in the user's guide is wrong
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 29/Mar/21 14:26
Affects Version/s: 3.2.0

For example, the guide for running JoinBenchmark as follows:

 
{code:java}
/**
 * Benchmark to measure performance for joins.
 * To run this benchmark:
 * {{{
 *   1. without sbt:
 *      bin/spark-submit --class <this class> --jars <spark core test jar> <spark sql test jar>
 *   2. build/sbt "sql/test:runMain <this class>"
 *   3. generate result:
 *      SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
 *      Results will be written to "benchmarks/JoinBenchmark-results.txt".
 * }}}
 */
object JoinBenchmark extends SqlBasedBenchmark {
{code}
 

 

but if we run JoinBenchmark with commnad

 
{code:java}
bin/spark-submit --class org.apache.spark.sql.execution.benchmark.JoinBenchmark --jars spark-core_2.12-3.2.0-SNAPSHOT-tests.jar spark-sql_2.12-3.2.0-SNAPSHOT-tests.jar 
{code}
 

The following exception will be thrown：

 
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/plans/SQLHelper
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
 

because SqlBasedBenchmark trait extends BenchmarkBase and SQLHelper, SQLHelper def in spark-catalyst-tests.jar.
Bug: Eliminate LEFT SEMI/ANTI join to its left child side with AQE
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 17/Mar/21 21:50
Affects Version/s: 3.2.0

In `EliminateJoinToEmptyRelation.scala`, we can extend it to cover more cases for LEFT SEMI and LEFT ANI joins:
 # Join is left semi join, join right side is non-empty and condition is empty. Eliminate join to its left side.
 # Join is left anti join, join right side is empty. Eliminate join to its left side.
Bug: Move partitioning and ordering to common limit trait
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 24/Mar/21 07:28
Affects Version/s: 3.2.0

Both local limit and global limit define the output partitioning and output ordering in the same way and this is duplicated ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala#L159-L175] ). We can move the output partitioning and ordering into their parent trait - BaseLimitExec. This is a minor code refactoring.
Bug: Replaces map and getOrElse(false/true) with exists/forall
Status: Resolved | Priority: Trivial | Resolution: Won't Fix
Created: 02/Mar/21 06:01
Affects Version/s: 3.2.0

Code Simpilefications:

*before*
{code:java}
option.map(booleanFunc).getOrElse(false)
option.map(booleanFunc).getOrElse(true)
{code}
*after*
{code:java}
option.exists(booleanFunc)
option.forall(booleanFunc)
{code}
Bug: Avoid unnecessary code generation in JoinCodegenSupport.genBuildSideVars
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 10/Mar/21 23:40
Affects Version/s: 3.2.0

As a followup from code review in [https://github.com/apache/spark/pull/31736#discussion_r588134104] , for `JoinCodegenSupport.genBuildSideVars`, we only need to generate build side variables with default values for LEFT OUTER and RIGHT OUTER join, but not for other join types (i.e. LEFT SEMI and LEFT ANTI). Create this Jira to followup clean up the code.
Bug: Exercise code-gen enable/disable code paths for SHJ in join test suites
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 11/Mar/21 06:54
Affects Version/s: 3.1.0

Per comment in [https://github.com/apache/spark/pull/31802#discussion_r592068440] , we would like to exercise whole stage code-gen enabled and disabled code paths in join unit test suites. This is for better test coverage of shuffled hash join.
Bug: Spark producing incorrect groupBy results when key is a struct with nullable properties
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 30/Jun/20 08:59
Affects Version/s: 3.0.0

I'm in the process of migrating from Spark 2.4.x to Spark 3.0.0 and I'm noticing a behaviour change in a particular aggregation we're doing, and I think I've tracked it down to how Spark is now treating nullable properties within the column being grouped by.
 
Here's a simple test I've been able to set up to repro it:
 

{code:scala}
case class B(c: Option[Double])
case class A(b: Option[B])
val df = Seq(
A(None),
A(Some(B(None))),
A(Some(B(Some(1.0))))
).toDF
val res = df.groupBy("b").agg(count("*"))
{code}

Spark 2.4.6 has the expected result:

{noformat}
> res.show
+-----+--------+
|    b|count(1)|
+-----+--------+
|   []|       1|
| null|       1|
|[1.0]|       1|
+-----+--------+

> res.collect.foreach(println)
[[null],1]
[null,1]
[[1.0],1]
{noformat}

But Spark 3.0.0 has an unexpected result:

{noformat}
> res.show
+-----+--------+
|    b|count(1)|
+-----+--------+
|   []|       2|
|[1.0]|       1|
+-----+--------+

> res.collect.foreach(println)
[[null],2]
[[1.0],1]
{noformat}

Notice how it has keyed one of the values in be as `[null]`; that is, an instance of B with a null value for the `c` property instead of a null for the overall value itself.

Is this an intended change?
Bug: SparkR CRAN check gives a warning with R 4.0.0 on OSX
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Jun/20 02:58
Affects Version/s: 2.4.6

When the SparkR package is run through a CRAN check (i.e. with something like R CMD check --as-cran ~/Downloads/SparkR_2.4.6.tar.gz), we rebuild the SparkR vignette as a part of the checks.

However this seems to be failing with R 4.0.0 on OSX -- both on my local machine and on CRAN https://cran.r-project.org/web/checks/check_results_SparkR.html

cc [~felixcheung]
Bug: Incorrect results for SUBSTRING when overflow
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 28/Jun/20 04:28
Affects Version/s: 1.6.3

SQL query SELECT SUBSTRING("abc", -1207959552, -1207959552) incorrectly returns "abc" against expected output of "".
 This is a result of integer overflow in addition [https://github.com/apache/spark/blob/8c44d744631516a5cdaf63406e69a9dd11e5b878/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L345]
Bug: Regression in handling NaN values in COUNT(DISTINCT)
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/Jun/20 23:44
Affects Version/s: 3.0.0

There seems to be a regression in Spark 3.0.0, with regard to how {{NaN}} values are normalized/handled in {{COUNT(DISTINCT ...)}}. Here is an illustration:
{code:scala}
case class Test( uid:String, score:Float)

val POS_NAN_1 = java.lang.Float.intBitsToFloat(0x7f800001)
val POS_NAN_2 = java.lang.Float.intBitsToFloat(0x7fffffff)

val rows = Seq(

 Test("mithunr",  Float.NaN),	
 Test("mithunr",  POS_NAN_1),
 Test("mithunr",  POS_NAN_2),
 Test("abellina", 1.0f),
 Test("abellina", 2.0f)

).toDF.createOrReplaceTempView("mytable")

spark.sql(" select uid, count(distinct score) from mytable group by 1 order by 1 asc ").show
{code}
Here are the results under Spark 3.0.0:
{code:java|title=Spark 3.0.0 (single aggregation)}
+--------+---------------------+
|     uid|count(DISTINCT score)|
+--------+---------------------+
|abellina|                    2|
| mithunr|                    3|
+--------+---------------------+
{code}
Note that the count against {{mithunr}} is {{3}}, accounting for each distinct value for {{NaN}}.
 The right results are returned when another aggregation is added to the GBY:
{code:scala|title=Spark 3.0.0 (multiple aggregations)}
scala> spark.sql(" select uid, count(distinct score), max(score) from mytable group by 1 order by 1 asc ").show
+--------+---------------------+----------+
|     uid|count(DISTINCT score)|max(score)|
+--------+---------------------+----------+
|abellina|                    2|       2.0|
| mithunr|                    1|       NaN|
+--------+---------------------+----------+
{code}
Also, note that Spark 2.4.6 normalizes the {{DISTINCT}} expression correctly:
{code:scala|title=Spark 2.4.6}
scala> spark.sql(" select uid, count(distinct score) from mytable group by 1 order by 1 asc ").show

+--------+---------------------+
|     uid|count(DISTINCT score)|
+--------+---------------------+
|abellina|                    2|
| mithunr|                    1|
+--------+---------------------+
{code}
Bug: K8s integration tests are failing with mockito errors
Status: Resolved | Priority: Blocker | Resolution: Duplicate
Created: 03/Jun/20 23:07
Affects Version/s: 3.1.0

The K8s integration tests are failing consistently with Mockito errors. Looking at the error it seems like we may have inadvertently (or intentionally) bumped our mockito or bytebuddy versions.
Bug: Streaming's state store compatibility is broken
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 14/Jun/20 19:10
Affects Version/s: 3.0.0

[This line|https://github.com/apache/spark/pull/28062/files#diff-7a46f10c3cedbf013cf255564d9483cdR2458] of [https://github.com/apache/spark/pull/28062] changed the order of groupCols in dropDuplicates(). Thus, the executor JVM could probably crash, throw a random exception or even return a wrong answer when using the checkpoint written by the previous version.
Bug: Loading jobs UI page takes 40 seconds
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 11/Jun/20 07:58
Affects Version/s: 2.4.6

In the latest master branch, I find that the job list page becomes very slow.
To reproduce in local setup:

{code:java}
spark.read.parquet("/tmp/p1").createOrReplaceTempView("t1")
spark.read.parquet("/tmp/p2").createOrReplaceTempView("t2")
(1 to 1000).map(_ =>  spark.sql("select * from t1, t2 where t1.value=t2.value").show())
{code}

And that, open live UI: http://localhost:4040/
The loading time is about 40 seconds.

If we comment out the function call for `drawApplicationTimeline`, then the loading time is around 1 second.
Bug: First day of week changed for non-MONDAY_START Lacales
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 01/Jun/20 02:43
Affects Version/s: 3.0.0

h1. cases
{code:sql}
spark-sql> select to_timestamp('2020-1-1', 'YYYY-w-u');
2019-12-29 00:00:00
spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
spark.sql.legacy.timeParserPolicy	legacy
spark-sql> select to_timestamp('2020-1-1', 'YYYY-w-u');
2019-12-30 00:00:00
{code}

h1. reasons

These week-based fields need Locale to express their semantics, the first day of the week varies from country to country.

From the Java doc of WeekFields
{code:java}
    /**
     * Gets the first day-of-week.
     * <p>
     * The first day-of-week varies by culture.
     * For example, the US uses Sunday, while France and the ISO-8601 standard use Monday.
     * This method returns the first day using the standard {@code DayOfWeek} enum.
     *
     * @return the first day-of-week, not null
     */
    public DayOfWeek getFirstDayOfWeek() {
        return firstDayOfWeek;
    }
{code}

But for the SimpleDateFormat, the day-of-week is not localized

```
u	Day number of week (1 = Monday, ..., 7 = Sunday)	Number	1
```

Currently, the default locale we use is the US, so the result moved a day backward.

For other countries, please refer to [First Day of the Week in Different Countries|http://chartsbin.com/view/41671]

h1. solution options

1. Use new Locale("en", "GB") as default locale.
2. For JDK10 and onwards, we can set locale Unicode extension 'fw'  to 'mon', but not work for lower JDKs
3. Forbid 'u', give user proper exceptions, and enable and document 'e/c'. Currently, the 'u' is internally substituted by 'e', but they are not equivalent.

1 and 2 can solve this with default locale but not for the functions with custom locale supported.

cc [~cloud_fan] [~dongjoon] [~maropu]
Bug: Use iloc for positional slicing instead of direct slicing in createDataFrame with Arrow
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 25/Jun/20 10:44
Affects Version/s: 2.4.6

When you use floats are index of pandas, it produces a wrong results:

{code}
>>> import pandas as pd
>>> spark.createDataFrame(pd.DataFrame({'a': [1,2,3]}, index=[2., 3., 4.])).show()
+---+
|  a|
+---+
|  1|
|  1|
|  2|
+---+
{code}

This is because direct slicing uses the value as index when the index contains floats:

{code}
>>> pd.DataFrame({'a': [1,2,3]}, index=[2., 3., 4.])[2:]
     a
2.0  1
3.0  2
4.0  3
>>> pd.DataFrame({'a': [1,2,3]}, index=[2., 3., 4.]).iloc[2:]
     a
4.0  3
>>> pd.DataFrame({'a': [1,2,3]}, index=[2, 3, 4])[2:]
   a
4  3
{code}
Bug: Unable to Set `spark.ui.port` configuration in Yarn Cluster Mode
Status: Closed | Priority: Critical | Resolution: Duplicate
Created: 20/Jun/20 09:44
Affects Version/s: 2.4.0

Spark Web UI port in Yarn cluster mode always gets a random number since we disable the configuration spark.ui.port by setting this property to be 0 always. 

There are some use cases where we need to run Web UI port on some specified range because of some security concerns and unable to do so with the configuration. 

 

[https://github.com/apache/spark/blob/842b1dcdff0ecab4af9f292c2ff7b2b9ae1ac40a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L216]
Bug: Use fine-grained read write lock for each database in HiveExternalCatalog
Status: In Progress | Priority: Major | Resolution: 
Created: 28/Jun/20 14:16
Affects Version/s: 3.0.0

In HiveExternalCatalog, all metastore operations are synchronized by a same object lock. In a heavy traffic Spark thriftserver or Spark Driver, users's queries may be stuck by any a long operation. For example, if a user is accessing a table which contains mass partitions, the operation loadDynamicPartitions() holds the object lock for a long time. All queries are blocking to wait for the lock.
Bug: Drop references to slave
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jun/20 18:33
Affects Version/s: 3.1.0

We have a lot of references to "slave" in the code base which doesn't match the terminology in the rest of our code base and we should clean it up. In many situations it would be clearer with "executor", "worker", or "replica" depending on the context (so this is not just a search and replace but actually read through the code and make it consistent).

 

We may want to (in a follow on) explore renaming master to something more precise.