av9ash/GitBugs huggingface dataset:
Bug: Downgrade scala-maven-plugin to 4.3.0
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 05/Jan/21 07:42
Affects Version/s: 3.1.0

After we upgraded scala-maven-plugin to 4.4.0 at SPARK-33512, the docker release script fails as below:

{code}
[INFO] Compiling 21 Scala sources and 3 Java sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes ...
[ERROR] ## Exception when compiling 24 sources to /opt/spark-rm/output/spark-3.1.0-bin-hadoop2.7/resource-managers/yarn/target/scala-2.12/test-classes
java.lang.SecurityException: class "javax.servlet.SessionCookieConfig"'s signer information does not match signer information of other classes in the same package
java.lang.ClassLoader.checkCerts(ClassLoader.java:891)
java.lang.ClassLoader.preDefineClass(ClassLoader.java:661)
java.lang.ClassLoader.defineClass(ClassLoader.java:754)
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
java.net.URLClassLoader.access$100(URLClassLoader.java:74)
java.net.URLClassLoader$1.run(URLClassLoader.java:369)
java.net.URLClassLoader$1.run(URLClassLoader.java:363)
java.security.AccessController.doPrivileged(Native Method)
java.net.URLClassLoader.findClass(URLClassLoader.java:362)
java.lang.ClassLoader.loadClass(ClassLoader.java:418)
java.lang.ClassLoader.loadClass(ClassLoader.java:351)
java.lang.Class.getDeclaredMethods0(Native Method)
java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
java.lang.Class.privateGetPublicMethods(Class.java:2902)
java.lang.Class.getMethods(Class.java:1615)
sbt.internal.inc.ClassToAPI$.toDefinitions0(ClassToAPI.scala:170)
sbt.internal.inc.ClassToAPI$.$anonfun$toDefinitions$1(ClassToAPI.scala:123)
scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
sbt.internal.inc.ClassToAPI$.toDefinitions(ClassToAPI.scala:123)
sbt.internal.inc.ClassToAPI$.$anonfun$process$1(ClassToAPI.scala:33)
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
{code}
Bug: Avro should read decimal values with the file schema
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 25/Jan/21 20:32
Affects Version/s: 2.4.7

{code:java}
scala> sql("SELECT 3.14 a").write.format("avro").save("/tmp/avro")
scala> spark.read.schema("a DECIMAL(4, 3)").format("avro").load("/tmp/avro").show
+-----+
|    a|
+-----+
|0.314|
+-----+ {code}
Bug: Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12
Status: Resolved | Priority: Blocker | Resolution: Done
Created: 28/Jan/21 04:34
Affects Version/s: 3.2.0

Before the release, we need to double check the unreleased/unresolved JIRAs/PRs of Parquet 1.11/1.12 and then decide whether we should upgrade/revert Parquet. At the same time, we should encourage the whole community to do the compatibility and performance tests for their production workloads, including both read and write code paths.

More details: [https://github.com/apache/spark/pull/26804#issuecomment-768790620]
Bug: For parquet table, after changing the precision and scale of decimal type in hive, spark reads incorrect value
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 23/Jan/21 13:09
Affects Version/s: 2.4.5

In Hive, 

{code}
create table test_decimal(amt decimal(18,2)) stored as parquet; 
insert into test_decimal select 100;
alter table test_decimal change amt amt decimal(19,3);
{code}


In Spark,

{code}
select * from test_decimal;
{code}

{code}
+--------+
|    amt |
+--------+
| 10.000 |
+--------+
{code}
Bug: NPE in ExecutorPodsSnapshot
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 06/Jan/21 00:39
Affects Version/s: 3.1.0

Currently the test (finishedExecutorWithRunningSidecar in test utils depends on nulls matching) passes because we match the null on the container state string. To fix this label both the statuses and ensure the ExecutorPodSnapshot starts with the default config to match.
Bug: "show create table" doesn't work for v2 table
Status: Resolved | Priority: Blocker | Resolution: Not A Problem
Created: 06/Jan/21 21:13
Affects Version/s: 3.1.0

I was QAing Spark 3.1.0 RC1 and found one regression: "show create table" doesn't work for v2 table.

But when using Spark 3.0.1, "show create table" works for v2 table.

Steps to test:
{code:java}
/bin/spark-shell --packages io.delta:delta-core_2.12:0.7.0 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

scala> spark.sql("create table foo(i INT) using delta")
 res0: org.apache.spark.sql.DataFrame = []

scala> spark.sql("show create table foo").show(false)

+-----------------------------------------------+
|createtab_stmt                                 |
+-----------------------------------------------+
|CREATE TABLE `default`.`foo` (
  )
USING delta
|
+-----------------------------------------------+
{code}
Looks like it's caused by [https://github.com/apache/spark/pull/30321]

 which blocks "show create table" for v2.
Bug: Deconflict the jars jakarta.servlet-api-4.0.3.jar and javax.servlet-api-3.1.0.jar
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 14/Jan/21 06:14
Affects Version/s: 3.1.0

After SPARK-33705, we now happened to have two jars in the release artifact with Hadoop 3:

{{dev/deps/spark-deps-hadoop-3.2-hive-2.3}}:

{code}
...
jakarta.servlet-api/4.0.3//jakarta.servlet-api-4.0.3.jar
...
javax.servlet-api/3.1.0//javax.servlet-api-3.1.0.jar
...
{code}

It can potentially cause an issue, and we should better remove {{javax.servlet-api-3.1.0.jar}} which is apparently only required for YARN tests.
Bug: Add UnivariateFeatureSelector to deprecate existing selectors
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 12/Jan/21 01:04
Affects Version/s: 3.1.1

In SPARK-26111, we introduced a few univariate feature selectors, which share a common set of params. And they are named after the underlying test, which requires users to understand the test to find the matched scenarios. It would be nice if we introduce a single class called UnivariateFeatureSelector that accepts a selection criterion and a score method (string names). Then we can deprecate all other univariate selectors.

For the params, instead of ask users to provide what score function to use, it is more friendly to ask users to specify the feature and label types (continuous or categorical) and we set a default score function for each combo. We can also detect the types from feature metadata if given. Advanced users can overwrite it (if there are multiple score function that is compatible with the feature type and label type combo). Example (param names are not finalized):

{code}
selector = UnivariateFeatureSelector(featureCols=["x", "y", "z"], labelCol=["target"], featureType="categorical", labelType="continuous", select="bestK", k=100)
{code}

cc: [~huaxingao] [~ruifengz] [~weichenxu123]
Bug: Please reduce GitHub Actions matrix or improve the build time
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 08/Jan/21 19:42
Affects Version/s: 3.0.1

GitHub Actions queue is very high for Apache projects, and it looks like a significant number of executors are occupied by Spark jobs :-(

Note: all Apache projects share the same limit of shared GitHub Actions runners, and based on the chart below Spark is consuming 20+ runners while the total limit (for all ASF projects) is 180.


See https://lists.apache.org/thread.html/r5303eec41cc1dfc51c15dbe44770e37369330f9644ef09813f649120%40%3Cbuilds.apache.org%3E

>number of GA workflows in/progress/queued per project and they clearly show the situation is getting worse by day: https://pasteboard.co/JIJa5Xg.png
Bug: Rule conflicts between PaddingAndLengthCheckForCharVarchar and ResolveAggregateFunctions
Status: Resolved | Priority: Critical | Resolution: Fixed
Created: 05/Jan/21 06:58
Affects Version/s: 3.1.0

ResolveAggregateFunctions is a hacky rule and it calls `executeSameContext` to generate a `resolved agg` to determine which unresolved sort attribute should be pushed into the agg. However, after we add the PaddingAndLengthCheckForCharVarchar rule which will rewrite the query output, thus, the `resolved agg` cannot match original attributes anymore. 

It causes some dissociative sort attribute to be pushed in and fails the query


{code:java}
[info]   Failed to analyze query: org.apache.spark.sql.AnalysisException: expression 'testcat.t1.`v`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;
[info]   Project [v#14, sum(i)#11L]
[info]   +- Sort [aggOrder#12 ASC NULLS FIRST], true
[info]      +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS sum(i)#11L, v#13 AS aggOrder#12]
[info]         +- SubqueryAlias testcat.t1
[info]            +- Project [if ((length(v#6) <= 3)) v#6 else if ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of length , cast(length(v#6) as string),  exceeds varchar type length limitation: 3)) as string) else rpad(rtrim(v#6, None), 3,  ) AS v#14, i#7]
[info]               +- RelationV2[v#6, i#7, index#15, _partition#16] testcat.t1
[info]
[info]   Project [v#14, sum(i)#11L]
[info]   +- Sort [aggOrder#12 ASC NULLS FIRST], true
[info]      +- !Aggregate [v#14], [v#14, sum(cast(i#7 as bigint)) AS sum(i)#11L, v#13 AS aggOrder#12]
[info]         +- SubqueryAlias testcat.t1
[info]            +- Project [if ((length(v#6) <= 3)) v#6 else if ((length(rtrim(v#6, None)) > 3)) cast(raise_error(concat(input string of length , cast(length(v#6) as string),  exceeds varchar type length limitation: 3)) as string) else rpad(rtrim(v#6, None), 3,  ) AS v#14, i#7]
[info]               +- RelationV2[v#6, i#7, index#15, _partition#16] testcat.t1
{code}
Bug: datasourceV1 VS  dataSourceV2 
Status: Closed | Priority: Critical | Resolution: Invalid
Created: 06/Jan/21 07:45
Affects Version/s: 3.0.0

I found that DataSourceV2 has been through many versions .So why hasn't datasourceV2 been used by default until now in the latest version.I want to know if it’s because there is a big difference in execution efficiency between v1 and v2. Or there are other reasons.

Thanks a lot
Bug: New type coercion syntax rules in ANSI mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Jan/21 15:30
Affects Version/s: 3.1.2

Add new implicit cast syntax rules in ANSI mode.
In Spark ANSI mode, the type coercion rules are based on the type precedence lists of the input data types. 
As per the section "Type precedence list determination" of "ISO/IEC 9075-2:2011
Information technology — Database languages - SQL — Part 2: Foundation (SQL/Foundation)", the type precedence lists of primitive
 data types are as following:
* Byte: Byte, Short, Int, Long, Decimal, Float, Double
* Short: Short, Int, Long, Decimal, Float, Double
* Int: Int, Long, Decimal, Float, Double
* Long: Long, Decimal, Float, Double
* Decimal: Any wider Numeric type
* Float: Float, Double
* Double: Double
* String: String
* Date: Date, Timestamp
* Timestamp: Timestamp
* Binary: Binary
* Boolean: Boolean
* Interval: Interval
As for complex data types, Spark will determine the precedent list recursively based on their sub-types.

With the definition of type precedent list, the general type coercion rules are as following:
* Data type S is allowed to be implicitly cast as type T iff T is in the precedence list of S
* Comparison is allowed iff the data type precedence list of both sides has at least one common element. When evaluating the comparison, Spark casts both sides as the tightest common data type of their precedent lists.
* There should be at least one common data type among all the children's precedence lists for the following operators. The data type of the operator is the tightest common precedent data type.

{code:java}
In
Except(odd)
Intersect
Greatest
Least
Union
If
CaseWhen
CreateArray
Array Concat
Sequence
MapConcat
CreateMap
{code}


* For complex types (struct, array, map), Spark recursively looks into the element type and applies the rules above. If the element nullability is converted from true to false, add runtime null check to the elements.
Bug: Faster way to display/render entries in HistoryPage (Spark history server summary page)
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/21 06:01
Affects Version/s: 3.2.0

Since a long time ago my team/company suffered from history server being very slow to display/search entries specially when entries grow over 50k entry, regardless there is a pagination there in that page already but still very slow to display the entries.

  

Current situation *Mustache Js* is used to render the entries and *datatables* is used to manipulate it (sort by column and search).

 

By getting rid of *Mustache*  (stop rendering the entries using *Mustache*) and using *datatables*  to display it proved to be faster.

 

Displaying > 100k entries (my case):

Existing takes at least 30 to 40 seconds to display the entries, searching takes at least 20 seconds and the page stop responding until it finishes.

Improved takes ~3 seconds to display the entries searching is very fast and the page stays responsive.

*(These numbers will be different for others since JS is executed on your browser)*

 

I am not sure why *Mustache* is used to display the data since data tables can do the job,

[~ajbozarth] [~sowen]  please elaborate more about this what is the reason to use *Mustache*? what are the drawbacks if it's not used anymore to display the entries (only this part)?
Bug: Review and fix issues in API docs
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Jan/21 04:47
Affects Version/s: 3.1.1

Compare the 3.1.1 API doc with the latest release version 3.0.1. Fix the following issues:
 * Add missing `Since` annotation for new APIs
 * Remove the leaking class/object in API doc
Bug: When use input_file_name() func all column from file appeared in physical plan of query, not only projection.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/21 12:52
Affects Version/s: 2.4.7

input_file_name() function damage applying projection to the physical plan of the query.
 if use this function and a new column, column-oriented formats like parquet and orc put all columns to Physical plan.
 While without it, only selected columns uploaded.
 In my case, performance influence is x30.
{code:java}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object TestSize {

  def main(args: Array[String]): Unit = {
    implicit val spark: SparkSession = SparkSession.builder()
      .master("local")
      .config("spark.sql.shuffle.partitions", "5")
      .getOrCreate()

    import spark.implicits._

    val query1 = spark.read.parquet(
      "s3a://part-00040-a19f0d20-eab3-48ef-be5a-602c7f9a8e58.c000.gz.parquet"
    )
      .select($"app_id", $"idfa", input_file_name().as("fileName"))
      .distinct()
      .count()

   val query2 = spark.read.parquet( "s3a://part-00040-a19f0d20-eab3-48ef-be5a- 602c7f9a8e58.c000.gz.parquet" ) 
      .select($"app_id", $"idfa")
      .distinct() 
      .count()

    Thread.sleep(10000000000L)

  }

}
{code}
`query1` has all columns in the physical plan, while `query2` only two.
Bug: Long runtime on many environment variables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/21 12:40
Affects Version/s: 2.4.0

I am not sure if this is a bug report or a feature request. The code is is the same in current versions of Spark and maybe this ticket saves someone some time for debugging.

We migrated some older code to Spark 2.4.0, and suddently the integration tests on our build machine were much slower than expected.

On local machines it was running perfectly.

At the end it turned out, that Spark was wasting CPU Cycles during DataFrame analyzing in the following functions
 * AnalysisHelper.assertNotAnalysisRule calling
 * Utils.isTesting

Utils.isTesting is traversing all environment variables.

The offending build machine was a Kubernetes Pod which automatically exposed all services as environment variables, so it had more than 3000 environment variables.

As Utils.isTesting is called very often throgh AnalysisHelper.assertNotAnalysisRule (via AnalysisHelper.transformDown, transformUp).

 

Of course we will restrict the number of environment variables, on the other side Utils.isTesting could also use a lazy val for

 
{code:java}
sys.env.contains("SPARK_TESTING") {code}
 

to not make it that expensive.
Bug: Intersect operator missing rowCount when CBO enabled
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/21 03:35
Affects Version/s: 3.2.0

https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/BasicStatsPlanVisitor.scala#L60 missing the row count.
Bug: Remove unused `getRawTable()` from `HiveExternalCatalog.alterPartitions()`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Jan/21 13:44
Affects Version/s: 3.2.0

This https://github.com/apache/spark/blob/157b72ac9fa0057d5fd6d7ed52a6c4b22ebd1dfc/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L1148 can be removed.
Bug: SQL running error, spark does not exit, resulting in data quality problems
Status: Open | Priority: Major | Resolution: 
Created: 15/Jan/21 08:39
Affects Version/s: 3.0.1

Spark SQL executes a SQL file containing multiple SQL segments. Because one of the SQL segments fails to run, but spark driver or spark context does not exit, an error will occur. The table written by the SQL segment is empty or old data. Depending on this problematic table, the subsequent SQL will have data quality problems even if it runs successfully.
Bug: Allow users to specify a maximum decommissioning time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 23:57
Affects Version/s: 3.1.0

We currently have the ability for users to set the predicted time of the cluster manager or cloud provider to terminate a decommissioning executor, but for nodes where Spark it's self is triggering decommissioning we should add the ability of users to specify a maximum time we want to allow the executor to decommission.

 

This is important especially if we start to in more places (like with excluded executors that are found to be flaky, that may or may not be able to decommission successfully).
Bug: In addition to killing exlcuded/flakey executors which should support decommissioning
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jan/21 23:59
Affects Version/s: 3.2.0

Decommissioning will give the executor a chance to migrate it's files to a more stable node.

 

Note: we want SPARK-34104 to be integrated as well so that flaky executors which can not decommission are eventually killed.
Bug: ALTER TABLE .. ADD PARTITION doesn't refresh cache
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/21 07:16
Affects Version/s: 3.0.1

Here is the example to reproduce the issue:
{code:sql}
spark-sql> create table tbl2 (col int, part int) partitioned by (part);
spark-sql> insert into tbl2 partition (part=0) select 0;
spark-sql> cache table tbl2;
spark-sql> select * from tbl2;
0	0
spark-sql> show table extended like 'tbl2' partition (part = 0);
default	tbl2	false	Partition Values: [part=0]
Location: file:/Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=0
...
{code}
Add new partition by copying the existing one:
{code}
cp -r /Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=0 /Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=1
{code}
 Add new partition and select the table:
{code}
spark-sql> alter table tbl2 add partition (part = 1) location '/Users/maximgekk/proj/add-partition-refresh-cache-2/spark-warehouse/tbl2/part=1';
spark-sql> select * from tbl2;
0	0
{code}
We see only old data.
Bug: Cancel the duplicated jobs only in PRs at GitHub Actions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jan/21 06:23
Affects Version/s: 3.2.0

Currently the jobs are being canceled in master branch.
If a commit is merged before the test finishes, it cancels the previous builds.
This is a problem because we cannot, for example, detect logical conflict properly. We should only cancel the jobs in PRs.
Bug: Column pruning is not working as expected for PERMISIVE mode
Status: Open | Priority: Major | Resolution: 
Created: 07/Jan/21 08:13
Affects Version/s: 2.4.7

In PERMISSIVE mode

Given a csv with multiple columns per row, if your file schema has a single column and you are doing a SELECT in SQL with a condition like '<corrupt_record_column_name> is null', the row is marked as corrupted

 

BUT if you add an extra column in the file schema and you are not putting that column in SQL SELECT , the row is not marked as corrupted

 

PS. I don't know exactly what is the right behavior, I didn't find it for PERMISSIVE mode the documentation.

What I found is: As an example, CSV file contains the "id,name" header and one row "1234". In Spark 2.4, the selection of the id column consists of a row with one column value 1234 but in Spark 2.3 and earlier, it is empty in the DROPMALFORMED mode. To restore the previous behavior, set {{spark.sql.csv.parser.columnPruning.enabled}} to {{false}}.

 

[https://spark.apache.org/docs/2.4.0/sql-migration-guide-upgrade.html]

 

I made a "unit" test in order to exemplify the issue: [https://github.com/butzy92/spark-column-mapping-issue/blob/master/src/test/java/spark/test/SparkTest.java]
Bug: Check isNull and isNotNull in tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/21 15:49
Affects Version/s: 3.2.0

Hive related tickets:
HIVE-11898 support default partition in metastoredirectsql
HIVE-13115 MetaStore Direct SQL getPartitions call fail when the columns schemas for a partition are null
Bug: Miscellaneous cleanup for new PySpark documentation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jan/21 07:14
Affects Version/s: 3.1.0

1. Add a link of quick start in PySpark docs into "Programming Guides" in Spark main docs
2. ML MLlib -> MLlib (DataFrame-based)" and "MLlib (RDD-based)"
3. Mention MLlib user guide (https://dist.apache.org/repos/dist/dev/spark/v3.1.0-rc1-docs/_site/ml-guide.html)
4. Mention other migration guides as well because PySpark can get affected by it.
Bug: Union operator missing rowCount when enable CBO
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/21 13:20
Affects Version/s: 3.2.0

{code:scala}
spark.sql("CREATE TABLE t1 USING parquet AS SELECT id FROM RANGE(10)")
spark.sql("CREATE TABLE t2 USING parquet AS SELECT id FROM RANGE(10)")
spark.sql("ANALYZE TABLE t1 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("ANALYZE TABLE t2 COMPUTE STATISTICS FOR ALL COLUMNS")
spark.sql("set spark.sql.cbo.enabled=true")
spark.sql("SELECT * FROM t1 UNION ALL SELECT * FROM t2").explain("cost")
{code}

Current:
{noformat}
== Optimized Logical Plan ==
Union false, false, Statistics(sizeInBytes=320.0 B)
:- Relation[id#5880L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
+- Relation[id#5881L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
{noformat}

Expected
{noformat}
== Optimized Logical Plan ==
Union false, false, Statistics(sizeInBytes=320.0 B, rowCount=20)
:- Relation[id#2138L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
+- Relation[id#2139L] parquet, Statistics(sizeInBytes=160.0 B, rowCount=10)
{noformat}