av9ash/GitBugs huggingface dataset:
Bug: Respect sql execution id when scheduling taskSets
Status: Open | Priority: Major | Resolution: 
Created: 10/Mar/20 09:40
Affects Version/s: 3.1.0

Currently, spark will sort taskSets by jobId and stageId and then schedule them in order for FIFO schedulingMode. In OLAP senerios, especially under high concurrency, the taskSets are always from different sql queries and several jobs can be submitted for execution at one time  for one query for adaptive execution. But now we order those taskSets without considering the execution group, which may causes the query being delayed.
So I propose to consider the sql execution id when scheduling jobs.
Bug: implement VarianceThresholdSelector
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Mar/20 00:55
Affects Version/s: 3.1.0

Implement a Feature selector that removes all low-variance features. Features with a
 variance lower than the threshold will be removed. The default is to keep all features with non-zero variance, i.e. remove the features that have the same value in all samples.
Bug: set spark.master and spark.app.name conf default value
Status: Closed | Priority: Major | Resolution: Not A Bug
Created: 19/Mar/20 15:04
Affects Version/s: 3.1.0

I see the default value of master setting in spark-submit client
{code:java}
// Global defaults. These should be keep to minimum to avoid confusing behavior. master = Option(master).getOrElse("local[*]") 
{code}
but during our development and debugging, We will encounter this kind of problem

Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration

This conflicts with the default setting

 
{code:java}
//If we do
 val sparkConf = new SparkConf().setAppName(“app”)
 //When using the client to submit tasks to the cluster, the matser will be overwritten by the local
 sparkConf.set("spark.master", "local[*]"){code}
 

so we have to do like this
{code:java}
val sparkConf = new SparkConf().setAppName(“app”)
 //Because the program runs to set the priority of the master, we have to first determine whether to set the master to avoid submitting the cluster to run.
 sparkConf.set("spark.master",sparkConf.get("spark.master","local[*]")){code}
 

 

so is spark.app.name

Is it better for users to handle it like submit client ?
Bug: Lots of columns and distinct aggregation functions triggers compile exception on Janino
Status: Open | Priority: Major | Resolution: 
Created: 11/Mar/20 07:07
Affects Version/s: 2.3.4

We got some report on failure on user's query which Janino throws error on compiling generated code. The issue is here: [janino-compiler/janino#113|https://github.com/janino-compiler/janino/issues/113] 

It contains the information of generated code, symptom (error), and analysis of the bug, so please refer the link for more details.

It would be ideal to upgrade Janino to the new version which contains the fix [janino-compiler/janino#114|https://github.com/janino-compiler/janino/pull/114] - SPARK-31101 tracks the effort to upgrade Janino, but given upgrading Janino to 3.1.1 seems to bring lots of test failures and there's no guarantee to have Janino 3.0.16, we may need to provide workaround to avoid hitting Janino bug.
Bug: Convert Catalyst's DATE/TIMESTAMP to Java Date/Timestamp via local date-time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 19:55
Affects Version/s: 3.0.0

By default, collect() returns java.sql.Timestamp/Date instances with offsets derived from internal values of Catalyst's TIMESTAMP/DATE that store microseconds since the epoch. The conversion from internal values to java.sql.Timestamp/Date based on Proleptic Gregorian calendar but converting the resulted values before 1582 year to strings produces timestamp/date string in Julian calendar. For example:
{code}
scala> sql("select date '1100-10-10'").collect()
res1: Array[org.apache.spark.sql.Row] = Array([1100-10-03])
{code} 

This can be fixed if internal Catalyst's values are converted to local date-time in Gregorian calendar,  and construct local date-time from the resulted year, month, ..., seconds in Julian calendar.
Bug: Disable useless and uncleaned hive SessionState initialization parts
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 06:33
Affects Version/s: 3.0.0

As a common usage and according to the spark doc, users may often just copy their hive-site.xml to Spark directly from hive projects. Sometimes, the config file is not that clean for spark and may cause some side effects

for example, hive.session.history.enabled will create a log for the hive jobs but useless for spark and also it will not be deleted on JVM exit.
Bug: Inconsistent rdd and output partitioning for bucket table when output doesn't contain all bucket columns
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Mar/20 09:35
Affects Version/s: 2.2.3

For a bucketed table, when deciding output partitioning, if the output doesn't contain all bucket columns, the result is `UnknownPartitioning`. But when generating rdd, current Spark uses `createBucketedReadRDD` because it doesn't check if the output contains all bucket columns. So the rdd and its output partitioning are inconsistent.
Bug: Spark Encoders.bean() should allow marking non-null fields in its Spark schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 15:38
Affects Version/s: 2.4.4

Spark _Encoders.bean()_ method should allow the generated StructType schema fields be *non-nullable*.

Currently, any non-primitive type is automatically _nullable_. It is hard-coded in the _org.apache.spark.sql.catalyst.JavaTypeReference_ class.  This can lead to rather interesting situations... For example, let's say I want to save a dataframe using an Avro format with my own non-spark generated Avro schema. Let's also say that my Avro schema has a field that is non-null (i.e., not a union type). Well, it appears *impossible* to store a dataframe using such an Avro schema since Spark would assume that the field is nullable (as it is in its own schema) which would conflict with Avro schema semantics and throw an exception.

I propose making a change to the _JavaTypeReference_ class to observe the JSR-305 _Nonnull_ annotation (and its children) on the provided bean class during StructType schema generation. This would allow bean creators to control the resulting Spark schema so much better.
Bug: Spark 2.4.4 count distinct query much slower than Spark 1.6.2 and Hive 1.2.1
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 13/Mar/20 06:28
Affects Version/s: 2.4.4

In our company, we are doing the migration of our ad-hoc query engine from Hive to Spark. We use Spark thrift server, and the version is 2.4.4. Many of the queries run well and faster than Hive. But we have a complex query with multiple count(distinct) expression. This query runs extremely slow on Spark, comparing to Hive. It runs much slower even comparing to Spark 1.6.2. The query is as flowing:
{code:java}
select
'All'as industry_name
,sum(shop_cnt)/31 as shop_cnt
,sum(sku_cnt)/31 as sku_cnt
,sum(limit_act_sku_cnt)/31 as limit_act_sku_cnt
,sum(discount_act_sku_cnt)/31 as discount_act_sku_cnt
,sum(sku_mid_price)/31 as sku_mid_price
,sum(sku_high_mid_price)/31 as sku_high_mid_price
FROM
(
select
cal_dt
,approx_count_distinct(shop_id) as shop_cnt
,sum(sku_cnt)/approx_count_distinct(shop_id) as sku_cnt
,sum(limit_act_sku_cnt)/approx_count_distinct(shop_id) as limit_act_sku_cnt
,sum(discount_act_sku_cnt)/approx_count_distinct(shop_id) as discount_act_sku_cnt
,percentile(cast(sku_mid_price as bigint) ,0.5) as sku_mid_price
,percentile(cast(sku_high_mid_price as bigint),0.75) as sku_high_mid_price
from
(
select
cal_dt
,vender_id
,shop_id
,approx_count_distinct(sku_id) as sku_cnt
,approx_count_distinct(case when is_limit_grab_act_sku=1 then sku_id end) as limit_act_sku_cnt
,approx_count_distinct(case when is_offer_act_sku=1 then sku_id end) as discount_act_sku_cnt
,percentile(cast(sku_price as bigint),0.5) as sku_mid_price
,percentile(cast(sku_price as bigint),0.75) as sku_high_mid_price
from bi_dw.dw_dj_prd_shop_sku_info
where cal_dt = '2019-12-01'
group by cal_dt, vender_id, shop_id
) a
group by cal_dt
) a;
{code}
The query took about 18 minutes to run on Spark 2.4.4. And it took only about 80 seconds to run on Hive 1.2.1. On Spark 1.6.2, it only took about 2 to 3 minutes (run on Spark shell, no accurate time taken output).

When investigating this, I found the Jira https://issues.apache.org/jira/browse/SPARK-9241, which optimizes count disctint. But when I look at the code of Spark 2.4.4, I found the related code is not there. 

So my question is: why the code is removed?
Bug: IntervalBenchmark and DateTimeBenchmark fails to run
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Mar/20 06:53
Affects Version/s: 3.0.0

[error] Caused by: java.time.format.DateTimeParseException: Text '2019-01-27 11:02:01.0' could not be parsed at index 20
[error] 	at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
[error] 	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
[error] 	at org.apache.spark.sql.catalyst.util.Iso8601TimestampFormatter.$anonfun$parse$1(TimestampFormatter.scala:71)
[error] 	... 19 more
Bug: .ClassNotFoundException CoarseGrainedClusterMessages$RetrieveDelegationTokens
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 09/Mar/20 03:28
Affects Version/s: 2.4.0

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages$RetrieveDelegationTokens$
Bug: outputOrdering should handle aliases correctly
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Mar/20 00:45
Affects Version/s: 3.0.0

Currently, `outputOrdering` doesn't respect aliases. Thus, the following would produce an unnecessary sort node:

{code:java}
withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "0") {
      val df = (0 until 20).toDF("i").as("df")
      df.repartition(8, df("i")).write.format("parquet")
        .bucketBy(8, "i").sortBy("i").saveAsTable("t")
      val t1 = spark.table("t")
      val t2 = t1.selectExpr("i as ii")
      t1.join(t2, t1("i") === t2("ii")).explain
    }
{code}

would produce an unnecessary sort node:
{code:java}
== Physical Plan ==
*(3) SortMergeJoin [i#8], [ii#10], Inner
:- *(1) Project [i#8]
:  +- *(1) Filter isnotnull(i#8)
:     +- *(1) ColumnarToRow
:        +- FileScan parquet default.t[i#8] Batched: true, DataFilters: [isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/..., PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int>, SelectedBucketsCount: 8 out of 8
+- *(2) Sort [ii#10 ASC NULLS FIRST], false, 0
   +- *(2) Project [i#8 AS ii#10]
      +- *(2) Filter isnotnull(i#8)
         +- *(2) ColumnarToRow
            +- FileScan parquet default.t[i#8] Batched: true, DataFilters: [isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/..., PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int>, SelectedBucketsCount: 8 out of 8
{code}
Bug: Expired SSL cert on Spark 2.4.5 download
Status: Closed | Priority: Major | Resolution: Fixed
Created: 09/Mar/20 18:43
Affects Version/s: 2.4.5

{code:java}
+ curl -L 'https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz' -o spark-2.4.5-bin-hadoop2.7.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
{code}
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/3091/HADOOP_VERSION=2.7.5,SCALAVER=2.11,SPARK_VERSION=2.4.5,label=ubuntu/console
Bug: New Parquet Predicate Filter APIs with multi-part Identifier Support
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Mar/20 04:08
Affects Version/s: 2.4.5

Parquet's *org.apache.parquet.filter2.predicate.FilterApi* uses *dots* as separators to split the column name into multi-parts of nested fields. The drawback is this causes issues when the field name contains *dot*.

The new APIs that will be added will take array of string directly for multi-parts of nested fields, so no confusion as using *dot* as a separator.

It's intended to move this code back to parquet community. See [PARQUET-1809]
Bug: Consolidate the implementation of quoteIfNeeded
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Mar/20 19:13
Affects Version/s: 2.4.5

There are two implementation of quoteIfNeeded, and one is in *org.apache.spark.sql.connector.catalog.CatalogV2Implicits.quote* and the other is in *OrcFiltersBase.quoteAttributeNameIfNeeded* This PR will consolidate them into one.
Bug: ShuffleBlockFetcherIterator may can't create request for last group
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 04/Mar/20 07:17
Affects Version/s: 3.0.0

When the size of all blocks is less than targetRemoteRequestSize and the size of last block group is less than maxBlocksInFlightPerAddress, ShuffleBlockFetcherIterator will not create a request for the last group.
Bug: Support time zone ids in casting strings to timestamps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 07:03
Affects Version/s: 3.0.0

Currently, Spark supports only time zone offsets in the formats:
* -[h]h:[m]m
* +[h]h:[m]m
* Z

The ticket aims to support any valid time zone ids at the end of timestamp strings, for instance:
{code}
2015-03-18T12:03:17.123456 Europe/Moscow
{code}
Bug: Thriftserver operations other than SparkExecuteStatementOperation do not call onOperationClosed
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 05/Mar/20 12:25
Affects Version/s: 3.0.0

In Spark 3.0 onOperationClosed was implemented in HIveThriftServer2Listener to track closing the operation in the thriftserver (after client finishes fetching).
However, it seems that only SparkExecuteStatementOperation calls it in it's close() function. Other operations need to do this as well.
Bug: Allow specifying session catalog name (spark_catalog) in qualified column names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/20 18:48
Affects Version/s: 3.0.0

Currently, the user cannot specify the session catalog name when using qualified column names for v1 tables:
{code:java}
SELECT spark_catalog.default.t.i FROM spark_catalog.default.t
{code}
fails with "cannot resolve '`spark_catalog.default.t.i`".

This is inconsistent with v2 tables where catalog name can be used:
{code:java}
SELECT testcat.ns1.tbl.id FROM testcat.ns1.tbl.id
{code}
Bug: Test for shuffle requests packaging with different size and numBlocks limit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/20 07:33
Affects Version/s: 3.0.0

When packaging shuffle fetch requests in ShuffleBlockFetcherIterator, there are two limitations: maxBytesInFlight and maxBlocksInFlightPerAddress. However, we don’t have test cases to test them both, e.g. the size limitation is hit before the numBlocks limitation.

We should add test cases in ShuffleBlockFetcherIteratorSuite to test:
 # the size limitation is hit before the numBlocks limitation

 # the numBlocks limitation is hit before the size limitation
Bug: An analysis failure in generators with aggregate functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/20 04:29
Affects Version/s: 3.0.0

We have supported generators in SQL aggregate expressions by SPARK-28782.
But, the generator(explode) query with aggregate functions in DataFrame failed as follows;


{code}
// SPARK-28782: Generator support in aggregate expressions
scala> spark.range(3).toDF("id").createOrReplaceTempView("t")
scala> sql("select explode(array(min(id), max(id))) from t").show()
+---+
|col|
+---+
|  0|
|  2|
+---+

// A failure case handled in this pr
scala> spark.range(3).select(explode(array(min($"id"), max($"id")))).show()
org.apache.spark.sql.AnalysisException:
The query operator `Generate` contains one or more unsupported
expression types Aggregate, Window or Generate.
Invalid expressions: [min(`id`), max(`id`)];;
Project [col#46L]
+- Generate explode(array(min(id#42L), max(id#42L))), false, [col#46L]
   +- Range (0, 3, step=1, splits=Some(4))

  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.failAnalysis(CheckAnalysis.scala:49)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.failAnalysis$(CheckAnalysis.scala:48)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:129)
{code}

The root cause is that `ExtractGenerator` wrongly replaces a project w/ aggregate functions
before `GlobalAggregates` replaces it with an aggregate as follows;

{code}
scala> sql("SET spark.sql.optimizer.planChangeLog.level=warn")
scala> spark.range(3).select(explode(array(min($"id"), max($"id")))).show()

20/03/01 12:51:58 WARN HiveSessionStateBuilder$$anon$1: 
=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences ===
!'Project [explode(array(min('id), max('id))) AS List()]   'Project [explode(array(min(id#72L), max(id#72L))) AS List()]
 +- Range (0, 3, step=1, splits=Some(4))                   +- Range (0, 3, step=1, splits=Some(4))
           
20/03/01 12:51:58 WARN HiveSessionStateBuilder$$anon$1: 
=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ExtractGenerator ===
!'Project [explode(array(min(id#72L), max(id#72L))) AS List()]   Project [col#76L]
!+- Range (0, 3, step=1, splits=Some(4))                         +- Generate explode(array(min(id#72L), max(id#72L))), false, [col#76L]
!                                                                   +- Range (0, 3, step=1, splits=Some(4))
           
20/03/01 12:51:58 WARN HiveSessionStateBuilder$$anon$1: 
=== Result of Batch Resolution ===
!'Project [explode(array(min('id), max('id))) AS List()]   Project [col#76L]
!+- Range (0, 3, step=1, splits=Some(4))                   +- Generate explode(array(min(id#72L), max(id#72L))), false, [col#76L]
!                                                             +- Range (0, 3, step=1, splits=Some(4))
          
// the analysis failed here...
{code}
Bug: ClassCastException when a generator having nested inner generators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Mar/20 05:02
Affects Version/s: 2.0.2

A query below failed in the master/branch-3.0/branch-2.4;

{code}
scala> sql("select array(array(1, 2), array(3)) ar").select(explode(explode($"ar"))).show()
20/03/01 13:51:56 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
java.lang.ClassCastException: scala.collection.mutable.ArrayOps$ofRef cannot be cast to org.apache.spark.sql.catalyst.util.ArrayData
	at org.apache.spark.sql.catalyst.expressions.ExplodeBase.eval(generators.scala:313)
	at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$8(GenerateExec.scala:108)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$ConcatIterator.hasNext(Iterator.scala:222)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    ...
{code}
Bug: Star(*) expression fails when used with fully qualified column names for v2 tables
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Mar/20 03:51
Affects Version/s: 3.0.0

For a v2 table,
{code:java}
CREATE TABLE testcat.ns1.ns2.tbl (id bigint, name string) USING foo
{code}

The following works
{code:java}
SELECT testcat.ns1.ns2.tbl.id FROM testcat.ns1.ns2.tbl
{code}
but this fails to resolve
{code:java}
SELECT testcat.ns1.ns2.tbl.* FROM testcat.ns1.ns2.tbl
[info]   org.apache.spark.sql.AnalysisException: cannot resolve 'testcat.ns1.ns2.tbl.*' given input columns 'id, name';
{code}
Bug: Fix incorrect use of assume() in tests
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Mar/20 02:46
Affects Version/s: 3.0.0

[~ahirreddy] and I found some cases where {{assume}} and {{assert}} were mixed up in test suites.

If an {{assume}} call fails then it will cause the test to be marked as skipped instead of failed: this is often used to skip tests if certain prerequisites are missing (e.g. to disable certain tests on Windows). If {{assume}} is mistakenly used in place of {{assert}} then this can cause logically-failing tests to be skipped, masking bugs.

This patch fixes several such cases, replacing certain {{assume}} calls with {{assert}}.
Bug: Support .r files in 2.x version
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 26/Mar/20 17:45
Affects Version/s: 2.3.4

Hello,

We are using Spark 2.3.4 currently that does not allow .r files in Spark-Submit. However, the latest versions of Spark do. It is a bit difficult for us at the moment to upgrade the Spark version we are using. 

Can you point me to the Jira that added support for .r? I am not able to find it in the issues.
Bug: Add ability to set table description in the catalog
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 01/Mar/20 18:31
Affects Version/s: 3.1.0

It seems that the catalog supports a {{description}} attribute on tables.

https://github.com/apache/spark/blob/86cc907448f0102ad0c185e87fcc897d0a32707f/sql/core/src/main/scala/org/apache/spark/sql/catalog/interface.scala#L68

However, the {{createTable()}} interface doesn't provide any way to set that attribute.
Bug: Use stringArgs in Expression.toString to respect hidden parameters
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Mar/20 09:19
Affects Version/s: 3.0.0

Currently, the top of https://github.com/apache/spark/pull/27657, 

{code}
val identify = udf((input: Seq[Int]) => input)
spark.range(10).select(identify(array("id"))).show()
{code}

shows hidden parameter `useStringTypeWhenEmpty`.

{code}
+---------------------+
|UDF(array(id, false))|
+---------------------+
|                  [0]|
|                  [1]|
...
{code}

This is a general problem and we should respect hidden parameters.