av9ash/GitBugs huggingface dataset:
Bug: Fix flakiness of CliSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jul/20 12:44
Affects Version/s: 3.1.0

A lot of build failures happened in recent builds, which showed that CliSuite is very flaky.

While the flakiness depends on the power of the machine, it'd be better if we can make the suite less flaky even on slow machine.
Bug: Support 'F'-suffixed Float Literals
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 08:34
Affects Version/s: 2.3.4

{code:sql}
select 1.1f;
{code}

currently thows

{code:java}

```
org.apache.spark.sql.AnalysisException: Can't extract value from 1: need struct type but got int;
```
{code}
Bug: Enable multi-line true could break the read csv in Azure Data Lake Storage gen2
Status: Open | Priority: Major | Resolution: 
Created: 07/Jul/20 07:51
Affects Version/s: 2.3.2

I'm using azure data lake gen2, when I'm loading data frame with certain options: 

var df = spark.read.format("csv")
 .option("ignoreLeadingWhiteSpace", "true")
 .option("ignoreTrailingWhiteSpace", "true")
 .option("parserLib", "UNIVOCITY")
 .option("multiline", "true")
 .option("inferSchema", "true")
 .option("mode", "PERMISSIVE")
 .option("quote", "\"")
 .option("escape", "\"")
 .option("timeStampFormat", "M/d/yyyy H:m:s a")
 .load("abfss://\{containername}@\{storage}.dfs.core.windows.net/\{somedirectory}/\{somefilesnapshotname}yyyy-MM-dd'T'hh:mm:ss")
 .limit(1)

It will load data correctly.

 

But if I use \{DirectoryWithColon}, it will thrown error:

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: \{somefilesnapshotname}yyyy-MM-dd'T'hh:mm:ss

 

Then if I remove .option("multiline", "true"), data can be loaded, but for sure that the dataframe is not handled correctly because there are newline character.

 

So I believe it is a bug.

 

And since our production is running correctly if we enable spark.read.schema(\{SomeSchemaList}).format("csv"), and we want to use inferschema feature on those file path with colon or other special characters, could you help fix this issue?
Bug: New udaf(Aggregator) has an integration bug with UnresolvedMapObjects serialization
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/20 18:05
Affects Version/s: 3.0.0

The new user defined aggregator feature (SPARK-27296) based on calling 'functions.udaf(aggregator)' works fine when the aggregator input type is atomic, e.g. 'Aggregator[Double, _, _]', however if the input type is an array, like 'Aggregator[Array[Double], _, _]',  it is tripping over the following:

/**
 * When constructing [[MapObjects]], the element type must be given, which may not be available
 * before analysis. This class acts like a placeholder for [[MapObjects]], and will be replaced by
 * [[MapObjects]] during analysis after the input data is resolved.
 * Note that, ideally we should not serialize and send unresolved expressions to executors, but
 * users may accidentally do this(e.g. mistakenly reference an encoder instance when implementing
 * Aggregator). Here we mark `function` as transient because it may reference scala Type, which is
 * not serializable. Then even users mistakenly reference unresolved expression and serialize it,
 * it's just a performance issue(more network traffic), and will not fail.
 */
 case class UnresolvedMapObjects(
 {color:#de350b}@transient function: Expression => Expression{color},
 child: Expression,
 customCollectionCls: Option[Class[_]] = None) extends UnaryExpression with Unevaluable {
 override lazy val resolved = false

override def dataType: DataType = customCollectionCls.map(ObjectType.apply).getOrElse

{ throw new UnsupportedOperationException("not resolved") }

}

 

*The '@transient' is causing the function to be unpacked as 'null' over on the executors, and it is causing a null-pointer exception here, when it tries to do 'function(loopVar)'*

object MapObjects {
 def apply(
 function: Expression => Expression,
 inputData: Expression,
 elementType: DataType,
 elementNullable: Boolean = true,
 customCollectionCls: Option[Class[_]] = None): MapObjects =

{ val loopVar = LambdaVariable("MapObject", elementType, elementNullable) MapObjects(loopVar, {color:#de350b}function(loopVar){color}, inputData, customCollectionCls) }

}

*I believe it may be possible to just use 'loopVar' instead of 'function(loopVar)', whenever 'function' is null, but need second opinion from catalyst developers on what a robust fix should be*
Bug: Nested pruning should still work for nested column extractors of attributes with cosmetic variations
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jul/20 02:14
Affects Version/s: 3.0.0

If the expressions extracting nested fields have cosmetic variations like qualifier difference, currently nested column pruning cannot work well.

For example, two attributes which are semantically the same, are referred in a query, but the nested column extractors of them are treated differently when we deal with nested column pruning.
Bug: RDD.takeOrdered can choose to merge intermediate results in executor or driver
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jul/20 18:17
Affects Version/s: 3.0.0

In the list of issues, I saw some discussions about exceeding the memory limit of the driver when using _RDD.takeOrdered_ or _SQL(order by xx limit xx)_. I think that the implementation of _RDD.takeOrdered_ can be improved.

In the original code implementation of _RDD.takeOrdered_, the QuickSelect algorithm in guava is used in the executor process to calculate the local TopK results of each RDD partition. These intermediate results are packaged into java.util.PriorityQueue and returned to the driver process. In the driver process, these intermediate results are merged to get the global TopK results.

The problem with this implementation is that if the intermediate results are too large and too many partitions, the intermediate results may accumulate in the memory of the driver process, causing excessive memory pressure.

We can use an optional config to determine whether the intermediate results(local TopK) of each partition in _RDD.takeOrdered_ will be merged in driver process or executor process. If set to true, merge in driver process(by util.PriorityQueue), which will get shorter waiting time for return. But if the intermediate results are too large and too many partitions, the intermediate results may accumulate in the memory of the driver process, causing excessive memory pressure. If set to false, merge in executor process(by guava.QuickSelect), intermediate results will not accumulate in memory, but will cause longer runtimes.

something like:
_(org.apache.spark.rdd.RDD)_
{code:scala}
  def takeOrdered(num: Int)(implicit ord: Ordering[T]): Array[T] = withScope {
    if (num == 0 || partitions.length == 0) {
      Array.empty
    } else {
      if (conf.getBoolean("spark.rdd.takeOrdered.mergeInDriver", true)) {
        val mapRDDs = mapPartitions { items =>
          // Priority keeps the largest elements, so let's reverse the ordering.
          val queue = new BoundedPriorityQueue[T](num)(ord.reverse)
          queue ++= collectionUtils.takeOrdered(items, num)(ord)
          Iterator.single(queue)
        }
        mapRDDs.reduce { (queue1, queue2) =>
          queue1 ++= queue2
          queue1
        }.toArray.sorted(ord)
      } else {
        mapPartitions { items =>
          collectionUtils.takeOrdered(items, num)(ord)
        }.repartition(1).mapPartitions { items =>
          collectionUtils.takeOrdered(items, num)(ord)
        }.collect()
      }
    }
  }
{code}
Bug: Pin mariadb-plugin-gssapi-server version to fix MariaDBKrbIntegrationSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Jul/20 13:46
Affects Version/s: 3.1.0

The test fails with the following error:
{code:java}
2020-07-07 12:43:13 8 [Warning] Aborted connection 8 to db: 'unconnected' user: 'unauthenticated' host: '172.17.0.1' (This connection closed normally without authentication)
{code}
This is because the docker image contains MariaDB version 1:10.4.12+maria~bionic but 1:10.4.13+maria~bionic came out and mariadb-plugin-gssapi-server installation triggered unwanted database upgrade inside the docker image.
Bug: More general skew join pattern matching
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jul/20 03:44
Affects Version/s: 3.0.0

Current the AQE skew join handling logic is very specified.
It can only handle the pattern like this:
{code}
  SMJ
     Sort
       Shuffle
     Sort
       Shuffle
{code}

We propose a more general skew Join pattern matching approach.
In this patch, we can handle 3-tablesjoin, join with aggregation, and so on.
Bug: tree models auto infer compact integer type
Status: In Progress | Priority: Major | Resolution: 
Created: 07/Jul/20 04:00
Affects Version/s: 3.1.0

a feature from [LightGBM|https://lightgbm.readthedocs.io/en/latest/Parameters.html#max_bin]:

 

"LightGBM will auto compress memory according to {{max_bin}}. For example, LightGBM will use {{uint8_t}} for feature value if {{max_bin=255}}"
Bug: Allow filter pushdown after a groupBy with collect_list
Status: Resolved | Priority: Major | Resolution: Not A Bug
Created: 04/Jul/20 06:21
Affects Version/s: 2.4.6

Consider the following statement:


{code:python}
import pyspark
import pyspark.sql.functions as f


with pyspark.SparkContext(conf=pyspark.SparkConf().setMaster('local[*]')) as sc:
    spark = pyspark.sql.SQLContext(sc)

    df = spark.createDataFrame([
        [2020, 1, 1, 1.0],
        [2020, 1, 2, 2.0],
        [2020, 1, 3, 3.0],
    ], schema=['year', 'id', 't', 'value'])

    df = df.groupBy(['year', 'id']).agg(f.collect_list('value'))
    df = df.where(f.col('year') == 2020)
    df.explain()
{code}
 

which yields the plan

{code:bash}
== Physical Plan ==
*(2) Filter (isnotnull(year#0L) AND (year#0L = 2020))
+- ObjectHashAggregate(keys=[year#0L, id#1L], functions=[collect_list(value#3, 0, 0)])
   +- Exchange hashpartitioning(year#0L, id#1L, 200), true, [id=#23]
      +- ObjectHashAggregate(keys=[year#0L, id#1L], functions=[partial_collect_list(value#3, 0, 0)])
         +- *(1) Project [year#0L, id#1L, value#3]
            +- *(1) Scan ExistingRDD[year#0L,id#1L,t#2L,value#3]
{code}

This issue is about allowing the `Filter` to be placed before `hashpartitioning` (in the plan), since there seems to be no reason to not allow so.
Bug: Refactor two similar test cases from SPARK-31061 in HiveExternalCatalogSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jul/20 09:58
Affects Version/s: 3.0.0

In `org.apache.spark.sql.hive.HiveExternalCatalogSuite`

`

test("SPARK-31061: alterTable should be able to change table provider") {
 val catalog = newBasicCatalog()
 val parquetTable = CatalogTable(
 identifier = TableIdentifier("parq_tbl", Some("db1")),
 tableType = CatalogTableType.MANAGED,
 storage = storageFormat.copy(locationUri = Some(new URI("file:/some/path"))),
 schema = new StructType().add("col1", "int").add("col2", "string"),
 provider = Some("parquet"))
 catalog.createTable(parquetTable, ignoreIfExists = false)

 val rawTable = externalCatalog.getTable("db1", "parq_tbl")
 assert(rawTable.provider === Some("parquet"))

 val fooTable = *parquetTable*.copy(provider = Some("foo")) <- `*parquetTable*` seems should be rawTable
 catalog.alterTable(fooTable)
 val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
 assert(alteredTable.provider === Some("foo"))
}

test("SPARK-31061: alterTable should be able to change table provider from hive") {
 val catalog = newBasicCatalog()
 val hiveTable = CatalogTable(
 identifier = TableIdentifier("parq_tbl", Some("db1")),
 tableType = CatalogTableType.MANAGED,
 storage = storageFormat,
 schema = new StructType().add("col1", "int").add("col2", "string"),
 provider = Some("hive"))
 catalog.createTable(hiveTable, ignoreIfExists = false)

 val rawTable = externalCatalog.getTable("db1", "parq_tbl")
 assert(rawTable.provider === Some("hive"))

 val fooTable = rawTable.copy(provider = Some("foo"))
 catalog.alterTable(fooTable)
 val alteredTable = externalCatalog.getTable("db1", "parq_tbl")
 assert(alteredTable.provider === Some("foo"))
}

`
Bug: Spark: PartitionBy changing the columns value 
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 01/Jul/20 08:26
Affects Version/s: 3.0.0

While saving dataframe as parquet or csv with partitionBy column having 'f' and 'd' with numbers are changing the values.

Below is the example 
{code:java}
scala> val df = Seq(
 | ("9q", 1),
 | ("3k", 2),
 | ("6f", 3),
 | ("7f", 4),
 | ("7d", 5)
 | ).toDF("value", "id")
df: org.apache.spark.sql.DataFrame = [value: string, id: int]
scala> df.show(false)
+-----+---+
|value|id |
+-----+---+
|  9q | 1 |
|  3k | 2 |
|  6f | 3 |
|  7f | 4 |
|  7d | 5 |
+-----+---+

scala> df.write.partitionBy("value").mode(SaveMode.Overwrite).parquet("tmp_parquet")
scala> spark.read.parquet("tmp_parquet").show(false)
+---+-----+
|id |value|
+---+-----+
|5  | 7.0 |
|3  | 6.0 |
|2  | 3k  |
|4  | 7.0 |
|1  | 9q  |
+---+-----+

{code}
Same with the other format too, Is this a bug or is it normal.

Taken from [SO|[https://stackoverflow.com/questions/62671684/spark-incorrectly-intepret-partition-name-ending-with-d-or-f-as-number-when]]
Bug: Fast fail when the AQE skew join produce too many splits
Status: In Progress | Priority: Major | Resolution: 
Created: 01/Jul/20 03:51
Affects Version/s: 3.0.0

In handling skewed SortMergeJoin, when matching partitions from the left side and the right side both have skew and split too many partitions, the plan generation may take a very long time.

Even fallback to normal SMJ, the query cannot success either. So we should fast fail this query.

In below logs we can see that it took over 1 hour to generate the plan in AQE when handle a skewed join which produced too many splits.
{quote}20/06/30 *12:31:26,271* INFO [HiveServer2-Background-Pool: Thread-821384] adaptive.OptimizeSkewedJoin:54 :
 20/06/30 12:31:26,299 INFO [HiveServer2-Background-Pool: Thread-821384] adaptive.OptimizeSkewedJoin:54 : Left side partition 1 (3 TB) is skewed, split it into *39150* parts.
 20/06/30 12:31:26,315 INFO [HiveServer2-Background-Pool: Thread-821384] adaptive.OptimizeSkewedJoin:54 : Right side partition 1 (11 TB) is skewed, split it into *17022* parts.
 20/06/30 12:32:24,952 INFO [HiveServer2-Background-Pool: Thread-821384] adaptive.OptimizeSkewedJoin:54 : Right side partition 8 (1 GB) is skewed, split it into 17 parts.

...
 20/06/30 *13:27:25,158* INFO [HiveServer2-Background-Pool: Thread-821384] adaptive.AdaptiveSparkPlanExec:54 : Final plan: CollectLimit 1000
{quote}
Bug: Expose confusion matrix elements by threshold in BinaryClassificationMetrics
Status: Open | Priority: Minor | Resolution: 
Created: 28/Jul/20 21:28
Affects Version/s: 3.0.0

Currently, the only thresholded metrics available from BinaryClassificationMetrics are precision, recall, f-measure, and (indirectly through roc()) the false positive rate.

Unfortunately, you can't always compute the individual thresholded confusion matrix elements (TP, FP, TN, FN) from these quantities. You can make a system of equations out of the existing thresholded metrics and the total count, but they become underdetermined when there are no true positives.

Fortunately, the individual confusion matrix elements by threshold are already computed and sitting in the confusions variable. It would be helpful to expose these elements directly. The easiest way would probably be by adding methods like 
{code:java}
def truePositivesByThreshold(): RDD[(Double, Double)] = confusions.map{ case (t, c) => (t, c.weightedTruePositives) }{code}
An alternative could be to expose the entire RDD[(Double, BinaryConfusionMatrix)] in one method, but BinaryConfusionMatrix is also currently package private.

The closest issue to this I found was this one for adding new calculations to BinaryClassificationMetrics https://issues.apache.org/jira/browse/SPARK-18844, which was closed without any changes being merged.
Bug: The type conversion function generated in makeFromJava for "other"  type uses a wrong variable.
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Jul/20 21:59
Affects Version/s: 2.4.6

`makeFromJava` in `EvaluatePython` create a type conversion function for some Java/Scala types.

For `other` type, the parameter of the type conversion function is named `obj` but `other` is mistakenly used rather than `obj` in the function body.
{code:java}
case other => (obj: Any) => nullSafeConvert(other)(PartialFunction.empty) {code}
Bug: Disable test-dependencies.sh from Jenkins jobs
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 06/Jul/20 02:18
Affects Version/s: 3.1.0

This issue aims to disable dependency tests(test-dependencies.sh) from Jenkins. GitHub Action provides the same test capability already and stabler.

Currently, `test-dependencies.sh` fails very frequently in AmpLab Jenkins environment. For example, in the following irrelevant PR, it fails 5 times during 6 hours.
https://github.com/apache/spark/pull/29001
Bug: Improve Pandas Grouped Map with Window test output
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jul/20 01:58
Affects Version/s: 3.0.0

The output of GroupedMapInPandasTests.test_grouped_over_window_with_key is not helpful, only gives 

{code}

======================================================================
FAIL: test_grouped_over_window_with_key (pyspark.sql.tests.test_pandas_grouped_map.GroupedMapInPandasTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/spark/python/pyspark/sql/tests/test_pandas_grouped_map.py", line 588, in test_grouped_over_window_with_key
    self.assertTrue(all([r[0] for r in result]))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 21 tests in 141.194s

FAILED (failures=1)
{code}
Bug: class 'DataFrame' returns instance of type(self) instead of DataFrame 
Status: Open | Priority: Minor | Resolution: 
Created: 24/Jul/20 06:37
Affects Version/s: 2.4.6

To allow for appropriate child classing of DataFrame, I propose the following change:

class 'DataFrame' returns instance of type(self) instead of  typeDataFrame 

 

Therefore child classes using methods such as '.limit()' will return an instance of the child class.
Bug: Remove project if output is subset of child
Status: Open | Priority: Minor | Resolution: 
Created: 20/Jul/20 01:08
Affects Version/s: 3.1.0

We can remove some redundant project after we completed pruning column.

e.g.,
{code:java}
create table t1(c1 int, c2 int) using parquet;

explain extended
select sum(c1) from (
  select * from t1
);
{code}
Currently we get this plan.
{code:java}
== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[sum(cast(c1#19 as bigint))], output=[sum(c1)#68L])
+- Exchange SinglePartition, true, [id=#86]
   +- *(1) HashAggregate(keys=[], functions=[partial_sum(cast(c1#19 as bigint))], output=[sum#70L])
      +- *(1) Project [c1#19]
         +- *(1) ColumnarToRow
            +- FileScan parquet default.t1[c1#19] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[hdfs:///user/hive/warehouse/t1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int>
{code}
We can remove the `Project`, like this
{code:java}
== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[sum(cast(c1#19 as bigint))], output=[sum(c1)#68L])
+- Exchange SinglePartition, true, [id=#86]
   +- *(1) HashAggregate(keys=[], functions=[partial_sum(cast(c1#19 as bigint))], output=[sum#70L])
       +- *(1) ColumnarToRow
          +- FileScan parquet default.t1[c1#19] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[hdfs:///user/hive/warehouse/t1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c1:int>
{code}
Bug: Unevaluable expr is set to FIRST/LAST ignoreNullsExpr in distinct aggregates
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 17/Jul/20 07:04
Affects Version/s: 2.2.3

{code}
scala> sql("SELECT FIRST(DISTINCT v) FROM VALUES 1, 2, 3 t(v)").show()
...
Caused by: java.lang.UnsupportedOperationException: Cannot evaluate expression: false#37
  at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:258)
  at org.apache.spark.sql.catalyst.expressions.AttributeReference.eval(namedExpressions.scala:226)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.ignoreNulls(First.scala:68)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions$lzycompute(First.scala:82)
  at org.apache.spark.sql.catalyst.expressions.aggregate.First.updateExpressions(First.scala:81)
  at org.apache.spark.sql.execution.aggregate.HashAggregateExec$$anonfun$15.apply(HashAggregateExec.scala:268)
{code}
Bug: Add a test case for toPandas to work with empty partitioned Spark DataFrame
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 14/Jul/20 08:46
Affects Version/s: 3.0.0

{code}
spark.sparkContext.emptyRDD().toDF("col1 int").toPandas()
{code}

empty partitioned DataFrame is not being tested with {{toPandas}}.
Bug: Parquet footer information is read twice
Status: Open | Priority: Minor | Resolution: 
Created: 08/Jul/20 08:55
Affects Version/s: 3.0.0

When running queries, spark reads parquet footer information twice. In cloud env, this would turn out to be expensive (depending on the jobs, # of splits). It would be nice to reuse the footer information already read via "ParquetInputFormat::buildReaderWithPartitionValues"

 

!spark_parquet_footer_reads.png|width=640,height=644!

Lines of interest:

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L271]

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L326]

 

[https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L105]

[https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L111]
Bug: Remove task result size check for shuffle map stage
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jul/20 18:04
Affects Version/s: 2.4.6

The task result of a shuffle map stage is not the query result but instead is only map status and metrics accumulator updates. Aside from the metrics that can vary in size, the total task result size solely depends on the number of tasks. And the number of tasks can get large regardless of the stage's output size. For example, the number of tasks generated by `CartesianProduct` is square of "spark.sql.shuffle.partitions", say if "spark.sql.shuffle.partitions" is set to 200, you get 40,000 tasks, if set to 500, you get 250,000 tasks, which can easily error on the default limit of `spark.driver.maxResultSize`:

 
{code:java}
org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 66496 tasks (4.0 GiB) is bigger than spark.driver.maxResultSize (4.0 GiB)
{code}
 

However, map status and accumulator updates are used by the driver to update the overall map stats and metrics of the query, and they are not cached on the driver, so they won't cause catastrophic memory issues on the driver. So we should remove this check for shuffle map stage tasks.