av9ash/GitBugs huggingface dataset:
Bug: EXPLAIN EXTENDED does not show detail information for aggregate operators
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Jan/20 15:47
Affects Version/s: 3.1.0

Currently EXPLAIN FORMATTED only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate. While EXPLAIN EXTENDED provides more information. We need to enhance EXPLAIN FORMATTED to follow the original behavior.
Bug: The native ALTER COLUMN syntax should change one thing at a time
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 04:21
Affects Version/s: 3.0.0

Our native ALTER COLUMN syntax is newly added in 3.0 and almost follows the SQL standard.

{code}
ALTER TABLE table=multipartIdentifier
  (ALTER | CHANGE) COLUMN? column=multipartIdentifier
  (TYPE dataType)?
  (COMMENT comment=STRING)?
  colPosition?   
{code}

The SQL standard (section 11.12) only allows changing one property at a time. This is also true on other recent SQL systems like snowflake(https://docs.snowflake.net/manuals/sql-reference/sql/alter-table-column.html) and redshift(https://docs.aws.amazon.com/redshift/latest/dg/r_ALTER_TABLE.html)

The snowflake has an extension that it allows changing multiple columns at a time, like ALTER COLUMN c1 TYPE int, c2 TYPE int. If we want to extend the SQL standard, I think this syntax is better. 

For now, let's be conservative and only allow changing one property at a time.
Bug: Document general file source options
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jan/20 01:01
Affects Version/s: 3.0.0

Write a new document for general file source options:
1. spark.sql.files.ignoreCorruptFiles
2. spark.sql.files.ignoreMissingFiles
3. pathGlobFilter
4. recursiveFileLookup
Bug: Mitigate SQL injections - can't parameterize query parameters for JDBC connectors
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/20 06:07
Affects Version/s: 3.1.0

One of the options to read from a JDBC connection is a query.

Sometimes, this query is parameterized (e.g. column name, values, etc).

The JDBC API does not support parameterizing SQL queries, which puts the burden of escaping SQL on the developer. This burden is unnecessary and a security risk.

Very often, drivers provide a specific API to securely parameterize SQL statements.

This issue proposes allowing the developers to pass "query" and "parameters" to the JDBC options, so that it is the driver, not the developer, that escape parameters.
Bug: HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/20 08:15
Affects Version/s: 3.1.0

In HiveTableScanExec, it will push down to hive metastore for partition pruning if _spark.sql.hive.metastorePartitionPruning_ is true, and then it will prune the returned partitions again using partition filters, because some predicates, eg. "b like 'xyz'", are not supported in hive metastore. But now this problem is already fixed in HiveExternalCatalog.listPartitionsByFilter, the HiveExternalCatalog.listPartitionsByFilter can return exactly what we want now. So it is not necessary any more to double prune in HiveTableScanExec.
Bug: Introduce AdmissionControl API to Structured Streaming
Status: Resolved | Priority: Major | Resolution: Done
Created: 29/Jan/20 02:20
Affects Version/s: 3.0.0

In Structured Streaming, we have the concept of Triggers. With a trigger like Trigger.Once(), the semantics are to process all the data available to the datasource in a single micro-batch. However, this semantic can be broken when data source options such as `maxOffsetsPerTrigger` (in the Kafka source) rate limit the amount of data read for that micro-batch.

We propose to add a new interface `SupportsAdmissionControl` and `ReadLimit`. A ReadLimit defines how much data should be read in the next micro-batch. `SupportsAdmissionControl` specifies that a source can rate limit its ingest into the system. The source can tell the system what the user specified as a read limit, and the system can enforce this limit within each micro-batch or impose it's own limit if the Trigger is Trigger.Once() for example.
Bug: Support the "minPartitions" option in Kafka batch source and streaming source v1
Status: Resolved | Priority: Major | Resolution: Done
Created: 27/Jan/20 23:19
Affects Version/s: 3.0.0

Right now, the "minPartitions" option only works in Kafka streaming source v2. It would be great that we can support it in batch and streaming source v1 (v1 is the fallback mode when a user hits a regression in v2) as well.
Bug: Improve error messages when Executor dies before registering with driver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/20 15:10
Affects Version/s: 3.0.0

currently when you give a bad configuration for accelerator aware scheduling to the executor, the Executors can die but its hard for the user to know why.  The executor dies and logs in its log files what went wrong but many times it hard to find those logs because the executor hasn't registered yet.  Since it hasn't registered the executor doesn't show up on UI to see log files.

One specific example is you give a discovery script that that doesn't find all the GPUs:

{code}
20/01/16 08:59:24 INFO YarnCoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.28.9.112:44403
20/01/16 08:59:24 ERROR Inbox: Ignoring error
java.lang.IllegalArgumentException: requirement failed: Resource: gpu, with addresses: 0 is less than what the user requested: 2)
 at scala.Predef$.require(Predef.scala:281)
 at org.apache.spark.resource.ResourceUtils$.$anonfun$assertAllResourceAllocationsMatchResourceProfile$1(ResourceUtils.scala:251)
 at org.apache.spark.resource.ResourceUtils$.$anonfun$assertAllResourceAllocationsMatchResourceProfile$1$adapted(ResourceUtils.scala:248)
{code}

 

Figure out a better way of logging or letting user know  what error occurred when the executor dies before registering
Bug: Enable dependency audit files to tell dependency classifier
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Jan/20 01:06
Affects Version/s: 2.4.4

Current dependency audit files under `dev/deps` only show jar names. And there isn't a simple rule on how to parse the jar name to get the values of different fields. For example, `hadoop2` is the classifier of `avro-mapred-1.8.2-hadoop2.jar`, in contrast, `incubating` is the version of `htrace-core-3.1.0-incubating.jar`.

Thus, I propose to enable dependency audit files to tell the value of artifact id, version, and classifier of a dependency. For example, `avro-mapred-1.8.2-hadoop2.jar` should be expanded to `avro-mapred/1.8.2/hadoop2/avro-mapred-1.8.2-hadoop2.jar` where `avro-mapred` is the artifact id, `1.8.2` is the version, and `haddop2` is the classifier.

In this way, dependency audit files are able to be consumed by automated tests or downstream tools. There is a good example of the downstream tool that would be enabled:
Say we have a Spark application that depends on a third-party dependency `foo`, which pulls in `jackson` as a transient dependency. Unfortunately, `foo` depends on a different version of `jackson` than Spark. So, in the pom of this Spark application, we use the dependency management section to pin the version of `jackson`. By doing this, we are lifting `jackson` to the top-level dependency of my application and I want to have a way to keep tracking what Spark uses. What we can do is to cross-check my Spark application's classpath with what Spark uses. Then, with a test written in my code base, whenever my application bumps Spark version, this test will check what we define in the application and what Spark has, and then remind us to change our application's pom if needed. In my case, I am fine to directly access git to get these audit files.
Bug: Prevent unnessary copies of data in Arrow to Pandas conversion with Timestamps
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 23:04
Affects Version/s: 2.4.4

During conversion of Arrow to Pandas, timestamp columns are modified to localize for the current timezone. If there are no timestamp columns, this can sometimes result in unnecessary copies of the data. See [https://www.mail-archive.com/dev@arrow.apache.org/msg17008.html] for discussion.
Bug: Codegen fails when xxHash seed is not an integer
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 10:47
Affects Version/s: 2.4.4

If the seed for xxHash is not an integer the generated code does not compile.

Steps to reproduce:
{code:java}
import org.apache.spark.sql.catalyst.expressions.XxHash64
import org.apache.spark.sql.Column

val file = "..."
val column = col("...")

val df = spark.read.csv(file)

def xxHash(seed: Long, cols: Column*): Column = new Column(
   XxHash64(cols.map(_.expr), seed)
)

val seed = (Math.pow(2, 32)+1).toLong
df.select(xxHash(seed, column)).show()
{code}
Appending an L to the seed when the datatype is long fixes the issue.
Bug: EXPLAIN EXTENDED does not show detail information for aggregate operators
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 27/Jan/20 15:53
Affects Version/s: 3.0.0

Currently EXPLAIN FORMATTED only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate. While EXPLAIN EXTENDED provides more information. We need to enhance EXPLAIN FORMATTED to follow the original behavior.
Bug: cleanClosure on recursive call leads to node stack overflow
Status: Open | Priority: Major | Resolution: 
Created: 24/Jan/20 03:02
Affects Version/s: 3.0.0

This problem surfaced while handling SPARK-22817. In theory there are tests, which cover that problem, but it seems like they have been dead for some reason.

Reproducible example

{code:r}
f <- function(x) {
  f(x)
}

SparkR:::cleanClosure(f)
{code}
Bug: Upgrade Jersey to 2.30
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Jan/20 22:08
Affects Version/s: 3.0.0

For better JDK11 support, this issue aims to upgrade **Jersey** and **javassist** to `2.30` and `3.35.0-GA` respectively.

*Jersey*:
- https://eclipse-ee4j.github.io/jersey.github.io/release-notes/2.30.html
  - https://github.com/eclipse-ee4j/jersey/issues/4245 (Java 11 java.desktop module dependency)

*javassist*: This is a transitive dependency upgrade from 3.20.0-CR2 to 3.25.0-GA.
- `javassist` officially supports JDK11 from [3.24.0-GA release note](https://github.com/jboss-javassist/javassist/blob/master/Readme.html#L308).
Bug: Disable all the V2 file sources in Spark 3.0 by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Jan/20 23:53
Affects Version/s: 3.0.0

There are still some missing parts in the file source V2 framework:
1. It doesn't support reporting file scan metrics such as "numOutputRows"/"numFiles"/"fileSize" like `FileSourceScanExec`. 
2. It doesn't support partition pruning with subqueries or dynamic partition pruning.

As we are going to code freeze on Jan 31st, I suggest disabling all the V2 file sources in Spark 3.0 by default.
Bug: Allow default merge command resolution to be bypassed by DSv2 sources
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 20:57
Affects Version/s: 2.4.4

Problem: Some DSv2 sources may want to customize the merge resolution logic. For example, a table that can accept any schema (TableCapability.ACCEPT_ANY_SCHEMA) may want to allow certain merge queries that are blocked (that is, throws AnalysisError) by the default resolution logic. So there should be a way to completely bypass the merge resolution logic in the Analyzer. 

Potential solution: Skip resolving the merge expressions if the target is a DSv2 table with  ACCEPT_ANY_SCHEMA capability.
Bug: Job History Storage Tab does not display RDD Table
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 10/Jan/20 10:00
Affects Version/s: 3.0.0

scala> import org.apache.spark.storage.StorageLevel._
import org.apache.spark.storage.StorageLevel._

scala> val rdd = sc.range(0, 100, 1, 5).setName("rdd")
rdd: org.apache.spark.rdd.RDD[Long] = rdd MapPartitionsRDD[1] at range at <console>:27

scala> rdd.persist(MEMORY_ONLY_SER)
res0: rdd.type = rdd MapPartitionsRDD[1] at range at <console>:27

scala> rdd.count
res1: Long = 100                                                                

scala> val df = Seq((1, "andy"), (2, "bob"), (2, "andy")).toDF("count", "name")
df: org.apache.spark.sql.DataFrame = [count: int, name: string]

scala> df.persist(DISK_ONLY)
res2: df.type = [count: int, name: string]

scala> df.count
res3: Long = 3

Open Storage Tab under Incomplete Jobs in Job History Page
UI will not display the RDD Table.
Bug:  Hive MetaException
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 10/Jan/20 15:01
Affects Version/s: 2.4.4

Hi ,

I am getting below error

INFO TransactionTableCreation: Exception Occurred - [Ljava.lang.StackTraceElement;@4fd7c296
20/01/10 14:09:07 INFO TransactionTableCreation: Exception Occurred - Caught Hive MetaException attempting to get partition metadata by filter from Hive. You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem, however this will result in degraded performance.
Bug: Applying the `like` function with 2 parameters fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Jan/20 19:14
Affects Version/s: 3.0.0

The `Like` expression is registered as the `like` function. Applying the function with 2 parameters fails with the exception:
{code}
spark-sql> select like('Spark', 'S%');

Invalid arguments for function like; line 1 pos 7
org.apache.spark.sql.AnalysisException: Invalid arguments for function like; line 1 pos 7
	at org.apache.spark.sql.catalyst.analysis.FunctionRegistry$.$anonfun$expression$7(FunctionRegistry.scala:618)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.catalyst.analysis.FunctionRegistry$.$anonfun$expression$4(FunctionRegistry.scala:602)
	at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry.lookupFunction(FunctionRegistry.scala:121)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction(SessionCatalog.scala:1412)
{code}
Bug: Add a fallback Maven repository
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/20 21:23
Affects Version/s: 2.4.4

This issue aims to add a fallback Maven repository when a mirror to `central` fail.
For example, when we use Google Maven Central in GitHub Action as a mirror of `central`,
this will be used when Google Maven Central is out of sync due to its late sync cycle.
Bug: Make conflict attributes resolution more scalable in ResolveReferences
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jan/20 09:38
Affects Version/s: 3.0.0

Currently, ResolveReferences rule only resolves conflict attributes of one single conflict plan pair in one iteration, which can be inefficient when there're many conflicts. We could make it more scalable by doing resolution in batch.
Bug: Run test suites for CSV v1 and JSON v1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Jan/20 15:20
Affects Version/s: 3.0.0

Currently, CSVSuite and JSONSuite test only CSV and JSON version 2 but version 1 which is still supported, and can be enabled by users. This ticket aims to add CSVv1Suite and JsonV1Suite similar to AvroSuite.
Bug: File source V2: Push data filters for file listing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jan/20 23:36
Affects Version/s: 3.0.0

Follow up on [SPARK-30428|https://github.com/apache/spark/pull/27112] which added support for partition pruning in File source V2.
 We should also pass the {{dataFilters}} to the {{listFiles method.}}

Datasources such as {{csv}} and {{json}} do not implement the {{SupportsPushDownFilters}} trait. In order to support data skipping uniformly for all file based data sources, one can override the {{listFiles}} method in a {{FileIndex}} implementation and use the {{dataFilters}} and partitionFilters to consult external metadata and prunes the list of files.
Bug: Uncache table in tempViews if needed on session closed
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 09/Jan/20 10:32
Affects Version/s: 2.3.2

Currently, Spark will not cleanup cached tables in tempViews produced by sql like following

`CACHE TABLE table1 as SELECT ....`

There are risks that the `uncache table` not called due to session closed unexpectedly, or user closed manually. Then these temp views will lost, and we can not visit them in other session, but the cached plan still exists in the `CacheManager`.

Moreover, the leaks may cause the failure of the subsequent query, one failure we encoutered in our production environment is as below:
{code:java}
Caused by: java.io.FileNotFoundException: File does not exist: /user/xxxx/xx/data__db60e76d_91b8_42f3_909d_5c68692ecdd4Caused by: java.io.FileNotFoundException: File does not exist: /user/xxxx/xx/data__db60e76d_91b8_42f3_909d_5c68692ecdd4It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:131) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:182) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:109) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage0.scan_nextBatch_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage0.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439)
{code}
The above exception happens when user update the data of the table, but spark still use the old cached plan.
Bug: Invalidate interval type as a field table schema
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 19/Jan/20 07:32
Affects Version/s: 3.0.0

After this commit https://github.com/apache/spark/commit/d67b98ea016e9b714bef68feaac108edd08159c9, we are able to create table or alter table with interval column types if the external catalog accepts which is varying the interval type's purpose for internal usage.
Bug: CSV load followed by "is null" filter produces incorrect results
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Jan/20 16:23
Affects Version/s: 3.0.0

Trying to filter on is null from values loaded from a CSV file has regressed recently and now produces incorrect results.

Given a CSV file with the contents:
{noformat:title=floats.csv}
100.0,1.0,
200.0,,
300.0,3.0,
1.0,4.0,
,4.0,
500.0,,
,6.0,
-500.0,50.5
 {noformat}
Filtering this data for the first column being null should return exactly two rows, but it is returning extraneous rows with nulls:
{noformat}
scala> val schema = StructType(Array(StructField("floats", FloatType, true),StructField("more_floats", FloatType, true)))
schema: org.apache.spark.sql.types.StructType = StructType(StructField(floats,FloatType,true), StructField(more_floats,FloatType,true))

scala> val df = spark.read.schema(schema).csv("floats.csv")
df: org.apache.spark.sql.DataFrame = [floats: float, more_floats: float]

scala> df.filter("floats is null").show
+------+-----------+
|floats|more_floats|
+------+-----------+
|  null|       null|
|  null|       null|
|  null|       null|
|  null|       null|
|  null|        4.0|
|  null|       null|
|  null|        6.0|
+------+-----------+
{noformat}
Bug: Bump lz4-java version to 1.7.1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jan/20 14:56
Affects Version/s: 3.0.0

lz4-java v1.7.0 has an issue on older macOS (e.g., v10.12 and v10.13). Since v1.7.1 will be released in the end of next week, we need to upgrade: https://github.com/lz4/lz4-java/issues/156#issuecomment-573063299
Bug: Avoid unnecessary WrappedArray roundtrip in GenericArrayData constructor
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jan/20 07:22
Affects Version/s: 3.0.0

GenericArrayData has a constructor which accepts a {{seqOrArray: Any}} parameter. This constructor was originally added for use in situations where we don't know the actual type at compile-time (e.g. when converting UDF outputs). It's also called (perhaps unintentionally) in code paths where we could plausibly and statically know that the type is {{Array[Any]}} (in which case we could simply call the primary constructor).

In the current version of this code there's an unnecessary performance penalty for going through this path when {{seqOrArray}} is an {{Array[Any]}}: we end up converting the array into a WrappedArray, then call a method to unwrap it back into an array: this results in a bunch of unnecessary method calls. See https://scastie.scala-lang.org/7jOHydbNTaGSU677FWA8nA for an example of situations where this can crop up.

Via a small modification to this constructor's implementation, I think we can effectively remove this penalty.
Bug: Disable comparison for interval type
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Jan/20 11:09
Affects Version/s: 3.0.0

As we are not going to follow ANSI, it is weird to compare the year-month part to the day-time part for our current implementation of interval. Additionally, the current ordering logic comes from PostgreSQL where the implementation of the interval is messy. And we are not aiming PostgreSQL compliance at all.
Bug: Use `NoOp` datasource in SQL benchmarks
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jan/20 13:29
Affects Version/s: 2.4.4

Currently, SQL benchmarks use `count()`, `collect()` and `foreach(_ => ())` actions. The actions have additional overhead. For example, `collect()` converts column values to external type values and pull data on the driver. Need to unify benchmark and the `NoOp` datasource except the benchmarks for `count()` or `collect()`
Bug: Precision and scale should be same for values between -1.0 and 1.0 in Decimal
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jan/20 08:36
Affects Version/s: 3.0.0

Currently, for values between -1.0 and 1.0, precision and scale is inconsistent between Decimal and DecimalType. For example, for numbers like 0.3, it has (precision, scale) as (2, 1) in Decimal, but (1, 1) in DecimalType. We should make Decimal be consistent with DecimalType.