av9ash/GitBugs huggingface dataset:
Bug: Perf regression of fromJavaDate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Apr/20 18:01
Affects Version/s: 3.0.0

DateTimeBenchmark shows the regression

Spark 2.4.6-SNAPSHOT at the PR https://github.com/MaxGekk/spark/pull/27
{code}
================================================================================================
Conversion from/to external types
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
To/from java.sql.Timestamp:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
From java.sql.Date                                  614            655          43          8.1         122.8       1.0X
{code}

Current master:
{code}
================================================================================================
Conversion from/to external types
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
To/from java.sql.Timestamp:               Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
From java.sql.Date                                 1154           1206          46          4.3         230.9       1.0X
{code}

The regression is ~x2.
Bug: Incorrect rebasing of BCE dates
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/20 20:18
Affects Version/s: 3.0.0

Dates of before common era are rebased incorrectly, see https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120679/testReport/org.apache.spark.sql/SQLQueryTestSuite/sql/
{code}
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: postgreSQL/date.sql
Expected "[-0044]-03-15", but got "[0045]-03-15" Result did not match for query #93
select make_date(-44, 3, 15)
{code}
Even such dates are out of the valid range of dates supported by the DATE type, there is a test in postgreSQL/date.sql for a negative year, and it would be nice to fix the issue.
Bug: Regression in loading/saving timestamps from/to ORC files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/20 14:54
Affects Version/s: 3.0.0

Here are results of DateTimeRebaseBenchmark on the current master branch:
{code}
Save timestamps to ORC:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
after 1582                                        59877          59877           0          1.7         598.8       0.0X
before 1582                                       61361          61361           0          1.6         613.6       0.0X

Load timestamps from ORC:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off                               48197          48288         118          2.1         482.0       1.0X
after 1582, vec on                                38247          38351         128          2.6         382.5       1.3X
before 1582, vec off                              53179          53359         249          1.9         531.8       0.9X
before 1582, vec on                               44076          44268         269          2.3         440.8       1.1X
{code}

The results of the same benchmark on Spark 2.4.6-SNAPSHOT:
{code}
Save timestamps to ORC:                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
after 1582                                        18858          18858           0          5.3         188.6       1.0X
before 1582                                       18508          18508           0          5.4         185.1       1.0X

Load timestamps from ORC:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
after 1582, vec off                               14063          14177         143          7.1         140.6       1.0X
after 1582, vec on                                 5955           6029         100         16.8          59.5       2.4X
before 1582, vec off                              14119          14126           7          7.1         141.2       1.0X
before 1582, vec on                                5991           6007          25         16.7          59.9       2.3X
{code}
 Here is the PR with DateTimeRebaseBenchmark backported to 2.4: https://github.com/MaxGekk/spark/pull/27
Bug: Performance regression with new TimestampFormatter for json and csv
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 11:16
Affects Version/s: 3.0.0

with benchmark original, where the timestamp values are valid to new parser

the result is 

{code:java}
[info] Running benchmark: Read dates and timestamps
[info]   Running case: timestamp strings
[info]   Stopped after 3 iterations, 5781 ms
[info]   Running case: parse timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 44764 ms
[info]   Running case: infer timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 93764 ms
[info]   Running case: from_json(timestamp)
[info]   Stopped after 3 iterations, 59021 ms
{code}

when we modify the benchmark to 

{code:java}
      def timestampStr: Dataset[String] = {
        spark.range(0, rowsNum, 1, 1).mapPartitions { iter =>
          iter.map(i => s"""{"timestamp":"1970-01-01T01:02:03.${i % 100}"}""")
        }.select($"value".as("timestamp")).as[String]
      }

      readBench.addCase("timestamp strings", numIters) { _ =>
        timestampStr.noop()
      }

      readBench.addCase("parse timestamps from Dataset[String]", numIters) { _ =>
        spark.read.schema(tsSchema).json(timestampStr).noop()
      }

      readBench.addCase("infer timestamps from Dataset[String]", numIters) { _ =>
        spark.read.json(timestampStr).noop()
      }
{code}

where the timestamp values are invalid for the new parser which cause fallback to legacy parser.
the result is 

{code:java}
[info] Running benchmark: Read dates and timestamps
[info]   Running case: timestamp strings
[info]   Stopped after 3 iterations, 5623 ms
[info]   Running case: parse timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 506637 ms
[info]   Running case: infer timestamps from Dataset[String]
[info]   Stopped after 3 iterations, 509076 ms
{code}

About 10x perf-regression
Bug: Fix failed tests due to result order changing when we enable AQE
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 07:43
Affects Version/s: 3.0.0

query #147 in SQLQueryTestSuite#"udf/postgreSQL/udf-join.sql - Scala UDF" and test sql/SQLQuerySuite#"check outputs of expression examples" will fail when enable AQE due to result order changing.
Bug: Speed up reading dates in ORC
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Apr/20 13:25
Affects Version/s: 3.0.0

Currently, ORC datasource converts values of DATE type to java.sql.Date and the result to days since the epoch in Proleptic Gregorian calendar. ORC datasource does such conversion when 
spark.sql.orc.enableVectorizedReader is set to false.

The conversion to java.sql.Date is not necessary because we can use DaysWritable which performs rebasing in much more optimal way.
Bug: NPE in OptimizeSkewedJoin when there's a inputRDD of plan has 0 partition
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 08:59
Affects Version/s: 3.0.0

When there's a inputRDD of a plan with 0 partitions, rule OptimizeSkewedJoin can hit NPE.

The issue can be reproduced by below test:
{code:java}
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true",
  SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
  withTempView("t2") {
    // create DataFrame with 0 partition
    spark.createDataFrame(sparkContext.emptyRDD[Row], new StructType().add("b", IntegerType))
      .createOrReplaceTempView("t2")
    // should run successfully without NPE
    runAdaptiveAndVerifyResult("SELECT * FROM testData2 t1 left semi join t2 ON t1.a=t2.b")
  }
}
{code}
Bug: Fix NPE when BlockManagerSource is used after BlockManagerMaster stops
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Apr/20 06:24
Affects Version/s: 2.3.4

In `SparkEnv.stop`, the following stop sequence is used.
{code}
blockManager.stop()
blockManager.master.stop()
metricsSystem.stop()
{code}

During `metricsSystem.stop`, some sink can invoke `BlockManagerSource` and ends up with NPE.
{code}
sinks.foreach(_.stop)
registry.removeMatching((_: String, _: Metric) => true)
{code}

{code}
java.lang.NullPointerException
	at org.apache.spark.storage.BlockManagerMaster.getStorageStatus(BlockManagerMaster.scala:170)
	at org.apache.spark.storage.BlockManagerSource$$anonfun$10.apply(BlockManagerSource.scala:63)
	at org.apache.spark.storage.BlockManagerSource$$anonfun$10.apply(BlockManagerSource.scala:63)
	at org.apache.spark.storage.BlockManagerSource$$anon$1.getValue(BlockManagerSource.scala:31)
	at org.apache.spark.storage.BlockManagerSource$$anon$1.getValue(BlockManagerSource.scala:30)
{code}

Since SparkContext registers and forgets `BlockManagerSource` without deregistering, we had better avoid NPE inside `BlockManagerMaster`.
{code}
    _env.metricsSystem.registerSource(new BlockManagerSource(_env.blockManager))
{code}
Bug: transformAllExpressions can not transform expressions in ScalarSubquery
Status: Open | Priority: Major | Resolution: 
Created: 11/Apr/20 05:10
Affects Version/s: 3.1.0

Example:
{code:scala}
spark.sql("CREATE TABLE t1 (c1 bigint, c2 bigint)")
val sqlStr = "SELECT COUNT(*) FROM t1 HAVING SUM(c1) > (SELECT AVG(c2) FROM t1)"
println(normalizeExprIds(spark.sql(sqlStr).queryExecution.optimizedPlan).treeString)
{code}
The output:
{noformat}
Project [count(1)#0L]
+- Filter (isnotnull(sum(c1#2L)#0L) AND (cast(sum(c1#2L)#0L as double) > scalar-subquery#0 []))
   :  +- Aggregate [avg(c2#3L) AS avg(c2)#6]
   :     +- Project [c2#3L]
   :        +- Relation[c1#2L,c2#3L] parquet
   +- Aggregate [count(1) AS count(1)#0L, sum(c1#0L) AS sum(c1#2L)#0L]
      +- Project [c1#0L]
         +- Relation[c1#0L,c2#0L] parquet
{noformat}

Another example query:
https://github.com/apache/spark/blob/master/sql/core/src/test/resources/tpcds/q14b.sql
Bug: Speed up timestamps rebasing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/20 11:29
Affects Version/s: 3.0.0

Currently, rebasing of timestamps is performed via conversions to local timestamps and back to microseconds. This is CPU intensive operation which can be avoid by converting via pre-calculated tables per each time zone. For example, the below is timestamps when diffs are changed in America/Los_Angeles time zone for the range 0001-01-01...2100-01-01
{code}
0001-01-01T00:00 diff = -2872 minutes
0100-03-01T00:00 diff = -1432 minutes
0200-03-01T00:00 diff = 7 minutes
0300-03-01T00:00 diff = 1447 minutes
0500-03-01T00:00 diff = 2887 minutes
0600-03-01T00:00 diff = 4327 minutes
0700-03-01T00:00 diff = 5767 minutes
0900-03-01T00:00 diff = 7207 minutes
1000-03-01T00:00 diff = 8647 minutes
1100-03-01T00:00 diff = 10087 minutes
1300-03-01T00:00 diff = 11527 minutes
1400-03-01T00:00 diff = 12967 minutes
1500-03-01T00:00 diff = 14407 minutes
1582-10-15T00:00 diff = 7 minutes
1883-11-18T12:22:58 diff = 0 minutes
{code}
It seems it is possible to build rebasing maps, and perform rebasing via the maps.
Bug: Support json_arrayAgg
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Apr/20 11:09
Affects Version/s: 3.1.0

Returns a JSON array by aggregating all the JSON arrays from a set of JSON arrays, or by aggregating the values of a Column.

Some of the Databases supporting this aggregate function are:
 * MySQL
 * PostgreSQL
 * Maria_DB
 * Sqlite
 * IBM Db2
Bug: Support json_objectAgg function
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 09/Apr/20 11:05
Affects Version/s: 3.1.0

Returns a JSON object containing the key-value pairs by aggregating the key-values of set of Objects or columns. 

 

This aggregate function is supported by: 
 * MySQL
 * PostgreSQL
 * IBM Db2
 * Maria_DB
 * Sqlite
Bug: stage level scheduling dynamic allocation bug with initial num executors
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Apr/20 14:38
Affects Version/s: 3.1.0

I found a bug in the stage level scheduling dynamic allocation code when you have a non default profile and it has an initial number of executors the same as what the number of executors needed for the first job, then we don't properly request the executors.
Bug: Fix hung-up issue in StagePage
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Apr/20 12:14
Affects Version/s: 3.1.0

StagePage will be hung-up with following operations.

1. Run a job with shuffle.
scala> sc.parallelize(1 to 10).map(x => (x, x)).reduceByKey(_ + _).collect

2. Visit StagePage for the stage writing shuffle data and check `Shuffle Write Time`.

3. Run a job with no shuffle.
scala> sc.parallelize(1 to 10).collect

4. Visit StagePage for the last stage.

This issue is caused by following reason.

In stagepage.js, an array `optionalColumns` has indices for columns for optional metrics.
If a stage doesn't perform shuffle read or write, the corresponding indices are removed from the array.
StagePage doesn't try to create column for such metrics, even if the state of corresponding optional metrics are preserved as "visible".
But, if a stage doesn't perform both shuffle read and write, the index for `Shuffle Write Time` isn't removed.
In that case, StagePage tries to create a column for `Shuffle Write Time` even though there are no metrics for shuffle write, leading hungup.
Bug: TreeNode asCode function incorrectly handles null literals
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 09/Apr/20 22:42
Affects Version/s: 2.4.4

In the TreeNode code in Catalyst the asCode function incorrectly handles null literals.  When it tries to render a null literal it will match {{null}} using the third case expression and try to call {{null.toString}} which will raise a NullPointerException.

I verified this bug exists in Spark 2.4.4 and the same code appears to be in master:

[https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L707]

The fix seems trivial - add an explicit case for null.

One way to reproduce this is via:
{code:java}

      val plan =
        spark
          .sql("select if(isnull(id), null, 2) from testdb_jdbc.users")
          .queryExecution
          .optimizedPlan
      println(plan.asInstanceOf[Project].projectList.head.asCode) {code}
However any other way which generates a Literal with the value null will cause the issue.

In this case the above SparkSQL will generate the literal: {{Literal(null, IntegerType)}} for the "trueValue" of the if statement.
Bug: Inconsistent result of df.head(1) and df.head()
Status: In Progress | Priority: Minor | Resolution: 
Created: 23/Apr/20 03:46
Affects Version/s: 2.4.6

In this line [https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L1339], if you are calling `df.head()` and dataframe is empty, it will return *None*

but if you are calling `df.head(1)` and dataframe is empty, it will return *empty list* instead.

This particular behaviour is not consistent and can create confusion. Especially when you are calling `len(df.head())` which will throw an exception for empty dataframe
Bug: Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 07/Apr/20 14:10
Affects Version/s: 3.1.0

For some combinations of join algorithm and join types there are no unit tests for the "number of output rows" metric.

A list of missing unit tests include the following.
 * ShuffledHashJoin: leftOuter, RightOuter, LeftAnti, LeftSemi
 * BroadcastNestedLoopJoin: RightOuter
 * BroadcastHashJoin: LeftAnti
Bug: Non-global sort support for structured streaming
Status: Resolved | Priority: Minor | Resolution: Invalid
Created: 07/Apr/20 13:08
Affects Version/s: 3.1.0

Currently, all sorting is disallowed with structured streaming queries. Not allowing global sorting makes sense, but could non-global sorting (i.e. sortWithinPartitions) be allowed? I'm running into this with an external source I'm using, but not sure if this would be useful to file sources as well. I have to foreachBatch so that I can do a sortWithinPartitions.

Two main questions:
 * Does a local sort cause issues with any exactly-once guarantees streaming queries provides? I can't say I know or understand how these semantics work. Or are there other issues I can't think of this would cause?
 * Is the change as simple as changing the unsupported operations check to only look for global sorts instead of all sorts?

I have built a version that simply changes the unsupported check to only disallow global sorts and it seems to be working. Anything I'm missing or is it this simple?
Bug: don't use stop(paste to build R errors
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Apr/20 06:21
Affects Version/s: 2.4.5

I notice for example this:

stop(paste0("Arrow optimization does not support 'dapplyCollect' yet. Please disable ",
            "Arrow optimization or use 'collect' and 'dapply' APIs instead."))

paste0 is totally unnecessary here -- stop itself uses ... (vararg) with default ''-sep combination, i.e., the above is equivalent to:

stop("Arrow optimization does not support 'dapplyCollect' yet. Please disable ",
      "Arrow optimization or use 'collect' and 'dapply' APIs instead.")

More generally, for portability, this will make it more difficult for user-contributed translations because the standard set of tools for doing this (namely tools::update_pkg_po('.")) would fail to capture these messages as being candidates for translation.

In fact, it's completely preferable IMO to keep the entire stop("") message as a single string -- I've found that breaking the string across multiple lines makes translation across different languages with different grammars quite difficult. Understand there are lint style constraints however so I wouldn't press on that for now.

If formatting is needed, I recommend using stop(gettextf(...)) instead.
Bug: support drop multiple functions
Status: Open | Priority: Minor | Resolution: 
Created: 30/Apr/20 14:00
Affects Version/s: 3.1.0

postgresql support dropping of multiple functions in one command.Better spark sql also can support this.

 

[https://www.postgresql.org/docs/12/sql-dropfunction.html]

Drop multiple functions in one command:
DROP FUNCTION sqrt(integer), sqrt(bigint);
Bug: Internal checkSchemaInArrow is inefficient
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Apr/20 15:21
Affects Version/s: 3.0.0

Current implementation is doubly inefficient:

Repeatedly doing the same (95%) sapply loop
Doing scalar == on a vector (== should be done over the whole vector for efficiency)

See existing PR:

https://github.com/apache/spark/pull/28372
Bug: Improve DataSourceRegister interface
Status: In Progress | Priority: Minor | Resolution: 
Created: 06/Apr/20 21:07
Affects Version/s: 2.4.5

As the DSv2 API evolves, some breaking changes are occasionally made to the API. It's possible to split a plugin into a "common" part and multiple version-specific parts and this works good to have a single artifact for users. The one part that can't be currently worked around is the DataSourceRegister trait. This is an issue because users cargo-cult configuration values, and choosing the wrong plugin version gives a particularly baroque error message that bubbles up through ServiceLoader.

Currently, the class implementing DataSourceRegister must also be the class implementing the "toplevel" DataSourceV2 interface (and mixins), and these various interfaces occasionally change as the API evolves. As a practical matter, this means that there's no opportunity to decide at runtime which class to pass along to Spark. Attempting to add multiple DataSourceV2 implementations to services/META-INF causes an exception when the ServiceLoader tries to load the DataSourceRegister who implements the "different" DataSourceV2.

I would like to propose a new DataSourceRegister interface which adds a level of indirection between the what ServiceLoader and DataSourceV2 loads. E.g. (strawman)

{{interface DataSourceRegisterV2 {}}
{{  public String shortName();}}
{{  public Class<? implements DataSourceV2> getImplementation();}}
{{ }}}

Then org.apache.spark.sql.execution.datasources.DataSource.lookupDataSource would have its search algorithm extended to look for DataSourceRegisterV2 objects, and if one is located for the given shortName, return the class object from getImplementation(). At this point, the plugin could decide based on the current runtime environment which class to prevent to Spark. There wouldn't be any changes to plugins who don't implement this API.

If this is an acceptable idea, I can put together a PR for further comment.

Thanks
 Andrew
Bug: R: gapply documentation could be clearer about what the func argument is
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 27/Apr/20 03:36
Affects Version/s: 2.4.5

copied from pre-existing GH PR:

https://github.com/apache/spark/pull/28350

Spent a long time this weekend trying to figure out just what exactly key is in gapply's func. I had assumed it would be a named list, but apparently not -- the examples are working because schema is applying the name and the names of the output data.frame don't matter.

As near as I can tell the description I've added is correct, namely, that key is an unnamed list.
Bug: Enhance JsonDataSource by replacing jackson with simdjson
Status: Open | Priority: Minor | Resolution: 
Created: 16/Apr/20 23:57
Affects Version/s: 3.0.0

I came across this VLDB paper: [https://arxiv.org/pdf/1902.08318.pdf] on how to improve json reading speed. We use Spark to process terabytes of JSON, so we try to find ways to improve JSON parsing speed. 

 

[https://lemire.me/blog/2020/03/31/we-released-simdjson-0-3-the-fastest-json-parser-in-the-world-is-even-better/]

 

[https://github.com/simdjson/simdjson/issues/93]

 

Anyone on the opensource communty interested in leading this effort to integrate simdjson in spark json data source api?
Bug: RLIKE and LIKE expression compiles every time when it used
Status: Resolved | Priority: Minor | Resolution: Duplicate
Created: 16/Apr/20 16:30
Affects Version/s: 2.3.0


org.apache.spark.sql.catalyst.expressions

regexpExpressions.scala

line: 41

{code:scala}
   // try cache the pattern for Literal
  private lazy val cache: Pattern = right match {
    case x @ Literal(value: String, StringType) => compile(value)
    case _ => null
  }
{code}

StringType Literal value is UTF8String by default