av9ash/GitBugs huggingface dataset:
Bug: Incrementally create and materialize query stage to avoid unnecessary local shuffle
Status: In Progress | Priority: Major | Resolution: 
Created: 17/Jun/20 01:54
Affects Version/s: 3.1.0

The current way of creating query stage in AQE is in batch. For example, the children of a sort merge join will be materialized as query stages in a batch. Then AQE brings the optimization in and optimize sort merge join to broadcast join. Except for the broadcasted exchange, we don't need do any exchange on another side of join but we already materialized the exchange. Currently AQE wraps the materialized exchange with local reader, but it still brings unnecessary I/O. We can avoid unnecessary local shuffle by incrementally creating query stage.
Bug: Make micros rebasing functions via local timestamps pure
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Jun/20 18:47
Affects Version/s: 3.0.1

The functions rebaseGregorianToJulianMicros(zoneId: ZoneId, ...) and rebaseJulianToGregorianMicros(zoneId: ZoneId, ...) accept the zone id as the first parameter but use it only while forming ZonedDateTime and ignore in Java 7 GregorianCalendar. The Calendar instance uses the default JVM time zone internally. This causes the following problems:
# The functions depend on the global state variable. And calling the functions from different threads can return wrong results if the the default JVM time zone is changed during the execution.
# It is impossible to speed up generation of JSON files with diff/switch via parallelisation.
# The functions don't fully use the passed ZoneId.
Bug: Test failure RebaseDateTimeSuite."optimization of micros rebasing - Julian to Gregorian"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Jun/20 11:19
Affects Version/s: 3.0.1

The test fails on 1945-09-14 23:30:00.0. The failure can be reproduced by modifying the test:
{code:scala}
  test("optimization of micros rebasing - Julian to Gregorian") {
    outstandingZoneIds.filter(_.getId.contains("Hong"))foreach { zid =>
      withClue(s"zone id = $zid") {
        withDefaultTimeZone(zid) {
          val start = rebaseGregorianToJulianMicros(
            instantToMicros(LocalDateTime.of(1, 1, 1, 0, 0, 0).atZone(zid).toInstant))
          val end = rebaseGregorianToJulianMicros(
            instantToMicros(LocalDateTime.of(2100, 1, 1, 0, 0, 0).atZone(zid).toInstant))
          var micros = -761211000000000L
          do {
            val rebased = rebaseJulianToGregorianMicros(zid, micros)
            val rebasedAndOptimized = rebaseJulianToGregorianMicros(micros)
            assert(rebasedAndOptimized === rebased)
            micros += (MICROS_PER_DAY * 30 * (0.5 + Math.random())).toLong
          } while (micros <= end)
        }
      }
    }
  }
{code}
{code}
zone id = Asia/Hong_Kong -761211000000000 did not equal -761207400000000
ScalaTestFailureLocation: org.apache.spark.sql.catalyst.util.RebaseDateTimeSuite at (RebaseDateTimeSuite.scala:236)
Expected :-761207400000000
Actual   :zone id = Asia/Hong_Kong -761211000000000
{code}
Bug: Test failure "RebaseDateTimeSuite.optimization of micros rebasing - Gregorian to Julian"
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 17:53
Affects Version/s: 3.0.1

See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123688/testReport/org.apache.spark.sql.catalyst.util/RebaseDateTimeSuite/optimization_of_micros_rebasing___Gregorian_to_Julian/
Bug: Fix concurrency issue for ThriftCLIService to getPortNumber
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Jun/20 08:30
Affects Version/s: 3.0.0

When org.apache.spark.sql.hive.thriftserver.HiveThriftServer2#startWithContext called,
it starts ThriftCLIService in the background with a new Thread, at the same time we call ThriftCLIService.getPortNumber, we might not get the bound port if it's configured with 0.
Bug: Benchmark the EXCEPTION rebase mode
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Jun/20 06:19
Affects Version/s: 3.0.1

Add benchmarks for the EXCEPTION rebase mode to DateTimeRebaseBenchmark. It is the default value of spark.sql.legacy.parquet.datetimeRebaseModeInWrite and spark.sql.legacy.parquet.datetimeRebaseModeInRead, and it would be nice to benchmark it as well.
Bug: Make MDC configuration step be consistent between setLocalProperty and log4j.properties
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 11/Jun/20 15:40
Affects Version/s: 3.1.0

It's weird that we use "mdc.XXX" as key to set MDC value via `setLocalProperty` while we use "XXX" as key to set MDC pattern in log4j.properties.
Bug: The SparkR regexp_replace function causes problems
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/20 20:12
Affects Version/s: 2.4.5

The SparkR regex functions in R are not working as they should. Here's a reprex.

 
{code:java}
# Load packages

library(tidyverse)
library(sparklyr)
library(SparkR)


# Create data
df <- data.frame(test = c("Less 2", "A1,2", "Over 2", "Resp1 1aa"))
# Transfer data to Spark memory
df <- copy_to(sc, df, "df", overwrite = TRUE)
# Modify data
df1 <- df %>%
 dplyr::mutate(
 test = as.character(test),
 test1 = regexp_replace(test, "Less ", "<"),
 test1 = regexp_replace(test1, "A1", "<1"),
 test1 = regexp_replace(test1, "Over ", ">"),
 test2 = regexp_replace(test1, "[a-zA-Z]+", ""),
 test3 = regexp_replace(test2, "[\\,]", "aa"),
 test3 = regexp_replace(test3, " ", ""))
df2 <- df %>%
 dplyr::mutate(
 test = as.character(test),
 test = regexp_replace(test, "Less ", "<"),
 test = regexp_replace(test, "A1", "<1"),
 test = regexp_replace(test, "Over ", ">"),
 test = regexp_replace(test, "[a-zA-Z]+", ""),
 test = regexp_replace(test, "[\\,]", "aa"),
 test = regexp_replace(test, " ", ""))
# Collect and print
df1_1 <- df1 %>% as.data.frame()
df1_1
df2_1 <- df2 %>% as.data.frame()
df2_1
{code}
The column test3 in df1_1 is correct but the column test in df2_1 is not, although the input, regex patterns and the replacements are identical.

I find SparkR really great, but would be eager to use R regex instead of java regex.
Bug: Where to find Prometheus metrics path about executors in Spark 3
Status: Open | Priority: Major | Resolution: 
Created: 14/Jun/20 11:58
Affects Version/s: 3.0.0

I know in Spark 3.0, it supports Prometheus Sink. But I don't know where to get metrics about executors. I want to fetch realtime executor's *_filesystem.hdfs.read_bytes,_* etc.

 

I have tried several paths, but it doesn't work. Could you please tell me the Prometheus metrics path about all executors?
Bug: Normalize all binary comparison expressions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 08:28
Affects Version/s: 3.1.0

This test will fail:
{code:scala}
  test("SPARK-31912 Normalize all binary comparison expressions") {
    val original = testRelation
      .where('a === 'b && Literal(13) >= 'b).as("x")
    val optimized = testRelation
      .where(IsNotNull('a) && IsNotNull('b) && 'a === 'b && 'b <= 13 && 'a <= 13).as("x")
    comparePlans(Optimize.execute(original.analyze), optimized.analyze)
  }
{code}
Bug: Add pagination support for all jobs timeline
Status: In Progress | Priority: Major | Resolution: 
Created: 11/Jun/20 20:26
Affects Version/s: 3.0.1

If there are lots of jobs, rendering performance of all jobs timeline can significantly goes down. This issue is reported in SPARK-31967.
 For example, the following operation can take >40 sec.
{code:java}
(1 to 1000).foreach(_ => sc.parallelize(1 to 10).collect) {code}
Although it's not the fundamental solution, pagination can mitigate the issue.
Bug: Support both pandas 0.23 and 1.0
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Jun/20 20:54
Affects Version/s: 3.1.0

This issue aims to fix a bug by supporting both pandas 0.23 and 1.0.

{code}
$ pip install pandas==0.23.2

$ python -c "import pandas.CategoricalDtype"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas.CategoricalDtype'

$ python -c "from pandas.api.types import CategoricalDtype"
{code}
Bug: Explicitly close iterator in KVStoreView
Status: Open | Priority: Major | Resolution: 
Created: 10/Jun/20 08:00
Affects Version/s: 3.1.0

While reviewing SPARK-31929 we realized there're places which retrieve KVStoreView and leverage it but don't close the underlying iterator not knowing that it's actually closeable.

The culprit is that KVStoreView implements Iterable and hides the fact that it should be closed. The only way to close the underlying iterator properly is calling closeableIterator to get underlying iterator.

I think this is the way to go; force caller to call closeableIterator and let caller deal with underlying iterator with responsibility to close the iterator.
Bug: Document the default JVM time zone in to/fromJavaDate and legacy date formatters
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/Jun/20 09:18
Affects Version/s: 3.1.0

Update comments for DateTimeUtils.toJavaDate and fromJavaDate, and for the date legacy formatters, and say explicitly that the default JVM time zone is used intentionally.
Bug: Wrong warning of "WARN Master: App app-xxx requires more resource than any of Workers could have."
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Jun/20 14:24
Affects Version/s: 3.0.0

When starting spark-shell using local cluster mode, e.g. ./bin/spark-shell --master "local-cluster[2, 1, 1024]", there will be a warning:

 
{code:java}
20/06/06 22:09:09 WARN Master: App app-20200606220908-0000 requires more resource than any of Workers could have.
{code}
which means the application can not get enough resources to launch at least one executor.

But that's not true since we can successfully complete a job.
Bug: Upgrade fabric8 to 4.10.2 to support Kubernetes 1.18
Status: In Progress | Priority: Major | Resolution: 
Created: 08/Jun/20 11:09
Affects Version/s: 2.4.6

Current Spark 2.4.x version uses Fabric Kubernetes client version 4.6.1, which supports Kubernetes version 1.16, but not 1.17 or 1.18. [Fabric8's compatibility matrix|https://github.com/fabric8io/kubernetes-client/#compatibility-matrix] doesn't mention anything about Kubernetes 1.18 support, but initial testing with the upgraded library suggests that it's compatible enough for Spark 2.4 (Github pull request coming soon).
Bug: When using GCS as checkpoint location for Structured Streaming aggregation pipeline, the Spark writing job is aborted
Status: Open | Priority: Major | Resolution: 
Created: 08/Jun/20 16:31
Affects Version/s: 2.4.5

Structured streaming checkpointing does not work with Google Cloud Storage when there are aggregations included in the streaming pipeline.

Using GCS as the external store works fine when there are no aggregations present in the pipeline (i.e. groupBy); however, once an aggregation is introduced, the attached error is thrown.

The error is only thrown when aggregating and pointing checkpointLocation to GCS. The exact code works fine when pointing checkpointLocation to HDFS.

Is it expected for GCS to function as a checkpoint location for aggregated pipelines? Are efforts currently in progress to enable this? Is it on a roadmap?
Bug: Push down more predicates through Join
Status: In Progress | Priority: Major | Resolution: 
Created: 06/Jun/20 07:39
Affects Version/s: 3.1.0

Currently, in `PushPredicateThroughJoin`, if the condition predicate of `Or` operator can't be entirely pushed down, it will be thrown away. 
In fact, the predicates under `Or` operators can be partially pushed down.
For example, says `a` and `b` are able to be pushed into one of the joined tables, while `c` can't be pushed down, the predicate
`a or (b and c)` 
can be converted as 
`(a or b) and (a or c)`
We can still push down `(a or b)`.
We can't push down disjunctive predicates only when one of its children is not partially convertible.
Bug: toPandas with Arrow enabled doesn't show metrics in Query UI.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Jun/20 23:46
Affects Version/s: 2.4.5

When calling {{toPandas}}, usually Query UI shows each plan node's metric and corresponding Stage ID and Task ID:
{code:java}
>>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', 'y', 'z'])
>>> df.toPandas()
   x   y    z
0  1  10  abc
1  2  20  def
{code}
!Screen Shot 2020-06-03 at 4.47.07 PM.png!

but if Arrow execution is enabled, it shows only plan nodes and the duration is not correct:
{code:java}
>>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> df.toPandas()
   x   y    z
0  1  10  abc
1  2  20  def{code}
 

!Screen Shot 2020-06-03 at 4.47.27 PM.png!
Bug: Spark planner is extremely slow when struct transformation is used
Status: Open | Priority: Major | Resolution: 
Created: 05/Jun/20 21:31
Affects Version/s: 2.4.5

When doing multiple struct transformations one after another, then the Spark planner becomes extremely slow, greatly increasing with the depth of the transformation.

See the following for replicating the issue: [SparkTest.java|https://gist.github.com/kelemen/1bafe46e898326252cfda224b98a0e07].

The following runs out of memory with 2 GB heap in _RelationalGroupedDataset.agg_, but _agg_ completes with 4 GB heap. However, in this case the _show_ call must be commented, because it fails even with 4 GB heap (I have not tried how much it needs given that _agg_ already takes a lot of time).

When applying all the _mapFields_ calls, then _agg_ took me 20 mins.

When commenting the 1st _mapFields_ call, then _agg_ took me 50 sec.

When commenting the 2nd and 3rd _mapFields_ call, then _agg_ took me 5 seconds.

After various tries, it seems that the time required is mainly dependent on how many transformations are applied. Even the aggregation can be left out, and replaced with another simple transformation.

Aside from the performance, the plan generated by Spark will be poor as well, creating plans like (simplified notation) _sum(struct(F.A1, F.A2, F.A3).A3)_. And the previous needless struct wrapping is sometimes nested even deeper in the optimized plan.
Bug: Enable Java 8 time API in Thrift server
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Jun/20 06:45
Affects Version/s: 3.1.0

Switch to Java 8 time API by turning on the SQL config spark.sql.datetime.java8API.enabled to address the following issues:
# Date and timestamp string literals are parsed by using Java 8 time API and Spark's session time zone. Before the changes, date/timestamp values were collected as legacy types `java.sql.Date`/`java.sql.Timestamp`, and the value of such types didn't respect the config `spark.sql.session.timeZone`. To have consistent view, users had to keep JVM time zone and Spark's session time zone in sync.
# After the changes, formatting of date values doesn't depend on JVM time zone.
# While returning dates/timestamps of Java 8 type, we can avoid dates/timestamps rebasing from Proleptic Gregorian calendar to the hybrid calendar (Julian + Gregorian), and the issues related to calendar switching.
# Properly handle negative years (BCE).
# Consistent conversion of date/timestamp strings to/from internal Catalyst types in both direction to and from Spark.
Bug: Apply SharedThriftServer to all ThriftServer related tests
Status: In Progress | Priority: Major | Resolution: 
Created: 05/Jun/20 12:36
Affects Version/s: 3.1.0

To add 
{code:java}
HiveThriftBinaryServerSuite
HiveThriftCleanUpScratchDirSuite
HiveThriftHttpServerSuite
JdbcConnectionUriSuite
SingleSessionSuite
SparkMetadataOperationSuite
SparkThriftServerProtocolVersionsSuite
UISeleniumSuite
{code}

exist ones

{code:java}
ThriftServerQueryTestSuite
ThriftServerWithSparkContextSuite
{code}
Bug: Spark UI shows wrong driver memory configurations if driver memory is provided at run time
Status: Open | Priority: Major | Resolution: 
Created: 04/Jun/20 20:09
Affects Version/s: 2.4.3

I discovered that in cluster mode , when driver memory is provided via spark.driver.memory configuration at run time after creating spark session, spark doesn't pick this configurations at run time as application master is already launched by that time and picks the default spark driver memory configuration(1GB). 
However, on spark UI page, in environment tab, it still shows driver memory as the value passed via configurations at run time, which makes identifying and debugging this scenario more difficult. Driver memory should be shown as the value which spark is actually using in job.
Bug: Disable week-based date filed for parsing
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 02/Jun/20 08:41
Affects Version/s: 3.0.0

It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.

A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674

The existing behavior itself in 2.4 is confusing, e.g. 

{code:sql}
spark-sql> select to_timestamp('1', 'w');
1969-12-28 00:00:00
spark-sql> select to_timestamp('1', 'u');
1970-01-05 00:00:00
{code}

the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.


{code:sql}
spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy');
2020-01-01 00:00:00
spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY');
2019-12-29 00:00:00
spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w');
NULL
spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w');
2019-12-29 00:00:00
{code}

I think we don't need to introduce all the weird behavior from Java.
Bug: Legacy date formatters don't use specified time zone
Status: In Progress | Priority: Major | Resolution: 
Created: 03/Jun/20 20:46
Affects Version/s: 3.0.0

The legacy date formatters LegacyFastDateFormatter and LegacySimpleDateFormatter create FastDateFormat and SimpleDateFormat w/o specified zoneId via DateFormatter.getLegacyFormatter. It means the formatters use the default system JVM time zone and don't respect Spark's session time zone. The ticket aims to support provided time zone id.
Bug: Support `java.time.Instant` in Parquet filter pushdown
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/20 17:23
Affects Version/s: 3.0.0

Currently, ParquetFilters supports only java.sql.Timestamp values of TimestampType, and explicitly casts Any to java.sql.Timestamp, see
https://github.com/apache/spark/blob/cb0db213736de5c5c02b09a2d5c3e17254708ce1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L180

So, any filters refer to timestamp values are not pushed down to Parquet when spark.sql.datetime.java8API.enabled is true.
Bug: Incorrect filtering of old millis timestamp in parquet
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 01/Jun/20 10:45
Affects Version/s: 3.0.0

{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
scala> spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")
scala> Seq(java.sql.Timestamp.valueOf("1000-06-14 08:28:53.123")).toDF("ts").write.mode("overwrite").parquet("/Users/maximgekk/tmp/ts_millis_old_filter")

scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").show(false)
+-----------------------+
|ts                     |
+-----------------------+
|1000-06-14 08:28:53.123|
+-----------------------+


scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").filter($"ts" === "1000-06-14 08:28:53.123")
res6: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [ts: timestamp]

scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").filter($"ts" === "1000-06-14 08:28:53.123").show(false)
+---+
|ts |
+---+
+---+
{code}
Bug: Should drop test_udtf table when SingleSessionSuite completed
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jun/20 05:17
Affects Version/s: 3.0.0

If we execute mvn test SingleSessionSuite and HiveThriftBinaryServerSuite in order,  the test case "SPARK-11595 ADD JAR with input path having URL scheme"  in HiveThriftBinaryServerSuite will failed as following:

 
{code:java}
- SPARK-11595 ADD JAR with input path having URL scheme *** FAILED *** java.sql.SQLException: Error running query: org.apache.spark.sql.AnalysisException: Can not create the managed table('`default`.`test_udtf`'). The associated location('file:/home/yarn/spark_ut/spark_ut/baidu/inf-spark/spark-source/sql/hive-thriftserver/spark-warehouse/test_udtf') already exists.; at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:385) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254) at org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite.$anonfun$new$65(HiveThriftServer2Suites.scala:603) at org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite.$anonfun$new$65$adapted(HiveThriftServer2Suites.scala:603) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite.$anonfun$new$63(HiveThriftServer2Suites.scala:603) at org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite.$anonfun$new$63$adapted(HiveThriftServer2Suites.scala:573) at org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.$anonfun$withMultipleConnectionJdbcStatement$3(HiveThriftServer2Suites.scala:1074) at org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(HiveThriftServer2Suites.scala:1074) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
{code}
 

because SingleSessionSuite do `create table test_udtf` and not drop it when test  completed and HiveThriftBinaryServerSuite want to re-create this table.

If we execute mvn test HiveThriftBinaryServerSuite and SingleSessionSuite in order,both test suites will succeed, but we shouldn't rely on their execution order
Bug: ExternalShuffleBlockResolverSuite failed on Windows
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 28/Jun/20 19:27
Affects Version/s: 3.0.0

The method {code}ExecutorDiskUtils.createNormalizedInternedPathname{code} should consider the Windows file separator.
{code}
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.132 s <<< FAILURE! - in org.apache.spark.network.shuffle.ExternalShuffleBlockResolverSuite
[ERROR] testNormalizeAndInternPathname(org.apache.spark.network.shuffle.ExternalShuffleBlockResolverSuite)  Time elapsed: 0 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:</foo[/bar/]baz> but was:</foo[\bar\]baz>
        at org.apache.spark.network.shuffle.ExternalShuffleBlockResolverSuite.assertPathsMatch(ExternalShuffleBlockResolverSuite.java:160)
        at org.apache.spark.network.shuffle.ExternalShuffleBlockResolverSuite.testNormalizeAndInternPathname(ExternalShuffleBlockResolverSuite.java:149)
{code}