av9ash/GitBugs huggingface dataset:
Bug: Pyspark.ml.clustering illegalArgumentException with dataframe created from rows
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 30/Mar/20 07:50
Affects Version/s: 2.4.0

I hope this is the right place and way to report a bug in (at least) the PySpark API:

BisectingKMeans in the following example is only exemplary, the error occurs with all clustering algorithms:
{code:python}
from pyspark.sql import Row
from pyspark.mllib.linalg import DenseVector
from pyspark.ml.clustering import BisectingKMeans

data = spark.createDataFrame([Row(test_features=DenseVector([43.0, 0.0, 200.0, 1.0, 1.0, 1.0, 0.0, 3.0])),
 Row(test_features=DenseVector([44.0, 0.0, 250.0, 1.0, 1.0, 1.0, 0.0, 1.0])),
 Row(test_features=DenseVector([23.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0])),
 Row(test_features=DenseVector([25.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 2.0])),
 Row(test_features=DenseVector([19.0, 0.0, 200.0, 1.0, 0.0, 1.0, 0.0, 1.0]))])

kmeans = BisectingKMeans(featuresCol='test_features').setK(4).setSeed(1)
model = kmeans.fit(data)
{code}
The .fit-call in the last line will fail with the following error:
{code:java}
Py4JJavaError: An error occurred while calling o51.fit.
: java.lang.IllegalArgumentException: requirement failed: Column test_features must be of type equal to one of the following types: [struct<type:tinyint,size:int,indices:array<int>,values:array<double>>, array<double>, array<float>] but was actually of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.
{code}
As can be seen, the data type reported to be passed to the function is the first data type in the list of allowed data types, yet the call ends in an error because of it.

See my [StackOverflow issue|[https://stackoverflow.com/questions/60884142/pyspark-py4j-illegalargumentexception-with-spark-createdataframe-and-pyspark-ml]] for more context
Bug: Split Parquet/Avro configs for rebasing dates/timestamps in read and in write
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Mar/20 17:46
Affects Version/s: 3.0.0

Currently, Spark provides 2 SQL configs to control rebasing of dates/timestamps in Parquet and Avro datasource: 

spark.sql.legacy.parquet.rebaseDateTime.enabled

spark.sql.legacy.avro.rebaseDateTime.enabled

The configs control rebasing in read and in write. That's can be inconvenient for users who want to read files saved by Spark 2.4 and earlier versions, and save dates/timestamps without rebasing.

The ticket aims to split the configs, and introduce separate SQL configs for read and for write.
Bug: Migrate the implementation of algorithms from MLlib to ML
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 30/Mar/20 08:52
Affects Version/s: 3.1.0

Migrate the implementation of algorithms from MLlib to ML and make MLlib a wrapper of the ML side. Including:

1, clustering: KMeans, BiKMeans, LDA, PIC

2, regression: AFT, Isotonic,

3, stat: ChiSquareTest, Correlation, KolmogorovSmirnovTest,

4, tree: remove the usage of OldAlgo/OldStrategy/etc

5, feature: Word2Vec, PCA

6, evaluation:

7, fpm: FPGrowth, PrefixSpan

8, linalg, MLUtils, etc
Bug: Incompatible Avro dates/timestamps with Spark 2.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 16:49
Affects Version/s: 3.0.0

Write dates/timestamps to Avro file in Spark 2.4.5:
{code}
$ export TZ="America/Los_Angeles"
$ bin/spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.5
{code}
{code:scala}
scala> df.write.format("avro").save("/Users/maxim/tmp/before_1582/2_4_5_ts_avro")

scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
+----------+
|date      |
+----------+
|1001-01-01|
+----------+


scala> df2.write.format("avro").save("/Users/maxim/tmp/before_1582/2_4_5_ts_avro")

scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
+--------------------------+
|ts                        |
+--------------------------+
|1001-01-01 01:02:03.123456|
+--------------------------+
{code}

Spark 3.0.0-preview2 ( and 3.1.0-SNAPSHOT) outputs different values from Spark 2.4.5:
{code}
$ export TZ="America/Los_Angeles"
$ /bin/spark-shell --packages org.apache.spark:spark-avro_2.12:2.4.5
{code}
{code:scala}
scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")

scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_date_avro").show(false)
+----------+
|date      |
+----------+
|1001-01-07|
+----------+

scala> spark.read.format("avro").load("/Users/maxim/tmp/before_1582/2_4_5_ts_avro").show(false)
+--------------------------+
|ts                        |
+--------------------------+
|1001-01-07 01:09:05.123456|
+--------------------------+
{code}
Bug: Incompatible Parquet dates/timestamps with Spark 2.4
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Mar/20 11:10
Affects Version/s: 3.0.0

Write dates/timestamps to Parquet file in Spark 2.4:
{code}
$ export TZ="UTC"
$ ~/spark-2.4/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_231)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.conf.set("spark.sql.session.timeZone", "UTC")

scala> val df = Seq(("1001-01-01", "1001-01-01 01:02:03.123456")).toDF("dateS", "tsS").select($"dateS".cast("date").as("d"), $"tsS".cast("timestamp").as("ts"))
df: org.apache.spark.sql.DataFrame = [d: date, ts: timestamp]

scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_micros")

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")

scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_micros")
scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_micros").show(false)
+----------+--------------------------+
|d         |ts                        |
+----------+--------------------------+
|1001-01-01|1001-01-01 01:02:03.123456|
+----------+--------------------------+
{code}
Spark 2.4 saves dates/timestamps in Julian calendar. The parquet-mr tool prints *1001-01-07* and *1001-01-07T01:02:03.123456+0000*:
{code}
$ java -jar /Users/maxim/proj/parquet-mr/parquet-tools/target/parquet-tools-1.12.0-SNAPSHOT.jar dump -m ./2_4_5_micros/part-00000-fe310bfa-0f61-44af-85ee-489721042c14-c000.snappy.parquet
INT32 d
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 1 ***
value 1: R:0 D:1 V:1001-01-07

INT64 ts
--------------------------------------------------------------------------------
*** row group 1 of 1, values 1 to 1 ***
value 1: R:0 D:1 V:1001-01-07T01:02:03.123456+0000
{code}
Spark 3.0.0-preview2 ( and 3.1.0-SNAPSHOT) prints the same as parquet-mr but different values from Spark 2.4:
{code}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-preview2
      /_/

Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_231)
scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_micros").show(false)
+----------+--------------------------+
|d         |ts                        |
+----------+--------------------------+
|1001-01-07|1001-01-07 01:02:03.123456|
+----------+--------------------------+
{code}
Bug: Benchmark date-time rebasing in Parquet datasource
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Mar/20 07:19
Affects Version/s: 3.0.0

* Add benchmarks for saving dates/timestamps to parquet when spark.sql.legacy.parquet.rebaseDateTime.enabled is set to true
* Add bechmark for loading dates/timestamps from parquet when rebasing is on
Bug: Remove useless conf set in pyspark context
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 13/Mar/20 03:29
Affects Version/s: 3.1.0

In Pyspark/context.py, we extract the configuration of the "spark.executorEnv" prefix from conf and put it in env. But in PythonWorkerFactory, we have already obtained these environment variables from ProgressBuilder, thus we do not need the extra copy.
In addition, in some special environment deployments, user-configured environment variables such as "spark.executorEnv.Path=/usr/lib:$PATH" will be automatically parsed as actual machine confs. If we still directly copy and overwrite it, the configuration will be a wrong config.
Bug: Perform propagating empty relation after RewritePredicateSubquery
Status: Reopened | Priority: Major | Resolution: 
Created: 27/Mar/20 05:38
Affects Version/s: 3.1.0

{code:java}
scala> spark.sql(" select * from values(1), (2) t(key) where key in (select 1 as key where 1=0)").queryExecution
res15: org.apache.spark.sql.execution.QueryExecution =
== Parsed Logical Plan ==
'Project [*]
+- 'Filter 'key IN (list#39 [])
   :  +- Project [1 AS key#38]
   :     +- Filter (1 = 0)
   :        +- OneRowRelation
   +- 'SubqueryAlias t
      +- 'UnresolvedInlineTable [key], [List(1), List(2)]

== Analyzed Logical Plan ==
key: int
Project [key#40]
+- Filter key#40 IN (list#39 [])
   :  +- Project [1 AS key#38]
   :     +- Filter (1 = 0)
   :        +- OneRowRelation
   +- SubqueryAlias t
      +- LocalRelation [key#40]

== Optimized Logical Plan ==
Join LeftSemi, (key#40 = key#38)
:- LocalRelation [key#40]
+- LocalRelation <empty>, [key#38]

== Physical Plan ==
*(1) BroadcastHashJoin [key#40], [key#38], LeftSemi, BuildRight
:- *(1) LocalTableScan [key#40]
+- Br...
{code}
Bug: Simplify ChiSq by adding a common method
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 07:10
Affects Version/s: 3.1.0

The logic in '{color:#c7a65d}chiSquaredDenseFeatures{color}' and '{color:#c7a65d}chiSquaredSparseFeatures{color}' can be unified.
Bug: Specify formats of time zone ID for JSON/CSV option and from/to_utc_timestamp
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Mar/20 11:36
Affects Version/s: 2.4.5

There are two distinct types of ID (see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html):
# Fixed offsets - a fully resolved offset from UTC/Greenwich, that uses the same offset for all local date-times
# Geographical regions - an area where a specific set of rules for finding the offset from UTC/Greenwich apply

For example three-letter time zone IDs are ambitious, and depend on the locale. They have been already deprecated in JDK, see https://docs.oracle.com/javase/8/docs/api/java/util/TimeZone.html :
{code}
For compatibility with JDK 1.1.x, some other three-letter time zone IDs (such as "PST", "CTT", "AST") are also supported. However, their use is deprecated because the same abbreviation is often used for multiple time zones (for example, "CST" could be U.S. "Central Standard Time" and "China Standard Time"), and the Java platform can then only recognize one of them.
{code}

The ticket aims to specify formats of the `timeZone` option in JSON/CSV datasource, and the `tz` parameter of the from_utc_timestamp() and to_utc_timestamp() functions.
Bug: Specify formats of `spark.sql.session.timeZone`
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/20 08:34
Affects Version/s: 2.4.5

There are two distinct types of ID (see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html):
# Fixed offsets - a fully resolved offset from UTC/Greenwich, that uses the same offset for all local date-times
# Geographical regions - an area where a specific set of rules for finding the offset from UTC/Greenwich apply

For example three-letter time zone IDs are ambitious, and depend on the locale. They have been already deprecated in JDK, see https://docs.oracle.com/javase/8/docs/api/java/util/TimeZone.html :
{code}
For compatibility with JDK 1.1.x, some other three-letter time zone IDs (such as "PST", "CTT", "AST") are also supported. However, their use is deprecated because the same abbreviation is often used for multiple time zones (for example, "CST" could be U.S. "Central Standard Time" and "China Standard Time"), and the Java platform can then only recognize one of them.
{code}

The ticket aims to specify formats of the SQL config *spark.sql.session.timeZone* in the 2 forms mentioned above.
Bug: HiveResult compatibility for DatasourceV2 command
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Mar/20 11:11
Affects Version/s: 3.0.0

HiveResult performs some compatibility matches and conversions for commands to be compatible with Hive output, e.g.:

{code}
    case ExecutedCommandExec(_: DescribeCommandBase) =>
      // If it is a describe command for a Hive table, we want to have the output format
      // be similar with Hive.
...
    // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
    case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended =>
{code}

It is needed for DatasourceV2 commands as well (eg. ShowTablesExec...).
Bug: sbt unidoc fail to resolve arvo dependency
Status: Open | Priority: Major | Resolution: 
Created: 26/Mar/20 01:33
Affects Version/s: 3.1.0


{code:java}
warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Main Scala API documentation to /home/jenkins/workspace/SparkPullRequestBuilder@6/target/scala-2.12/unidoc...
[info] Main Java API documentation to /home/jenkins/workspace/SparkPullRequestBuilder@6/target/javaunidoc...
[error] /home/jenkins/workspace/SparkPullRequestBuilder@6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123: value createDatumWriter is not a member of org.apache.avro.generic.GenericData
[error]     writerCache.getOrElseUpdate(schema, GenericData.get.createDatumWriter(schema))
[error]                                                         ^
[info] No documentation generated with unsuccessful compiler run
[error] one error found
{code}
Bug: Test case import another test case contains bracketed comments, can't display bracketed comments in golden files well.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Mar/20 03:14
Affects Version/s: 3.1.0

The content of 
{code:java}
nested-comments.sql 
{code} show below:

{code:java}
-- This test case just used to test imported bracketed comments.

-- the first case of bracketed comment
--QUERY-DELIMITER-START
/* This is the first example of bracketed comment.
SELECT 'ommented out content' AS first;
*/
SELECT 'selected content' AS first;
--QUERY-DELIMITER-END
{code}
The test case 
{code:java}
comments.sql 
{code} imports 
{code:java}
nested-comments.sql
{code}
 below:

{code:java}
--IMPORT nested-comments.sql
{code}

The output will be:

{code:java}
-- !query
/* This is the first example of bracketed comment.
SELECT 'ommented out content' AS first
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

mismatched input '/' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 
'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', '
ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
/* This is the first example of bracketed comment.
^^^
SELECT 'ommented out content' AS first


-- !query
*/
SELECT 'selected content' AS first
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

extraneous input '*/' expecting {'(', 'ADD', 'ALTER', 'ANALYZE', 'CACHE', 'CLEAR', 'COMMENT', 'COMMIT', 'CREATE', 'DELETE', 'DESC', 'DESCRIBE', 'DFS', 'DROP', 'EXPLAIN', 'EXPORT', 'FROM', 'GRANT', 'IMPORT', 'INSERT', 'LIST', 'LOAD', 'LOCK', 'MAP', 'MERGE', 'MSCK', 'REDUCE', 'REFRESH', 'REPLACE', 'RESET', 'REVOKE', 'ROLLBACK', 'SELECT', 'SET', 'SHOW', 'START', 'TABLE', 'TRUNCATE', 'UNCACHE', 'UNLOCK', 'UPDATE', 'USE', 'VALUES', 'WITH'}(line 1, pos 0)

== SQL ==
*/
^^^
SELECT 'selected content' AS first
{code}
Bug: Make display of stageId/stageAttemptId/taskId of sql metrics toggleable
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Mar/20 10:06
Affects Version/s: 3.0.0

It makes metrics harder to read after SPARK-30209 and user may not interest in extra info({{stageId/StageAttemptId/taskId }}) when they do not need debug.
Bug: Clone SparkSession should respect spark.sql.legacy.sessionInitWithConfigDefaults
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 05:57
Affects Version/s: 3.1.0

In SQL test, a conf specified by `withSQLConf` can be reverted to "original value" after cloning SparkSession if the "original value" is already set in SparkConf level. Because in `WithTestConf`, it doesn't  respect spark.sql.legacy.sessionInitWithConfigDefaults and always merge SQLConf with SparkConf.
Bug: `HiveResult.toHiveString` does not use the current session time zone
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Mar/20 20:21
Affects Version/s: 3.0.0

Currently, date/timestamp formatters in `HiveResult.toHiveString` are initialized once on instantiation of the `HiveResult` object, and pick up the session time zone. If the sessions time zone is changed, the formatters still use the previous one.

See the discussion there https://github.com/apache/spark/pull/23391#discussion_r397347820
Bug: Replace 3-letter time zones by zone offsets
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 24/Mar/20 20:14
Affects Version/s: 3.0.0

3-letter time zones are ambitious, and have been already deprecated in JDK, see [https://docs.oracle.com/javase/8/docs/api/java/util/TimeZone.html] . Also, some short names are mapped to region-based zone IDs, and don't conform to actual definitions. For example, the PST short name is mapped to America/Los_Angeles. It has different zone offsets in Java 7 and Java 8 APIs:
{code:scala}
scala> TimeZone.getTimeZone("PST").getOffset(Timestamp.valueOf("2016-11-05 23:00:00").getTime)/3600000.0
res11: Double = -7.0
scala> TimeZone.getTimeZone("PST").getOffset(Timestamp.valueOf("2016-11-06 00:00:00").getTime)/3600000.0
res12: Double = -7.0
scala> TimeZone.getTimeZone("PST").getOffset(Timestamp.valueOf("2016-11-06 01:00:00").getTime)/3600000.0
res13: Double = -8.0
scala> TimeZone.getTimeZone("PST").getOffset(Timestamp.valueOf("2016-11-06 02:00:00").getTime)/3600000.0
res14: Double = -8.0
scala> TimeZone.getTimeZone("PST").getOffset(Timestamp.valueOf("2016-11-06 03:00:00").getTime)/3600000.0
res15: Double = -8.0
{code}
and in Java 8 API https://github.com/apache/spark/pull/27980#discussion_r396287278

By definition, PST must be a constant and equals to UTC-08:00, see https://www.timeanddate.com/time/zones/pst

The ticket aims to replace all short time zone names by zone offsets in tests.
Bug: Support getTablesByType API of Hive Client
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 18/Mar/20 17:48
Affects Version/s: 3.0.0

Hive 2.3+ supports getTablesByType API, which is a precondition to implement SHOW VIEWS in HiveExternalCatalog. Currently, without this API, we can not get hive table with type HiveTableType.VIRTUAL_VIEW directly.
Bug: Document the backward incompatibility issue for Encoders.kryo and Encoders.javaSerialization
Status: Open | Priority: Major | Resolution: 
Created: 25/Mar/20 06:35
Affects Version/s: 3.0.0

We should document that when a user uses Encoders.kryo or Encoders.javaSerialization for state store, we don't guarantee binary compatibility across different Spark versions because these two types of encoders can be broken easily. Maybe add a new section after http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovery-semantics-after-changes-in-a-streaming-query
Bug: Failure on loading 1000-02-29 from parquet saved by Spark 2.4.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Mar/20 10:13
Affects Version/s: 3.0.0

Save valid date in Julian calendar by Spark 2.4.5 in a leap year, for instance 1000-02-29:
{code}
$ export TZ="America/Los_Angeles"
{code}
{code:scala}
scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
scala> df.write.mode("overwrite").format("avro").save("/Users/maxim/tmp/before_1582/2_4_5_date_avro_leap")

scala> val df = Seq(java.sql.Date.valueOf("1000-02-29")).toDF("dateS").select($"dateS".as("date"))
df: org.apache.spark.sql.DataFrame = [date: date]

scala> df.show
+----------+
|      date|
+----------+
|1000-02-29|
+----------+

scala> df.write.mode("overwrite").parquet("/Users/maxim/tmp/before_1582/2_4_5_date_leap")
{code}

Load the parquet files back by Spark 3.1.0-SNAPSHOT:
{code:scala}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_231)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_date_leap").show
+----------+
|      date|
+----------+
|1000-03-06|
+----------+


scala> spark.conf.set("spark.sql.legacy.parquet.rebaseDateTime.enabled", true)

scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_date_leap").show
20/03/21 03:03:59 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.time.DateTimeException: Invalid date 'February 29' as '1000' is not a leap year
	at java.time.LocalDate.create(LocalDate.java:429)
	at java.time.LocalDate.of(LocalDate.java:269)
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.rebaseJulianToGregorianDays(DateTimeUtils.scala:1008)
{code}
Bug: Constant fold deterministic Scala UDFs with foldable arguments
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 25/Mar/20 01:07
Affects Version/s: 3.0.0

Constant fold deterministic Scala UDFs with foldable arguments, conservatively.

ScalaUDFs that meet all following criteria are subject to constant folding in this feature:
* deterministic
* all arguments are foldable
* does not throw an exception when evaluating the UDF for constant folding
Bug: Rebase all dates/timestamps in conversion in Java types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Mar/20 05:27
Affects Version/s: 3.0.0

Currently, the fromJavaDate(), toJavaDate(), toJavaTimestamp() and fromJavaTimestamp() methods of DateTimeUtils perform rebase only dates before Gregorian cutover date 1582-10-15 assuming that Gregorian calendar has the same behavior in Java 7 and Java 8 API. The assumption is incorrect, in particular, in getting zone offsets, for instance:
{code:scala}
scala> java.time.ZoneId.systemDefault
res16: java.time.ZoneId = America/Los_Angeles
scala> java.sql.Timestamp.valueOf("1883-11-10 00:00:00").getTimezoneOffset / 60.0
warning: there was one deprecation warning; re-run with -deprecation for details
res17: Double = 8.0
scala> java.time.ZoneId.of("America/Los_Angeles").getRules.getOffset(java.time.LocalDateTime.parse("1883-11-10T00:00:00"))
res18: java.time.ZoneOffset = -07:52:58
{code}
Java 7 is not accurate, America/Los_Angeles changed time zone shift from
{code}
-7:52:58
{code}
to
{code}
-8:00 
{code}
The ticket aims to perform rebasing for any dates/timestamps independently from calendar cutover date.