av9ash/GitBugs huggingface dataset:
Bug: Infer IsNotNull for non null intolerant child of null intolerant in join condition
Status: In Progress | Priority: Major | Resolution: 
Created: 25/May/20 07:31
Affects Version/s: 3.1.0

We should infer {{IsNotNull}} for all children of {{NullIntolerant}} expressions. For example:
{code:sql}
CREATE TABLE t1(c1 string, c2 string);
CREATE TABLE t2(c1 string, c2 string);
EXPLAIN SELECT t1.* FROM t1 JOIN t2 ON coalesce(t1.c1, t1.c2)=t2.c1;
{code}

{noformat}
== Physical Plan ==
*(4) Project [c1#5, c2#6]
+- *(4) SortMergeJoin [coalesce(c1#5, c2#6)], [c1#7], Inner
   :- *(1) Sort [coalesce(c1#5, c2#6) ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(coalesce(c1#5, c2#6), 200), true, [id=#33]
   :     +- Scan hive default.t1 [c1#5, c2#6], HiveTableRelation `default`.`t1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#5, c2#6], Statistics(sizeInBytes=8.0 EiB)
   +- *(3) Sort [c1#7 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(c1#7, 200), true, [id=#46]
         +- *(2) Filter isnotnull(c1#7)
            +- Scan hive default.t2 [c1#7], HiveTableRelation `default`.`t2`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#7, c2#8], Statistics(sizeInBytes=8.0 EiB)
{noformat}

We should infer {{coalesce(t1.c1, t1.c2) IS NOT NULL}} to improve query performance:
{noformat}
== Physical Plan ==
*(5) Project [c1#23, c2#24]
+- *(5) SortMergeJoin [coalesce(c1#23, c2#24)], [c1#25], Inner
   :- *(2) Sort [coalesce(c1#23, c2#24) ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(coalesce(c1#23, c2#24), 200), true, [id=#95]
   :     +- *(1) Filter isnotnull(coalesce(c1#23, c2#24))
   :        +- Scan hive default.t1 [c1#23, c2#24], HiveTableRelation `default`.`t1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#23, c2#24], Statistics(sizeInBytes=8.0 EiB)
   +- *(4) Sort [c1#25 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(c1#25, 200), true, [id=#103]
         +- *(3) Filter isnotnull(c1#25)
            +- Scan hive default.t2 [c1#25], HiveTableRelation `default`.`t2`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#25, c2#26], Statistics(sizeInBytes=8.0 EiB)
{noformat}

Real performance test case:
 !default.png!  !infer.png!
Bug: spark double count issue when table is recreated from hive
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 20/May/20 09:05
Affects Version/s: 2.3.2

Spark sets serde property named path when a table is created. Along with it , it will add a property called sql.source.provider that ensures the consistency. But if we recreate the same table using hive , then the hive will not respect the serde properties set by spark named sql.source.provider. This will lead to conflict if we read the newly created table using spark . 

 

It can be argued that the issue is with hive because it do not properly ensures that the table properties are being copied. But these are the properties set by spark on a hive table. and spark should be the one adhering to the standards of hive. From a corporate standpoint , we cannot tell users to avoid creating or altering tables on hive because spark will end up giving wrong results. *Worst of all , even if it failed then it was okay. but it simply gives double count without even a failure message and that is very dangerous for critical applications*. Either end up giving error message if the serde property does not have path as well as sql.source.provider (or) do not even try to create the property path in first place.  Few related issues : 
 # SPARK-31751
 # SPARK-28266 was resolved as non-reproducible which is not entirely true

 

*steps to reproduce :*

 
{code:java}
df = spark.createDataFrame([{"a": "x", "b": "y", "c": "3"}])
df.createOrReplaceTempView("test1")
spark.sql("CREATE TABLE test1_using_orc USING ORC AS (SELECT * from test1)")
{code}
 
----
Run the below from hive ( preferably tez )

 
{code:java}
create table test2_like_test1 like test1_using_orc;
insert into test2_like_test1 select * from test1_using_orc;{code}

-------from spark again : Now test2_like_test1 will have serde property called path , but will not have sql.source.provider. 

 

 
{code:java}
>>> spark.sql("set spark.sql.hive.convertMetastoreOrc=false").show()
+--------------------+-----+
| key|value|
+--------------------+-----+
|spark.sql.hive.co...|false|
+--------------------+-----+
>>> spark.sql("select count(*) from test2_like_test1").show()
+--------+
|count(1)|
+--------+
| 1|
+--------+
>>> spark.sql("set spark.sql.hive.convertMetastoreOrc=true").show()
+--------------------+-----+
| key|value|
+--------------------+-----+
|spark.sql.hive.co...| true|
+--------------------+-----+
>>> spark.sql("select count(*) from test2_like_test1").show()
+--------+
|count(1)|
+--------+
| 2|
+--------+
{code}
Bug: Spread partitions evenly to spark executors
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 06/May/20 07:02
Affects Version/s: 2.4.4

The year is 2020 and I am still trying to find a solution to this. I totally understand what [~thunderstumpges] was trying to achieve and I am trying to achieve the same. For a tool like spark, it is unacceptable not to be able to distribute the created partitions to the executors evenly. You know, we can create a custom partitioner to distribute the data to the partitions evenly by creating our own partition index. I was under the impression that a similar approach could be applied to spread these partitions to the executors evenly(using some sort of executor index for selection of executors during partition distribution). I have been googling this for a day now and I am very disappointed to say that up to now this seems to be not possible.

Note: I am disappointed that the issue below was put into resolved state without actually doing anything about it.

https://issues.apache.org/jira/browse/SPARK-19371
Bug: Include all columns in the summary report
Status: Resolved | Priority: Major | Resolution: Incomplete
Created: 16/May/20 17:30
Affects Version/s: 2.4.5

Dates and other columns are excluded:

 

{{from datetime import datetime, timedelta, timezone}}
{{from pyspark.sql import types as T}}
{{from pyspark.sql import Row}}
{{from pyspark.sql import functions as F}}{{START = datetime(2014, 1, 1, tzinfo=timezone.utc)}}{{n_days = 22}}{{date_range = [Row(date=(START + timedelta(days=n))) for n in range(0, n_days)]}}{{schema = T.StructType([T.StructField(name="date", dataType=T.DateType(), nullable=False)])}}
{{rdd = spark.sparkContext.parallelize(date_range)}}{{df = spark.createDataFrame(data=rdd, schema=schema)}}
{{df.agg(F.max("date")).show()}}{{df.summary().show()}}
{{+-------+}}
{{|summary|}}
{{+-------+}}
{{| count |}}
{{| mean  |}}
{{| stddev|}}
{{| min   |}}
{{| 25%   |}}
{{| 50%   |}}
{{| 75%   |}}
{{| max   |}}
{{+-------+}}
Bug: Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/20 14:43
Affects Version/s: 3.1.0

1.Add and register three new functions: {{TIMESTAMP_SECONDS}}, {{TIMESTAMP_MILLIS}} and {{TIMESTAMP_MICROS}}

Reference: [BigQuery|https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions?hl=en#timestamp_seconds]

2.People will have convenient way to get timestamps from seconds,milliseconds and microseconds.
Bug: Using complex type in Aggregation with cube failed Analysis error
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 09/May/20 14:52
Affects Version/s: 2.4.0

Will wrong with below SQL
{code:java}
test("TEST STRUCT FIELD WITH GROUP BY with CUBE") {
  withTable("t1") {
    sql(
      """create table t1(
        |a string,
        |b int,
        |c array<struct<row_id:int,json_string:string>>)
        |using orc""".stripMargin)

    sql(
      """
        |select a, coalesce(get_json_object(each.json_string,'$.iType'),'-127') as iType, sum(b)
        |from t1
        |LATERAL VIEW explode(c) x AS each
        |group by a, get_json_object(each.json_string,'$.iType')
        |with cube
        |""".stripMargin).explain(true)
  }
}
{code}
Error 
{code:java}
expression 'x.`each`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;
Aggregate [a#230, get_json_object(each#222.json_string AS json_string#223, $.iType)#231, spark_grouping_id#229L], [a#230, coalesce(get_json_object(each#222.json_string, $.iType), -127) AS iType#218, sum(cast(b#220 as bigint)) AS sum(b)#226L]
+- Expand [List(a#219, b#220, c#221, each#222, a#227, get_json_object(each#222.json_string AS json_string#223, $.iType)#228, 0), List(a#219, b#220, c#221, each#222, a#227, null, 1), List(a#219, b#220, c#221, each#222, null, get_json_object(each#222.json_string AS json_string#223, $.iType)#228, 2), List(a#219, b#220, c#221, each#222, null, null, 3)], [a#219, b#220, c#221, each#222, a#230, get_json_object(each#222.json_string AS json_string#223, $.iType)#231, spark_grouping_id#229L]
   +- Project [a#219, b#220, c#221, each#222, a#219 AS a#227, get_json_object(each#222.json_string, $.iType) AS get_json_object(each#222.json_string AS json_string#223, $.iType)#228]
      +- Generate explode(c#221), false, x, [each#222]
         +- SubqueryAlias spark_catalog.default.t1
            +- Relation[a#219,b#220,c#221] orc
{code}
Bug: Reduce the memory usage in file scan location metadata
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/May/20 07:19
Affects Version/s: 3.1.0

Currently, the data source scan node stores all the paths in its metadata. The metadata is kept when a SparkPlan is converted into SparkPlanInfo. SparkPlanInfo can be used to construct the Spark plan graph in UI.
However, the paths can be very large (e.g. it can be many partitions after partition pruning), while UI pages only require up to 100 bytes for the location metadata. We can reduce the paths stored in metadata to reduce memory usage.
Bug: "show tables like" support for SQL wildcard characters (% and _)
Status: In Progress | Priority: Major | Resolution: 
Created: 04/May/20 05:22
Affects Version/s: 3.1.0

https://docs.snowflake.com/en/sql-reference/sql/show-tables.html
https://clickhouse.tech/docs/en/sql-reference/statements/show/
https://www.mysqltutorial.org/mysql-show-tables/
https://issues.apache.org/jira/browse/HIVE-23359
Bug: Disable some flaky tests temporarily
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/May/20 06:01
Affects Version/s: 3.0.0

https://issues.apache.org/jira/browse/SPARK-31722
https://issues.apache.org/jira/browse/SPARK-31723
https://issues.apache.org/jira/browse/SPARK-31729
https://issues.apache.org/jira/browse/SPARK-31728
https://issues.apache.org/jira/browse/SPARK-31730
https://issues.apache.org/jira/browse/SPARK-31731
Bug: Register shuffle map output metadata with a shuffle output tracker
Status: In Progress | Priority: Major | Resolution: 
Created: 23/May/20 01:38
Affects Version/s: 3.1.0

Part of the design as discussed in [this document|https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit#].

Establish a {{ShuffleOutputTracker}} API that resides on the driver, and handle accepting map output metadata returned by the map output writers and send them to the output tracker module accordingly.

Requires https://issues.apache.org/jira/browse/SPARK-31798.
Bug: Use KVStore to cache stream query progress
Status: In Progress | Priority: Major | Resolution: 
Created: 11/May/20 09:23
Affects Version/s: 2.4.5

1. Streaming query progress information are cached twice in *StreamExecution* and *StreamingQueryStatusListener*.  It is memory-wasting. We can make this two usage unified.
2. Use *KVStore* instead to cache streaming query progress information.
Bug: Fail casting numeric to timestamp by default
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/May/20 10:27
Affects Version/s: 2.4.5

Hi Team

Steps to reproduce.
{code:java}
create table test(id bigint);
insert into test select 1586318188000;
create table test1(id bigint) partitioned by (year string);
insert overwrite table test1 partition(year) select 234,cast(id as TIMESTAMP) from test;
{code}
let's check the result. 

Case 1:

*select * from test1;*

234 | 52238-06-04 13:06:400.0

--the result is wrong

Case 2:

*select 234,cast(id as TIMESTAMP) from test;*

 

java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
 at java.sql.Timestamp.valueOf(Timestamp.java:237)
 at org.apache.hive.jdbc.HiveBaseResultSet.evaluate(HiveBaseResultSet.java:441)
 at org.apache.hive.jdbc.HiveBaseResultSet.getColumnValue(HiveBaseResultSet.java:421)
 at org.apache.hive.jdbc.HiveBaseResultSet.getString(HiveBaseResultSet.java:530)
 at org.apache.hive.beeline.Rows$Row.<init>(Rows.java:166)
 at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43)
 at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1756)
 at org.apache.hive.beeline.Commands.execute(Commands.java:826)
 at org.apache.hive.beeline.Commands.sql(Commands.java:670)
 at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:974)
 at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:810)
 at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:767)
 at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:480)
 at org.apache.hive.beeline.BeeLine.main(BeeLine.java:463)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
 Error: Unrecognized column type:TIMESTAMP_TYPE (state=,code=0)

 

I try hive,it works well,and the convert is fine and correct
{code:java}
select 234,cast(id as TIMESTAMP) from test;
 234   2020-04-08 11:56:28
{code}
Two questions:

q1:

if we forbid this convert,should we keep all cases the same?

q2:

if we allow the convert in some cases, should we decide the long length, for the code seems to force to convert to ns with times*1000000 nomatter how long the data is,if it convert to timestamp with incorrect length, we can raise the error.
{code:java}
// // converting seconds to us
private[this] def longToTimestamp(t: Long): Long = t * 1000000L{code}
 

Thanks!
Bug: Use a fallback version in HiveExternalCatalogVersionsSuite
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/May/20 05:32
Affects Version/s: 2.4.6

Currently, HiveExternalCatalogVersionsSuite is aborted in all Jenkins jobs except JDK11 Jenkins jobs which don't have old Spark releases supporting JDK11.

{code}
HiveExternalCatalogVersionsSuite:
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED ***
  Exception encountered when invoking run on a nested suite - Fail to get the lates Spark versions to test. (HiveExternalCatalogVersionsSuite.scala:180)
{code}
Bug: sequence producing inconsistent intervals for month step
Status: Resolved | Priority: Major | Resolution: Cannot Reproduce
Created: 06/May/20 23:41
Affects Version/s: 2.4.4

Taking an example from [https://spark.apache.org/docs/latest/api/sql/]
{code:java}
> SELECT sequence(to_date('2018-01-01'), to_date('2018-03-01'), interval 1 month);{code}
[2018-01-01,2018-02-01,2018-03-01]

if one is to expand `stop` till the end of the year some intervals are returned as the last day of the month whereas first day of the month is expected
{code:java}
> SELECT sequence(to_date('2018-01-01'), to_date('2019-01-01'), interval 1 month){code}
[2018-01-01, 2018-02-01, 2018-03-01, *2018-03-31, 2018-04-30, 2018-05-31, 2018-06-30, 2018-07-31, 2018-08-31, 2018-09-30, 2018-10-31*, 2018-12-01, 2019-01-01]
Bug: Add SupportsPartitions Catalog APIs on DataSourceV2
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/May/20 01:47
Affects Version/s: 3.1.0

There are no partition Commands, such as AlterTableAddPartition supported in DatasourceV2, it is widely used in mysql or hive or other places. Thus it is necessary to defined Partition Catalog API to support these Commands.

We add SupportsPartitions to use basic APIs to support these commands.
Bug: Support composed type of case class for typed Scala UDF
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/May/20 13:16
Affects Version/s: 3.1.0

After SPARK-30127, typed Scala UDF now supports to accept case class as input parameter. However, it still does not support types like Seq[T], Array[T], assuming T is a case class.
Bug: Spark SQL Sort fails when sorting big data points
Status: Open | Priority: Major | Resolution: 
Created: 04/May/20 08:22
Affects Version/s: 2.3.2

 Please have a look at the example below: 
{code:java}
case class Point(x:Double, y:Double)
case class Nested(a: Long, b: Seq[Point])
val test = spark.sparkContext.parallelize((1L to 100L).map(a => Nested(a,Seq.fill[Point](250000)(Point(1,2)))), 100)
test.toDF().as[Nested].sort("a").take(1)
{code}
 *Sorting* big data objects using *Spark Dataframe* is failing with following exception: 
{code:java}
2020-05-04 08:01:00 ERROR TaskSetManager:70 - Total size of serialized results of 14 tasks (107.8 MB) is bigger than spark.driver.maxResultSize (100.0 MB)
[Stage 0:======>                                                 (12 + 3) / 100]org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 13 tasks (100.1 MB) is bigger than spark.driver.maxResu
{code}
However using the *RDD API* is working and no exception is thrown: 
{code:java}
case class Point(x:Double, y:Double)
case class Nested(a: Long, b: Seq[Point])
val test = spark.sparkContext.parallelize((1L to 100L).map(a => Nested(a,Seq.fill[Point](250000)(Point(1,2)))), 100)
test.sortBy(_.a).take(1)
{code}
For both code snippets we started the spark shell with exactly the same arguments:
{code:java}
spark-shell --driver-memory 6G --conf "spark.driver.maxResultSize=100MB"
{code}
Even if we increase the spark.driver.maxResultSize, the executors still get killed for our use case. The interesting thing is that when using the RDD API directly the problem is not there. *Looks like there is a bug in dataframe sort because is shuffling too much data to the driver?* 

Note: this is a small example and I reduced the spark.driver.maxResultSize to a smaller size, but in our application I've tried setting it to 8GB but as mentioned above the job was killed.
Bug: Unregister application from YARN resource manager outside the shutdown hook
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 02/May/20 02:44
Affects Version/s: 3.1.0

Currently, an application is unregistered from YARN resource manager as a shutdown hook. In the scenario where the shutdown hook does not run (e.g., timeouts, etc.), the application is not unregistered, resulting in YARN resubmitting the application even if it succeeded.

For example, you could see the following on the driver log:
{code:java}
20/04/30 06:20:29 INFO SparkContext: Successfully stopped SparkContext
20/04/30 06:20:29 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
20/04/30 06:20:59 WARN ShutdownHookManager: ShutdownHook '$anon$2' timeout, java.util.concurrent.TimeoutException
java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask.get(FutureTask.java:205)
	at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124)
	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
{code}
On the YARN RM side:
{code:java}
2020-04-30 06:21:25,083 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1588227360159_0001_01_000001 Container Transitioned from RUNNING to COMPLETED
2020-04-30 06:21:25,085 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1588227360159_0001_000001 with final state: FAILED, and exit status: 0
2020-04-30 06:21:25,085 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1588227360159_0001_000001 State change from RUNNING to FINAL_SAVING on event = CONTAINER_FINISHED
{code}
You see that the final state of the application becomes FAILED since the container is finished before the application is unregistered.
Bug: DAGSchedulerSuite: For the pattern of runevent + assert, extract the general method
Status: In Progress | Priority: Major | Resolution: 
Created: 28/May/20 03:26
Affects Version/s: 3.1.0

DAGSchedulerSuite exists many test case(e.g. late fetch failures / task events always posted / ignore late map task completions) contains the pattern of runevent + assert, such as:

{code:java}
    runEvent(makeCompletionEvent(
      taskSets(1).tasks(0),
      FetchFailed(makeBlockManagerId("hostA"), shuffleId, 0L, 0, 0, "ignored"),
      null))
    assert(sparkListener.failedStages.contains(1))
{code}
We should extract the general method like runEventAndCheck(events*, predicate)
Bug: Upgrade to Zstd 1.4.5
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/May/20 15:32
Affects Version/s: 3.1.0

This issue aims to upgrade to Zstd 1.4.5

Zstd 1.4.5 improves performance.

[https://github.com/facebook/zstd/releases/tag/v1.4.5]
Bug: DAGSchedulerSuite: For the pattern of cancel + assert, extract the general method
Status: Open | Priority: Major | Resolution: 
Created: 28/May/20 03:53
Affects Version/s: 3.1.0

DAGSchedulerSuite exists many test case(e.g. trivial job cancellation | job cancellation no-kill backend) contains the pattern of cancel + assert, such as:

{code:java}
  test("trivial job cancellation") {
    val rdd = new MyRDD(sc, 1, Nil)
    val jobId = submit(rdd, Array(0))
    cancel(jobId)
    assert(failure.getMessage === s"Job $jobId cancelled ")
    assert(sparkListener.failedStages === Seq(0))
    assertDataStructuresEmpty()
  }
{code}
We should extract the general method like cancelAndCheck(jobId: Int, predicate)
Bug: DAGSchedulerSuite: For the pattern of failed + assert, extract the general method
Status: Open | Priority: Major | Resolution: 
Created: 28/May/20 03:43
Affects Version/s: 3.1.0

DAGSchedulerSuite exists many test case(e.g. trivial job failure | run shuffle with map stage failure) contains the pattern of failed + assert, such as:

{code:java}
  test("trivial job failure") {
    submit(new MyRDD(sc, 1, Nil), Array(0))
    failed(taskSets(0), "some failure")
    assert(failure.getMessage === "Job aborted due to stage failure: some failure")
    assert(sparkListener.failedStages === Seq(0))
    assertDataStructuresEmpty()
  }
{code}
We should extract the general method like failedAndCheck(taskSet: TaskSet, message: String, predicate)
Bug: Thriftserver with spark.sql.datetime.java8API.enabled=true
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/May/20 18:48
Affects Version/s: 3.0.0

{code}
  test("spark.sql.datetime.java8API.enabled=true") {
    withJdbcStatement() { st =>
      st.execute("set spark.sql.datetime.java8API.enabled=true")
      val rs = st.executeQuery("select timestamp '2020-05-28 00:00:00'")
      rs.next()
      // scalastyle:off
      println(rs.getObject(1))
    }
  }
{code}
fails with 
{code}
HiveThriftBinaryServerSuite:
java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
        at java.sql.Timestamp.valueOf(Timestamp.java:204)
        at org.apache.hive.jdbc.HiveBaseResultSet.evaluate(HiveBaseResultSet.java:444)
        at org.apache.hive.jdbc.HiveBaseResultSet.getColumnValue(HiveBaseResultSet.java:424)
        at org.apache.hive.jdbc.HiveBaseResultSet.getObject(HiveBaseResultSet.java:464
{code}

It seems it might be needed in HiveResult.toHiveString?
cc [~maxgekk]