Bug: Incorrect results from from_utc_timestamp function
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/24 02:43
Affects Version/s: 3.4.0

The values returned as output from_utc_timestamp are erratic and are not consistent in case of values before year 1850

 
{code:java}
❯ JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-17.jdk/Contents/Home/ bin/spark-shell --master local --conf spark.sql.session.timeZone=UTC
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
{"ts":"2024-09-06T02:42:45.333Z","level":"WARN","msg":"Your hostname, RINMAC2772, resolves to a loopback address: 127.0.0.1; using 192.168.28.3 instead (on interface en0)","context":{"host":"RINMAC2772","host_port":"127.0.0.1","host_port2":"192.168.28.3","network_if":"en0"},"logger":"Utils"}
{"ts":"2024-09-06T02:42:45.336Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if you need to bind to another address","logger":"Utils"}
{"ts":"2024-09-06T02:42:45.536Z","level":"INFO","msg":"Registering signal handler for INT","logger":"SignalUtils"}
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-preview1
      /_/Using Scala version 2.13.14 (OpenJDK 64-Bit Server VM, Java 17.0.11)
Type in expressions to have them evaluated.
Type :help for more information.
{"ts":"2024-09-06T02:42:48.551Z","level":"INFO","msg":"Found configuration file null","logger":"HiveConf"}
{"ts":"2024-09-06T02:42:48.597Z","level":"INFO","msg":"Running Spark version 4.0.0-preview1","logger":"SparkContext"}
{"ts":"2024-09-06T02:42:48.598Z","level":"INFO","msg":"OS info Mac OS X, 14.5, aarch64","logger":"SparkContext"}
{"ts":"2024-09-06T02:42:48.598Z","level":"INFO","msg":"Java version 17.0.11","logger":"SparkContext"}
{"ts":"2024-09-06T02:42:48.647Z","level":"WARN","msg":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable","logger":"NativeCodeLoader"}
{"ts":"2024-09-06T02:42:48.683Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-09-06T02:42:48.683Z","level":"INFO","msg":"No custom resources configured for spark.driver.","logger":"ResourceUtils"}
{"ts":"2024-09-06T02:42:48.683Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-09-06T02:42:48.684Z","level":"INFO","msg":"Submitted application: Spark shell","logger":"SparkContext"}
{"ts":"2024-09-06T02:42:48.697Z","level":"INFO","msg":"Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)","logger":"ResourceProfile"}
{"ts":"2024-09-06T02:42:48.698Z","level":"INFO","msg":"Limiting resource is cpu","logger":"ResourceProfile"}
{"ts":"2024-09-06T02:42:48.698Z","level":"INFO","msg":"Added ResourceProfile id: 0","logger":"ResourceProfileManager"}
{"ts":"2024-09-06T02:42:48.724Z","level":"INFO","msg":"Changing view acls to: ankit.gupta","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:48.724Z","level":"INFO","msg":"Changing modify acls to: ankit.gupta","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:48.725Z","level":"INFO","msg":"Changing view acls groups to: ","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:48.725Z","level":"INFO","msg":"Changing modify acls groups to: ","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:48.727Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: ankit.gupta; groups with view permissions: EMPTY; users with modify permissions: ankit.gupta; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:48.845Z","level":"INFO","msg":"Successfully started service 'sparkDriver' on port 58034.","logger":"Utils"}
{"ts":"2024-09-06T02:42:48.858Z","level":"INFO","msg":"Registering MapOutputTracker","logger":"SparkEnv"}
{"ts":"2024-09-06T02:42:48.862Z","level":"INFO","msg":"Registering BlockManagerMaster","logger":"SparkEnv"}
{"ts":"2024-09-06T02:42:48.870Z","level":"INFO","msg":"Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-09-06T02:42:48.871Z","level":"INFO","msg":"BlockManagerMasterEndpoint up","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-09-06T02:42:48.872Z","level":"INFO","msg":"Registering BlockManagerMasterHeartbeat","logger":"SparkEnv"}
{"ts":"2024-09-06T02:42:48.888Z","level":"INFO","msg":"Created local directory at /private/var/folders/8h/6blq52c15c303rydrrb15tkj20sffw/T/blockmgr-27c3b3f9-785e-45e2-a417-8f6bd0ff3a44","logger":"DiskBlockManager"}
{"ts":"2024-09-06T02:42:48.899Z","level":"INFO","msg":"Registering OutputCommitCoordinator","logger":"SparkEnv"}
{"ts":"2024-09-06T02:42:48.967Z","level":"INFO","msg":"Start Jetty 0.0.0.0:4040 for SparkUI","logger":"JettyUtils"}
{"ts":"2024-09-06T02:42:48.999Z","level":"INFO","msg":"Successfully started service 'SparkUI' on port 4040.","logger":"Utils"}
{"ts":"2024-09-06T02:42:49.026Z","level":"INFO","msg":"Changing view acls to: ankit.gupta","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:49.026Z","level":"INFO","msg":"Changing modify acls to: ankit.gupta","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:49.026Z","level":"INFO","msg":"Changing view acls groups to: ","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:49.026Z","level":"INFO","msg":"Changing modify acls groups to: ","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:49.026Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: ankit.gupta; groups with view permissions: EMPTY; users with modify permissions: ankit.gupta; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
{"ts":"2024-09-06T02:42:49.063Z","level":"INFO","msg":"Starting executor ID driver on host 192.168.28.3","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.063Z","level":"INFO","msg":"OS info Mac OS X, 14.5, aarch64","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.064Z","level":"INFO","msg":"Java version 17.0.11","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.067Z","level":"INFO","msg":"Starting executor with user classpath (userClassPathFirst = false): ''","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.067Z","level":"INFO","msg":"Using REPL class URI: spark://192.168.28.3:58034/classes","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.070Z","level":"INFO","msg":"Created or updated repl class loader org.apache.spark.executor.ExecutorClassLoader@41bc501b for default.","logger":"Executor"}
{"ts":"2024-09-06T02:42:49.079Z","level":"INFO","msg":"Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58036.","logger":"Utils"}
{"ts":"2024-09-06T02:42:49.080Z","level":"INFO","msg":"Server created on 192.168.28.3:58036","logger":"NettyBlockTransferService"}
{"ts":"2024-09-06T02:42:49.081Z","level":"INFO","msg":"Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy","logger":"BlockManager"}
{"ts":"2024-09-06T02:42:49.086Z","level":"INFO","msg":"Registering BlockManager BlockManagerId(driver, 192.168.28.3, 58036, None)","logger":"BlockManagerMaster"}
{"ts":"2024-09-06T02:42:49.089Z","level":"INFO","msg":"Registering block manager 192.168.28.3:58036 with 434.4 MiB RAM, BlockManagerId(driver, 192.168.28.3, 58036, None)","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-09-06T02:42:49.090Z","level":"INFO","msg":"Registered BlockManager BlockManagerId(driver, 192.168.28.3, 58036, None)","logger":"BlockManagerMaster"}
{"ts":"2024-09-06T02:42:49.091Z","level":"INFO","msg":"Initialized BlockManager: BlockManagerId(driver, 192.168.28.3, 58036, None)","logger":"BlockManager"}
Spark context Web UI available at http://192.168.28.3:4040
Spark context available as 'sc' (master = local, app id = local-1725590569043).
Spark session available as 'spark'.scala> :paste
// Entering paste mode (ctrl-D to finish)java.util.TimeZone.setDefault(java.util.TimeZone.getTimeZone("UTC"))
val df = Seq(java.sql.Timestamp.valueOf("0001-01-01 00:00:00"), java.sql.Timestamp.valueOf("1900-01-01 00:00:00"), java.sql.Timestamp.valueOf("1799-12-31 00:00:00"), java.sql.Timestamp.valueOf("1850-12-31 00:00:00"), new java.sql.Timestamp(0)).toDF("ts")
df.withColumn("ts_trans", from_utc_timestamp($"ts", "JST")).show
// Exiting paste mode... now interpreting.
warning: 1 deprecation (since 2.13.3); for details, enable `:setting -deprecation` or `:replay -deprecation`
{"ts":"2024-09-06T02:42:55.091Z","level":"INFO","msg":"Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.","logger":"SharedState"}
{"ts":"2024-09-06T02:42:55.096Z","level":"INFO","msg":"Warehouse path is 'file:/Users/ankit.gupta/Workspace/softwares/spark-4.0.0-preview1-bin-hadoop3/spark-warehouse'.","logger":"SharedState"}
{"ts":"2024-09-06T02:42:55.573Z","level":"INFO","msg":"Code generated in 110.8475 ms","logger":"CodeGenerator"}
{"ts":"2024-09-06T02:42:56.179Z","level":"INFO","msg":"Code generated in 5.344709 ms","logger":"CodeGenerator"}
{"ts":"2024-09-06T02:42:56.196Z","level":"INFO","msg":"Code generated in 7.283917 ms","logger":"CodeGenerator"}
+-------------------+-------------------+
|                 ts|           ts_trans|
+-------------------+-------------------+
|0001-01-01 00:00:00|0001-01-01 09:18:59|
|1900-01-01 00:00:00|1900-01-01 09:00:00|
|1799-12-31 00:00:00|1799-12-31 09:18:59|
|1850-12-31 00:00:00|1850-12-31 09:18:59|
|1970-01-01 00:00:00|1970-01-01 09:00:00|
+-------------------+-------------------+val df: org.apache.spark.sql.DataFrame = [ts: timestamp]
 {code}