av9ash/GitBugs huggingface dataset:
Bug: dataframe API filter criteria throwing ParseException when reading a JDBC column name with special characters
Status: Resolved | Priority: Minor | Resolution: Cannot Reproduce
Created: 26/Jan/23 02:29
Affects Version/s: 3.3.0

*On Spark 3.3.0,* when reading from a JDBC table(used SQLite to repro) using spark.read.jdbc command with sqlite-jdbc:3.34.0.jar on a table and column name containing special characters. Dataframe API filter criteria fails with parse Exception 

*[#Script:]*
{code:java}
from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Databricks Support") \
    .config("spark.jars.packages", "org.xerial:sqlite-jdbc:3.34.0") \
    .getOrCreate()

columns = ["id", "/abc/column", "value"]
data = [(1, 'A', 100), (2, 'B', 200), (3, 'B', 300)]

rdd = spark.sparkContext.parallelize(data)
df = spark.createDataFrame(rdd).toDF(*columns)

options = {"url": "jdbc:sqlite:/<local-path>/spark-3.3.1-bin-hadoop3/jars/test.db", "dbtable": '"/abc/table"', "driver": "org.sqlite.JDBC"}

df.coalesce(1).write.format("jdbc").options(**options).mode("append").save()

df_1 = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlite:/<local-path>/spark-3.3.1-bin-hadoop3/jars/test.db") \
    .option("dbtable", '"/abc/table"') \
    .option("driver", "org.sqlite.JDBC") \
    .load()

df_2 = df_1.filter("`/abc/column` = 'B'")

df_2.show() {code}
Error:
{code:java}
``` Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/opt/homebrew/Cellar/apache-spark/3.3.1/libexec/python/pyspark/sql/dataframe.py", line 606, in show
  print(self._jdf.showString(n, 20, vertical))
 File "/opt/homebrew/Cellar/apache-spark/3.3.1/libexec/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
 File "/opt/homebrew/Cellar/apache-spark/3.3.1/libexec/python/pyspark/sql/utils.py", line 196, in deco
  raise converted from None
pyspark.sql.utils.ParseException: 
Syntax error at or near '/': extra input '/'(line 1, pos 0)

== SQL ==
/abc/column
^^^```  {code}
However, when using Spark 3.2.1, we are able to successfully apply dataframe.filter option
{code:java}
>>> df_2.show()
+---+-----------+-----+
| id|/abc/column|value|
+---+-----------+-----+
|  2|          B|  200|
|  3|          B|  300|
+---+-----------+-----+ {code}
*Repro steps:*
 # Download [Spark 3.2.1 in local |https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz]
 # Download and Copy the sqlite-jdbc:3.34.0.jar into the jar folder present in the local spark download folder
 # Run the above [#script] by providing the jar path 
 # This will create a */abc/table* with column */abc/column*  and returns result when applying filter criteria
 # Download spark ** [3.3.0 in local|https://www.apache.org/dyn/closer.lua/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz]
 # Repeat #2, #3 
 # Fails with parse exception. 

could you please let us know how we can filter on the special characters column or escape them on spark version 3.3.0?
Bug: Missing typing for pandas_udf
Status: In Progress | Priority: Minor | Resolution: 
Created: 30/Jan/23 05:45
Affects Version/s: 3.3.0

The typing stub {{site-packages/pyspark/sql/pandas/functions.pyi}} has a list of possible signatures of {{pandas_udf}}. It is missing a case

{code:python}
import pyspark.sql.functions as F

# PySpark3's typing stub error
@F.pandas_udf(
    returnType=LongType()
)
def my_udf(dummy_col: pd.Series) -> pd.Series:
    ...
{code}

The stub defined {{pandas_udf(f, returnType, functionType)}} but not {{pandas_udf(f, returnType)}}. The official documentation recommends using a return type hint instead of {{returnType}}.
 
{code:python}
# defined
@overload
def pandas_udf( f: PandasScalarToScalarFunction, returnType: Union[AtomicDataTypeOrString, ArrayType], functionType: PandasScalarUDFType, ) -> UserDefinedFunctionLike: ...
 
# not defined
@overload
def pandas_udf( f: PandasScalarToScalarFunction, returnType: Union[AtomicDataTypeOrString, ArrayType]) -> UserDefinedFunctionLike: ...
{code}
Bug: change binary to unsupported dataType in csv format
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 30/Jan/23 09:19
Affects Version/s: 2.4.8

When a binary colunm is written into csv files, actual content of this colunm is {*}object.toString(){*}, which is meaningless.
{code:java}
val df = Seq(Array[Byte](1,2)).toDF
df.write.csv("/Users/guowei/Desktop/binary_csv")
{code}
The csv file's content is as follows:

!image-2023-01-30-17-21-09-212.png|width=141,height=29!

Meanwhile, if a binary colunm saved as table with csv fileformat, the table can't be read back successfully.
{code:java}
val df = Seq((1, Array[Byte](1,2))).toDF
df.write.format("csv").saveAsTable("binaryDataTable")spark.sql("select * from binaryDataTable").show()
{code}
!https://rte.weiyun.baidu.com/wiki/attach/image/api/imageDownloadAddress?attachId=82da0afc444c41bdaac34418a1c89963&docGuid=Eiscz4oMI45Sfp&sign=eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwiYXBwSWQiOjEsInVpZCI6IjgtVWkzU0lMY2wiLCJkb2NJZCI6IkVpc2N6NG9NSTQ1U2ZwIn0..z1O-00hE1tTua9co.RmL0GxEQyNVQbIMYOvyAmQY18NMCxHdGdEPtulFiV3BuqsVlJODgA9-xFY9H9yer_Ckpbt4aG2ZrqgohIq43_ywzj-8u8SKKZnnzm7Dt-EhQBwrA7EhwUveE4-MRcAmsgqRKneN0gUJIu78ogR-M5-GAYqiyd-C-PH0LTaHDhNBWFBkF01kVOLJ18c2VTT6_lbc9j9Drmxj56ouymFgfhdUtpA.cTYqsEvvnKDcIPiah99f_A!

So I think it' better to change binary to unsupported dataType in csv format, both for datasource v1(CSVFileFormat) and v2(CSVTable).
Bug: Support udf luhn_check 
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 25/Jan/23 19:31
Affects Version/s: 3.4.0

Support UDF to check if a given number string is a valid Luhn number. It shall return true if the number string is a valid Luhn number, and false otherwise.

This checksum function is widely applied to credit card numbers and government identification numbers to distinguish valid numbers from mistyped, incorrect numbers

Ref : [trino|https://trino.io/docs/current/functions/string.html]
         [Algorithm|https://en.wikipedia.org/wiki/Luhn_algorithm]
Bug: Remove duplicate branches in CASE_WHEN and COALESCE function
Status: In Progress | Priority: Minor | Resolution: 
Created: 28/Jan/23 08:16
Affects Version/s: 3.4.0

We can remove the duplicate branches in CASE_WHEN and COALESCE function.

For example:
{code:sql}
SELECT
  CASE WHEN id = 200 THEN 1
       WHEN id = 200 THEN 2 -- same condition, can remove
       WHEN id = 200 THEN 3 -- same condition, can remove
       ELSE 0 END as c1,
  coalesce(
    id + 1,
    id + 1, -- same expression, can remove
    id) as c2
FROM t1
{code}
Bug: Change the default value of argument of Mask udf from -1 to NULL
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 15/Jan/23 04:47
Affects Version/s: 3.4.0

In the udf 'mask', using -1 as ignore parameter in String type argument is not a standard way, hence, it is better to change the value of ignore argument from -1 to NULL.

Note: SPARK-40687 , has recently implemented udf *mask* , which uses -1 as the default argument to ignore the masking option, As no Spark version release has occurred since then, this new change will not cause backward compatibility issues
Bug: Add null check before `ContinuousWriteRDD#compute` method close dataWriter
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 19/Jan/23 08:17
Affects Version/s: 3.4.0

Run 

 

mvn clean test -pl sql/core -Dtest=none -DwildcardSuites=org.apache.spark.sql.streaming.continuous.ContinuousSuite -am

 

NPE exists in the test log

 
{code:java}
- repeatedly restart
...
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 is aborting.
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 aborted.
16:07:39.892 WARN org.apache.spark.util.Utils: Suppressing exception in finally: null
java.lang.NullPointerException
    at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$7(ContinuousWriteRDD.scala:91)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1558)
    at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.compute(ContinuousWriteRDD.scala:91)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
    at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
    at org.apache.spark.scheduler.Task.run(Task.scala:139)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1502)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750) {code}
 

 

The test did not fail because Utils.tryWithSafeFinallyAndFailureCallbacks function suppressed Exception in finally block
Bug: [Pyspark] Hide parameters when re-printing user provided remote URL in REPL
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 18/Jan/23 10:16
Affects Version/s: 3.4.0

The Spark Connect client is initialized in the PySpark REPL by using the {{--remote}} option. The option takes a Spark Connect endpoint URL. 

 

The URL may contain auth tokens as URL parameters or query parameters.



Hide these values when the URL is re-printed as part of the REPL start-up.
Bug: SPJ: Remove Option in KeyGroupedPartitioning#partitionValues
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 13/Jan/23 00:24
Affects Version/s: 3.3.1

Currently {{KeyGroupedPartitioning#partitionValuesOpt}} is an {{{}Option[Seq[InternalRow]]{}}}. This is unnecessary since it is always set. This propose to just replace it with {{Seq[InternalRow]}}
Bug: Throw ExecutorDeadException in fetchBlocks when executor dead
Status: Open | Priority: Minor | Resolution: 
Created: 17/Jan/23 19:12
Affects Version/s: 3.3.1

When fetchBlocks failed due to IOException, ExecutorDeadException will be thrown when executor is dead.

There're other cases that executor dead will cause TimeoutException or other Exceptions.
{code:java}
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Waited 30000 milliseconds (plus 143334 nanoseconds delay) for SettableFuture@624de392[status=PENDING]
    at org.sparkproject.guava.base.Throwables.propagate(Throwables.java:243)
    at org.apache.spark.network.client.TransportClient.sendRpcSync(TransportClient.java:293)
    at org.apache.spark.network.crypto.AuthClientBootstrap.doSparkAuth(AuthClientBootstrap.java:113)
    at org.apache.spark.network.crypto.AuthClientBootstrap.doBootstrap(AuthClientBootstrap.java:80)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:300)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:218)
    at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(NettyBlockTransferService.scala:126)
    at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:154)
    at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:184)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) {code}
Bug: Running python3 setup.py sdist on windows reports a permission error
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 16/Jan/23 08:02
Affects Version/s: 3.4.0

My system version is windows 10, and I can run setup.py with administrator permissions, so there will be no error. However, it may be troublesome for us to upgrade permissions with Windows Server, so we need to modify the code of setup.py to ensure no error. To avoid the hassle of compiling for the user, I suggest modifying the following code to enable the out-of-the-box effect
{code:python}
def _supports_symlinks():
    """Check if the system supports symlinks (e.g. *nix) or not."""
    return getattr(os, "symlink", None) is not None and ctypes.windll.shell32.IsUserAnAdmin() != 0 if sys.platform == "win32" else True
{code}
Bug: Clean up remove methods that do not need override
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 12/Jan/23 13:24
Affects Version/s: 3.4.0

Java 8 began to provide the default remove method implementation for the `java.util.Iterator` interface.


https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Iterator.java#L92-L94
Bug: SparkR datetime column compare with Sys.time() throws error in R (>= 4.2.0)
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 08/Jan/23 02:37
Affects Version/s: 3.3.0

Base R 4.2.0 introduced a change ([[Rd] R 4.2.0 is released|https://stat.ethz.ch/pipermail/r-announce/2022/000683.html]), "{{{}Calling if() or while() with a condition of length greater than one gives an error rather than a warning.{}}}"

The below code is a reproducible example of the issue. If it is executed in R >=4.2.0 then it will generate an error, or else just a warning message. `{{{}Sys.time()`{}}} is a multi-class object in R, and throughout the Spark R repository '{{{}if{}}}' statement is used as: `{{{}if(class(x) == "Column"){}}}` - this causes error in the latest R version >= 4.2.0. Note that R allows an object to have multiple '{{{}class{}}}' names as a character vector ([R: Object Classes|https://stat.ethz.ch/R-manual/R-devel/library/base/html/class.html]); hence this type of check itself was not a good idea in the first place.

The below chunks are executed on R version 4.1.3.
{code:java}
{
 SparkR::sparkR.session()
 t <- Sys.time()
 sdf <- SparkR::createDataFrame(data.frame(x = t + c(-1, 1, -1, 1, -1)))
 SparkR::collect(SparkR::filter(sdf, SparkR::column('x') > t))
}
#> Warning in if (class(e2) == 'Column') {: the condition has length > 1 
#> and only the first element will be used
#> x
#> 1 2023-01-07 20:40:20
#> 2 2023-01-07 20:40:20 

{code}
 

 
{code:java}
{
 Sys.setenv(`_R_CHECK_LENGTH_1_CONDITION_` = "true")
 SparkR::sparkR.session()
 t <- Sys.time()
 sdf <- SparkR::createDataFrame(data.frame(x = t + c(-1, 1, -1, 1, -1)))
 SparkR::collect(SparkR::filter(sdf, SparkR::column('x') > t))
}
#> Error in h(simpleError(msg, call)): error in evaluating the argument 'x' 
#> in selecting a method for function 'collect': error in evaluating the 
#> argument 'condition' in selecting a method for function 'filter': the
#> condition has length > 1 {code}
 

Similar issue is noted for these SparkR functions where {{Sys.time()}} type of multi-class data might be used: {{lit, fillna, when, otherwise, contains, ifelse }}

The suggested change is to add the `{{{}all{}}}` function (or `{{{}any{}}}`, as appropriate) while doing the check of whether `{{{}class(.){}}}` is `{{{}Column{}}}` or not: `{{{}if(all(class(.) == "Column")){}}}`. Or, better to use `{{{}base::inherits{}}}` for this check as `{{{}if(inherits(., "Column")){}}}`.
Bug: Use Map in place of SortedMap for ErrorClassesJsonReader
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 03/Jan/23 02:53
Affects Version/s: 3.2.3

The use of SortedMap in ErrorClassesJsonReader was mostly for making tests easier to write.

This PR replaces SortedMap with Map since SortedMap is slower compared to Map.
Bug: Make AvroScanBuilder and JsonScanBuilder case classes
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 03/Jan/23 15:21
Affects Version/s: 3.3.1

Most of the existing `ScanBuilder`s inheriting from `FileScanBuilder` are case classes which is very nice and useful. However the Json and the Avro ones aren't, which makes it quite inconvenient when working with those. I don't see any particular reason why these two would not be case classes as well given they are very similar to the other ones and very simple and stateless
Bug: Make v2 ScanBuilders' build() return typed scan
Status: Resolved | Priority: Trivial | Resolution: Fixed
Created: 03/Jan/23 16:02
Affects Version/s: 3.3.1

The `ScanBuilder` interface has the `build()` method to return `Scan` objects. All the different implementations of it return scans that are of the type of the builder itself. Eg `ParquetScanBuilder` will return a `ParquetScan`, `TextScanBuilder` a `TextScan` etc. However, in the method overrides declaration, the return type is not changed to the more specific type but left as a generic `Scan`. This is a bit problematic when one has to work with scan object because a manual cast is required, even if we know for a fact that a `ParquetScanBuilder`'s `build()` method will return a `ParquetScan`. For ease of developement (and for stricter correctness checks, as we wouldn't want a `ParquetScanBuilder` to return a non-parquet scan I would assume), it would be very nice if the `build()` methods of each implementation of `ScanBuilder`s returned a more strictly typed object instead of the generic `Scan` object
Bug: IDEA开发连接远程Spark集群报错
Status: Resolved | Priority: Blocker | Resolution: Invalid
Created: 15/Mar/20 04:12
Affects Version/s: 1.5.1

IDEA开发Spark程序连接远程Spark集群报异常如下：

WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@192.168.159.129:7077] has failed, address is now gated for [5000] ms. Reason: [Disassociated] 
ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main]
Bug: Support setuptools 46.1.0+ in PySpark packaging
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Mar/20 07:35
Affects Version/s: 2.4.5

PIP packaging test started to fail (see https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120218/testReport/) as of  setuptools 46.1.0 release.

In https://github.com/pypa/setuptools/issues/1424, they decided to don't keep the modes in {{package_data}}. In PySpark pip installation, we keep the executable scripts in {{package_data}} https://github.com/apache/spark/blob/master/python/setup.py#L199-L200, and expose their symbolic links as executable scripts.

So, the symbolic links (or copied scripts) executes the scripts copied from {{package_data}}, which didn't keep the modes:

{code}
/tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: Permission denied
/tmp/tmp.UmkEGNFdKF/3.6/bin/spark-submit: line 27: exec: /tmp/tmp.UmkEGNFdKF/3.6/lib/python3.6/site-packages/pyspark/bin/spark-class: cannot execute: Permission denied
{code}

The current issue is being tracked at https://github.com/pypa/setuptools/issues/2041
Bug: Amend Spark's Semantic Versioning Policy
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/Mar/20 04:22
Affects Version/s: 3.0.0

This issue tracks all the activity for the following discussion and vote.
- https://lists.apache.org/thread.html/r82f99ad8c2798629eed66d65f2cddc1ed196dddf82e8e9370f3b7d32%40%3Cdev.spark.apache.org%3E

- https://lists.apache.org/thread.html/r683dbb0481adb1944461b6e1a60aafc44a66423c6e9fa2bab24a07db%40%3Cdev.spark.apache.org%3E
Bug: Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
Status: Closed | Priority: Blocker | Resolution: Won't Do
Created: 12/Mar/20 16:56
Affects Version/s: 3.0.0

We need to consider the behavior change of SPARK-30098 .
This is a placeholder to keep the discussion and the final decision.

`CREATE TABLE` syntax changes its behavior silently.

The following is one example of the breaking the existing user data pipelines.
*Apache Spark 2.4.5*
{code}
spark-sql> CREATE TABLE t(a STRING);

spark-sql> LOAD DATA INPATH '/usr/local/spark/README.md' INTO TABLE t;

spark-sql> SELECT * FROM t LIMIT 1;
# Apache Spark
Time taken: 2.05 seconds, Fetched 1 row(s)
{code}

{code}
spark-sql> CREATE TABLE t(a CHAR(3));

spark-sql> INSERT INTO TABLE t SELECT 'a ';

spark-sql> SELECT a, length(a) FROM t;
a  	3
{code}

*Apache Spark 3.0.0-preview2*
{code}
spark-sql> CREATE TABLE t(a STRING);

spark-sql> LOAD DATA INPATH '/usr/local/spark/README.md' INTO TABLE t;
Error in query: LOAD DATA is not supported for datasource tables: `default`.`t`;
{code}

{code}
spark-sql> CREATE TABLE t(a CHAR(3));

spark-sql> INSERT INTO TABLE t SELECT 'a ';

spark-sql> SELECT a, length(a) FROM t;
a 	2
{code}
Bug: ResetCommand should not wipe out all configs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Mar/20 13:12
Affects Version/s: 2.0.2

Currently, ResetCommand clear all configurations, including sql configs, static sql configs and spark context level configs.
for example:
```
spark-sql> set xyz=abc;
xyz	abc
spark-sql> set;
spark.app.id	local-1585055396930
spark.app.name	SparkSQL::10.242.189.214
spark.driver.host	10.242.189.214
spark.driver.port	65094
spark.executor.id	driver
spark.jars
spark.master	local[*]
spark.sql.catalogImplementation	hive
spark.sql.hive.version	1.2.1
spark.submit.deployMode	client
xyz	abc
spark-sql> reset;
spark-sql> set;
spark-sql> set spark.sql.hive.version;
spark.sql.hive.version	1.2.1
spark-sql> set spark.app.id;
spark.app.id	<undefined>
```
Bug: Incompatible ORC dates with Spark 2.4
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 24/Mar/20 20:50
Affects Version/s: 3.0.0

Using Spark 2.4.5, write pre-1582 date to ORC file and then read it:
{noformat}
$ export TZ=UTC
$ bin/spark-shell --conf spark.sql.session.timeZone=UTC
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5-SNAPSHOT
      /_/
         
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.

scala> sql("select cast('1200-01-01' as date) dt").write.mode("overwrite").orc("/tmp/datefile")

scala> spark.read.orc("/tmp/datefile").show
+----------+
|dt        |
+----------+
|1200-01-01|
+----------+

scala> :quit
{noformat}
Using Spark 3.0 (branch-3.0 at commit a934142f24), read the same file:
{noformat}
$ export TZ=UTC
$ bin/spark-shell --conf spark.sql.session.timeZone=UTC
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0-SNAPSHOT
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.orc("/tmp/datefile").show
+----------+
|dt        |
+----------+
|1200-01-08|
+----------+

scala>
{noformat}
Dates are off.

Timestamps, on the other hand, appear to work as expected.