av9ash/GitBugs huggingface dataset:
Bug: RocksDB consumes excessive disk space when many concurrent streaming queries are using dedup
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 04/Oct/23 13:44
Affects Version/s: 3.3.2

Our spark environment features a number of parallel structured streaming jobs, many of which use state store. Most use state store for dropDuplicates and work with a tiny amount of information, but a few have a substantially large state store requiring use of RocksDB. In such a configuration, spark allocates a minimum of {{spark.sql.shuffle.partitions * queryCount}} partitions, each of which pre-allocate about 74mb (observed on EMR/Hadoop) disk storage for RocksDB. This allocation is due to pre-allocation of log files space using [fallocate|https://github.com/facebook/rocksdb/blob/main/include/rocksdb/options.h#L871-L880], requiring users to either unnaturally reduce shuffle partitions, split running spark instances, or allocate a large amount of wasted storage.
Bug: Replace `Proxy.getProxyClass()` with `Proxy.newProxyInstance().getClass`
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 09/Oct/23 05:10
Affects Version/s: 4.0.0

{code:java}
 * @deprecated Proxy classes generated in a named module are encapsulated
 *      and not accessible to code outside its module.
 *      {@link Constructor#newInstance(Object...) Constructor.newInstance}
 *      will throw {@code IllegalAccessException} when it is called on
 *      an inaccessible proxy class.
 *      Use {@link #newProxyInstance(ClassLoader, Class[], InvocationHandler)}
 *      to create a proxy instance instead.
 *
 * @see <a href="#membership">Package and Module Membership of Proxy Class</a>
 * @revised 9
 */
@Deprecated
@CallerSensitive
public static Class<?> getProxyClass(ClassLoader loader,
                                     Class<?>... interfaces)
    throws IllegalArgumentException {code}
Bug: Fix the compilation warning related to other-pure-statement
Status: Resolved | Priority: Minor | Resolution: Fixed
Created: 11/Oct/23 03:32
Affects Version/s: 4.0.0

{code:java}
"-Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv",
"-Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv",
"-Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite:wv",
"-Wconf:cat=other-pure-statement&site=org.apache.spark.sql.streaming.sources.StreamingDataSourceV2Suite.testPositiveCase.\\$anonfun:wv", {code}
Bug: Error when creating UnionRDD of PairRDDs
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 21/May/20 20:08
Affects Version/s: 3.0.0

Union RDD of Pair RDD's seems to have issues

SparkSession available as 'spark'.

{code}
rdd1 = sc.parallelize([1,2,3,4,5])
rdd2 = sc.parallelize([6,7,8,9,10])
pairRDD1 = rdd1.zip(rdd2)
unionRDD1 = sc.union([pairRDD1, pairRDD1])
{code}

{code}
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/gs/spark/latest/python/pyspark/context.py", line 870,
in union jrdds[i] = rdds[i]._jrdd
File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 238, in _setitem_ File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 221,
in __set_item File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 332, in get_return_value py4j.protocol.Py4JError: An error occurred while calling None.None. Trace: py4j.Py4JException: Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD at py4j.commands.ArrayCommand.convertArgument(ArrayCommand.java:166) at py4j.commands.ArrayCommand.setArray(ArrayCommand.java:144) at py4j.commands.ArrayCommand.execute(ArrayCommand.java:97) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)
{code}


{code}
rdd3 = sc.parallelize([11,12,13,14,15])
pairRDD2 = rdd3.zip(rdd3)
unionRDD2 = sc.union([pairRDD1, pairRDD2])
{code}

{code}
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/gs/spark/latest/python/pyspark/context.py", line 870, in union jrdds[i] = rdds[i]._jrdd File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 238, in _setitem_ File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/java_collections.py", line 221, in __set_item File "/home/gs/spark/latest/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 332, in get_return_value py4j.protocol.Py4JError: An error occurred while calling None.None. Trace: py4j.Py4JException: Cannot convert org.apache.spark.api.java.JavaPairRDD to org.apache.spark.api.java.JavaRDD at py4j.commands.ArrayCommand.convertArgument(ArrayCommand.java:166) at py4j.commands.ArrayCommand.setArray(ArrayCommand.java:144) at py4j.commands.ArrayCommand.execute(ArrayCommand.java:97) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)
{code}

2.4.5 does not have this regression as below:

{code}
rdd4 = sc.parallelize(range(5))
pairRDD3 = rdd4.zip(rdd4)
unionRDD3 = sc.union([pairRDD1, pairRDD3])
unionRDD3.collect()
{code}

{code}
[(1, 6), (2, 7), (3, 8), (4, 9), (5, 10), (0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]
{code}
Bug: Sql Div operator can result in incorrect output for int_min
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 19/May/20 12:42
Affects Version/s: 3.0.0

Input  in csv : -2147483648,-1  --> (_c0, _c1)
{code}
val res = df.selectExpr("_c0 div _c1")
res.collect
res1: Array[org.apache.spark.sql.Row] = Array([-2147483648])
{code}
The result should be 2147483648 instead.
Bug: Changes made by SPARK-26985 break reading parquet files correctly in BigEndian architectures (AIX + LinuxPPC64)
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 13/May/20 13:04
Affects Version/s: 2.4.5

tTrying to upgrade to Apache Spark 2.4.5 in our IBM systems (AIX and PowerPC) so as to be able to read data stored in parquet format, we notice that values associated with DOUBLE and DECIMAL types are parsed in the wrong form.

According toe parquet documentation, they always opt to store the values using little-endian representation for values:
 [https://github.com/apache/parquet-format/blob/master/Encodings.md]
{noformat}
The plain encoding is used whenever a more efficient encoding can not be used. It
stores the data in the following format:

BOOLEAN: Bit Packed, LSB first
INT32: 4 bytes little endian
INT64: 8 bytes little endian
INT96: 12 bytes little endian (deprecated)
FLOAT: 4 bytes IEEE little endian
DOUBLE: 8 bytes IEEE little endian
BYTE_ARRAY: length in 4 bytes little endian followed by the bytes contained in the array
FIXED_LEN_BYTE_ARRAY: the bytes contained in the array

For native types, this outputs the data as little endian. Floating
point types are encoded in IEEE.
For the byte array type, it encodes the length as a 4 byte little
endian, followed by the bytes.{noformat}
Bug: Fix silent data change for datetime formatting 
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 29/May/20 07:57
Affects Version/s: 3.0.0

{code:java}
spark-sql> select from_unixtime(1, 'yyyyyyyyyyy-MM-dd');
NULL
spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
spark.sql.legacy.timeParserPolicy	legacy
spark-sql> select from_unixtime(1, 'yyyyyyyyyyy-MM-dd');
00000001970-01-01
spark-sql>
{code}
For patterns that support `SignStyle.EXCEEDS_PAD`, e.g. `y..y`(len >=4), when using the `NumberPrinterParser` to format it

{code:java}
switch (signStyle) {
  case EXCEEDS_PAD:
    if (minWidth < 19 && value >= EXCEED_POINTS[minWidth]) {
      buf.append(decimalStyle.getPositiveSign());
    }
    break;
   
           ....
{code}
the `minWidth` == `len(y..y)`
the `EXCEED_POINTS` is 

{code:java}
/**
         * Array of 10 to the power of n.
         */
        static final long[] EXCEED_POINTS = new long[] {
            0L,
            10L,
            100L,
            1000L,
            10000L,
            100000L,
            1000000L,
            10000000L,
            100000000L,
            1000000000L,
            10000000000L,
        };
{code}

So when the `len(y..y)` is greater than 10, ` ArrayIndexOutOfBoundsException` will be raised.

 And at the caller side, for `from_unixtime`, the exception will be suppressed and silent data change occurs. for `date_format`, the `ArrayIndexOutOfBoundsException` will continue.
Bug: Wrong results for week-based-year 
Status: Resolved | Priority: Blocker | Resolution: Won't Fix
Created: 29/May/20 08:42
Affects Version/s: 3.0.0

{code:sql}
spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
spark.sql.legacy.timeParserPolicy	exception
spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd')
         > ;
== Physical Plan ==
*(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#37]
+- *(1) Scan OneRowRelation[]


spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
spark.sql.legacy.timeParserPolicy	legacy
spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
== Physical Plan ==
*(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#53]
+- *(1) Scan OneRowRelation[]


spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
spark.sql.legacy.timeParserPolicy	exception
spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
== Physical Plan ==
*(1) Project [-28800000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#69]
+- *(1) Scan OneRowRelation[]


spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
1970-01-01 00:00:00
spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
spark.sql.legacy.timeParserPolicy	legacy
spark-sql> explain select to_timestamp('1969-01-01', 'YYYY-MM-dd');
== Physical Plan ==
*(1) Project [-31824000000000 AS to_timestamp(1969-01-01, YYYY-MM-dd)#87]
+- *(1) Scan OneRowRelation[]


spark-sql> select to_timestamp('1969-01-01', 'YYYY-MM-dd');
1968-12-29 00:00:00
spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
1969-01-01 00:00:00
spark-sql> set spark.sql.legacy.timeParserPolicy=exception;
spark.sql.legacy.timeParserPolicy	exception
spark-sql> select to_timestamp('1969-01-01', 'yyyy-MM-dd');
1969-01-01 00:00:00
{code}


{code:java}

== Physical Plan ==
*(1) Project [-28800000000 AS to_timestamp(2000-01-01, YYYY-MM-dd)#73]
+- *(1) Scan OneRowRelation[]

spark-sql> select to_timestamp('2000-01-01', 'YYYY-MM-dd');
1970-01-01 00:00:00
{code}


Seems that `-28800000000` is static
Bug: Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 13/May/20 20:04
Affects Version/s: 3.0.0

According to the latest status of discussion in the dev@ mailing list, [[DISCUSS] Resolve ambiguous parser rule between two "create table"s|http://apache-spark-developers-list.1001551.n3.nabble.com/DISCUSS-Resolve-ambiguous-parser-rule-between-two-quot-create-table-quot-s-td29051i20.html], we'd want to revert the change of SPARK-30098 first to unblock Spark 3.0.0.

This issue tracks the effort of revert.
Bug: Use GitHub instead of GitBox in release script
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 12/May/20 16:16
Affects Version/s: 3.0.0

Currently, Spark Packaing jobs are broken.

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/2906/console

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-3.0-maven-snapshots/105/console

 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-branch-2.4-maven-snapshots/439/console
{code:java}
fatal: unable to access 'https://gitbox.apache.org/repos/asf/spark.git/': Failed to connect to gitbox.apache.org port 443: Connection timed out{code}
Bug: Saving and loading HashingTF leads to hash function changed
Status: Resolved | Priority: Blocker | Resolution: Fixed
Created: 09/May/20 01:50
Affects Version/s: 3.0.0

If we use spark 2.x save HashingTF, and then use spark 3.0 load it, and then use spark 3.0 to save it again, and then use spark 3.0 to load it again, the hash function will be changed.

This bug is hard to debug, we need to fix it.
Bug: Redesign PySpark documentation
Status: Resolved | Priority: Critical | Resolution: Done
Created: 28/May/20 06:10
Affects Version/s: 3.1.0

Currently, PySpark documentation (https://spark.apache.org/docs/latest/api/python/index.html) is pretty much poorly written compared to other projects.
See, for example, see Koalas https://koalas.readthedocs.io/en/latest/ as an example.

PySpark is being more and more important in Spark, and we should improve this documentation so people can easily follow.

Reference: 
- https://koalas.readthedocs.io/en/latest/
- https://pandas.pydata.org/docs/
Bug: Investigate AmpLab Jenkins server network issue
Status: Closed | Priority: Critical | Resolution: Fixed
Created: 12/May/20 22:56
Affects Version/s: 2.4.6

Given the series of failures in Spark packaging Jenkins job, it seems that there is a network issue in AmbLab Jenkins cluster.

- https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/

- The node failed to talk to GitBox. (SPARK-31687) -> GitHub is okay.
- The node failed to download the maven mirror. (SPARK-31691) -> The primary host is okay.
- The node failed to communicate repository.apache.org. (Current master branch Jenkins job failure)
{code}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0-M1:deploy (default-deploy) on project spark-parent_2.12: ArtifactDeployerException: Failed to retrieve remote metadata org.apache.spark:spark-parent_2.12:3.1.0-SNAPSHOT/maven-metadata.xml: Could not transfer metadata org.apache.spark:spark-parent_2.12:3.1.0-SNAPSHOT/maven-metadata.xml from/to apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots): Transfer failed for https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-parent_2.12/3.1.0-SNAPSHOT/maven-metadata.xml: Connect to repository.apache.org:443 [repository.apache.org/207.244.88.140] failed: Connection timed out (Connection timed out) -> [Help 1]
{code}
Bug: Redesign PySpark documentation
Status: Resolved | Priority: Critical | Resolution: Duplicate
Created: 28/May/20 06:13
Affects Version/s: 3.1.0

Currently, PySpark documentation (https://spark.apache.org/docs/latest/api/python/index.html) is pretty much poorly written compared to other projects.
See, for example, see pandas https://pandas.pydata.org/docs/

PySpark is being more and more important in Spark, and we should improve this documentation so people can easily follow.

Reference: 
- https://pandas.pydata.org/docs/
- https://koalas.readthedocs.io/en/latest/
Bug: Enable Hive related test cases of SparkR on Windows
Status: Open | Priority: Major | Resolution: 
Created: 18/May/20 01:28
Affects Version/s: 3.1.0

Hive related tests look being skipped in AppVeyor:

{code}
test_sparkSQL.R:307: skip: create DataFrame from RDD
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:1341: skip: test HiveContext
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:2813: skip: read/write ORC files
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:2834: skip: read/write ORC files - compression option
Reason: Hive is not build with SparkSQL, skipped
test_sparkSQL.R:3727: skip: enableHiveSupport on SparkSession
Reason: Hive is not build with SparkSQL, skipped
{code}
Bug: Incorrect distribution with repartitionByRange and repartition column expression
Status: Open | Priority: Major | Resolution: 
Created: 22/May/20 08:31
Affects Version/s: 2.3.2

Both repartitionByRange and  repartition(<num>, <column>)  resulting in wrong distribution within the resulting partition.  

 

In the Range partition one of the partition has 2x volume and last one with zero.  In repartition this is more problematic with some partition with 4x, 2x the avg and many partitions with zero volume.  

 

This distribution imbalance can cause performance problem in a concurrent environment.

Details from testing in 3 different versions.
|Verion 2.3.2|Version 2.4.5|Versoin 3.0 Preview2|
|Spark Version 2.3.2.3.1.4.0-315|Spark Version 2.4.5|Spark Version 3.0.0-preview2|
|Default Partition Length:2|Default Partition Length:2|Default Partition Length:80|
|Default Partition getNumPartitions:2|Default Partition getNumPartitions:2|Default Partition getNumPartitions:80|
|Default Partition groupBy spark_partition_id:200|Default Partition groupBy spark_partition_id:200|Default Partition groupBy spark_partition_id:200|
|repartitionByRange Length:24|repartitionByRange Length:24|repartitionByRange Length:24|
|repartitionByRange getNumPartitions:24|repartitionByRange getNumPartitions:24|repartitionByRange getNumPartitions:24|
|repartitionByRange groupBy spark_partition_id:200|repartitionByRange groupBy spark_partition_id:200|repartitionByRange groupBy spark_partition_id:200|
|repartitionByRange: List(83, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41, 41, 41, 0)|repartitionByRange: List(83, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41, 41, 41, 0)|repartitionByRange: List(83, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 41, 41, 41, 41, 41, 41, 0)|
|repartition by column expr Length:24|repartition by column expr Length:24|repartition by column expr Length:24|
|repartition by column expr getNumPartitions:24|repartition by column expr getNumPartitions:24|repartition by column expr getNumPartitions:24|
|repartition by column expr groupBy spark_partitoin_id:200|repartition by column expr groupBy spark_partitoin_id:200|repartition by column expr groupBy spark_partitoin_id:200|
|repartition by column expr:List(83, 42, 0, 84, 0, 42, 125, 0, 42, 84, 0, 42, 0, 82, 0, 124, 42, 83, 84, 42, 0, 0, 0, 0)|repartition by column expr:List(83, 42, 0, 84, 0, 42, 125, 0, 42, 84, 0, 42, 0, 82, 0, 124, 42, 83, 84, 42, 0, 0, 0, 0)|repartition by column expr:List(83, 42, 0, 84, 0, 42, 125, 0, 42, 84, 0, 42, 0, 82, 0, 124, 42, 83, 84, 42, 0, 0, 0, 0)|
Bug: input_file_name() gives wrong value following Python UDF usage
Status: Open | Priority: Major | Resolution: 
Created: 27/May/20 12:28
Affects Version/s: 2.4.5

I'm using PySpark for Spark 3.0.0 RC1 with Python 3.6.8.

The following commands demonstrate that the input_file_name() function sometimes returns the wrong filename following usage of a Python UDF:

{code}
$ for i in `seq 5`; do echo $i > /tmp/test-file-$i; done

$ pyspark

>>> import pyspark.sql.functions as F

>>> spark.readStream.text('file:///tmp/test-file-*', wholetext=True).withColumn('file1', F.input_file_name()).withColumn('udf', F.udf(lambda x:x)('value')).withColumn('file2', F.input_file_name()).writeStream.trigger(once=True).foreachBatch(lambda df,_: df.select('file1','file2').show(truncate=False, vertical=True)).start().awaitTermination()
{code}

A few notes about this bug:
 * It happens with many different files, so it's not related to the file contents
 * It also happens loading files from HDFS, so storage location is not a factor
 * It also happens using .csv() to read the files instead of .text(), so input format is not a factor
 * I have not been able to cause the error without using readStream, so it seems to be related to streaming
 * The bug also happens using spark-submit to send a job to my cluster
 * I haven't tested an older version, but it's possible that Spark pulls 24958 and 25321([https://github.com/apache/spark/pull/24958], [https://github.com/apache/spark/pull/25321]) to fix issue 28153 (https://issues.apache.org/jira/browse/SPARK-28153) introduced this bug?
Bug: Cannot map hostPath volumes to container
Status: Closed | Priority: Major | Resolution: Not A Problem
Created: 08/May/20 20:49
Affects Version/s: 2.4.5

I'm trying to mount additional hostPath directories as seen in a couple of places:

[https://aws.amazon.com/blogs/containers/optimizing-spark-performance-on-kubernetes/]

[https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#using-volume-for-scratch-space]

[https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes]

 

However, whenever I try to submit my job, I run into this error:


{code:java}
Uncaught exception in thread kubernetes-executor-snapshots-subscribers-1 │
 io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default.svc/api/v1/namespaces/my-spark-ns/pods. Message: Pod "spark-pi-1588970477877-exec-1" is invalid: spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be unique. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, message=Invalid value: "/tmp1": must be unique, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, name=spark-pi-1588970477877-exec-1, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "spark-pi-1588970477877-exec-1" is invalid: spec.containers[0].volumeMounts[1].mountPath: Invalid value: "/tmp1": must be unique, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).{code}
 

This is my spark-submit command (note: I've used my own build of spark for kubernetes as well as a few other images that I've seen floating around (such as this one seedjeffwan/spark:v2.4.5) and they all have this same issue):
{code:java}
bin/spark-submit \
 --master k8s://https://my-k8s-server:443 \
 --deploy-mode cluster \
 --name spark-pi \
 --class org.apache.spark.examples.SparkPi \
 --conf spark.executor.instances=2 \
 --conf spark.kubernetes.container.image=my-spark-image:my-tag \
 --conf spark.kubernetes.driver.pod.name=sparkpi-test-driver \
 --conf spark.kubernetes.namespace=my-spark-ns \
 --conf spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.mount.path=/tmp1 \
 --conf spark.kubernetes.executor.volumes.hostPath.spark-local-dir-2.options.path=/tmp1 \
 --conf spark.local.dir="/tmp1" \
 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark
 local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 20000{code}
Any ideas on what's causing this?
Bug: Spark Sql Function year does not extract year from date/timestamp
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 30/May/20 16:11
Affects Version/s: 2.4.5

There is a Spark SQL function

org.apache.spark.sql.functions.year which fails in below case


 
{code:java}
// Code to extract year from Timestamp
val df = Seq(
  ("1300-01-03 00:00:00")
).toDF("date_val")
  .withColumn("date_val_ts", to_timestamp(col("date_val")))
  .withColumn("year_val", year(to_timestamp(col("date_val"))))

df.show()
//Output of the above code
+-------------------+-------------------+--------+
|           date_val|        date_val_ts|year_val|
+-------------------+-------------------+--------+
|1300-01-03 00:00:00|1300-01-03 00:00:00|    1299|
+-------------------+-------------------+--------+

{code}
 

The above code works perfectly for all the years greater than 1300