av9ash/GitBugs huggingface dataset:
Bug: Investigate the difference between JDK and Spark's time zone offset calculation
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 15/Apr/20 06:47
Affects Version/s: 2.4.5

Spark 2.4 calculates time zone offsets from wall clock timestamp using `DateTimeUtils.getOffsetFromLocalMillis()` (see https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L1088-L1118):
{code:scala}
  private[sql] def getOffsetFromLocalMillis(millisLocal: Long, tz: TimeZone): Long = {
    var guess = tz.getRawOffset
    // the actual offset should be calculated based on milliseconds in UTC
    val offset = tz.getOffset(millisLocal - guess)
    if (offset != guess) {
      guess = tz.getOffset(millisLocal - offset)
      if (guess != offset) {
        // fallback to do the reverse lookup using java.sql.Timestamp
        // this should only happen near the start or end of DST
        val days = Math.floor(millisLocal.toDouble / MILLIS_PER_DAY).toInt
        val year = getYear(days)
        val month = getMonth(days)
        val day = getDayOfMonth(days)

        var millisOfDay = (millisLocal % MILLIS_PER_DAY).toInt
        if (millisOfDay < 0) {
          millisOfDay += MILLIS_PER_DAY.toInt
        }
        val seconds = (millisOfDay / 1000L).toInt
        val hh = seconds / 3600
        val mm = seconds / 60 % 60
        val ss = seconds % 60
        val ms = millisOfDay % 1000
        val calendar = Calendar.getInstance(tz)
        calendar.set(year, month - 1, day, hh, mm, ss)
        calendar.set(Calendar.MILLISECOND, ms)
        guess = (millisLocal - calendar.getTimeInMillis()).toInt
      }
    }
    guess
  }
{code}

Meanwhile, JDK's GregorianCalendar uses special methods of ZoneInfo, see https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/aa318070b27849f1fe00d14684b2a40f7b29bf79/jdk/src/share/classes/java/util/GregorianCalendar.java#L2795-L2801:
{code:java}
            if (zone instanceof ZoneInfo) {
                ((ZoneInfo)zone).getOffsetsByWall(millis, zoneOffsets);
            } else {
                int gmtOffset = isFieldSet(fieldMask, ZONE_OFFSET) ?
                                    internalGet(ZONE_OFFSET) : zone.getRawOffset();
                zone.getOffsets(millis - gmtOffset, zoneOffsets);
            }
{code}

Need to investigate are there any differences in results between 2 approaches.
Bug: java.lang.IllegalArgumentException: Frame length should be positive
Status: Resolved | Priority: Major | Resolution: Won't Do
Created: 29/Apr/20 10:46
Affects Version/s: 2.4.4

Hi,

I am currently facing the below error when I run my code to stream data from Couchbase in spark cluster.

2020-04-29 00:04:06,061 WARN server.TransportChannelHandler: Exception in connection from /[host]:56910 

java.lang.IllegalArgumentException: Frame length should be positive: -9223371863711366549
 at org.spark_project.guava.base.Preconditions.checkArgument(Preconditions.java:119)
 at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:134)
 at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:81)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 at java.lang.Thread.run(Thread.java:748)

 

When ever I run the spark-submit command with all the arguments on the host I get the above error.

  

Thanks,

Divya
Bug: FileStreamSource: Decide seen files on the checksum, instead of filename.
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/20 08:41
Affects Version/s: 2.4.5

At the moment structured streaming's file source, ignores updates to the same file, it has processed earlier. However, for reasons beyond our control, a software might update the same file with new data. A case in point can be rolling logs, where the latest log file is always e.g. log.txt and the rolled logs could be log-1.txt etc... 
So by supporting this, it may not actually be a special casing but supporting a genuine use case.
Bug: extracting day from intervals should be interval.days + days in interval.microsecond
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Apr/20 13:53
Affects Version/s: 3.0.0

checked with both Presto and PostgresSQL, one is implemented intervals with ANSI style year-month/day-time, the other is mixed and Non-ANSI. They both add the exceeded days in interval time part to the total days of the operation which extracts day from interval values

 

```sql

presto> SELECT EXTRACT(DAY FROM (cast('2020-01-15 00:00:00' as timestamp) - cast('2020-01-01 00:00:00' as timestamp)));
 _col0
-------
 14
(1 row)

Query 20200428_135239_00000_ahn7x, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:01 [0 rows, 0B] [0 rows/s, 0B/s]

presto> SELECT EXTRACT(DAY FROM (cast('2020-01-15 00:00:00' as timestamp) - cast('2020-01-01 00:00:01' as timestamp)));
 _col0
-------
 13
(1 row)

Query 20200428_135246_00001_ahn7x, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [0 rows, 0B] [0 rows/s, 0B/s]

presto>

```

```scala

postgres=# SELECT EXTRACT(DAY FROM (cast('2020-01-15 00:00:00' as timestamp) - cast('2020-01-01 00:00:00' as timestamp)));
 date_part
-----------
 14
(1 row)

postgres=# SELECT EXTRACT(DAY FROM (cast('2020-01-15 00:00:00' as timestamp) - cast('2020-01-01 00:00:01' as timestamp)));
 date_part
-----------
 13

```
Bug: Pyspark CrossValidator/TrainValidationSplit with pipeline estimator cannot save and load model
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 20/Apr/20 09:50
Affects Version/s: 2.4.5

Pyspark CrossValidator/TrainValidationSplit with pipeline estimator cannot save and load model.

Reproduce code run in pyspark shell:

1) Train model and save model in pyspark:
{code:python}

from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.feature import HashingTF, Tokenizer
from pyspark.ml.tuning import CrossValidator, CrossValidatorModel, ParamGridBuilder

training = spark.createDataFrame([
    (0, "a b c d e spark", 1.0),
    (1, "b d", 0.0),
    (2, "spark f g h", 1.0),
    (3, "hadoop mapreduce", 0.0),
    (4, "b spark who", 1.0),
    (5, "g d a y", 0.0),
    (6, "spark fly", 1.0),
    (7, "was mapreduce", 0.0),
    (8, "e spark program", 1.0),
    (9, "a e c l", 0.0),
    (10, "spark compile", 1.0),
    (11, "hadoop software", 0.0)
], ["id", "text", "label"])

# Configure an ML pipeline, which consists of tree stages: tokenizer, hashingTF, and lr.
tokenizer = Tokenizer(inputCol="text", outputCol="words")
hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
lr = LogisticRegression(maxIter=10)
pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])

paramGrid = ParamGridBuilder() \
    .addGrid(hashingTF.numFeatures, [10, 100, 1000]) \
    .addGrid(lr.regParam, [0.1, 0.01]) \
    .build()
crossval = CrossValidator(estimator=pipeline,
                          estimatorParamMaps=paramGrid,
                          evaluator=BinaryClassificationEvaluator(),
                          numFolds=2)  # use 3+ folds in practice

# Run cross-validation, and choose the best set of parameters.
cvModel = crossval.fit(training)

cvModel.save('/tmp/cv_model001') # save model failed. Rase error.
{code}

2): Train crossvalidation model in scala with similar code above, and save to '/tmp/model_cv_scala001', run following code in pyspark:
{code:python}
from pyspark.ml.tuning import CrossValidatorModel
CrossValidatorModel.load('/tmp/model_cv_scala001') # raise error
{code}
Bug: Flaky test: AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 25/Apr/20 11:56
Affects Version/s: 3.0.0

the org.apache.spark.sql.execution.ui.AllExecutionsPageSuite.SPARK-27019:correctly display SQL page when event reordering happens is flaky for just checking the html not containing 1970. I will add a ticket to check and fix that.
In the specific failure https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121799/testReport, it failed because

...
<td sorttable_customkey="1587806019707">
...
contained 1970
Bug: Backport SPARK-27981       Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
Status: Closed | Priority: Major | Resolution: Won't Do
Created: 23/Apr/20 20:06
Affects Version/s: 2.4.6

SPARK-27981       Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+
Bug: Backport SPARK-30199       Recover `spark.(ui|blockManager).port` from checkpoint
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Apr/20 20:10
Affects Version/s: 2.4.6

Backport SPARK-30199       Recover `spark.(ui|blockManager).port` from checkpoint

cc [~dongjoon] for if you think this is a good candidate
Bug: Backport SPARK-25692       Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 23/Apr/20 20:08
Affects Version/s: 2.4.6

Backport SPARK-25692       Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well.

While the test was only flaky in the 3.0 branch, it seems possible the same code path could be triggered in 2.4 so consider for backport.
Bug: Hive metastore client initialization related configurations should be static 
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/20 15:53
Affects Version/s: 2.0.2

The following configurations defined in HiveUtils should be considered static:
 # spark.sql.hive.metastore.version - used to determine the hive version in Spark
 # spark.sql.hive.version - the fake of the above
 # spark.sql.hive.metastore.jars - hive metastore related jars location which is used by spark to create hive client
 # spark.sql.hive.metastore.sharedPrefixes and spark.sql.hive.metastore.barrierPrefixes -  packages of classes that are shared or separated between SparkContextLoader and hive client class loader

Those are used only once when creating the hive metastore client. They should be static in SQLConf for retrieving them correctly. We should avoid them being changed by users with SET/RESET command. They should
Bug: string type compare with numberic cause data inaccurate
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 21/Apr/20 10:56
Affects Version/s: 2.4.5

Hi all

 

Sparksql may should convert values to double if string type compare with number type.the cases shows as below
1, create table
create table test1(id string);
 
2,insert data into table
insert into test1 select 'avc';
insert into test1 select '2';
insert into test1 select '0a';
insert into test1 select '1111111111111111';
insert into test1 select '22222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222222';
3.Let's check what's happening
select * from test_gf13871.test1 where id > 0
the results shows below
*2*
*11111111*
Really amazing,the big number 222...cannot be selected.
while when i check in hive,the 222...shows normal.
4.try to explain the command,we may know what happened,if the data is big enough than max_int_value，it will not selected,we may need to convert to double instand.
!image-2020-04-21-18-49-58-850.png!
I wanna know if we have fixed or planned in 3.0 or later version.,please feel free to give any advice,
 
Many Thanks
Bug: Expose filterByRange in JavaPairRDD
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Apr/20 08:40
Affects Version/s: 3.1.0

The `filterByRange` method makes it possible to efficiently filter a sorted RDD using bounds on its keys. It prunes out partitions instead of scanning them if a RangePartitioner is available in the RDD.

This method is part of the Scala API, defined in OrderedRDDFunctions, but is not exposed in the Java API as far as I can tell. All other methods defined in OrderedRDDFunctions are exposed in JavaPairRDD, therefore it seems natural to expose `filterByRange` there too.
Bug: Remove millennium, century, decade, millisecond, microsecond and epoch from extract fucntion
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Apr/20 09:38
Affects Version/s: 3.0.0

Extracting millennium, century, decade, millisecond, microsecond and epoch from datetime is neither ANSI standard nor quite common in modern SQL platforms. None of the systems listing below support these except PostgreSQL.

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF

https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions050.htm

https://prestodb.io/docs/current/functions/datetime.html

https://docs.cloudera.com/documentation/enterprise/5-8-x/topics/impala_datetime_functions.html

https://docs.snowflake.com/en/sql-reference/functions-date-time.html#label-supported-date-time-parts

https://www.postgresql.org/docs/9.1/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT

https://docs.teradata.com/reader/kmuOwjp1zEYg98JsB8fu_A/SIkE2wnHyQBnU4AGWRZSRw
Bug: Upgrade built-in Hive 2.3.6 to 2.3.7
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 08/Apr/20 01:37
Affects Version/s: 3.0.0

Hive 2.3.7 fixed these issues:

HIVE-21508: ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
HIVE-21980:Parsing time can be high in case of deeply nested subqueries
HIVE-22249: Support Parquet through HCatalog
Bug: Consistancy between dayofweek/dow in extract expression and dayofweek function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 18:14
Affects Version/s: 3.0.0


{code:sql}
spark-sql> SELECT extract(dayofweek from '2009-07-26');
1
spark-sql> SELECT extract(dow from '2009-07-26');
0
spark-sql> SELECT extract(isodow from '2009-07-26');
7
spark-sql> SELECT dayofweek('2009-07-26');
1
spark-sql> SELECT weekday('2009-07-26');
6
{code}
Currently, there are 4 types of day-of-week range: 
the function dayofweek(2.3.0) and extracting dayofweek(2.4.0) result as of Sunday(1) to Saturday(7)
extracting dow(3.0.0) results as of Sunday(0) to Saturday(6)
extracting isodow (3.0.0)  results as of Monday(1) to Sunday(7)
the function weekday(2.4.0) results as of Monday(0) to Sunday(6)

Actually, extracting dayofweek and dow are both derived from PostgreSQL but have different meanings.
https://issues.apache.org/jira/browse/SPARK-23903
https://issues.apache.org/jira/browse/SPARK-28623
Bug: Sparse BLAS native optimization
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/20 07:11
Affects Version/s: 3.1.0

BLAS are highly optimized routines by the industry. If take advantage of it, we can leverage hardware vendors' low-level optimization when available. 

[Sparse BLAS|https://math.nist.gov/spblas] also has [native optimization|https://software.intel.com/en-us/mkl-developer-reference-c-sparse-blas-level-2-and-level-3-routines] , but current BLAS in Spark only has native optimization for Dense and naive Java implementation for Sparse.

We would like to introduce native optimization support for Sparse BLAS operations related to machine learning and produce benchmarks.
Bug: Single precision floating point support in machine learning
Status: Open | Priority: Major | Resolution: 
Created: 21/Apr/20 06:47
Affects Version/s: 3.1.0

MLlib assumed the algorithm data type to "double" in all its BLAS and algorithm code. In some situations, there is no need to use high precision, end users may want to trade precision with performance given machines have much more single precision FP bandwiths than double precision ones.

It may require templating all FP data type and rewrite all existing code. We would like to fast prototype some key algorithms and produce benchmarks on using float instead of double. Feel free to comment on this if there is a need!
Bug: Add more column entries in bulit-in function tables of SQL references
Status: Open | Priority: Major | Resolution: 
Created: 20/Apr/20 12:37
Affects Version/s: 3.1.0

In SPARK-31429, we are planning to automatically generate a list of built-in functions in SQL references. In the PR, only function names and descriptions are shown in a function table. So, we could improve the table by adding more fields (e.g., input types, arguments, ....) in the table.
Bug: Make extract interval field ANSI compliance
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 08:48
Affects Version/s: 3.0.0

Currently, we can extract millennium/century/decade/year/quarter/month/week/day/hour/minute/second(with fractions)//millisecond/microseconds and epoch from interval values

millennium/century/decade/year means how many the interval months part can be converted to that unit-value.

while month/day and so on, the integral remainder of the previous unit.

while getting epoch we have treat month as 30 days which varies the natural Calendar rules we use.

To avoid ambiguity,  I suggest we should only support those extract field defined ANSI with their abbreviations.
Bug: Null types should be implicitly casted to Decimal types
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Apr/20 08:21
Affects Version/s: 3.0.0

A query below fails in master/branch-3.0 and passed in v2.4.5;
{code}
scala> Seq(BigDecimal(10)).toDF("v1").selectExpr("v1 = NULL").explain(true)
org.apache.spark.sql.AnalysisException: cannot resolve '(`v1` = NULL)' due to data type mismatch: differing types in '(`v1` = NULL)' (decimal(38,18) and null).; line 1 pos 0;
'Project [(v1#5 = null) AS (v1 = NULL)#7]
+- Project [value#2 AS v1#5]
   +- LocalRelation [value#2]
...
{code}
Bug: Infinite timeline redraw in job details page
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Apr/20 22:57
Affects Version/s: 2.0.2

In the job page, the timeline section keeps changing the position style and shaking. We can see that there is a warning "infinite loop in redraw" from the console, which can be related to https://github.com/visjs/vis-timeline/issues/17

I am using the history server with the events under "core/src/test/resources/spark-events" to reproduce.
I have also uploaded a screen recording.
Bug: UnsafeKVExternalSorter/VariableLengthRowBasedKeyValueBatch should also respect UnsafeAlignedOffset
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 12/Apr/20 14:42
Affects Version/s: 2.1.0

Since BytesToBytesMap respects UnsafeAlignedOffset when writing the record, UnsafeKVExternalSorter should also respect UnsafeAlignedOffset when reading the record otherwise it will causes a data correctness issue.
Bug: Peak Execution Memory Quantile is not displayed in Spark History Server UI
Status: Open | Priority: Major | Resolution: 
Created: 07/Apr/20 21:09
Affects Version/s: 3.0.0

Peak Execution Memory Quantile is displayed in the regular Spark UI correctly. If the same application is viewed in Spark History Server UI, Peak Execution Memory is always displayed as zero.

Spark event log for the application seem to contain Peak Execution Memory(under the tag "internal.metrics.peakExecutionMemory") correctly.  However this is not reflected in the History Server UI.

*Steps to produce non-zero Peak Execution Memory*

spark.range(0, 200000).map\{x => (x , x % 20)}.toDF("a", "b").createOrReplaceTempView("fred")

spark.range(0, 200000).map\{x => (x , x + 1)}.toDF("a", "b").createOrReplaceTempView("phil")

sql("select p.**,* f.* from phil p join fred f on f.b = p.b").count
Bug: spark jdbc read hive created the wrong PreparedStatement
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 16/Apr/20 05:07
Affects Version/s: 2.3.2

{code:java}
val res = spark
  .read
  .format("jdbc")
  .option("url", "jdbc:hive2://host:10000/default")
  .option("dbtable", "user_info2")
  .option("driver","org.apache.hive.jdbc.HiveDriver")
  .option("user", "")
  .option("password","")
  .load()

res.show(){code}
get wrong result

+--------------+--------------+-------------------+
|user_info2.age|user_info2.sex|user_info2.birthday|
+--------------+--------------+-------------------+
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
|user_info2.age|user_info2.sex|user_info2.birthday|
+--------------+--------------+-------------------+
Bug: Perf regression of toJavaDate
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Apr/20 13:13
Affects Version/s: 3.0.0

DateTimeBenchmark shows the regression

Spark 2.4.6-SNAPSHOT at the PR [https://github.com/MaxGekk/spark/pull/27]
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
To/from Java's date-time:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
From java.sql.Date                                  559            603          38          8.9         111.8       1.0X
Collect dates                                      2306           3221        1558          2.2         461.1       0.2X
{code}
Current master:
{code:java}
OpenJDK 64-Bit Server VM 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 on Linux 4.15.0-1063-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
To/from Java's date-time:                 Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
From java.sql.Date                                 1052           1130          73          4.8         210.3       1.0X
Collect dates                                      3251           4943        1624          1.5         650.2       0.3X
{code}
If we subtract preparing DATE column:
* Spark 2.4.6-SNAPSHOT is (461.1 - 111.8) = 349.3 ns/row
* master is (650.2 - 210.3) = 439 ns/row

The regression of toJavaDate in master against Spark 2.4.6-SNAPSHOT is (439 - 349.3)/349.3 = 25%