av9ash/GitBugs huggingface dataset:
Bug: pyspark dataframe methods (i.e. show()) won't run in VSCode debug console
Status: Open | Priority: Major | Resolution: 
Created: 21/Sep/22 20:35
Affects Version/s: 3.1.2

when debugging pyspark codes in VSCode using the python debugger, when execution is paused on a breakpoint, you can issue statement/expression in VSCode debug console to check dataframe's content, etc.

However, those statement related to df operation always gets stuck and then debugger throws a timeout error in the debug console. 

 

This issue is initially reported in stack overflow: [https://stackoverflow.com/questions/65739467/pyspark-dataframe-methods-i-e-show-can-not-be-printed-in-vs-code-debug-cons]

there are some workaround suggestions as well in that thread.

OS: win 10
VSCode: 1.64.0

Python extension in VScode: v2022.4.1
Bug: Keep struct field names or map keys in CreateStruct
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/22 07:16
Affects Version/s: 3.4.0

Using index-like notation when extracting columns in a struct produces generated column names like {{{}col1{}}}, {{{}col2{}}}, etc. However, when dot-like syntax is used, the column name is correctly selected. 

 

Example:
{code:java}
select struct(a['x'], a['y']), struct(a.x, a.y) from (select named_struct('x', 1, 'y', 2) as a) {code}
Produces the following schema and result:
{code:java}
root
 |-- struct(a.x, a.y): struct (nullable = false)
 |    |-- col1: integer (nullable = false)
 |    |-- col2: integer (nullable = false)
 |-- struct(a.x, a.y): struct (nullable = false)
 |    |-- x: integer (nullable = false)
 |    |-- y: integer (nullable = false)

+----------------+----------------+
|struct(a.x, a.y)|struct(a.x, a.y)|
+----------------+----------------+
|{1, 2}          |{1, 2}          |
+----------------+----------------+ {code}
 

It would be good to have {{x}} and {{y}} column names in the first struct.
Bug: Support dataframe API use jdbc data source in PySpark
Status: Open | Priority: Major | Resolution: 
Created: 20/Sep/22 09:29
Affects Version/s: 3.3.0

When i using pyspark, i wanna get data from mysql database.  so i want use JDBCRDD like java\scala.

But that is not be supported in PySpark.

 

For some reasons, i can't using DataFrame API, only can use RDD(datastream) API. Even i know the DataFrame can get data from jdbc source fairly well.

 
So i want to implement functionality that can use rdd to get data from jdbc source for PySpark.
 
*But i don't know if that are necessary for PySpark.   so we can discuss it.*
 
{*}If it is necessary for PySpark{*}{*}, i want to contribute to Spark.{*}  
*i hope this Jira task can assigned to me, so i can start working to implement it.*
 
*if not, please close this Jira task.*
 
 
*thanks a lot.*
Bug: Integrate Grouped Aggregate Pandas UDFs into *.sql test cases
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 14/Sep/22 04:38
Affects Version/s: 3.4.0

We ported Python UDF, Scala UDF and Scalar Pandas UDF into SQL test cases from SPARK-27921, but Grouped Aggregate Pandas UDF is not tested from SQL at all.

We should also leverage this to test pandas aggregate UDFs too.
Bug: arrays_zip output unexpected alias column names when using GetMapValue and GetArrayStructFields
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 05:30
Affects Version/s: 3.2.2

This is a follow-up for https://issues.apache.org/jira/browse/SPARK-40292. 

I forgot to fix the case when GetMapValue and GetArrayStructFields are used in arrays_zip function instead of structs.
Bug: Update gpg's keyserver
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 01:44
Affects Version/s: 3.4.0

{noformat}
yumwang@LM-SHC-16508156 create-release % gpg --keyserver keyserver.ubuntu.com --recv-key E298A3A825C0D65DFD57CBB651716619E084DAB9
gpg: keyserver receive failed: End of file
{noformat}
Bug: Set upperbound for pyzmq 24.0.0 for Python linter
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Sep/22 00:45
Affects Version/s: 3.1.3

Error: https://github.com/apache/spark/actions/runs/3063515551/jobs/4947782771
New release (10 hours ago): https://github.com/zeromq/pyzmq/commit/2d3327d2e50c2510d45db2fc51488578a737b79b
Bug: Spark Master UI - SSL implementation
Status: Open | Priority: Major | Resolution: 
Created: 08/Sep/22 10:51
Affects Version/s: 2.4.8

While I tried to implement SSL for the master UI, we were getting the below error while having the spark-core (spark-core_2.11-{*}2.4.8{*}.jar) :

22/09/08 03:45:03 ERROR MasterWebUI: Failed to bind MasterWebUI
java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.spark_project.jetty.util.ssl.SslContextFactory. (Use org.spark_project.jetty.util.ssl.SslContextFactory$Server or org.spark_project.jetty.util.ssl.SslContextFactory$Client instead)
        at org.spark_project.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1283)
        at org.spark_project.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1265)hread.run(Thread.java:745)

 

But, while having the spark-core (spark-core_2.11-{*}2.4.3{*}.jar), we are not having any issues.

Please suggest how we can mitigate this issue in the latest version of the spark-core_2.11-2.4.8
Bug: Refactor FlatMapGroupsWithStateExec to have a parent trait
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 13/Sep/22 02:58
Affects Version/s: 3.4.0

There are a lot of stuff to share if we implement another version of {{FlatMapGroupsWithStateExec}}. Should better factor them out. This is part of https://github.com/apache/spark/pull/37285
Bug: Add SQL misc function to assert/check column value
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 06/Sep/22 22:43
Affects Version/s: 3.3.0

SQL function that allows to assert a condition on a column that:
* fails when condition is not met
* returns original value otherwise

Related: SPARK-32793

But {{assert_true}} and {{raise_error}} do not really cut it. In case of {{assert_true}} you have to actually collect the empty column, and the check might no happen if you drop the assertion column, which you will likely do since it's empty. Having a function that returns some value as part of the check, in most cases it would be the checked column would be handy.

I'm working with pyspark, so here's python implementation:

{code:python}
@overload
def assert_col_condition(
    col: Union[str, Column],
    cond: Callable[[Column], Column],
    error_msg: Optional[str] = None,
) -> Column:
    """Asserts condition on a column, IFF it holds returns the original value under `col`"""
    ...


@overload
def assert_col_condition(
    col: Union[str, Column], cond: Column, error_msg: Optional[str] = None
) -> Column:
    """Asserts condition on a column, IFF it holds returns the original value under `col`"""
    ...


def assert_col_condition(
    col: Union[str, Column],
    cond: Union[Column, Callable[[Column], Column]],
    error_msg: Optional[str] = None,
) -> Column:
    col = str_to_col(col)
    if not isinstance(cond, Column):
        cond = cond(col)
    return F.when(
        ~cond, F.raise_error(error_msg or f"Assertion failed: {cond}")
    ).otherwise(col)
{code}
Bug: shuffle data recovery on the reused PVCs  no effect
Status: Reopened | Priority: Major | Resolution: 
Created: 01/Sep/22 03:26
Affects Version/s: 3.2.2

I use spark3.2.2 to test the [ Support shuffle data recovery on the reused PVCs (SPARK-35593) ] feature.I found that when shuffle read fails, data is still read from source.

It can be confirmed that the pvc has been multiplexed by other pods, and the Index and data data information has been sent

*This is my spark configuration information：*

--conf spark.driver.memory=5G 
--conf spark.executor.memory=15G 
--conf spark.executor.cores=1
--conf spark.executor.instances=50
--conf spark.sql.shuffle.partitions=50
--conf spark.dynamicAllocation.enabled=false
--conf spark.kubernetes.driver.reusePersistentVolumeClaim=true
--conf spark.kubernetes.driver.ownPersistentVolumeClaim=true
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=OnDemand
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass=gp2
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit=100Gi
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/tmp/data
--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly=false
--conf spark.executorEnv.SPARK_EXECUTOR_DIRS=/tmp/data
--conf spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.KubernetesLocalDiskShuffleDataIO
--conf spark.kubernetes.executor.missingPodDetectDelta=10s
--conf spark.kubernetes.executor.apiPollingInterval=10s
--conf spark.shuffle.io.retryWait=60s
--conf spark.shuffle.io.maxRetries=5
Bug: Fix `ps.read_parquet` when pandas_metadata is True
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 28/Sep/22 01:12
Affects Version/s: 3.4.0

The behavior of `ps.read_parquet` is not matched with latest pandas when `pandas_metadata` is set True.

We should make the test pass to support pandas 1.5.x.
Bug: SELECT * shouldn't be empty project list in proto.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 27/Sep/22 21:10
Affects Version/s: 3.4.0

Current proto uses empty project list for `SELECT *`.  However, this is an implicit way that it is hard to differentiate `not set` and `set but empty`. For longer term proto compatibility, we should always use explicit fields for passing through information.
Bug: Result of a single task in collect() must fit in 2GB
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Sep/22 17:54
Affects Version/s: 3.3.0

when collecting results, data from single partition/task is serialized through byte array or ByteBuffer(which is backed by byte array as well), therefore it's subject to java array max size limit(in terms of byte array, it's 2GB).

 

Construct a single partition larger than 2GB and collect it can easily reproduce the issue
{code:java}
// create data of size ~3GB in single partition, which exceeds the byte array limit
// random gen to make sure it's poorly compressed
val df = spark.range(0, 3000, 1, 1).selectExpr("id", s"genData(id, 1000000) as data")

withSQLConf("spark.databricks.driver.localMaxResultSize" -> "4g") {
  withSQLConf("spark.sql.useChunkedBuffer" -> "true") {
    df.queryExecution.executedPlan.executeCollect()
  }
} {code}
 will get a OOM error from [https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/io/ByteArrayOutputStream.java#L125]

 

Consider using ChunkedByteBuffer to replace byte array in order to bypassing this limit
Bug: Make yarn appmaster load config from client
Status: Closed | Priority: Major | Resolution: Invalid
Created: 20/Sep/22 10:17
Affects Version/s: 3.0.1

In yarn federation mode, config in client side and nm side may be different. AppMaster should override config from client side.

For example: 

In client side, yarn.resourcemanager.ha.rm-ids are yarn routers.

In nm side, yarn.resourcemanager.ha.rm-ids are the rms of subcluster.
Bug: Migrate failures of array type checks onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:38
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in collection expressions:
1. SortArray (3): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L1027-L1035
2. ArrayContains (2): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L1259-L1264
3. ArrayPosition (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2035
4. ElementAt (3): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2177-L2187
5. Concat (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2385-L2388
6. Flatten (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2593-L2595
7. Sequence (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2773
8. ArrayRemove (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3445-L3447
9. ArrayDistinct (1): https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L3642
Bug: Migrate type check failures of type creators onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:48
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the complex type creator expressions:
1. CreateMap(3): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala#L205-L214
2. CreateNamedStruct(3): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala#L445-L457
3. UpdateFields(2): https://github.com/apache/spark/blob/1431975723d8df30a25b2333eddcfd0bb6c57677/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala#L670-L673
Bug: Migrate type check failures of NthValue and NTile onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:35
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in window expressions:
1. NthValue (2): [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L729-L738|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L697-L700]
2. NTile (3): [https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L827-L840|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala#L793-L804]
Bug: When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 03/Sep/22 02:26
Affects Version/s: 3.0.0

*Reproduce step:*
set `spark.plugins=ErrorSparkPlugin`
`ErrorSparkPlugin` && `ErrorExecutorPlugin` class as below (I abbreviate the code to make it clearer):
{code:java}
class ErrorSparkPlugin extends SparkPlugin {
  /**
   */
  override def driverPlugin(): DriverPlugin =  new ErrorDriverPlugin()

  /**
   */
  override def executorPlugin(): ExecutorPlugin = new ErrorExecutorPlugin()
}{code}
{code:java}
class ErrorExecutorPlugin extends ExecutorPlugin {
  private val checkingInterval: Long = 1

  override def init(_ctx: PluginContext, extraConf: util.Map[String, String]): Unit = {
    if (checkingInterval == 1) {
      throw new UnsatisfiedLinkError("My Exception error")
    }
  }
} {code}
The Executor is active when we check in spark-ui, however it was broken and doesn't receive any task.

*Root Cause:*

I check the code and I find in `org.apache.spark.rpc.netty.Inbox#safelyCall` it will throw fatal error (`UnsatisfiedLinkError` is fatal erro ) in method `dealWithFatalError` . Actually the  `CoarseGrainedExecutorBackend` JVM process  is active but the  communication thread is no longer working ( please see  `MessageLoop#receiveLoopRunnable` , `receiveLoop()` was broken, so executor doesn't receive any message)

Some ideas:
I think it is very hard to know what happened here unless we check in the code. The Executor is active but it can't do anything. We will wonder if the driver is broken or the Executor problem.  I think at least the Executor status shouldn't be active here or the Executor can exitExecutor (kill itself)
Bug: Check unsupported data type when decorrelating subqueries
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/22 15:39
Affects Version/s: 3.4.0

Currently, we don't check if the join condition after subquery decorrelation is resolved or not. This can lead to confusing error messages to users. We should throw proper exceptions when one or more join conditions after decorrelation are not resolved.

For example:
{code:java}
create temp view v1(x) as values from_json('{"a":1, "b":2}', 'map<string,int>') t(x)

select (select a + a from (select upper(x['a']) as a)) from v1 

After applying rule org.apache.spark.sql.catalyst.optimizer.PullupCorrelatedPredicates in batch Pullup Correlated Expressions, the structural integrity of the plan is broken.{code}
Bug: Upgrade hive to 4.0.0
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 14/Sep/22 07:01
Affects Version/s: 3.3.0

Upgrade hive to 4.0.0 to avoid security vulnerability CVE-2022-25647 through google-gson:2.2.4. In hive:4.0.0, the google-gson is upgraded to 2.8.9 for which CVE is not reported yet.
Bug: Introduce "Arbitrary Stateful Processing" in Structured Streaming with Python
Status: Open | Priority: Major | Resolution: 
Created: 15/Sep/22 02:01
Affects Version/s: 3.4.0

This is a part of effort for SPARK-39590, python API parity in Structured Streaming.

Most of public APIs are available in both Scala/Java Spark and PySpark, but we have a huge gap on streaming workload in PySpark as we don't have matching API for flatMapGroupsWithState in PySpark.

This ticket is to track the effort.
Bug: Migrate the type check failures of calls via reflection onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:15
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the expressions of invokes a method on a class via reflection:
CallMethodViaReflection (5): https://github.com/apache/spark/blob/0945baf90660a101ae0f86a39d4c91ca74ae5ee3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala#L62-L75
Bug: Migrate Bloom Filter type check failures onto error classes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Sep/22 09:12
Affects Version/s: 3.4.0

Replace TypeCheckFailure by DataTypeMismatch in type checks in the Bloom Filter expressions:
BloomFilterMightContain (2): https://github.com/apache/spark/blob/a241256ed0778005245253fb147db8a16105f75c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BloomFilterMightContain.scala#L61-L67
Bug: Convert some DDL exception to new error framework
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 06/Sep/22 16:50
Affects Version/s: 3.4.0

Tackling the following files:

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlreadyExistException.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CannotReplaceMissingTableException.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NonEmptyException.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala

Here is the doc with proposed text:
https://docs.google.com/document/d/1TpFx3AwcJZd3l7zB1ZDchvZ8j2dY6_uf5LHfW2gjE4A/edit?usp=sharing
Bug: Add dockerfile template
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/22 07:36
Affects Version/s: 3.4.0

Python addtion:

 
{code:java}
$ diff  scala2.12-java11-r-ubuntu/Dockerfile scala2.12-java11-python3-r-ubuntu/Dockerfile

31a32,33
>     apt install -y python3 python3-pip && \
>     pip3 install --upgrade pip setuptools && \
33a36
>     mkdir /opt/spark/python && \
68a72,73
>     mv python/pyspark /opt/spark/python/pyspark/; \
>     mv python/lib /opt/spark/python/lib/; \ {code}
 

R specific:

 
{code:java}
$ diff  scala2.12-java11-python3-ubuntu/Dockerfile scala2.12-java11-python3-r-ubuntu/Dockerfile

33a34
>     apt install -y r-base r-base-dev && \
72a74
>     mv R /opt/spark/; \
78a81
> ENV R_HOME /usr/lib/R {code}
 

We'd better
 * add a base template and apply R/python3 on demand.
 * Support specify spark version, java version, scala version
Bug: NPE from observe of collect_list
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 22/Sep/22 13:03
Affects Version/s: 3.4.0

The code below reproduces the issue:

{code:scala}
import org.apache.spark.sql.functions._
val df = spark.range(1,10,1,11)
df.observe("collectedList", collect_list("id")).collect()
{code}
instead of

{code}
Array(1, 2, 3, 4, 5, 6, 7, 8, 9)
{code}
it fails with the NPE:

{code:java}
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:641)
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.getBufferObject(interfaces.scala:602)
	at org.apache.spark.sql.catalyst.expressions.aggregate.TypedImperativeAggregate.serializeAggregateBufferInPlace(interfaces.scala:624)
	at org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:205)
	at org.apache.spark.sql.execution.AggregatingAccumulator.withBufferSerialized(AggregatingAccumulator.scala:33)
{code}
Bug: Add spark.sql.legacy.groupingIdWithAppendedUserGroupBy
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Sep/22 09:26
Affects Version/s: 3.2.3

{code:java}
scala> sql("SELECT count(*), grouping__id from (VALUES (1,1,1),(2,2,2)) AS t(k1,k2,v) GROUP BY k1 GROUPING SETS (k2) ").show()
+--------+------------+
|count(1)|grouping__id|
+--------+------------+
|       1|           2|
|       1|           2|
+--------+------------+

scala> sql("set spark.sql.legacy.groupingIdWithAppendedUserGroupBy=true")
res1: org.apache.spark.sql.DataFrame = [key: string, value: string]scala> 

sql("SELECT count(*), grouping__id from (VALUES (1,1,1),(2,2,2)) AS t(k1,k2,v) GROUP BY k1 GROUPING SETS (k2) ").show()
+--------+------------+
|count(1)|grouping__id|
+--------+------------+
|       1|           1|
|       1|           1|
+--------+------------+ {code}
Bug: Improve the performance for setInterval & getInterval of UnsafeRow
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Sep/22 12:25
Affects Version/s: 3.4.0

As we know, UnsafeRow stores each field with 8 bytes or Long.
UnsafeRow supports read and store CalendarInterval.
After my investigation, getLong/putLong of Platform have better performance than call  getInt/putInt of Platform twice.