av9ash/GitBugs huggingface dataset:
Bug: Spark throws NPE without useful message when NULL value appears in non-null schema
Status: In Progress | Priority: Major | Resolution: 
Created: 24/Aug/22 00:09
Affects Version/s: 3.2.2

Currently in some cases, if Spark encounters a NULL value where the schema indicates that the column/field should be non-null, it will throw a {{NullPointerException}} with no message and thus no way to debug further. This can happen, for example, if you have a UDF which is erroneously marked as {{asNonNullable()}}, or if you read input data where the actual values don't match the schema (which could happen e.g. with Avro if the reader provides a schema declaring non-null although the data was written with null values).

As an example of how to reproduce:
{code:scala}
    val badUDF = spark.udf.register[String, Int]("bad_udf", in => null).asNonNullable()
    Seq(1, 2).toDF("c1").select(badUDF($"c1")).collect()
{code}

This throws an exception like:
{code}
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (xxxxxxxxxx executor driver): java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.codegen.UnsafeWriter.write(UnsafeWriter.java:110)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
	at org.apache.spark.scheduler.Task.run(Task.scala:139)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1490)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
{code}

As a user, it is very confusing -- it looks like there is a bug in Spark. We have had many users report such problems, and though we can guide them to a schema-data mismatch, there is no indication of what field might contain the bad values, so a laborious data exploration process is required to find and remedy it.

We should provide a better error message in such cases.
Bug: Finalize shuffle merge slow due to connection creation fails
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 16/Aug/22 05:20
Affects Version/s: 3.3.0

*How to reproduce this issue*
 * Enable push based shuffle
 * Remove some merger nodes before sending finalize RPCs
 * Driver try to connect those merger shuffle services and send finalize RPC one by one, each connection creation will timeout after SPARK_NETWORK_IO_CONNECTIONCREATIONTIMEOUT_KEY (120s by default)

 
We can send these RPCs in *shuffleMergeFinalizeScheduler*  thread pool and handle the connection creation exception
Bug: Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark
Status: Open | Priority: Major | Resolution: 
Created: 09/Aug/22 22:47
Affects Version/s: 3.2.2

Project Lightspeed is an umbrella project aimed at improving a couple of key aspects of Spark Streaming:
 * Improving the latency and ensuring it is predictable
 * Enhancing functionality for processing data with new operators and APIs

 

Umbrella Jira to track all tickets under Project Lightspeed

SPARK-39585 - Multiple Stateful Operators in Structured Streaming
SPARK-39586 - Advanced Windowing in Structured Streaming
SPARK-39587 - Schema Evolution for Stateful Pipelines
SPARK-39589 - Asynchronous I/O support
SPARK-40431 - Python API for Arbitrary Stateful Processing
SPARK-39591 - Offset Management Improvements

SPARK-40849 - Async log purge

SPARK-39592 - Asynchronous State Checkpointing
SPARK-39593 - Configurable State Checkpointing Frequency
Bug: Support list type for pyspark.sql.functions.lit
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 06:26
Affects Version/s: 3.4.0

Currently, `pyspark.sql.functions.lit` doesn't support for Python list type as below:


{code:python}
>>> df = spark.range(3).withColumn("c", lit([1,2,3]))
Traceback (most recent call last):
...
: org.apache.spark.SparkRuntimeException: [UNSUPPORTED_FEATURE.LITERAL_TYPE] The feature is not supported: Literal for '[1, 2, 3]' of class java.util.ArrayList.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:302)
	at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:100)
	at org.apache.spark.sql.functions$.lit(functions.scala:125)
	at org.apache.spark.sql.functions.lit(functions.scala)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:833)
{code}

We should make it supported.
Bug: Load Data using Spark by a single partition moves entire dataset under same location in S3
Status: Reopened | Priority: Major | Resolution: 
Created: 31/Aug/22 07:03
Affects Version/s: 3.2.1

Hello,

I'm experiencing an issue in PySpark when creating a hive table and loading in the data to the table. So I'm using an Amazon s3 bucket as a data location and I'm creating a table as parquet and trying to load data into that table by a single partition, and I'm seeing some weird behavior. When selecting the data location in s3 of a parquet file to load into my table. All of the data is moved into the specified location in my create table command including the partitions I didn't specify in the load data command. For example:
{code:java}
# create a data frame in pyspark with partitions
df = spark.createDataFrame([("a", 1, "x"), ("b", 2, "y"), ("c", 3, "y")], ["c1", "c2", "p"])
# save it to S3
df.write.format("parquet").mode("overwrite").partitionBy("p").save("s3://bucket/data/")
{code}

In the current state S3 should have a new folder `data` with two folders which contain a parquet file in each partition. 
  
 - s3://bucket/data/p=x/
    - part-00001.snappy.parquet
 - s3://bucket/data/p=y/
    - part-00002.snappy.parquet
    - part-00003.snappy.parquet

 
{code:java}
# create new table
spark.sql("create table src (c1 string,c2 int) PARTITIONED BY (p string) STORED AS parquet LOCATION 's3://bucket/new/'")
# load the saved table data from s3 specifying single partition value x
spark.sql("LOAD DATA INPATH 's3://bucket/data/'INTO TABLE src PARTITION (p='x')")
spark.sql("select * from src").show()
# output: 
# +---+---+---+
# | c1| c2|  p|
# +---+---+---+
# +---+---+---+
{code}


After running the `load data` command, and looking at the table I'm left with no data loaded in. When checking S3 the data source we saved earlier is moved under `s3://bucket/new/` oddly enough it also brought over the other partitions along with it directory structure listed below. 

- s3://bucket/new/
    - p=x/
        - p=x/
            - part-00001.snappy.parquet
        - p=y/
            - part-00002.snappy.parquet
            - part-00003.snappy.parquet

Is this the intended behavior of loading the data in from a partitioned parquet file? Is the previous file supposed to be moved/deleted from source directory?
Bug: AQE should use available column statistics from completed query stages
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 05/Aug/22 17:51
Affects Version/s: 3.3.0

In QueryStageExec.computeStats we copy partial statistics from materlized query stages by calling QueryStageExec#getRuntimeStatistics, which in turn calls ShuffleExchangeLike#runtimeStatistics or BroadcastExchangeLike#runtimeStatistics.

Only dataSize and numOutputRows are copied into the new Statistics object:

 {code:scala}
  def computeStats(): Option[Statistics] = if (isMaterialized) {
    val runtimeStats = getRuntimeStatistics
    val dataSize = runtimeStats.sizeInBytes.max(0)
    val numOutputRows = runtimeStats.rowCount.map(_.max(0))
    Some(Statistics(dataSize, numOutputRows, isRuntime = true))
  } else {
    None
  }
{code}

I would like to also copy over the column statistics stored in Statistics.attributeMap so that they can be fed back into the logical plan optimization phase. This is a small change as shown below:

{code:scala}
  def computeStats(): Option[Statistics] = if (isMaterialized) {
    val runtimeStats = getRuntimeStatistics
    val dataSize = runtimeStats.sizeInBytes.max(0)
    val numOutputRows = runtimeStats.rowCount.map(_.max(0))
    val attributeStats = runtimeStats.attributeStats
    Some(Statistics(dataSize, numOutputRows, attributeStats, isRuntime = true))
  } else {
    None
  }
{code}

The Spark implementations of ShuffleExchangeLike and BroadcastExchangeLike do not currently provide such column statistics, but other custom implementations can.
Bug: Repeat calls to `PartitionIterator.hasNext` can timeout
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 31/Aug/22 19:46
Affects Version/s: 3.3.0

Repeat calls to {{PartitionIterator.hasNext}} where both calls return {{false}} can result in timeouts. For example, {{{}KafkaBatchPartitionReader.next(){}}}, which calls {{consumer.get}} (which can potentially timeout with repeat calls), is called by {{{}PartitionIterator.hasNext{}}}. Thus, repeat calls to {{PartitionIterator.hasNext}} by its parent could timeout.
Bug: Fix the issue with Parquet column index and predicate pushdown in Data source V1
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 21/Aug/22 22:47
Affects Version/s: 3.2.3

This is a follow for SPARK-39833. In [https://github.com/apache/spark/pull/37419,] we disabled column index for Parquet due to correctness issues that we found when filtering data on the partition column overlapping with data schema.

 

This ticket is for permanent and thorough fix for the issue and re-enablement of the column index. See more details in the PR linked above.
Bug: Simplify join condition of form (a==b) || (a==null&&b==null) to a<=>b
Status: In Progress | Priority: Major | Resolution: 
Created: 22/Aug/22 08:13
Affects Version/s: 3.2.0

If the join condition is like key1==key2 || (key1==null && key2==null), join is executed as Broadcast Nested Loop Join as this condition doesn't satisfy equi join condition. BNLJ takes more time as compared to Sort merge or broadcast join. This condition can be converted to key1<=>key2 to make the join execute as Broadcast or sort merge join.
Bug: arrays_zip output unexpected alias column names
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 31/Aug/22 17:26
Affects Version/s: 3.4.0

For the below query:
{code:sql}
with q as (
  select
    named_struct(
      'my_array', array(named_struct('x', 1, 'y', 2))
    ) as my_struct
)
select
  arrays_zip(my_struct.my_array)
from
  q {code}
The latest spark gives the below schema, the field name "my_array" was changed to "0"
{code:java}
root
 |-- arrays_zip(my_struct.my_array): array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- 0: struct (nullable = true)
 |    |    |    |-- x: integer (nullable = true)
 |    |    |    |-- y: integer (nullable = true){code}
While Spark 3.1 gives the expected result
{code:java}
root
 |-- arrays_zip(my_struct.my_array): array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- my_array: struct (nullable = true)
 |    |    |    |-- x: integer (nullable = true)
 |    |    |    |-- y: integer (nullable = true)
{code}
Bug: Failure to create parquet predicate push down for ints and longs on some valid files
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 20:03
Affects Version/s: 3.1.0

The [parquet format|https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers] specification states that...

bq. {{{}INT(8, true){}}}, {{{}INT(16, true){}}}, and {{INT(32, true)}} must annotate an {{int32}} primitive type and {{INT(64, true)}} must annotate an {{int64}} primitive type. {{INT(32, true)}} and {{INT(64, true)}} are implied by the {{int32}} and {{int64}} primitive types if no other annotation is present and should be considered optional.

But the code inside of [ParquetFilters.scala|https://github.com/apache/spark/blob/296fe49ec855ac8c15c080e7bab6d519fe504bd3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala#L125-L126] requires that for {{int32}} and {{int64}} that there be no annotation. If there is an annotation for those columns and they are a part of a predicate push down, the hard coded types will not match and the corresponding filter ends up being {{None}}.

This can be a huge performance penalty for a valid parquet file.

I am happy to provide files that show the issue if needed for testing.
Bug: mergedShuffleCleaner should have been shutdown before db closed
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 03:27
Affects Version/s: 3.4.0

Should ensure `RemoteBlockPushResolver#mergedShuffleCleaner` have been shutdown before `RemoteBlockPushResolver#db` closed, otherwise, `RemoteBlockPushResolver#applicationRemoved` may perform delete operations on a closed db.

 

https://github.com/apache/spark/pull/37610#discussion_r951185256
Bug: Remove column suggestion when the candidate list is empty for unresolved column/attribute/map key
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 02:17
Affects Version/s: 3.4.0

For unresolved column, attribute or map key an error message might contain suggestions from the list. However, when the list is empty the error message looks incomplete:

`[UNRESOLVED_COLUMN] A column or function parameter with name 'YrMo' cannot be resolved. Did you mean one of the following? []`

This issue is to make final suggestion to show only if suggestion list is non empty.
Bug: Fix the documents "Contributing and Maintaining Type Hints".
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 07:55
Affects Version/s: 3.4.0

Since we don't use `*.pyi` for type hinting anymore (it's all ported as inline type hints in the `*.py` files), we also should fix the related documents accordingly (https://spark.apache.org/docs/latest/api/python/development/contributing.html#contributing-and-maintaining-type-hints)
Bug: Don't simplify multiLike if child is not attribute
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 26/Aug/22 07:46
Affects Version/s: 3.4.0

{code:scala}
sql("create table t1(name string) using parquet")
sql("select * from t1 where substr(name, 1, 5) like any('%a', 'b%', '%c%')").explain(true)
{code}

{noformat}
== Physical Plan ==
*(1) Filter ((EndsWith(substr(name#0, 1, 5), a) OR StartsWith(substr(name#0, 1, 5), b)) OR Contains(substr(name#0, 1, 5), c))
+- *(1) ColumnarToRow
   +- FileScan parquet default.t1[name#0] Batched: true, DataFilters: [((EndsWith(substr(name#0, 1, 5), a) OR StartsWith(substr(name#0, 1, 5), b)) OR Contains(substr(n..., Format: Parquet, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<name:string>

{noformat}
Bug: Expensive UDF evaluation pushed down past a join leads to performance issues 
Status: Open | Priority: Major | Resolution: 
Created: 29/Aug/22 23:24
Affects Version/s: 3.4.0

Consider a Spark job with an expensive UDF which looks like follows:
{code:scala}
val expensive_udf = spark.udf.register("expensive_udf", (i: Int) => Option(i))

spark.range(10).write.format("orc").save("/tmp/orc")

val df = spark.read.format("orc").load("/tmp/orc").as("a")
    .join(spark.range(10).as("b"), "id")
    .withColumn("udf_op", expensive_udf($"a.id"))
    .join(spark.range(10).as("c"), $"udf_op" === $"c.id")
{code}
This creates a physical plan as follows:
{code:java}
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastHashJoin [cast(udf_op#338 as bigint)], [id#344L], Inner, BuildRight, false
   :- Project [id#330L, if (isnull(cast(id#330L as int))) null else expensive_udf(knownnotnull(cast(id#330L as int))) AS udf_op#338]
   :  +- BroadcastHashJoin [id#330L], [id#332L], Inner, BuildRight, false
   :     :- Filter ((isnotnull(id#330L) AND isnotnull(cast(id#330L as int))) AND isnotnull(expensive_udf(knownnotnull(cast(id#330L as int)))))
   :     :  +- FileScan orc [id#330L] Batched: true, DataFilters: [isnotnull(id#330L), isnotnull(cast(id#330L as int)), isnotnull(expensive_udf(knownnotnull(cast(i..., Format: ORC, Location: InMemoryFileIndex(1 paths)[file:/tmp/orc], PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
   :     +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]),false), [plan_id=416]
   :        +- Range (0, 10, step=1, splits=16)
   +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]),false), [plan_id=420]
      +- Range (0, 10, step=1, splits=16)
{code}
In this case, the expensive UDF call is duplicated thrice. Since the UDF output is used in a future join, `InferFiltersFromConstraints` adds an `IS NOT NULL` filter on the UDF output. But the pushdown rules duplicate this UDF call and push the UDF past a previous join. The duplication behaviour [is documented|https://github.com/apache/spark/blob/c95ed826e23fdec6e1a779cfebde7b3364594fb5/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala#L196] and in itself is not a huge issue. But given a highly restrictive join, the UDF gets evaluated on many orders of magnitude more rows than it should have slowing down the job.

Can we avoid this duplication of UDF calls? In SPARK-37392, we made a [similar change|https://github.com/apache/spark/pull/34823/files] where we decided to only add inferred filters if the input is an attribute. Should we use a similar strategy for `InferFiltersFromConstraints`?
Bug: Fix the inconsistent behavior for Index.intersection.
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 30/Aug/22 00:39
Affects Version/s: 3.4.0

There is inconsistent behavior on `Index.intersection` when `other` is list of tuple for pandas API on Spark as below:


{code:python}
>>> pidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
MultiIndex([], )
>>> pidx.intersection([(1, 2), (3, 4)]).sort_values()
Traceback (most recent call last):
...
ValueError: Names should be list-like for a MultiIndex
{code}

We should fix it to follow pandas.
Bug: Nested schema pruning support through element_at
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 10/Aug/22 07:39
Affects Version/s: 3.4.0

The semantics of element_at is similar with GetArrayItem and GetMapValue, so we can support do nested schema pruning if the inside data type is struct.

For example:
For a column schema: `c: array<struct<s1: int, s2: string>>`
With the query: `SELECT element_at(c, 1).s1`
The final pruned schema should be `c: array<struct<s1: int>>`
Bug: Uncatchable exceptions in SparkSession Java API
Status: Resolved | Priority: Major | Resolution: Won't Fix
Created: 31/Aug/22 11:27
Affects Version/s: 3.2.1

SparkSession#sql may raise exceptions extending org.apache.spark.sql.AnalysisException like org.apache.spark.sql.catalyst.analysis.NoSuchPartitionException for instance. These exceptions extend java.lang.Exception.

As a result, they are considered as *checked* Exception by the Java compiler, they can not be caught from Java code because the Java compiler considers SparkSession#sql doesn't throw this kind of method (there is not throws AnalysisException in the signature).

The only workaround is to catch java.lang.Exception which produces a very wide catch.

A cleaner solution would be to make org.apache.spark.sql.AnalysisException extends java.lang.RuntimeException
Bug: Add doc for using Apache YuniKorn as a customized scheduler
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 23/Aug/22 05:04
Affects Version/s: 3.3.0

Add a section under https://spark.apache.org/docs/latest/running-on-kubernetes.html#customized-kubernetes-schedulers-for-spark-on-kubernetes to explain how to run Spark with Apache YuniKorn. This is based on [this PR review comment|https://github.com/apache/spark/pull/35663#issuecomment-1220474012].
Bug: Improve the TO_BINARY() function
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 17/Aug/22 01:18
Affects Version/s: 3.4.0

Original description SPARK-37507:
{quote}to_binary(expr, fmt) is a common function available in many other systems to provide a unified entry for string to binary data conversion, where fmt can be utf8, base64, hex and base2 (or whatever the reverse operation to_char()supports).

[https://docs.aws.amazon.com/redshift/latest/dg/r_TO_VARBYTE.html]

[https://docs.snowflake.com/en/sql-reference/functions/to_binary.html]

[https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#format_string_as_bytes]

[https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/etRo5aTAY9n5fUPjxSEynw]

Related Spark functions: unbase64, unhex
{quote}
 

Expected improvement:
 * `base64` behaves more strictly, i.e. does not allow symbols not included in base64 dictionary (A-Za-z0-9+/) and verifies correct padding and symbol groups (see RFC 4648 § 4). Whitespaces are ignored.
 ** Current implementation allows arbitrary strings and invalid symbols are skipped.
 * `hex` converts only valid hexadecimal strings and throws errors otherwise. Whitespaces are not allowed.
 * `utf-8` and `utf8` are interchangeable.
 * Correct errors are thrown and classified for invalid input (CONVERSION_INVALID_INPUT) and invalid format (CONVERSION_INVALID_FORMAT)
Bug: DirectTaskResult meta should not be counted into result size
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 29/Aug/22 20:41
Affects Version/s: 3.4.0

This issue exists for a long time (since [https://github.com/liuzqt/spark/commit/c33e55008239f417764d589c1366371d18331686)]

when calculating whether driver fetching result exceed `spark.driver.maxResultSize` limit, the whole serialized result task size is taken into account, including task metadata overhead size([accumUpdates|https://github.com/apache/spark/blob/c95ed826e23fdec6e1a779cfebde7b3364594fb5/core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala#L41]) as well. However, the metadata should not be counted because they will be discarded by the driver immediately after being processed.

This will lead to exception when running jobs with tons of task but actually return small results.

Therefore we should only count `[valueBytes|https://github.com/apache/spark/blob/c95ed826e23fdec6e1a779cfebde7b3364594fb5/core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala#L40]` when calculating result size limit.

cc [~joshrosen]