Updating to Spark 3.0 in production
By: Louis Fruleux
Tags: Production, Spark, Data Engineering, Programming, Big Data

Breaking Changes

Let’s first discuss the four main breaking changes we faced when updating our codebase to Spark 3:

Depreciation of UserDefinedAggregateFunction

Depreciation of untyped UserDefinedFunction, a.k.a. udf(AnyRef, DataType)

Change in behavior of spark.emptyDataFrame

Cassandra driver incompatibilities with third-party libraries.

Deprecation of UserDefinedAggregateFunction

A UserDefinedAggregateFunction , commonly called UDAF, is particularly useful to define custom aggregations, such as averaging sparse arrays.

In Spark 3, the sql.expressions.UserDefinedAggregateFunction API has been deprecated in favor of the Aggregator API (based on algebird Aggregator ). The former suffers from a serialization and deserialization overhead with each merge into the buffer (merge of rows or temporary buffer). In this gist experiment, for 1000 rows, there is 1006 ser/deser. The newly introduced API aims to remove this cost.

On top of the fact that it is supposedly faster, this new API is also easier to write, read, and maintain. That makes it preferable from an engineering point of view.

As an example, here is a gist of a Mean aggregation function with both UDAF and Aggregator.

Concrete impacts of this change observed in production are presented later in this post.

Deprecation of untyped UserDefinedFunction

We also had to face a change in the default behavior of untyped UserDefinedFunction (aka udf ) between Spark 2.4 and Spark 3.0. This change is explained in this PR and can be illustrated as follows.

Given a UDF defined this way val f = udf((x: Int) => x, IntegerType) . For a null value in Spark 2, it returns null and in Spark 3 it returns the default value of the Java type. In this case, 0 for an Int .

One should then be especially careful and rely on strong and wide tests as this difference of behavior can change the results. For instance, this code would behave differently depending on the Spark versions:

import org.apache.spark.sql.types._

import org.apache.spark.sql.functions._ val f = udf((x: Int) => x + 1, IntegerType) Seq((None), (Some(0))).

toDF("value").

withColumn("incremented", f($"value")).

agg(sum("incremented")).

first.getLong(0) // 1 in Spark2, 2 in Spark3

This deprecation warning raises an exception, that can be removed (if you are sure you want the new behavior) by setting the configuration spark.sql.legacy.allowUntypedScalaUDF=true .

Change in behavior of `spark.emptyDataFrame`

There is a known hack in Spark 2 that allows you to make a side effect on each executor of a cluster at runtime. The idea of this hack is to create an empty DataFrame , repartition it by the number of executors and use foreachPartitions to trigger your side effect as in the following example:

val nbExecutors = sparkSession.sparkContext.getConf.get("spark.executor.instances").toInt sparkSession.

emptyDataframe.

toDF.

repartition(nbExecutors).

foreachPartition {

(_: Iterator[Row]) => idempotentSideEffect()

}

Unfortunately, this hack does not work anymore in Spark 3. To understand why let’s compare the physical plans in both versions.

In Spark 2:

spark.emptyDataFrame.explain(true) == Optimized Logical Plan ==

Repartition 8, true

+- LogicalRDD false == Physical Plan ==

Exchange RoundRobinPartitioning(8)

+- Scan ExistingRDD empty[]

In Spark 3:

== Optimized Logical Plan ==

LocalRelation <empty> == Physical Plan ==

LocalTableScan <empty>

We can observe with the OptimizedLogicalPlan that the exchange on an empty Dataframe is now optimized in Spark3 making the trick inefficient.

In a few words, the optimization comes from the inner representation of an empty Dataframe . It used to be represented with an empty RDD which prevents optimizations such as PropagateEmptyRelation . In Spark 3, it is now using an empty LocalRelation . You can dig more with this commit.

As a workaround, one can now use spark.range(nbExecutors).repartition(nbExecutors) instead of spark.emptyDataframe.repartition(nbExecutors) to trigger a side effect on each machine of a cluster at runtime.

Cassandra driver incompatibilities between third-party libraries

In the latest version of spark-cassandra-connector (3.0), the java driver was bumped to version 4.7.

Be careful before spending time to bump it to make sure all your third party libraries using Spark and Cassandra are up to date with the Java driver version 4.