Dynamic data file compaction in Apache Spark
By: Himani Gadve
Tags: Delta Lake, Small File Issue, Python, Apache Spark, Data Compaction

Summary

This document illustrates a strategy required for the small file problem identification and mitigation in Apache Spark. This document describes when you need small file compaction and workaround to generate large data files dynamically on the Apache Spark framework.

Background and Motivation

This document serves the purpose of dynamic data compaction using the delta table in the Spark API. Dynamic compaction is implemented for partitioned as well as unpartitioned data.

What is a small file issue

Apache Spark takes too much time reading a dataset on s3 where it takes a while to even see the executors launched.

This is because it is the driver that performs the listing of files in the dataset, estimating the size/partitions and then it distributes the actual processing to executors.

Need for dynamic compaction of partitioned and unpartitioned delta files

Spark has to pull those files over the network, it also has to uncompress them. Each file then becomes one partition in the resulting resilient distributed dataset (RDD), which end up with an RDD with more than a million tiny partitions.

Having too many small files could lead to a performance bottleneck and in worst cases, the driver could fail with Out-of-Memory exceptions

To overcome a small file issue in Apache Spark, dynamic compaction of data files is required.

How small file issue arises

The data ingestion job will append a new batch of data to the destination location hourly which will have at least one new file. File size depends upon how much data we collect in that one hour. It could be 1 GB, 100 GB, or even just a few MBs. Data Ingestion job creates hundreds of small >1 GB files every hour which creates too many small files. Small files are neither efficiently handled by the storage systems nor it can be efficient for the Spark because the Spark API would internally need to query the storage system such as s3 to get the list of files. A large number of files would slow down the listing operation. Spark would also need to open all of these files and close them after reading them. This will result in the job spending a lot of time opening and closing files rather than reading data. A small file doesn’t compress well due to reduced compression effectiveness. If the Spark table, that means, you are storing metadata about the table which grows significantly for a large number of files. That will again slow down our job. So, a large number of small files is a one of the problems.

Problems due to small files

Besides creating insufficient storage, small files affect the performance of the job. The reason being small files results in more disk seeks while running computations through these compute engines. In Spark, the task within an executor reads and processes one partition at a time. Each partition is one block by default. Hence, a single concurrent task can run for every partition in a Spark RDD which means that if there are a lot of small files, each file is read in a different partition and this will cause a substantial task scheduling overhead compounded by lower throughput per CPU core.

What is a compacted data:

Data is well compacted if it is stored in a reasonable number of files relative to its storage size. Example of efficiently compacted file storage is, store 100 MB of data in just 1 file. On the other hand, we could store 100 MB of data across 1000 files which is an example of poorly compacted data. The difference is in the number of files generated to store the same volume of data.

Small files also cause a major performance hit for read processing as the consumer process will need to spend additional handles for opening/closing of many more files than is optimal for reading. To overcome a small file issue dynamically repartition data before writing it to data lake AWS S3.

Solution for small file issue

Repartition data before writing varies for different volumes of data per partition. it should vary as per the data size. It is not a practical solution to write each file of the same size in each partition. For example, in partition date_year=2019 if it creates 20 partitions of 20 MB each, the same number of partitions is not practical way to apply for another partition such as date_year=2020 having total file size of 20 GB which will create each file of 2 GB.

Adapt dynamic partitioning for each compacted file as per the unique file size for each partition to accommodate below user requirements

Getting Started:

Assumptions:

User will provide minimum and maximum file size in Megabytes desired for each file size after compaction In repartition, by default shuffle partition is 200 which physically rearranges and shuffles data from several data files and hence the total size after repartition may get higher or lower compared to the total size of data before repartition. Therefore, the expected file size will be approximate but not exact megabytes. This script assumes that the input directory ‘partitionedpath’ contains subdirectories that actually have the partitioned delta table files along with delta log which stores the metadata of delta operations. S3 bucket full access to read and write data.

Step 1: Find the partition size

This method has a purpose to get the number of repartitions based on the size of data files in each partition. Casting column datatypes based on the Method requirement — Using MapType to accommodate multiple partitions based on Key:Value pairs in descending order of partition keys

In this method, I am collecting the snapshot of the delta log for columns such as path, partitionValues, size as path is the actual delta data files, size having a unit of bytes of data for the specific partition.

While listing, I am finding the size of each object and then returning a Map<Partition-name, Partition-value> to the caller. Outcome of this method looks like below

I want file size between minimum and maximum size in MB in the respective partition.

This method takes five arguments

DeltaLog DataFrame : It contains snapshot of DeltaLog having size, partitionValue, and Path as a data file

PartitionKey: It is the partition column name, example: date_year

minFileSizeInMB: User provided minimum file size in Megabytes

maxFileSizeInMB: User provided maximum file size in Megabytes

minNumOfPartitions (default): Its defaults value is 1 if the calculated file size is less than minimum file size required by user then this will be returned by to the caller.

def withBytesToMegabytes(bytesColName: String, outputColName: String = "megabytes")(df: DataFrame): DataFrame = {

df.withColumn(outputColName, col(bytesColName) / lit(1048576L))

}



def rePartitionOnMinAndMaxInMB(deltaLogDF: DataFrame, partitionKey: String, minFileSizeInMB: Long, maxFileSizeInMB: Long, minNumOfPartitions: Int = 1): DataFrame = {

deltaLogDF

.withColumn("partitionValue", col("partitionValues")(partitionKey))

.groupBy("partitionValue")

.sum()

.transform(withBytesToMegabytes("sum(size)"))

.withColumn(

"number_of_partitions",

when(

col("megabytes") < minFileSizeInMB,

minNumOfPartitions)

.when(

(col("megabytes") > minFileSizeInMB && col("megabytes") < maxFileSizeInMB), (floor(col("megabytes").cast(DoubleType) / minFileSizeInMB.toDouble)).cast("int"))

.otherwise(ceil(col("megabytes").cast(DoubleType) / maxFileSizeInMB.toDouble).cast("int"))

)

}

​​​​I want fixed file size in the respective partition.

This method takes four arguments

DeltaLog DataFrame : It contains snapshot of DeltaLog having size, partitionValue, and Path as a data file

PartitionKey: It is the partition column name, example: date_year

FixedFileSizeInMB: User provided fixed file size in Megabytes

minNumOfPartitions (default): Its defaults value is 1 if the calculated file size is less than minimum file size required by user then this will be returned by to the caller.

def withBytesToMegabytes(bytesColName: String, outputColName: String = "megabytes")(df: DataFrame): DataFrame = {

df.withColumn(outputColName, col(bytesColName) / lit(1048576L))

}



def rePartitionOnFixedFileSizeInMB(deltaLogDF: DataFrame, partitionKey: String, FixedFileSizeInMB: Long, minNumOfPartitions: Int = 1): DataFrame = {

deltaLogDF

.withColumn("partitionValue", col("partitionValues")(partitionKey))

.groupBy("partitionValue")

.sum()

.transform(withBytesToMegabytes("sum(size)"))

.withColumn(

"number_of_partitions",

when(

col("megabytes") < FixedFileSizeInMB,

minNumOfPartitions)

.otherwise(ceil(col("megabytes").cast(DoubleType) / FixedFileSizeInMB.toDouble).cast("int"))

)

}​

I want each partition to have a provided number of data files.

This method takes three arguments

DeltaLog DataFrame : It contains snapshot of DeltaLog having size, partitionValue, and Path as a data file

PartitionKey: It is the partition column name, example: date_year

numofPartitions: User provided fixed number of repartition file in each partitioned column

def withBytesToMegabytes(bytesColName: String, outputColName: String = "megabytes")(df: DataFrame): DataFrame = {

df.withColumn(outputColName, col(bytesColName) / lit(1048576L))

}



def rePartitionOnFixedNumOfPartition(deltaLogDF: DataFrame, partitionKey: String, numofPartitions: Int = 1): DataFrame = {

deltaLogDF

.withColumn("partitionValue", col("partitionValues")(partitionKey))

.groupBy("partitionValue")

.sum()

.transform(withBytesToMegabytes("sum(size)"))

.withColumn(

"number_of_partitions", lit(numofPartitions))

}

These methods returns the number of repartitions along with size and the partitionValue, its outcome looks like below-

partition value sum(size) is in Bytes, converted to MB size number of repartition in integer

Step 2: Rewriting data in each partition as a compacted large files

Above outcome as a DataFrame is passed as input to the method to perform the write operation on delta table per the number of repartitions calculated for each unique partition This method takes four arguments-

DataFrame: It contains the structure of data having partition value, sum(size) is in Bytes, converted to MB size, number of repartition in integer

TargetPartitionKey: Partition key provided by the user, example: ‘date_year,date_day’

PartitionKey: “colon : separated” Partition key of the data, example: date_year:date_day

PartitionedPath: Target directory where compacted files will be written

def rewriteDataPartitions(writeDf:DataFrame, targetPartitionKey: String, partitionKey: String, partitionedpath: String): Unit = {

val targetPartitionKeycolList = targetPartitionKey.split(",").toList

val targetdf = spark.read.format("delta").load(partitionedpath)

validationLogDF = writeDf.withColumnRenamed("sum(size)","size")

// joining DataFrame to get partition values back which will be given as input in partitionBy column

val partitionvValDf = writeDf.join(explodedf, writeDf("partitionValue") === explodedf("partitionVals"),"left").drop("size","megabytes","path","partitionVals","sum(size)").distinct() val parlen = partitionKey.split(":").length val result = for (lst <- partitionvValDf.distinct().rdd.collect.map(x=>x.mkString(" "))) {

var filterCond = lit(true)

var filterWhere = lit(true)

var numofpartition: Int = 1

for (i <- 0 to parlen-1) {

numofpartition = lst.split(" ")(1).toInt

val partkey = partitionKey.split(":")(i)

val partval = lst.split(" ")(0).split(":")(i)

val addedCondition = col(partkey) === "%s".format(partval)

filterCond = filterCond && addedCondition

filterWhere = filterWhere && "%s=%s".format(partkey,partval)

}

targetdf.where(filterCond).repartition(numofpartition).write.partitionBy(targetPartitionKeycolList:_*).format("delta").mode("overwrite").option("replaceWhere", "%s".format(filterWhere)).option("dataChange", "false").option("overwriteSchema", "true").save(partitionedpath)

}

vacuumAndSymlinkManifest(partitionedpath)

}