Merging or Rolling up multiple intraday files in an HDFS directory to daily files using Scala and Spark
By: Sujesh Chirackkal
Tags: Spark, Hdfs, Scala, Merge

Data engineering pipelines create hundreds or thousands of files every day in its raw or processed storage layers. Often it creates lot of small files too in every directory. Sometimes it is necessary to merge many of these files with some conditions and make the directory look clean and also help optimising the read operations thereafter.

Hundreds of files get generated in an HDFS directory every day as there are intra day jobs writing output to a folder and a regular roll up or merge job need to run and combine those files based on its timestamp. As an example, if there are 10 files created / modified at 2020–10–26 with a naming pattern part-*.csv, all of them need to be merged to form a file 20201026.csv.

Here is the code snippet that would help to achieve this. Solution demanded the use of Scala and Spark to achieve this outcome. The code snippet is testing with `Spark 3.0.0` and Scala version `2.12.10`

Disclaimer: This may not be the best method to solve this problem. Feel free to provide the optimisations or alternatives.

import org.apache.hadoop.conf.Configuration

import org.apache.spark.sql.SparkSession



import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}

import org.apache.spark.SparkContext

import org.joda.time.format.DateTimeFormat





object MergeFiles {



def main(args: Array[String]): Unit = {

val spark = SparkSession.builder()

.appName("Merging files day wise in a directory")

.master("local[2]")

.getOrCreate()



val inputDir = "/Users/myhome/test_data"

val outputDir = "/Users/myhome/output_data"



val hadoopConf = spark.sparkContext.hadoopConfiguration

val fs = FileSystem.get(hadoopConf)



val filesPerDate = getFiles(inputDir, fs)



filesPerDate

.foreach { m =>

spark

.read

.format("csv")

.option("inferSchema", false)

.option("header", false)

.load(m._2:_*)

.repartition(1)

.write

.format("csv")

.save(s"$outputDir/${m._1}")



val file = fs.globStatus(new Path(s"$outputDir/${m._1}/part*.csv"))(0).getPath.getName

fs.rename(new Path(s"$outputDir/${m._1}/$file"), new Path(s"$outputDir/${m._1}.csv"))

fs.delete(new Path(s"$outputDir/${m._1}"), true)

}

}



/*

Get the list of files group by date

date is taken from file's modification timestamp

*/

def getFiles(dir: String, fs: FileSystem) = {

fs

.globStatus(new Path(s"$dir/part*.csv"))

.map { f: FileStatus =>

(DateTimeFormat.forPattern("yyyyMMdd").print(f.getModificationTime), f.getPath.toUri.getRawPath)

}.groupBy(_._1)

.map { case (k,v) => (k -> v.map(_._2).toSeq) } // convert tuple to Map

}

}

First step is to the get the list of files per date as a Map of type (Map[String, List[String]]) where key is Date and value is list of files with a same date. Date is taken from the modification timestamp of the HDFS file.

Note: Tested the code using local path, users have to give the right HDFS path / url as required.

While writing the output, as there is no direct option to specify the target filename, but you can specify the target directory specific to each date. Code makes use of FileSystem APIs to rename the file to the desired target name and delete the temp output folders created per date.

Summary

The code snippet need to be tuned and modified according to the requirements in every specific use case. It could be across multiple input directories or within partitioned directories etc. The search patterns and File rename snippet would require appropriate changes. In short, this is a skeleton to start with.