Emulating the move functionality in S3 using Spark
By: Shitij Goyal
Tags: Apache Spark, Aws S3, Hadoop, Technology, Big Data

I was recently working on a scenario where I had to move files between buckets using Spark. Now, Spark does not have native support for S3 but uses the Hadoop FileSystem API to treat S3 as a distributed FileSystem. The API is pretty good and can handle most of the use cases of S3 from Spark pretty well, with many methods provided to solve frequent use cases. But surprisingly there is not direct method to move files from one bucket to another. It has to be done using the copy() function provided and has some intricacies which must be kept in mind before starting the movement across buckets.

The difference between move and copy is that move transfers the data from source to destination while copy keeps the original data at the source and only makes an additional copy at the destination.

The first thing to remember in Spark while moving files is the configuration that must be done for the FileSystem. The FileSystem has to be declared as s3native and the Access keys and Secret keys for AWS have to be entered. The “mapreduce.fileoutputcommitter.algorithm.version” parameter is optional, but it highly recommended to use this as it speeds up the write process from Spark to S3. Before this parameter, Spark used to write the data from executor to the driver and then from the driver to S3. This parameter bypasses the transfer to driver and directly writes from executors to S3, increasing parallelism many folds and consequently the speed of writing.

sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")

sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.awsAccessKeyId", "XXXXXXXXXXX")

sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.awsSecretAccessKey","XXXXXXXXXXX")

sparkSession.sparkContext.hadoopConfiguration.set("mapreduce.fileoutputcommitter.algorithm.version", "2")

Once the configuration has been set, it is time to code the process. Here each bucket of S3 is treated as a separate FileSystem and thus the FileSystems to each bucket have to be created separately. An example of rawPath could be “s3://test-bucket/data”. The FileSystem.get() method expects the raw path and the configuration which we declared above. A thing to remember while using the copy() method is that it copies one file at a time. Hence you have to give the full source path of a file and the full destination path of the file you want to move. This can be done by listing the files in the source FileSystem, iterating over them and storing their paths in a List. Now it’s trivial to set the destination path by iterating over the paths above and replacing the source bucket name with the destination bucket name. This ensures that the destination bucket has the same directory structure as the source bucket. Finally, the copy() method is called for each source path and the relevant parameters are passed to it as shown below.

Even though the parameters given to copy() are pretty clear, I’ll explain what the ‘deleteSource’ and ‘overwrite’ parameters are for. ‘deleteSource’ deletes the files from the source bucket once the files have been copied and ‘overwrite’ replaces any previous file with the same exact path in the destination bucket.

def copyS3DataBetweenBuckets(sparkSession: SparkSession, rawPath: String, processingPath: String, deleteSource : Boolean, overwrite : Boolean): Unit = {

import org.apache.hadoop.fs.{FileSystem, Path}



val srcPath: Path = new Path(rawPath)

val srcFs = FileSystem.get(

srcPath.toUri, sparkSession.sparkContext.hadoopConfiguration)

val dstPath: Path = new Path(processingPath)

val dstFs = FileSystem.get(

dstPath.toUri, sparkSession.sparkContext.hadoopConfiguration)



val paths: ListBuffer[Path] = ListBuffer()

val fileStatusListIterator: RemoteIterator[LocatedFileStatus] = srcFs.listFiles(new Path(rawPath), true)

while (fileStatusListIterator.hasNext) {

val fileStatus: LocatedFileStatus = fileStatusListIterator.next()

paths += fileStatus.getPath

}



paths.foreach { path =>

val finalPath = processingPath + path.toUri.getRawPath



FileUtil.copy(srcFs, path, dstFs, new Path(finalPath), deleteSource, overwrite, sparkSession.sparkContext.hadoopConfiguration)

}

}

Hadoop FileSystem in Spark emulates the move() option across S3 buckets by utilising the copy() function. Though a little prior setup has to be done to ensure the move behaviour, the end result is completely satisfactory.

P.S — Ensure the hadoop-aws jar needed for the FileSystem API is above 2.7.0, if setting the “mapreduce.fileoutputcommitter.algorithm.version” parameter to 2.

For more stories check out my publication- https://medium.com/polar-tropics