Using AWS S3 Batch Operations with Lambda in Scala
By: Kirsey Fam
Tags: Technology, AWS Lambda, Scala, S3 Batch Operations, AWS

S3 Batch Operations is a relatively recent feature of Amazon Simple Storage Service (S3) that enables processing of hundreds, thousands, millions, or even billions of S3 objects in a simple and straightforward fashion. It allows for copying objects from one bucket to another, setting tags or access control lists (ACLs), initiating a restore from Glacier, or invoking an AWS Lambda function on each object. AWS Lambda is an event-driven, serverless computing platform that runs pieces of code called Lambda functions.

This post will focus on how to use S3 Batch Operations to invoke a Lambda function written in Scala, with sbt as our build tool. Note: the same result is achievable with other Scala build tools.

Request and Response

When configured correctly, an S3 Batch Operations job sends a JSON request in a particular format to the Lambda function to be invoked on the S3 objects. In turn, it expects the Lambda function to return a JSON response in a specific format indicating the success or failure of the invocation.

Request

The following is an example of a JSON request that is sent from S3 Batch Operations to the Lambda function:

{

"invocationSchemaVersion": "1.0",

"invocationId": "YXNkbGZqYWRmaiBhc2RmdW9hZHNmZGpmaGFzbGtkaGZza2RmaAo",

"job": {

"id": "f3cc4f60-61f6-4a2b-8a21-d07600c373ce"

},

"tasks": [

{

"taskId": "dGFza2lkZ29lc2hlcmUK",

"s3Key": "mediaImage1.png",

"s3VersionId": "1",

"s3BucketArn": "arn:aws:s3:us-east-1:0123456789:sample-bucket"

}

]

}

The most important fields for processing S3 objects in the Lambda function are s3Key , s3VersionId , and s3BucketArn , which are wrapped in a task object. Each task references an S3 object that the Lambda function should process. The other fields are metadata about the S3 Batch Operations job.

At the time of writing, S3 Batch Operations only sends one task per request.

Response

The following is an example of a JSON response that the Lambda function must return to the S3 Batch Operations job:

{

"invocationSchemaVersion": "1.0",

"treatMissingKeysAs": "PermanentFailure",

"invocationId": "YXNkbGZqYWRmaiBhc2RmdW9hZHNmZGpmaGFzbGtkaGZza2RmaAo",

"results": [

{

"taskId": "dGFza2lkZ29lc2hlcmUK",

"resultCode": "Succeeded",

"resultString": "Processed mediaImage1.png"

}

]

}

Response and Result Codes

Amazon S3 Batch Operations expects two different status codes in the response payload. The first is the response code for the entire request, and the second is a per-task result code. The following table contains the response codes and their associated behaviors.

Lambda Request Handler

AWS Lambda Functions require the implementation of an interface that consumes and processes data. The specific details vary between different languages, but for a Lambda function written in Java or Scala, there are two interfaces for handler methods. Those interfaces are documented here.

To handle the S3 Batch Operations request/response format, the RequestStreamHandler will be implemented in this Lambda function. In addition, circe will be used for serialization and deserialization of the request and response.

Dependencies

To use these libraries, the respective dependencies should be added to the build.sbt file.

libraryDependencies ++= Seq(

"com.amazonaws" % "aws-lambda-java-core" % "1.2.1",

"io.circe" %% "circe-core" % "0.12.3",

"io.circe" %% "circe-generic" % "0.12.3",

"io.circe" %% "circe-parser" % "0.12.3"

)

Model

First, the models for the request need to be defined. The definition of these models captures the structure of the data and allows for ease of deserialization with circe. Circe is a popular JSON library in Scala that allows semi-automatic generation of JSON codecs for data types using annotations. The @JsonCodec annotation is used to derive the JSON encoders and decoders for the models defined below.

import io.circe.generic.JsonCodec // Input from S3 Batch Operations to Lambda

@JsonCodec

case class S3BatchRequest(

invocationSchemaVersion: String,

invocationId: String,

job: S3BatchJob,

tasks: List[S3BatchTask]

) // S3 Batch Job details

@JsonCodec

case class S3BatchJob(id: String) // S3 Batch Task model

@JsonCodec

case class S3BatchTask(

taskId: String,

s3Key: String,

s3VersionId: String,

s3BucketArn: String

) {

lazy val s3BucketName: String = s3BucketArn.split(":").last

}

And similarly for the response:

// Output from Lambda to S3 Batch Operations

@JsonCodec

case class S3BatchLambdaResponse(

invocationSchemaVersion: String,

treatMissingKeysAs: String,

invocationId: String,

results: List[S3BatchResult]

) // Output Result for each S3 Batch task

@JsonCodec

case class S3BatchResult(

taskId: String,

resultCode: String,

resultString: String

)

Handler

With the RequestStreamHandler interface, Lambda passes an InputStream and OutputStream to the handleRequest method, which needs to be overridden to implement the interface. The handler method should read bytes from the input stream, do some processing, then write the expected result to the output stream.

import com.amazonaws.services.lambda.runtime.{Context, RequestStreamHandler} class S3BatchOperationsLambdaHandler extends RequestStreamHandler { override def handleRequest(

is: InputStream,

os: OutputStream,

context: Context

): Unit = {

// processing logic here

}

}

This class extends the RequestStreamHandler interface and overrides the handleRequest method as required. In addition, the implementation of this method must handle the request and response format expected from S3 Batch Operations.

import io.circe.parser.decode

import io.circe.syntax._

import software.amazon.awssdk.services.s3.model.S3Exception import scala.io.Source override def handleRequest(

is: InputStream,

os: OutputStream,

context: Context

): Unit = {

val reqString = Source.fromInputStream(is).mkString

val s3BatchRequest = decode[S3BatchRequest](reqString) match {

case Right(batchReq) =>

batchReq

case Left(ex) =>

// if an exception is thrown during the Lambda invocation,

// then the ResponseCode is automatically set to

// "PermanentFailure".

throw ex

} val results = processTasks(s3BatchRequest.tasks)

returnResults(os, s3BatchRequest, results)

}

Here, the handleRequest method reads in the bytes from the input stream as a string, then decodes that JSON string into the S3BatchRequest model defined earlier. Having deserialized the data, the S3 objects that appear in the task list from the request can then be retrieved and processed.

Additionally, helper methods are defined which contain logic for processing the objects and communicating the results back to the job.

def processTasks(

tasks: List[S3BatchTask]

): List[S3BatchResult] =

for {

task <- tasks

} yield {

val res = processObject(task.s3BucketName, task.s3Key)

val (resultCode, resultString) = res match {

case Right(res) => ("Succeeded", res)

case Left(ex: S3Exception)

if ex.awsErrorDetails().errorCode() == "RequestTimeout" =>

("TemporaryFailure", ex.getMessage)

case Left(ex) => ("PermanentFailure", ex.getMessage)

} S3BatchResult(

taskId = task.taskId,

resultCode = resultCode,

resultString = resultString

)

} def processObject(

s3BucketName: String,

s3Key: String

): Either[Exception, String] = {

val path = s"$s3BucketName/$s3Key"

println(path) Right(path)

}

The processObject method is called for each task, receiving the name of the S3 bucket and the key of the target object as parameters. In a real use-case, the object could be retrieved from S3 to perform further processing, such as updating JSON fields or removing sensitive data. To keep things simple in this post, the object’s path is simply printed and returned as the result string.

After the object has been processed, an indication as to whether the operation succeeded or failed is required. Earlier in the post, three possible response codes were listed: Succeeded , TemporaryFailure , and PermanentFailure . If a task is marked as TemporaryFailure , S3 Batch Operations automatically retries that task before the job completes.

The results of the processing are then wrapped in an S3BatchResult object, as defined earlier. The result of each task is then returned in the response to S3 Batch Operations.

def returnResults(

os: OutputStream,

req: S3BatchRequest,

results: List[S3BatchResult]

): Unit = {

output = S3BatchLambdaResponse(

invocationSchemaVersion = req.invocationSchemaVersion,

treatMissingKeysAs = "PermanentFailure",

invocationId = req.invocationId,

results = results

)

outputBytes = output.asJson.noSpaces.getBytes()

os.write(outputBytes)

}

In this method, the results from processing, as well as the metadata relevant to the job, are included in the S3BatchLambdaResponse model. The response is then serialized as a JSON string using circe, the bytes of which are written to the output stream.

At this point, the implementation of the Lambda function is complete.

Deployment

To use the function with S3 Batch Operations, it must be packaged and deployed. To package the code into a JAR, the sbt-assembly plugin can be used. Following setup of the plugin, the project can be packaged by running the following command:

sbt assembly

Now that the project has been assembled into a JAR file, it can be uploaded as a Lambda function via the AWS console or the AWS CLI, as outlined in the documentation.