AWS StepFunctions: Fine Tuning Serverless Workflows Using the Result Selector
By: Robert Bulmer
Tags: Serverless, AWS, Aws Step Functions, AWS Lambda, Cloud Computing

Introduction

In this example, I will be talking about how you can improve your workflows when using StepFunctions by customising the results from AWS Lambda at each step.

AWS recently released, in August 2020, a way to do this using the ResultSelector; read more about this release on the AWS blog here. 🎉

There is another alternative to using the result selector and getting around large data payloads; Writing the data to a file in S3. This option would give you the opportunity to store in S3 the specific data needed for the workload which could be read directly in a Lambda function and therefore mitigate the StepFunctions payload limit.

Firstly, I must admit that this article is not just for serverless applications and I hope that you can utilise this example in any AWS environment you may have. However, I personally believe it leans itself nicely towards a serverless solution for a number of reasons, I’ll explain.

Suppose you have a serverless application that consists of a set of AWS Lambda functions that are invoked by hitting various requests on AWS API Gateway or AWS AppSync endpoints. Wouldn’t it be nice to reuse the lambda functions you have already created and configured inside your StepFunction workloads.

Example System Architecture

In this example, we have a ‘Get Post’ Lambda function that retrieves the data from AWS DynamoDB and we can reuse this same Lambda function inside our ‘Report Post Metrics’ workload. Perhaps we want to fetch the meta data on the post, evaluate the data and release it to another area in the service.

Why Reuse the same Lambda? 🤓

There are many reasons why you would want to use the same lambda function inside the workload. For example, within a serverless application you have account limits that AWS enforce to guide you on best practices and design. One of these default limits currently sets the number of concurrent Lambdas running per account to 1000. This may seem a lot however, consider a large application with high throughput and lambdas scaling out.

Not only have you got such limits within your account, if you have multiple lambdas to retrieve the same data from the datasource you’ll end up with higher inefficiency and extra functions and code to maintain.

So, Why do we need the Result Selector? 💡

As we are reusing our ‘Get Post’ Lambda throughout the application where possible, let’s imagine that the object that the Lambda returns contains all the post details, meta data, authorization access and much more data. For the StepFunction we only need the postId and all or some of the meta data on the object. All of the other information is redundant for this use case.

Up until recently AWS StepFunctions had a payload limit of 32,768 characters, so if you were getting multiple posts, each returning the entire data object, you could quite easily hit this character limit and receive the following error:

ERROR: The state/task returned a result with a size exceeding the maximum number of characters service limit.

Fortunately, AWS release another recent update, in September 2020, to increase this limit to 256kb. Read more here. 🎉

Okay so, AWS have increased the limit… do we still need to worry about the data returned from the lambda?

Maybe not for a small number of posts, but if you’re running a workload for all your posts or another large data set you may still see the error. Plus, it is good practice not to be fetching or logging any data you don’t need.