Unleash Your Serverless Superpowers Using AWS CDK
By: Sagi Eliyahu
Tags: Aws Cdk, AWS Lambda, AWS

Are you still maintaining complicated YAML files for your serverless application?



Whether you’re currently researching various serverless development frameworks for a new application or trying to improve your existing serverless deployment strategy, this post is for you.

There’s a real alternative to the declarative approach for handling serverless deployment (utilized by AWS SAM and Serverless Framework), which we will demonstrate using AWS CDK. In this post we’ll cover:

The major downsides of the more common, declarative approach for dealing with serverless deployments.

Show a real-life example, utilizing AWS Step Functions, to illustrate the power of AWS CDK and the imperative approach. We’ll be using Python despite the fact that CDK for Python is not currently well documented.

It’s important to note that while we will be focusing on AWS serverless platform, but the general concept can be applied to any serverless platform.

Downsides of the Declarative Approach

Serverless technology is in the process of completely transforming the way we think about software development. Unfortunately, during this transformation period we are faced with unique challenges which arise from its very nature. One of the bigger challenges is the increase of deployment complexity. And it is a big challenge — especially when increasingly complex systems are being developed using serverless technology.

Thing is, when we’re maintaining our deployment by following a declarative approach, we are constantly violating some of the most basic and timeless software development principles:

We want to avoid repeating ourselves (aka the DRY principle), but we can’t resolve to even the simplest of practices, such as extracting a method.

We can’t even start talking about creating abstract constructs within our deployment code by introducing classes etc.

SOLID principles (such as the open-close principle) are not even in our realm, since our YAML file is not even a real code

What we’re actually practicing is “Infrastructure as Configuration”, which is maintained as code, rather than actual “Infrastructure as Code”.

Don’t get me wrong, if your serverless deployment is simple enough (and is not about to change anytime soon), or when your deployment configuration is maintained by non-programmers, the declarative approach will probably work just fine.

However, when deployment complexity starts raising, you will probably want to look for a better approach. Here are a few examples:

When your dev team is spending more time automating the deployment of resources in the cloud, than writing business logic code.

When you start incorporating Serverless Framework plugins, such as the popular “If-Else” plugin.

All those issues relate to a single quality attribute, which is flexibility. That’s where the imperative approach and AWS CDK come into play.

A Real World Example

Suppose we wanted to build a system for managing various ops workflows on a production system, like running a backup operation or executing system tests. Let’s also assume that in order to start a backup operation we will integrate with an external backup system. So, we come up with the following design, which implements the callback pattern:

We make a few adjustments to the original example implementation of the callback pattern as it described by AWS (note that you can skip this design notes section, and still benefit from this post):

We design the state machine to handle backup operations exclusively, rather than creating a generic state machine for handling all sorts of tasks. The trade-off is to have better visibility of the system by having separate state machines for each type of operation. That way, we can easily inspect state machine executions (which is a great benefit of Step Functions), will benefit from better visibility of usage metrics for each type of operation, and have easier time analyzing AWS costs more granularly.

We invoke a Lambda function directly from the state machine, instead of using SQS trigger (in which SQS queue serves as an event source for the Lambda function). With this specific use case we don’t require the capabilities which are offered by SQS: there is no point buffering messages on very low throughput (no risk of throttling errors on lambda executions). Also, there is no need for the added resiliency which SQS offers, since we assume that the handling of failed requests is the responsibility of the invoker of the state machine execution.

We add an additional step — the “Fail” step — in order to make sure that when the backup operation fails, the entire state machine execution will fail as well. It not only makes more sense, but it will also give us the added visibility when we troubleshoot failed executions.

Let’s look at how we would develop the deployment of this state machine using AWS CDK:

Deployment of “Start Backup” state machine using CDK

To mix things up, let’s also assume we want to schedule our backup workflow to run nightly. We can do that easily using CDK to deploy a CloudWatch Events rule that will trigger our state machine:

Deployment of CloudWatch Events rule to trigger our backup workflow

When we want to implement additional ops workflows into our system, we create additional state machines - repeating the same callback pattern. For example, a system-test workflow would look very similar to the backup workflow:

Since those are two distinct state machines (“Start Backup” state machine and “System Test” state machine). When we deploy them using a tool that applies a declarative approach, we would need to duplicate and adjust existing sections within the YAML configuration file for each additional workflow we introduce into the system.

With that in mind, let’s look at how we can utilize AWS CDK Custom Constructs to create an abstraction, which we would be able to reuse each time we want to deploy an additional state machine.

What we currently identify as the elements which varies between those state machines are the following:

The state machine name

The lambda function of the actual task

Whether of not we want to trigger the state machine nightly

The configured timeout of the state machine execution

With that in mind, we create a helper class to represent the construct settings:

Help class for the workflow construct

Now let’s create our generic workflows construct, which will be configured according to the injected settings:

Custom construct example

So all we have to do, in order to add additional state machines, is to instantiate workflow construct objects.

For example, in order to deploy the system test state machine, we can do that by simply adding the following lines of code to our stack:

Instantiate a construct for the system test workflow

Assuming we want to deploy a new ops workflow for rotating passwords and configure it to run nightly, we just add the following lines of code to the stack:

Instantiate a construct for the rotate secrets workflow

We can theoretically evolve our design to enhance our custom construct and create whatever our hearts desire!

Try to do that with a declarative approach and YAML files manipulation…

Let’s reiterate on what we just accomplished:

We recognized a pattern within deployment logic and were able to create a generic construct within our deployment code. We were now able to use this construct to generate complicated step functions easily, without rewriting similar deployment code repeatedly.

This is the power of the imperative approach, and it will greatly enhance your serverless deployment capabilities.

With this approach you will be able to improve the maintainability of your deployment code, especially as your serverless applications evolve, and your deployment complexity increases.