Extending CloudFormation using lambda-backed custom resources | by Harry Daniels | Towards Data Science

This article is split into two parts:
Background on CloudFormation and an introduction to Custom Resources.
Monitoring as Code: Managing Datadog monitors with CloudFormation.
If you’d like to skip ahead, all the code discussed in this post is available on my GitHub HERE.
CloudFormation is a fantastic ‘infrastructure as code’ offering from AWS which allows you to efficiently model a collection of resources and manage them throughout their lifecycles. The AWS ‘Spaghetti Bowl’ continues to grow at a rate of knots, meaning even CloudFormation cannot keep up with them all, but do not fear as we’re able to plug any gaps and more using custom resources.
Infrastructure as code is the new normal when it comes to the cloud, it gives you an insight into how your environment came to be and the confidence to update your stack without causing any critical errors. Examples include CloudFormation and Terraform.
CloudFormation provides a common language to describe and provision all infrastructure in your cloud environment, it provisions resources in a repeatable manner, and since everything is written as code can be version controlled. CloudFormation provisions and configures resources by making calls to the AWS services that are described in your template. After all the resources have been created, it reports that your stack has been created and is ready for use. If stack creation fails, CloudFormation rolls back your changes.
Sometimes you’ll want to do something that isn’t supported out of the box by CloudFormation, this could be a manual step post deployment or even provision some infrastructure outside the AWS ecosystem. Fortunately, custom resources make this very easy to do.
Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update or delete a stack. If a Custom Resource has been defined in your template, CloudFormation will send an external request to the resource provider endpoint during a stack operation and wait for a response. Data from the response can subsequently be used to provision and configure other resources. A custom resource can be defined in your Cloudformation template as follows:
CustomResource:    Type: 'Custom::MyCustomResourceTypeName'    Properties:       ServiceToken: RequestAddress      var1: variable1      var2: variable2
Custom resources only require one property: ServiceToken, which will tell Cloudformation where to send the request, however additional properties can also be included. For lambda-backed Custom resources the ServiceToken will be the lambda Arn and Cloudformation will send a request in the form of an event. An example request is shown below:
{    "RequestType": "Create",    "ServiceToken": "lambda-arn",    "ResponseURL": "http://pre-signed-S3-url-for-response",    "StackId": "cloudformation-stack-id",    "RequestId": "request-id",    "LogicalResourceId": "CustomResource",    "ResourceType": "Custom::MyCustomResourceTypeName",    "ResourceProperties": {        "ServiceToken": "lambda-arn",        "var1": "variable1",        "var2": "variable2"    }}
Once a request has been sent, Cloudformation will wait for a response in the pre-signed URL: ResponseURL. The RequestType can be either Create, Update or Delete depending on the type of operation. The Custom resource provider will process the request, perform whatever task you require, and return a response of SUCCESS or FAILED to the pre-signed URL. If a no response or a Failed status is returned the CloudFormation operation will fail and rollback.
{    "Status": "SUCCESS",    "PhysicalResourceId": "function-return-value",    "StackId": "cloudformation-stack-id",    "RequestId": "request-id",    "LogicalResourceId": "CustomResource",    "Data": {        "out1": "output1",        "out2": "output2"    }}
Any values included in the Data field are stored in the pre-signed url location and can be referenced in the template using the !GetAtt function. To retrieve the out1 value from the above output, you would use the following command: !GetAtt CustomResource.out1 .
When a lambda is used as a custom resource provider, the function is invoked whenever the custom resource is created, updated or deleted. CloudFormation invokes the function with the request data (as above) and waits for a response. Fortunately, there are a number of libraries that make writing custom resources very easy. The one I will be using is the custom-resource-helper: a Python-based library provided by AWS that uses decorators.
The Custom Resource Helper is a wonderful package which drastically reduces the complexity of deploying lambda-backed custom resources. The code below can be used as a starting point for lambda development. It can be broken down into the following bullet points:
First the CfnResource class is imported. A CfnResource object is instantiated and called helper.
The create, update and delete decorators determine which function will be invoked for the difference CloudFormation stack actions.
The resource properties, which are defined in the CloudFormation template are available in the ResourceProperties object within the lambda event.
Anything saved in the Data object in the helper object can be referenced by other resources in the CloudFormation template using the !Get Att intrinsic function.
Upon each successful execution of the functions, the helper object takes care of uploading the response to the ResponseUrl as defined in the lambda event.
To truly demonstrate some of the benefits of CloudFormation custom resources, I’ve included a walkthrough of how you might use them to provision monitors in Datadog to send alerts if any of your resources fail. The example below only monitor for failed lambda invocations.
The prerequisites to this walkthrough are as follows:
All the resources in this post have been deployed using the AWS SAM CLI. To follow along, the CLI must be installed and the config file: samconfig.toml must be defined in the root of your project.
version=0.1 [default.deploy.parameters] profile = "personal" stack_name = "datadog-monitor-dev" s3_bucket = "custom-resources-dev-20200318" s3_prefix = "datadog" region = "eu-west-1" capabilities = "CAPABILITY_IAM" confirm_changeset = true tags = "project=\"datadog-monitor-dev\" stage=\"dev\""
Setup the AWS Services integration in your Datadog account and ensure that the lambda tile is selected. This enables Datadog to collect Amazon lambda metrics. Once completed all lambda functions will be available in the Datadog serverless view.
Retrieve the DataDog API key and create an Application key. These must be stored in AWS secrets manager as follows, you can choose any name for the secrets which is added to the CF template as a parameter:
{  "DD_CLIENT_API_KEY": "API KEY",  "DD_CLIENT_APP_KEY": "APP KEY"}
Datadog has a HTTP REST API that allows you to interact with the platform programmatically. The API has been used in the code below to create the Monitor class which will be used by the custom resource lambda. The code below has been summarised into the following key points:
The base url of the API is: https://api.datadoghq.eu/api/v1
To authenticate calls to the API, the datadog API key and Application Key must be provided in the request headers. The keys are used to initialise the class object.
The class has three methods to create, update and delete Datadog monitors, and one helper method to construct a simple monitor query.
The create_monitor method takes the following arguments: name, message, priority, functionname and tags. The name is simply the name of the monitor as it appears in the Datadog UI. The message is what gets sent with each alert. The priority indicates the severity of the alert. The functionname is used by the _create_query method to construct a simple monitor query and the tags are used to tag the monitor. After a monitor is created its monitor_id is returned.
The _create_query method uses the functionname argument to create a monitor query which checks for any lambda errors within the last hour.
The update_monitor method takes all the same arguments as the create_monitor method, except with the addition of the monitor_id so it knows which monitor to update.
The delete_monitor method uses the monitor_id to delete the monitor.
Using the Datadog Monitor class and also the Custom resource skeleton code defined above, it is now possible to create a Custom resource which can create, update and manage Datadog monitors in tandem with CloudFormation stack operations.
As mentioned in the prerequisites, the Datadog API and App keys are stored in AWS secrets manager. These are retrieved using the aws_lambda_powertools package.
The CfnResource and DD_monitor objects are created.
The create_monitor method is called within the create function which is executed when the stack is first created; as indicated by the @helper.create decorator. The required arguments are defined in the CloudFormation template and available in the ResourceProperties field in the event object.
The update_monitor and delete_monitor methods are called within the update and delete function respectively.
Both the create and delete functions return the monitor_id which will be assigned as the physical id of the resource.
The test lambda is the lambda which will be monitored in Datadog. The lambda will only be used to generate some successful and failed invocation metrics.
With the all the lambdas now defined, the next step is to create the CloudFormation (SAM) template to deploy them all. The template explicitly defines three resources, the DataDogMonitorLambda, aTestLambda, and the TestLambdaDDMonitor .
The DataDogMonitorLambda is the lambda function that will be called by the custom resource. This is evident as the ServiceToken in the custom resource points to the DataDogMonitorLambda Arn.
The TestLambda is the lambda that will be monitored.
The TestLambdaDDMonitor is the Datadog monitor that is created by invoking the DataDogMonitorLambda. The arguments required to create the monitor are passed as additional properties.
To deploy the stack with SAM, execute the following commands:
sam build --use-container
sam deploy
Check everything looks right in the changeset and press y to continue with the deployment.
Once the stack creation is complete, all the resources will have both Logical and Physical IDs. Since the create function in the Custom Resource lambda returns the monitor_id of the Datadog monitor it just created, this is set as the resource Physical ID. The monitor can be viewed in Datadog at the following address: https://datadoghq.eu/monitors/{MonitorID}.
Initially the Datadog monitor will have no data as shown below, this is because the monitor requires either successful or failed lambda executions.
The easiest way to generate some data is to manually invoke the TestLambda in the AWS UI. Since the monitor only checks for failed invocations each hour, its best to invoke the lambda successfully a few times and wait for the data to populate in Datadog and then cause the lambda to fail.
Finally, if the monitor is working as expected, after a failed lambda execution, it should remain in an alert state.
I hope you enjoyed reading this post as much as I enjoyed writing it. The simplicity of Lambda Functions and Custom Resources really open up the possibilities of CloudFormation. I’m planning on writing another article of how you might use Custom Resources to manage your Snowflake account, so watch this space!
Checkout my other engineering posts:
Connecting to an ec2 instance in a private subnet on AWS