AWS Serverless Provisioning: Terraform + Serverless Framework
By: Jorge Luis Vargas
Tags: AWS, Terraform, Provisioning, DevOps, Serverless

Law of the instrument

You may be wondering now: Why do you two different tools for IaC, particularly if Terraform is already quite Robust and you can deploy Lambdas using Terraform?

Terraform is an incredibly robust tool, and you can use it to deploy all your infrastructure. You can specify at a very granular level configurations and settings for each specific resource, associate the resource with other resources, modularize your IaC, between other things.

resource "aws_lambda_function" "test_lambda" {

filename = "lambda_function_payload.zip"

function_name = "lambda_function_name_${var.environment}"

role = aws_iam_role.iam_for_lambda.arn

handler = "exports.test"

source_code_hash = filebase64sha256("lambda_function_payload.zip")

runtime = "nodejs12.x"

environment {

variables = {

foo = "bar"

}

}

}

lambda.tf - Terraform Lambda declaration

Depending on your use case I would recommend utilizing a single tool to reduce complexity. But In this case, and especially when working on Serverless projects, it’s more convenient to use a more optimized tool like Serverless Framework for certain parts of your workflow.

Serverless Framework is much better and faster with Lambda/Serverless workflows. And covers an area that Terraform doesn’t cover too well, the deployment of Lambda functions and associated resources. While a Terraform deployment can take up to 15–20 minutes (this will depend of course a lot on what you’re deploying, and which services you use, some take longer than others), Serverless Framework takes a couple of minutes.

service: sms-api

provider:

name : aws

runtime : nodejs12.x

region : us-east-1

functions:

send:

handler: src/index.send

description: API that sends SMS to a specified recipient

environment:

region: ${self:provider.region}

events:

- http:

path : "/sendMessage"

method: post

serverless.yml — Serverless API Declaration

In the example above we are not only declaring a Lambda function, but also an API Gateway and all the roles and permissions required for the API Gateway to communicate with the Lambda. This means, this tool additionally automatically creates and connects components associated with your Lambda functions, so it simplifies drastically the need to define complicated and long resources as code.

Which tool should you use?

After using both tools extensively I’ve discovered that these tools excel at different scenarios. That means before deciding from one of the other, decide which kind of application you’re building and which resources you’ll need to provision.

Terraform

Doesn’t need to change very often

Doesn’t change with code/implementation

Used by many projects

Data can’t be regenerated (Database, files, etc.)

Highly customized/specific configurations

Serverless Framework

Highly coupled to Serverless project or application

Has Lambda functions

Coupled to code/implementation

Data can be regenerated (Or re-populated)

Need to deploy very fast or very often

Knowing the weaknesses and strengths of these tools can help us assess how to effectively manage our cloud resources. You can now decide which tool suits better your project or new resource.

Resource Ownership

What I explained above sounds great, and if you create stand-alone systems probably you won’t have any issues. But if you create systems that interact with each other, like microservices, eventually until you run into this scenario:

System 1 is managed by Serverless Framework, while System 2 is managed by Terraform and now they need to communicate with each other via a shared resource.

So which system should own this resource?

This will be a fairly common scenario if you use more than one tool to provision cloud infrastructure, and as I mentioned before, sometimes it’s better to use a single tool.

There are multiple ways of approaching this situation, so I can’t give you a single answer for the example stated above, as there are cons and pros of declaring the shared resource as part of one system or the other. On top, there are other factors like your integration architecture and dependencies, but you may also have specific business requirements that will influence the decision.

Although, a good rule of thumb to define your resources ownership is to keep your resource as part of a system that:

Connectivity : Needs constant access to it

: Needs constant access to it Hard dependencies : Breaks without it

: Breaks without it Business domain: The resource will process only certain domain data associated with the system.

Connecting the dots

Once you have defined where the new resource will go and which project and tool will manage it, you will still need to figure how to connect these different systems.

Luckily for us AWS already supports a simple yet powerful reference system, by using Amazon Resource Names (ARNs) we can easily configure references between the different projects:

Example of Terraform — Serverless Framework resource reference

So how would it look like in practice? Well it depends on who owns the resource, but in any case, it’s very similar:

Terraform owns the resource

In this case, you just declare your resource on Terraform:

resource "aws_s3_bucket" "awesome_bucket" {

bucket = "s3-awesome-bucket"

...

}

s3.tf

And then, simply reference it on Serverless Framework, using its ARN:

service: awesome-api

provider:

name: aws

...

functions:

run:

handler: src/index.run

description: API that is Awesome!

environment:

s3-bucket: arn:aws:s3:::s3-awesome-bucket

serverless.yml

Configuration Service

A most scalable alternative to this approach, is using an intermediate configuration manager, to avoid having to hardcode values or passwords. For those cases you can use intermediate tools like the AWS Parameter Store or AWS Secrets Manager that are supported out of the box by both tools.

resource "aws_ssm_parameter" "awesome_s3_bucket_arn" {

name = "s3-awesome-bucket-arn"

description = "Awesome S3 Bucket ARN"

type = "String"

value = aws_s3_bucket.awesome_bucket.arn

}

ssm.tf

And can be easily referenced from the Serverless Framework configuration:

...

environment:

s3-bucket: ${ssm:s3-awesome-bucket-arn}

serverless.yml

Serverless Framework owns the resource

Serverless Framework uses a CloudFormation Stack to create and manage the resources. You can output the values to a CloudFormation stack output to keep it clean and avoid hardcoding values. But you can of course use just hardcoded ARNs.

service: awesome-api

provider:

name: aws

...

functions:

run:

handler: src/index.run

description: API that is Awesome!

resources:

Outputs:

LambdaFunctionQualifiedArn:

Export:

Name: LambdaFunctionQualifiedArn

serverless.yml

This is particularly useful in cases in which you need a specific versioned ARN, like Lambda qualified versions, or values that are generated on deployment time.

data "aws_cloudformation_export" "awesome-api-qualified-arn" {

name = "LambdaFunctionQualifiedArn"

}

data.tf

And then you can use the data anywhere else in your project referencing the data value:

...

lambda_arn = data.aws_cloudformation_export.awesome-api-qualified-arn.value

cloudfront.tf

Caveats

It’s worth mentioning, that there are a few caveats with integrating these tools.

First, references should be accessible to Serverless Framework, which means that you need to give the deployment AWS role, extra permissions to access, read, or associate the external resource(s). Not giving it these permissions could make the tool fail.

Second, Terraform may overwrite some configurations for resources that it manages. This is an issue with how Terraform manages its state , in which any resource out of its persisted state, is not visible for the tool and it will overwrite any other configuration not specified in Terraform. In those cases you may have to duplicate any information on both projects to avoid Terraform replacing the values, or you can abstract that configuration to a shared configuration tool, so there’s a single source of truth for both tools.

Final thoughts

What I shared above are just the hard learnings from the past couple of years, after working with the tools mentioned. There may be some other approaches, although I’ve found this is a structured and scalable approach that will save you a lot of pain and refactoring.

Hopefully, you’ve learned more about Infrastructure as Code, and how to handle these more advanced scenarios in which you need to orchestrate and handle complex references between systems and projects.

Please let me know if you have any comments or other ideas about these amazing tools like Terraform and Serverless Framework, which are making provisioning, configuring, and deploying cloud infrastructure, a much simpler and more organized process, than what it used to be.