Testing AWS Lambdas Locally with a HTTPS Proxy
By: Grant Seward
Tags: AWS, Development, AWS Lambda, Programming, Developer

AWS Lambda is generally one of the easiest ways to deploy and execute code in the cloud, especially when deploying code with the sam CLI. The simplicity of serverless resource definitions coupled with the ability to package resources locally and ensure they run on AWS provides a beautiful development experience.

Except sometimes this beautiful process can turn into a beast of a deployment when the build and package steps grow to ten, fifteen or (gasp) more minutes. There are a few places where the sam build / sam deploy paradigm breaks down and starts to cause runaway deployment times:

The number of Lambdas in a deployment becomes large (e.g. to more than half a dozen)

Some or all of your Lambdas have a large number of resources to install and / or a large Lambda package size (increasing time to build and for sam to put the code on S3)

You are making coordinated changes to two Lambdas at once (multiple stacks need to be updated in AWS before you can test)

Multiple people are developing on the same Lambda stack at the same time

Your internet is slow (again, extending the time to place Lambda packages in S3)

Since sam deploys all resources in a CFT, you cannot selectively choose which ones to incrementally update. The impact is that you need to build and upload all of your code to AWS for each deployment that you want to test in the cloud. This becomes particularly annoying when you push code to AWS only to realize that you misspelled a string and you need to redeploy the entire stack.

Photo by Andrea Piacquadio from Pexels

Yes, sam does provide a way to invoke lambdas locally, but this does not help when you want to test coordinated changes between Lambdas. With sam You cannot deploy two lambdas locally and route traffic between them (as far as I know). Sure, there may be better ways to structure and deploy your code to prevent the monolithic deployment issues but there will always be tradeoffs. How much functional code should go into a single Lambda? How to choose when to separate functions into different stacks? How to organize repositories around stacks?

What can be done if you have long package and deployment steps but you want to speed up the time it takes to build and test your Lambdas? Or if you have lambdas from different stacks that invoke each other and you’d like test them locally at the same time?

Reroute Traffic with a Proxy

The basic principle behind rerouting traffic with a proxy is that we want to capture the in-flight request that is going to AWS and then send it back to our local machine instead of allowing it to go to the cloud. This process is generally called Man in the Middle (MITM). The name gets a bad rep for being widely used for hacking but it has practical use in helping to reduce the time it takes to test Lambdas.

Basic Man in the Middle Process (Image by Author)

The key here is that we will only want to reroute pre-configured endpoints back to our local machine. If, for example, we are deploying a stack that has five Lambdas and we’re only making changes to one of those Lambdas then the traffic for the other four should continue to go to AWS.

Capture Traffic Going to AWS

When you invoke an AWS Lambda, with boto3 or another AWS package, you are just calling an API via HTTPS; the AWS package handles the authentication, endpoint creation, and serialization for you. Because all of this traffic is sent via HTTPS you can view all your machine’s HTTPS outgoing traffic through a proxy!

If you’re on a Mac you can update your HTTPS proxy settings under system preferences and then network > advanced > proxies . Here I have turned the local proxy on for port 9090.

(Image by Author)

You can see the traffic flowing through the traffic yourself if you download a tool such as Proxyman and then invoke lambdas from your local machine.

Screenshot of HTTPS request to AWS as seen in Proxyman (Image by Author)

Match Traffic From Endpoints to Reroute Them Locally

Now that the HTTPS traffic is flowing through our proxy we have the ability to manipulate it. This is done with the man-in-the-middle program. This MITM will read all events that come through the proxy and optionally apply some action to them. This is the key to testing our Lambdas locally.

I am using mitmproxy, a super powerful tool that allows scripting logic to be applied to any any records — aka flows — that come through the proxy. The mitmproxy cli accepts a python file as an argument where it will feed HTTPS events to the python script to take action. Here is an example of a script that routes all traffic from google to bing and from reddit to medium.

This code can be run with the command (note — 9090 is the port that my HTTPS proxy is running on):

mitmdump -p 9090 reroute_traffic.py

Rerouting Lambda Traffic

You may have noticed in the screenshot above that all of the requests being made to AWS follow the same naming convention; the region, lambda version date, and lambda function name are concatenated to build the endpoint. When you run sam local start-lambda … a similar process occurs locally! AWS creates a local endpoint on your machine that maps to a specific port. You can even specify that port like this:

sam local start-lambda -p 54321 --template /path/to/template.yaml

Your Lambda will be running locally and you can see the local endpoint:

Thus far we have learned how to capture and reroute traffic with a man in the middle as well as how to start a lambda locally and select the port it runs on. The only thing left to do is to build a map that points from the endpoint in AWS back to our local port where this Lambda is running. This is where things may get tricky for you. When sam starts a lambda locally you can only invoke the lambda with the resource name but as we saw above the Lambda URL contains the function name.

CFT Lambda Definition (Image by Author)

I always use a standardized naming conventions for my Lambdas and my team is very strict about following it, as you can see above that naming convention is “{StackName}-{ResourceName}-{Env}”. This allows me to simply strip the stack name and environment from the function name when routing traffic to the locally running Lambda. This allows the man-in-the-middle script to replace flows using this basic logic!

reroute_flow = {

'from_route':'my-stack-MyLambda1-dev',

'to_route': 'MyLambda1'

}

mf = MatchedFlow(**reroute_flow)

FM = FlowMatcher([mf])

And voila! Once plugged into the man-in-the-middle script, we are now routing traffic to our local Lambdas. We can even have multiple reroutes defined in order to route traffic between local versions of our Lambdas!

reroute_flow1 = {

'from_route':'my-stack-MyLambda1-dev',

'to_route': 'MyLambda1'

}

reroute_flow2 = {

'from_route':'my-stack-MySecondLambda-dev',

'to_route': 'MySecondLambda'

}

mf1 = MatchedFlow(**reroute_flow1)

mf2 = MatchedFlow(**reroute_flow2)

FM = FlowMatcher([mf1, mf2])

LocaLambda — A CLI to Simplify Things

There is a lot to do in order to test Lambdas locally: setting up the HTTPS proxy, defining matching rules, serving Lambdas locally, etc. LocaLambda, or lola for short, is a lightweight CLI that is here to simplify the process. Lola provides three functions:

Setup : set up the required resource and configuration files in order to get lola to run, this is a one-time activity after installing lola

: set up the required resource and configuration files in order to get lola to run, this is a one-time activity after installing lola Build : lola will build the specific resources defined in your lola.yaml file. It does this by replicating the actual template.yaml file with a slimmed down version that only contains the specified resources

: lola will build the specific resources defined in your lola.yaml file. It does this by replicating the actual template.yaml file with a slimmed down version that only contains the specified resources Serve: lola runs the sam local start-lambda... for each Lambda defined in your configuration file, keeping track of the specific port used. It then will automatically start your local HTTPS proxy server and execute the man in the middle script. It should be noted that the man in the middle runs in a different executable than lola. In order to tell the man in the middle what matching logic to apply it serializes the mappings for the locally deployed resources and places them on Redis (which you must have running locally as well).

LocaLambda Setup

LocaLambda is available on pypi and can be installed with:

pip install localambda

When first installed, LocaLambda needs to setup a few resource and config files that will be specific to your machine. You can have LocaLambda guide you through the one-time setup with:

lola --setup

The output of this is a .lolarc resource file located in your home directory and a new lola directory located in the location of your choice. You will need to create a lola.yaml file, such as the one below, in the lola directory that will tell LocaLambda which resources to build and serve. This file can also be found in the GitHub repo under the examples section.

A lot of these configurations are based off of what we use and it is likely that you’ll have slightly different needs. Because of the standardized naming convention I use for Lambdas, I only need to tell lola about three things:

The location of the template.yaml file: this is a concatenation of repo_home and location, or a fully qualified location can be provided. This should be the directory where the code and template are located. The stack name: lola needs to strip the stack name and environment name that are in the HTTPS request going to AWS in order to find the specific resource name. The environment is going to be static (dev), so that is hard coded for now. Resources: a list of resource names in the CFT template.yaml file that should be deployed or served. If you have ten lambdas in your CFT this is where you can select one, two or however many to build and test instead of having to build and deploy all ten.

Running lola

LocaLambda does two main things: it builds resources and serves them locally. You can build and serve lambdas with lola independently or you can do both at the same time. Building simply requires the -b flag where serving requires -s . To build and serve you can run:

lola -bs

When running this command using the sample configuration file, you will get the following output:

LocaLambda build and serve output (Image by Author)

LocaLambda is going to run through a handful of steps in sequential order, some of which may be skipped if you are only building or deploying:

(build) Create a slimmed down version of your deployment template for each stack in your lola.yaml config, only the resources selected are included in the build — not the entire stack (build) Run sam build ... for each of your slimmed down templates (serve) For each stack defined in lola.yaml, run sam local start-lambda .. (serve) For each stack defined in lola.yaml, serialize the matching logic for the man in the middle and place them on Redis (serve) Launch the man in the middle which reads the configs from Redis and begins matching the HTTPS events

Turn Off SSL Verification

Because we are capturing HTTPS traffic in-flight and rerouting it to a different host, we will inevitably run into SSL Certificate issues since there is no way for your local machine to have certs that cover the AWS domain. To get around this, the devs behind AWS Lambda have provided a nifty way to turn off SSL verification. When setting up your Lambda client simply set the verify flag to False , as seen here with boto3:

import boto3

lambda_client = boto3.client('lambda', verify=False)

Realistically, you do not want to hard-code this value because in all other environments you will want to have the lambda client verify the certificate. One suggestion here is to just use an environment variable for the environment (e.g. local, dev, staging, prod, etc.), and to only set this to verify when deployed locally.

Test to Ensure Events Are Intercepted & Rerouted

Now that everything is set up and running, we can test that traffic is properly being routed to the local Lambda. The following script can be run in a shell while lola is serving your local resources.

Behind the scenes I have set it up so that this Lambda is deployed locally and it is invoking another lambda that is deployed in AWS. You can see here that traffic is being routed to the first Lambda served locally instead of to AWS!

LocaLambda Rerouting Events Output (Image by Author)

Closing Thoughts

LocaLambda can be an amazingly string tool in your toolbox to expedite your development process, but it’s by no means a silver bullet. It is more lightweight than localstack but it’s probably still a bit too heavy if it only takes a moment or two to build and push code to AWS. There are a few places where it has room for improvement and (and feedback / PRs are always welcome):

Allow custom matching logic for function name to resource mapping

Handle layers integration, both local layers and imported layers from AWS

Manage ssh tunnels; many lambdas that need to access databases in private networks are also deployed in private networks or at least in security groups that have access to the databases. Having the ability to quickly test lambdas that need DB access without the headache of manually configuring the tunnel would be a big improvement.

Be able to deploy with different structures for templates in a repo (e.g. template.yaml is not at the repo root, multiple templates, embedded template, etc.)

Allow dynamic linking and unlinking for different lola.yaml config files to prevent needing to update a single config

All in all, it’s a nifty and fun little tool that meets the need for the niche use-cases that our teams run into from time to time. I hope you can get some value out of this project as well!