Python packages with AWS layers — The right way
By: Jose Ignacio Castelli
Tags: Serverless, Python3, AWS, Python, AWS Lambda

Step1: Requirements

Basically, we are going to be using Serverless framework for our deployments on AWS (both layers and the lambda function itself)

Install AWS CLI

Download AWS CLI, then just to run and configure your AWS credentials

aws configure

Install Pipenv

We are going to use Pipenv to manage your local dependencies working with a virtual environment and allow serverless to know what needs to download before deploying the layer

Install Docker

You must have Docker up and running to allow a serverless-python-requirements plugin to build dependencies in a dockerized environment

Install NPM

In order to install Serverless framework you must have installed NPM before

Install serverless

Finally set up Serverless framework to make AWS lambda and layers deployments

Step2: Splitting your dependencies on separates folders

The idea is to have a separate folder for every dependency just to keep every AWS layer light as possible and to be able to reuse those dependencies.

So go to your root directory and make your folder layer dependencies

mkdir -p Layers/Pandas && mkdir Layers/SQLAlchemy

Step3: Create your Pipfile for every dependency folder

Working with Pipenv, we are going to install and create python dependencies files. Those Pipenv files will be consumed by serverless-python-requirements plugin. We will create two files, one inside Layers/Pandas folder and another one inside into Layers/SQLAlchemy folder, both under ‘Pipfile’ name.

To do it you can just execute into Layers/Pandas

pipenv install pandas

And the same into Layers/SQLAlchemy directory

pipenv install sqlalchemy

Step4: Create your serverless file for every dependency folder

Before working with a serverless framework we are going to install a serverless-python-requirements plugin. Go to the root folder and execute

npm i --save serverless-python-requirements

Then we need to create a serverless.yml file to be able to deploy on every layer directory. So for our example create the ‘serverless.yml’ file into Layers/SQLAlchemy and Layers/Pandas and copy on those the next configuration

Then replace YOUR-LAYER-NAME with panda and Sqlalchemy on each file respectively

Step5: Deploy your layers

To be able to consume those layers on our lambda function we need to make dependency deployment first. Ideally, you need to do this process just once, however, if you want to renew or change a python package version you need to deploy it again

To deploy, execute inside Layers/SQLAlchemy and Layers/Pandas directory

serverless deploy

Grab both deployment layer ARN outputs that will be used on next steps. Both ARN will look like

arn:aws:lambda:AWS_ACCOUNT_ZONE:AWS_ACCOUNT_ID:layer:LAYER_NAME:REVISION_NUMBER

Step6: Create your lambda function to consume those deployed layers

We will be creating a pretty straightforward lambda function just to consume those dependencies

On your root directory create a simple python script file called your_lambda_fn.py and paste there the next content

Finally, we need to create the deployment file for the lambda function itself. So again, on the root directory create a serverless.yml file and paste

At the bottom, you must replace both layers ARNs with those you grabbed before

Remember that every time that you decide to change a python dependency you need to update the REVISION_NUMBER

Step7: Deploy your lambda function

Again, on the root directory just execute

serverless deploy

You can go to your AWS Lambda Console and execute manually or using AWS CLI instead

Conclusion

You will not have any dependency issue when deploying to AWS

Your lambda functions will be pretty light

Working on this way will be able to reuse those AWS layers across multiple lambdas functions

If you keep working with layers, AWS Lambda console will allow you to edit onsite, so doing tests will be pretty straightforward if something happens

You can clone or fork this final repository where you can check how everything should go

https://github.com/JicLotus/python-aws-serverless-dependencies

Resources

Support my content by buying me a coffee

Please write to me at joseignaciocastelli92@gmail.com