Modern CI/CD Pipeline: Github Actions with AWS Lambda Serverless Python Functions and API Gateway
By: Ary Sharifian
Tags: API, Python, DevOps, Data Science, AWS Lambda

Modern CI/CD Pipeline: Github Actions with AWS Lambda Serverless Python Functions and API Gateway

Modernizing web application development and deployment

Photo by Morning Brew on Unsplash

Cloud is here to stay and more and more developers are seeking ways to effectively incorporate the cloud. Whether you are a startup recognizing limitations of your on-premise hardware and local machines or a large enterprise curious about how to slowly offload on-prem workloads, this tutorial will be insightful. I describe a phase 1 AWS architecture including Github, API Gateway, and AWS Lamba python functions. This represents an initial tutorial exposing developers to the AWS cloud adoption learning curve.

Outline:

Notional Architecture Purpose and Goals How to Setup Limitations and Problems of AWS Lambda Thoughts and Conclusion

Notional Architecture:

Architecture Diagram

The architecture above describes the basic CI/CD pipeline for deploying a python function with AWS Lambda. The developer represented above can pull and push their git repository to github using git. We configured the github actions YAML file to automatically update the AWS Lambda function once a pull request is merged to the master branch. The API Gateway represents a trigger which runs the AWS Lambda python function and returns a result. In this way, a Data Scientist (or analyst, frontend developer, or another developer) can trigger and access results in a quick and succinct fashion.

N ote: AWS Lambda Functions exists in AWS Lambda, a compute resource inaccessible by the developer.

Purpose and Goals

This architecture represents an effective way for teams to build a robust CI/CD pipeline for application development and deployment. Though this architecture is “incomplete” with respect to a full-blown web application, this design can represent the first phase of building a web application. For anyone interested in offloading local compute resources, AWS Lambda serverless functions can be an effective way to leverage cloud in a cost-effective manner (AWS Lambda Functions are part of AWS always free tier). So many times, development teams design a lofty cloud based-architecture for application deployment (or migration) and fail. Conducting Proof-Of-Concepts and slowly incorporating cloud is more prudent.

How to Setup

The primary challenge is understanding the YAML file and correctly formatting the “main.py” file which is executed in AWS Lambda. To setup a new workflow where developers can configure YAML file for function deployment, click “actions” in the github repository. This will provide instructions for creating a new workflow or selecting from a pre-existing workflow consistent with the deployment architecture. I created a default workflow and then searched other workflows to find a template that deployed to AWS Lambda (but there is probably an easier way leveraging a pre-configured workflow for AWS Lambda).

Note: line 57 is actually zipping github repo into a zip file called “test-dep-dev”. Ideally, to organize a Lambda function deployment, I recommend creating a folder in your repo and zipping that folder for deployment. For example, you might create a folder called “Lambda_Function_Scrapping_Data” which contains all the dependencies needed to deploy your function. Line 57 would look something like “zip -r test-dep-dev.zip ./Lambda_Funtions_Scrapping_Data”

Once you have configured the YAML file, check the AWS Lambda Function page to see if the function has been updated. To troubleshoot, create a test case within the AWS Lambda Function. The issue I first faced was a malformed syntax for the python return value (here is some documentation that might help). Below is a simple example of what a AWS Lambda Python function should look like. The response format is most important.

Note: in AWS Lambda function, the handler, is defined as the file_name.function_name. For example, if the file name is main.py and the function is my_serverless_function, the handle should be defined as main.my_serverless_function.

Instead of recreating the wheel, this is a great video demonstrating how to create an API Gateway for the AWS Lambda Function:

Limitations and Problems of AWS Lambda and possible alternatives

Before deciding to use AWS Lambda its important to consider the limitations. AWS Lambda functions will timeout after 15 minutes which is fairly generous, but for more involved enterprise level workloads, this might not be enough. More importantly your packaged functions are limited to 250mb unzipped and 50mb zipped which include the size of the packages, coded functions, and other dependencies. These are just some of the limitations that were especially applicable to the use case I am addressing. You can find other limitations here.

Given that AWS Lambda is essentially a shared compute instance running a containerized function, for more flexibility you can provision an EC2 instance. Obviously this results in higher costs. If you are worried about high availability, I am not sure what the SLA offers are for AWS Lambda, but generally, AWS compute instance users should assume 90% availability.

Conclusion

There is a bit of learning curve to AWS Lambda and serverless functions, but given their general applicability in most modern web application development, I consider it a worthwhile investment of time. If you are working on a project and want to incorporate some cloud, AWS Lambda is fairly worthwhile from a process and cost standpoint. End users can easily call the endpoint and retrieve data. There seems to be plenty of documentation and tutorials to address nearly any use case. Overall, I would recommend AWS Lambda. For more flexibility with a compute instance, I recommend Oracle Cloud Compute Instances. They are free (less than 1 OCPU), but documentation might be difficult to find and understand.