AWS CodePipeline Serverless Deployment | Towards Data Science

Serverless applications are needed for machine learning and data science. They are lightweight, pay-as-you-go functions that reduce time spent on setting up servers or instances for your machine learning infrastructure. AWS has made it easier to construct a CI/CD pipeline with CodeCommit, CodeBuild, CodeDeploy, and CodePipeline. Data scientists can spend less time on cloud architecture and DevOps, and spend more time fine-tuning their models/analyzing data. Plus, the pay-as-you-go model is cheaper than paying for cloud servers/EC2 instances to run 24/7 just to host the infrastructure.
In this article, we’ll go over using AWS native tools to construct a CI/CD pipeline for deploying a simple machine learning model as a serverless endpoint. This pipeline (using CodePipeline) reads all code from a CodeCommit repository and triggers a CodeBuild. This CodeBuild uses Serverless Framework (a software package for YAML + CLI development) to deploy the code in CodeCommit to AWS Lambda Function. The goal is to trigger an AWS Lambda deployment whenever we make changes to a CodeCommit repository.
This article will cover a lot of concepts, that may be unfamiliar to many of you. Please view the Terminology and Suggested Reading to understand more details.
DISCLAIMER: Because AWS Lambda is lightweight, it may not be appropriate for huge, deep learning models. See Why We Don’t Use Lambda for Serverless Machine Learning for more information.
AWS Lambda and Serverless are normally used interchangeably. To avoid confusion, I’ll refer to AWS Lambda when talking about the pay-as-you-go architecture. I’ll refer to Serverless Framework when talking about the CLI tool used to deploy to Lambda.
AWS SQS — Simple Queue Service. A fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
AWS SageMaker — helps data scientists and developers to prepare, build, train, and deploy high-quality machine learning (ML) models quickly by bringing together a broad set of capabilities purpose-built for ML. Used to deploy ML models separately from AWS Lambda.
AWS Lambda — a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes.
Serverless Framework — a software package for YAML + CLI development and deployment to AWS Lambda.
AWS CloudWatch — A tool for monitoring and observability. Keeps track of Lambda output in logs.
To understand more on AWS Lambda, I highly recommend reading Serverless Functions and Using AWS Lambda with S3 Buckets.
To understand more about deploying in the Serverless Framework, see the Serverless Framework Documentation.
To understand CI/CD and CodePipeline development in AWS, I highly recommend reading CI/CD- Logical and Practical Approach to Build Four Step pipeline on AWS.
To understand more about SageMaker and SQS, see the documentation on AWS.
The following steps are as follows
The developer deploys a model on SageMaker
The developer updates via git the CodeCommit repository that contains logic for the architecture
CodePipeline notices a change in CodeCommit repository via CloudWatch. It then triggers CodeBuild to create/update, link, and deploy all 4 services (3 SQS queues and 1 lambda function) in AWS.
The developer then sends a message from TriggerLambdaSQS. This TriggerLambdaSQS then triggers the ModelPredictLambda function with input data. The Lambda will use the model deployed from SageMaker. If there is an error in the message format/lambda, the failed message will be outputted to the DeadletterSQS. If the lambda function processes the message successfully, it will output the result of SageMaker to the CloudWatch Logs.
Correct, CodeDeploy allows lambda deployment. For every commit to CodeCommit, you can manually call a CodeBuild and then a CodeDeploy. We want to automate the build and deployment stages via CodePipeline, and not worry about manual triggers.
In CodeDeploy, deploying packages on lambda functions is different than deploying packages on EC2 instances. In deploying on a lambda function, CodeBuild takes only the YAML file that lists lambda deployment configurations and stores it in an S3 Bucket. CodeDeploy downloads the YAML file and deploys it on a lambda function. In deploying on an EC2, CodeBuild takes all the files from CodeCommit, zips them, and stores the zip file in an S3 Bucket. CodeDeploy downloads and unzips the file on an EC2. CodePipeline specifies that artifacts generated from CodeBuild MUST be in zip format for them to be passed to CodeDeploy. This makes automated lambda deployment tricky, as it depends on a YAML file for deployment. This is a known problem in AWS discussion forums and Amazon did promise to deliver a fix for this bug. As of 2021, there still hasn’t been an update.
The workaround is to create a CodePipeline that uses both CodeCommit and CodeBuild. In CodeBuild, you install all packages (including the serverless framework) and run the serverless framework to deploy the application to AWS Lambda. Since you’re relying on the serverless framework package instead of CodeDeploy for deployment, you’ll need to create a different YAML file called serverless.yml (no need for appspec.yml, a default YAML file for CodeDeployment). We’ll discuss more later on.
For this tutorial, we’ll use scikit-learn’s Boston home prices dataset. We’ll train an XGBoost model to predict the median home price in Boston. We’ll deploy the model on SageMaker, and use it for predictions in the AWS Lambda architecture.
We’ll first want to create a folder called repo. This will contain a folder (lambda_deployment) and a Python script (boston_sagemaker_deployment.py) on the same level. lambda_deployment will contain all the relevant files needed to successfully deploy a lambda function in CodePipeline. Below is a folder structure of the files.
buildspec.yml — a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build and deploy using the serverless framework
model_predict_lambda.py — the lambda function to call the SageMaker model for predictions
requirements.txt — list of python packages to download from pip
serverless.yml — configurations in YAML to define a serverless service. This also includes additional components (SQS) that are needed to be deployed along with the lambda
boston_sagemaker_deployment.py — script to build and deploy a model on AWS using SageMaker
First thing is to deploy a simple XGBoost Model to AWS. Note: we’ll want to create an IAM role called sagemaker-role, with full permissions to access SageMaker. I won’t go into any detail about XGBoost, as it is outside the scope of this tutorial.
After that, we’ll execute the script.
python boston_sagemaker_deployment.py
We only have to do this once. It’ll take a few minutes, but it’ll give a SUCCESS or FAILED message. If SUCCESS, it’ll return an endpoint for our model. Check on the SageMaker AWS console that the endpoint is there.
CodeCommit is Amazon’s source control service that hosts secure Git-based repositories. We’ll upload the lambda_deployment folder there. We can either use git to add the files or add them manually via the console.
UPDATE: CodeBuild only recognizes files from the head of repo, not from the head of repo/lambda_deployment. To work around this, move all 4 files in lambda_deployment up to the root of repo. Once those are moved, delete the rest of the files/folders.
CodeBuild allows us to build and test code with continuous scaling. As mentioned before, we’ll also use the serverless framework to deploy the lambda function and services in CodeBuild.
We’ll add in buildspec.yml, which contains configurations for CodeBuild. Since we’re using a clean environment/image, we want to specify the commands needed to download certain packages. This is assuming a clean Ubuntu environment:
version: 0.2phases:  pre_build:    commands:      - echo "Running pre build commands"      - apt-get update      - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7EA0A9C3F273FCD8      - add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"      - apt-get -y install docker-ce docker-ce-cli containerd.io      - npm install -g serverless --unsafe      - npm i -D serverless-dotenv-plugin      - npm install serverless-plugin-aws-alerts --save-dev      - sls plugin install -n serverless-python-requirements      - apt-get -y install python3-pip      - pip3 install awscli --upgrade --user      - pip3 install -r requirements.txt      - echo "Reducing size of SageMaker to run on lambda"      - pip3 install sagemaker --target sagemaker-installation      - cd sagemaker-installation      - find . -type d -name "tests" -exec rm -rfv {} +      - find . -type d -name "__pycache__" -exec rm -rfv {} +      - zip -r sagemaker_lambda_light.zip      - cd ..  build:    commands:.      - echo "Running build commands"      - sls deploy --verbose      - echo "Finished deploying to Lambda and SQS"
Because it is a clean image, we are responsible for installing pip, awscli, serverless, docker, etc. Note: lambda has a limit to how much we can install on the function. SageMaker has a lot of dependencies, so we’re removing the components of SageMaker that are not relevant for deployment.
The prebuild commands focus on installing the packages, including the pip3 packages in requirements.txt (shown below):
boto3
Now that we have the CodeBuild commands, we next want to set up serverless.yml. Serverless.yml contains configurations to deploy the lambda function and associated services to trigger the function, store the function outputs, send alarms on threshold limits, log function behavior, etc. The file will look like this.
service: model-predictpackage:  include:    - ./model_predict_lambda.pyprovider:  name: aws  runtime: python3.6  iamRoleStatements:    - Effect: "Allow"      Action:         - "lambda:InvokeFunction"        - "iam:GetRole"        - "sqs:CreateQueue"        - "sqs:GetQueueUrl"        - "sqs:SendMessage"        - "ecr:DescribeRepositories"                - "cloudformation:DescribeStacks"        - "sagemaker:InvokeEndpoint"       Resource: "*"functions:  get_model_predictions:    handler: model_predict_lambda.model_predict_handler    provisionedConcurrency: 1 # optional, Count of provisioned lambda instances    reservedConcurrency: 1 # optional, reserved concurrency limit for this function. By default, AWS uses account concurrency limit    events:      - sqs:          arn:            Fn::GetAtt:              - TriggerLambdaSQS              - Arn          batchSize: 1resources:  Resources:    TriggerLambdaSQS:      Type: "AWS::SQS::Queue"      Properties:        QueueName: "TriggerLambdaSQS"        VisibilityTimeout: 30        MessageRetentionPeriod: 60        RedrivePolicy:          deadLetterTargetArn:            "Fn::GetAtt":              - DeadletterSQS              - Arn          maxReceiveCount: 1    DeadletterSQS:      Type: "AWS::SQS::Queue"      Properties:        QueueName: "DeadletterSQS"        VisibilityTimeout: 30        MessageRetentionPeriod: 60plugins:  - serverless-python-requirements
Here’s a brief overview of what serverless framework is creating
a service called model-predict, which will encapsulate the lambda function and its resources
a package of all python files required for the lambda function (in this case, model_predict_lambda.py)
an AWS provider and permissions assigned to the IAM role created for this function
a lambda function called get_model_predictions, which includes a handler that points to the model_predict_handler function in model_predict_lambda.py. It also contains concurrency limits and a reference to the SQS queue listed in resources
a resource section that contains TriggerLambdaSQS and DeadletterSQS. TriggerLambdaSQS is where we send the messages to the lambda function. DeadletterSQS contains all the messages that failed when being processed in lambda. A dead letter queue is added to make sure that TriggerLambdaSQS doesn’t hold onto the failed messages (which can be resent to the lambda function and trigger a continuous loop of failed messages, resulting in a lot of failed lambda invocations and a surge in prices)
Now, we want to create the python file to store the lambda function.
Because we’re using an SQS to trigger a Lambda function, we’re dealing with a queue of asynchronous events. Hence, we’re traversing through the “Records” parameter to process a list of events in the queue at a current moment. For now, we’re going to print out the result and return a successful message (which will be deleted immediately as we set the batchSize in serverless.yml to 1).
Now that we uploaded repo to CodeCommitt, let’s create a CodeBuild. Navigate to Developer Tools -> CodeBuild -> Build Projects.
From there, click on the orange button, Create build project.
Let’s name the project serverless-build. We then scroll down to initiate the source of the CodeBuild.
Select the appropriate branch where the code is.
Next, we’ll configure the environment for the build.
We have the choice of using a custom Docker image on Amazon ECS or creating a new image. I recommend using Ubuntu, as it is compatible with a lot of machine learning libraries. Because we haven’t created a service role for the CodeBuild earlier, we’ll create a new service role for the account.
We’ll then specify the buidspec file, batch configurations, and artifacts.
Buildspec, we already defined the buildspec.yml file in repo. So we just point to that file. Artifacts are zip files stored in S3 that are outputted from CodeBuild and sent to other stages in the pipeline. Since we are just deploying using serverless in the build step, we can leave this empty.
Next, we’ll instantiate CloudWatch logs to monitor the CodeBuild process.
We’ll leave everything blank, and check the CloudWatch logs checkbox. Finally, we click on Create build project.
So our build project is created. We can click on Start Build orange button, but we can let CodePipeline handle that.
Remember how we created a new service role in CodeBuild titled codebuild-serverless-build-service-role? While this creates a new service role with that name, we still need to add permissions to that role so that it can access other AWS components in our architecture (Lambda, SQS, CloudFormation, CloudWatch logs, etc). Before creating the CodePipeline, please check that the following permissions added in the service role.
AmazonSQSFullAccess
IAMFullAccess
AmazonS3FullAccess
CloudWatchLogFullAccess
AWSLambdaSQSQueueExecutionRole
AWSCloudFormationFullAccess
AWSLambda_FullAccess
CodePipeline watches repo in CodeCommit. It triggers a pipeline whenever we add a commit to repo. This is convenient because we automated a deployment whenever we make changes. Navigate to Developer Tools -> CodePipeline -> Pipeline.
Click on the orange button Create pipeline.
We now can configure settings for the pipeline. In Step 1, name the pipeline serverless-pipeline. Because we haven’t created a service role for the CodePipeline earlier, we’ll create a new service role for the account. Use default values for everything else. Use the default values for Advanced Settings and click Next.
We now add the source stage of the pipeline, which points to repo in CodeCommit. Use default values and branch of choosing, and click on Next.
We now add the build stage of the pipeline. We set the build provider to CodeBuild and the project name to serverless-build (which is what we created earlier). We do a single build and click Next.
Because we didn’t add a CodeDeploy stage (and because we’re doing serverless deployment in CodeBuild), we can skip this step. Click on Skip deploy stage.
This is just to review everything before creating the pipeline. It looks good, so let’s click on Create pipeline.
Our CodePipeline is now completed. This has the Source and Build stage. Source stage is blue because it is in progress. It will be green if it is successful, and red if it failed. Once both stages are green, we can begin testing our architecture.
If everything builds correctly, we should see Lambda -> Function-> model-predict-dev-get_model_predictions created. We should also see two queues created (DeadletterSQS and TriggerLambdaSQS).
If we click on TriggerLambdaSQS -> Send and receive messages, we’ll see a message we can send to the lambda function. Let’s send a JSON format to pass into the model.
This is what we send in the Message Body.
{ "payload": "14.0507,0,18.1,0,0.597,6.657,100,1.5275,24,666,20.2,35.05,21.22"}
After clicking on Send Message, we navigate to CloudWatch -> Log Groups -> /aws/lambda/model-predict-dev-get_model_predictions. We get the latest time stamp and inspect the logs.
We can see the output 75.875, which represents the Boston Housing mean sample price, based on the input we sent.
We went over how to construct a CI/CD pipeline in AWS to deploy a lambda architecture. The architecture contains SQS queues for trigger/failed messages and CloudWatch to output the results to logs. We went over a workaround for creating a serverless pipeline by utilizing CodeBuild and serverless framework for deployment.
We also created a script to utilize SageMaker for hosting the XGBoost model. We referred to the SageMaker endpoint in our lambda function and got the prediction of the Boston Housing median price with input fed from SQS. We are able to see the prediction in CloudWatch logs.
AWS CodePipeline has made it easier for data scientists to perform MLOps. Hopefully, Amazon can fix the bug in CodePipeline to allow YAML files to be passed in between CodeBuild and CodeDeploy stages within the pipeline.
Note: Make sure to delete SageMaker endpoint when you’re done
github.com
Thanks for reading! If you want to read more of my work, view my Table of Contents.
If you’re not a Medium paid member, but are interested in subscribing to Towards Data Science just to read tutorials and articles like this, click here to enroll in a membership. Enrolling in this link means I get paid for referring you to Medium.