A Custom SpaCy Model Deployment with AWS ECR, SageMaker, and Lambda | by Cheng | Towards Data Science

SpaCy is one of my favourite NLP libraries. And I have been using spaCy to perform a lot of Named Entity Recognition (NER) tasks. Generally, we first need to load a spaCy pre-trained model of a specific language and fine-tune the model with our training dataset. The training process can be done offline with a local computer and we can even test the fine-tuned model performance by hosting it locally through Flask / Streamlit.
towardsdatascience.com
Although I have found many great tutorials on deploying a spaCy model locally with Flask / Streamlit, there are not many tutorials on how to deploy it on a larger scale, for example, how to deploy a spaCy model with AWS.
It’s a very interesting topic and after a great amount of work I summarized my solution into this article; hopefully it can be useful for people facing the same question.
In this article, I will explain my solution on how to deploy a custom spaCy model with AWS services including:
AWS ECR (Elastic Container Registry)
AWS SageMaker
AWS Lambda
AWS S3 Bucket (Optional)
Here is my plan 🧗🏻:
First, data input can be sent as an event into the AWS Lambda.
Then, within the Lambda, we invoke a SageMaker endpoint. The endpoint takes the data input coming from the Lambda event and returns a response containing the results from the spaCy model.
Finally, this endpoint response will display as an execution result. Hence we are able to check the response on the AWS Lambda result tab or debug on the AWS CloudWatch.
Through this process, the spaCy model is hosted on AWS SageMaker and can be invoked as an endpoint at any time. To create the SageMaker endpoint, we need to create our custom spaCy model container on the AWS ECR first and prepare our model artifact ready in the format of tar.gz.
To create the model artifact, we could do it either offline or online. Every time we finish training a spaCy model, the trained model is actually saved into a folder. The folder usually contains the following files:
*file names might be different
meta.json
ner (folder)
tokenizer
vocab (folder)
In terms of creating the model artifact, we could simply compress this folder into a tar.gz format and upload it to a S3 bucket. Alternatively, we could also train the model online with the AWS SageMaker by creating a training job under the SageMaker Training section. But we need to provide our custom training image from AWS ECR and training data from AWS S3 bucket.
Hopefully, you are now a bit clearer about the whole workflow. Now, let’s begin from creating our custom spaCy model container and uploading it onto the AWS ECR as our training image.
Because spaCy is not one of the SageMaker built-in algorithms. We have to create a spaCy model container first on the AWS ECR, and specify the container as the training image while creating a model endpoint on the SageMaker.
SageMaker provides 2 options wherein the first option is to use built-in algorithms that SageMaker offers that includes KNN, XgBoost, Linear Learner, etc. while the other option is to use your custom docker container from ECR.
towardsdatascience.com
The above article is an amazing reference that inspired me on how to deploy a custom spaCy container. I strongly encourage to read through this article and have a deeper understanding on how this docker container works internally. It was quite complicated to understand at first for me. 🧠
In brief, it will include the following steps:
Edit the files under the Github repo (mainly the docker file, train.py and predictor.py) in order to train a custom spaCy model and return predictions based on the data input.
Check the results by performing a local test for the spaCy docker container and debug the code if any error happens.
Finally, upload the spaCy container to the AWS ECR.
A bit more explanation about the train.py and predictor.py, because these are the two files that determine how we want to train the model and what we expect the input & output of the model to be like.
In the train file, the objective is to modify the code to train a custom spaCy model. After reading the data from CSV files into a dataframe called train_data, we need to convert the dataframe into a format that spaCy model can take as input. I would recommend to read a previous article I wrote about how to train a spaCy model for an NER task.
towardsdatascience.com
Essentially, we need to convert a dataframe into the following spaCy format:
TRAIN_DATA = [('Amazon co ca', {'entities': [(0, 6, 'BRD')]}),('AMZNMKTPLACE AMAZON CO', {'entities': [(13, 19, 'BRD')]}),('APPLE COM BILL', {'entities': [(0, 5, 'BRD')]}),('BOOKING COM New York City', {'entities': [(0, 7, 'BRD')]}),('STARBUCKS Vancouver', {'entities': [(0, 9, 'BRD')]}),('Uber BV', {'entities': [(0, 4, 'BRD')]}),('Hotel on Booking com Toronto', {'entities': [(9, 16, 'BRD')]}),('UBER com', {'entities': [(0, 4, 'BRD')]}),('Netflix com', {'entities': [(0, 7, 'BRD')]})]]
Then, in the train file, we define our train_spacy function to train the model. The train_spaCy function is similar to the one in the following article. Just be aware that the code within the article is written in spaCy version 2.
manivannan-ai.medium.com
In the predictor file, we need to edit the code under the predict function. The predict function only takes test_data as input. The test_data is in format of a dataframe already so we could apply the trained spaCy model onto the column which contains the source text input. Then we could create a new column for the model’s predictions. If it’s an NER task, then the new column would contain the extracted entities from each input. Also, we could adjust whether the predict function returns just a list of predictions or a whole dataframe that contains both the source and the prediction columns.
At the end of the predictor file, it requires a bit knowledge of Flask app to understand. Although it’s recommended to know how it works, in fact, we don’t need to modify many parts in the original Flask app. Just be sure that the output data format from the predict function is in consistent with the format used in the Flask app. The original function returns prediction in format of a list. For me, I changed it to return a dataframe so I modified the code in the Flask app to keep the consistency.
Finally, in the docker file, simply edit the code accordingly based on the libraries needed. For me, I added RUN pip3 install -U spacy==2.3.5 to install the correct version of spaCy I used.
Congratulations if you got here! 🥳
To push our container image to Amazon ECR, we could follow the code inside (from the above article’s Github)
build_and_push.sh file
Bring_Your_Own-Creating_Algorithm_and_Model_Package notebook
These files are well explained in that article. We could simply run through the code to push our own custom spaCy container onto the AWS ECR.
Until this step, we have finally uploaded our custom spaCy container to the AWS ECR. This is a critical step for the model deployment. Now, we will switch to AWS SageMaker and create a model endpoint.
As described in the AWS SageMaker developer guide:
Amazon SageMaker is a fully-managed service that enables data scientists and developers to quickly and easily build, train, and deploy machine learning models at any scale. Amazon SageMaker includes modules that can be used together or independently to build, train, and deploy your machine learning models.
Essentially, it has an interface structured as below:
Sections on the AWS SageMaker...--->Training------>Algorithms------>Training jobs ------>Hyperparameter tuning jobs--->Inference------>Compilation jobs------>Model packages------>Models------>Endpoint configurations------>Endpoints------>Batch transform jobs...
To create a spaCy model endpoint, we need to go through the following steps:
Create a training job under AWS SageMaker — Training. Or instead, simply train your spaCy model locally, compress it into tar.gz format and upload it to a S3 bucket. The goal is to prepare a model artifact at this step.
Create a model under AWS SageMaker — Inference. We need to specify the container image from AWS ECR and the model artifact from a AWS S3 bucket.
Create an endpoint configuration under AWS SageMaker — Inference. We need to specify a model for the endpoint configuration.
Create an endpoint under AWS SageMaker — Inference. We need to specify an endpoint configuration for the endpoint.
If you have any trouble during the above steps on AWS SageMaker, the following tutorial will be helpful! 🏂🏻
aws.amazon.com
As described in the AWS Lambda developer guide:
Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availability compute infrastructure and performs all of the administration of the compute resources, including server and operating system maintenance, capacity provisioning and automatic scaling, code monitoring and logging.
In terms of the spaCy deployment, we will use the AWS Lambda as an interface that can communicate between the user input and the spaCy model endpoint. Specifically, we want to input the inference text as a Lambda event and receive the model response on the Lambda execution tab.
Some details about how to invoke a SageMaker endpoint on the AWS Lambda can be found at the tutorial same as the one above. Essentially, we want to write a python handler function on the Lambda that can do the following steps:
Input event: a list of inference texts
Read the event into a dataframe (optional)
Write the dataframe into a CSV buffer (keep data format consistency with the Flask app)
Convert the values of the buffer into ASCII format (keep consistency with the AWS requirement otherwise may cause some errors)
Invoke the SageMaker endpoint and receive the response
Decode the response into UTF-8 format
Read the response into a dataframe (optional)
Output response: a list of model predictions
Some details about creating a CSV buffer or ASCII / UTF-8 conversion steps may not be necessary but eventually it works for me. If you have any better ideas, let me know! 😸
If everything works, now finally, we should be able to see that on the AWS Lambda execution results tab, the list of model predictions is printed out. 💯 And the model input is based on the AWS Lambda configure test event.
Finally, my article on deploying a custom spaCy model with the AWS will conclude here. As a summary, the workflow includes services:
1st step: AWS ECR
2nd step: AWS S3 Bucket (Optional)
3rd step: AWS SageMaker
4th step: AWS Lambda
Although this design may not be the best, it’s still the most stable one I could think of after a large amount of research. Hope it can help / inspire anyone who is working on the similar tasks 😋
Thank you for reading!