Saving 95% on infrastructure costs using AWS Lambda for scikit-learn predictions | by Yoav Zimmerman | Towards Data Science

Disclaimer: I am the developer behind Model Zoo, a model deployment platform focused on ease-of-use.
If you’re an AWS customer that needs to deploy machine learning models for real-time inference, you might have considered using AWS SageMaker Inference Endpoints. However, there is another option for model deployment that is sometimes overlooked: deploying directly on AWS Lambda. Although it comes with some caveats, the simplicity and cost-efficiency of Lambda make it worthwhile to consider over SageMaker endpoints for model deployment, especially when using scikit-learn, xgboost, or spaCy. In this article, we’ll go over some of the benefits and caveats of using AWS Lambda for ML inference and dive into some relevant benchmarks. We show that in scenarios of low usage (<2M predictions per month), you can save up to 95% on infrastructure costs when moving models from SageMaker to Lambda. We’ll also present scikit-learn-lambda, our open-source toolkit for easily deploying scikit-learn on AWS Lambda.
SageMaker inference endpoints are one of many pieces of an impressive end-to-end machine learning toolkit offered by AWS, from data labeling (AWS SageMaker Ground Truth) to model monitoring (AWS SageMaker Model Monitor). SageMaker inference endpoints offer features around GPU acceleration, autoscaling, AB testing, integration with training pipelines, and integration with offline scoring (AWS Batch Transform). These features come at a steep cost — the cheapest possible inference endpoint (ml.t2.medium) will run you $50/month to run 24/7. The next best endpoint (ml.t2.xlarge) is $189.65/month.
AWS Lambda is a pioneer of the serverless computing movement, letting you run arbitrary functions without provisioning or managing servers. It executes your code only when needed and scales automatically, from a few requests per day to hundreds per second. Lambda is a generic function execution engine without any machine learning specific features. It has inspired a growing community of tooling, some from AWS themselves (Serverless Application Model) and some externally affiliated (Serverless framework).
Lambda has a pay-per-request model that scales with your growth. Whether this is a pro or con depends on your usage level, but is especially cost-efficient for serving under 2M predictions per month. SageMaker, as of this writing, does not support auto-scaling to zero during periods of low activity.
Lambda requires a simpler implementation computational model where concurrency/autoscaling can be handled transparently outside of your logic.
Lambda requires lower maintenance effort, without any underlying servers or logic to manage.
Lambda has a rich developer ecosystem (both open-source and SaaS) for monitoring, logging, and testing serverless applications, such as Serverless. With SageMaker, you’re relying on AWS-specific resources such as the SageMaker-compatible containers and SageMaker Python SDK for tooling.
Lambda has constraints on the resources available, with a maximum of 3,008 MB physical memory and no support for GPU acceleration. This can be a deal-breaker for infrastructures that use large models with stringent latency requirements.
Lambda incurs a cold start latency when the function is invoked after a 5–10 minute period of inactivity. Cold Starts on AWS Lambda is a great blog post diving into some of the factors that affect cold start latency.
Package size limits in AWS Lambda are stringent at 50 MB zipped and 250 MB unzipped. There are workarounds (AWS Lambda Layers), but this can be an annoying hurdle for bundling common ML dependencies such as TensorFlow (~400 MB) and PyTorch (~250 MB).
Lambda cannot directly use Docker containers as a dependency, and instead has its own dependency management system AWS Lambda Layers. This sometimes requires users to put in a little bit of extra work into packaging.
PharmAssess is a Model Zoo user whose mission is to treat more patients, more confidently, and more efficiently. They have developed a proprietary model that can automate minor ailment assessment, diagnosis, treatment & documentation with scikit-learn, and are now working with an initial set of users to test it. The size of their model is ~50 MB, making it a great candidate for deployment with Lambda.
With Model Zoo, PharmAssess is able to deploy its scikit-learn model with a simple Python API. Behind the scenes, Model Zoo will package and deploy the scikit-learn model to an AWS Lambda function (see our quickstart for a complete example).
import modelzoo.sklearnmodel = ...  # train modelmodelzoo.sklearn.deploy(model)
If you prefer maintaining your own model storage, deployment, and monitoring stack, you can use our open-source package scikit-learn-lambda to deploy a model onto your own AWS cloud with the Serverless framework:
$ git clone https://github.com/model-zoo/scikit-learn-lambda$ cd scikit-learn-lambda$ cp <model file> scikit-learn-lambda/model.joblib$ serverless deploy
One big caveat of AWS Lambda is the “cold start”, or initialization overhead that occurs after 5–10 minutes of inactivity. To get an understanding of the severity, we generate MLPClassifier models with three hidden layers each using layer sizes of 100, 250, 500, 750, and 900. Each model is trained on the iris dataset and serialized to disk using the joblib library. This results in model files sized 0.68 MB, 4.08 MB, 16.17 MB, 36.25 MB, and 52.13 MB respectively. We deploy each to a lambda function with 1024 MB memory using scikit-learn-lambda and measure an average cold-start latency over three samples to generate the benchmark above.
We also plot the “warm” latency below averaged over fifteen samples, which hovers around 2–5 milliseconds for each model.
There are some workarounds for cold-start latency, such as hitting your function every few minutes to keep it warm or using Provisioned Concurrency. However, if optimizing for p90+ latency is an important business requirement, the serverless paradigm might not be right for your use-case.
AWS Lambda usage can be difficult to measure as a combination of time per request, memory usage, and total requests per month. We created this interactive visualization to help us understand the cost per month under various usage scenarios, benchmarked against SageMaker endpoints.
The cost difference speaks for itself: using an average warm latency observed above and an allocated memory of 1024 MB, Lambda is an order of magnitude cheaper than a single SageMaker endpoint from the lowest cost tier. The pay-per-request model also makes it great for AB testing and prototyping — you can deploy every single model you train without ever worrying about underutilization.
As a general rule, AWS Lambda makes sense for smaller models (<50MB) that can use CPUs for inference and for which you expect a usage pattern of <2M requests/month. This makes it ideal for building prototypes or new products that use lightweight model libraries[2] like scikit-learn, xgboost, or spaCy.