FastAPI + AWS = robust API (Part 1) | by Anna Geller | Towards Data Science

FastAPI revolutionized the way of developing modern Python-based REST APIs. Since its inception, the project has been adopted by large companies, such as Microsoft, Uber, and Netflix, and it’s increasingly gaining popularity, what we can observe by the number of Github stars growing every day.
Being able to build fast, robust, secure, and scalable APIs is a must-have skill for any backend (data) engineer. In this article, we’ll build a simple REST API that will show the market sentiment and current prices of selected cryptocurrencies in a currency of your choice (euro, dollar, etc.).
We’ll use the top-down approach: from packaging and deploying the API to diving into details on what it is about. In Part 1, we’ll deploy our FastAPI code to AWS by leveraging Amazon Lambda and API Gateway. In the future article (Part 2), we’ll make sure to secure our API by using API keys. We’ll also demonstrate how to enable distributed tracing and logging to track our application’s health. Let’s dive in!
Create a Pycharm project with a virtual environmentInstall the required librariesTest the API locallyPackage your code & prepare the deployment pipeline (incl. creation of Lambda function)Create the REST API in API GatewayConfigure your Lambda function as a proxy to forward requests from API Gateway to Amazon LambdaDeploy the API to dev stageTest the endpointDiving deeper into how it all works togetherDemo of the cryptocurrency API by using Swagger UIConclusion
Create a Pycharm project with a virtual environment
Install the required libraries
Test the API locally
Package your code & prepare the deployment pipeline (incl. creation of Lambda function)
Create the REST API in API Gateway
Configure your Lambda function as a proxy to forward requests from API Gateway to Amazon Lambda
Deploy the API to dev stage
Test the endpoint
Diving deeper into how it all works together
Demo of the cryptocurrency API by using Swagger UI
Conclusion
Let’s start by creating a Project with a virtual environment in Pycharm. You can either use a Virtualenv or Conda — whichever you prefer. I usually use Virtualenv on macOS and Conda on Windows.
If you want to follow along, this Github repository contains the code for this project.
Let’s change the current working directory to api folder and install the required packages:
cd api && pip install -r requirements.txt
After having installed all package dependencies, we can start the Uvicorn ASGI server in our terminal:
uvicorn main:app --reload
If you configured your virtual environment and downloaded (or cloned) the code from Github, you should be able to access the interactive Swagger UI from your browser:
http://localhost:8000/docs
In order to make our API accessible from API Gateway, we need to:
zip our API code with all site-packages dependencies into a zip file that will be used by our Lambda function
upload this zip file to S3 (we could, in theory, upload it directly to Lambda, but there is a 10MB zip file size limit → to prepare our application for growth, we will upload it directly to S3, and during the Amazon Lambda creation, we’ll make sure that Amazon Lambda uses this zip file from S3)
Note: this part requires that you installed and configured AWS CLI
create the Amazon Lambda function with Python 3.8 runtime, choose the option “Create a new role with basic Lambda permissions”, and configure the Lambda handler as main.handler, as shown in the GIF:
update the function to make sure that it uses the packaged code from S3 (we could have done it from the management console, as well):
aws lambda update-function-code --function-name medium-demo \--s3-bucket $bucket_name --s3-key lambda.zip
From the management console, make sure to choose the same region for your API in API Gateway as the one you selected for the Lambda function. Even though it’s possible to have API Gateway and Lambda function in different regions, it could potentially lead to some extra latency.
Then: Create API → Choose REST API (not the private one) → Build → enter the name for your API → Create API.
Now we need to configure the integration point for our request methods. To use a Lambda function as our integration point for ANY type of request (i.e., GET, POST, PATCH, DELETE, etc.), we will create a Method (to handle the root path) and a child Resource (to handle all child paths). We will configure them to handle any requests made to API Gateway by using the Lambda proxy integration [1].
From the dropdown menu Actions: 1. Create Method 2. Choose ANY 3. Select Lambda Function as your integration type and make sure to check the box “Use Lambda Proxy Integration”4. Add the name of your Lambda function (and its corresponding region) and keep the defaults for everything else → Save
When the pop-up window asks you whether you want to grant API Gateway permission to invoke your Lambda function, confirm. Without it, API Gateway will not be authorized to invoke your Lambda function.
Similarly to the “Method” configuration, go to Actions → Create Resource. Make sure to select the checkbox “Configure as proxy resource” and keep the defaults to enable Lambda proxy integration (as before), then select your Lambda function and Save.
Again, confirm that you allow API Gateway to invoke the Lambda function on behalf of your API requests:
Since our Lambda is now configured, we can deploy the API. We can name it dev stage. The deployment is crucial to make the Lambda function integration active.
Deploying your API to any deployment stage ensures that API Gateway creates a URL that serves as a front door to your API. You can click on this URL and test that the API has been properly configured.
In your browser, you should see a response corresponding to the root GET path of our FastAPI code:
{"Hello Medium Reader": "from FastAPI & API Gateway"}
We can continue testing further endpoints directly from our browser:
Side note: If the endpoint dev/v1/prices/ETH doesn’t work for you, and you get a message {"message": "Missing Authentication Token"}, you should retry the creation of the Resource proxy and redeploy the API.
Already at this point, we can see that our API is publicly accessible by anyone! In Part 2, we’ll change it by creating an API key directly from the API Gateway console.
By now, you may have many questions. I will try to guess and answer them in this Q&A form:
1. How did we make the local FastAPI code work with Amazon Lambda?
We used the Mangum library [5], which serves as a wrapper for ASGI APIs running inside of AWS Lambda and API Gateway. It provides an adapter, which:
routes requests made to the API Gateway to our Lambda function,
routes Lambda function responses back to the API Gateway.
We only had to add two lines (line 1 and 7) to turn our FastAPI code within main.py to an Amazon Lambda handler:
2. What is the difference between a Method and a Resource in API Gateway?
The ANY Method will route requests to the root path / to the Lambda function that executes our FastAPI code. In contrast, the child Resource will route all paths below the root path / in a greedy mode (greedy path parameter: {proxy+}) [7], i.e., it will route all other endpoints defined in your FastAPI code to Lambda, for example, the path /v1/prices/BTC/.
3. Why do we need {proxy+}?
Using the proxy, we don’t need to write any mapping templates that would route the specific paths to different Lambda functions - instead, we have a single Lambda function that serves all endpoints [6]. Then, API Gateway will route all types of requests (GET, POST, DELETE, PATCH, ...) from API Gateway to the paths defined within FastAPI code. And the FastAPI code gets executed inside of the Lambda function thanks to the Mangum adapter.
4. How does this architecture scale?
Each request made to API Gateway will invoke a Lambda function. This means that we can have thousands of parallel requests made to our API, as each will be served from a separate Lambda function invocation.
Note that there are soft concurrency limits of how many parallel Lambda executions we can have per account (at the time of writing, 1000). If you need more than that, you can use the Support Center console to request an increase to this limit [2].
5. Why is this architecture attractive?
It provides a completely serverless API infrastructure with many benefits:
all advantages of Python and the amazing FastAPI library combined with the benefits of a serverless architecture
scalability & no backend server to manage — API Gateway will spin up as many Lambda functions as needed to handle all requests made to our API
very cost-effective, especially for smaller APIs that may not yet get millions of requests
performant: due to Lambda’s context reuse, consecutive requests (within 15 minutes from a previous request) reuse the frozen execution context, allowing to mitigate the cold start of a serverless function
if enabled, API Gateway caching allows further latency improvements
easy management of API keys
API Gateway provides integration with Active Directories and third-party auth providers such as “Login with Google/Amazon account” via Cognito
out of the box centralized logging of all API requests and responses, as well as access logs via CloudWatch
distributed tracing capability by enabling X-Ray traces.
6. Why did we use site-packages from the virtual environment to install package dependencies?
We could have installed the libraries into some specific directory (ex. libs directory) by adding the flag -t, which stands for target:
pip install -r requirements.txt -t ../libs/
And then we could have added this directory to our lambda.zip.
However, this way, we would clutter our project with Python packages that would have been installed into the project directory, and it could make it harder for other developers later to distinguish between the actual API code and its dependencies. We would likely have to add the libs directory to .gitignore. Overall, it would add extra work, which we don’t need to do if we are working with a virtual environment and its site-packages.
7. Could we have used the Serverless Framework or AWS SAM to create a deployment pipeline?
Yes, this setup works well with Serverless Framework. Here is an example of how it could look like:
github.com
However, I prefer to minimize dependencies in my project and serverless requires additional components such as the Node package manager.
You can also use this setup with AWS SAM.
8. How much does it costs?
With the free tier, you get monthly (free) access to:
1 million API Gateway requests
1 million Lambda invocations with up to 3.2 million seconds of compute time per month
Apart from that, the prices may vary depending on your AWS region and your number of requests (bulk discount), but they are in ranges of 1–4.25 dollars per one million requests (at the time of writing). Lambda is priced based on the number of invocations and how long your code is running, measured in milliseconds. You can find out more here and here.
Finally, here is what the API is doing — all functionality is based on information coming from CryptoCompare [4]:
we are getting the trading signals to get the information about the market sentiment for a specific cryptocurrency, i.e., whether there is a current upward or downward tendency
we are getting real-time prices of selected cryptocurrencies and can display them in a specific currency of our choice.
In this article, we discussed the process of deploying a REST API to AWS by leveraging FastAPI, API Gateway, and Amazon Lambda. Along the way, we learned how to package the code and create a deployment pipeline, and we discussed how all those microservice components work together.
In Part 2 article, we’ll continue building this API by adding authentication via API Keys, distributed logging and tracing, as well as how to redeploy the API once we made changes to our FastAPI code.
towardsdatascience.com
If this article was useful, follow me to see my next articles.
References:
[1] Blog by Sean Baier
[2] Amazon Lambda docs
[3] API Gateway docs
[4] CryptoCompare API docs
[5] Mangum docs
[6] About the Lambda proxy integration
[7] How to use Amazon API Gateway {proxy+}