Develop and sell a Python API — from start to end tutorial | by Daniel Deutsch | Towards Data Science

I recently read a blog post about setting up your own API and selling it.
I was quite inspired and wanted to test if it works. In just 5 days I was able to create an API from start to end. So I thought I share issues I came across, elaborate on concepts that the article was introducing, and provide a quick checklist to build something yourself. All of this by developing another API.
About this article
Disclaimer
Stack used
1. Create project formalities
2. Create a solution for a problem
3. Deploy to AWS
4. Set up Rapidapi
End result
Inspiration
About
This article can be considered as a tutorial and comprehension of other articles (listed in my “Inspiration” section).
It paints a picture for developing a Python API from start to finish and provides help in more difficult areas like the setup with AWS and Rapidapi.
I thought it will be useful for other people trying to do the same. I had some issues on the way, so I thought I share my approach. It is also a great way to build side projects and maybe even make some money.
As the Table of content shows, it consists of 4 major parts, namely:
Setting up the environmentCreating a problem solution with PythonSetting up AWSSetting up Rapidapi
Setting up the environment
Creating a problem solution with Python
Setting up AWS
Setting up Rapidapi
You will find all my code open sourced on Github:
https://github.com/Createdd/pandas_transform_format
You will find the end result here on Rapidapi:
https://rapidapi.com/Createdd/api/excel-to-other-formats
If you found this article helpful let me know and/or buy the functionality on Rapidapi to show support.
I am not associated with any of the services I use in this article.
I do not consider myself an expert. If you have the feeling that I am missing important steps or neglected something, consider pointing it out in the comment section or get in touch with me. Also, always make sure to monitor your AWS costs to not pay for things you do not know about.
I am always happy for constructive input and how to improve.
We will use
Github (Code hosting),
Anaconda (Dependency and environment management),
Jupyter Notebook (code development and documentation),
Python (programming language),
AWS (deployment),
Rapidapi (market to sell)
It’s always the same but necessary. I do it along with these steps:
Create a local folder mkdir NAMECreate a new repository on Github with NAMECreate conda environment conda create --name NAME python=3.7Activate conda environment conda activate PATH_TO_ENVIRONMENTCreate git repo git initConnect to Github repo. Add Readme file, commit it and
Create a local folder mkdir NAME
Create a new repository on Github with NAME
Create conda environment conda create --name NAME python=3.7
Activate conda environment conda activate PATH_TO_ENVIRONMENT
Create git repo git init
Connect to Github repo. Add Readme file, commit it and
git remote add origin URL_TO_GIT_REPOgit push -u origin master
Now we have:
local folder
github repository
anaconda virtual environment
git version control
Then we need to create a solution to some problem. For the sake of demonstration, I will show how to convert an excel csv file into other formats. The basic functionality will be coded and tested in a Jupyter Notebook first.
Install packages
Develop a solution to a problem
Download data
Create functionality
Build server to execute a function with REST
Install jupyter notebook and jupytext:
pip install notebook jupytext
set a hook in .git/hooks/pre-commit for tracking the notebook changes in git properly:
touch .git/hooks/pre-commitcode  .git/hooks/pre-commit
copy this in the file
#!/bin/sh# For every ipynb file in the git index, add a Python representationjupytext --from ipynb --to py:light --pre-commit
afterwards for making the hook executable (on mac)
chmod +x .git/hooks/pre-commit
pip install pandas requests
Add a .gitignore file and add the data folder (data/) to not upload the data to the hosting.
Download an example dataset (titanic dataset) and save it into a data folder:
def download(url: str, dest_folder: str):    if not os.path.exists(dest_folder):        os.makedirs(dest_folder)    filename = url.split('/')[-1].replace(" ", "_")    file_path = os.path.join(dest_folder, filename)    r = requests.get(url, stream=True)    if r.ok:        print("saving to", os.path.abspath(file_path))        with open(file_path, 'wb') as f:            for chunk in r.iter_content(chunk_size=1024 * 8):                if chunk:                    f.write(chunk)                    f.flush()                    os.fsync(f.fileno())    else:        print("Download failed: status code {}\n{}".format(r.status_code, r.text))url_to_titanic_data = 'https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/stuff/titanic.csv'download(url_to_titanic_data,'./data')
Transform format
df = pd.read_csv('./data/titanic.csv')df.to_json(r'./data/titanic.json')
After developing the functionality in jupyter notebook we want to actually provide the functionality in a python app.
There are ways to use parts of the jupyter notebook, but for the sake of simplicity we create it again now.
Add an app.py file.
We want the user to upload an excel file and return the file converted into JSON for example.
Browsing through the internet we can see that there are already packages that work with flask and excel formats. So let's use them.
pip install Flask
Start Flask server with
env FLASK_APP=app.py FLASK_ENV=development flask run
Tipp: Test your backend functionality with Postman. It is easy to set up and allows us to test the backend functionality quickly. Uploading an excel is done in the “form-data” tab:
Here you can see the uploaded titanic csv file and the returned column names of the dataset.
Now we simply write the function to transform the excel into json, like:
import jsonimport pandas as pdfrom flask import Flask, requestapp = Flask(__name__)@app.route('/get_json', methods=['GET', 'POST'])def upload_file():    if request.method == 'POST':        provided_data = request.files.get('file')        if provided_data is None:            return 'Please enter valid excel format ', 400        data = provided_data        df = pd.read_csv(data)        transformed = df.to_json()        result = {            'result': transformed,        }        json.dumps(result)        return resultif __name__ == '__main__':    app.run()
(Check out my repository for the full code.)
Now we have the functionality to transform csv files into json for example.
After developing it locally we want to get it in the cloud.
Set up zappa
Set up AWS
AWS credentials
Set up credentials with users and roles in IAM
Add credentials in your project
AWS API Gateway
After we created the app locally we need to start setting up the hosting on a real server. We will use zappa.
Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as “serverless” web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance — and at a fraction of the cost of your current deployments!
pip install zappa
As we are using a conda environment we need to specify it:
which python
will give you /Users/XXX/opt/anaconda3/envs/XXXX/bin/python (for Mac)
remove the bin/python/ and export
export VIRTUAL_ENV=/Users/XXXX/opt/anaconda3/envs/XXXXX/
Now we can do
zappa init
to set up the config.
Just click through everything and you will have a zappa_settings.json like
{    "dev": {        "app_function": "app.app",        "aws_region": "eu-central-1",        "profile_name": "default",        "project_name": "pandas-transform-format",        "runtime": "python3.7",        "s3_bucket": "zappa-pandas-transform-format"    }}
Note that we are not yet ready to deploy. First, we need to get some AWS credentials.
First, you need te get an AWS access key id and access key
You might think it is as easy as:
To get the credentials you need to
Go to: http://aws.amazon.com/
Sign Up & create a new account (they’ll give you the option for 1 year trial or similar)
Go to your AWS account overview
Account menu; sub-menu: Security Credentials
But no. There is more to permissions in AWS!
I found this article from Peter Kazarinoff to be very helpful. He explains the next section in great detail. My following bullet point approach is a quick summary and I often quote his steps. Please check out his article for more details if you are stuck somewhere.
I break it down as simple as possible:
Within the AWS Console, type IAM into the search box. IAM is the AWS user and permissions dashboard.Create a groupGive your group a name (for example zappa_group)Create our own specific inline policy for your groupIn the Permissions tab, under the Inline Policies section, choose the link to create a new Inline PolicyIn the Set Permissions screen, click the Custom Policy radio button and click the “Select” button on the right.Create a Custom Policy written in json formatRead through and copy a policy discussed here: https://github.com/Miserlou/Zappa/issues/244Scroll down to “My Custom policy” see a snippet of my policy.After pasting and modifying the json with your AWS Account Number, click the “Validate Policy” button to ensure you copied valid json. Then click the “Apply Policy” button to attach the inline policy to the group.Create a user and add the user to the groupBack at the IAM Dashboard, create a new user with the “Users” left-hand menu option and the “Add User” button.In the Add user screen, give your new user a name and select the Access Type for Programmatic access. Then click the “Next: Permissions” button.In the Set permissions screen, select the group you created earlier in the Add user to group section and click “Next: Tags”.Tags are optional. Add tags if you want, then click “Next: Review”.Review the user details and click “Create user”Copy the user’s keysDon’t close the AWS IAM window yet. In the next step, you will copy and paste these keys into a file. At this point, it’s not a bad idea to copy and save these keys into a text file in a secure location. Make sure you don’t save keys under version control.
Within the AWS Console, type IAM into the search box. IAM is the AWS user and permissions dashboard.
Create a group
Give your group a name (for example zappa_group)
Create our own specific inline policy for your group
In the Permissions tab, under the Inline Policies section, choose the link to create a new Inline Policy
In the Set Permissions screen, click the Custom Policy radio button and click the “Select” button on the right.
Create a Custom Policy written in json format
Read through and copy a policy discussed here: https://github.com/Miserlou/Zappa/issues/244
Scroll down to “My Custom policy” see a snippet of my policy.
After pasting and modifying the json with your AWS Account Number, click the “Validate Policy” button to ensure you copied valid json. Then click the “Apply Policy” button to attach the inline policy to the group.
Create a user and add the user to the group
Back at the IAM Dashboard, create a new user with the “Users” left-hand menu option and the “Add User” button.
In the Add user screen, give your new user a name and select the Access Type for Programmatic access. Then click the “Next: Permissions” button.
In the Set permissions screen, select the group you created earlier in the Add user to group section and click “Next: Tags”.
Tags are optional. Add tags if you want, then click “Next: Review”.
Review the user details and click “Create user”
Copy the user’s keys
Don’t close the AWS IAM window yet. In the next step, you will copy and paste these keys into a file. At this point, it’s not a bad idea to copy and save these keys into a text file in a secure location. Make sure you don’t save keys under version control.
My Custom policy:
{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "iam:AttachRolePolicy",        "iam:GetRole",        "iam:CreateRole",        "iam:PassRole",        "iam:PutRolePolicy"      ],      "Resource": [        "arn:aws:iam::XXXXXXXXXXXXXXXX:role/*-ZappaLambdaExecutionRole"      ]    },    {      "Effect": "Allow",      "Action": [        "lambda:CreateFunction",        "lambda:ListVersionsByFunction",        "logs:DescribeLogStreams",        "events:PutRule",        "lambda:GetFunctionConfiguration",        "cloudformation:DescribeStackResource",        "apigateway:DELETE",        "apigateway:UpdateRestApiPolicy",        "events:ListRuleNamesByTarget",        "apigateway:PATCH",        "events:ListRules",        "cloudformation:UpdateStack",        "lambda:DeleteFunction",        "events:RemoveTargets",        "logs:FilterLogEvents",        "apigateway:GET",        "lambda:GetAlias",        "events:ListTargetsByRule",        "cloudformation:ListStackResources",        "events:DescribeRule",        "logs:DeleteLogGroup",        "apigateway:PUT",        "lambda:InvokeFunction",        "lambda:GetFunction",        "lambda:UpdateFunctionConfiguration",        "cloudformation:DescribeStacks",        "lambda:UpdateFunctionCode",        "lambda:DeleteFunctionConcurrency",        "events:DeleteRule",        "events:PutTargets",        "lambda:AddPermission",        "cloudformation:CreateStack",        "cloudformation:DeleteStack",        "apigateway:POST",        "lambda:RemovePermission",        "lambda:GetPolicy"      ],      "Resource": "*"    },    {      "Effect": "Allow",      "Action": [        "s3:ListBucketMultipartUploads",        "s3:CreateBucket",        "s3:ListBucket"      ],      "Resource": "arn:aws:s3:::zappa-*"    },    {      "Effect": "Allow",      "Action": [        "s3:PutObject",        "s3:GetObject",        "s3:AbortMultipartUpload",        "s3:DeleteObject",        "s3:ListMultipartUploadParts"      ],      "Resource": "arn:aws:s3:::zappa-*/*"    }  ]}
NOTE: Replace XXXXXXXXXXX in the inline policy by your AWS Account Number.
Your AWS Account Number can be found by clicking “Support → “Support Center. Your Account Number is listed in the Support Center on the upper left-hand side. The json above is what worked for me. But, I expect this set of security permissions may be too open. To increase security, you could slowly pare down the permissions and see if Zappa still deploys. The settings above are the ones that finally worked for me. You can dig through this discussion on GitHub if you want to learn more about specific AWS permissions needed to run Zappa: https://github.com/Miserlou/Zappa/issues/244.
Create a .aws/credentials folder in your root with
mkdir ~/.awscode ~/.aws/credentials
and paste your credentials from AWS
[dev]aws_access_key_id = YOUR_KEYaws_secret_access_key = YOUR_KEY
Same with the config
code ~/.aws/config[default]region = YOUR_REGION (eg. eu-central-1)
Note that code is for opening a folder with vscode, my editor of choice.
Save the AWS access key id and secret access key assigned to the user you created in the file ~/.aws/credentials. Note the .aws/ directory needs to be in your home directory and the credentials file has no file extension.
Now you can do deploy your API with
zappa deploy dev
There shouldn’t be any errors anymore. However, if there are still some, you can debug with:
zappa statuszappa tail
The most common errors are permission related (then check your permission policy) or about python libraries that are incompatible. Either way, zappa will provide good enough error messages for debugging.
If you update your code don’t forget to update the deployment as well with
zappa update dev
To set up the API on a market we need to first restrict its usage with an API-key and then set it up on the market platform.
I found this article from Nagesh Bansal to be helpful. He explains the next section in great detail. My following bullet point approach is a quick summary and I often quote his steps. Please check out his article for more details if you are stuck somewhere.
Again, I break it down:
go to your AWS Console and go to API gatewayclick on your APIwe want to create an x-api-key to restrict undesired access to the API and also have a metered usagecreate a Usage plan for the API, with the desired throttle and quota limitscreate an associated API stageadd an API keyin the API key overview section, click “show” at the API key and copy itthen associate the API with the key and discard all requests that come without the keygo back to the API overview. under resources, click the “/ any” go to the “method request”. then in settings, set “API key required” to truedo the same for the “/{proxy+} Methods”
go to your AWS Console and go to API gateway
click on your API
we want to create an x-api-key to restrict undesired access to the API and also have a metered usage
create a Usage plan for the API, with the desired throttle and quota limits
create an associated API stage
add an API key
in the API key overview section, click “show” at the API key and copy it
then associate the API with the key and discard all requests that come without the key
go back to the API overview. under resources, click the “/ any” go to the “method request”. then in settings, set “API key required” to true
do the same for the “/{proxy+} Methods”
it looks like this
Now you have restricted access to your API.
Create API on Rapidapi
Test your own API
Create private plan for testing
Test endpoint with rapidapi
Create code to consume API
Go to “My APIs” and “Add new API”Add the name, description, and category. Note that you cannot change your API name afterward anymoreIn settings, add the URL of your AWS API (it was displayed when you deployed with zappa)In the section “Access Control” under “Transformations”, add the API key you added in AWS
Go to “My APIs” and “Add new API”
Add the name, description, and category. Note that you cannot change your API name afterward anymore
In settings, add the URL of your AWS API (it was displayed when you deployed with zappa)
In the section “Access Control” under “Transformations”, add the API key you added in AWS
5. In the security tab you can check everything
6. Then go to “endpoints” to add the routes from you Python app by clicking “create REST endpoint”
7. Add an image for your API
8. Set a pricing plan. Rapidapi published an own article on pricing options and strategies. As they conclude, it is up to your preferences and product on how to price it.
9. I created a freemium pricing plan. The reason for that is that I want to give the chance to test it without cost, but add a price for using it regularly. Also, I want to create a plan for supporting my work. For example:
10. Create some docs and a tutorial. This is pretty self-explaining. It is encouraged to do so as it is easier for people to use your API if it is documented properly.
11. The last step is to make your API publicly available. But before you do that it is useful to test it for yourself.
Having set up everything, you of course should test it with the provided snippets. This step is not trivial and I had to contact the support to understand it. Now I am simplifying it here.
Create a private plan for yourself, by setting no limits.
The go to the “Users” section of your API, then to “Users on free plans”, select yourself and “invite” you to the private plan.
Now you are subscribed to your own private plan and can test the functionality with the provided snippets.
Upload an example excel file and click on “test endpoint”. Then you will get a 200 ok response.
To consume the API now you can simply copy the snippet that Rapidapi provides. For example with Python and the requests library:
import requestsurl = "https://excel-to-other-formats.p.rapidapi.com/upload"payload = ""headers = {    'x-rapidapi-host': "excel-to-other-formats.p.rapidapi.com",    'x-rapidapi-key': "YOUR_KEY",    'content-type': "multipart/form-data"    }response = requests.request("POST", url, data=payload, headers=headers)print(response.text)
The article “API as a product. How to sell your work when all you know is a back-end” by Artem provided a great idea, namely to
Make an API that solves a problem
Deploy it with a serverless architecture
Distribute through an API Marketplace
For the setting everything I found the articles from Nagesh Bansal very helpful:
https://medium.com/@bansalnagesh/how-to-sell-your-apis-b4b5c9a273f8
https://medium.com/@bansalnagesh/launch-your-api-on-aws-with-0-upfront-cost-using-zappa-in-10-minutes-eb6d00623842
Also this article from Peter Kazarinoff: https://pythonforundergradengineers.com/deploy-serverless-web-app-aws-lambda-zappa.html
I encourage you to have a look at those articles as well.
You can also read my article directly on Github (for better code formatting)
Daniel is an entrepreneur, software developer, and lawyer. He has worked at various IT companies, tax advisory, management consulting, and at the Austrian court.
His knowledge and interests currently revolve around programming machine learning applications and all its related aspects. To the core, he considers himself a problem solver of complex environments, which is reflected in his various projects.
Don’t hesitate to get in touch if you have ideas, projects, or problems.