How to Get Data from APIs with Python 🐍 | by Jeff Hale | Towards Data Science

It’s a bit of a cliche to say that data is the new oil. I like to think of it more as a renewable resource like wind.💨 Whatever energy source you choose as your metaphor, you want to harness some of its power! ⚡️
Ideally, you have direct access to the data you want in a file or a database you control. When that’s not the case, if you’re lucky, the data will be available through a public-facing an API. ☘️
In this article, I’ll show you the steps to get the data from a public API using Python. 🐍 First I’ll show you how and where to look for a Python API wrapper and share the largest repository of Python API wrappers. 🎉
Then I’ll show you how to use the requests library to get the data you want from an API that doesn’t have a Python wrapper.
If the data you want is on a website, but not available through a public-facing API, there are several options for scraping it. When to use which scraping package is a whole other article I have in the works. Follow me to make sure you don’t miss it! 😁
Let’s drill down!
An external facing Application Programming Interface (API) is often intended to provide data in large chunks. You just need to know how to work with the API.
An organization creates a public-facing API with the intent that you use it. Their motivations vary from idealistic to mercenary and might include the following:
Hoping you’ll build something that will improve the world. 🌍Hoping you’ll use their free plan and then need so much data or need the data so frequently you’ll pay them for access to more data. 💵Figuring you’ll scrape the data from the website if they don’t give you a direct line to it, so they might as well reduce their server overhead and make your experience better. 😀
Hoping you’ll build something that will improve the world. 🌍
Hoping you’ll use their free plan and then need so much data or need the data so frequently you’ll pay them for access to more data. 💵
Figuring you’ll scrape the data from the website if they don’t give you a direct line to it, so they might as well reduce their server overhead and make your experience better. 😀
APIs can be documented well, poorly, or somewhere in between. If you’re lucky, they are well-documented. If you’re really lucky there is a Python wrapper for the API that works and is well documented. 🎉
It can be tricky to find Python wrappers for the API you need. Here’s my suggested approach.
I am maintaining what I believe is the largest list of Python API wrappers over at GitHub. Real Python made a nice list that was forked and updated by johnwmiller. I cleaned the list up a bit and then, given that coronavirus quarantine ended my children’s ability to earn money from soccer refereeing ⚽️ and cat sitting 🐈, I paid them to help improve the list. We updated and augmented this outdated list because I couldn’t find a good list of functioning API wrappers elsewhere. 😀
If you find a Python API wrapper that is missing from the list, please edit the ReadMe file and submit a pull request. Here’s a quick guide to editing GitHub Markdown files in the GUI, if you’re new to this:
Click the pencil icon in the right corner and make changes (here’s a lovely Markdown tutorial if you need it). Then click the green Propose file change button at the bottom of the page. Then click on the green Create pull request button, summarize the changes, and click on the green Create pull request button at the bottom. Thank you! 🎉
FYI, I often highlight Python packages on my Data Awesome mailing list. In the next installment I’ll be highlighting yfinance by Ran Aroussi. yfinance wraps the Yahoo Finance API. You can use it to read stock market data into a pandas DataFrame with one line of code. 🚀
If the Python API wrapper list doesn’t have what you need, I suggest you use the usual method for finding things on the interwebs. 🕸
A good search engine is a developer or data scientist’s best friend 😁
Specifically, I would search for Python wrapper the_name_of_the_api_I’m_looking_for. GitHub links are likely to be the most fruitful. If a repo hasn’t been updated in the past several years or has been archived, your odds of being able to successfully use the wrapper aren’t high. In that case, I suggest you keep looking. 👓
If you’re lucky, the website for the API you are using might list wrappers available in various programming languages. It’s worth a try. 😃
When there isn’t an API wrapper, you have toto query the API directly. I suggest you use the Python requests library.
The venerable requests library is the battle-tested way to get information from an API. Requests was created by Kenneth Reitz and is protected by the Python Software Foundation. It’s the most downloaded Python package as of this writing. 👍
Install requests into your environment from the command line with pip install requests
Then import it and use it. Use the HTTP verbs get and post as methods to return the information you desire. Mostly, you’ll use get. Here’s how to query the GitHub API:
import requestsr = requests.get('https://api.github.com/events')
You can pass parameters to the get method as a dictionary. Here’s the quick start guide.
You’ll often get back JSON when you make a get request. You can use the requests .json() method to change JSON into a dictionary quickly.
my_dict = r.json()
Requests uses the urllib3 library under the hood and enhances it. There are other libraries you could use, but requests is super solid, is nice to use, and is familiar to most Python coders. I suggest you use requests whenever you want to make an API call and the API doesn’t have a Python wrapper.
Speaking of which, if your favorite API doesn’t have a Python wrapper, I encourage you to considering making one. 👍
Making a an API wrapper is a great way to learn Python packaging skills. Here’s a guide I wrote to get you started with making a Python package and releasing it on PyPi.
I created Pybraries, a Python wrapper for the Libraries.io API. Making it was a great learning experience, and it was cool to give back to the open source community that I’ve benefited from. 🚀
API keys, rate limits, and cURL are three other terms you should be familiar with.
You’ll often need an API key to query the API. The API docs should make it clear how you can procure a key. Most of the time, you can get one for free by signing up at the website of the organization whose data you want. If you want a lot of data or want it frequently, you might have to pay for the privilege.
You can store your API key in an environment variable. Here’s a guide for setting up environment variables on Mac, Linux, and Windows. Environment variable names are ALL CAPS by convention. Speaking of all caps:
⚠️ DO NOT, UNDER ANY CIRCUMSTANCES, STORE AN API KEY IN GITHUB OR ANOTHER PUBLICLY ACCESSIBLE ONLINE VERSION CONTROL SYSTEM. ⚠️
Your key could get stolen and abused — especially if your account has a credit card attached to it. ☹️
To access the environment variable that holds your API key in Python, import the os module and get the value from the dictionary with the key that matches the name of your environment variable. For example:
import osos.environ.get('MYENVIRONMENTVARIABLE')
This code returns the value of your MYENVIRONMENTVARIABLE environment variable.
If you will be using your key in applications on the cloud, you should look into secrets management, as discussed here.
Speaking of staying out of trouble, let’s discuss rate limits.
Many APIs limit how many times you can ping them in a given amount of time to avoid needing to pay for lots of extra servers. To avoid going over these limits, you can use the Python time library and put your request into a loop. For example, here’s a brief snippet of code that waits for five seconds between requests.
import timeimport requestst = 0my_url = 'https://example.com'while t < 100:    r = requests.get(my_url)    time.sleep(5)            # wait five seconds    t += 1    print(r.json())
The point of this code is to show you how you can use time.sleep() to avoid nonstop API endpoint pings that might push you over a rate limit. In a real use case, you would probably want to save your data in some way and use a try...except block.
This is a guide for using Python with APIs, but you should know that you can query an API from the command line with the popular cURL program. cURL is included on Macs and machines running recent Windows 10 versions by default. If you want to learn cURL, I suggest you check out the free ebook Conquering the Command Line, by Mark Bates. 🚀
If you want to get data from an API, try to find a Python wrapper first. Check out the list of Python wrappers here. Google search if that fails, and check out the API website. If you find a Python wrapper missing from my list on GitHub please add it. 😀
If there isn’t a Python wrapper for an API, use the requests library. If you have the time, consider creating a wrapper so all Python users can benefit. ❤️
Don’t forget to keep your API keys safe and avoid hitting rate limits. 🔑
You’ve learned a workflow for finding and using APIs with Python. You’re on your way to getting the data you need to do awesome things! 🚀
I hope you found this guide to using APIs with Python useful. If you did, please share it on your favorite social media so other folks can find it, too. 👍
I write about Python, SQL, Docker, data science, and other tech topics. If any of that’s of interest to you, follow me and read more here. 😀