Getting started with APIs in Python to Gather Data | by Nik Piepenbreier | Towards Data Science

APIs, or Application Programming Interfaces, provide easy ways to retrieve (and post) data. They are the interfaces provided by servers that you can use to, among others, retrieve and send data using code. In essence, they streamline complex instructions to provide back a request from a server. They can be compared to a waiter at a restaurant. As a patron, you give the waiter instructions for a meal, who then communicates the request back to the kitchen, where the chefs undertake complex steps to prepare a dish, without you ever knowing all the hard work he or she put into it! You then get back the meal you ordered, without having to figure out the steps it took to get there!
APIs provide many efficiencies over using static data downloads (such as CSV files). These include the ability to work with rapidly changing data or working with data from which you only want a small chunk (say, today’s temperature, compared to downloading a whole trove of weather data).
In the real, abstracted world, you likely use APIs every day, without knowing it. It’s this abstraction that makes APIs so useful — websites may rely on Google Maps’ API to link you to directions or your Twitter client lets you post data directly to your account.
But how do we leverage APIs to collect data?
In this tutorial, we’ll explore gathering data first without requiring authentication, and then another API that requires us to authenticate. We’ll start by finding where the International Space Station (ISS) currently is using the Open Notify API, and then use authentication to Dark Sky’s weather API to collect interesting weather data. We’ll close off the post by exploring how to add data about the Frankfurt Stock Exchange into a Pandas DataFrame!
Similarly to interacting with regular websites, when you want to get data from an API, you need to make a request from the server. To be able to pull data from an API using Python, we’ll need to use the requests library (go figure!). Requests is more or less the standard package for making HTTP requests in Python. It’s immensely simple to use due to its abstractions, especially when working with APIs.
To get started, you may need to install the Requests library, as it’s not part of the standard Python package. This can be accomplished either via pip or conda.
pip install requestsconda install requests
A request is made up of four different pieces:
An endpoint — which looks like the URL to the dataA method — either GET, PUT, POST, or DELETE. We’ll only be exploring GET requests in this post.The headers — which provide information such as authentication keys.The data/body — which isn’t part of a GET request (so, pretend I didn’t mention it for now)
An endpoint — which looks like the URL to the data
A method — either GET, PUT, POST, or DELETE. We’ll only be exploring GET requests in this post.
The headers — which provide information such as authentication keys.
The data/body — which isn’t part of a GET request (so, pretend I didn’t mention it for now)
When we run a request using the requests library, it returns a request object, which includes the data we’re hoping to extract, but also a requests status code. The status code tells us what happened with the request and they are a part of every request that we make. We’ll be working with GET requests and there’s a number of response codes that are relevant for our work. The codes are split into different values in hundreds, depending on the information they’re returning:
1xx: Provide information
2xx: Generally are successful
3xx: Provide information on redirects
4xx: Refer to a client error (our bad)
5xx: Refer to a server error (their bad)
You may be familiar with seeing the (sometimes comical) 404 page when a page can’t be found. This is what that status code is.
Similarly, a status code of 200 lets us know that everything is OK! This is the one we’re going for and will hopefully get!
For the first part of this tutorial, we’ll take a look at the Open Notify API that provides data on the International Space Station. You can find more information on the API by visiting their documentation.
One of the great things about this API is that it doesn’t require authentication, meaning we can dive right into pulling some data without additional set up.
Let’s begin by generating a URL request with the Requests library, and accessing its status code attribute.
We can do this by using the code outlined below. The output is printed under the dashed line for simplicity.
The .get() function requests a request object. Here, we can use the .status_code attribute to discover what the status code returns. In this example, the status_code returned was 200, meaning we are good to move forward!
The data that we get back comes back in what’s known as JSON format. A JSON file is similar to a Python dictionary and ensures that data is easily readable by machines. An easy way to understand JSON files is to think of them as Python dictionaries represented as strings.
To begin working with JSON files, let’s import the JSON library, which is part of the standard package (so we don’t need to install anything). I’ve included the output of the code under the dashed line for ease of reading.
The string that this response returned is quite short and quite easy to read. However, many API responses will be lengthy and may require a bit of parsing to determine where you want to look for your data (if you’re interested in a particular piece). Let’s use the .dumps() function to convert a JSON string into a Python object. We’ll repeat some of the code for continuity.
This is a much easier-to-read format, allowing us to parse out the data much more readily. Say we were only interested in knowing the latitude and longitude, we could parse this out in the same manner that we would parse data out of a dictionary in Python.
An important thing to note here is that we are calling the response.json() object here — not the one that we returned using the .dumps() function.
From here, we can now use this data to easily append data to a data set, using whatever method we want.
Let’s now turn our attention to the Dark Sky API, which requires a bit of set up before we can get started.
To get started, we’ll need to get what’s known as an API secret key. Keys are ways that allow a user to be authenticated without needing to provide a username and password. As such, they should be treated securely and not be shared. To get your own Dark Sky secret key, sign up for a Dark Sky account at https://darksky.net/dev/register. A free tier will be fine for experimenting with.
Once you’re signed up, navigate to the Account Page https://darksky.net/dev/account, which will give you your API Key. Make note of this key, and be sure to keep it secret.
As part of the free tier, you can make up to 1,000 calls per day.
Dark Sky provides a thorough resource for what data is included in their API, which you can access by visiting: https://darksky.net/dev/docs
There are two main types of calls: (1) the Forecast request, and (2) the Time Machine Request. The former provides current weather (up to the next week), and the latter provides data for the past and the future.
Let’s assume that we wanted to pull out the current weather conditions (summary, temperature, and wind speed) for Toronto, Ontario.
Forecast requests take the following format: https://api.darksky.net/forecast/[key]/[latitude],[longitude]
That means we need three things:
Our API key,Our latitude, andOur longitude.
Our API key,
Our latitude, and
Our longitude.
To easily find your latitude and longitude, you can put the address you’re looking for into Google Maps. For example, I searched for Toronto’s CN Tower. The URL returned will include the coordinates: https://www.google.com/maps/place/CN+Tower/@43.6425701,-79.3892455,17z/
Following the @ sign, the longitude comes first (43.6425701), followed by the latitude (-79.3892455).
Let’s write the code out in a verbose, but easy to follow manner. We’ll make use of f-strings to make the code even easier to read. If you’re not familiar with f-strings, check out the Python docs for them.
Based on the documentation, we can see that the generic request pulls more data than we need. While not required, we can trim this down using Dark Sky’s exclude block, which will reduce latency on the request. We’ll only be using the currently block, so we will exclude all the others.
Earlier we mentioned we were going to be interested in the current conditions (specifically, the summary, temperature, and windspeed). By reviewing the structure above, we can see that we can readily pull this data by accessing the key-value pairs for those items.
The Dark Sky API lets you also set units. Right now, our statement isn’t clear as to what units the data we’re extracting is in. Let’s be more explicit. As we’re pulling data for a Canadian location, let’s specify for their locale. The Dark Sky API documentation tells u that we should set units=si.
So far, we’ve learned how to make API calls with and without authentication. Let’s take it one step further by adding this into a Pandas dataframe.
Let’s say we’re working on a project, and we need to generate a data frame with financial data. We can do this easily using built-in functions in Pandas.
For this, we’ll use Quandl’s financial API. In particular, we’ll use the free API for Frankfurt’s Stock Exchange, for which the documentation can be found here.
Let’s explore the data a bit. We’ll load the data following the instructions for generating the URL on the website and print the data using the json.dumps() function described above. We’ll need to import the same libraries as before, as well as Pandas.
The data actually keeps going for a long, long time, but it’s truncated here.
From this, we can see that:
Column names are found in [‘dataset’][‘column_names’]
The data is found in [‘dataset’][‘data’]
Let’s create a dataframe using this knowledge:
In this post, we explored what APIs are, how to use them in Python, both with and without authentication, and how to add data from APIs to pandas dataframes. There’s still a whole lot more to APIs that we didn’t cover here, but hopefully this gives you a good understanding of where to take your learning from here!