Query API’s with Json Output in Python | by Alexandra Yanina | Towards Data Science

If you are a Data Science beginner, you will often work in courses and tutorials with .csv files that are easy to read into Pandas dataframes. In practice, however, you often need to access API’s and get data in Json format. This data often contains nested lists and dictionaries.
So today I will explain how to query an API with JSON output and process this data to read it into a pandas dataframe. As an example I decided to use the cat facts API. This API contains funny information on the one hand, but on the other hand it is also relatively simple structured so it is well suited for beginners. With the Cat Facts API we can learn how to query the API in Python and how to convert the JSON output into a pandas dataframe.
To follow this guide, basic Python knowledge as well as knowledge of the pandas package is required. The knowledge of what an API is, is assumed.
First you need to look at the API documentation and find out the URL of the API interface and the endpoints. Then we need the specification what type of facts we want to get returned. In this example we query the Endpoint:
GET /facts/random
The base URL of all endpoints is https://cat-fact.herokuapp.com.
More information about the Fact endpoint can be found at this URL Here you will also find information about the query parameters and how an API response will look like and how it will be structured. There are even sample queries available. We query this endpoint to retrieve the individual facts, with their ID, the user who uploaded the fact, and the creation date.
Now let’s start creating the query in Python:
import requestsimport pandas as pdimport jsonimport pprintimport seaborn as snsimport matplotlib.pyplot as plt
First we have to import the necessary packages. In this tutorial we use requests to access the API via HTTP request. Pandas to read the JSON data into a DataFrame and do an explorative analysis. Seaborn and matplotlib are used to visualize the data.
base_url = "https://cat-fact.herokuapp.com"facts = "/facts/random?animal_type=cat&amount=500"
Now we save the base URL and the used endpoint in variables.
first_response = requests.get(base_url+facts)response_list=first_response.json()
To get the data as Json output you can use the requests package. To do this we call the request.get method with the base URL and the endpoint and store the returned values in the variable first_response. The Json output can then be parsed with the .json() method and stored in a list. Printing response_list shows us the structure of the obtained data:
[{'used': False,  'source': 'api',  'type': 'cat',  'deleted': False,  '_id': '591f97e88dec2e14e3c20b06',  '__v': 0,  'text': 'A female Amur leopard gives birth to one to four cubs in each litter.',  'updatedAt': '2020-08-23T20:20:01.611Z',  'createdAt': '2018-05-16T20:20:03.145Z',  'status': {'verified': True, 'sentCount': 1},  'user': '5a9ac18c7478810ea6c06381'}, {'used': False,  'source': 'user',  'type': 'cat',  'deleted': False,  '_id': '5c87d03dec1e5c0aa37e0a6b',  'user': '5a9ac18c7478810ea6c06381',  'text': 'On October 24, 1963, a tuxedo cat named Félicette entered outer space aboard a French Véronique AG1 rocket and made feline history. Félicette returned from the 15-minute trip in once piece and earned the praise of French scientists, who said she made "a valuable contribution to research.".',  'createdAt': '2019-03-12T15:29:01.505Z',  'updatedAt': '2020-08-23T20:20:01.611Z',  '__v': 0,  'status': {'verified': True, 'sentCount': 1}}, {'used': False,  'source': 'api',  'type': 'cat',  'deleted': False,  '_id': '591f98783b90f7150a19c1b5',  '__v': 0,  'text': "Cats have 32 muscles that control the outer ear (compared to human's 6 muscles each). A cat can rotate its ears independently 180 degrees, and can turn in the direction of sound 10 times faster than those of the best watchdog.",  'updatedAt': '2020-08-23T20:20:01.611Z',  'createdAt': '2018-01-04T01:10:54.673Z',  'status': {'verified': True, 'sentCount': 1},  'user': '5a9ac18c7478810ea6c06381'}...]
Now that we have the data in a list with sublists and dictionaries, we have to iterate through them and append the individual values of the respective keys to a new list called data. Getting the values from the subdictionaries requires using the .get method of the dictionary object.
There are possibly innumerable ways to perform this step. This is one possible way to get the values from the sub-dictionaries.
data=[]for response in response_list:      data.append({        "used": response.get('used'),        "source": response.get('source'),        "text": response.get('text'),        "updatedAt": response.get('updatedAt'),        "createdAt": response.get('createdAt'),        "user": response.get('user')            })
Then we will simply transform our list into a pandas data frame with the pd.Dataframe method.
catfacts_df=pd.DataFrame(data)catfacts_df.head()
Now that we have successfully imported the data into a pandas dataframe, we can access more data via the API. It would be interesting to know which user has added the catfacts. To do this, we have to retrieve the endpoint “GET /facts?” again and merge the dataframes about the users with the catfacts_df data. Here you have to watch out that you dont get duplicate user values. We only want to use a list with the username and the user id, and use these information to join the two dataframes. But by obtaining data from the endpoint facts, we have to drop duplicate rows based on the user id. By doing that we get a list where every user is included once.
We must also note that it is necessary to check whether a data point is complete and does not have missing values. We do this with an if-condition. This way we insert only complete data points into the list, otherwise the code would return an error. After creating a DataFrame with the data and dropping duplicate values based on the “user” column, we also reset the index.
user_facts="/facts?"user_response=requests.get(base_url+user_facts)user_list=user_response.json()users=[]for response in user_list.values():    for user in response:        if user.get('user'):            users.append({              "user": user.get('user').get('_id'),              "firstname":user.get('user').get('name').get('first'),               "lastname":user.get('user').get('name').get('last')})userfacts_df= pd.DataFrame(users)userfacts_df = userfacts_df.drop_duplicates(subset = ["user"])userfacts_df.reset_index(inplace=True,drop=True)userfacts_df.tail()
Printing the DataFrame shows that there are 83 unique catfact submitters.
Now is the next important step to merge both data frames in one. For this we use the .merge method and specifiy that it should be merged on the left dataset (catfacts_df) on the column “user”.
df=catfacts_df.merge(userfacts_df, how='left', on= "user")
After joining both data frames it is important to confirm that the merge was correct.
Before we continue with the visualization, it makes sense to insert the already written code into a clearly arranged function. Here we can also set the parameter “amount” to get a certain amount of catfacts.
def get_cat_facts(amount):    base_url="https://cat-fact.herokuapp.com"        # get cat facts    response=requests.get(base_url+ "/facts/random?animal_type=cat&amount=" + str(amount))    response_json=response.json()        data=[]    for response in response_json:          data.append({            "used": response.get('used'),            "source": response.get('source'),            "text": response.get('text'),            "updatedAt": response.get('updatedAt'),            "createdAt": response.get('createdAt'),            "user": response.get('user')})            #create catdf    catfacts_df=pd.DataFrame(data)         # get user data        user_response=requests.get(base_url+ "/facts?")    user_json=user_response.json()        users=[]    for response in user_json.values():        for user in response:            if user.get('user'):users.append({                    "user": user.get('user').get('_id'),                    "firstname":user.get('user').get('name').get('first'),                     "lastname":user.get('user').get('name').get('last')})    #create userdf    userfacts_df=pd.DataFrame(users)     userfacts_df = userfacts_df.drop_duplicates(subset = ["user"])    #joining both datframes    df=catfacts_df.merge(userfacts_df, how='left', on= "user")        return dfcat_facts_df = get_cat_facts(amount=500)
In the last step we can visualize the data. Here I simply visualized how many facts a user has submitted.
# catfacts per useragg_user_df=cat_facts_df.groupby(['user', 'firstname', 'lastname']).agg({    "text": 'count'})top_ten=agg_user_df.nlargest(10, 'text')bar_plot=sns.barplot(x = 'lastname', y = 'text', data = top_ten,            palette = 'hls',              capsize = 0.05,                         saturation = 8,                         errcolor = 'gray', errwidth = 2,              ci = 'sd'                        )bar_plot.set_title('Facts by User')bar_plot.set_ylabel('Amount Cat Facts')bar_plot.set_xlabel('Lastname')plt.xticks(rotation=45)plt.show()
In the GitHub repository you can get the whole code. I hope you enjoyed this Tutorial and I hope it will help you when working with APIs.