COVID-19 Data Collection: A Python & API Story. | by Somesh Routray | Towards Data Science

This context will be a brief discussion on the collection of data from an API using python. The agenda for this will be as below.
Connect to API using pythonParse the data collected into a CSV formatA little touch of Python Pandas
Connect to API using python
Parse the data collected into a CSV format
A little touch of Python Pandas
API- Application Programming Interface:
Pretend you want to use other website data for your program. You have different options to do that. You can scrape the website using BeautifulSoup and Scrapy or you can use an interface that will help you to talk with your program. That interface is API.
More elaborately if I say if the website has API, you can connect your connect application to the internet and it will send data to the server. Then the server retrieves the data and performs certain actions according to your application. This happens primarily through the API
The modern APIs have layers of security features. When your application sends a request to the server and when the target website sends a response, they never fully exposed to the internet. Either of the communication happens with small packets of data and the information shared just as per the requirement. This is so trusted that many big companies get revenues through APIs. Below are a few points about why APIs are helpful.
The modern APIs follow the standards of HTTP and REST, developer-friendly kinds of stuff.These are more than a piece of code and built for consumption by a specific audience.The building of an API follows proper security, governance. It is developed in an SDLC and monitored thoroughly.
The modern APIs follow the standards of HTTP and REST, developer-friendly kinds of stuff.
These are more than a piece of code and built for consumption by a specific audience.
The building of an API follows proper security, governance. It is developed in an SDLC and monitored thoroughly.
This blog is dedicated to connecting one API using python and along with I will let you know some of the techniques of Pandas library as well. Here I will be using COVID-19 API. You can get that API from the rapid-api website. These guys are doing a great job in making different API. I will recommend visiting the website or you can make an API for your own use. The good thing about this website is, you can get a sample code in different programming languages.
Prerequisites:
local development environment for python 3API URL and key
local development environment for python 3
API URL and key
here Key is used as first security to identify that the requested data is genuine. It is used to prevent unethical use of APIs.
Let’s get started with the coding.
Step1 -Import these necessary libraries
import json import pandas as pdimport requests
Step2- Get API URL and Key and send a request
url = “your url"headers = { ‘x-rapidapi-host’: “api host”, ‘x-rapidapi-key’: “api token” }response = requests.request("GET", url, headers=headers)print(response)
if the response is 200 means request success.
Generally, API has documentation regarding API details and its purpose. Some API services have API wrappers that need to be installed in your PC to access API.
The APIs may have the data in JSON, XML or any custom format. To parse data you can use different libraries in python. The request and response headers often include request headers and a key. This is the current information about your use of the API service.
Voila !!! you got the results.
But the data I got is like a nested JSON structure of data. Let’s process it further.
Step3- Parse the JSON data
parsed_data = json.loads(response.text)print(parsed_data)
JSON data seems more organized. But these are still in nested form. Let’s flatten this JSON
def flatten_json(json):   dict1 = {}   def flatten(i, name=’’):      if type(i) is dict:         for a in i:             flatten(i[a], name + a + ‘_’)      else:         dict1[name[:-1]] = i   flatten(json)   return dict1df = pd.DataFrame.from_dict(flatten_json(parsed_data), orient=’index’)
flatten_json() will help you to make the nested json in a single json structure.
This JSON is easy to convert to a DataFrame. The csv format data you get there is one column with numerical values. The alphabetical data you are seeing there is actually the index.
Let's apply some Pandas techniques in the dataset
State-wise confirmed cases
State-wise confirmed cases
states = [‘Andhra Pradesh’,’Arunachal Pradesh’, ‘Assam’, ‘Bihar’, ‘Chhattisgarh’, ‘Goa’, ‘Gujarat’, ‘Haryana’, ‘Himachal Pradesh’, ‘Jharkhand’, ‘Karnataka’, ‘Kerala’, ‘Madhya Pradesh’, ‘Maharashtra’, ‘Manipur’, ‘Meghalaya’, ‘Mizoram’, ‘Nagaland’, ‘Odisha’, ‘Punjab’, ‘Rajasthan’, ‘Sikkim’, ‘Tamil Nadu’, ‘Telangana’, ‘Tripura’, ‘Uttar Pradesh’, ‘Uttarakhand’, ‘West Bengal’, ‘Andaman and Nicobar Islands’, ‘Chandigarh’, ‘Dadra and Nagar Haveli’, ‘Daman and Diu’, ‘Delhi’, ‘Lakshadweep’, ‘Puducherry’, ‘Jammu and Kashmir’, ‘Ladakh’]list1 = []list2 = []for i in states: list1.append(df.at[‘state_wise_{}_confirmed’.format(i), 0]) list2.append(i)df1 = pd.DataFrame(list(zip(list2,list1)), columns=[‘state’,’state_value’])
This will help you to get confirmed case details according to states
2.reset_index()
df.reset_index()
Here index is changed. The previous index in the DataFrame has been converted to a column.
3. df.rename()- rename the column names
df = df.rename(columns={“index”: “Statewise_details”, 0 : “Values”})
4. at[]
print(df.at[4,’Statewise_details’],”:”,df.at[4,’Values’])
at[] helps you to access the cell values whenever you require. The output is as below
total_values_deltaconfirmed : 1370
5. Create separate data frames for cured, recovered, deaths and confirmed case according to states and merge it
If you extend the logic of case-1(State-wise confirmed cases), you will end up with the below results
You can perform different operations using pandas. Later you can visualize the data using matplotlib, seaborn and many other libraries and tools.
This blog is a way of collecting data that has been continuously generating in a server. Soon I will be back with another story of web scraping.
I hope you have enjoyed the Natural Language Processing Part-I & Part-2. Follow me up at Medium or Subscribe to my blogs to be informed about them. As always, I welcome feedback and constructive criticism and can be reached on Twitter @RoutraySomesh.