Tracking Coronavirus(COVID-19) Spread in India using Python | by Pratik Nabriya | Towards Data Science

Coronavirus or COVID-19 needs no introduction. It has already been declared as a pandemic by WHO and in past couple of weeks it’s impact has been deleterious from both health perspective and an economic one. Plenty has been written about it, especially statistical reports on its exponential growth and the importance of “flattening the curve”.
As of now, most of us are staying and working from home to avoid the spread of corona virus. I decided to utilize the surplus time to write a Python Script that pulls the latest Statewise data of COVID-19 cases from the official website of Ministry of Health and Family Welfare, Government of India and turn it into insightful visualizations using popular Python packages like GeoPandas, Seaborn and Matplotlib.
Beautifulsoup — A library for pulling data out of html and xml files.
Requests — A library for making HTTP requests in python.
GeoPandas — A library for working with geospatial data in python.
PrettyTable — quick and easy to represent tabular data in visually appealing ASCII tables.
and other regular packages like Pandas, Matplotlib and Seaborn.
If you don’t have any of the above mentioned packages installed on your system, please follow the installation instructions that are mentioned in the respective links.
(Note that Geopandas further depends on fiona for file access and descartes and matplotlib for plotting)
To scrape a website using Python, you need to perform these four basic steps:
Sending an HTTP GET request to the URL of the webpage that you want to scrape, which will respond with the HTML content. We can do this by using the Request library of Python.Analyzing the HTML tags and their attributes, such as class, id, and other HTML tag attributes. Also, identifying your HTML tags where your content lives.Fetching and parsing the data using Beautifulsoup library and maintain the data in some data structure such as Dictionary or List.Output data in any file format such as csv, xlsx, json, etc. or use this tabulated data to make visualizations using Seaborn/Matplotlib libraries.
Sending an HTTP GET request to the URL of the webpage that you want to scrape, which will respond with the HTML content. We can do this by using the Request library of Python.
Analyzing the HTML tags and their attributes, such as class, id, and other HTML tag attributes. Also, identifying your HTML tags where your content lives.
Fetching and parsing the data using Beautifulsoup library and maintain the data in some data structure such as Dictionary or List.
Output data in any file format such as csv, xlsx, json, etc. or use this tabulated data to make visualizations using Seaborn/Matplotlib libraries.
If you are new to web scraping, check this blog for the step-by-step explanation that can help you get started with web scarping using Python.
www.pluralsight.com
Import necessary libraries —
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport requestsfrom bs4 import BeautifulSoupimport geopandas as gpdfrom prettytable import PrettyTable
Web Scraping —
url = 'https://www.mohfw.gov.in/'# make a GET request to fetch the raw HTML contentweb_content = requests.get(url).content# parse the html contentsoup = BeautifulSoup(web_content, "html.parser")# remove any newlines and extra spaces from left and rightextract_contents = lambda row: [x.text.replace('\n', '') for x in row]# find all table rows and data cells withinstats = [] all_rows = soup.find_all('tr')for row in all_rows:    stat = extract_contents(row.find_all('td')) # notice that the data that we require is now a list of length 5    if len(stat) == 5:        stats.append(stat)#now convert the data into a pandas dataframe for further processingnew_cols = ["Sr.No", "States/UT","Confirmed","Recovered","Deceased"]state_data = pd.DataFrame(data = stats, columns = new_cols)state_data.head()
Output:
Before we proceed further, notice that the scraped data columns are actually of ‘string’ datatype. We need to convert them into ‘int’ datatype.
state_data[‘Confirmed’] = state_data[‘Confirmed’].map(int)state_data[‘Recovered’] = state_data[‘Recovered’].map(int)state_data[‘Deceased’] = state_data[‘Deceased’].map(int)
You may also choose to present the data using PrettyTable
table = PrettyTable()table.field_names = (new_cols)for i in stats:    table.add_row(i)table.add_row([“”,”Total”,                sum(state_data[‘Confirmed’]),                sum(state_data[‘Recovered’]),               sum(state_data[‘Deceased’])print(table)
Output:
(This is the latest available data at the time of writing this article)
Plotting horizontal barplot to show the statewise total confirmed cases —
sns.set_style(“ticks”)plt.figure(figsize = (15,10))plt.barh(state_data[“States/UT”],    state_data[“Confirmed”].map(int),align = ‘center’, color = ‘lightblue’, edgecolor = ‘blue’)plt.xlabel(‘No. of Confirmed cases’, fontsize = 18)plt.ylabel(‘States/UT’, fontsize = 18)plt.gca().invert_yaxis()plt.xticks(fontsize = 14)plt.yticks(fontsize = 14)plt.title(‘Total Confirmed Cases Statewise’, fontsize = 18 )for index, value in enumerate(state_data[“Confirmed”]):    plt.text(value, index, str(value), fontsize = 12)plt.show()
Output:
group_size = [sum(state_data[‘Confirmed’]),              sum(state_data[‘Recovered’]),              sum(state_data[‘Deceased’])]group_labels = [‘Confirmed\n’ + str(sum(state_data[‘Confirmed’])),                ‘Recovered\n’ + str(sum(state_data[‘Recovered’])),                ‘Deceased\n’ + str(sum(state_data[‘Deceased’]))]custom_colors = [‘skyblue’,’yellowgreen’,’tomato’]plt.figure(figsize = (5,5))plt.pie(group_size, labels = group_labels, colors = custom_colors)central_circle = plt.Circle((0,0), 0.5, color = ‘white’)fig = plt.gcf()fig.gca().add_artist(central_circle)plt.rc(‘font’, size = 12)plt.title(‘Nationwide total Confirmed, Recovered and Deceased Cases’, fontsize = 20)plt.show()
Output:
The shape files used in this article to plot the India map with state boundaries can be downloaded from here.
# reading the shape file of map of India in GeoDataFramemap_data = gpd.read_file(‘Indian_States.shp’)map_data.rename(columns = {‘st_nm’:’States/UT’}, inplace = True)map_data.head()
Output:
I noticed that the names of some of the States and Union Territories(UT) and in the shape file were not consistent with the state names on the government website. So, I modified the State/UT names in the GeoDataFrame to match with the ones on our State dataframe.
Correcting spellings of states in map_data dataframe —
map_data[‘States/UT’] = map_data[‘States/UT’].str.replace(‘&’,‘and’)map_data[‘States/UT’].replace(‘Arunanchal Pradesh’,                              ‘Arunachal Pradesh’, inplace = True)map_data[‘States/UT’].replace(‘Telangana’,                               ‘Telengana’, inplace = True)map_data[‘States/UT’].replace(‘NCT of Delhi’,                               ‘Delhi’, inplace = True)map_data['States/UT'].replace('Andaman and Nicobar Island',                               'Andaman and Nicobar Islands',                                inplace = True)
Merge the two dataframes state_data and map_data on States/UT names —
merged_data = pd.merge(map_data, state_data,                        how = ‘left’, on = ‘States/UT’)merged_data.fillna(0, inplace = True)merged_data.drop(‘Sr.No’, axis = 1, inplace = True)merged_data.head()
Output:
Display the Statewise data on the map of India —
fig, ax = plt.subplots(1, figsize=(20, 12))ax.axis(‘off’)ax.set_title(‘Covid-19 Statewise Data — Confirmed Cases’,              fontdict =  {‘fontsize’: ‘25’, ‘fontweight’ : ‘3’})merged_data.plot(column = ‘Confirmed’, cmap=’YlOrRd’,                  linewidth=0.8, ax=ax, edgecolor=’0.8',                  legend = True)plt.show()
Output:
Covid-19 hasn’t yet hit India in a widespread way. The missing pieces of data are making it impossible to predict how the outbreak will unfold in the coming months. As I conclude this article, I pray for the safety and well-being of everyone in India and around the world.
Note from the editors: Towards Data Science is a Medium publication primarily based on the study of data science and machine learning. We are not health professionals or epidemiologists, and the opinions of this article should not be interpreted as professional advice. To learn more about the coronavirus pandemic, you can click here.