Spreadsheets to Python : Monitoring Covid Made Easy, Using Code | by Clive Siviour | Towards Data Science

The massive availability of data on the internet gives us an unprecedented opportunity to explore the underlying numbers behind different phenomena and events. Often these data are updated as time goes on, and we want to take advantage of this by updating our analysis. For me, one of the most important reasons to move from spreadsheets to code is that this process becomes so much easier.
A few months ago, I wrote a blog post examining the start of the ‘third wave’ of Covid in the UK. For the post I wrote code to read in data from the UK government website [1]. I’m now interested in how the now dominant Omicron variant is affecting case numbers; the good news is that it is really easy to reuse the code that I produced in order to find out.
I’m writing this blog in Filament, a cloud-based Python platform, which allows me to re-run the code easily from my phone or computer whenever I am interested. At this point I should say that I am CSO of Filament. Of course, the same code will work on other Python platforms.
There has been a big increase recently in the number of Covid cases in the UK; let’s see how it compares to previous waves. To do this, I’m going to use two libraries, Pandas for storing the data and Matplotlib for plotting them, so first I import the libraries that I need. To avoid breaking up the text, in this post all of the code is presented at the end. I encourage readers to copy and run this code themselves; alternatively, users of Filament can launch a workspace containing code from a range of blog-posts, including the ones in this series.
I then import a csv file containing the number of cases per day. I’ve chosen to use a csv organised by the date the specimen was taken, because I think this will be more accurate for my later calculations. This means that the data for the last few days are incomplete. The other option would be to download a file organised by the date the result was reported. This is more up-to-date, but might affect calculations of hospitalisation rates.
Once the file is imported, it’s simple to produce a plot showing the number of cases per day. We can see the big uptick in cases over the past few weeks!
For the next graph, I have downloaded the number of hospital admissions per day. We expect these to lag cases a bit (we’ll explore this a bit more later), but even so, we can see a big increase at the end.
And, finally, deaths. There are a number of different measures of deaths; this graph shows deaths within 28 days of a positive test; the death was not necessarily from Covid. It’s easy to change the code to read one of the other measures instead.
These data are interesting in themselves, but what is more important is that it is now really easy to re-run the code and update the graphs whenever I want to: I just have to hit run again! This would be a lot more difficult if I manually downloaded the spreadsheets each day.
Let’s now do some simple calculations with these data. The first thing I am going to do is look at the ratio of cases to hospital admissions and see how this has changed over time. I am going to miss out the first Covid wave, in spring 2020, because the testing was limited and it is likely that a lot of cases were missed. Instead, I will start on the 1st September 2020. Before calculating a ratio, the first thing I need to do is align the two sets of data: it turns out that the peak in hospitalisations is 8 days after the peak in cases. This implies that, on average, people who go to hospital do so 8 days after their positive test. I account for this by shifting the hospital admissions by 8 days. To make things easier to view, I have smoothed the data with a 7-day rolling average.
Now let’s have a look at the ratio of cases to admissions. The graph below shows this (on the right-hand axis). In late 2020, about one in 15 people who caught Covid-19 in the UK went to hospital; in mid-2021, presumably as vaccination took effect, this improved to about one in 40. The ratio seems to be improving again now; however, it’s too early to be sure and the data may have been affected by reporting delays over Christmas. I’ll keep checking back to see how things evolve.
Finally, lets take a look at death rates, for the graph below I have shifted these by 15 days to overlap with the cases; I also scaled them by multiplying by 50 so that cases and deaths can be viewed on the same axis. Around a year ago, there was one death for every 50 cases, this improved rapidly before settling at around one for every 250 cases or so, the very high ratio around July 2021 is probably a result of the Euro 2020 football tournament, during which I expect a large number of cases were among young people who were less likely to be seriously ill. It appears as though the recent rise in cases has not been associated with a rise in deaths, which will be good news if it continues, but may also be an artifact of the shifting: more data are needed before firm conclusions can be drawn.
We have seen how Python can be used to visualise and analyse data obtained from an internet source. Unfortunately, a bit more time is required before these data can be used to draw reliable conclusions about the Omicron variant of Covid: the new wave of cases is too recent and the available data are probably not yet up-to-date because of disruption over Christmas and the New Year.
The great thing about using code, however, is that I can now rerun the analysis every few days simply by, quite literally, pressing a single ‘run’ button. This is a huge advantage of performing this analysis in Python. The analysis is also adaptable, I can change the amount of averaging, the shifts between different sets of data, or even the data set used just by changing one or two lines and running the code again.
One of the main assumptions in the analysis above is that the time delay between cases and hospital admissions (or deaths) is the same in the current (fourth) wave as it was in the second. This is a difficult assumption to justify — most people are not vaccinated, and Omicron appears to be a very different virus. If cases stabilise, this won’t matter; whilst they are rising quickly, small mis-alignments might make a large difference to the calculated ratios. This is something to continue tweaking in the future, again made easier by code.
I hope this simple example has again shown the value of performing data analyses and visualisation in Python.
The text, code and images for this post were created using Filament, the all-in-one workspace for data, analysis and reporting. To find out more please visit our website. Filament is currently running a closed Beta program; the first 100 people to request access using referral code TDSFILAMENT can skip the waitlist and get early access. All code for this and other blogs is available to all Filament users.
Follow me on Medium for more stories like this
Connect on LinkedIn
[1] See Fair Use Policy here: https://coronavirus.data.gov.uk/details/download
#%% Code to import and plot UK Covid Dataimport matplotlib.pyplot as pltimport matplotlib.ticker as tkrfrom matplotlib.dates import DateFormatterimport pandas as pd#%% Read case data into a DataFramecovid_cases=pd.read_csv('https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=overview&structure=%7B%22areaType%22:%22areaType%22,%22areaName%22:%22areaName%22,%22areaCode%22:%22areaCode%22,%22date%22:%22date%22,%22newCasesBySpecimenDate%22:%22newCasesBySpecimenDate%22,%22cumCasesBySpecimenDate%22:%22cumCasesBySpecimenDate%22%7D&format=csv')# Change date formats in the DataFrame, and reorder so that the most recent data are lastcovid_cases['date']=pd.to_datetime(covid_cases['date'])covid_cases=covid_cases[::-1]covid_cases.reset_index(drop=True, inplace=True)#%% Plotfig, ax1 = plt.subplots(figsize = [4,3], dpi=200)ax1.plot(covid_cases['date'],          covid_cases['newCasesBySpecimenDate'],          label='Full data',          color='b'         )# Format the axesax1.tick_params(axis='both', which='major', labelsize=8)ax1.set_xlabel('Date', fontsize=10)ax1.set_ylabel('UK Cases per day', fontsize=10)ax1.set_xticks(ax1.get_xticks())ax1.set_xticklabels(ax1.get_xticklabels(), rotation=45, ha='center')# this code adds a comma at the thousand separator on the y-axisax1.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ',')))# these two lines are used to set the date format to MMM-YYdate_form = DateFormatter("%b-%y")       ax1.xaxis.set_major_formatter(date_form) #%% Read hospital admission data into a new DataFramecovid_hospital=pd.read_csv('https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=overview&structure=%7B%22areaType%22:%22areaType%22,%22areaName%22:%22areaName%22,%22areaCode%22:%22areaCode%22,%22date%22:%22date%22,%22newAdmissions%22:%22newAdmissions%22,%22cumAdmissions%22:%22cumAdmissions%22%7D&format=csv')covid_hospital['date']=pd.to_datetime(covid_hospital['date'])covid_hospital=covid_hospital[::-1]covid_hospital.reset_index(drop=True, inplace=True)#%% Plotfig, ax2 = plt.subplots(figsize=[4,3] ,dpi=200)ax2.plot(covid_hospital['date'],          covid_hospital['newAdmissions'],          label='Full data',          color='r'         )# Format the axesax2.tick_params(axis='both', which='major', labelsize=8)ax2.set_xlabel('Date', fontsize=10)ax2.set_ylabel('UK Hospital admissions per day', fontsize=10)ax2.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax2.xaxis.set_major_formatter(date_form)#%% Read death data into a new DataFramecovid_deaths=pd.read_csv('https://coronavirus.data.gov.uk/api/v1/data?filters=areaType=overview&structure=%7B%22areaType%22:%22areaType%22,%22areaName%22:%22areaName%22,%22areaCode%22:%22areaCode%22,%22date%22:%22date%22,%22newDeaths28DaysByDeathDate%22:%22newDeaths28DaysByDeathDate%22,%22cumDeaths28DaysByDeathDate%22:%22cumDeaths28DaysByDeathDate%22%7D&format=csv')                         covid_deaths['date']=pd.to_datetime(covid_deaths['date'])covid_deaths=covid_deaths[::-1]covid_deaths.reset_index(drop=True, inplace=True)#%% Plotfig, ax3 = plt.subplots(figsize=[4,3] ,dpi=200)ax3.plot(covid_deaths['date'],          covid_deaths['newDeaths28DaysByDeathDate'],          label='Full data',          color='g'         )# Format the axesax3.tick_params(axis='both', which='major', labelsize=8)ax3.set_xlabel('Date', fontsize=10)ax3.set_ylabel('UK Deaths per day', fontsize=10)ax3.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ',')))            ax3.xaxis.set_major_formatter(date_form)#%% Compare cases and hospital admissions, align admissions# Choose shift and averaging periodshift_1 = 8 # number of days to shift admissions data to compare to case dataav_period = 7 # days over which to average# In order to align the two datasets, I find the index of the date I want to start plotting fromstart_1=covid_cases[covid_cases['date']=='2020-09-01'].index[0]start_2=covid_hospital[covid_hospital['date']=='2020-09-01'].index[0] # Create figure and axesfig, ax4 = plt.subplots(figsize=[4,3] ,dpi=200)ax4_2=ax4.twinx() # allows me to plot the data using two y-axes# Now draw the plotsax4.plot(covid_cases['date'][start_1:-shift_1],         covid_cases['newCasesBySpecimenDate'][start_1:-shift_1].rolling(av_period).mean(),        label='Cases',         color='b'        )ax4_2.plot(covid_hospital['date'][start_2:],          covid_hospital['newAdmissions'][start_2:].rolling(av_period).mean(),           label='Hospital Admissions         ',           color=(1,0,0,0.5)          )ax4_2.plot(covid_hospital['date'][start_2:-shift_1],          covid_hospital['newAdmissions'][start_2+shift_1:].rolling(av_period).mean(),           label='Hospital Admissions         \nShifted',           color='r'          )# Format the axesax4.legend(fontsize=6, frameon=False, loc='upper left' )ax4_2.legend(fontsize=6, frameon=False, loc='upper right' )#ax4_2.legend(fontsize=6, facecolor='white', edgecolor='white',framealpha=1, loc='upper right')ax4.tick_params(axis='both', which='major', labelsize=8)ax4_2.tick_params(axis='both', which='major', labelsize=8)ax4.set_xlabel('Date', fontsize=10)ax4.set_ylabel('Cases per day', fontsize=10)ax4_2.set_ylabel('Hospital admissions per day', fontsize=10)        ax4.set_ylim([0,65000])ax4_2.set_ylim([0,4500])ax4.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax4_2.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax4.set_xticks(ax4.get_xticks())ax4.set_xticklabels(ax4.get_xticklabels(), rotation=45, ha='center')ax4.xaxis.set_major_formatter(date_form)#%% Plot hospitalisation ratios# create a DataFrame with the required datahospitalisation_rate=pd.DataFrame({'Date':covid_cases['date'][start_1:]})hospitalisation_rate['Cases']=covid_cases['newCasesBySpecimenDate'][start_1:]hospitalisation_rate.reset_index(drop=True, inplace=True)temp=pd.DataFrame({'Admissions': covid_hospital['newAdmissions'][start_2:]})temp.reset_index(drop=True, inplace=True)hospitalisation_rate['Admissions']=temptemp=pd.DataFrame({'Admissions_shifted': covid_hospital['newAdmissions'][start_2+shift_1:]})temp.reset_index(drop=True, inplace=True)hospitalisation_rate['Admissions_shifted']=tempdel temphospitalisation_rate['Ratio']=hospitalisation_rate['Cases'].rolling(7).mean()/hospitalisation_rate['Admissions_shifted'].rolling(7).mean()#%% Plotfig, ax5 = plt.subplots(figsize=[4,3] ,dpi=200)ax5.plot(hospitalisation_rate['Date'],          hospitalisation_rate['Cases'].rolling(7).mean(),          label='Cases',         color='b'         )ax5.plot(hospitalisation_rate['Date'],          hospitalisation_rate['Admissions'].rolling(7).mean(),          label='Hospital Admissions',         color=(1,0,0,0.5)         )ax5.plot(hospitalisation_rate['Date'],          hospitalisation_rate['Admissions_shifted'].rolling(7).mean(),         label='Hospital Admissions \nShifted',         color='r'         )ax5_2 = ax5.twinx()ax5_2.plot(hospitalisation_rate['Date'],            hospitalisation_rate['Ratio'],            label='Ratio            ',           color='k'           )ax5.legend(fontsize=6, frameon=False, loc='upper left' )ax5_2.legend(fontsize=6, frameon=False, loc='upper right')ax5.tick_params(axis='both', which='major', labelsize=8)ax5_2.tick_params(axis='both', which='major', labelsize=8)ax5.set_xlabel('Date', fontsize=10)ax5.set_ylabel('Cases / Admission per day', fontsize=10)ax5_2.set_ylabel('Ratio', fontsize=10)    ax5.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax5_2.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax5.set_xticks(ax5.get_xticks())ax5.set_xticklabels(ax5.get_xticklabels(), rotation=45, ha='center')ax5.xaxis.set_major_formatter(date_form)ax5.set_ylim(bottom=0)ax5_2.set_ylim(bottom=0)#%% Plot death ratios# create a DataFrame with the required datastart_3=covid_deaths[covid_deaths['date']=='2020-09-01'].index[0]shift_2=15temp=pd.DataFrame({'Deaths_shifted': covid_deaths['newDeaths28DaysByDeathDate'][start_3+shift_2:]})temp.reset_index(drop=True, inplace=True)hospitalisation_rate['Deaths_shifted']=tempdel temphospitalisation_rate['Deaths_Ratio']=hospitalisation_rate['Cases'].rolling(7).mean()/hospitalisation_rate['Deaths_shifted'].rolling(7).mean()#%% Plotfig, ax6 = plt.subplots(figsize=[4,3] ,dpi=200)ax6.plot(hospitalisation_rate['Date'],          hospitalisation_rate['Cases'].rolling(7).mean(),          label='Cases',         color='b'         )ax6.plot(hospitalisation_rate['Date'],          hospitalisation_rate['Deaths_shifted'].rolling(7).mean()*50,          label='Deaths Shifted x 50',         color='lightgreen'         )ax6_2 = ax6.twinx()ax6_2.plot(hospitalisation_rate['Date'],            hospitalisation_rate['Deaths_Ratio'],            label='Ratio            ',           color='k'           )ax6.legend(fontsize=6, frameon=False, loc='upper left' )ax6_2.legend(fontsize=6, frameon=False, loc='upper right')ax6.tick_params(axis='both', which='major', labelsize=8)ax6_2.tick_params(axis='both', which='major', labelsize=8)ax6.set_xlabel('Date', fontsize=10)ax6.set_ylabel('Cases / Deaths per day', fontsize=10)ax6_2.set_ylabel('Ratio', fontsize=10)    ax6.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax6_2.get_yaxis().set_major_formatter(    tkr.FuncFormatter(lambda x, p: format(int(x), ','))) ax6.set_xticks(ax6.get_xticks())ax6.set_xticklabels(ax6.get_xticklabels(), rotation=45, ha='center')ax6.xaxis.set_major_formatter(date_form)ax6.set_ylim(bottom=0)