How to: Automate live data to your website with Python | by Truett Bloxsom | Towards Data Science

This tutorial will be helpful for people who have a website that hosts live data on a cloud service but are unsure how to completely automate the updating of the live data so the website becomes hassle free. For example: I host a website that shows Texas COVID case counts by county in an interactive dashboard, but everyday I had to run a script to download the excel file from the Texas COVID website, clean the data, update the pandas data frame that was used to create the dashboard, upload the updated data to the cloud service I was using, and reload my website. This was annoying, so I used the steps in this tutorial to show how my live data website is now totally automated.
I will only be going over how to do this using the cloud service pythonanywhere, but these steps can be transferred to other cloud services. Another thing to note is that I am new to building and maintaining websites so please feel free to correct me or give me constructive feedback on this tutorial. I will be assuming that you have basic knowledge of python, selenium for web scraping, bash commands, and you have your own website. Lets go through the steps of automating live data to your website:
web scraping with selenium using a cloud serviceconverting downloaded data in a .part file to .xlsx filere-loading your website using the os python packagescheduling a python script to run every day in pythonanywhere
web scraping with selenium using a cloud service
converting downloaded data in a .part file to .xlsx file
re-loading your website using the os python package
scheduling a python script to run every day in pythonanywhere
I will not be going through some of the code I will be showing because I use much of the same code from my last tutorial on how to create and automate an interactive dashboard using python found here. Lets get started!
web scraping with selenium using a cloud service
web scraping with selenium using a cloud service
So in your cloud service of choice (mine being pythonanywhere), open up a python3.7 console. I will be showing the code in chunks but all the code can be combined into one script which is what I have done. Also, all the file paths in the code you will have to change to your own for the code to work.
from pyvirtualdisplay import Displayfrom selenium import webdriverimport timefrom selenium.webdriver.chrome.options import Optionswith Display():    # we can now start Firefox and it will run inside the virtual display    browser = webdriver.Firefox()# these options allow selenium to download files    options = Options()    options.add_experimental_option("browser.download.folderList",2)    options.add_experimental_option("browser.download.manager.showWhenStarting", False)    options.add_experimental_option("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream,application/vnd.ms-excel")# put the rest of our selenium code in a try/finally    # to make sure we always clean up at the end    try:        browser.get('https://www.dshs.texas.gov/coronavirus/additionaldata/')# initialize an object to the location on the html page and click on it to download        link = browser.find_element_by_xpath('/html/body/form/div[4]/div/div[3]/div[2]/div/div/ul[1]/li[1]/a')        link.click()# Wait for 30 seconds to allow chrome to download file        time.sleep(30)print(browser.title)    finally:        browser.quit()
In the chunk of code above, I open up a Firefox browser within pythonanywhere using their pyvirtualdisplay library. No new browser will pop on your computer since its running on the cloud. This means you should test out the script on your own computer without the display() function because error handling will be difficult within the cloud server. Then I download an .xlsx file from the Texas COVID website and it saves it in my /tmp file within pythonanywhere. To access the /tmp file, just click on the first “/” of the files tab that proceeds the home file button. This is all done within a try/finally blocks, so after the script runs, we close the browser so we do not use any more cpu time on the server. Another thing to note is that pythonanywhere only supports one version of selenium: 2.53.6. You can downgrade to this version of selenium using the following bash command:
pip3.7 install --user selenium==2.53.6
2. converting downloaded data in a .part file to .xlsx file
import shutilimport globimport os# locating most recent .xlsx downloaded filelist_of_files = glob.glob('/tmp/*.xlsx.part')latest_file = max(list_of_files, key=os.path.getmtime)print(latest_file)# we need to locate the old .xlsx file(s) in the dir we want to store the new xlsx file inlist_of_files = glob.glob('/home/tsbloxsom/mysite/get_data/*.xlsx')print(list_of_files)# need to delete old xlsx file(s) so if we download new xlsx file with same name we do not get an error while moving itfor file in list_of_files:    print("deleting old xlsx file:", file)    os.remove(file)# move new data into data dirshutil.move("{}".format(latest_file), "/home/tsbloxsom/mysite/get_data/covid_dirty_data.xlsx")
When you download .xlsx files in pythonanywhere, they are stored as .xlsx.part files. After some research, these .part files are caused when you stop a download from completing. These .part files cannot be opened with typical tools but there is a easy trick around this problem. In the above code, I automate moving the new data and deleting the old data in my cloud directories. The part to notice is that when I move the .xlsx.part file, I save it as a .xlsx file. This converts it magically, and when you open this new .xlsx file, it has all the live data which means that my script did download the complete .xlsx file but pythonanywhere adds a .part to the file which is weird but hey it works.
3. re-loading your website using the os python package
import pandas as pdimport relist_of_files = glob.glob('/home/tsbloxsom/mysite/get_data/*.xlsx')latest_file = max(list_of_files, key=os.path.getctime)print(latest_file)df = pd.read_excel("{}".format(latest_file),header=None)# print out latest COVID data datetime and notesdate = re.findall("- [0-9]+/[0-9]+/[0-9]+ .+", df.iloc[0, 0])print("COVID cases latest update:", date[0][2:])print(df.iloc[1, 0])#print(str(df.iloc[262:266, 0]).lstrip().rstrip())#drop non-data rowsdf2 = df.drop([0, 1, 258, 260, 261, 262, 263, 264, 265, 266, 267])# clean column namesdf2.iloc[0,:] = df2.iloc[0,:].apply(lambda x: x.replace("\r", ""))df2.iloc[0,:] = df2.iloc[0,:].apply(lambda x: x.replace("\n", ""))df2.columns = df2.iloc[0]clean_df = df2.drop(df2.index[0])clean_df = clean_df.set_index("County Name")clean_df.to_csv("/home/tsbloxsom/mysite/get_data/Texas county COVID cases data clean.csv")df = pd.read_csv("Texas county COVID cases data clean.csv")# convert df into time series where rows are each date and clean updf_t = df.Tdf_t.columns = df_t.iloc[0]df_t = df_t.iloc[1:]df_t = df_t.iloc[:,:-2]# next lets convert the index to a date time, must clean up dates firstdef clean_index(s):    s = s.replace("*","")    s = s[-5:]    s = s + "-2020"    #print(s)    return sdf_t.index = df_t.index.map(clean_index)df_t.index = pd.to_datetime(df_t.index)# initalize df with three columns: Date, Case Count, and Countyanderson = df_t.T.iloc[0,:]ts = anderson.to_frame().reset_index()ts["County"] = "Anderson"ts = ts.rename(columns = {"Anderson": "Case Count", "index": "Date"})# This while loop adds all counties to the above ts so we can input it into plotlyx = 1while x < 254:    new_ts = df_t.T.iloc[x,:]    new_ts = new_ts.to_frame().reset_index()    new_ts["County"] = new_ts.columns[1]    new_ts = new_ts.rename(columns = {new_ts.columns[1]: "Case Count", "index": "Date"})    ts = pd.concat([ts, new_ts])    x += 1ts.to_csv("/home/tsbloxsom/mysite/data/time_series_plotly.csv")time.sleep(5)#reload website with updated dataos.utime('/var/www/tsbloxsom_pythonanywhere_com_wsgi.py')
Most of the above code I explained in my last post which deals with cleaning excel files using pandas for inputting into a plotly dashboard. The most important line for this tutorial is the very last one. The os.utime function shows access and modify times of a file or python script. But when you call the function on your Web Server Gateway Interface (WSGI) file it will reload your website!
4. scheduling a python script to run every day in pythonanywhere
Now for the easy part! After you combine the above code into one .py file, you can make it run every day or hour using pythonanywhere’s Task tab. All you do is copy and paste the bash command, with the full directory path, you would use to run the .py file into the bar in the image above and hit the create button! Now you should test the .py file using a bash console first to see if it runs correctly. But now you have a fully automated data scraping script that your website can use to have daily or hourly updated data displayed without you having to push one button!
If you have any questions or critiques please feel free to say so in the comments and if you want to follow me on LinkedIn you can!