Automatically Update Data Sources in Python | by Joe T. Santhanavanich | Towards Data Science

In Data Sciences, several times, you get an external data source such as a csv table for your Python project from the open-source website or git repository. Some of these datasets may update periodically. Then, you might end up with open a web browser to download that dataset before running your Python script to get the most updated result. This article will show you an essential guide on how to automate this process to keep your data up-to-date in an easy way.
Several open-source datasets can be retrieved from the URL. You may useurllib module to open and read the URL directly in Python. It is a pre-installed module in most Python versions. So you can just import this module and use the following codes to download the data from any URL.
import urllib.requesturl = '<Your URL Here>'output = '<Your output filename/location>'urllib.request.urlretrieve(url, output)#.. Continue data analysis with updated output file
For instance, an example script below shows an example of updating German COVID-19 data from Robert Koch-Institut (RKI, a German Research Institute for disease control and prevention) The data is in csv format and available at this URL. In real practice, some URLs may change or depreciate over time, which makes the Python script run to an error, so we may include try/except to check if the URL is still valid or not. Assume that you want to keep all versions of downloaded files, you may use thedatetime module to add a timestamp to the output filename too.
* Please note that if you use macOS, you should have a Python certificate installed before you can use the urllib library. You can easily do this by running a install certificates.command file in your Macintosh HD > Applications > Python3.x folder.
If your python project relies on the data from the git repository. After you clone the repository and get the dataset to your local machine, you might have to periodically do git fetch or pull to keep your dataset up-to-date. In this case, you may use the GitPython module to automatically update your project dataset automatically. To use this module, you have to install it first, which you can easily use pip to do so.
pip install gitpython
If you run to an error, you may check this full GitPython document to check the requirements you might not have of your os environment. After you have the module installed, you can just easily follow these codes to automatically do the git pull every time you run your code.
import gitrepo = git.Repo('<your repository folder location>')repo.remotes.origin.pull()#... Continue with your Code.
As an example, the JHU CSSE Github repository has provided the COVID-19 time-series dataset in csv format in which the datasets are daily updated. After you clone the repository to your machine, you may add the above codes to the beginning of your Python script to automatically pull the last version every time. In this way, you can make sure that your result is up-to-date.
For the git authentication, if you use ssh, it would be straightforward. In case you use HTTPs, it will be more convenient to save your username and password to your environment. In Mac and Windows, the authentication information is saved automatically after you input for the first time. In Linux, you may force a machine to remember your credentials by changing the credential timeout. For example, you can use the following command to make your machine remember your credentials for one year. (31,536,000 seconds)
$ git config --global credential.helper 'cache --timeout=31536000'
This article shows step-by-step guidelines for updating the dataset from any URL or git repository directly at the beginning of your Python script. In this way, you can make sure that you run a script based on the up-to-date dataset.
I hope you like this article and found it useful for your daily work or projects. Feel free to leave me a message if you have any questions or comments.
About me & Check out all my blog contents: Link
Be Safe and Healthy! 💪
Thank you for Reading. 📚