Downloading Kaggle datasets directly into Google Colab
By: Anna D'Angela
Tags: Kaggle Api, Kaggle, Colab, Virtual Machine, Google Drive

Downloading Kaggle datasets directly into Google Colab

A data scientist leaving behind the limited world of her personal computer and embarking on a journey to the virtual machine. (Photo by The New York Public Library on Unsplash)

There are many perks to working in Google Colaboratory, or “Colab” for short, for data science projects. For those like me, who are plugging away on an older laptop — the main highlight is the free access to accelerated hardware GPU/TPU for larger projects.

For the Phase 4 Project of my data science boot camp, I set out to build an image classifier to aide in diagnosing pneumonia from chest X-rays. Excited to embark on my first computer vision assignment (cue laser eyes), I hurried over to Kaggle to download the dataset. My computer informed me it would take 4 hours to download the zip file of almost 6,000 JPEGs. Oh no. As I watched the “estimated time remaining” continue to increase, I began to fill with dread about what this bode for the upcoming model training times. In the tenet of the greyscale society trapped in infomercials: “There’s gotta be a better way!”

Enter Google Colab. Set up to run Python in a similar user style as Jupyter, Colab is optimized to assist with high processing needs by providing access to hosted GPU/TPU (see more here about runtimes). All you need to get started is a Google account. Your Colab notebooks will be saved in your Google Drive, but to access your files from Colab you will need to connect to (or ‘mount’) your Drive into your notebook at the start of each session.

The virtual machine operates out of ‘/content’. This is where your Colab working data needs to be. Screenshot by author.

Colab’s virtual machine operates out of the ‘content’ folder (shown left). Any data you wish to work with during your session should be stored in this folder.

To keep the servers available for the maximum amount of users there are limits to usage at a free or paid tier:

For Free Users

The kernel will time out after 30 minutes of inactively — meaning all variables will need to be reloaded.

The entire contents of the virtual machine ( ‘/content’) will be cleared after 12 hours. Thus, the data will need to be reimported at the start of each 12 hour stretch.

For PRO (Paid) Users

For about $10 monthly, the kernel inactivity time-out will increase to 90 minutes. The contents of the virtual machine will now be cleared after 24 hours.

Alright, I’ve got my driving gloves on and seatbelt buckled. I am ready to go full throttle. But how do I get my data into Colab when I can barely get it onto my computer? One solution is to download the data directly from Kaggle into Colab — bypassing my local computer entirely.

Since Colab clears all data from ‘/content’ after 12/24 hours, I will download the dataset into my Google Drive for storage (and access from anywhere). I will also need to add some code to the top of my Colab project notebook to load it in from Drive for each session. The most efficient way to do this, is to keep the file on Drive in its compressed, zipped state. Even though they are both Google external machines, they do not operate on the same servers. You will need to copy the zip file from Drive to the Colab virtual machine (‘/content’) at the start of each new session and unzip it there. Otherwise, you will need to load in each, individual file from Drive — and didn’t we arrive here for speed?

Once everything has been configured properly it is quite simple and quick to use. Below is the code to get it set up.