Downloading Datasets into Google Drive via Google Colab | by Kevin Luk | Towards Data Science

Update: (Coming Soon) TensorFlow 2.0 will be launching new distribution strategy for Keras so that you can use the same code to distribute your model on TPUs. Google Colab can directly access Google Cloud TPUs after the release! Watch TF Dev Summit ’19 for more information.
If you are working on an old MacBook Pro like me (Late 2013, with 120GB HD), limited storage will be my greatest hurdle in working on data science project. For those who are also working on data science project with large dataset, I am sure that saving the dataset and training the model on the cloud will definitely ease your mind.
In this tutorial, I will share with you my experience in:
Mounting the Google Drive to Google ColabDownloading the dataset directly to Google Drive via Google Colab
Mounting the Google Drive to Google Colab
Downloading the dataset directly to Google Drive via Google Colab
with the use of Kaggle API;
from competition website which username and password is required while requesting to download
BONUS: One click to enable FREE GPU support in Google Colab to train with Tensorflow.
First, go to your Google Colab then type the below:
from google.colab import drivedrive.mount('/content/gdrive')
The cell will return the following and your needs to go to the link to retrieve the authorization code. Then you are good to go!
If you are able to access Google Drive, your google drive files should be all under:
/content/gdrive/My Drive/
while your current directory will be /content/
For your easy usage, just save the below code snippet and paste it into the Google Colab and you can mount your Google Drive to the notebook easily.
In this section I will share with you my experience in downloading dataset from Kaggle and other competition.
Downloading Kaggle datasets via Kaggle API
Visit www.kaggle.com ⇨ login ⇨ My Account ⇨ Create New API Token
The “kaggle.json” file will be auto downloaded.
Use these code snippets in Google Colab for the task:
from google.colab import filesfiles.upload()  #this will prompt you to upload the kaggle.json
The below will create the necessary folder path.
!pip install -q kaggle!mkdir -p ~/.kaggle!cp kaggle.json ~/.kaggle/!ls ~/.kaggle!chmod 600 /root/.kaggle/kaggle.json  # set permission
Simply download the required dataset with the syntax:
!kaggle competitions download -c ‘name_of_competition’ -p “target_colab_dir”
!kaggle competitions download -c histopathologic-cancer-detection -p /content/gdrive/My\ Drive/kaggle/cancer
Bonus: please see the git gist below for searching Kaggle dataset
For dataset with multiple zip files like the example, I tend to change directory to the designated folder and unzip them one by one.
!unzip -q file[.zip] -d [exdir]
-q suppress the printing of the file names being extracted-d [exdir] optional directory to which to extract files
import osos.chdir('gdrive/My Drive/kaggle/cancer')  #change dir!mkdir train  #create a directory named train/!mkdir test  #create a directory named test/!unzip -q train.zip -d train/  #unzip data in train/!unzip -q test.zip -d test/  #unzip data in test/!unzip sample_submission.csv.zip!unzip train_labels.csv.zip
Go to here to read more from Kaggle API docs.
For competition like ICIAR2018, you will need to provide the username and password while downloading the dataset.
To do this in Google Colab, first you can change your current directory to the folder you wish to save your dataset. Then, use wget instead of using curl command.
!wget --user=your_username --password=your_password http://cdn1.i3s.up.pt/digitalpathology/ICIAR2018_BACH_Challenge.zip
After downloading, you can unzip the file using the same approach above.
After you mounted your Google Drive to Google Colab and downloaded the required dataset, let’s enable GPU in your Colab notebook and train your model.
From task bar: Runtime ⇨ Change runtime type
Hardware accelerator: None ⇨ GPU
Hope you find this tutorial useful and happy cloud computing!
Thanks for Matt Gleeson, Finlay Macrae suggestions to make the content better.