Copy Kaggle data into Google Cloud Storage | Towards Data Science

So, you have picked your Kaggle competition, and you want to start training your model and make yourself known on the Kaggle leaderboard.
If like us, you use Google Cloud AI Platform for your data science workloads, one of the first steps in a Kaggle competition is to upload the Kaggle training data into Google Cloud Storage.
We will be using this recently announced (late Nov 2020) Kaggle competition as an example.
www.kaggle.com
This particular competition is an image classification problem with circa 20k training images (jpeg files) in Kaggle. Here’s a simple way to upload these into a Google Cloud Storage bucket.
We assume you already have a Google Cloud Project. If not, you can easily create one (for free) using the link below.
We assume you already have a Google Cloud Project. If not, you can easily create one (for free) using the link below.
cloud.google.com
2. Make sure you have permissions on your Google Cloud Project to create new storage buckets in Google Cloud Storage (GCS) and upload files. You can read the GSC documentation here if you are unsure.
3. We assume you have registered with Kaggle, and have signed up for a competition.
In a browser, navigate to kaggle.comCreate a new Kaggle Notebook. To do this, click <> Notebooks on the left sidebar and click + New NotebookGive your notebook a name so you can easily find it again. I called mine “Cassava — Copy Kaggle Data to GCS”
In a browser, navigate to kaggle.com
Create a new Kaggle Notebook. To do this, click <> Notebooks on the left sidebar and click + New Notebook
Give your notebook a name so you can easily find it again. I called mine “Cassava — Copy Kaggle Data to GCS”
Tip: I prefix my notebooks in Kaggle with the competition name to make them easy to find later on
4. Add the following imports in a new Notebook cell (click the +Code button to add a new cell:
import os from google.cloud import storage
Run the cell by clicking into the cell, and then clicking the play button that appears on the left.
3. Add a reference in your Notebook to the Casava Kaggle training images. To do this, click Add data.
In the window that appears, select the Competition Data tab, find the Casava competition, and click the Add button.
You should now see the Casava data listed under Input in the sidebar:
4. Grant Kaggle access to your Google Cloud Storage service. To do this, select Google Cloud Services from the Add-ons menu:
In the window that appears, tick Cloud Storage and then click the Link Account button.
5. Add a new code cell and declare the following two functions; the first will create a storage bucket, and the second will upload all files to a bucket that are found in the source folder path.
Replace the GCP_PROJECT_ID with your Google Cloud project id.
Once added, run the cell and confirm no errors.
storage_client = storage.Client(project=’GCP_PROJECT_ID')def create_bucket(bucket_name): bucket = storage_client.create_bucket(bucket_name)def upload_files(bucket_name, source_folder): bucket = storage_client.get_bucket(bucket_name) for filename in os.listdir(source_folder):   blob = bucket.blob(filename)  blob.upload_from_filename(source_folder + filename)
6. Add a new cell to create your new GCS bucket. Replace BUCKET_NAME with a suitable, unique name. Run the cell.
bucket_name = ‘BUCKET_NAME’ create_bucket(bucket_name)
Tip: GCS bucket names must be globally unique. An easy way to ensure this, is to add a suffix of your GCP project id (which is also unique).
7. In a browser, open the Google Cloud Console. Type “storage” in the search bar and click on Storage. Confirm you see your new bucket above in the list.
8. Finally, upload the training resources from Kaggle into your new bucket. To do this, add a new cell with the following code:
local_data = ‘../input/cassava-leaf-disease-classification/train_images/’upload_blob(bucket_name, local_data)
Replace the string in bold, with the location of your training data. To get this, click on the Copy file path button (shown below). And then add the folder that contains the data you want to copy (mine was in train_images).
Run that cell and your data will be copied! It can take a little while, depending on your file volumes and size. The Cassava data took about 15 minutes.
Hopefully, you found this walkthrough useful.
Since Google acquired Kaggle in 2019 we are seeing tighter integration with Google Cloud, for example, you can now submit models trained in AutoML. Keep an eye out for a follow-up article to this, where I will do just that using Google Cloud AI Platform Unified and AutoML.
1. In Part 2, we will show how to train a model for this competition using Google Cloud AI Platform Unified and AutoML.
2. View the Cassava Kaggle competition used in this walkthrough.
3. Learn more about Ancoris Data, Analytics & AI