The End is Nigh — Part 1: Python analysis on Climate Change Database
By: Marco Aurélio Gemaque Cantuária
Tags: Kaggle Api, Climate Change, Pandas

How to Download the Dataset from Kaggle directly on Google Colab?

We’ll use this dataset from Kaggle in case you’re wondering. There are a couple of alternatives to using Kaggle files on Colab. The most common one, for example, is downloading the data to your drive and connecting that folder to Google Colab. That’s the methodology I usually use, mainly because after processing files and rearranging datasets, you can store them directly into Drive and reuse them without having to rerun the code again and again.

Create a Kaggle Account/Sign in:

Well, I mean, obviously, right?

Figure 1: Click on the black button if you don’t have an account.

2. Download your API Token from Kaggle

Under profile settings, there is the Account option which holds many relevant information, including our dear API key:

Figure 2: How to find the account.

Figure 3: Generate your new API token.

This will download a ‘.json’ file containing your username and key.

3. Connect your Google Colab with your Kaggle API key

Ok, so you have some alternatives in here:

a) first of you can upload it to your google drive and then simply reference it whenever you want for all the kaggle datasets you can download; or

b) upload it from your local machine every time you wish to download it.

We’ll go with path a) in here since it makes our job easier!

On Google Colab, just write the following code lines:

from google.colab import drive

drive.mount('/content/drive')

A brief explanation: the import drive will import the drive class which will connect to drive with the method drive.mount().

This will open a new tab asking you to validate your gmail account, and you’ll have to write your password, so don’t be scared of that. Also, don’t forget to upload your ‘.json’ file to your drive! It’s better to save it on a specific folder, for example, mine is:

path = "/content/drive/My Drive/A Duck's Dream/Climate Change/kaggle.json"

The “/content/drive/My Drive” is standard, so you’ll have to replicate that, but the remaining part will be the folder’s path of wherever you saved your file in.

We’ll have to use python’s json package in order to read the file and its content, so:

import json

path = "/content/drive/My Drive/A Duck's Dream/Climate Change/kaggle.json"

with open(path) as json_file:

data = json.load(json_file)

If you want to check whether the upload was successful, then just print the content of it like this:

print(data)

You should have something like this:

Figure 4: Did you really thought I was going to show it?

4. Install Kaggle Library and download kaggle dataset to your notebook

To install Kaggle you just use the commands from the prompt:

# Install Kaggle library

!pip install -q kaggle

The “-q” command means quiet, with as less verbose as possible.

So, our next should be to download our dataset, but in order to do this, Kaggle requires that our ‘kaggle.json’ API key is located in the folder, otherwise it won’t work:

/root/.kaggle

Our strategy in here is very simple: we create that folder, then we copy the ‘kaggle.json’ file in it, and finally we download our dataset. The command lines we need for that are below

!mkdir /root/.kaggle #create the folder

!ls /root/.kaggle #check if we actually created it

!cp "/content/drive/My Drive/A Duck's Dream/Climate Change/kaggle.json" /root/.kaggle #copy the file here

In case Python complains that there is already a “/root/.kaggle” folder, then just delete that line and copy the file. Notice that my path is different than yours! So replace it properly hehehe.

With all well documented, now the only thing we actually need to do is download our dataset:

!kaggle datasets download -d berkeleyearth/climate-change-earth-surface-temperature-data

The “-d” command in UNIX returns the path, and not the content itself, so we’re using that command to download the files from that path.

The whole operations takes about 2 seconds and then a new zip file containing the dataset will be available for you under the path:

Figure 5: Straight out of the oven.

5. Unzip our files and start using it:

First of all consider we don’t want to extract all of the files, let’s say we only want two of the ‘.csv’ files, for instance. We’ll need to use the ZipFile library in order to deal with the zips, and also Pandas in order to convert the file into a dataframe:

import zipfile

z = zipfile.ZipFile('/content/climate-change-earth-surface-temperature-data.zip','r')

df_city = pd.read_csv(z.open('GlobalLandTemperaturesByMajorCity.csv'))

df_country = pd.read_csv(z.open('GlobalLandTemperaturesByCountry.csv'))

And there you have it, everything in its right place.

Now, stay tuned for Part 2 where we start processing the file and applying EDA to it in order to extract relevant information from it!