How to Search and Download Data using Kaggle API? | by Satyam Kumar | Towards Data Science

Kaggle is the world’s largest data science community with powerful tools, datasets, and other resources to help you achieve your data science goals. Kaggle contains tons of freely available datasets used for educational purposes. It also hosts competitions and has freely available notebook to explore and run data science and machine learning models.
To use Kaggle resources and participate in Kaggle competitions you need to log in to the Kaggle website and search correspondingly. To download a dataset from Kaggle one needs to search for the dataset and download it manually and move to the desired folder to further explore.
All interactions with Kaggle can be done using a Kaggle API via the command-line tool (CLI) implemented in Python.
To install the Kaggle library using pip:
pip install kaggle
To use Kaggle’s publicly available API to interact with Kaggle resources, firstly you need to authenticate using an API token. Follow the steps to download a new authentication token onto your machine to complete the authentication:
Click on your profile picture and click on Account from the drop-down menu.
Scroll down to the API section.
Click on the ‘Create New API Token’ button to download a fresh token as a JSON file having a username and API key.
Copy the JSON file to ~/.kaggle/ the directory. In the windows system, go to the root directory, then to .kaggle folder, and copy the downloaded file to this directory.
If you are using the Kaggle API directly, where you keep the token doesn’t matter, so long as you can provide your credentials at runtime.
You can interact using Kaggle API to use its resources without login into its website now and then, below is the list of interactions available using API via command line statement:
Searching for the dataset
Download the dataset
Creating and maintaining the dataset
Searching for published notebook
Download a published notebook
Creating and Running a notebook
Interacting with competitions
Submitting to a competition
Using CLI arguments you can search for any keyword to find the corresponding datasets. CLI statement to get the list of datasets using search statement:
kaggle datasets list -s [KEYWORD]
After you have searched for the appropriate dataset using CLI arguments for searching, this API provides an advantage to download any datasets from Kaggle to your local machine. Commands to download the files associated with the datasets using CLI:
kaggle datasets download -d [DATASET]
Kaggle API can be used to upload new datasets and versions of datasets using CLI arguments. This can ease the sharing of datasets and projects on Kaggle.
To create a new dataset follow the below steps:
Assemble the dataset files in a folder to upload on Kaggle.
To generate metadata, run: kaggle datasets init -p /path/to/dataset
Add the metadata to the generated file: datapackage.json
To finally create the dataset run: kaggle datasets create -p /path/to/dataset
To upload a new version of the existing dataset follows the below steps:
To generate metadata, run: kaggle datasets init -p /path/to/dataset
Make sure the id field in the metadata file datapackage.json points to your dataset.
To finally create the dataset run:kaggle datasets version -p /path/to/dataset -m "MESSAGE"
Using Kaggle API you can search using a keyword to find corresponding published notebooks. It enables to search for published Notebooks and their metadata as well as workflows for creating and running Notebooks.
CLI statement to get the list of published notebooks using search keyword:
kaggle kernels list -s [KEYWORD]
Kaggle API provides the advantage to download any published notebooks from Kaggle to your local machine. Commands to download the files associated with the notebooks using CLI:
kaggle kernels pull -k [KERNEL] -p /path/to/download -m
Kaggle API can be used to upload new notebooks and maintain versions of notebooks using CLI arguments. This can ease sharing notebooks and projects on Kaggle.
To create new notebooks follow the below steps:
Assemble the code files (notebooks) in a folder to upload on Kaggle.
To generate metadata, run: kaggle kernels init -p /path/to/kernel
Add the metadata to the generated file: kernel-metadata.json
To finally create the dataset run: kaggle kernels push -p /path/to/kernel
To upload a new version of the existing dataset follows the below steps:
Download the last version of your notebook and corresponding metadata to your local machine: kaggle kernels pull -k [KERNEL] -p /path/to/download -m
Make sure the id field in the metadata file kernel-metadata.json points to your notebook.
To finally push the new version of the notebook run:kaggle kernels version -p /path/to/kernel
Kaggle API tools provide an easy way to interact with the competitions hosted on Kaggle. To accept the rules for any competition, you need to login to the competition site and accept the rules to download the dataset and make the submission. You need to visit the Kaggle website and accept the rules there, as it is not possible through the API.
Commands to interact with the competition hosted on Kaggle:
List of all ongoing competitions: kaggle competitions list
Download the files associated with a competition: kaggle competitions download -c [COMPETITION]
Submitting to any competition is only possible after you have accepted the submission rules by visiting the Kaggle competition website page.
CLI arguments to submit to competition and get yourself scored:
kaggle competitions submit -c [COMPETITION NAME] -f [FILE PATH]
To list all the previous submission to a competition run:
kaggle competitions submissions -c [COMPETITION NAME]
In this article, we have discussed how to use Kaggle API to use Kaggle resources via CLI arguments. Using CLI commands we can interact with datasets, interact with notebooks, or interact with any competitions.
To explore additional CLI arguments, you can always append -h after any call to see the help menu for that command.
[1] Kaggle API Documentation: https://www.kaggle.com/docs/api
[2] Kaggle API GitHub Repo: https://github.com/Kaggle/kaggle-api
Thank You for Reading