Python + Azure Cosmos DB. Create Datasets and Containers. Query... | by Richard Peterson | Towards Data Science

BONUS: Azure CLI commands.
Azure Cosmos DB is a globally distributed, multi-model database service that is elastically scalable and extremely fast. The fact that it is globally distributed means that it is an excellent solution for applications where the data needs to be close to the user, and able to scale quickly for peak usage. It has been marketed as a solution for web, mobile, gaming and IOT applications with massive incoming data and the need for real-time instant queries across the planet. As an example, it is the database for the social gameplay for Halo 5!
In this article, we are going to create an Azure Cosmos DB account and use the Azure Cosmos Python SDK to create a database, create a container, populate the container with data, and query the container using the SQL API. The data we will be working with comes from the UN Human Development Index — Subnational Index, hence the HDI prefixes on resources.
Check out my Github repo for this article, where you can follow along with a Jupyter Notebook and view an executable python script.
When working with Azure, it is always a good idea to house your resources in a Resource Group. The simplest way to do this is from the Azure CLI or cloud shell:
First, log into the Azure CLI with:
az login
Next, create a Resource Group named HDI-cosmosdb-group with the az group create command.
az group create --location eastus \                --name HDI-cosmosdb-group \                --subscription MySubscription
Create a Cosmos DB named hdicosmosdb (must be all lowercase) with az cosmosdb create command. This will take about 10 minutes to complete.
az cosmosdb create --name hdicosmosdb \                   --resource-group HDI-cosmosdb-group \                   --locations regionName=eastus \                   --subscription MySubscription
When this is complete, you will see a Cosmos DB resource in your newly created Resource Group.
Now that the Cosmos DB account has been created, we need the URI Endpoint and the Primary Key to interact with our account. You can find your URI and access key by going to your Cosmos DB account, and scrolling down on the left to find Settings >> Keys. Copy the URI and Primary key to a safe file.
Install the two packages we need into your python environment.
pip install azure-cosmospip install pandas
Now, taking tips from both the azure-cosmos-python github sample and the official Microsoft docs, we will begin to interact with our Cosmos DB using Python. We will start by importing the necessary packages and creating the Cosmos client. By initializing the Cosmos client with our URI endpoint and access key, we will be able to use the azure-cosmos API to interact with our Cosmos DB account.
Using the client initialized to our account, create a Cosmos DB database. This database will house containers, which house items. Items are the individual documents or records/rows that comprise the database.
Containers, also referred to as collections, house the items (documents) that make up the records of the database. The partition key path (‘/country’) is an attribute of the container that Cosmos DB will use to group and partition the dataset behind the scenes. This C#Corner article gives a good explanation of how partition keys could be used to group records by “department.” In our dataset, the column “country” will be used to group records in partitions defined by the record’s value for “country.”
Now, download the Human Development Index — Subnational index using Pandas. Azure Cosmos DB needs one column to identify a unique id for each record/row. We will create a column ‘id’ based on the index of the pandas DataFrame. Also, since each row will end up as a json document in the Cosmos DB, we will need to convert the ‘id’ column to type string.
Here, we will iterate over the rows of the pandas DataFrame and create a dictionary of each row. That dictionary is already nicely formatted for json extraction, which is accomplished with json.dumps(). Each row is then written to the Cosmos DB as the for loop executes. For a 45,000 record dataset this took about 15 minutes.
Now that our dataset has been seeded with data, let’s run a test query.
We want to query all National-level data for the country of Afghanistan.
All of our hard work has paid off. The .head() method shows that the data has been returned successfully and is in a pandas DataFrame.
Let’s celebrate with a line chart of the three main indexes — Health, Income, and Education indexes.
Thanks for coming this far! Azure Cosmos DB is surprisingly easy to set up and it’s json-oriented records are a breeze with python and pandas.