How to do advanced analytics on graph data in Azure Cosmos DB
By: René Bremer
Tags: Azure, Programming, Spark, Graph Database, Data Science

How to do advanced analytics on graph data in Azure Cosmos DB

Learn to write graph data in Cosmos DB using Gremlin and then to read/analyze data in Azure Databricks with GraphFrames.

0. Introduction

Azure Cosmos DB is a fully managed multi-database service. It enables you to build highly responsive applications worldwide. As part of Cosmos DB, Gremlin is supported for graph databases. Since Cosmos DB is optimized for fast processing (OLTP), traversal limits may apply for heavy analytic workloads (OLAP). In that case, Azure Databricks and GraphFrames can be used as an alternative to do advanced analytics, see also architecture below.

0. Architecture (image by Author)

In the remaining of blog, the following is done:

OLTP: write graph data to Cosmos DB using the Gremlin API and Python

OLAP: read data from Cosmos DB, analyze data in Azure Databricks

Finally, a conclusion is drawn in the last chapter.

1. OLTP: write data to Cosmos DB using Gremlin API

In this chapter, the Gremlin API and Python will be used to write data to Cosmos DB. Since Azure Databricks is used in the next chapter for OLAP, an Azure Databricks cluster is also used to write the data to Cosmos DB using the Gremlin API. However, any Python3 environment can be used to write the data to Cosmos DB, e.g. Visual Code, PyCharm or Azure Functions. The following steps are executed:

1.1. Install prerequisites

1.2. Install Gremlin Python Library in Azure Databricks

1.3. Get and run notebook

See also architecture below.

1. OLTP: write data to Cosmos DB using Gremlin API (image by Author)

1.1. Install prerequisites

The following prerequisites need to be installed:

When creating the database and graph in Cosmos DB, /name can be used as partition key. Take PeopleDB as name of the database and friends as name of the graph, see also below.

1.1. /name as Partition key in friends graph and PeopleDB data (image by Author)

1.2. Install Gremlin Python Library in Azure Databricks

Start your Azure Databricks workspace and go to Cluster. Create a new cluster with Databricks runtime version 6.4 and only 1 worker node, see also below.

1.2a. Create Databricks cluster (image by Author)

Subsequently, go to your cluster and click install libraries. Then select Pypi and search for gremlinpython and click install, see also below.

1.2b. Install gremlinpython libary on cluster (image by Author)

1.3. Get and run notebook

Go to your Azure Databricks workspace, right-click and then select import. In the radio button, select to import the following notebook using URL:

Once the notebook is imported, you need to change the URL of your Cosmos DB endpoint and your Cosmos DB key which can be found in the portal, see also below.

1.3a. Notebook with gremlin API writing to Cosmos DB (image by Author)

Now select to run the notebook and the data will be written to Cosmos DB. Verify whether the data is there, see also below.

1.3b. Data in Cosmos DB (image by Author)

In this chapter, graph data was added using the Gremlin API, Python and Azure Databricks. In the next chapter, the data is analyzed in which the Databricks cluster is reused.

2. OLAP: get data, analyze with Azure Databricks

In this chapter, the data is retrieved from Cosmos DB and analyzed in Azure Databricks using GraphFrames. In this, the data is directly read from Cosmos DB in Azure Databricks. Since Cosmos DB is typically used as OLTP it may be necessary to stage the data first in ADLSgen2 to minimize load on Cosmos DB, prevent time outs when massive data loads are needed and save costs. However, for simplicity, in this blog data is directly read from Cosmos DB to Azure Databricks. The following steps are executed:

2.1. Install Cosmos DB & GraphFrames libaries in Azure Databricks

2.2. Get and run notebook

See also architecture below.

2. OLAP: get graph data from Cosmos DB and analyze using GraphFrames (image by Author)

2.1. Install Cosmos DB & GraphFrames libraries in Azure Databricks

Cosmos DB connector and GraphFrames needs to installed as a jar file to the cluster that was created in the previous chapter. Download the lastest uber Cosmos DB connector using this link to your desktop. Subsequently, go the your Azure Databricks cluster and click install libraries. Then select Jar and drop the jar file, see also below.

2.1a. Install Cosmos DB connector to cluster (image by Author)

Subsequently, download the lastest GraphFreams using this link and install this in the same was as the Cosmos DB connector. Verify that all libraries are installed correctly, see also below.

2.1b. Libraries installed correctly (image by Author)

2.2. Get and run notebook

Go to your Azure Databricks workspace, right-click and then select import. In the radio button, select to import the following notebook using URL:

Once the notebook is imported, you need to change the URL of your cosmos DB endpoint and your Cosmos DB key which can be found in the portal, see also below. Finally you can run the notebook. In the last cell you can find the result of the Label Propagation Algorithm this is used from Community Detection, see also below.

2.2. More advanced queries on Graphs in Azure Databricks (image by Author)

3. Conclusion

Cosmos DB is the multi-model database service in Azure and graph databases are supported. Since Cosmos DB is optimized for OLTP, the traversal limits may apply for heavy OLAP workloads. In that case, Azure Databricks and GraphFrames can be used as an alternative to do advanced analytics, see also architecture below.