How to do advanced analytics on graph data in Azure Cosmos DB | by René Bremer | Towards Data Science

Azure Cosmos DB is a fully managed multi-database service. It enables you to build highly responsive applications worldwide. As part of Cosmos DB, Gremlin is supported for graph databases. Since Cosmos DB is optimized for fast processing (OLTP), traversal limits may apply for heavy analytic workloads (OLAP). In that case, Azure Databricks and GraphFrames can be used as an alternative to do advanced analytics, see also architecture below.
In the remaining of blog, the following is done:
OLTP: write graph data to Cosmos DB using the Gremlin API and Python
OLAP: read data from Cosmos DB, analyze data in Azure Databricks
Finally, a conclusion is drawn in the last chapter.
In this chapter, the Gremlin API and Python will be used to write data to Cosmos DB. Since Azure Databricks is used in the next chapter for OLAP, an Azure Databricks cluster is also used to write the data to Cosmos DB using the Gremlin API. However, any Python3 environment can be used to write the data to Cosmos DB, e.g. Visual Code, PyCharm or Azure Functions. The following steps are executed:
1.1. Install prerequisites
1.2. Install Gremlin Python Library in Azure Databricks
1.3. Get and run notebook
See also architecture below.
The following prerequisites need to be installed:
Azure Databricks
Cosmos DB with Gremlin API
When creating the database and graph in Cosmos DB, /name can be used as partition key. Take PeopleDB as name of the database and friends as name of the graph, see also below.
Start your Azure Databricks workspace and go to Cluster. Create a new cluster with Databricks runtime version 6.4 and only 1 worker node, see also below.
Subsequently, go to your cluster and click install libraries. Then select Pypi and search for gremlinpython and click install, see also below.
Go to your Azure Databricks workspace, right-click and then select import. In the radio button, select to import the following notebook using URL:
https://raw.githubusercontent.com/rebremer/cosmosdb-databricks-olap/master/insert_data_CosmosDB_OLTP_Python_Gremlin.py
Once the notebook is imported, you need to change the URL of your Cosmos DB endpoint and your Cosmos DB key which can be found in the portal, see also below.
Now select to run the notebook and the data will be written to Cosmos DB. Verify whether the data is there, see also below.
In this chapter, graph data was added using the Gremlin API, Python and Azure Databricks. In the next chapter, the data is analyzed in which the Databricks cluster is reused.
In this chapter, the data is retrieved from Cosmos DB and analyzed in Azure Databricks using GraphFrames. In this, the data is directly read from Cosmos DB in Azure Databricks. Since Cosmos DB is typically used as OLTP it may be necessary to stage the data first in ADLSgen2 to minimize load on Cosmos DB, prevent time outs when massive data loads are needed and save costs. However, for simplicity, in this blog data is directly read from Cosmos DB to Azure Databricks. The following steps are executed:
2.1. Install Cosmos DB & GraphFrames libaries in Azure Databricks
2.2. Get and run notebook
See also architecture below.
Cosmos DB connector and GraphFrames needs to installed as a jar file to the cluster that was created in the previous chapter. Download the lastest uber Cosmos DB connector using this link to your desktop. Subsequently, go the your Azure Databricks cluster and click install libraries. Then select Jar and drop the jar file, see also below.
Subsequently, download the lastest GraphFreams using this link and install this in the same was as the Cosmos DB connector. Verify that all libraries are installed correctly, see also below.
Go to your Azure Databricks workspace, right-click and then select import. In the radio button, select to import the following notebook using URL:
https://raw.githubusercontent.com/rebremer/cosmosdb-databricks-olap/master/get_data_cosmosDB_OLAP_Scala_GraphFrames.scala
Once the notebook is imported, you need to change the URL of your cosmos DB endpoint and your Cosmos DB key which can be found in the portal, see also below. Finally you can run the notebook. In the last cell you can find the result of the Label Propagation Algorithm this is used from Community Detection, see also below.
Cosmos DB is the multi-model database service in Azure and graph databases are supported. Since Cosmos DB is optimized for OLTP, the traversal limits may apply for heavy OLAP workloads. In that case, Azure Databricks and GraphFrames can be used as an alternative to do advanced analytics, see also architecture below.