Using Neo4j with PySpark on Databricks | by Lukas Böhres | Towards Data Science

With the recent release of the official Neo4j Connector for Apache Spark leveraging the Spark DataSource API, there has been a fundamental change in the way that Neo4j data can be queried from within an Apache Spark environment. Alongside this change, the previous Neo4j Spark Connector was marked as deprecated. In this article, I’d like to share an updated end-to-end workflow of setting up a fully interconnected pairing of Neo4j and Spark that makes use of the new connector’s capabilities.
In the process, we will first set up a Neo4j cloud instance using an Azure virtual machine. Afterwards, we will set up an Azure Databricks instance running Spark before finally establishing a connection between both resources using the new Neo4j Connector for Apache Spark. If you already have an up-and-running instance of Neo4j or Databricks, you might of course want to skip the respective steps. However, please note the compatibility information at the top of each step.
First, a quick word on compatibility: The connector we are going to use supports Neo4j versions 3.5 and above. Versions prior to 3.5 are not supported. It does however support both Neo4j Enterprise and Community as well as single instance VMs, Causal Clusters and Neo4j Aura. This article is going to focus on the workflow for a single instance VM.
As our Neo4j instance, we will use the official Neo4j Enterprise VM image. The latest version is listed as Neo4j Enterprise VM version 4.1 in the Azure marketplace. If you don’t have an Enterprise license, there are images of the Community version offered by Websoft9 as well as Container Images by Bitnami:
After the VM is deployed, navigate to it’s Networking tab to make sure that its port settings are correct: In order to query it from Spark, a Bolt port must allow inbound traffic. By default, this is port 7687. Additionally, we will use the Neo4j Web interface to populate the database, for which we need an open HTTP or HTTPS port. By default, they are mapped to port numbers 7474 and 7473. If any of these port rules are missing, add them by clicking the Add Inbound Port Rule button.
Once you verified the port settings, grab the VM’s public IP from its Overview tab and connect to it via HTTP by navigating to http://YOUR.VM.IP.ADDRESS:7474 in your browser. With a freshly started VM, it might take a few minutes before Neo4j has started up and accepts inbound connections. If you use VPN or Proxies, make sure they are configured accordingly, otherwise you might receive a “This site can’t be reached” error.
If all settings are correct, the Neo4j Browser Web UI will pop up:
You will be prompted for login credentials. Enter the default username and password to log in to your database:
username: neo4j
password: neo4j
On your first access, you will immediately have to change the default password for security reasons. After choosing a solid password and logging in, you can populate the database. In this guide, we will use the Cypher query language to create a simple dummy dataset by typing the following into the command line:
UNWIND range(1,100) as idCREATE (p:Person {id:id}) WITH collect(p) as peopleUNWIND people as p1UNWIND range(1,10) as friendWITH p1, people[(p1.id + friend) % size(people)] as p2CREATE (p1)-[:KNOWS {years: abs(p2.id - p2.id)}]->(p2)
As a result, a data structure similar to the following will be created, where each Person node is tagged with an ID and each KNOWS relation has an additional (arbitrary) attribute years that describes how long the two parties have known each other:
That’s it! Our Neo4j graph database is up and running and can’t wait to be queried from Spark. We will now move on to setting up an appropriate Spark environment.
Again, an important note on compatibility: At the time of writing, Neo4j does not support a connector for Spark 3.0. As such, we will have to fall back to a Spark 2.4 environment in order to communicate with Neo4j.
For our setup, we will use an Azure Databricks instance. Search for databricks on the Azure marketplace and create a new resource. Apart from the usual settings (resource group, name, location and pricing tier), no special configuration settings are required.
Once deployed, open up your Databricks workspace by navigating to your Azure resource’s overview tab and clicking the Launch Workspace button.
First, we will need a cluster to work with. Navigate to the Clusters tab and click the Create Cluster button to display the cluster configuration tool:
Choose a cluster name and availability mode and configure the size and amount of worker nodes to your liking. Now, remember that we are forced to use a Spark 2 setup — luckily, Databricks still offers a variety of Spark 2.4.5 distributions. Make sure to select one of them in the Databricks Runtime Version field, e.g. Runtime Version 6.6 running Spark 2.4.5 and Scala 2.11.
Start your cluster and you’re good to go!
Since we now have both Neo4j and Databricks up and running, it’s time to focus on the connection between them. In order to do so, we will need to add the Neo4j Connector for Apache Spark to our Databricks cluster.
In Databricks, navigate to the cluster tab. Select the previously created cluster and access its libraries options:
Now, add the Neo4j Connector for Apache Spark by clicking the Install New button, select Maven and clicking Search Packages. Type ‘neo4j’ to see all available options. At the time of writing, there are three packages coming up in the search:
neo4j-spark-connector: This is the deprecated connector version (2.4). It is no longer actively supported by the developers.
neo4j-connector-apache-spark_2.11: This is the current connector version (4.0) and the one we are going to use.
neo4j-connector-apache-spark_2.12: This is the current connector version as well, the only difference being that is is written for Scala version 2.12. Since our Databricks environment runs Scala 2.11, this one is not suitable for our purpose.
According to the developers, the split in two separate connectors is necessary due to API differences:
Because of the differences in the APIs, different JAR files are needed depending on your scala version. Ensure that you have the appropriate JAR file for your environment.
After the connector is installed, create a new Jupyter notebook in the Workspace tab (right click → Create New). Try reading the node data of your Neo4j database by running the following command after inserting your Neo4j VM’s IP, username and password:
df = spark.read.format("org.neo4j.spark.DataSource")\ .option("url", "bolt://XXX.XXX.XXX.XXX:7687")\ .option("authentication.type", "basic")\ .option("authentication.basic.username", "neo4j")\ .option("authentication.basic.password", "password")\ .option("labels", "Person")\ .load()display(df)
In case you receive an Unable to connect error, make sure your Neo4j VM is still running and its Bolt port accepts inbound traffic. If the query returns
java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
you most likely installed the connector for the wrong Scala version — make sure you select the Scala 2.11 version (see above).
If all worked out, the query will return a DataFrame similar to this:
As an alternative to Python, you can also use Scala for your queries by adding the %scala magic command at the top of your Notebook cells:
%scalaimport org.apache.spark.sql.{SaveMode, SparkSession}val spark = SparkSession.builder().getOrCreate()val df = spark.read.format("org.neo4j.spark.DataSource") .option("url", "bolt://XXX.XXX.XXX.XXX:7687") .option("authentication.type", "basic") .option("authentication.basic.username", "neo4j") .option("authentication.basic.password", "password") .option("labels", "Person") .load()display(df)
Writing to your Graph Database works similar in a similar way. In the following query, we will update our original data to include names for the first two nodes:
df = spark.createDataFrame( [(1, "John"),(2, "Thomas")], ["id", "name"])df.write.format("org.neo4j.spark.DataSource")\ .option("url”, "bolt://XXX.XXX.XXX.XXX:7687")\ .option("authentication.type", "basic")\ .option("authentication.basic.username", "neo4j")\ .option("authentication.basic.password", "password")\ .option("labels", ":Person")\ .option("node.keys", "id")\ .mode("Overwrite")\ .save()
Note that we use the “Overwrite” mode along with setting the node.keys option to the DataFrame’s id column in order to append the new values to our existing nodes. In fact, running the read query again returns our previous result updated with a name property for the first two nodes:
And that’s it! 🎉 You have successfully set up a fully cloud-based and interconnected pair of Apache Spark and Neo4j, allowing you to leverage the full potential of “traditional” Big Data and Graph Databases working hand in hand.
For more query examples and syntax overview, take a deep dive into the official Neo4j Connector for Apache Spark documentation or check the quick guides on reading and writing from/to Neo4j. Additional examples can be found in the Zeppelin notebook example repository. The connector also supports Cypher queries, allowing you to re-use existing queries from Neo4j Desktop / Web applications. Examples for Cypher queries can be found on the official Cypher page.