Preview of the Neo4j Graph Data Science plugin with examples from the “Graph Algorithms: Practical Examples in Apache Spark and Neo4j” book | by Tomaz Bratanic | Towards Data Science

In the past couple of years, the field of data science has gained much traction. It has become an essential part of business and academic research. Combined with the increasing popularity of graphs and graph databases, folks at Neo4j decided to release the Graph Data Science (GDS) plugin. It is the successor of the Graph Algorithms plugin, that is to be deprecated. However, this is still a preview version of the GDS and the official launch will follow in the coming weeks.
Those of you who are familiar with Graph Algorithms plugin will notice that the syntax hasn’t changed much to allow for a smoother transition. To show what has changed, I have prepared the migration guides in the form of Apache Zeppelin notebooks that can be found on GitHub.
Neo4j connector for Apache Zeppelin was developed by Andrea Santurbano, who also designed the beautiful home page notebook of this project and helped with his ideas. In the migrations guides, we used the examples from the Graph Algorithms: Practical Examples in Apache Spark and Neo4j book written by Mark Needham and Amy Hodler. As the book also demonstrates the Apache Spark implementations of the graph algorithms, we decided it is only appropriate that we include them as well. When setting up the Apache Spark environment, I ran into the same issues as the co-author of this book, who was kind enough to write a blog post about it. Shout out to Mark!
Here is what you can expect from this project:
16 single graph algorithms notebooks with implementations in Apache Spark as well as the migration guides to GDS
Link prediction analysis from the book ported to GDS
Neo4j Graph Data Science and Graph Algorithms plugins are not compatible, so they do not and will not work together on a single instance of Neo4j. In this project, we used two Neo4j instances to demonstrate both the old and the new syntax.
To get a better feeling of what the notebooks have to offer, we will now go through the PageRank example notebooks. All the quotes that you shall see in this blog post are excerpts from the book mentioned above.
Each graph algorithm has a main notebook that includes a short presentation of the algorithm as well as its use-cases. The following is the introduction to the PageRank algorithm:
PageRank is named after Google cofounder Larry Page, who created it to rank websites in Google’s search results. The basic assumption is that a page with more incoming and more influential incoming links is more likely a credible source. PageRank measures the number and quality of incoming relationships to a node to determine an estimation of how important that node is. Nodes with more sway over a network are presumed to have more incoming relationships from other influential nodes.
On the bottom of the main notebook, you will find links to Neo4j and/or Apache Spark implementation of the algorithm.
In the actual notebooks, you will find both the Graph Algorithms and the Graph Data Science algorithms examples. Still, for the clarity of this blog post, I decided to show only the new GDS syntax.
The example graph is available as a CSV file on GitHub and can be easily fetched using a LOAD CSV cypher statement.
// import nodesWITH “https://raw.githubusercontent.com/neo4j-graph-analytics/book/master/data/social-nodes.csv"AS uriLOAD CSV WITH HEADERS FROM uri AS rowMERGE (:User {id: row.id});// import relationshipsWITH “https://raw.githubusercontent.com/neo4j-graph-analytics/book/master/data/social-relationships.csv"AS uriLOAD CSV WITH HEADERS FROM uri AS rowMATCH (source:User {id: row.src})MATCH (destination:User {id: row.dst})MERGE (source)-[:FOLLOWS]->(destination);
We can visualize the example graph in Zeppelin by using the following cypher statement:
MATCH p=(:User)-[:FOLLOWS]->(:User)RETURN p;
Results
The recommended way of using the GDS library is with pre-loaded named graphs. A projected graph can be stored in memory and be later retrieved by any of the graph algorithms using its name. This allows for more efficient graph analytics pipelines, where we run many graph algorithms in sequence on the same in-memory graph.
The general syntax is:
CALL gds.graph.create('nameOfGraph','NodeLabel','RelationshipType')
In this example, we will be using the following cypher statement to load the graph in memory:
CALL gds.graph.create(‘pagerank_example’, ‘User’, ‘FOLLOWS’);
The general syntax for streaming results from algorithms is:
CALL gds.<algorithm>.stream()
For the PageRank example, the cypher procedure looks like:
CALL gds.pageRank.stream(‘pagerank_example’,    {maxIterations: 20, dampingFactor: 0.85}) YIELD nodeId, scoreRETURN gds.util.asNode(nodeId).id AS name, scoreORDER BY score DESC LIMIT 10
Results
As we might expect, Doug has the highest PageRank because he is followed by allother users in his subgraph. Although Mark only has one follower, that follower isDoug, so Mark is also considered important in this graph. It’s not only the number of followers that is important, but also the importance of those followers.
Sometimes, we want to write back the results of the algorithm to the graph instead of streaming them. The general syntax for the write-back algorithms is:
CALL gds.<algorithm>.write({writeProperty:'pagerank'})
Where the writePropertyparameter is mandatory if you want to write-back the results. This is to prevent any unwanted writing to the graph.
In our PageRank example we use:
CALL gds.pageRank.write(‘pagerank_example’,    {maxIterations: 20, dampingFactor: 0.85,          writeProperty:’pageRank’})
After we are done with our graph analysis, we want to release the GDS graph from memory using the following cypher statement.
CALL gds.graph.drop(‘pagerank_example’);
As mentioned before, the book also offers implementations of graph algorithms in Apache Spark, so I just copied them to the notebooks.
In this example, we are using GraphFrames library to load the graph and calculate the PageRank centrality.
def create_social_graph():    nodes = spark.read.csv(“spark-warehouse/social-nodes.csv”, header=True)    relationships = spark.read.csv(“spark-warehouse/social-relationships.csv”, header=True)    return GraphFrame(nodes, relationships)g = create_social_graph()
Let’s see an example of the fixed iterations approach.Notice in Spark that the damping factor is more intuitively calledthe reset probability, with the inverse value. In other words, resetProbability=0.15 in this example is equivalent to dampingFactor:0.85 in Neo4j.
results = g.pageRank(resetProbability=0.15, maxIter=20)results.vertices.sort(“pagerank”, ascending=False).show()
Results
Results are similar to the Neo4j results.
PageRank implementations vary, so they can produce differentscoring even when the ordering is the same. Neo4j initializes nodesusing a value of 1 minus the dampening factor whereas Spark usesa value of 1. In this case, the relative rankings (the goal of Pag‐eRank) are identical but the underlying score values used to reachthose results are different.
And now let’s try the convergence implementation that will run PageRank until it closes in on a solution within the set tolerance:
results = g.pageRank(resetProbability=0.15, tol=0.01)results.vertices.sort(“pagerank”, ascending=False).show()
Results
The PageRank scores for each person are slightly different than with the fixed number of iterations variant, but as we would expect, their order remains the same.
We can calculate the personalized PageRank score for a given node by passing in the sourceId parameter. The following code calculates the PPR for Doug:
me = “Doug”results = g.pageRank(resetProbability=0.15, maxIter=20, sourceId=me)people_to_follow = results.vertices.sort(“pagerank”, ascending=False)already_follows = list(g.edges.filter(“src = ‘{me}’”).toPandas()[“dst”])people_to_exclude = already_follows + [me]people_to_follow[~people_to_follow.id.isin(people_to_exclude)].show()
Results
The results of this query could be used to make recommendations for people whoDoug should follow. Notice that we are also making sure that we exclude people who Doug already follows, as well as himself, from our final result.
Alice is the best suggestion for somebody that Doug should follow, but we might suggest Michael and Bridget as well.
Again, thanks to Amy and Mark for writing this excellent book, and thanks to the Neo4j graph data science team for going all the way to deliver such a fantastic tool to help us find more insights within our graph. As I mentioned in the intro of this blog posts, currently this is still a preview of the GDS plugin, and the official launch will be coming in the next couple of weeks. Try the Graph Data Science plugin yourself, and please share any comments, feedback, or suggestion.
As always, you can find the notebooks on GitHub.