Preview of the Neo4j Graph Data Science plugin with examples from the “Graph Algorithms: Practical Examples in Apache Spark and Neo4j” book
By: Tomaz Bratanic
Tags: Neo4j, Spark, Data Science, Graph

In the past couple of years, the field of data science has gained much traction. It has become an essential part of business and academic research. Combined with the increasing popularity of graphs and graph databases, folks at Neo4j decided to release the Graph Data Science (GDS) plugin. It is the successor of the Graph Algorithms plugin, that is to be deprecated. However, this is still a preview version of the GDS and the official launch will follow in the coming weeks.

Those of you who are familiar with Graph Algorithms plugin will notice that the syntax hasn’t changed much to allow for a smoother transition. To show what has changed, I have prepared the migration guides in the form of Apache Zeppelin notebooks that can be found on GitHub.

Neo4j connector for Apache Zeppelin was developed by Andrea Santurbano, who also designed the beautiful home page notebook of this project and helped with his ideas. In the migrations guides, we used the examples from the Graph Algorithms: Practical Examples in Apache Spark and Neo4j book written by Mark Needham and Amy Hodler. As the book also demonstrates the Apache Spark implementations of the graph algorithms, we decided it is only appropriate that we include them as well. When setting up the Apache Spark environment, I ran into the same issues as the co-author of this book, who was kind enough to write a blog post about it. Shout out to Mark!

Here is what you can expect from this project:

16 single graph algorithms notebooks with implementations in Apache Spark as well as the migration guides to GDS

Link prediction analysis from the book ported to GDS

Neo4j Graph Data Science and Graph Algorithms plugins are not compatible, so they do not and will not work together on a single instance of Neo4j. In this project, we used two Neo4j instances to demonstrate both the old and the new syntax.

To get a better feeling of what the notebooks have to offer, we will now go through the PageRank example notebooks. All the quotes that you shall see in this blog post are excerpts from the book mentioned above.

PageRank algorithm introduction

Each graph algorithm has a main notebook that includes a short presentation of the algorithm as well as its use-cases. The following is the introduction to the PageRank algorithm:

PageRank is named after Google cofounder Larry Page, who created it to rank websites in Google’s search results. The basic assumption is that a page with more incoming and more influential incoming links is more likely a credible source. PageRank measures the number and quality of incoming relationships to a node to determine an estimation of how important that node is. Nodes with more sway over a network are presumed to have more incoming relationships from other influential nodes.

On the bottom of the main notebook, you will find links to Neo4j and/or Apache Spark implementation of the algorithm.

PageRank implementation in Neo4j

In the actual notebooks, you will find both the Graph Algorithms and the Graph Data Science algorithms examples. Still, for the clarity of this blog post, I decided to show only the new GDS syntax.

Load the example graph:

The example graph is available as a CSV file on GitHub and can be easily fetched using a LOAD CSV cypher statement.



WITH “

AS uri

LOAD CSV WITH HEADERS FROM uri AS row

MERGE (:User {id: row.id}); // import nodesWITH “ https://raw.githubusercontent.com/neo4j-graph-analytics/book/master/data/social-nodes.csv AS uriLOAD CSV WITH HEADERS FROM uri AS rowMERGE (:User {id: row.id});

WITH “

AS uri

LOAD CSV WITH HEADERS FROM uri AS row

MATCH (source:User {id: row.src})

MATCH (destination:User {id: row.dst})

MERGE (source)-[:FOLLOWS]->(destination); // import relationshipsWITH “ https://raw.githubusercontent.com/neo4j-graph-analytics/book/master/data/social-relationships.csv AS uriLOAD CSV WITH HEADERS FROM uri AS rowMATCH (source:User {id: row.src})MATCH (destination:User {id: row.dst})MERGE (source)-[:FOLLOWS]->(destination);

We can visualize the example graph in Zeppelin by using the following cypher statement:

MATCH p=(:User)-[:FOLLOWS]->(:User)

RETURN p;

Results

Example graph visualized in Apache Zeppelin

Graph data science plugin

Create a pre-loaded named graph

The recommended way of using the GDS library is with pre-loaded named graphs. A projected graph can be stored in memory and be later retrieved by any of the graph algorithms using its name. This allows for more efficient graph analytics pipelines, where we run many graph algorithms in sequence on the same in-memory graph.

The general syntax is:

CALL gds.graph.create('nameOfGraph','NodeLabel','RelationshipType')

In this example, we will be using the following cypher statement to load the graph in memory:

CALL gds.graph.create(‘pagerank_example’, ‘User’, ‘FOLLOWS’);

Stream the results of algorithms

The general syntax for streaming results from algorithms is:

CALL gds.<algorithm>.stream()

For the PageRank example, the cypher procedure looks like:

CALL gds.pageRank.stream(‘pagerank_example’,

{maxIterations: 20, dampingFactor: 0.85}) YIELD nodeId, score

RETURN gds.util.asNode(nodeId).id AS name, score

ORDER BY score DESC LIMIT 10

Results

Neo4j Pagerank results

As we might expect, Doug has the highest PageRank because he is followed by all

other users in his subgraph. Although Mark only has one follower, that follower is

Doug, so Mark is also considered important in this graph. It’s not only the number of followers that is important, but also the importance of those followers.

Write back the results of algorithms

Sometimes, we want to write back the results of the algorithm to the graph instead of streaming them. The general syntax for the write-back algorithms is:

CALL gds.<algorithm>.write({writeProperty:'pagerank'})

Where the writeProperty parameter is mandatory if you want to write-back the results. This is to prevent any unwanted writing to the graph.

In our PageRank example we use:

CALL gds.pageRank.write(‘pagerank_example’,

{maxIterations: 20, dampingFactor: 0.85,

writeProperty:’pageRank’})

Release the loaded graph from memory

After we are done with our graph analysis, we want to release the GDS graph from memory using the following cypher statement.

CALL gds.graph.drop(‘pagerank_example’);

PageRank implemented with Apache Spark

As mentioned before, the book also offers implementations of graph algorithms in Apache Spark, so I just copied them to the notebooks.

Load the example graph:

In this example, we are using GraphFrames library to load the graph and calculate the PageRank centrality.

def create_social_graph():

nodes = spark.read.csv(“spark-warehouse/social-nodes.csv”, header=True)

relationships = spark.read.csv(“spark-warehouse/social-relationships.csv”, header=True)

return GraphFrame(nodes, relationships) g = create_social_graph()

PageRank with a fixed number of iterations

Let’s see an example of the fixed iterations approach.

Notice in Spark that the damping factor is more intuitively called

the reset probability, with the inverse value. In other words, reset

Probability=0.15 in this example is equivalent to dampingFactor:0.85 in Neo4j.

results = g.pageRank(resetProbability=0.15, maxIter=20)

results.vertices.sort(“pagerank”, ascending=False).show()

Results

Spark PageRank with a fixed number of iterations

Results are similar to the Neo4j results.

PageRank implementations vary, so they can produce different

scoring even when the ordering is the same. Neo4j initializes nodes

using a value of 1 minus the dampening factor whereas Spark uses

a value of 1. In this case, the relative rankings (the goal of Pag‐

eRank) are identical but the underlying score values used to reach

those results are different.

PageRank until convergence

And now let’s try the convergence implementation that will run PageRank until it closes in on a solution within the set tolerance:

results = g.pageRank(resetProbability=0.15, tol=0.01)

results.vertices.sort(“pagerank”, ascending=False).show()

Results

Spark PageRank with until convergence strategy

The PageRank scores for each person are slightly different than with the fixed number of iterations variant, but as we would expect, their order remains the same.

Personalized PageRank

We can calculate the personalized PageRank score for a given node by passing in the sourceId parameter. The following code calculates the PPR for Doug:

me = “Doug”

results = g.pageRank(resetProbability=0.15, maxIter=20, sourceId=me)

people_to_follow = results.vertices.sort(“pagerank”, ascending=False) already_follows = list(g.edges.filter(“src = ‘{me}’”).toPandas()[“dst”])

people_to_exclude = already_follows + [me] people_to_follow[~people_to_follow.id.isin(people_to_exclude)].show()

Results

Spark Personalized PageRank

The results of this query could be used to make recommendations for people who

Doug should follow. Notice that we are also making sure that we exclude people who Doug already follows, as well as himself, from our final result. Alice is the best suggestion for somebody that Doug should follow, but we might suggest Michael and Bridget as well.

Conclusion

Again, thanks to Amy and Mark for writing this excellent book, and thanks to the Neo4j graph data science team for going all the way to deliver such a fantastic tool to help us find more insights within our graph. As I mentioned in the intro of this blog posts, currently this is still a preview of the GDS plugin, and the official launch will be coming in the next couple of weeks. Try the Graph Data Science plugin yourself, and please share any comments, feedback, or suggestion.

As always, you can find the notebooks on GitHub.