Node embeddings: Node2vec with Neo4j
By: Tomaz Bratanic
Tags: Graph, Neo4j, Node2vec, Word2vec, Graphs Are Fun

Node embeddings: Node2vec with Neo4j

Learn how to train your custom node2vec algorithm with Neo4j Graph Data Science

My last blog post about combining graphs with NLP techniques was the most successful by far. It motivated me to write more about this topic. During my research, I stumbled upon the node2vec algorithm and noticed how easy it would be to implement it with Neo4j and Graph Data Science library. I guess that left me no other choice than to put on my Neo4j Data Science glasses and demonstrate how easy it is to implement it.

Graph import

Today we will be using the Spoonacular Food Dataset that is available on Kaggle. It contains nutritional information alongside the ingredients used in 1600+ dishes. Unfortunately, it contains no recipe for sourdough bread.

Graph schema

There are three types of nodes in the graph schema. A dish consists of one or more ingredients, which we represent as a connection between a dish and its ingredients. Recipes fall into categories or types such as lunch, breakfast, and so on. We use the apoc.schema.assert procedure to define the graph schema. It allows us to describe multiple indexes and unique constraints in a single query.

CALL apoc.schema.assert(

// define indexes

null,

// define unique constraints

{Ingredient:['name'], Dish:['id'], DishType:['name']})

Before we can execute the import query, we have to download the dataset and copy it to the Neo4j import folder. In the import query, we do a tiny bit of preprocessing as we lowercase the names of ingredients and replace dash characters ( - ) with whitespace.

LOAD CSV WITH HEADERS FROM "file:///newfood.csv" as row

CREATE (d:Dish{id:row.id})

SET d += apoc.map.clean(row, ['id','dishTypes','ingredients'],[])

FOREACH (i in split(row.ingredients,',') |

MERGE (in:Ingredient{name:toLower(replace(i,'-',' '))})

MERGE (in)<-[:HAS_INGREDIENT]-(d))

FOREACH (dt in split(row.dishTypes,',') |

MERGE (dts:DishType{name:dt})

MERGE (dts)<-[:DISH_TYPE]-(d))

Basic graph exploration

To start off, we will do a bit of graph exploration. Let’s look at which ingredients are used the most.

MATCH (n:Ingredient)

RETURN n.name as ingredient,

size((n)<--()) as mentions

ORDER BY mentions DESC

LIMIT 10

Results

Olive oil is by far the most popular as it is used in more than half of the recipes. Slowly following are garlic, salt, and butter. I didn’t know that butter was so popular. I also found it quite surprising that anchovies are so widely used. Or maybe the dataset is just biased towards dishes that contain anchovies.

We could build an application on top of this dataset to search for recipes based on the desired ingredients we want to cook. I borrowed this cypher query from the What’s cooking series written by Mark Needham and Lju Lazarevic. Let’s say, for example, you want to eat zucchini and feta cheese today, but don’t have any idea which recipe to go for. Luckily, our application could help us solve this problem with the following cypher query.

WITH ["feta cheese", "zucchini"] as ingredients

MATCH (d:Dish)

WHERE

all(i in ingredients WHERE exists(

(d)-[:HAS_INGREDIENT]->(:Ingredient {name: i})))

RETURN d.title AS dish

ORDER BY size(ingredients)

LIMIT 10

Results

We could go with a salad or a fish. I think I’ll pass and skip over to node2vec for dessert.

Node2vec algorithm

The node2vec algorithm is relatively new. It was proposed only in the year 2016 by Jure Leskovac and Aditya Grover in an article node2vec: Scalable Feature Learning for Networks. To understand how it works, we must first understand the word2vec algorithm. Word2vec algorithm was proposed in the year 2013 by a team of researchers led by Tomas Mikolov at Google. It is a popular technique using neural networks to learn the word embedding. It takes a list of sentences as input and produces a vector or an embedding for each word that appears in the text corpus. Words with similar meanings should be closer in the embedding space. For example, apple and pear should be more similar than apple and car. There are two training algorithms for word2vec. The first method is called the continuous bag of words (CBOW), which uses the context of the word to predict a target term. The context is defined as words that appear near the target word in the text. The second method is called skip-gram. Instead of trying to predict the target word from the context, it tries to predict the context of a given term. If you want to learn more about the word2vec algorithm, there is plenty of good literature about it on the internet.

You might ask how we get from word2vec to node2vec. It is actually very straightforward. Instead of using a list of sentences as input, we use a list of random walks. This is the only difference.

Neo4j Graph Data Science library

Neo4j Graph Data Science library supports the random walk algorithm, which makes it very easy for us to implement the node2vec algorithm. If you need a quick refresher on how the GDS library works, you can check out my previous blog post. We will start by projecting the in-memory graph. We describe all three node labels and project relationships as undirected.

CALL gds.graph.create('all',

['Dish', 'Ingredient'],

{undirected:{type:'*', orientation:'UNDIRECTED'}})

Now we are ready to train our first node2vec model. The process will consist of three parts:

Execute the random walk algorithm starting from each node in the graph Feed the random walks to word2vec algorithm Inspect results by looking at the most similar neighbors

The random walk algorithm has an optional start parameter, which can be used to define the starting node of the walk. We can also specify how long the walk should be with the steps setting and how many times it should be repeated with the walks parameter. Note that every time random walk is executed, we expect a different result.

We will use the Word2vec algorithm implementation in the gensim library. It also has a couple of hyperparameters we can define. Most notable are:

size : Dimensionality of the embedding vectors

: Dimensionality of the embedding vectors window : Maximum distance between the current and predicted word

: Maximum distance between the current and predicted word min_count : The minimum count of words to consider when training the model; words with occurrence less than this count will be ignored.

: The minimum count of words to consider when training the model; words with occurrence less than this count will be ignored. sg: The training algorithm: 1 for skip-gram; otherwise default CBOW

Check out the official documentation for more information about the word2vec hyperparameters

Results

[('anchovy fillet', 0.6236759424209595),

('pork shoulder roast', 0.6039043068885803),

('penne', 0.5999650955200195),

('cherry', 0.5930663347244263),

('cooked quinoa', 0.5898399353027344),

('turkey', 0.5864514112472534),

('asiago cheese', 0.5858502388000488),

('pasillas', 0.5852196216583252),

('fresh marjoram', 0.5819133520126343),

('prunes', 0.5735701322555542)]

If I knew it was that easy, I would have written about the node2vec algorithm before. On the other hand, the results smell kind of fishy. I have no idea what is it with this dataset and anchovies. It seems like the recipes were mostly written by someone who really likes them. You will probably get quite different results though.

In the original node2vec paper, the authors defined two parameters that control the random walks execution. The first one is the return parameter.

Return parameter, p. Parameter p controls the likelihood of immediately revisiting a node in the walk. Setting it to a high value (> max(q, 1)) ensures that we are less likely to sample an already visited node in the following two steps (unless the next node in the walk had no other neighbor). This strategy encourages moderate exploration and avoids 2-hop redundancy in sampling. On the other hand, if p is low (< min(q, 1)), it would lead the walk to backtrack a step (Figure 2) and this would keep the walk “local” close to the starting node u.

And the second parameter is called the in-out parameter.

In-out parameter, q. Parameter q allows the search to differentiate between “inward” and “outward” nodes. Going back to Figure 2, if q > 1, the random walk is biased towards nodes close to node t. Such walks obtain a local view of the underlying graph with respect to the start node in the walk and approximate BFS behavior in the sense that our samples comprise of nodes within a small locality. In contrast, if q < 1, the walk is more inclined to visit nodes which are further away from the node t. Such behavior is reflective of DFS which encourages outward exploration. However, an essential difference here is that we achieve DFS-like exploration within the random walk framework. Hence, the sampled nodes are not at strictly increasing distances from a given source node u, but in turn, we benefit from tractable preprocessing and superior sampling efficiency of random walks. Note that by setting πv,x to be a function of the preceeding node in the walk t, the random walks are 2nd order Markovian.

In summary, the return parameter directs how often random walk backtracks a step or two. The in-out parameter controls if the random walk is more focused on local exploration, similar to BFS, or inclined more towards outward exploration like the DFS. Even though the random walk algorithm is still in the alpha tier, it supports these two node2vec parameters. Let’s try them out in practice.

Results

[('leg of lamb', 0.7168825268745422),

('anise seeds', 0.6833588480949402),

('basic bruschetta', 0.6759517192840576),

('dried chilli flakes', 0.6719993352890015),

('tuna in olive oil', 0.6697120666503906),

('spaghetti pasta', 0.669307291507721),

('prime rib roast', 0.661544919013977),

('baby artichokes', 0.6588324308395386),

('rice vermicelli', 0.6581511497497559),

('whole wheat fusilli', 0.6571477651596069)]

Looking at results makes me hungry. It is really hard to say if the leg of lamb and olive oil should be regarded as similar ingredients. If we inspect the graph, out of eight recipes that include the leg of lamb, seven of those also use olive oil. By this logic, they are quite similar.

In our next example, we will show how to run the node2vec algorithm and store the result embeddings back to Neo4j. Instead of returning the titles of dishes and ingredients, we will return the internal Neo4j node ids. This will help us to link the results back to Neo4j efficiently.

The embeddings are now available in the vocabulary of the word2vec model. We will store them to Neo4j in a single batch using the UNWIND cypher statement. If possible, try to avoid committing a single transaction per row as this is not very performant.

Word2vec model uses the cosine similarity to find the most similar words. The Graph Data Science library also supports the Cosine similarity algorithm, which can be used to infer a similarity algorithm. As with all similarity algorithms, we have to fine-tune the similarityCutoff and topK parameters to get the best results. They directly influence how sparse the inferred similarity graph will be.

MATCH (node)

WITH id(node) as id, node.embedding as weights

WITH {item:id, weights: weights} as dishData

WITH collect(dishData) as data

CALL gds.alpha.similarity.cosine.write({

nodeProjection: '*',

relationshipProjection: '*',

similarityCutoff:0.5,

topK:5,

data: data,

writeRelationshipType:'COSINE_SIMILARITY'})

YIELD nodes, similarityPairs

RETURN nodes, similarityPairs

Results

To finish this analysis, we will inspect the community structure of the inferred network with the label propagation algorithm. As we are only interested in a rough outline of the community structure, we can use the stats mode of the algorithm to provide us some basic community structure statistics.

CALL gds.labelPropagation.stats({

nodeProjection:'*',

relationshipProjection:'COSINE_SIMILARITY',

maxIterations:20})

YIELD communityCount, communityDistribution

RETURN communityCount,

apoc.math.round(communityDistribution.p50,2) as p50,

apoc.math.round(communityDistribution.p75,2) as p75,

apoc.math.round(communityDistribution.p90,2) as p90,

apoc.math.round(communityDistribution.p90,2) as p95,

apoc.math.round(communityDistribution.mean,2) as mean,

apoc.math.round(communityDistribution.max,2) as max

Results

The label propagation algorithm found 118 groups in the similarity network. Most of them have less than 40 members. There are a handful of massive communities with the largest containing 393 members.

Conclusion

The node2vec algorithm is a useful way of learning low-dimensional representations of the nodes in a graph that can be used downstream in a machine learning pipeline. During this blog post, I realized that changing the random walk algorithm parameters as well as the word2vec hyperparameters can produce very different results. Play around with them and see what works best for you.

As always, the code is available on GitHub.