Extracting ML-Features from Graph Data with DeepGL on Neo4j | by Mark Needham | Towards Data Science

In 2013 Tomas Mikolov and his Google colleagues released a paper describing word2vec, and popularised the idea of generating embeddings to represent pieces of data.
An embedding is an array or vector of numbers used to represent something, in word2vec’s case: a word.
Adrian Colyer has a nice diagram showing a very simple hot encoding representation of a vocabulary of words (with one element “hot” below):
In this example we have one column for each word and the vector for one word will have a 1 in the appropriate column and 0s everywhere else.
With this simple representation we can’t really do any meaningful comparisons between our embeddings, but word2vec goes much further than this and comes up with a representation for each word that is a distribution of weights across those elements.
For example, we might end up with a representation for words that looks like this:
Since word2vec was released other people have used similar approaches to come up with embeddings for things other than words. Pinterest created pin2vec which is used to recommend pins to users and Airbnb use embeddings to find similar property listings.
Embeddings can be considered as an implementation of Representational learning where we automatically come up with features to feed into our machine learning models rather than creating them by hand.
For example, the output of these algorithms could be used as the input to a Tensorflow model.
We could also build a kNN similarity graph from the embeddings using the Cosine Similarity procedure introduced in the 3.4.7.0 release of the Graph Algorithms library.
The similarity graph could then be used to make recommendations as part of a k-Nearest Neighbors query.
There are several algorithms that can be used to generate graph embeddings. These are the ones that we’re aware of:
node2Vec
struc2vec: Learning Node Representations from Structural Identity
DeepWalk — Online Learning of Social Representations
The idea is the same as for word embedding algorithms. We want to determine a vector representation for each entity (usually nodes) in our graph and then feed those representations into a machine learning algorithm.
A recent addition to the list is Deep Feature Learning for Graphs (DeepGL), created by Ryan A. Rossi, Rong Zhou, and Nesreen K. Ahmed.
This one looked very interesting to us as it’s specifically designed to keep memory usage low and also returns the names of the features that it comes up with, which is helpful in these days of machine learning model interpretability and feature extraction. Another nice feature is, that it allows you to also pass numeric node attributes to the algorithm whereas the others only rely on the graph structure.
We start by building a set of base features for each node: in-degree, out-degree, and both-degree. We can also choose to include any other numeric properties (or projections) of the node.
The paper describes more base features that can be generated through local graphlet decomposition, but we haven’t implemented that yet.
For a very simple graph containing 4 nodes, these base features may look like this:
We then apply logarithmic binning by column on these values. This results in the following embedding:
Next we apply a set of operators over this set of features for the (in, out, whole) neighbourhoods of each node. The operators we apply are
Hadamard,
mean,
sum,
maximum,
L1 norm,
and RBF.
This means that for, say, the sum operator we will create 9 new columns for each node:
sum of the in-degree of our in neighbourhood
sum of the out-degree of our in neighbourhood
sum of the both-degree of our in neighbourhood
sum of the in-degree of our out neighbourhood
sum of the out-degree of our out neighbourhood
sum of the both-degree of our out neighbourhood
sum of the in-degree of our whole neighbourhood
sum of the out-degree of our whole neighbourhood
sum of the both-degree of our whole neighbourhood
Let’s have a look at a what would happen if we just apply the sum operator to the in neighbourhood. We end up with 3 extra columns, summing up the scores of each nodes’ neighbors for in-degree, out-degree, and both-degree.
Our resulting set of features would look like this:
We calculate the same values for our other neighbourhoods and for all the other operators as well which give us a matrix of 57 columns and 4 rows.
We then apply a diffusion process where each node distributes its score to its neighbours. The way we’ve implemented this is that each node takes the mean value of all of its neighbors for each feature.
The diffusion process excludes the base features, so in our example we’ll only diffuse sum-in-in, sum-in-out, and sum-in-both.
Our resulting set of features would look like this:
Before we continued to the next stage we apply logarithmic binning again, which results in this set of features:
After that step, we apply feature pruning, which involves running the weakly connected components algorithm over a feature similarity graph and then keeping only a single column per connected component.
We work out the similarity of features by doing an item wise comparison of values in each column. If all values across two columns are identical then we’d have a similarity score of 1 for that column. This is the case for diffused-sum-in-out and diffused-sum-in-both in our example, as well as a few other pairs of columns.
Assuming a pruning lambda of 0.7 to remove lower ranked relationships, this is the feature graph that we end up with:
The connected component algorithm would give us 3 components:
sum-in-in
sum-in-both, in-degree, sum-in-out
both-degree, diffused-sum-in-both, diffused-sum-in-out, out-degree, diffused-sum-in-in
We bias towards keeping features that we had from an earlier iteration, which in this case means the base features. We keep one feature per component, which in this example means that we keep:
sum-in-in
in-degree
out-degree or both-degree (we randomly choose between these)
We repeat this process for as many iterations as we want. On the 2nd and subsequent iterations, we’ll apply the operators to the features that we have left from the previous iteration rather than applying them to the base features again.
Once the algorithm has completed we’ll have vectors of equal length for each node in our graph.
In our example these vectors would be:
So the vector for node 0 is [0,0,0], for node 1 it’s [1,1,0], for node 2 it’s [2,0,2] and for node 3 it’s [0,2,1]
Over the last couple of months Pete Meltzer, who works at BrainTree as a Research Associate and is s`tudying for a PhD at UCL, and I have been implementing this algorithm as a Neo4j procedure and now have a version ready for you to try out!
The DeepGL algorithm is available as part of the ml-models library. You’ll want to grab the 1.0.2 release.
Once you’ve got that installed (instructions in the repository) you’ll find two versions of the algorithm that you can use:
embedding.deepgl — this stores the embeddings as a property on each node
embedding.deepgl.stream — this returns a stream of node, embedding
While you’re playing around with the algorithm the streaming probably makes sense, but once you want to use it on a proper dataset I’d suggest using the first version.
This is the signature of the algorithm:
CALL embedding.deepgl("NodeLabel" ,"RelationshipType", {  nodeFeatures: [string]  pruningLambda: double,  diffusions: integer,   iterations: integer,  writeProperty: string})
nodeFeatures — contains an array of additional property names that you want to use as part of the algorithms base features (defaults to [])
pruningLambda — how strict the algorithm should be when removing similar features. Lower means prune morse aggressively (defaults to 0.7)
diffusions — how many diffusions the algorithm should execute (defaults to 10)s
iterations — how many iterations the algorithm should run for (defaults to 10)
writeProperty — the name of the property in which the embedding array will be stored (defaults to “deepgl”)
You can also utilize Graph projections (views) by passing the graph: “cypher” config parameter and providing Cypher queries for node-id- and relationship-lists, instead of NodeLabel and RelationshipType.
The defaults we’ve set seemed to work well on the datasets we’ve tested against, but they are different than the ones suggested in the paper so YMMV.
We’ve tried the algorithm out on the Enzyme and EU network datasets referenced in the paper and it seems to work reasonably well when compared with embeddings created using other algorithms.
We hope this algorithm is useful and that you’re able to use it against your own datasets.
If you have any questions or suggestions on what we should do next let us know in the comments or send us an email to devrel@neo4j.com. You can also ask any questions in our new Neo4j community forum.
If you want to get in contact with Pete Meltzer you can reach him at p.meltzer@braintree.com.