Embedding Graphs with Deep Learning | by Connor Shorten | Towards Data Science

Sparse representations are the natural killer of classifiers. Current graph data structures such as adjacency matrices and lists are plagued with this sparsity. This article will discuss techniques such as Matrix decomposition, DeepWalk, and Node2Vec which convert sparse graph data into low-dimensional continuous vector spaces.
Historically, matrix decomposition has been a popular way of reducing the dimensionality of graph data. This includes techniques such as SVD (Singular Value Decomposition) and MDS (Multi-dimensional Scaling). The problem with these techniques is that they essentially have the running time of Matrix Multiplication (typically n3, although n^(2.37) and n^(2.77) are achievable with clever tricks such as the Coopersmith-Winograd or Strassen algorithms).
An n3 running time isn’t too terrible for many problems. However, this is catastrophic in the case of graph representations with thousands or even millions of users. Additionally, this must be recomputed as new users and connections are added to the graph. Social network graphs such as twitter retweets or medium followers are dynamic and constantly evolving.
Instead of using Matrix Decompositions to represent graphs, we can use Deep Learning!
One of the first papers to elegantly formulate the use of Deep Learning for representation graphs was DeepWalk (linked below).
arxiv.org
DeepWalk is inspired by similarities between graph representations and text tokens. Naively, text tokens are encoded as one-hot vectors. For example, you would take a very long vector the size of the vocabulary, and reserve an index for each word. The word “tiger” would be encoded as something like:
One-hot encoding of text[ 0 0 .. 0 0 1 0 0 .. 0 0 ] = "tiger"
Naturally, these sparse representations of text wrecked havoc for classification models on downstream tasks such as sentiment prediction. The savior of this was the introduction of the Skip-gram model (pictured below):
The Skip-gram model works by sliding context windows across sentences and building a Deep Neural Network that takes as input the word token and maps it to a prediction of other words appearing in the context window. The intermediate representation (denoted as “projection” in the diagram above) is then the low-dimensional representation of the text tokens.
DeepWalk took this idea and extended it to graph data. However, text data naturally contains some kind of structure that is well fitted to the context window. The authors of DeepWalk simulate this structure by constructing neighborhoods through random walks, hence the name “DeepWalk”.
The diagram above is used to depict a Random Walk. Starting from node “Connor”, we either travel to Erika, Daniel, Ben, or James with the probabilities a, b, c, and d. In DeepWalk, a = b = c = d, but in Node2Vec these probabilities are parameterized throughout the walk.
Random walking along graphs results in neighborhoods such as:
Examples of Vertex Neighborhoods(Connor, James, Jon, Bob, Joe)(Connor, James, Jon, James, Connor)(Connor, Erika, Connor, Daniel, Connor)
Context windows are then slid across these neighborhoods and a skip-gram model is trained.
Two important ideas for training skip-gram models are Hierarchical Softmax and Negative Sampling. The image highlights the idea of Hierarchical Softmax.
Imagine a facebook network where you have millions of users to predict in the vertex neighborhoods. This would result in a softmax layer with millions of hidden units. This disaster can be avoided by constructing the vertex space in a binary tree and predicting a path along the tree to traverse. For example, if there are 8 users in the network, you only need 3 values to traverse from the root to any individual node.
The next idea (which is a crucial idea in Word2Vec) is negative sampling. Negative sampling basically throws out the idea of directly predicting the words in the context window, and rather treating the context prediction task as a binary classification problem. Once you have a context pair, you randomly sample negative pairs and then have the model label wether the word appears in the context or not.
Another interesting paper building on DeepWalk is Node2Vec (linked below):
arxiv.org
Node2Vec takes apart the idea of using random walks to construct vertex neighborhoods. This begins by looking at two extreme ways of traversing a graph, Bread-First and Depth-First Search:
They find that if you traverse nodes in a Depth-First Search, the resulting embedding will capture homophily. Homophily refers to the notion that nodes which have a small shortest-path distance should be embedded near each other in the new representation. They also find that if you traverse nodes in a Breadth-First Search, the resulting embedding will capture structural equivalence. Structural equivalence is easily explained in the image above, even though u and s6 do not share any neighbors, they both serve as the hub of their respective communities.
The authors of Node2Vec thus propose to parameterize the random walk so that it can construct more semantic vertex neighborhoods:
In Conclusion, this article has introduced DeepWalk and Node2Vec and showed how they use Deep Learning to embed graphs into low-dimensional representations. The skip-gram model approach has the advantage of being adaptable to changes in the network, compared to Matrix Decomposition methods which must be recomputed. Thanks for reading, please check out the video below if you would like a recap of this article!