An introduction to Graph Neural Networks
By: Joao Schapke
Tags: Graph, Neural Networks, Data Science, Deep Learning, Machine Learning

Graph structured data is common across various domains, examples such as molecules, { social, citation, road } networks, are just a few of the vast array of data which can be represented with a graphs. With the advancements of machine learning we witness the potential for applying intelligent algorithms on the data which is available. Graph Neural Network is the branch of Machine Learning which concerns on building neural networks for graph data in the most effective manner.

Notwithstanding the progress made with ML in the computer vision domain with convolutional networks, Graph Neural Networks (GNNs) face a more challenging problem, they deal with the awkward nature of graphs. Differently from images and text, graphs do not have a well defined structure. A graph’s node might have no connections or many, of which could be directed or undirected. Graphs in a dataset may have a variable number of nodes and edges and cannot be properly ‘resized’, and they could be either acyclic, cyclic, joint, disjoint. All in all it makes the process of handling the data more challenging.

A simple approach to deal with graphs is to concatenate nodes, and build a more friendly structured dataset which could be fed to a Multi Layer Perceptron (MLP), and although is not a quite elegant method it could provide good results. Although it’s not sufficient to handle problems in which you deal with multiple graphs of variable sizes and topologies or with large graphs which would severely increase the dimensionality of the input. Variable sized graphs could be handled by using Recurrent Neural Networks (RNN’s) by converting graphs into a sequences of nodes. But, this strategy is doomed to failure as well, one of the most important information within a graph is its structure, which would be left no trace of in the node sequence. Graph data require more robust approaches which are tailored to handle their structure.

Nevertheless the difficulty of dealing with graphs, the progress of convolutional networks inspired much research and new approaches on the field of GNN. Convolutional layers enjoy some useful properties such as: local connectivity, they assume that pixels which are close together are highly correlated. Shift invariance, convolutions learn the features of objects notwithstanding their spatial position. They also are able to convolute over images of any size. All of which are really attractive features for neural networks on graphs, which should value local connectivity, because a node’s neighbors are correlated with the node itself, they should be shift invariant, in the sense that a neighborhood should conceive the same meaning to the convolution despite of the particular place on the graph where is located, and they also should be invariant to the size of graphs.

This gave rise to different approaches of applying convolutions in graphs, which can mainly be classified as either spatial based graph convolutions or spectral based graph convolutions.

Photo by Zak Sakata on Unsplash

Spectral Graph Convolutions

Spectral GNN’s is a more principled approach towards convolutions, which have a foundation on signal preprocessing theory.

Without going into much details on the theory behind spectral convolutions, it has been shown that a convolution in the spatial domain equals to multiplication in the frequency domain (convolution theorem). The same theorem can be applied to graphs, but instead of using the Discrete Fourier Transform matrix as basis we use the eigenvectors of the graph Laplacian.

Where the laplacian of a graph is defined as: L = I — D-1/2 A D-1/2

where I is the identity matrix, D is the diagonal order matrix and A is the adjacency matrix of the graph.

The main limitation of spectral convolutions is their dependency on the eigenvectors of the Laplacian Matrix, which are needed in every calculation, which have the time complexity of O(n2) for backward propagation and O(n3) for calculating the eigen-decomposition.

Further work alleviated the time complexity by approximating the equations using Chebyshev polynomials and Cayley Polynomials.

GCN (Graph Convolutional Networks) made further simplifications into the ChebNet networks which used Chebyshev polynomials.

Photo by Ashley Whitlatch on Unsplash

Spatial Graph Convolutions

Spatial convolutions are a more relaxed version of graph convolutions, which got popularized by the simplifications made by GCN on previous spectral GNN’s.

Spatial convolutions are similar to regular convolution in the way they spatially convolute over nodes. Convolutions generate a new pixel from the pixel itself ( the pixel placed in the same position as the new one ) and it’s surrounding pixels, spatial graph convolutions makes convolutions by aggregating a node and it’s neighbors into a new node.

Image illustrating the difference between a regular convolution in a grid structured image and a spatial graph convolution in a graph.

A spatial graph convolution for a single node can be defined as:

hi = (sum(N) + ni) * W

Where ni represents node i which belongs to Rc , N represents the neighbors of ni, w is the weight matrix belonging to Rc x f. hi is the new representation of node i.

We note this convolution will aggregate the local information of a node into a new node. From the GCN paper we have that:

Which is similar to the previous equations. Theta is a non-linear activation function applied to the convolution.

Ã is the graph adjacency matrix plus the identity matrix, this will make take the role to aggregate every node with its neighbors.

D is the order diagonal matrix. If we note in the first equation that we are summing over all the neighbors of node i, and nodes in the graph can have an arbitrary number of neighbors we should note that we must normalize the results of the summation. Multiplying by the inverse diagonal matrix does the job.

This simple operation is the basis for spatial convolutions. And stacking a couple of these with dropout layers, L2 regularization and using ReLU as the activation function and Adam as the optimizer, GCN was able to achieve SOTA for 3 node classification benchmarks on the field.

Note that the de facto approach used today are spatial based convolutions. Which provide a more efficient and simple approach to graph convolutions.