Limitations of Graph Neural Networks
By: Sergei Ivanov
Tags: Gnn, Graphs, Gml, Machine Learning, Theory

On the way to more powerful GNN. Source.

There are two paradigms for graph representations: graph kernels and graph neural networks. Graph kernels typically create an embedding of a graph, based on decomposition, in an unsupervised manner. For example, we can count the number of triangles or more generally triplets of each type a graph has and then use these counts to get embeddings. This is known to be an instance of a graphlet kernel.

All graphlets of size 4. Counting the number of each graphlet among all quadruples in a graph will result in a graphlet kernel. Source: Efficient graphlet kernels for large graph comparison

The main research motivation of this paradigm was to create an embedding that preserves isomorphism between graphs, i.e. two graphs are isomorphic if and only if their corresponding embeddings are the same. Needless to say, if we have such embedding we solve graph isomorphism problem, which is, for now, known to be harder than the class of problems P. Yet, there are embeddings such as Anonymous Walk Embeddings that preserve isomorphism, of course at the cost of running time computation. Nonetheless, the main message here is that graph kernels were designed to solve graph isomorphism problem. The more graphs your embeddings can distinguish, the better your embeddings. That was the principle.

With graph neural networks the principle has changed. Instead of solving one problem, graph isomorphism, we can try to solve any given problem, such as finding the shortest path or detecting the cycles. This is quite promising as it allows us to guide our network design by the problem that it can solve. It sounds like magic: instead of using some well-established combinatorial algorithms, you can just train your network and it will find the solution for you. But it is also suspicious: neural networks are searching the solution by SGD and involve many other technicalities and what if you are stuck in a bad local optimum — how can it solve any problem then? Well, in fact, there are some limitations to graph neural networks that I will describe next.

Conditions on GNN to be powerful.

I will start from the work, How Powerful are Graph Neural Networks, that triggered a lot of research on the theoretical explanation of GNN. In particular, GNNs are compared to a well-studied graph isomorphism algorithm, Weisfeiler-Lehman (WL) algorithm.

About WL algorithm.

This algorithm is quite simple to describe. You start with a graph, where each node has some color (if none, put a degree). In each iteration, each node gets a set of colors of its neighbors and updates its color in a specific way. In particular, there is an injective function that creates a new color c’ of the node from its previous color c and the ordered list of the neighborhood’s colors X. The algorithm stops after n iterations with updated coloring for a graph.

Note: it’s important that WL algorithm uses injective function as it guarantees that different inputs will get different outputs. A particular injective function that WL uses is that it creates a new color that was not encountered before for each of the input arguments. Because it operates in the categorical (countable) domain (colors) it can always create such mapping.

Injective, bijective, and surjective mappings (left to right). Source.

The main use of this algorithm is to test isomorphism between two graphs. If the final colorings are different, then two graphs are not isomorphic. If two graphs have the same final coloring, then WL algorithm outputs that they are possibly isomorphic, which means there is still a slight chance that they are not isomorphic.

This algorithm was designed in 70’s in the secret Soviet lab, when the computers still used punched cards. But since then researchers across the world have studied its properties and in particular, we know different families of graphs when WL algorithm fails. For example, for any two d -regular graphs with n vertices, the final colorings will be the same. Nonetheless, it’s a very powerful test of isomorphism and there are theorems that state as n tends to infinity, the likelihood that WL fails goes to 0. So it’s a reasonably strong algorithm.

Back to GNN.

If you studied GNN you noticed many similarities between how GNN updates nodes’ features and how WL algorithm updates nodes’ colors. In particular, GNN updates features using a message-passing mechanism.

The difference between different GNN is on which aggregation and readout functions they use. But it’s easy to understand that if aggregation functions are injective, then if WL maps graphs to different colorings then GNN will also map these graphs to different embeddings. Theorem 3 is a formal way to say this.

In other words, there are parameterized functions phi and f of GNN that if injective, then they guarantee strong power of the GNN. This should not be a surprise, as WL algorithm also requires its function to be injective and in other aspects the two procedures are equivalent.

Note here a particular way we update node’s embeddings. We get previous embedding h_v^(k-1) and a multiset of previous embeddings of the neighbors as two different arguments, and not as one argument when you merge these two together. This will be important down the line.

So, you can use GNN to decide whether graphs are isomorphic and it will be equivalent to using WL algorithm.

This is the magical part of it. GNN suddenly became equivalent to well-known algorithms. But where is the limitation of it?

About the limitations of GNN.

The main limitation from the above is that you need to have injective functions phi and f . What are these functions? These are the functions that map a multiset of embeddings into new embeddings. You can, for example, use the mean function. This function will take an average of embeddings and assign it as a new embedding. However, it’s easy to see that for some different graphs these will give the same embeddings, so mean function is not injective.

Mean aggregation of embeddings (embeddings correspond to different colors here) for nodes v and v’ will give the same embedding, even though the graphs are different. Source: How Powerful are Graph Neural Networks?

But if you take the sum and transform embeddings in a specific manner it’s possible to get injective function. Here is the Lemma 5:

What’s really important here is that you can first map each embedding under the sum to a new embedding using some function f(x) and then take a sum of this and get a function that is injective. In the proof, they actually explicitly state this function f, which requires two additional conditions that X is countable and any multiset is bounded. I think both these assumptions are not strong, as anyway, we apply our GNN to finite graphs, where the cardinality of features and neighborhoods is finite. But at least we know now that if we use transformation f and then addition we can get an injective mapping.

However, there should be a specific aggregation scheme from the Theorem 3 above (condition a) which in addition to the aggregation of the neighbors, uses the previous embedding for the current node h_v^(k-1) . To include it we need another statement:

Note that here a function h takes the sum of transformed neighbors’ features as before, but in addition, adds (1+eps)f(c) and this eps is any irrational number. This way the function h is injective.

Okay, so what do we know? We know that our aggregation functions phi and f should be injective and we have the function h which is injective. If we were aiming at building powerful embeddings, then we are done. But we attempt not just to build embeddings, but to solve the downstream task such as node classification in a supervised manner. And the function h does not have learnable parameters that could fit the data (except maybe eps ).

What GIN architecture proposes instead is to replace functions phi and f with MLP and thanks to the universal approximation theorem, we know that MLP can approximate any function, including the injective function. So, in particular, the update of embeddings of GIN has the following form:

Note that the thing inside MLP is not guaranteed to be injective anymore and moreover MLP itself is not guaranteed to be injective. In fact, for the first layer, if input features are one-hot encoded, then the sum inside MLP will be injective and in principle, MLP can learn an injective function. But on the second layer and beyond, node embeddings become irrational and it’s easy to come up with an example where the sum of embeddings is not injective anymore (e.g. having one neighbor with embedding equals to 2 or having two neighbors with embeddings equal to 1).

So, GIN is as powerful as WL algorithm if MLP and the sum of embeddings will be injective functions.

But in fact, there is nothing in the training that would guarantee this injectivity and there could be graphs that GIN does not distinguish, that WL can. So this is quite a strong assumption on GIN and if it’s violated, then the power of GIN is limited.

This limitation has been later discussed in the paper Discriminative structural graph classification, where it’s shown that the size of the output embeddings should be exponential to the size of the input features in order to make MLP injective, although the analysis is done for the unbounded neighborhoods (infinite graphs). Finding an architecture that would have an injective aggregation and be sufficiently expressive for the downstream tasks is an open question, even though there are several architectures that generalize GIN to higher-dimensional WL algorithm as well to other problems; however, there are no guarantees yet that the learned GNN architecture will solve a particular task for all the input graphs. A Survey on The Expressive Power of Graph Neural Networks makes a very good job of explaining recent advancements in the theoretical explanation of the power of GNN.

Conclusion.

The studies of properties of GNN are now an active area of research (and you can check on the latest trends) and there are many open questions to tackle. The main message of this post is to show that GNN currently are not guaranteed to converge to the state that will be as powerful as WL algorithm, even though in general there is a set of parameters for GNN when it becomes as powerful. GNN can solve different problems on graphs, but so far the studies have been focused on what they can or cannot solve, and not on how it can have some guarantees of the obtained solutions, which I think will be a focus of next research papers.

P.S. I will continue to write about graph machine learning, so if you are interested, follow me on medium or subscribe to my telegram channel (which I update every day) or my twitter (which I update once a week).