Title: The Logical Expressiveness of Graph Neural Networks
Authors: Pablo Barceló
Egor V. Kostylev
Mikael Monet
Jorge Pérez
Juan Reutter
Juan Pablo Silva
Venue: ICLR 2020
Keywords: Graph Neural Networks
First Order Logic
Expressiveness

Abstract: The ability of graph neural networks (GNNs) for distinguishing nodes in graphs has been recently characterized in terms of the Weisfeiler-Lehman (WL) test for checking graph isomorphism. This characterization, however, does not settle the issue of which Boolean node classifiers (i.e., functions classifying nodes in graphs as true or  false) can be expressed by GNNs.  We tackle this problem by focusing on Boolean classifiers expressible as formulas in the logic FOC2, a well-studied fragment of first order logic. FOC2 is tightly related to the WL test, and hence to GNNs. We start by studying a popular class of GNNs, which we call AC-GNNs, in which the features of each node in the graph are updated, in successive layers, only in terms of the features of its neighbors.  We show that this class of GNNs is too weak to capture all FOC2 classifiers, and provide a syntactic characterization of  the largest subclass of FOC2 classifiers that can be captured by AC-GNNs. This subclass coincides with a logic heavily used by the knowledge representation community. We then look at what needs to be added to AC-GNNs for capturing all FOC2 classifiers. We show that it suffices to add readout functions, which allow to update the features of a node not only in terms of its neighbors, but also in terms of a global attribute vector. We call GNNs of this kind ACR-GNNs. We experimentally validate our findings showing that, on synthetic data conforming to FOC2 formulas, AC-GNNs struggle to fit the training data while ACR-GNNs can generalize even to graphs of sizes not seen during training.

TL;DR: We characterize the expressive power of GNNs in terms of classical logical languages, separating different GNNs and showing connections with standard notions in Knowledge Representation.

Review discussion:
The paper utilizes recent insights into the relationship between the Weisfeiler-Lehman (WL) test for checking graph isomorphism and Graph Neural Networks (GNNs) in order to characterize the class of node classifiers that can be captured by a specific GNN architecture, called aggregate-combine GNN (AC-GNN).
The primary contribution of this work is the identification of the logical classifiers that can be represented within an AC-GNN, a fragment of first order logic called graded modal logic, as well as an extention of AC-GNNs with the ability to capture a strictly more expressive fragment of first order logic, called ACR-GNN. Both of these results are supported by formal proofs and derivations, adding not only theoretical value to the presented work, but also intuitive insights behind the reasons of the AC-GNN limitations.
In addition, the presented experiments demonstrate the practical implications of the results, with the exception of some datasets where no significance difference in performance between AC-GNNs and ACR-GNNs was found.
I believe that the contributions of this paper are significant.
On the one hand, studying theoretical properties of GNNs facilitates their transparency and highlights the range of applications they can be used in.
On the other hand, although the GNN variant introduced by the author is a special case of an existing class of GNNs, the motivation behind its introduction is different and follows naturally from the discussion within the paper.
The fact that no actual difference in performance between AC-GNNs and ACR-GNNs was noticed in the only non-synthetic dataset used in the experiment should prompt the author to run experiments with more real life datasets, in order to empirically verify the results, but this is a minor point.
Thank you for your review. You correctly point out that more empirical results on real life data are needed and this is part of our current and future work. We just add that there is some work showing that current benchmarks might not be very helpful when differentiating specific changes for GNNs [1]. It is even common practice to only keep and train over the biggest connected component of the graph, and discard the rest, so our ACR-GNN might not benefit from those situations. Recent work has also started studying that current real life datasets might not be adequate for some tasks [2].
Moreover, most common benchmarks do not encode global constraints in the graph features nor labels, and thus it might not be easy to find a good benchmark to show for our global approach. We comment on this only to emphasize that finding a good benchmark for our approach has not been an easy task and that is why we focused more on synthetic data in this submission.
[1] Chen, Ting, Song Bian, Yizhou Sun. "Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classification." https://arxiv.org/abs/1905.04579
[2] Ivanov, Sergei, Sergei Sviridov, and Evgeny Burnaev. "Understanding Isomorphism Bias in Graph Data Sets." https://arxiv.org/abs/1910.12091
The paper elaborates on the expressivity of graph neural networks (GNNs).
More precisely, the authors show that expressivity of AC-GNNs (aggregate and combine) can only express logical classifiers that can be expressed in graded modal logic. By adding readouts, ACR-GNNs (aggregate, combine and readout) can capture FOC2 which is logical classifiers expressed with 2 variables and counting quantifiers. The second theorem leaves open the question of whether ACR-GNNs can capture logical classifiers beyond FOC2.
The paper is written nicely, its easy on the eyes, and delegates the proofs to the appendix. I was a bit surprised by the lack of a discussion connecting the choice of the aggregate and combine operations to the representation power of GNNs. One has to delve deep into the proofs to find out if the choice of these operations affects expressivity.
Thank you very much for your comments. In the final version of the paper, we plan to add in the body of the paper more discussion on the aggregate and combine operation used in the proofs and how they affect the expressiveness of GNNs.
This paper establishes novel theoretical connections between Boolean node classifiers on aggregate-combine Graph Neural Networks (AC-GNNs) and first-order predicate logic (FOC2). It shows that current boolean node classifiers on AC-GNNs can only represent a subset of FOC2 but that a simple extension taking into global information can generalise AC-GNNs to the full FOC2.
The style of the manuscript is mixed: the abstract and introduction are quite dense for non-experts; a motivating real-world example could help here. The other sections, on the other hand, are quite good to follow and most concepts have a good high-level introduction as well as a little example to follow the arguments.
The theoretical connections connection between GNNs and first-order logic strike me as interesting. I did not, however, understand the results reported in table 2: ACR reaches optimal performance only on alpha_1 but not on alpha_2 and alpha_3. Does this happen because the latter two expressions are not part of FOC2?
I increased my rating due to the author rebuttal.
Thanks a lot for your positive comments. We will do our best to improve the readability of the introduction, and to try to fit an example.
In any case we will delay this change for an eventual final version of the paper as it might require some more general changes to the flow of the intro.
But thank you very much for pointing this out.
Regarding your question, all \alpha_i do belong to FOC2.
In fact, it is not difficult to show that there exists an ACR-i GNN that captures alpha_i for each i, but none of ACR-j GNNs with j < i can do this.
Nonetheless, the existence of such a GNN does not imply the ability to learn this GNN from examples.
So, besides other things, our experiments demonstrate that, for i = 2 and i = 3, it is possible to learn an ACR-i GNN very close (~80-90%) to alpha_i, in contrast to AC-GNNs that achieve a lower performance (60%) even if 10 layers are allowed; The question whether and how it is possible to learn an ACR-i GNN exactly equivalent to alpha_i is still open. We will add a discussion about this into the final version of the paper.

Meta-review:
role: PC
data: comment: The paper focuses on characterizing the expressiveness of graph neural networks. The reviewers were satisfied that the authors answered their questions suffciiently and uniformly agree that this is a strong paper that should be accepted.

Decision: Accept (Spotlight)