The Logical Expressiveness of Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) ( Merkwirth & Lengauer , 2005 ; Scarselli et al. , 2009 ) are a class of neural network architectures that has recently become popular for a wide range of applications dealing with structured data , e.g. , molecule classification , knowledge graph completion , and Web page ranking ( Battaglia et al. , 2018 ; Gilmer et al. , 2017 ; Kipf & Welling , 2017 ; Schlichtkrull et al. , 2018 ) . The main idea behind GNNs is that the connections between neurons are not arbitrary but reflect the structure of the input data . This approach is motivated by convolutional and recurrent neural networks and generalize both of them ( Battaglia et al. , 2018 ) . Despite the fact that GNNs have recently been proven very efficient in many applications , their theoretical properties are not yet well-understood . In this paper we make a step towards understanding their expressive power by establishing connections between GNNs and well-known logical formalisms . We believe these connections to be conceptually important , as they permit us to understand the inherently procedural behavior of some fragments of GNNs in terms of the more declarative flavor of logical languages . Two recent papers ( Morris et al. , 2019 ; Xu et al. , 2019 ) have started exploring the theoretical properties of GNNs by establishing a close connection between GNNs and the Weisfeiler-Lehman ( WL ) test for checking graph isomorphism . The WL test works by constructing a labeling of the nodes of the graph , in an incremental fashion , and then decides whether two graphs are isomorphic by comparing the labeling of each graph . To state the connection between GNNs and this test , consider the simple GNN architecture that updates the feature vector of each graph node by combining it with the aggregation of the feature vectors of its neighbors . We call such GNNs aggregate-combine GNNs , or AC-GNNs . The authors of these papers independently observe that the node labeling produced by the WL test always refines the labeling produced by any GNN . More precisely , if two nodes are labeled the same by the algorithm underlying the WL test , then the feature vectors of these nodes produced by any AC-GNN will always be the same . Moreover , there are AC-GNNs that can reproduce the WL labeling , and hence AC-GNNs can be as powerful as the WL test for distinguishing nodes . This does not imply , however , that AC-GNNs can capture every node classifier—that is , a function assigning true or false to every node—that is refined by the WL test . In fact , it is not difficult to see that there are many such classifiers that can not be captured by AC-GNNs ; one simple example is a classifier assigning true to every node if and only if the graph has an isolated node . Our work aims to answer the question of what are the node classifiers that can be captured by GNN architectures such as AC-GNNs . To start answering this question , we propose to focus on logical classifiers—that is , on unary formulas expressible in first order predicate logic ( FO ) : such a formula classifies each node v according to whether the formula holds for v or not . This focus gives us an opportunity to link GNNs with declarative and well understood formalisms , and to establish conclusions about GNNs drawing upon the vast amount of work on logic . For example , if one proves that two GNN architectures are captured with two logics , then one can immediately transfer all the knowledge about the relationships between those logics , such as equivalence or incomparability of expressiveness , to the GNN setting . For AC-GNNs , a meaningful starting point to measure their expressive power is the logic FOC2 , the two variable fragment of first order predicate logic extended with counting quantifiers of the form ∃≥Nϕ , which state that there are at least N nodes satisfying formula ϕ ( Cai et al. , 1992 ) . Indeed , this choice of FOC2 is justified by a classical result due to Cai et al . ( 1992 ) establishing a tight connection between FOC2 and WL : two nodes in a graph are classified the same by the WL test if and only if they satisfy exactly the same unary FOC2 formulas . Moreover , the counting capabilities of FOC2 can be mimicked in FO ( albeit with more than just two variables ) , hence FOC2 classifiers are in fact logical classifiers according to our definition . Given the connection between AC-GNNs and WL on the one hand , and that between WL and FOC2 on the other hand , one may be tempted to think that the expressivity of AC-GNNs coincides with that of FOC2 . However , the reality is not as simple , and there are many FOC2 node classifiers ( e.g. , the trivial one above ) that can not be expressed by AC-GNNs . This leaves us with the following natural questions . First , what is the largest fragment of FOC2 classifiers that can be captured by AC-GNNs ? Second , is there an extension of AC-GNNs that allows to express all FOC2 classifiers ? In this paper we provide answers to these two questions . The following are our main contributions . • We characterize exactly the fragment of FOC2 formulas that can be expressed as ACGNNs . This fragment corresponds to graded modal logic ( de Rijke , 2000 ) , or , equivalently , to the description logicALCQ , which has received considerable attention in the knowledge representation community ( Baader et al. , 2003 ; Baader & Lutz , 2007 ) . • Next we extend the AC-GNN architecture in a very simple way by allowing global readouts , where in each layer we also compute a feature vector for the whole graph and combine it with local aggregations ; we call these aggregate-combine-readout GNNs ( ACR-GNNs ) . These networks are a special case of the ones proposed by Battaglia et al . ( 2018 ) for relational reasoning over graph representations . In this setting , we prove that each FOC2 formula can be captured by an ACR-GNN . We experimentally validate our findings showing that the theoretical expressiveness of ACR-GNNs , as well as the differences between AC-GNNs and ACR-GNNs , can be observed when we learn from examples . In particular , we show that on synthetic graph data conforming to FOC2 formulas , ACGNNs struggle to fit the training data while ACR-GNNs can generalize even to graphs of sizes not seen during training . 2 GRAPH NEURAL NETWORKS . In this section we describe the architecture of AC-GNNs and introduce other related notions . We concentrate on the problem of Boolean node classification : given a ( simple , undirected ) graph G = ( V , E ) in which each vertex v ∈ V has an associated feature vector xv , we wish to classify each graph node as true or false ; in this paper , we assume that these feature vectors are one-hot encodings of node colors in the graph , from a finite set of colors . The neighborhood NG ( v ) of a node v ∈ V is the set { u | { v , u } ∈ E } . The basic architecture for GNNs , and the one studied in recent studies on GNN expressibility ( Morris et al. , 2019 ; Xu et al. , 2019 ) , consists of a sequence of layers that combine the feature vectors of every node with the multiset of feature vectors of its neighbors . Formally , let { AGG ( i ) } Li=1 and { COM ( i ) } Li=1 be two sets of aggregation and combination functions . An aggregate-combine GNN ( AC-GNN ) computes vectors x ( i ) v for every node v of the graph G , via the recursive formula x ( i ) v = COM ( i ) ( x ( i−1 ) v , AGG ( i ) ( { { x ( i−1 ) u | u ∈ NG ( v ) } } ) ) , for i = 1 , . . . , L ( 1 ) where each x ( 0 ) v is the initial feature vector xv of v. Finally , each node v ofG is classified according to a Boolean classification function CLS applied to x ( L ) v . Thus , an AC-GNN withL layers is defined as a tupleA = ( { AGG ( i ) } Li=1 , { COM ( i ) } Li=1 , CLS ) , and we denote byA ( G , v ) the class ( i.e. , true or false ) assigned by A to each node v in G.1 There are many possible aggregation , combination , and classification functions , which produce different classes of GNNs ( Hamilton et al. , 2017 ; Kipf & Welling , 2017 ; Morris et al. , 2019 ; Xu et al. , 2019 ) . A simple , yet common choice is to consider the sum of the feature vectors as the aggregation function , and a combination function as COM ( i ) ( x1 , x2 ) = f ( x1C ( i ) + x2A ( i ) + b ( i ) ) , ( 2 ) where C ( i ) and A ( i ) are matrices of parameters , b ( i ) is a bias vector , and f is a non-linearity function , such as relu or sigmoid . We call simple an AC-GNN using these functions . Furthermore , we say that an AC-GNN is homogeneous if all AGG ( i ) are the same and all COM ( i ) are the same ( share the same parameters across layers ) . In most of our positive results we construct simple and homogeneous GNNs , while our negative results hold in general ( i.e. , for GNNs with arbitrary aggregation , combining , and classification functions ) . The Weisfeiler-Lehman ( WL ) test is a powerful heuristic used to solve the graph isomorphism problem ( Weisfeiler & Leman , 1968 ) , or , for our purposes , to determine whether the neighborhoods of two nodes in a graph are structurally close or not . Due to space limitations , we refer to ( Cai et al. , 1992 ) for a formal definition of the underlying algorithm , giving only its informal description : starting from a colored graph , the algorithm iteratively assigns , for a certain number of rounds , a new color to every node in the graph ; this is done in such a way that the color of a node in each round has a one to one correspondence with its own color and the multiset of colors of its neighbors in the previous round . An important observation is that the rounds of the WL algorithm can be seen as the layers of an AC-GNN whose aggregation and combination functions are all injective ( Morris et al. , 2019 ; Xu et al. , 2019 ) . Furthermore , as the following proposition states , an AC-GNN classification can never contradict the WL test . Proposition 2.1 ( Morris et al. , 2019 ; Xu et al. , 2019 ) . If the WL test assigns the same color to two nodes in a graph , then every AC-GNN classifies either both nodes as true or both nodes as false . 3 CONNECTION BETWEEN GNNS AND LOGIC .

Summary: The paper elaborates on the expressivity of graph neural networks (GNNs). More precisely, the authors show that expressivity of AC-GNNs (aggregate and combine) can only express logical classifiers that can be expressed in graded modal logic. By adding readouts, ACR-GNNs (aggregate, combine and readout) can capture FOC2 which is logical classifiers expressed with 2 variables and counting quantifiers. The second theorem leaves open the question of whether ACR-GNNs can capture logical classifiers beyond FOC2.