On Size Generalization in Graph Neural Networks

1 INTRODUCTION . Graphs are a flexible representation , widely used for representing diverse data and phenomena . Graph neural networks ( GNNs ) – Deep models that operate over graphs – have emerged as a prominent learning model ( Bruna et al. , 2013 ; Kipf and Welling , 2016 ; Veličković et al. , 2017 ) . They are used in natural sciences ( Gilmer et al. , 2017 ) , social network analysis ( Fan et al. , 2019 ) , for solving difficult mathematical problems ( Luz et al. , 2020 ) and for approximating solutions to combinatorial optimization problems ( Li et al. , 2018 ) . In many domains , graphs data vary significantly in size . This is the case for molecular biology , where molecules – represented as graphs over atoms as nodes – span from small compounds to proteins with many thousands of nodes . It is even more severe in social networks , which can reach billions of nodes . The success of GNNs for such data stems from the fact that the same GNN model can process input graphs regardless of their size . Indeed , it has been proposed that GNNs can generalize to graphs whose size is different from what they were trained on , but it is largely unknown in what problems such generalization occurs . Empirically , several papers report good generalization performance on specific tasks ( Li et al. , 2018 ; Luz et al. , 2020 ) . Other papers , like Veličković et al . ( 2019 ) , show that size generalization can fail on several simple graph algorithms , and can be improved by using task-specific training procedures and specific architectures . Given their flexibility to operate on variable-sized graphs , A fundamental question arises about generalization in GNNs : ” When do GNNs trained on small graphs generalize to large graphs ? ” Aside from being an intriguing theoretical question , this problem has important practical implications . In many domains , it is hard to label large graphs . For instance , in combinatorial optimization problems , labeling a large graph boils down to solving large and hard optimization problems . In other domains , it is often very hard for human raters to correctly label complex networks . One approach to this problem could have been to resize graphs into a homogeneous size . This is the strategy taken in computer vision , where it is well understood how to resize an image while keeping its content . Unfortunately , there are no effective resizing procedures for graphs . It would therefore be extremely valuable to develop techniques that can generalize from training on small graphs . As we discuss below , a theoretical analysis of size generalization is very challenging because it depends on several different factors , including the task , the architecture , and the data . For tasks , we argue that it is important to distinguish two types of tasks , local and global . Local tasks can be solved by GNNs whose depth does not depend on the size of the input graph . For example , the task of finding a constant-size pattern . Global tasks require that the depth of the GNN grows with the size of the input graph . For example , calculating the diameter of a graph . While there are a few previous works that explore depth-dependant GNNs ( e.g. , Tang et al . ( 2020 ) , constant depth GNNs are by far the most widely used GNN models today and are therefore the focus of this paper . In this paper , we focus on GNNs with constant depth and study the ability of the most expressive message passing neural networks ( Xu et al. , 2018 ; Morris et al. , 2019 ) to generalize to unseen sizes . Our key observation is that generalization to graphs of different sizes is strongly related to the distribution of patterns around nodes in the graphs of interest . These patterns , dubbed d-patterns ( where d is the radius of the local neighborhood ) , describe the local feature-connectivity structure around each node , as seen by message-passing neural networks and are defined in Section 3 . We study the role of d-patterns both empirically and theoretically . First , we theoretically show that when there is a significant discrepancy between the d-pattern distributions , GNNs have multiple global minima for graphs of a specific size range , out of which only a subset of models can generalize well to larger graphs . We complement our theoretical analysis with an experimental study and show that GNNs tend to converge to non-generalizing global minima , when d-patterns from the large graph distribution are not well-represented in the small graph distribution . Furthermore we demonstrate that the size generalization problem is accentuated in deeper GNNs . Following these observations , in the final part of this paper , we discuss two learning setups that help improve size-generalization by formulating the learning problem as a domain adaptation problem : ( 1 ) Training the GNNs on self-supervised tasks aimed at learning the d-pattern distribution of both the target ( large graphs ) and source ( small graphs ) domains . We also propose a novel SSL task that addresses over-fitting of d-patterns . ( 2 ) A semi-supervised learning setup with a limited number of labeled examples from the target domain . The idea behind both setups is to promote convergence of GNNs to local/global minima with good size generalization properties . We show that both setups are useful in a series of experiments on synthetic and real data . To summarize , this paper makes the following contributions . ( 1 ) We identify a size generalization problem when learning local tasks with GNNs and analyze it empirically and theoretically . ( 2 ) We link the size-generalization problem with the distribution of d-patterns and suggest to approach it as a domain adaptation problem ( 3 ) We empirically show how several learning setups help improve size generalization . 2 RELATED WORK . Size generalization in set and graph learning . Several papers observed successful generalization across graph sizes , but the underlying reasons were not investigated ( Li et al. , 2018 ; Maron et al. , 2018 ; Luz et al. , 2020 ) . More recently , ( Veličković et al. , 2019 ) showed that when training GNNs to perform simple graph algorithms step by step they generalize better to graphs of different sizes . Unfortunately , such training procedures can not be easily applied to general tasks . Knyazev et al . ( 2019 ) studied the relationship between generalization and attention mechanisms . Tang et al . ( 2020 ) observed two issues that can harm generalization : ( 1 ) There are tasks for which a constant number of layers is not sufficient . ( 2 ) Some graph learning tasks are homogeneous functions . They then suggest a new GNN architecture to deal with these issues . Our work is complementary to these works as it explores another fundamental size generalization problem , focusing on constant depth GNNs . For more details on the distinction between constant depth and variable depth tasks see Appendix A . Several works also studied size generalization and expressivity when learning set-structured inputs ( Zweig and Bruna , 2020 ; Bueno and Hylton , 2020 ) . On the more practical side , Joshi et al . ( 2019 ) , Joshi et al . ( 2020 ) study the combinatorial problem of traveling salesman and whether it is possible to generalize to larger sizes . Corso et al . ( 2020 ) study several multitask learning problems on graphs and evaluate how the performance changes as the size of the graphs change . Expressivity and generalization in graph neural networks . ( Xu et al. , 2018 ; Morris et al. , 2019 ) established a fundamental connection between message-passing neural networks and the WeisfeilerLeman ( WL ) graph-isomorphism test . We use similar arguments to show that GNNs have enough expressive power to solve a task on a set of small graphs and to fail on it on a set of large graphs . Several works studied generalization bounds for certain classes of GNNs ( Garg et al. , 2020 ; Puny et al. , 2020 ; Verma and Zhang , 2019 ) , but did not discuss size-generalization . Sinha et al . ( 2020 ) proposed a benchmark for assessing the logical generalization abilities of GNNs . 3 THE SIZE GENERALIZATION PROBLEM . We now present the main problem discussed in the paper , that is , what determines if a GNN generalizes well to graphs of sizes not seen during training . We start with a simple motivating example showing the problem on single layer GNNs . We then show that the question of size generalization actually depends on d-patterns , the local patterns of connectivity and features of the graphs , and not only on their actual size . Setup . We are given two distributions over graphs P1 , P2 that contain small and large graphs accordingly , and a task that can be solved with 0 error for all graph sizes using a constant depth GNN . We train a GNN on a training set S sampled i.i.d from P1 and study its performance on P2 . GNN model . We focus on the first order GNN ( 1-GNN ) architecture from Morris et al . ( 2019 ) defined in the following way : h ( t ) v = σ W ( t ) 2 h ( t−1 ) v + ∑ u∈N ( v ) W ( t ) 1 h ( t−1 ) u + b ( t ) . Here , h ( t ) v is the feature vector of node v after t layers , W ( t ) 1 , W ( t ) 2 ∈ Rdt−1×dt , b ( t ) ∈ Rdt denotes the parameters of the t-th layer of the GNN , and σ is some non-linear activation ( e.g ReLU ) . It was shown in Morris et al . ( 2019 ) that GNNs composed from these layers have maximal expressive power with respect to all message-passing neural networks . In the experimental section we also experiment with Graph Isomorphism Network ( GIN ) ( Xu et al. , 2018 ) . For further details on GNNs see Appendix A . In this work we use the most expressive GNN variants that use the ” sum ” aggregation function . Using a ” max ” or ” mean ” reduces the expressive power of the network , making it not powerful enough to solve simple counting problems ( e.g . counting edges or computing node degrees ) . On the other hand , these networks give rise to slightly different definitions of patterns and can generalize better in some cases as shown in ( Veličković et al. , 2019 ) , yet still suffer from size overfit . Exploring these networks is beyond the scope of this work . 3.1 SIZE GENERALIZATION IN SINGLE-LAYER GNNS . We start our discussion on size generalization with a theoretical analysis of a simple setup . We consider a single-layer GNN and an easy task and show that : ( 1 ) The training objective has many different solutions , but only a small subset of these solutions generalizes to larger graphs ( 2 ) Simple regularization techniques can not mitigate the problem . This subsection serves as a warm-up for the next subsections that contain our main results . Assume we train on a distribution of graphs with a fixed number of nodes n and a fixed number of edges m. Our goal is to predict the number of edges in the graph using a 1-GNN with a single linear layer and additive readout function , for simplicity also consider the squared loss . The objective boils down to the following function for any graph G in the training set : : L ( w1 , w2 , b ; G ) = ( ∑ u∈V ( G ) ( w1 · xu + ∑ v∈N ( u ) w2 · xv + b ) − y ) 2 . Here , G is an input graph , V ( G ) are the nodes of G , N ( v ) are all the neighbors of node v , w1 , w2 and b are the trainable parameters , y is the target ( m in this case ) and xv is the node feature for node v. Further , assume that we have no additional information on the nodes , so we can just embed each node as a one-dimensional feature vector with a fixed value of 1 . In this simple case , the trainable parameters are also one-dimensional . We note that the training objective can also be written in the following form L ( w1 , w2 , b ; G ) = ( nw1 + 2mw2 + nb−m ) 2 , and that one can easily find its solutions space , which is an affine subspace defined by w2 = m−n ( w1+b ) 2m . In particular , the solutions with b+w1 = 0 , w2 = 1/2 are the only ones which do not depend on the specific training set graph size n , and generalize to graphs of any size . It can be readily seen that when training the model on graphs of fixed size ( fixed m , n ) , gradient descent will have no reason to favor one solution over another and we will not be able to generalize . We also note that the generalizing solution is not always the least norm solution ( with respect to both L1 and L2 norms ) so simple regularization will not help here . On the other hand , it is easy to show that training on graphs with different number of edges will favor the generalizing solution . As we will see next , the problem gets worse when considering GNNs with multiple non-linear layers , and this simple solution will not help in this case : we can train deeper GNNs on a wide variety of sizes and the solution will not generalize to other sizes . 3.2 d-PATTERNS We wish to understand theoretically when does a GNN which was trained on on graphs with a small number of nodes can generalize to graphs with a large number of nodes . To answer that question , we first analyze what information is received by each node in the graph from its neighboring nodes after a graph is processed by a GNN with T layers . It is easy to see that every node can receive information about its neighbors which are at most T hops away . We also know that nodes do not have full information about their order T environment . For example , GNNs can not determine if a triangle is present in a neighborhood of a given node Chen et al . ( 2020 ) . In order to characterize the exact information that can be found in each node after a T layer GNN , we use the definition of the WL test , specifically its iteration structure , which has the same representational power as GNNs ( see Xu et al . ( 2018 ) ; Morris et al . ( 2019 ) ) , For more details on the WL test see Appendix A . Definition 3.1 ( d-patterns ) . Let C be a finite set of node features and N ∈ N. For d ≥ 0 we define the set of d-patterns Pd on graphs with maximal degree N and node features from C. The definition is recursive in the following way : For d = 0 , P0 = C. We define Pd to be the set of all tuples ( a , b ) where a ∈ Pd−1 and b is in multisets of size at most N consisting of elements from Pd−1 . Let G = ( V , E ) be a graph with maximal degree N and a node feature cv ∈ C for every node v ∈ V . We define the d-pattern of a node v ∈ V for d ≥ 0 recursively : For d = 0 , its 0- pattern is cv . For d > 0 we say that v with ` neighboring d − 1 patterns has a d-pattern p = ( pv , { ( pi1 , mpi1 ) , . . . , ( pi ` , mpi ` ) } ) iff node v has ( d − 1 ) -pattern pv and for every j ∈ { 1 , . . . , ` } the number of neighbors of v with ( d− 1 ) -pattern pij is exactly mpij . The d-pattern of a node is an encoding of the ( d − 1 ) -patterns of itself and of its neighbors . For example , assume a graph has a maximal degree of N and all the nodes start with the same node feature . The 1-pattern of each node is its degree . The 2-pattern of each node is for each possible degree i ∈ { 1 , . . . , N } the number of neighbors with degree i . In the same manner , the 3-pattern of a node is for each possible 2-pattern , the number of its neighbors with this exact 2-pattern . The definition of d-patterns can naturally be extended to the case of unbounded degrees . We have the following theorem which connects the d-patterns with the expressive power of GNNs : Theorem 3.2 . Any function that can be represented by a d-layer GNN is constant on d-patterns . In particular , the theorem shows that for any two graphs ( of any size ) and two nodes , one in each graph , if the nodes have the exact same d-pattern , then any d-layer GNN will output the same result for the two nodes . The full proof can be found in Appendix B , and follows directly from the analogy between the WL algorithm ( see Appendix A ) and d-patterns . Thm . 3.2 implies that d-patterns don ’ t represent more expressive power than GNN . In the next subsection , we prove that GNNs can exactly compute d-patterns , and show that this capacity is tightly related to size generalization . It is also easy to see from the definition of d-patterns and the proof of Theorem 2 from Morris et al . ( 2019 ) that d-patterns exactly represent the expressive power of GNNs ( with additive aggregation ) , thus this definition is a natural tool to study the properties of GNNs , such as size generalization . 3.3 GNNS MAY OVERFIT d-PATTERNS We can now connect the size generalization problem to the concept of d-patterns . We start with an example : consider a node prediction task in which an output is specified for each node in an input graph , and is solvable by a d-layer GNN . To perfectly solve this task , the model should produce the correct output for the d-pattern of all the nodes in the training set . Testing this GNN on a different set of graphs will succeed if the test set has graphs with similar d-patterns to those in the training set . Note that this requirement is not related to the size of the graphs but to the distribution of the d-patterns of the nodes in the test set . In the following theorem we show rigorously , that given a set of d-patterns and output for each such pattern , there is an assignment of weights to a GNN with O ( d ) layers that perfectly fits the output for each pattern . We will then use this theorem in order to show that , under certain assumptions on the distribution of d-patterns of the large graphs , GNNs can perfectly solve a task on a set of small graphs , and completely fail on a set on large graphs . In other words , we show that there are multiple global minima for the training objective that do not generalize to larger graphs . Theorem 3.3 . Let C be a finite set of node features , P be a finite set of d-patterns on graphs with maximal degree N ∈ N , and for each pattern p ∈ P let yp ∈ [ −1 , 1 ] be some target label . Then there exists a 1-GNN F with d + 2 layers , width bounded by max { ( N + 1 ) d · |C| , 2 √ |P | } and ReLU activation such that for every graph G with nodes v1 , . . . , vn , corresponding d-patterns p1 , . . . , pn ⊆ P and node features from C , the output of F on node vi is exactly ypi . The full proof is in Appendix B . Note that the width of the required GNN from the theorem is not very large if d is small , where d represents the depth of the 1-GNN . In practice , shallow GNNs are very commonly used and are proven empirically successful , while training deep GNNs was shown to be hard due to many problems like over-smoothing ( Zhao and Akoglu , 2019 ) . Using the above theorem we can claim that there are assignments of weights to GNN that can not ” size-generalize ” , that is , given a specific task , the GNN succeeds on the task for small graphs ( up to some bound ) and fails on larger graphs , as long as there is a notable discrepancy between their d-patterns distributions : Corollary 3.4 . Let P1 and P2 be distributions of small and large graphs respectively with finite support , and let P d−pat1 be the distribution of d-patterns over small graphs and similarly P d−pat 2 for large graphs . For any node prediction task which is solvable by a 1-GNN with depth d and > 0 there exists a 1-GNN with depth at most d+ 2 that has 0-1 loss smaller then on P1 and 0-1 loss ∆ on P2 , where ∆ ( ) = max A : Pd−pat1 ( A ) < P d−pat2 ( A ) . ( 1 ) Here , A is a set of d-patterns and P ( A ) is the total probability mass for that set under P . Intuitively , large ∆ means that there exists a set of d-patterns that have a low probability for small graphs and high probability for large graphs . Corollary 3.4 implies that the major factor in the success of GNN to generalize to larger graphs is not the graph size , but the distribution of the dpatterns . Different distributions of d-patterns lead to large ∆ and thus to bad generalization to larger graphs . On the other hand , from Thm . 3.2 we immediately get that similar distributions of d-patterns imply that every GNN model that succeeds on small graphs will also succeed on large graphs , since GNNs are constant on d-patterns : Corollary 3.5 . In the setting of Corollary 3.4 , also assume that all the patterns that have a positive probability in P d−pat2 also have a positive probability in P d−pat 1 . Then , for any node prediction task solvable by a depth d GNN , any 1−GNN that have 0 loss ( w.r.t the 0−1 loss ) on P1 will also have 0 loss on P2 . Examples . Corollary 3.4 shows that even for simple tasks , GNN may fail , here are two simple examples . ( i ) Consider the task of calculating the node degree . From Corollary 3.4 there is a GNN that successfully output the degree of nodes with max degree up to N and fails on nodes with larger degrees . Note that this problem can easily be solved for any node degree with a 1-layer GNN . ( ii ) Consider some node regression task , when the training set consists of graphs sampled i.i.d from an Erdos-Renyi graph G ( n , p ) 1 , and the test set contains graphs sampled i.i.d from G ( 2n , p ) . In this case , a GNN trained on the training set will be trained on graphs with an average degree np , while the test set contains graphs with an average degree 2np . This means that the d-patterns in the train and test set are very different , and by Corollary 3.4 the GNN may overfit . 1Graphs with n nodes such that each edge exists with probability p. Graph prediction tasks . Our theoretical results discuss node prediction tasks . We note that they are also relevant for graph prediction tasks where there is a single output to each input graph . The reason for that is that in order to solve graph prediction tasks , a GNN first calculates node features and then pools them into a single global graph feature . Our analysis shows that the first part of the GNN , which is responsible for calculating the node features , might not generalize to large graphs . As a result , the GNN will generate an uninformative global graph feature and the GNN will fail on the original graph prediction task . In the experimental sections , we show that the size generalization problem is indeed relevant for both node and graph prediction tasks . Here is a formal statement regarding graph prediction tasks , the full proof can be found in Appendix B. Corollary 3.6 . Let P1 and P2 be distributions of small and large graphs respectively with finite support . Let P d−pat1 be the distribution of d-patterns over small graphs and similarly P d−pat 2 for large graphs , and assume that the supports of P d−pat1 and P d−pat 2 are disjoint . For any graph prediction task solvable by a 1-GNN with depth d and summation readout function , there exists a 1-GNN with depth at most d+ 3 that perfectly solves the task on P1 and fails on all graphs from P2 . Relation to Morris et al . ( 2019 ) ; Xu et al . ( 2018 ) . We note that Theorem 3.3 and Corollary 3.4 are somewhat related to the expressivity results in ( Xu et al. , 2018 ; Morris et al. , 2019 ) that show that GNNs can be as powerful as the WL test . Here , we show that the expressive power of GNNs can cause negative effects when there is a discrepancy between the training and test sets .

Summary: The paper considers the important problem of lack of generalization of GNNs to graphs whose sizes are much larger than the graphs on which they were trained on. The authors propose a theoretical explanation by arguing that this lack of generalization is due to a mismatch in the “d-patterns” (essentially the pattern of neighborhood around nodes) between the training and test graphs. To improve generalization performance, they borrow ideas from domain adaptation and propose a pattern-tree based pretext learning task which improves classification accuracy compared to baseline pretext tasks in a range of datasets and tasks.