Reconstruction for Powerful Graph Representations

1 Introduction . Supervised machine learning for graph-structured data , i.e. , graph classification and regression , is ubiquitous across application domains ranging from chemistry and bioinformatics [ 8 , 94 ] to image [ 89 ] , and social network analysis [ 33 ] . Consequently , machine learning on graphs is an active research area with numerous proposed approaches—notably GNNs [ 21 , 42 , 45 ] being the most representative case of GRL methods . Arguably , GRL ’ s most interesting results arise from a cross-over between graph theory and representation learning . For instance , the representational limits of GNNs are upper-bounded by a simple heuristic for the graph isomorphism problem [ 78 , 105 ] , the 1-dimensional Weisfeiler-Leman algorithm ( 1- WL ) [ 44 , 73 , 75 , 101 , 102 ] , which might miss crucial structural information in the data [ 5 ] . Further works show how GNNs can not approximate graph properties such as diameter , radius , girth , and subgraph counts [ 25 , 38 ] , inspiring architectures [ 6 , 66 , 77 , 78 ] based on the more powerful κ-dimensional Weisfeiler-Leman algorithm ( κ-WL ) [ 44 ] .1 On the other hand , despite the limited expressiveness of GNNs , they still can overfit the training data , offering limited generalization performance [ 105 ] . Hence , devising GRL architectures that are simultaneously sufficiently expressive and avoid overfitting remains an open problem . 1We opt for using κ instead of k , i.e. , κ-WL instead of k-WL , to not confuse the reader with the hyperparameter k of our models . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . An under-explored connection between graph theory and GRL is graph reconstruction , which studies graphs and graph properties uniquely determined by their subgraphs . In this direction , both the pioneering work of Shawe-Taylor [ 88 ] and the more recent work of Bouritsas et al . [ 18 ] , show that assuming the reconstruction conjecture ( see Conjecture 1 ) holds , their models are most-expressive representations ( universal approximators ) of graphs . Unfortunately , Shawe-Taylor ’ s computational graph grows exponentially with the number of vertices , and Bouritsas et al. ’ s full representation power requires performing multiple graph isomorphism tests on potentially large graphs ( with n− 1 vertices ) . Moreover , these methods were not inspired by the more general subject of graph reconstruction ; instead , they rely on the reconstruction conjecture to prove their architecture ’ s expressive powers . Contributions . In this work , we directly connect graph reconstruction to GRL . We first show how the k-reconstruction of graphs—reconstruction from induced k-vertex subgraphs—induces a natural class of expressive GRL architectures for supervised learning with graphs , denoted k-Reconstruction Neural Networks . We then show how several existing works have their expressive power limited by k-reconstruction . Further , we show how the reconstruction conjecture ’ s insights lead to a provably most expressive representation of graphs . Unlike Shawe-Taylor [ 88 ] and Bouritsas et al . [ 18 ] , which , for graph tasks , require fixed-size unattributed graphs and multiple ( large ) graph isomorphism tests , respectively , our method represents bounded-size graphs with vertex attributes and does not rely on isomorphism tests . To make our models scalable , we propose k-Reconstruction GNNs , a general tool for boosting the expressive power and performance of GNNs with graph reconstruction . Theoretically , we characterize their expressive power showing that k-Reconstruction GNNs can distinguish graph classes that the 1-WL and 2-WL can not , such as cycle graphs and strongly regular graphs , respectively . Further , to explain gains in real-world tasks , we show how reconstruction can act as a lower-variance risk estimator when the graph-generating distribution is invariant to vertex removals . Empirically , we show that reconstruction enhances GNNs ’ expressive power , making them solve multiple synthetic graph property tasks in the literature not solvable by the original GNN . On real-world datasets , we show that the increase in expressive power coupled with the lower-variance risk estimator boosts GNNs ’ performance up to 25 % . Our combined theoretical and empirical results make another important connection between graph theory and GRL . 1.1 Related work . We review related work from GNNs , their limitations , data augmentation , and the reconstruction conjecture in the following . See Appendix A for a more detailed discussion . GNNs . Notable instances of this architecture include , e.g. , [ 31 , 47 , 97 ] , and the spectral approaches proposed in , e.g. , [ 19 , 30 , 56 , 72 ] —all of which descend from early work in [ 10 , 57 , 69 , 70 , 71 , 87 , 90 ] . Aligned with the field ’ s recent rise in popularity , there exists a plethora of surveys on recent advances in GNN methods . Some of the most recent ones include [ 21 , 104 , 114 ] . Limits of GNNs . Recently , connections to Weisfeiler-Leman type algorithms have been shown [ 9 , 27 , 39 , 40 , 64 , 66 , 77 , 78 , 105 ] . Specifically , the authors of [ 78 , 105 ] show how the 1-WL limits the expressive power of any possible GNN architecture . Morris et al . [ 78 ] introduce κ-dimensional GNNs which rely on a more expressive message-passing scheme between subgraphs of cardinality κ . Later , this was refined in [ 6 , 66 ] and in [ 76 ] by deriving models equivalent to the more powerful κ-dimensional Weisfeiler-Leman algorithm . Chen et al . [ 27 ] connect the theory of universal approximation of permutation-invariant functions and graph isomorphism testing , further introducing a variation of the 2- WL . Recently , a large body of work propose enhancements to GNNs , e.g. , see [ 3 , 11 , 14 , 18 , 80 , 98 , 109 ] , making them more powerful than the 1-WL ; see [ 75 ] for a survey and see Appendix A for a in-depth discussion . For clarity , throughout this work , we will use the term GNNs to denote the class of message-passing architectures limited by the 1-WL algorithm , where the class of distinguishable graphs is well understood [ 5 ] . Data augmentation , generalization and subgraph-based inductive biases . There exist few works proposing data augmentation for GNNs for graph classification . Kong et al . [ 59 ] introduces a simple feature perturbation framework to achieve this , while Rong et al . [ 85 ] , Feng et al . [ 34 ] focus on vertex-level tasks . Garg et al . [ 38 ] study the generalization abilities of GNNs showing bounds on the Rademacher complexity , while Liao et al . [ 62 ] offer a refined analysis within the PAC-Bayes framework . Recently , Bouritsas et al . [ 18 ] proposed to use subgraph counts as vertex and edge features in GNNs . Although the authors show an increase in expressiveness , the extent , e.g. , which graph classes their model can distinguish , is still mostly unclear . Moreover , Yehudai et al . [ 107 ] investigate GNNs ’ ability to generalize to larger graphs . Concurrently , Bevilacqua et al . [ 12 ] show how subgraph densities can be used to build size-invariant graph representations . However , the performance of such models in in-distribution tasks , their expressiveness , and scalability remain unclear . Finally , Yuan et al . [ 111 ] show how GNNs ’ decisions can be explained by ( often large ) subgraphs , further motivating our use of graph reconstruction as a powerful inductive bias for GRL . Reconstruction conjecture . The reconstruction conjecture is a longstanding open problem in graph theory , which has been solved in many particular settings . Such results come in two flavors . Either proving that graphs from a specific class are reconstructible or determining which graph functions are reconstructible . Known results of the former are , for instance , that regular graphs , disconnected graphs , and trees are reconstructible [ 16 , 53 ] . In particular , we highlight that outerplanar graphs , which account for most molecule graphs , are known to be reconstructible [ 41 ] . For a comprehensive review of graph reconstruction results , see Bondy [ 16 ] . 2 Preliminaries . Here , we introduce notation and give an overview of the main results in graph reconstruction theory [ 16 , 43 ] , including the reconstruction conjecture [ 96 ] , which forms the basis of the models in this work . Notation and definitions . As usual , let [ n ] = { 1 , . . . , n } ⊂ N for n ≥ 1 , and let { { . . . } } denote a multiset . In an abuse of notation , for a set X with x in X , we denote by X − x the set X \ { x } . We also assume elementary definitions from graph theory , such as graphs , directed graphs , vertices , edges , neighbors , trees , isomorphism , et cetera ; see Appendix B . The vertex and the edge set of a graph G are denoted by V ( G ) and E ( G ) , respectively . The size of a graph G is equal to its number of vertices . Unless indicated otherwise , we use n : = |V ( G ) | . If not otherwise stated , we assume that vertices and edges are annoted with attributes , i.e. , real-valued vectors . We denote the set of all finite and simple graphs by G. The subset of G without edge attributes ( or edge directions ) is denoted G ⊂ G. We write G ' H if the graphs G and H are isomorphic . Further , we denote the isomorphism type , i.e. , the equivalence class of the isomorphism relation , of a graph G as I ( G ) . Let S ⊆ V ( G ) , then G [ S ] is the induced subgraph with edge set E ( G ) [ S ] = { S2 ∩ E ( G ) } . We will refer to induced subgraphs simply as subgraphs in this work . Let R be a family of graph representations , such that for d ≥ 1 , r in R , r : G → Rd , assigns a d-dimensional representation vector r ( G ) for a graph G in G. We say R can distinguish a graph G if there exists r in R that assigns a unique representation to the isomorphism type of G , i.e. , r ( G ) = r ( H ) if and only if G ' H . Further , we say R distinguishes a pair of non-isomorphic graphs G and H if there exists some r in R such that r ( G ) 6= r ( H ) . Moreover , we write R1 R2 if R2 distinguishes between all graphs R1 does , and R1 ≡ R2 if both directions hold . The corresponding strict relation is denoted by ≺ . Finally , we say R is a most-expressive representation of a class of graphs if it distinguishes all non-isomorphic graphs in that class . Graph reconstruction . Intuitively , the reconstruction conjecture states that an undirected edgeunattributed graph can be fully recovered up to its isomorphism type given the multiset of its vertexdeleted subgraphs ’ isomorphism types . This multiset of subgraphs is usually referred to as the deck of the graph , see Figure 1 for an illustration . Formally , for a graph G , we define its deck as Dn−1 ( G ) = { { I ( G [ V ( G ) − v ] ) : v ∈ V ( G ) } } . We often call an element in Dn−1 ( G ) a card . We define the graph reconstruction problem as follows . Definition 1 . Let G and H be graphs , then H is a reconstruction of G if H and G have the same deck , denoted H ∼ G. A graph G is reconstructible if every reconstruction of G is isomorphic to G , i.e. , H ∼ G implies H ' G. Similarly , we define function reconstruction , which relates functions that map two graphs to the same value if they have the same deck . Definition 2 . Let f : G → Y be a function , then f is reconstructible if f ( G ) = f ( H ) for all graphs in { ( H , G ) ∈ G2 : H ∼ G } , i.e. , G ∼ H implies f ( G ) = f ( H ) . We can now state the reconstruction conjecture , which in short says that every G in G with |V | ≥ 3 is reconstructible . Conjecture 1 ( Kelly [ 52 ] , Ulam [ 96 ] ) . Let H and G in G be two finite , undirected , simple graphs with at least three vertices . If H is a reconstruction of G , then H and G are isomorphic . We note here that the reconstruction conjecture does not hold for directed graphs , hypergraphs , and infinite graphs [ 16 , 92 , 93 ] . In particular , edge directions can be seen as edge attributes . Thus , the reconstruction conjecture does not hold for the class G. In contrast , the conjecture has been proved for practical-relevant graph classes , such as disconnected graphs , regular graphs , trees , and outerplanar graphs [ 16 ] . Further , computational searches show that graphs with up to eleven vertices are reconstructible [ 68 ] . Finally , many graph properties are known to be reconstructible , such as every size subgraph count , degree sequence , number of edges , and the characteristic polynomial [ 16 ] . Graph k-reconstruction . Kelly et al . [ 53 ] generalized graph reconstruction , considering the multiset of subgraphs of size k instead of n− 1 , which we denote Dk ( G ) = { { I ( H ) : H ∈ S ( k ) ( G ) } } , where S ( k ) is the set of all ( n k ) k-size subsets of V . We often call an element in Dk ( G ) a k-card . From the k-deck definition , it is easy to extend the concept of graph and function reconstruction , cf . Definitions 1 and 2 , to graph and function k-reconstruction . Definition 3 . Let G and H be graphs , then H is a k-reconstruction of G if H and G have the same k-deck , denoted H ∼k G. A graph G is k-reconstructible if every k-reconstruction of G is isomorphic to G , i.e. , H ∼k G implies H ' G. Accordingly , we define k-function reconstruction as follows . Definition 4 . Let f : G → Y be a function , then f is k-reconstructible if f ( G ) = f ( H ) for all graphs in { ( H , G ) ∈ G2 : H ∼k G } , i.e. , G ∼k H implies f ( G ) = f ( H ) . Results for k-reconstruction usually state the least k as a function of n such that all graphs G in G ( or some subset ) are k-reconstructible [ 84 ] . There exist extensive partial results in this direction , mostly describing k-reconstructibility ( as a function of n ) for a particular family of graphs , such as trees , disconnected graphs , complete multipartite graphs , and paths , see [ 84 , 60 ] . More concretely , Nỳdl [ 83 ] , Spinoza and West [ 91 ] showed graphs with 2k vertices that are not k-reconstructible . In practice , these results imply that for some fixed k there will be graphs with not many more vertices than k that are not k-reconstructible . Further , k-reconstructible graph functions such as degree sequence and connectedness have been studied in [ 65 , 91 ] depending on the size of k. In Appendix C , we discuss further such results .

Summary: This paper proposed several kinds of GNN models that integrate the representations of subgraphs to obtain graph representations.  First, the k-reconstruction NN is defined using the representation of a subgraph of size k for the input graph G. Under the reconstruction conjecture, the k-reconstruction NN is proven to be most expressive.   Next, this paper defined the k-reconstruction GNN, which is made by specializing k-reconstruction NN by obtaining representations using GNNs. The computational complexity of a k-reconstruction GNN is exponential with respect to k. Therefore, we proposed a method to approximate it by sampling subgraphs. The relationship between GNNs and k-reconstruction GNNs is discussed. Concretely, the k-reconstruction GNN is not less expressive than GNN if k is not too small. Also, k-reconstruction GNN is not more expressive than GNNs when k is small.  Finally, the proposed k-reconstruction GNN was applied to synthetic and real datasets (OGB). This paper showed that the proposed model improved the accuracy of the proposed GNN compared to baseline GNNs.