Reconstruction for Powerful Graph Representations

1 Introduction . Supervised machine learning for graph-structured data , i.e. , graph classification and regression , is ubiquitous across application domains ranging from chemistry and bioinformatics [ 8 , 94 ] to image [ 89 ] , and social network analysis [ 33 ] . Consequently , machine learning on graphs is an active research area with numerous proposed approaches—notably GNNs [ 21 , 42 , 45 ] being the most representative case of GRL methods . Arguably , GRL ’ s most interesting results arise from a cross-over between graph theory and representation learning . For instance , the representational limits of GNNs are upper-bounded by a simple heuristic for the graph isomorphism problem [ 78 , 105 ] , the 1-dimensional Weisfeiler-Leman algorithm ( 1- WL ) [ 44 , 73 , 75 , 101 , 102 ] , which might miss crucial structural information in the data [ 5 ] . Further works show how GNNs can not approximate graph properties such as diameter , radius , girth , and subgraph counts [ 25 , 38 ] , inspiring architectures [ 6 , 66 , 77 , 78 ] based on the more powerful κ-dimensional Weisfeiler-Leman algorithm ( κ-WL ) [ 44 ] .1 On the other hand , despite the limited expressiveness of GNNs , they still can overfit the training data , offering limited generalization performance [ 105 ] . Hence , devising GRL architectures that are simultaneously sufficiently expressive and avoid overfitting remains an open problem . 1We opt for using κ instead of k , i.e. , κ-WL instead of k-WL , to not confuse the reader with the hyperparameter k of our models . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . An under-explored connection between graph theory and GRL is graph reconstruction , which studies graphs and graph properties uniquely determined by their subgraphs . In this direction , both the pioneering work of Shawe-Taylor [ 88 ] and the more recent work of Bouritsas et al . [ 18 ] , show that assuming the reconstruction conjecture ( see Conjecture 1 ) holds , their models are most-expressive representations ( universal approximators ) of graphs . Unfortunately , Shawe-Taylor ’ s computational graph grows exponentially with the number of vertices , and Bouritsas et al. ’ s full representation power requires performing multiple graph isomorphism tests on potentially large graphs ( with n− 1 vertices ) . Moreover , these methods were not inspired by the more general subject of graph reconstruction ; instead , they rely on the reconstruction conjecture to prove their architecture ’ s expressive powers . Contributions . In this work , we directly connect graph reconstruction to GRL . We first show how the k-reconstruction of graphs—reconstruction from induced k-vertex subgraphs—induces a natural class of expressive GRL architectures for supervised learning with graphs , denoted k-Reconstruction Neural Networks . We then show how several existing works have their expressive power limited by k-reconstruction . Further , we show how the reconstruction conjecture ’ s insights lead to a provably most expressive representation of graphs . Unlike Shawe-Taylor [ 88 ] and Bouritsas et al . [ 18 ] , which , for graph tasks , require fixed-size unattributed graphs and multiple ( large ) graph isomorphism tests , respectively , our method represents bounded-size graphs with vertex attributes and does not rely on isomorphism tests . To make our models scalable , we propose k-Reconstruction GNNs , a general tool for boosting the expressive power and performance of GNNs with graph reconstruction . Theoretically , we characterize their expressive power showing that k-Reconstruction GNNs can distinguish graph classes that the 1-WL and 2-WL can not , such as cycle graphs and strongly regular graphs , respectively . Further , to explain gains in real-world tasks , we show how reconstruction can act as a lower-variance risk estimator when the graph-generating distribution is invariant to vertex removals . Empirically , we show that reconstruction enhances GNNs ’ expressive power , making them solve multiple synthetic graph property tasks in the literature not solvable by the original GNN . On real-world datasets , we show that the increase in expressive power coupled with the lower-variance risk estimator boosts GNNs ’ performance up to 25 % . Our combined theoretical and empirical results make another important connection between graph theory and GRL . 1.1 Related work . We review related work from GNNs , their limitations , data augmentation , and the reconstruction conjecture in the following . See Appendix A for a more detailed discussion . GNNs . Notable instances of this architecture include , e.g. , [ 31 , 47 , 97 ] , and the spectral approaches proposed in , e.g. , [ 19 , 30 , 56 , 72 ] —all of which descend from early work in [ 10 , 57 , 69 , 70 , 71 , 87 , 90 ] . Aligned with the field ’ s recent rise in popularity , there exists a plethora of surveys on recent advances in GNN methods . Some of the most recent ones include [ 21 , 104 , 114 ] . Limits of GNNs . Recently , connections to Weisfeiler-Leman type algorithms have been shown [ 9 , 27 , 39 , 40 , 64 , 66 , 77 , 78 , 105 ] . Specifically , the authors of [ 78 , 105 ] show how the 1-WL limits the expressive power of any possible GNN architecture . Morris et al . [ 78 ] introduce κ-dimensional GNNs which rely on a more expressive message-passing scheme between subgraphs of cardinality κ . Later , this was refined in [ 6 , 66 ] and in [ 76 ] by deriving models equivalent to the more powerful κ-dimensional Weisfeiler-Leman algorithm . Chen et al . [ 27 ] connect the theory of universal approximation of permutation-invariant functions and graph isomorphism testing , further introducing a variation of the 2- WL . Recently , a large body of work propose enhancements to GNNs , e.g. , see [ 3 , 11 , 14 , 18 , 80 , 98 , 109 ] , making them more powerful than the 1-WL ; see [ 75 ] for a survey and see Appendix A for a in-depth discussion . For clarity , throughout this work , we will use the term GNNs to denote the class of message-passing architectures limited by the 1-WL algorithm , where the class of distinguishable graphs is well understood [ 5 ] . Data augmentation , generalization and subgraph-based inductive biases . There exist few works proposing data augmentation for GNNs for graph classification . Kong et al . [ 59 ] introduces a simple feature perturbation framework to achieve this , while Rong et al . [ 85 ] , Feng et al . [ 34 ] focus on vertex-level tasks . Garg et al . [ 38 ] study the generalization abilities of GNNs showing bounds on the Rademacher complexity , while Liao et al . [ 62 ] offer a refined analysis within the PAC-Bayes framework . Recently , Bouritsas et al . [ 18 ] proposed to use subgraph counts as vertex and edge features in GNNs . Although the authors show an increase in expressiveness , the extent , e.g. , which graph classes their model can distinguish , is still mostly unclear . Moreover , Yehudai et al . [ 107 ] investigate GNNs ’ ability to generalize to larger graphs . Concurrently , Bevilacqua et al . [ 12 ] show how subgraph densities can be used to build size-invariant graph representations . However , the performance of such models in in-distribution tasks , their expressiveness , and scalability remain unclear . Finally , Yuan et al . [ 111 ] show how GNNs ’ decisions can be explained by ( often large ) subgraphs , further motivating our use of graph reconstruction as a powerful inductive bias for GRL . Reconstruction conjecture . The reconstruction conjecture is a longstanding open problem in graph theory , which has been solved in many particular settings . Such results come in two flavors . Either proving that graphs from a specific class are reconstructible or determining which graph functions are reconstructible . Known results of the former are , for instance , that regular graphs , disconnected graphs , and trees are reconstructible [ 16 , 53 ] . In particular , we highlight that outerplanar graphs , which account for most molecule graphs , are known to be reconstructible [ 41 ] . For a comprehensive review of graph reconstruction results , see Bondy [ 16 ] . 2 Preliminaries . Here , we introduce notation and give an overview of the main results in graph reconstruction theory [ 16 , 43 ] , including the reconstruction conjecture [ 96 ] , which forms the basis of the models in this work . Notation and definitions . As usual , let [ n ] = { 1 , . . . , n } ⊂ N for n ≥ 1 , and let { { . . . } } denote a multiset . In an abuse of notation , for a set X with x in X , we denote by X − x the set X \ { x } . We also assume elementary definitions from graph theory , such as graphs , directed graphs , vertices , edges , neighbors , trees , isomorphism , et cetera ; see Appendix B . The vertex and the edge set of a graph G are denoted by V ( G ) and E ( G ) , respectively . The size of a graph G is equal to its number of vertices . Unless indicated otherwise , we use n : = |V ( G ) | . If not otherwise stated , we assume that vertices and edges are annoted with attributes , i.e. , real-valued vectors . We denote the set of all finite and simple graphs by G. The subset of G without edge attributes ( or edge directions ) is denoted G ⊂ G. We write G ' H if the graphs G and H are isomorphic . Further , we denote the isomorphism type , i.e. , the equivalence class of the isomorphism relation , of a graph G as I ( G ) . Let S ⊆ V ( G ) , then G [ S ] is the induced subgraph with edge set E ( G ) [ S ] = { S2 ∩ E ( G ) } . We will refer to induced subgraphs simply as subgraphs in this work . Let R be a family of graph representations , such that for d ≥ 1 , r in R , r : G → Rd , assigns a d-dimensional representation vector r ( G ) for a graph G in G. We say R can distinguish a graph G if there exists r in R that assigns a unique representation to the isomorphism type of G , i.e. , r ( G ) = r ( H ) if and only if G ' H . Further , we say R distinguishes a pair of non-isomorphic graphs G and H if there exists some r in R such that r ( G ) 6= r ( H ) . Moreover , we write R1 R2 if R2 distinguishes between all graphs R1 does , and R1 ≡ R2 if both directions hold . The corresponding strict relation is denoted by ≺ . Finally , we say R is a most-expressive representation of a class of graphs if it distinguishes all non-isomorphic graphs in that class . Graph reconstruction . Intuitively , the reconstruction conjecture states that an undirected edgeunattributed graph can be fully recovered up to its isomorphism type given the multiset of its vertexdeleted subgraphs ’ isomorphism types . This multiset of subgraphs is usually referred to as the deck of the graph , see Figure 1 for an illustration . Formally , for a graph G , we define its deck as Dn−1 ( G ) = { { I ( G [ V ( G ) − v ] ) : v ∈ V ( G ) } } . We often call an element in Dn−1 ( G ) a card . We define the graph reconstruction problem as follows . Definition 1 . Let G and H be graphs , then H is a reconstruction of G if H and G have the same deck , denoted H ∼ G. A graph G is reconstructible if every reconstruction of G is isomorphic to G , i.e. , H ∼ G implies H ' G. Similarly , we define function reconstruction , which relates functions that map two graphs to the same value if they have the same deck . Definition 2 . Let f : G → Y be a function , then f is reconstructible if f ( G ) = f ( H ) for all graphs in { ( H , G ) ∈ G2 : H ∼ G } , i.e. , G ∼ H implies f ( G ) = f ( H ) . We can now state the reconstruction conjecture , which in short says that every G in G with |V | ≥ 3 is reconstructible . Conjecture 1 ( Kelly [ 52 ] , Ulam [ 96 ] ) . Let H and G in G be two finite , undirected , simple graphs with at least three vertices . If H is a reconstruction of G , then H and G are isomorphic . We note here that the reconstruction conjecture does not hold for directed graphs , hypergraphs , and infinite graphs [ 16 , 92 , 93 ] . In particular , edge directions can be seen as edge attributes . Thus , the reconstruction conjecture does not hold for the class G. In contrast , the conjecture has been proved for practical-relevant graph classes , such as disconnected graphs , regular graphs , trees , and outerplanar graphs [ 16 ] . Further , computational searches show that graphs with up to eleven vertices are reconstructible [ 68 ] . Finally , many graph properties are known to be reconstructible , such as every size subgraph count , degree sequence , number of edges , and the characteristic polynomial [ 16 ] . Graph k-reconstruction . Kelly et al . [ 53 ] generalized graph reconstruction , considering the multiset of subgraphs of size k instead of n− 1 , which we denote Dk ( G ) = { { I ( H ) : H ∈ S ( k ) ( G ) } } , where S ( k ) is the set of all ( n k ) k-size subsets of V . We often call an element in Dk ( G ) a k-card . From the k-deck definition , it is easy to extend the concept of graph and function reconstruction , cf . Definitions 1 and 2 , to graph and function k-reconstruction . Definition 3 . Let G and H be graphs , then H is a k-reconstruction of G if H and G have the same k-deck , denoted H ∼k G. A graph G is k-reconstructible if every k-reconstruction of G is isomorphic to G , i.e. , H ∼k G implies H ' G. Accordingly , we define k-function reconstruction as follows . Definition 4 . Let f : G → Y be a function , then f is k-reconstructible if f ( G ) = f ( H ) for all graphs in { ( H , G ) ∈ G2 : H ∼k G } , i.e. , G ∼k H implies f ( G ) = f ( H ) . Results for k-reconstruction usually state the least k as a function of n such that all graphs G in G ( or some subset ) are k-reconstructible [ 84 ] . There exist extensive partial results in this direction , mostly describing k-reconstructibility ( as a function of n ) for a particular family of graphs , such as trees , disconnected graphs , complete multipartite graphs , and paths , see [ 84 , 60 ] . More concretely , Nỳdl [ 83 ] , Spinoza and West [ 91 ] showed graphs with 2k vertices that are not k-reconstructible . In practice , these results imply that for some fixed k there will be graphs with not many more vertices than k that are not k-reconstructible . Further , k-reconstructible graph functions such as degree sequence and connectedness have been studied in [ 65 , 91 ] depending on the size of k. In Appendix C , we discuss further such results .

Summary: The paper uses principles of the reconstruction conjecture to devise powerful graph representations addressing the current challenges in graph representation learning. The authors first present a theoretical construction of reconstruction-neural-networks, which is impractical because a universal graph function is not learnable. They then construct an approximate k-reconstruction-GNNs, which replace the graph representation function with GNN layers. The authors also discuss the hierarchy of expressive power and the relation of k-reconstruction networks to GNNs expressive power. They show theoretical guarantees regarding the expressive power of the suggested model, observing that under some conditions, k-reconstruction networks can distinguish graphs that GNNs cannot and even 2-GNNs cannot. While under other conditions, the opposite is true. The authors conduct an extensive set of experiments showing the model’s ability to distinguish several types of structures GNNs cannot, and also conduct some experiments on real-world tasks.