Learning on Random Balls is Sufficient for Estimating (Some) Graph Parameters

1 Introduction . Going beyond regular structural inputs such as grids ( images ) , sequences ( time series , sentences ) , or general feature vectors is an important research direction of machine learning and computational sciences . Arguably , most interesting objects and problems in nature can be described as graphs [ 37 ] . For such reason , graph learning methods , especially Graph Neural Networks ( GNN ) [ 60 ] , have recently proven to be a useful solution to many problems in computer vision [ 10 , 12 , 24 , 63 ] , complex network analyses [ 22 , 30 , 73 ] , molecule modeling [ 17 , 32 , 41 , 45 ] , and physics simulations [ 3 , 31 , 54 ] . The significant value of graph learning models in practice has inspired a large amount of theoretical work dedicated to exploring their representational limits and the possibilities of improving them . Most notably , the representational capability of GNNs has been in the spotlight of recent years . To answer the question “ Can GNNs approximate all functions on graphs ? ” , researchers discussed universal invariant and equivariant neural networks [ 27 , 40 , 43 , 51 ] as theoretical upper limits for neural architectures or showed the correspondence between message-passing GNNs ( MP-GNNs ) to the Weisfeiler-Leman ( WL ) algorithm [ 68 ] as practical upper limits [ 46 , 70 ] . Given an extremely large graph as an input , it is often impractical to keep the whole graph in the working memory . Therefore , practical graph learning methods often utilize neighborhood samplings [ 22 , 73 ] or random walks [ 53 ] to handle this scalability issue . Because existing analyses assumed a complete observation of the input graphs [ 27 , 51 , 57 ] , it is unclear what can be learned if we combine graph learning models with random samplings . Thus , the relevant question in this scenario is “ What graph functions are representable by GNNs when we can only observe random neighborhoods ? . ” This question adds another dimension to the discussion of GNN expressivity ; even if we have a powerful GNN ( in both theoretical and practical senses ) , what kind of graph functions can we learn if the input graphs are too large to be computed as a whole ? Contributions This study proposes a theoretical approach to address graph learning problems on large graphs by identifying a novel topology of the graph space . We discuss the graph classification 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . problem in the main part of the paper and extend the discussion to the vertex classification problem in Appendix C. The extension to vertex classification can be realised by viewing it as a rooted-graph classification problem . We first introduce a random ball sampling GNN ( RBS-GNN ) , which is a mathematical model of GNNs implementable in a random neighborhood computational model , and prove that the model is universal in the class of estimable functions ( Theorem 4 ) . Our main contribution is introducing randomized Benjamini–Schramm topology in the space of all graphs and identifying the estimability of the function as the uniform continuity in this topology ( Theorem 7 ) . By applying our main theorem , we obtain the following learning-theoretic results . • We show the equivalent of estimability and continuity ( Theorems 4 and 7 ) . This implies the continuity assumption is a sufficient condition for the mini-batch learning on graphs . • We prove that the functions representable by RBS-GNNs are generalizable by showing an upper bound of the Rademacher complexity of Lipschitz graph functions ( Theorem 10 ) . • We identify size-generalizable functions with estimable functions ( Theorem 11 ) . Then , by recognizing the size-generalization as a domain adaptation , we provide a size-generalization error based on the Wasserstein distance ( Theorem 13 ) . Unlike existing studies , which assumed a random graph model [ 26 , 28 ] or boundedness [ 11 , 27 , 51 , 58 ] , our framework does not assume anything about the graph class ; instead , we assume the continuity of the graph functions . Our results listed above are model-agnostic , i.e. , we only discuss the property of the function space , regardless of how GNN models are implemented . The model-agnostic nature of our results gives a systematic view to general graph parameters learning ; their generality is especially useful as there are many different GNN architectures in practice [ 57 , 69 , 78 ] . 2 Related Work . Large-scale GNNs The success of GNNs , especially vertex classification models like GCN [ 30 ] and GraphSAGE [ 22 ] , has led to various large-scale industrial GNN systems ( see [ 1 ] and references therein ) . Aiming to increase computational throughput while maintaining the predictive performance , most of these systems implemented fixed-size neighborhood sampling [ 22 ] to enable large-scale batching [ 23 , 73 , 76 , 77 ] . GNNs have also been applied to the 3D point clouds classification problem [ 67 ] , which translates a computer vision problem to the large graph classification problem , and the random sampling was empirically shown to be effective [ 35 ] . In this context , our work contributes a theoretical justification for the random sampling procedure . Graph Parameter Learning Graph function , graph parameter , or graph invariant refer to a ( real or integer value ) property of graphs , which only depends on the graph structure . In other words , they are functions defined on isomorphism classes of graphs [ 37 ] . Determining graph properties from data has long been a topic of interest in theoretical computer science [ 8 , 37 ] and is an important machine learning task in computational chemistry [ 9 , 17 ] and biology [ 6 , 16 ] . Recently , GNNs have been proven successful on a wide range of graph learning benchmark datasets . Current literature analyzed their expressivity to gain a better understanding of the architectures [ 27 , 44 , 51 ] . Several works identify MP-GNNs to the 1-dimensional WL isomorphism test [ 70 ] and further improve the GNN architectures to more expressive variants such as k-dimensional WL [ 46 ] , port-numbered message passing [ 58 ] , and sparse WL [ 48 ] . GNNs are also linked to the representational power of logical expressions [ 2 ] . These theoretical results assumed the complete observation of the input graph ; therefore , it is difficult to see to what extent these results would hold when the only partial observation is available . By studying the RBS-GNN model , we give an answer to this issue . We use GNNs because they are the most expressive graph learning methods [ 27 , 70 ] . Nonetheless , our results generalize for other universal ( Theorem 4 ) and partially-universal ( Theorem 17 ) methods . Generalization Besides expressivity , another challenge in graph learning is to understand the generalization bounds . Scarselli et al . [ 61 ] introduced an upper bound for the VC-dimension of functions computable by GNNs , in which the output is defined on a special supervised vertex . Garg et al . [ 15 ] derived tighter Rademacher complexity bounds for similar MP-GNNs by considering the local computational tree structures . Liao et al . [ 36 ] obtained a generalization gap of MP-GNNs and GCNs [ 30 ] using PAC-Bayes techniques . Du et al . [ 11 ] obtained a sample complexity using a result in the kernel method for their graph neural tangent kernel model in learning propagation-based functions . Verma and Zhang [ 66 ] obtained a generalization gap of single-layer GCNs by analyzing the stability and dependency on the largest eigenvalue of the graph ; Lv [ 39 ] derived a Rademacher bound for a similar GCN model with a similar dependency . Keriven et al . [ 28 ] assumed an underlying random kernel ( similar to graphons [ 37 ] ) and analyzed the stability of discrete GCN using a continuous counterpart c-GCN . They derived the convergence bounds by looking at stability when diffeomorphisms [ 42 ] are applied to the underlying graph kernel , the distribution , and the signals . All these methods placed some assumptions on the graph space ; either bounded degree [ 15 , 36 , 39 ] , bounded number of vertices [ 11 , 61 , 66 ] , or graphs belong to a random model [ 28 ] . Therefore , all these results become either inapplicable or unbounded in the general graph space . Our Theorem 10 contributes a complexity bound without assumptions on the graphs . Property Testing and Constant-Time Local Algorithms Property testing on graphs is a task to identify whether the input graph satisfies a graph property Π or -far from Π [ 18 ] . Often a researcher in this area tries to derive an algorithm whose complexity is constant ( i.e. , only depends on ) or sublinear in the input size [ 56 ] . Several graph properties admit sub-linear ( or constant-time ) algorithms ; the examples include bipartite testing , triangle-free testing , edge connectivity , and matching [ 49 , 74 ] . Recently , by bridging the constant-time algorithms and the GNN literature , Sato et al . [ 59 ] showed that , for each vertex , the neighborhood aggregation procedure of a GNN layer ( they called it “ node embedding ” ) can be approximated in constant time . However , this does not result in a constant-time learning algorithm for GNNs because we still need to access all the vertices to get the desired outputs . Our results provide the first “ fully constant-time ” GNNs in the sense that the whole learning and prediction process runs in time independent of the size of the graphs ( Section 4 ) . Statistical Theory for Network Analysis Learning graph property from samples is a traditional topic in statistical network analysis [ 14 , 21 , 34 ] . Klusowski and Wu [ 33 ] proved that it is hard to estimate the number of connected components using sublinear-size samples . Bhattacharya et al . [ 5 ] showed that the Horvitz–Thompson estimator for the number of subgraphs of constant size is consistent and asymptotic normal if the fourth-moment condition holds . Our result is consistent with these results as the number of connected components is non-continuous and the number of motifs is continuous in the randomized Benjamini–Schramm topology . These studies indicate a future direction of this study ; for example , the asymptotic normality and consistency of RBS-GNNs . 3 Preliminaries . 3.1 Graphs . A ( directed ) graph G is a tuple ( V , E ) of the set of vertices V and the set of edges E ⊆ V × V . We use V ( G ) for V and E ( G ) for E when the graph is unclear from the context . A graph is weakly connected if the underlying undirected graph has a path between any two vertices . A weakly connected component is a maximal weakly connected subgraph . Two graphs G and H are isomorphic if there is a bijection φ : V ( G ) → V ( H ) such that ( φ ( u ) , φ ( v ) ) ∈ E ( H ) if and only if ( u , v ) ∈ E ( G ) . Let G be the set of all directed graphs . A ball of radius r centered at v , Br ( v ) ( also simply B ) , is the set of vertices whose shortest path distance from v is bounded by r. For U ⊆ V ( G ) , G [ U ] is the subgraph of G induced by U . A rooted graph ( G , v ) is a graph G augmented with a vertex v in V ( G ) . The isomorphism between ( G , v ) and ( H , u ) is defined in the same way as for graphs with the extra requirement that it maps v to u . A k-rooted graph ( G , v1 , . . . , vk ) is defined similarly . We often recognize the graph induced by the ball Br ( v ) as a rooted graph whose root is v and by the union of k balls as a k-rooted graph . Modern graph learning problems ask for a function p : G → D from training data , where D is a “ learning-friendly ” domain such as the set of real numbers R , a d-dimensional real vector space Q ⊆ Rd , or some finite sets . In most cases , the function p is required to be isomorphism-invariant ( or invariant for short ) . This notion of graph functions coincides with the definition of graph parameters . Another term used in the literature is graph property , which can be formalized as a graph function whose co-domain is { 0 , 1 } . Our work focuses on the case in which the co-domain is R .

Summary: To take the words of the authors directly, this paper answers the question "What graph functions are representable by GNNs when we can only observe random neighborhoods?" In answering this question, the authors consider a notion of an estimable function in a random neighborhood model, which they then show is equivalent to uniform continuity in a metric topology, which they dub the randomized Benjamini-Schramm topology. With this perspective in hand, they show that sufficiently powerful graph neural networks coupled with universal approximators are capable of estimating uniformly continuous functions in this topology. They then discuss implications of this in terms of robustness, sampling complexity, and generalization.