Title: Optimality of Message-Passing Architectures for Sparse Graphs
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

message passing

bayesian inference

node classification

contextual stochastic block model
Abstract: We study the node classification problem on feature-decorated graphs in the sparse setting, i.e., when the expected degree of a node is $O(1)$ in the number of nodes, in the fixed-dimensional asymptotic regime, i.e., the dimension of the feature data is fixed while the number of nodes is large. Such graphs are typically known to be locally tree-like. We introduce a notion of Bayes optimality for node classification tasks, called asymptotic local Bayes optimality, and compute the optimal classifier according to this criterion for a fairly general statistical data model with arbitrary distributions of the node features and edge connectivity. The optimal classifier is implementable using a message-passing graph neural network architecture. We then compute the generalization error of this classifier and compare its performance against existing learning methods theoretically on a well-studied statistical model with naturally identifiable signal-to-noise ratios (SNRs) in the data. We find that the optimal message-passing architecture interpolates between a standard MLP in the regime of low graph signal and a typical convolution in the regime of high graph signal. Furthermore, we prove a corresponding non-asymptotic result.

Optimality of Message-Passing Architectures for Sparse Graphs

Abstract

We study the node classification problem on feature-decorated graphs in the sparse setting, i.e., when the expected degree of a node is O (1) in the number of nodes, in the fixed-dimensional asymptotic regime, i.e., the dimension of the feature data is fixed while the number of nodes is large. Such graphs are typically known to be locally tree-like. We introduce a notion of Bayes optimality for node classification tasks, called asymptotic local Bayes optimality, and compute the optimal classifier according to this criterion for a fairly general statistical data model with arbitrary distributions of the node features and edge connectivity. The optimal classifier is implementable using a message-passing graph neural network architecture. We then compute the generalization error of this classifier and compare its performance against existing learning methods theoretically on a well-studied statistical model with naturally identifiable signal-to-noise ratios (SNRs) in the data. We find that the optimal message-passing architecture interpolates between a standard MLP in the regime of low graph signal and a typical convolution in the regime of high graph signal. Furthermore, we prove a corresponding non-asymptotic result.

1 Introduction

Graph Neural Networks (GNNs) have rapidly emerged as a powerful tool for learning on graphstructured data, where along with features of the entities, there also exists a relational structure among them. They have found numerous applications to a wide range of domains such as social networks [Backstrom and Leskovec, 2011], recommendation systems [Ying et al., 2018, Hao et al., 2020], chip design [Mirhoseini et al., 2021], bioinformatics [Scarselli et al., 2009, Zhang et al., 2021], computer vision [Monti et al., 2017], quantum chemistry [Gilmer et al., 2017], statistical physics [Battaglia et al., 2016, Bapst et al., 2020], and financial forensics [Zhang et al., 2017, Weber et al., 2019]. Most of the success with these applications has been possible due to the advent of the

message-passing paradigm in GNNs, however, designing optimal GNN architectures for such a wide variety of applications still remains a challenging task.

In this work, we are interested in the node classification problem on very sparse feature-decorated graphs that are locally tree-like. We focus on the regime where the dimension of the node features is fixed and the number of nodes is large. Our motivation for considering this regime is that many major benchmark datasets for node classification appear to scale in this fashion. For example, in the popular Open Graph Benchmark collection [Hu et al., 2020], the medium and large-scale node-property prediction datasets have roughly 10 6 nodes ( ogbn-products , ogbn-mag ) to about 10 8 nodes ( ogbn-papers100M ), each with roughly 10 2 features. Graphs with such properties exist naturally in social, informational and biological networks; for motivational examples, see Stelzl et al. [2005], Adcock et al. [2013]. We present a precise definition of optimality for node classification tasks on locally tree-like graphs in this scaling regime and compute the optimal classifier according to this definition, for a multi-class statistical data model where node features can have arbitrary continuous or discrete distributions. Subsequently, we show that a message-passing GNN architecture is able to realize the optimal classifier. Furthermore, we provide a theoretical analysis, comparing the generalization error of the optimal classifier with other architectures like GCN and simple MLPs. Our results support a recent work [Veliˇ ckovi´ c, 2022] in the context of classification on sparse graphs. In particular, we show that when node features are accompanied by sparse graphical side information, message-passing graph neural networks are able to realize the optimal classification scheme, and as such, there does not exist a better architecture beyond the message-passing paradigm.

Related Work. There has been a tremendous amount of work on GNN architecture design, where the most popular designs are based on a convolutional architecture, with each layer of the neural network performing a weighted convolution (averaging) operation with immediate neighbours, e.g., graph convolutional networks (GCN) [Kipf and Welling, 2017, Chen et al., 2020] or graph attention networks (GAT) [Veliˇ ckovi´ c et al., 2018]. These architectures are known to have several limitations regarding their expressive power (see, for e.g., Li et al. [2018], Oono and Suzuki [2020], Balcilar et al. [2021], Xu et al. [2021], Keriven [2022]).

An interesting line of research consists of both theoretical and empirical works that attempt to address these limitations by developing an understanding of GNN architectures within the scope of message-passing [Rong et al., 2020, Liu et al., 2022, Maskey et al., 2022], as well as beyond it [Maron et al., 2019, Murphy et al., 2019, Chen et al., 2019]. For example, Xu et al. [2018] propose an architecture with a technique called skip-connections, that flexibly leverages different ranges of neighbourhoods for each node to enable structure-awareness in node representations, Chen et al. [2020] propose a modification of the vanilla GCN with an initial residual that effectively relieves the problem of oversmoothing [Oono and Suzuki, 2020], and Keriven et al. [2021] study the universality of structural GNNs in the large random graph limit. However, this area of research still lacks a clear understanding of optimality in the context of graph learning problems, making it hard to design architectures for which a well-defined notion of optimality can be theoretically justified.

Several works have studied traditional message-passing GNN architectures like GCN and GAT using the binary contextual stochastic block model, see for example, Baranwal et al. [2021], Chien et al. [2022], Fountoulakis et al. [2022a,b], Javaloy et al. [2022], Baranwal et al. [2023]. These analyses rely heavily on two assumptions: first, the graph is not too sparse, i.e., for a graph with n nodes, the expected degree of a node is Ω n (log 2 n/n ) , and second, the node features are modelled as a Gaussian mixture. The work by Wei et al. [2022] is of particular interest to us, where the authors take a Bayesian inference perspective to investigate the functions of non-linearity in GNNs for binary node classification. They characterize the max-a-posterior estimation of a node label given the features of itself and its immediate neighbours. A similar perspective to that of Wei et al. [2022] is discussed in Gosch et al. [2023], where the latter authors derive insights into the robustness-accuracy trade-off in GNNs for node classification. In contrast to these inspiring works, we study the highly sparse regime where the expected degree of a node is O n (1) and consider nodes beyond the immediate neighbours, at any fixed distance. (In fact, our non-asymptotic results allow distances of order c log n for small enough c > 0 , see Section 3.5 below.) Furthermore, our main result holds for a general multi-class statistical model with arbitrary continuous or discrete feature distributions and arbitrary edge-connectivity probabilities between all pairs of classes.

Our Contributions. In this paper, we use a multi-class statistical model with arbitrary node features and edge-connectivity profiles among all pairs of classes to study the node classification problem in the regime where the graph component of the data is very sparse, i.e., the expected degree is O (1) . The data model is described in Section 3.2. We state the following main results and findings:

We introduce a family of graph neural network architectures that are asymptotically (in the number of nodes n → ∞ ) Bayes optimal in a local sense for a general multi-class data model with arbitrary feature distributions. The optimality is stated precisely in Theorem 1.

We analyze the architecture in the simpler two-class setting with Gaussian features, explicitly characterizing the generalization error in terms of the natural signal-to-noise ratio (SNR) in the data, and perform a comparative study against other learning methods analyzed using the same statistical model (Theorems 2 and 3). We find two key insights:

When the graph SNR is very low, the architecture reduces to a simple MLP that does not consider the graph, while if it is very high, our architecture reduces to a typical convolutional network that averages information from all nodes in the local neighbourhood. In the regime between the low and high SNRs, the architecture interpolates and performs better than both a simple MLP and a typical GCN.

If the information in the graph is larger than a threshold, then a simple convolution is able to perform better than all methods that do not utilize the graph. Not surprisingly, this threshold aligns with the Kesten-Stigum weak-recovery threshold for community detection in sparse networks [Massoulié, 2014, Mossel et al., 2018].

In the non-asymptotic setting with a fixed number of nodes, we show that even for a logarithmic depth, the neighbourhoods of an overwhelming fraction of nodes are treelike with high probability. Subsequently, we show that the optimal classifier in the nonasymptotic setting obtains an error that is close to that incurred by the optimal classifier in the asymptotic setting. This is formalized in Theorem 4.

Let us end this section by reiterating that we work in the fixed-dimensional regime. It is natural to wonder as to the performance of this architecture as compared to optimal algorithms in the highdimensional setting. We present a numerical comparison of our work to the AMP-BP algorithm from Deshpande et al. [2018] in low and high-dimensional settings in Appendix B.

2 Architecture

This section explains the design of our GNN architecture for node classification. We perform two modifications to existing message-passing architectures. First, we decouple the layers in the neural network from the neighbourhood radius in the message-passing framework. This style of decoupled architecture has previously been studied, see for example, Nikolentzos et al. [2020], Feng et al. [2022], Baranwal et al. [2023]. Second, we introduce a learnable parameter that models edge connectivity between each pair of classes and helps construct the messages to propagate. In the following, for any matrix M , we denote row i of M by M i, : and column i of M by M : ,i .

Before stating our architecture, we need the following additional notation and pre-processing. Let ℓ ≥ 0 and L > 0 be fixed integers. Let C ≥ 2 be the number of classes. For given data ( A , X ) where A ∈ R n × n is the adjacency matrix of an unweighted undirected graph, and X ∈ R n × d is the node feature matrix, we perform a pre-computation on the graph to construct a tensor ˜ A as follows:

where f ( M ) for a matrix M returns the entry-wise flattened matrix with f ( M ) ij = 1 ( M ij > 0) , and ( ∧ , ¬ ) denote the entry-wise bit-wise operators ('and', 'negation') respectively. Note here that ˜ A ( k ) is an n × n binary matrix with ˜ A ( k ) uv = 1 if and only if v is present in the distance k neighbourhood of u but not within the distance ( k -1) neighbourhood. The idea behind this pre-processing step is the following: for each node u and each k ∈ [ ℓ ] , we want to divide the radius ℓ neighbourhood of u into ℓ groups of nodes, where each group k ∈ [ ℓ ] consists of nodes that are within discovered the neighbourhood at each distance from a given node. ˜ A ( k ) u, : models a non-backtracking walk of length k that considers new nodes in the distancek neighbourhood that were not discovered.

We can now define the graph neural network architecture as follows.

Architecture 1. Given input data ( A , X ) where A ∈ { 0 , 1 } n × n is the adjacency matrix and X ∈ R n × d is the node feature matrix, define:

Then the predicted label is given by ˆ y = { ˆ y u } u ∈ [ n ] , where

Let us pause here to comment on the interpretation of the terms arising in this architecture. Here, H ( L ) is viewed as the output of a simple L -layer MLP with { σ l } l ∈ [ L ] being a set of non-linear functions. We have ( W ( l ) , b ( l ) ) l ∈ [ L ] as the learnable parameters of this MLP, with suitable dimensions so that H ( L ) ∈ R n × C . In addition, we introduce the learnable parameter Z ∈ R C × C which is used to model edge connectivity among all pairs of classes. The quantity ˜ A ( k ) u, : M ( k ) : ,i = ∑ v ∈ [ n ] ˜ A ( k ) u,v M ( k ) v,i is viewed as the sum of messages M ( k ) v,i passed by all distance k neighbours of node u .

Although Architecture 1 follows the style of convolutional architectures like GCN and GAT for collecting messages within a local neighbourhood, the novelty lies in the construction of the messages M . Intuitively, Q = sigmoid( Z ) learns the probabilities of edge connectivity between all pairs of classes so that Q k models the probability of observing a distance k path between a pair of nodes in two classes. To predict the label of node u , the messages from other nodes v are constructed based on their features X v , their distance k from node u , and the path probabilities Q k . We show in Theorem 1 that this architecture is in a sense (made precise in Definition 3.2) universally optimal among all node-classification schemes for sparse graphs. Our result thus aligns with the observations in Veliˇ ckovi´ c [2022], showing that optimal neural network architectures for node classification on sparse graphs are implementable using the message-passing paradigm.

3 Theoretical Analysis and Discussion

In this section, we present a theoretical analysis of the message-passing GNN given in Architecture 1. We begin by defining a natural notion of optimality in our setting and show that among local learning methods on graphs, Architecture 1 is optimal according to this definition on a very general statistical model. We then compute the generalization error and compare the architecture to other well-studied methods like a simple MLP and a GCN.

3.1 Asymptotic Local Bayes Optimality

For classification tasks, it is natural to use a notion of generalization error in a 'per sample' or online sense. Without graphical side information, the natural choice is the Bayes risk. With graphical information, however, there is an important obstruction: the number of samples is equal to the size of the corresponding graph. As such, a naive extension of the Bayes risk does not have this property.

A natural approach would be to consider the Bayes risk for estimators that take in the node, the data set, and the graph, i.e., ˆ y v = ˆ y ( v, ( X,G )) . In this case, however, the risk necessarily implicitly depends on the sample size, n , through G . One might try to remove this dependence by taking the infinite sample size limit, but for a class of estimators this general, it is not clear that such a limit is well defined. To circumvent this issue, we restrict attention to node classifiers that are only allowed 'local' information around the node. The large graph limit of the generalization error is then naturally interpreted via local weak convergence . (For the convenience of the reader, we briefly recall the notion of local weak convergence of sparse graphs in Appendix A.1. See also Ramanan [2021, Chapter 1] or Bordenave [2016, Section 3] for more detailed expositions.) In this limit, one can then interpret the generalization error as a per-sample error for the randomly rooted graph ( G,u ) where u is a uniform random vertex in V ( G ) . (Here and in the following a rooted graph is a pair of a graph G

and a distinguished vertex, u , called the root .) With these observations, we are led to a natural notion of Bayes optimality, namely asymptotic local Bayes optimality which we define presently. 1

Before turning to this definition, we must first recall the notion of ℓ -local classifiers . For a node v in a graph G , let η k ( v ) = { u ∈ V ( G ) : d ( u, v ) ≤ k } denote the ball of radius k for the canonical graph distance metric.

Definition 3.1 ( ℓ -local classifier) . Let G = ( A , X ) be a feature-decorated graph of n vertices with d -dimensional features X u for each vertex u . For a fixed radius ℓ > 0 , an ℓ -local node-classifier is a function f that takes as input a root vertex u ∈ [ n ] , the adjacency matrix A and the features of all nodes within the ℓ -neighbourhood of u , i.e., { X v } v ∈ η ℓ ( u ) ) , and outputs a classification label for u .

Suppose now that we have a sequence of (random) feature decorated graphs ( X n , G n ) with | V ( G ) | = n . Let u n denote a uniform at random vertex in G n . Suppose finally that the rooted feature-decorated graphs ( X n , G n , u n ) locally weakly converge to ( X,G,u ) . We can then define the notion of asymptotically ℓ -locally Bayes optimal classifiers for this sequence of problems.

Definition 3.2. We say that a classifier h ∗ ℓ ∈ C ℓ is asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence { ( X n , G n , u n ) } if it minimizes the probability of misclassification of the root of the local weak limit, ( X,G,u ) , over the class C ℓ , i.e.,

̸

Before turning to our data model, we note here that the reader may ask whether or not the asymptotically ℓ -locally Bayes optimal classifier is in any sense the limit of optimal ℓ -local classifier of the random root, u n . We show this in an appropriate sense in Theorem 4.

3.2 Data Model

Let us now turn to the data model that we use for our analysis. We work with the general multi-class contextual stochastic block model (CSBM) where each node belongs to one of C different classes labelled 1 , . . . , C , and the node features have arbitrary continuous or discrete distributions. This model with C = 2 , along with a specialization to Gaussian features has been extensively studied in several works on (semi)-supervised node classification and unsupervised community detection, see, for example, Deshpande et al. [2018], Lu and Sen [2020], Baranwal et al. [2021], Wei et al. [2022], Fountoulakis et al. [2022b], Baranwal et al. [2023]. Informally, a CSBM consists of a coupling of a stochastic block model (SBM) [Holland et al., 1983] with a mixture model where the components of the mixture have arbitrary distributions and are associated with the blocks of the SBM.

More formally, let n, d be positive integers such that n denotes the number of nodes and d denotes the dimension of the node features. Define y 1 , . . . , y n ∈ { 1 , . . . , C } as the latent variables (class labels) to be inferred. We will assume that the latent variables have a uniform prior, i.e., y u ∼ Unif( { [ C ] } ) for all u . For the relational part of the data, we have an undirected unweighted graph of n nodes, G = ( V, E ) with adjacency matrix A = ( a uv ) u,v ∈ [ n ] ∼ SBM( n, Q ) , where Q = { q ij } ∈ [0 , 1] C × C is the edge-probability matrix, meaning that

The node attributes, X ∈ R n × d are sampled from a mixture of C arbitrary continuous or discrete distributions, P = { P i } i ∈ [ C ] , where corresponding to the y u , we have X u ∼ P y u for all u ∈ [ n ] .

We will view n as large and study the setting where d is fixed (does not grow with n ). We note here that in previous related works [Baranwal et al., 2021, Wei et al., 2022, Baranwal et al., 2023], crucial assumptions have been made about the distribution of the node features and the sparsity of the graph, i.e., q ij = Ω n (log 2 n/n ) . In contrast, we work in the extremely sparse setting where q ij = b ij /n for constants b ij > 1 , so we write Q = B /n where B = { b ij } i,j ∈ [ C ] . Furthermore, the only assumption we need about the distributions P i is that P i are absolutely continuous with respect to some base measure, in which case their densities exist, denoted by ρ i . For ease of reading, we encourage the

reader to consider the case where P i are continuous or discrete, therefore, the base measure is simply the Lebesgue measure on R or the counting measure on Z respectively.

For a feature-decorated graph G = ( A , X ) = ( { a uv } u,v ∈ [ n ] , { X u } u ∈ n ) sampled from the model described above, we say that G ∼ CSBM( n, d, P , Q ) or G ∼ CSBM( n, d, P , B /n ) .

3.3 Optimal Classifier

We are now ready to state our first main result that characterizes the asymptotically ℓ -locally Bayes optimal classifier on the CSBM data described in Section 3.2.

Theorem 1 (Bayes optimal message-passing) . For any ℓ ≥ 1 , the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , Q ) is

where { ρ i } i ∈ [ C ] are the densities associated with the distributions P i ∈ P , and

Let us briefly discuss the meaning of Theorem 1. It states that universally among all ℓ -local classifiers, h ∗ ℓ is asymptotically Bayes optimal for the sparse CSBM data. We view M ik ( X v ) as the message gathered from node v that is distance k away from node u . In particular, M ik ( X v ) naturally maximizes the likelihood of observing node v in class j at distance k from node u in class i , over all j ∈ [ C ] . Furthermore, this optimal classifier is realizable using Architecture 1 (see for example, Lu et al. [2017, Theorem 1], where it is shown that any Lebesgue measurable function can be approximated arbitrarily closely by standard neural networks). Consequently, this result shows that in the sparse setting, the message-passing paradigm can realize the optimal node classification scheme irrespective of the distributions of the node features or the inter-class edge probabilities.

For an intuitive understanding of Theorem 1, it helps to consider two extreme cases. First, if Q = p I for some p ∈ [0 , 1] , then the classifier reduces to a simple convolution, h ∗ ℓ ( u ) = argmax i ∈ [ C ] { ∑ v ∈ η k ( u ) log ρ i ( X v ) } . Second, if Q = p 11 ⊤ , then q ij = p for all i, j ∈ [ C ] , meaning that the graph component of the data is Erdös-Rényi, and hence, completely uninformative for the purposes of node classification. In this case, the classifier reduces to h ∗ ℓ ( u ) = argmax i ∈ [ C ] { log ρ i ( X u ) } , i.e., it is optimal to look at only the features of node u to predict its label since the neighbourhood does not provide any meaningful information. We formalize this intuition later for a simpler case (see Theorem 3).

3.4 Comparative Study

̸

In this section, we perform a theoretical analysis of the classifier in Theorem 1 using a well-studied specialization of the CSBM data model described in Section 3.2. For ease of discussion, let us restrict ourselves to the setting where there are two classes. Formally, we have C = 2 , and without loss of generality, the class labels y u ∈ {± 1 } for all u ∈ [ n ] . The distributions of the node features are given by X u ∼ P y u with corresponding density ρ y u . Furthermore, Q = { q ij } is a 2 × 2 matrix with q ii = p = a/n and q ij = q = b/n with constants a > 1 , b ≥ 0 for classes i = j . For a data sample G = ( A , X ) from this model, we write G ∼ CSBM( n, d, { P ± } , Q ) or G ∼ CSBM( n, d, { P ± } , a n , b n ) . We also recognize the quantity associated with the signal-to-noise ratio (SNR) in the graph structure for this case, which is given by

Note that the quantity Γ has been recognized as the meaningful SNR in several related works where the underlying random graph model is the binary symmetric stochastic block model, for example, Baranwal et al. [2021], Fountoulakis et al. [2022b], Wei et al. [2022], Baranwal et al. [2023].

Let us now state Theorem 1 in the case of two classes. For given input x ∈ R and c > 0 , let φ ( x, c ) = min(max( x, -c ) , c ) denote the value of x clipped between the range [ -c, c ] .

Corollary 1.1 (Optimal classifier for binary symmetric CSBM) . For any ℓ ≥ 1 , the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , a n , b n ) is

In this simplified setting, we note that the messages propagated from nodes in the ℓ -local neighbourhood of node u are clipped proportional to a function of their distance k from node u . In particular, the clip threshold c k can be expressed in terms of the graph SNR Γ from (1). It is interesting to observe in Corollary 1.1 that c k decreases rapidly as k increases. Since Γ < 1 , this means that to predict the label of node u , the value of the message propagated from node v at distance k from u decreases exponentially in k .

The above simplification helps us interpret the classifier in terms of the graph SNR Γ . We will now impose an assumption on the distribution of node features. This will help us analyze the generalization error in terms of the SNR in both the features and the graph, and enable us to compare the performance with other learning methods that are well-studied in the same statistical settings. We will resort to the setting where the features of the CSBM follow a Gaussian mixture. Note that this specialized statistical model has been studied extensively in previous works for benchmarking existing GNN architectures, see for example, Baranwal et al. [2021], Fountoulakis et al. [2022b], Wei et al. [2022], Baranwal et al. [2023].

In principle, one could compute the generalization error of h ∗ ℓ for arbitrary distributions on the node features (see Appendix A.3.2), however, we report the error for Gaussian features for expository reasons. The generalization error is defined for a classifier h to be the probability of disagreement between the true label y u and the output of the classifier h u for node u . We characterize the error for h ∗ ℓ in the case where P -, P + correspond to the Gaussian mixture with components N ( -µ , σ 2 I ) and N ( µ , σ 2 I ) for fixed µ ∈ R d and σ > 0 2 .

In this case, a notion of the signal-to-noise ratio of the features naturally exists, i.e., γ = ∥ µ ∥ 2 /σ , a quantity proportional to the ratio of the distance between the means of the mixture and the standard deviation. The log-likelihood ratio in this setting is ψ ( x ) = log ρ + ( x ) ρ -( x ) = 2 σ 2 ⟨ x , µ ⟩ .

Consider a sequence { ( G n , u n ) } n ≥ 1 with G n = ( V ( G n ) , E ( G n )) from this model where u n ∼ Unif( V ( G n )) . In this setting, in the absence of features, it is known that ( G n , u n ) converges locally weakly to a Poisson Galton-Watson tree (see for example, Mossel et al. [2015, Section 4]). Here, for every node, we additionally have features that are independent of the graph, and hence, as a straightforward consequence of Mossel et al. [2015, Section 4], ( G n , u n ) in our case converges to a feature-decorated Poisson Galton-Watson tree ( G,u ) .

For the root node u , let α k and β k denote the number of children at generation k in class y u and -y u respectively, where y u denotes the label of node u . Then { α k } k ≥ 0 and { β k } k ≥ 0 are characterized by

α

0

= 1

, β

0

= 0

,

For a classifier h acting on G , let E ( h ) denote the probability of misclassification of the root u in G , i.e., E ( h ) = Pr ( h u y u < 0) . Correspondingly, in the case of finite n , we denote by E n ( h ) the probability of misclassification of a uniform random node u n in G n . We are now ready to state the generalization error of h ∗ ℓ .

Theorem 2 (Generalization error) . For any ℓ ≥ 1 , the generalization error of the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , Q ) with Gaussian features is given by

where α k , β k are as in (2) , Z ( a ) k,i = φ ( -2 γ 2 +2 γg k,i , c k ) , Z ( b ) k,i = φ (2 γ 2 +2 γg k,i , c k ) , and g, { g k,i } are mutually independent standard Gaussian random variables.

Let us now understand how the error described in Theorem 2 behaves in terms of the two SNRs γ (for the features), and Γ (for the graph). Note that E ( h ∗ ℓ ) → 0 as γ →∞ , and E ( h ∗ ℓ ) → 1 / 2 as γ → 0 . This means that if the signal in the features is large, the number of mistakes made by the classifier vanishes, while if the signal is very small, then roughly half of the nodes are misclassified (equivalent to making a uniform random guess for each node).

To see how Γ affects the error, we begin by looking at two extreme settings: first, where the graph is complete noise, i.e., Γ = 0 , and second, where the graph signal is very strong, i.e., Γ → 1 , followed by a discussion on how h ∗ ℓ interpolates between these extremes. Let ϕ ± denote the Gaussian density functions with means ± µ and variance σ 2 I d . Define the random variable

where α k , β k follow (2). In the following, we denote the vanilla GCN classifier from Kipf and Welling [2017] by h gcn . We then have the following result.

Theorem 3 (Extreme graph signals) . Let h ∗ ℓ be the classifier from Corollary 1.1, h ∗ 0 ( u ) = sgn( ⟨ X u , µ ⟩ ) be the Bayes optimal classifier given only the feature information of the root node u , and h gcn be the one-layer vanilla GCN classifier. Then we have that for any fixed ℓ :

If Γ = 0 then E ( h ∗ ℓ ) = E ( h ∗ 0 ) = Φ( -γ ) , where Φ is the standard Gaussian CDF.

If Γ → 1 then ξ ℓ ≥ 1 a.s. and E ( h ∗ ℓ ) → Pr ( g > γξ ℓ ) , where g ∼ N (0 , 1) .

E ( h gcn ) = Pr ( g > γξ 1 ) .

Theorem 3 shows that in the regime of extremely low graph SNR, the optimal classifier h ∗ ℓ reduces to a linear classifier h ∗ 0 ( u ) = sgn( ⟨ X u , µ ⟩ ) , which can be realized by a simple MLP that does not use the graph component of the data at all. On the other hand, in the regime of extremely strong graph SNR, h ∗ ℓ reduces to a simple convolution over all nodes in the ℓ -neighbourhood and is comparable to a typical GCN. Furthermore, we note that in the strong graph SNR regime Γ → 1 , E ( h ∗ ℓ ) → Pr ( g > γξ ℓ ) ≤ Φ( -γ ) since ξ ℓ ≥ 1 . The clip operation during the propagation of messages makes things interesting between these two extremes, where h ∗ ℓ interpolates between a simple MLP and an ℓ -hop convolutional network. This interpolation is characterized by the graph signal Γ , since the messages are clipped in the range [ -c k , c k ] , where c k = log ( 1+Γ k 1 -Γ k ) .

In addition, Theorem 3 concludes that if ξ 1 ( a, b ) > 1 , then a GCN can perform better than every classifier that does not see the graph. On the other hand, if ξ 1 ( a, b ) < 1 , then a GCN incurs more errors on the data than the best methods that do not use the graph. Interestingly, but not surprisingly, this result aligns with the Kesten-Stigum weak recovery threshold for the community-detection problem on the sparse stochastic block model [Massoulié, 2014, Mossel et al., 2018], meaning that if weak recovery is possible on the graph component of the data, then a GCN is able to exploit it to perform better than methods that do not use the graph, e.g., a simple MLP.

We now demonstrate our results through experiments using pytorch and pytorch-geometric [Fey and Lenssen, 2019]. The following simulations are for the setting n = 10000 and d = 4 for binary classification on the CSBM. We implement Architecture 1 for the binary case, and perform full-batch training on a graph sampled from the CSBM with certain signals (mentioned in the figures), followed by an evaluation of the architecture on a new graph sampled from the same distribution.

In Fig. 1, we show that the accuracy obtained by the optimal classifier is higher than both a simple MLP and a vanilla GCN [Kipf and Welling, 2017]. We plot the test accuracy of Architecture 1 against the SNR in the node features, γ = ∥ µ ∥ /σ in Fig. 1a, and against the graph SNR Γ = | a -b | / ( a + b ) in Fig. 1b. We fix Γ = 0 . 42 and γ = 1 for the two plots, respectively. We chose these specific values because they generate relatively clearer plots where the accuracy metrics for the three architectures are easily visible and distinguished from each other. The results are similar for other values for Γ and γ , i.e., the Bayes optimal architecture is superior to both MLP and GCN.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

GMM signal

γ

0.6

0.7

0.8

0.9

1.0

MLP

GCN

(a) Varying

γ

with fixed

Γ = 0

.

42

.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Graph signal Γ

0.6

0.7

0.8

0.9

1.0

MLP

GCN

(b) Varying

Γ

with fixed

γ

= 1

.

Furthermore, Fig. 2 shows that as claimed in Theorem 3, when the graph signal is at the extremes, i.e., Γ = 0 and Γ = 1 , Architecture 1 behaves like a simple MLP and performs a typical convolution (averaging) over all nodes in the ℓ -neighbourhood, respectively. In the regime of poor graph SNR, i.e., Γ = 0 , a GCN is worse than a simple MLP, as inferred from part three of Theorem 3.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

GMM signal

γ

0.5

0.6

0.7

0.8

0.9

1.0

MLP

GCN

(a) Fixed graph signal

Γ = 0

.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

GMM signal

γ

0.6

0.7

0.8

0.9

1.0

MLP

GCN

(b) Fixed graph signal

Γ = 1

.

Finally, we observe that for the binary setting when the parameters of the architecture are initialized uniformly at random, gradient descent converges and the neural network learns the right parameters such that Architecture 1 realizes the optimal classifier in Corollary 1.1. This convergence, along with a comparison of our architecture to the Approximate Message-passing Belief Propagation (AMP-BP) algorithm from Deshpande et al. [2018] is presented in Appendix B.

3.5 Non-asymptotic Setting

We now turn to the non-asymptotic regime and argue that for fixed n , the classifier in Corollary 1.1 is still in a formal sense, Bayes optimal for an overwhelming fraction of nodes. We begin by exploiting the fact that for up to logarithmic depth neighbourhoods, a sparse CSBM graph is tree-like.

Proposition 3.1 (Tree neighbourhoods) . Let G = ( V, E ) ∼ CSBM( n, d, P , a n , b n ) for constants a, b > 1 . Then for any ℓ = c log n such that c log(( a + b ) / 2) < 1 / 4 , with probability 1 -O (1 / log 2 n ) , the number of nodes u ∈ V whose ℓ -neighbourhood is cycle-free is n (1 -o (log 4 ( n ) / √ n )) .

In particular, Proposition 3.1 states that for ℓ = c log n for a suitable constant c , the ℓ -neighbourhood of an overwhelming fraction of nodes is a tree. This implies that the classifier h ∗ ℓ is Bayes optimal for roughly all of the nodes. Moreover, since the diameter of a sparse graph (as in our setting) is O (log n ) almost surely [Chung and Lu, 2001, Theorem 6], any learning mechanism can only look as far as O (log n ) -hops away from a node to gather new information. This shows that for such graphs, GNNs that are not very deep and look at only up to logarithmic distance in the neighbourhood are sufficient.

Let us now turn to the misclassification error in the non-asymptotic setting. Recall that for a classifier h ∈ C ℓ , we denote by E n ( h ) and E ( h ) the misclassification error of h on the data model with n nodes,

and on the limiting data model with n →∞ , respectively. Furthermore, recall from Corollary 1.1 that min h ∈C ℓ E ( h ) = E ( h ∗ ℓ ) . Our next result shows that the optimal misclassification error in the nonasymptotic setting across all ℓ -local classifiers, i.e., min h ∈C ℓ E n ( h ) , is close to the misclassification error obtained in the non-asymptotic setting by h ∗ ℓ . Moreover, min h ∈C ℓ E n ( h ) is also close to E ( h ∗ ℓ ) which is explicitly computed in Theorem 2.

Theorem 4 (Misclassification error for fixed n ) . For any 1 ≤ ℓ ≤ c log n such that the positive constant c satisfies c log( a + b 2 ) < 1 / 4 , we have that

Recall that Corollary 1.1 implies that h ∗ ℓ performs optimally on the limiting data model (asymptotic setting) among the class of ℓ -local classifiers C ℓ , but it may not be optimal for the non-asymptotic data model where we have a finite feature-decorated graph with n nodes. However, Theorem 4 helps us conclude that even in the non-asymptotic setting, h ∗ ℓ performs almost as well as the actual optimal classifier among C ℓ in this case, as long as we compare with classifiers that can only look at moderate logarithmic depths in the local neighbourhood, i.e., ℓ ≤ c log n for a suitable c .

4 Conclusion and Future Work

In this work, we present a comprehensive theoretical characterization of the Bayes optimal node classification architecture for sparse feature-decorated graphs and show that it can be realized using the message-passing framework. Utilizing a well-established and well-studied statistical model, we interpret its performance in terms of the SNR in the data and validate our findings through empirical analysis of synthetic data. Additionally, we identify the following limitations as prospects for future work: (1) We consider neighbourhoods up to distance ℓ = c log n for a small enough c . Extending ℓ to the graph's diameter (known to be O (log n ) with high probability) by removing the restriction on c poses challenges due to the presence of cycles. (2) More insights can be provided through experiments on real data to benchmark the architecture in cases where we have a significant gap between the theoretical assumptions (sparse and locally tree-like graph) and the real-world data.

Acknowledgments and Disclosure of Funding

A.J. acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada Research Chairs programme. Cette recherche a été enterprise grâce, en partie, au soutien financier du Conseil de Recherches en Sciences Naturelles et en Génie du Canada (CRSNG), [RGPIN-2020-04597, DGECR-2020-00199], et du Programme des chaires de recherche du Canada.

K. Fountoulakis would like to acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Cette recherche a été financée par le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG), [RGPIN-2019-04067, DGECR-2019-00147].

References

B. Adcock, B. D. Sullivan, and M. W. Mahoney. Tree-like structure in large social and information networks. In 2013 IEEE 13th International Conference on Data Mining , pages 1-10, 2013. doi: 10.1109/ICDM.2013.77.

Backstrom and J. Leskovec. Supervised random walks: predicting and recommending links in social networks. In Proceedings of the fourth ACM international conference on Web search and data mining , pages 635-644, 2011.

Balcilar, G. Renton, P. Héroux, B. Gaüzère, S. Adam, and P. Honeine. Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective. In International Conference on Learning Representations , 2021.

Bapst, T. Keck, A. Grabska-Barwi´ nska, C. Donner, E. D. Cubuk, S. S. Schoenholz, A. Obika, A. W. Nelson, T. Back, D. Hassabis, et al. Unveiling the predictive power of static structure in glassy systems. Nature Physics , 16(4):448-454, 2020.

Baranwal, K. Fountoulakis, and A. Jagannath. Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proc. of Mach. Learn. Res. , pages 684-693. PMLR, 18-24 Jul 2021.

Baranwal, K. Fountoulakis, and A. Jagannath. Effects of Graph Convolutions in Multi-layer Networks. In The Eleventh International Conference on Learning Representations , 2023.

D. Barbour and L. H. Y. Chen. An introduction to Stein's method , volume 4. World Scientific, 2005.

Battaglia, R. Pascanu, M. Lai, D. J. Rezende, and K. Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Bordenave. Lecture notes on random graphs and probabilistic combinatorial optimization, 2016.

H. Y. Chen. Poisson Approximation for Dependent Trials. The Annals of Probability , 3(3):534 545, 1975.

Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li. Simple and Deep Graph Convolutional Networks. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1725-1735. PMLR, 13-18 Jul 2020.

Chen, S. Villar, L. Chen, and J. Bruna. On the equivalence between graph isomorphism testing and function approximation with GNNs. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Chien, W.-C. Chang, C.-J. Hsieh, H.-F. Yu, J. Zhang, O. Milenkovic, and I. S. Dhillon. Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. In International Conference on Learning Representations , 2022.

Chung and L. Lu. The diameter of sparse random graphs. Advances in Applied Mathematics , 26 (4):257-279, 2001.

Deshpande, S. Sen, A. Montanari, and E. Mossel. Contextual Stochastic Block Models. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Dreier, P. Kuinke, B. Le Xuan, et al. Local Structure Theorems for Erdos-Rényi Graphs and Their Algorithmic Applications. SOFSEM 2018: Theory and Practice of Computer Science LNCS 10706 , page 125, 2018.

Feng, Y. Chen, F. Li, A. Sarkar, and M. Zhang. How powerful are k-hop message passing graph neural networks. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Fey and J. E. Lenssen. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.

Fountoulakis, D. He, S. Lattanzi, B. Perozzi, A. Tsitsulin, and S. Yang. On classification thresholds for graph attention with edge features. arXiv preprint arXiv:2210.10014 , 2022a.

Fountoulakis, A. Levi, S. Yang, A. Baranwal, and A. Jagannath. Graph Attention Retrospective. arXiv preprint arXiv:2202.13060 , 2022b.

Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning , 2017.

Gosch, D. Sturm, S. Geisler, and S. Günnemann. Revisiting robustness in graph machine learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=h1o7Ry9Zctm .

Hao, T. Zhao, J. Li, X. L. Dong, C. Faloutsos, Y. Sun, and W. Wang. P-companion: A principled framework for diversified complementary product recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 2517-2524, 2020.

W. Holland, K. B. Laskey, and S. Leinhardt. Stochastic blockmodels: First steps. Social networks , 5(2):109-137, 1983.

Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In Advances in Neural Information Processing Systems , 2020.

Javaloy, P. S. Martin, A. Levi, and I. Valera. Learnable graph convolutional attention networks. In Has it Trained Yet? NeurIPS 2022 Workshop , 2022.

Keriven. Not too little, not too much: a theoretical analysis of graph (over)smoothing. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Keriven, A. Bietti, and S. Vaiter. On the universality of graph neural networks on large random graphs. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6960-6971. Curran Associates, Inc., 2021.

N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Li, Z. Han, and X.-M. Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence , 2018.

Liu, S. Jing, T. Zhao, Z. Huang, and D. Wu. Enhancing Multi-hop Connectivity for Graph Convolutional Networks. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , 2022.

Lu and S. Sen. Contextual Stochastic Block Model: Sharp Thresholds and Contiguity. ArXiv , 2020. arXiv:2011.09841.

Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and equivariant graph networks. In International Conference on Learning Representations , 2019.

Maskey, R. Levie, Y. Lee, and G. Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Massoulié. Community Detection Thresholds and the Weak Ramanujan Property. In Proceedings of the Forty-Sixth Annual ACM Symposium on Theory of Computing , page 694-703, 2014.

Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, A. Nazi, et al. A graph placement methodology for fast chip design. Nature , 594(7862): 207-212, 2021.

Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Mossel, J. Neeman, and A. Sly. Reconstruction and estimation in the planted partition model. Probability Theory and Related Fields , 162:431-461, 2015.

Mossel, J. Neeman, and A. Sly. A proof of the block model threshold conjecture. Combinatorica , 38(3):665-708, 2018.

Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational pooling for graph representations. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4663-4673. PMLR, 09-15 Jun 2019.

Nikolentzos, G. Dasoulas, and M. Vazirgiannis. k-hop graph neural networks. Neural Networks , 130:195-205, 2020.

Oono and T. Suzuki. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. In International Conference on Learning Representations , 2020.

Ramanan. CRM-PIMS Summer School 2021: Background Material For Mini Course on Asymptotics of Interacting Stochastic Processes on Sparse Graphs, 2021.

Rong, W. Huang, T. Xu, and J. Huang. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In International Conference on Learning Representations , 2020.

Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. IEEE Transactions on Neural Networks , 20(1), 2009.

Stein. A bound for the error in the normal approximation to the distribution of a sum of dependent random variables. In Proc. Sixth Berkeley Symp. Math. Stat. Prob. , pages 583-602, 1972.

Stelzl, U. Worm, M. Lalowski, C. Haenig, F. H. Brembeck, H. Goehler, M. Stroedicke, M. Zenkner, A. Schoenherr, S. Koeppen, et al. A human protein-protein interaction network: a resource for annotating the proteome. Cell , 122(6):957-968, 2005.

Veliˇ ckovi´ c. Message passing all the way up. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning , 2022.

Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks. In International Conference on Learning Representations , 2018.

Veliˇ ckovi´ c. Message passing all the way up, 2022.

Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science , volume 47. Cambridge University Press, 2018.

Weber, G. Domeniconi, J. Chen, D. K. I. Weidele, C. Bellei, T. Robinson, and C. E. Leiserson. Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics. arXiv preprint arXiv:1908.02591 , 2019.

Wei, H. Yin, J. Jia, A. R. Benson, and P. Li. Understanding Non-linearity in Graph Neural Networks from the Bayesian-Inference Perspective. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka. Representation Learning on Graphs with Jumping Knowledge Networks. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 5453-5462. PMLR, 10-15 Jul 2018.

Xu, M. Zhang, J. Li, S. S. Du, K.-I. Kawarabayashi, and S. Jegelka. How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks. In International Conference on Learning Representations , 2021.

Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. KDD '18: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 974-983, 2018.

Zhang, D. Zhou, M. Y. Yildirim, S. Alcorn, J. He, H. Davulcu, and H. Tong. Hidden: hierarchical dense subgraph detection with application to financial fraud detection. In Proceedings of the 2017 SIAM International Conference on Data Mining , pages 570-578. SIAM, 2017.

X.-M. Zhang, L. Liang, L. Liu, and M.-J. Tang. Graph neural networks and their current applications in bioinformatics. Frontiers in genetics , 12:690049, 2021.

A Supplementary Discussion and Proofs

A.1 Local Weak Convergence

We briefly recall here the notion of local weak convergence of random rooted graphs. The notion of local weak convergence of random, feature-decorated, rooted graphs is defined analogously.

Let us begin first with the case of rooted graphs. A rooted graph ( G,u ) = (( E,V ) , u ) is a graph G with a distinguished vertex u called the root. We say that two rooted graphs ( G 1 , u 1 ) = (( E 1 , V 1 ) , u 1 ) , ( G 2 , u 2 ) = (( E 2 , V 2 ) , u 2 ) are isomorphic if there is a bijection ϕ : V 1 → V 2 such that ϕ ( u ) = v and such that if ( x, y ) ∈ E 1 then ( ϕ ( x ) , ϕ ( y )) ∈ E 2 . In this case we write ( G 1 , u 1 ) ∼ = ( G 2 , u 2 ) . For a rooted graph ( G,u ) , we denote its isomorphism class as [( G,u )] .

Let G ∗ denote the set of isomorphism classes of (locally finite) rooted graphs. For a vertex v ∈ G we let η k ( v ) denote the collection of neighbours of v of distance at most k in the canonical edge distance metric, and G [ η k ( v )] denote the subgraph induced on this collection of vertices. We then have the notion of local convergence in G ∗ .

Definition A.1 (Local convergence on G ∗ ) . We say that a sequence [( G n , u n )] ∈ G ∗ converges locally to [( G,u )] ∈ G ∗ if for each k > 0 , we have that G [ η k ( u n )] ∼ = G [ η k ( u )] eventually.

It can be shown [Bordenave, 2016, Lemma 3.4] that G ∗ equipped with the topology of local convergence is a Polish space. We are then in the position to define local weak convergence on G ∗ . In brief, the topology of local weak convergence of random graphs is the topology of weak convergence of measures on the space of probability measures on G ∗ , namely M 1 ( G ∗ ) .

Definition A.2 (local weak convergence of rooted graphs) . A sequence { [ G n , u n ] } of random rooted graphs with corresponding laws { µ n } ⊆ M 1 ( G ∗ ) is said to locally weakly converge to a random rooted graph [ G,u ] with law µ ∈ M 1 ( G ∗ ) if µ n → µ weakly.

We note here that it is common also to talk about the notion of local weak convergence of a sequence of (finite) random graphs G n . In this case G n → G locally weakly if [( G n , u n )] → [( G,u )] locally weakly where u n ∼ Unif( V ( G n )) .

A.2 Preliminary Results

In this section, we state two important preliminary results and a fact that are used to establish our results in the paper.

Lemma A.1. [Dreier et al., 2018, Lemma 3]. Let G ∼ CSBM( n, d, { P -, P + } , a n , b n ) and r be a fixed constant. Then the probability that there exists an r -neighbourhood in G with m more edges than the number of vertices is bounded as follows:

Lemma A.2. [Massoulié, 2014, Lemma 4.2]. Assume ℓ = c log( n ) with c log ∆ < 1 / 4 , where ∆ = ( a + b ) / 2 . Then with high probability, no node i has more than one cycle in its ℓ -neighbourhood. Moreover, for any m > 0 , with probability at least 1 -O (1 /m 2 ) the number of nodes i whose ℓ -neighbourhood contains at least one cycle is bounded by O ( m log 3 ( n )∆ 2 ℓ ) .

Fact A.3. For any non-negative u, v, x, y such that x, y ≤ 1 ,

A.3 Bayes Optimal Classifier

In this section, we compute the asymptotically ℓ -locally Bayes optimal classifier for the general CSBM described in Section 3.2 and establish Theorem 1, followed by a proof of Corollary 1.1. Next, we compute the generalization error for the two-class case with arbitrary node features.

A.3.1 Computing the Classifier

For the proofs, we introduce the notation N k ( u ) for a given graph to mean the set of vertices that are at a distance of exactly k from node u in the graph. Thus, η k ( u ) = { u } ∪ k j =1 N k ( u ) .

Theorem (Restatement of Theorem 1) . For any ℓ ≥ 1 , the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , Q ) is

where { ρ i } i ∈ [ C ] are the densities associated with the distributions P i ∈ P , and

Furthermore, there exists a choice of parameters for Architecture 1 such that it realizes h ∗ ℓ .

Proof. We begin by writing the MAP estimation for this problem. Note that the features X v ∈ R d for every node v ∈ [ n ] follow the law P i if y v = i ∈ [ C ] . In addition, recall that we have the edge-probability matrix Q = { q ij } i,j ∈ [ C ] with Pr (( u, v ) is an edge | y u = i, y v = j ) = q ij , where q ij = b ij /n for absolute constants b ij . Then we can write the likelihood of the ℓ -neighbourhood of node u as the joint function:

In the above, Q k ij denotes the i, j -th entry of the matrix Q k and quantifies the probability that a node in class j is at a distance k from a node in class i ; while P { y v } v ∈ η ℓ ( u ) denotes the prior distribution of the node labels, which by our assumption is uniform. Let us now compute the MAP estimator.

where M ik ( X v ) = log( ρ j ( X v )) + log( Q k ij ) . Furthermore, note that an instance of Architecture 1 with L = 1 , σ 1 = { ρ i } i ∈ [ C ] , and Q for the edge-probabilities realizes the function h ∗ ℓ for a given root node u and its ℓ -neighbourhood η ℓ ( u ) , in the sense that ˆ y u = h ∗ ℓ ( u, { X v } v ∈ η ℓ ( u ) ) .

Next, we obtain a simpler version of the classifier for the two-class symmetric CSBM with an arbitrary distribution of node features. Recall that ψ is the log of the likelihood ratio.

Corollary (Restatement of Corollary 1.1) . For any ℓ ≥ 1 , the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , a n , b n ) is

where M k ( x ) = sgn( a -b ) · φ ( ψ ( x ) , c k ) with c k = log ( 1+Γ k 1 -Γ k ) , and ψ ( x ) = log ρ + ( x ) ρ -( x ) .

Proof. The proof follows directly from Theorem 1, by taking C = 2 . In this two-class case, the features X v ∈ R d for every node follow the law P y v where the class labels y v ∈ {∓ 1 } are Unif( {-1 , +1 } ) . In addition, we have

̸

where p = a/n and q = b/n for absolute constants a > 1 , b ≥ 0 . Define the quantities p k , q k as follows for k ∈ [ ℓ ] :

Then we can simplify the likelihood of the ℓ -neighbourhood of node u from (4) as follows:

Then maximizing the likelihood over possible class labels, we have

where M k ( x ) = log(max( p k ρ + ( x ) , q k ρ -( x ))) -log(max( p k ρ -( x ) , q k ρ + ( x ))) . Next, we observe that for any w,x,y, z ∈ R ,

Hence, M k ( X v ) is simply sgn( a -b ) · φ (log ρ + ( X v ) ρ -( X v ) , c k ) , i.e., the signed likelihood ratio clipped between -c k and c k .

A.3.2 Generalization Error

Let us now compute the generalization error of h ∗ ℓ . Formally, given a data instance ( u, { X v } v ∈ η ℓ ( u ) ) along with the neighbourhood η ℓ ( u ) , h ∗ ℓ outputs a label ˆ y u ∈ {± 1 } , and the generalization error is defined as the probability Pr ( y u ˆ y u < 1) . For a simple calculation, let us assume that the latent labels y i are uniformly distributed, i.e., Pr ( y i = -1) = Pr ( y i = 1) = 1 2 . It is straightforward to generalize to unbalanced settings. Recall that the features in classes ± 1 follow the law P ± , and denote by the log likelihood ratio by ψ ( x ) = log( ρ + ( x ) /ρ -( x )) . Then we have that for a fixed u ,

where Z (1) k = ∑ j ∈ [ α k ] M k ( Y (1) k,j ) + ∑ j ∈ [ β k ] M k ( Y (2) k,j ) and Z (2) k = ∑ j ∈ [ α k ] M k ( Y (2) k,j ) + ∑ j ∈ [ β k ] M k ( Y (1) k,j ) are independent random variables with Y (1) k,j ∼ P -and Y (2) k,j ∼ P + .

The above expression is not particularly insightful. For this reason, we now specialize to the case of Gaussian features and interpret the error in terms of the natural SNRs associated with the Gaussian mixture and the graph, i.e., the quantities γ and Γ .

A.4 Specialization to Gaussian Features

In this section, we look at the specialized setting where the node features are sampled from a symmetric binary Gaussian mixture model. Let us begin with the generalization error in this case.

A.4.1 Generalization Error

Theorem (Restatement of Theorem 2) . For any ℓ ≥ 1 , the generalization error of the asymptotically ℓ -locally Bayes optimal classifier of the root for the sequence ( G n , u n ) ∼ CSBM( n, d, P , Q ) with Gaussian features is given by

where α k , β k are as in (2) , Z ( a ) k,i = φ ( -2 γ 2 +2 γg k,i , c k ) , Z ( b ) k,i = φ (2 γ 2 +2 γg k,i , c k ) , and g, { g k,i } are mutually independent standard Gaussian random variables.

Proof. For the Gaussian mixture, the log of the likelihood ratio for a node u is given by 2 σ 2 ⟨ X u , µ ⟩ D = 2 y u γ 2 +2 γg , where g ∼ N (0 , 1) . Replacing every { X i } i ∈ [ n ] as X i = E X i + σ g i = y i µ + σ g i , we obtain the expression in Theorem 2.

A.4.2 Extreme Graph SNRs

Let us now turn to the next result, where we analyze the generalization error in the cases where the graph SNR Γ takes extreme values.

Theorem (Restatement of Theorem 3) . Let h ∗ ℓ be the classifier from Corollary 1.1, h ∗ 0 ( u ) = sgn( ⟨ X u , µ ⟩ ) be the Bayes optimal classifier given only the feature information of the root node u , and h gcn be the one-layer vanilla GCN classifier. Then we have that for any fixed ℓ :

If Γ = 0 then E ( h ∗ ℓ ) = E ( h ∗ 0 ) = Φ( -γ ) , where Φ is the standard Gaussian CDF.

Proof. Note that when Γ = 0 , i.e., when a = b , then a k = b k for all k ∈ [ ℓ ] , hence, c k = log( a k /b k ) = 0 . This implies that all information from the k -hop neighbours is truncated to 0 for all k ∈ [ ℓ ] . Thus, the classifier reduces to h ∗ u = h ∗ ℓ ( u, { X v } v ∈ η ℓ ( u ) ) = sgn( ψ ( X u )) = sgn ( g + y u γ ) = h ∗ 0 ( u, { X u } ) . Thus, the probability that h ∗ u y u < 0 is

where Φ( · ) is the standard Gaussian CDF.

For the other case where Γ → 1 , we have two sub-cases: Either a → 0 with b = 0 , or a = 0 with b → 0 . In this case, c k →∞ for all k , so the classifier takes the form

̸

̸

Hence, the probability of making a mistake is

where η ( a ) ℓ , η ( b ) ℓ denote the total number of nodes in the ℓ -neighbourhood η ℓ ( u ) that are in the same class as u and different class as u , respectively. The last equation is obtained by using the fact that ( g, { g k,v } ) are i.i.d. standard Gaussians. Note that in this case since either b → 0 or a → 0 (but not both), we have η ( b ) ℓ → 0 or η ( a ) ℓ → 0 using (2) for any fixed ℓ . Thus, ξ ℓ ( a, b ) > 1 a.s. Following a similar analysis, one can find that E ( h gcn ) = Pr ( g > γ · ξ 1 ( a, b )) .

It is interesting to note that we may not have ξ 1 ( a, b ) > 1 in general, meaning that a GCN is better than methods that do not use a graph only in the case where ξ 1 ( a, b ) > 1 .

A.5 Non-asymptotic Analysis

First, consider the case where ℓ , the total depth of the neighbourhood is a constant independent of n , the number of nodes.

Putting m = 1 in Lemma A.1, we see that the probability is bounded by (8 ℓ 2 ( a + b )) 4 ℓ +1 /n = O (1 /n ) . Hence, we conclude that in the limit n → ∞ , there are no cycles in any constant-depth neighbourhoods in the graph. In particular, we obtain that the local weak limit ( G,u ) is a tree.

We now turn to the case where the depth of the neighbourhood is logarithmic in n .

Proposition (Restatement of Proposition 3.1) . Let G = ( V, E ) ∼ CSBM( n, d, P , a n , b n ) . Then for any ℓ = c log n such that c log( a + b 2 ) < 1 / 4 , with probability 1 -O (1 / log 2 n ) , the number of nodes u ∈ V whose ℓ -neighbourhood is cycle-free is n ( 1 -o ( log 4 ( n ) √ n ) ) .

Proof. In Lemma A.2, observe that since c log ∆ < 1 / 4 , we have ℓ < log n 4 log ∆ = 1 4 log ∆ n . Thus, putting m = log n , we find that with probability at least 1 -O (1 / log 2 n ) , the number of nodes whose ℓ -neighbourhood contains at least one cycle is bounded by O (log 4 ( n )∆ 2 ℓ ) = o (log 4 ( n ) √ n ) . Hence, the fraction of nodes whose ℓ -neighbourhood is cycle-free is 1 -o ( log 4 n √ n ) .

For a fixed node u ∈ [ n ] , let us denote the number of nodes at distance k (respectively ≤ k ) from u with class label ± y u by U ± k ( u ) (respectively, U ± ≤ k ( u ) ). Also let n ± denote the number of nodes with class label ± y u , so that n = n + + n -. Note that U + 0 ( u ) = 1 , U -0 ( u ) = 0 , and conditionally on the sigma-field F k -1 = σ ( U ± t ( u ) , t ≤ k -1) , we have

Define S k ( u ) = U + k ( u ) + U -k ( u ) to be the number of nodes at distance exactly k from u , and denote ∆ = a + b 2 to be the expected degree of a node. Correspondingly, recall from (2) that we have

Let us now state a useful high-probability bound on S k ( u ) = U + k ( u ) + U -k ( u ) .

Lemma A.4. [Massoulié, 2014, Theorem 2.3]. For any ℓ = c log n with c log ∆ < 1 / 4 , there exist constants C, ϵ > 0 such that with probability at least 1 -O ( n -ϵ ) , S k ( u ) ≤ C ∆ k log( n ) for all u ∈ [ n ] and all k ∈ [ ℓ ] .

We now obtain a total variation bound between the sequences { U ± k } k ≥ 0 and { α k , β k } k ≥ 0 .

Lemma A.5. Let u ∈ [ n ] be fixed with label y u ∈ {± 1 } . Let ℓ = c log n with c log ∆ < 1 / 4 . Then the total variation distance between the collections of variables { U + k ( u ) , U -k ( u ) } k ≤ ℓ and { α k ( u ) , β k ( u ) } k ≤ ℓ is bounded by O (log 3 n/n 1 / 4 ) .

Proof. Define the following events for C as in Lemma A.4:

Conditionally on the sigma-field F k -1 = σ ( U ± t ( u ) , t ≤ k -1) and the event Ω k -1 , we compute the total variation distance between the variables ( U + k ( u ) , U -k ( u )) and ( α k ( u ) , β k ( u )) . Since u is fixed, we omit it from the notation for brevity. Define the following random variables:

We now apply the Stein-Chen method to bound d TV ( U ± k , W ± k ) . For more details on this technique, we refer to Stein [1972], Chen [1975], Barbour and Chen [2005]. In particular, we use the fact that for X 1 ∼ Bin( n, λ/n ) , X 2 ∼ Poi( λ ) and X 3 ∼ Poi( λ ′ ) , d TV ( X 1 , X 2 ) ≤ λ/n and d TV ( X 2 , X 3 ) ≤ | λ -λ ′ | . Let us focus on d TV ( U + k , W + k ) as the other case for d TV ( U -k , W -k ) is similar. Construct an intermediate random variable based on the distributions of U ± k as in Eqs. (6) and (7),

Denote T t = 1 -( 1 -a n ) U + t ( 1 -b n ) U -t for brevity. Note that using triangle inequality,

where in the last inequality we used Fact A.3. Then we obtain the variation distance:

Consider now a choice of c such that c log ∆ < 1 / 4 . We have ℓ = c log n < 1 4 log ∆ n , implying that ∆ ℓ ≤ n 1 / 4 . Recalling (9) corresponding to Lemma A.4, we have that under the event Ω k -1 for k ≤ ℓ , the number of nodes at distance k -1 is

Observe now that from Fact A.3, T k -1 ≤ aU + k -1 + bU -k -1 n . Recalling that y u have a uniform prior, by the Chernoff bound [Vershynin, 2018, Theorem 2.3.1] on n + , we have | n + -n 2 | = O ( √ n log n ) with probability at least 1 -1 / poly( n ) . Thus, we obtain that under this event,

where in the last step we used the bound from (10). Now recall that the variables { U ± k , α k , β k } k ∈ [ ℓ ] are defined as in Eqs. (6) to (8) for all k ≤ ℓ . For a fixed u ∈ [ n ] , we have the base cases U + 0 = α 0 = 1

and U -0 = β 0 = 0 . Then following an induction argument with a union bound over all k ∈ { 1 , . . . , ℓ } , we have that the variation distance between the sequences { U + k , U -k } k ≤ ℓ and { α k , β k } k ≤ ℓ is upper bounded by O ( log 3 n n 1 / 4 ) .

We now obtain a relationship between the misclassification error on the data model with finite n , i.e., E n and the error on the limit of the model with n →∞ , i.e., E .

Theorem (Restatement of Theorem 4) . For any 1 ≤ ℓ ≤ c log n such that the positive constant c satisfies c log( a + b 2 ) < 1 / 4 , we have that

Proof. Consider a random feature-decorated graph G n ∼ CSBM( n, d, { P ± } , a/n, b/n ) , where P ± correspond to the distributions N ( ± µ , σ 2 I ) for the node features given by { X u } u ∈ [ n ] . For a classifier h ∈ C ℓ , the class of all ℓ -local classifiers, define E n ( h ) to be the probability of misclassification for a uniform at random node u ∈ [ n ] , i.e., E n ( h ) = Pr ( y u · h ( u, { X v } v ∈ η ℓ ( u ) , η ℓ ( u )) < 0) . Since it is known that all classifiers in C ℓ operate on u given the information in its ℓ -neighbourhood η ℓ ( u ) , we will omit η ℓ ( u ) from the notation and say h ( u ) instead of h ( u, { X v } v ∈ η ℓ ( u ) , η ℓ ( u )) when it is understood. Let P be the joint measure of the variables { U ± k } k ≤ ℓ from Eqs. (6) and (7), and P ′ be the joint measure of the variables { α k , β k } k ≤ ℓ from Eq. (8). Then Lemma A.5 gives us that d TV ( P , P ′ ) ≤ O ( log 3 n n 1 / 4 ) = o n (1) .

Recall E ( h ∗ ℓ ) computed in Theorem 2 for the limiting data model ( G,u ) .

Similarly, we have

Thus, we obtain that

Let us now focus on the case with finite n . Let A denote the event from Proposition 3.1 where the number of nodes with cycle-free ℓ -neighbourhoods is 1 -o ( log 4 n √ n ) . For a node u ∈ G n , let E u denote the event that the subgraph induced by the ℓ -neighbourhood of u , η ℓ ( u ) is a tree. Then observe that for a uniform random node u ∈ G n ,

In the above, we used from Proposition 3.1 that Pr ( E u ) = Pr ( E u ∩ A ) + Pr ( E u ∩ A c ) = 1 -O ( 1 log 2 n ) , and that E n ( h ∗ ℓ ) = min h ∈C ℓ Pr ( y u h ( u ) < 0 | E u ) . This establishes the first part:

Combining the above display with (11), we obtain the second part, i.e.,

B Additional Empirical Observations

B.1 Convergence of Parameters

We observe empirically that gradient descent ( SGD and Adam implementations in the pytorch library) converges in the binary setting. In this case, the neural network learns the right parameters corresponding to the parameters of the CSBM, i.e., ρ and Q , such that Architecture 1 realizes the optimal classifier in Corollary 1.1. In Fig. 3, the x-axis denotes the number of epochs elapsed since

0

100

200

300

400

500

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

⟨

μ

,

θ

1⟩

0

100

200

300

400

500

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

|

log

( 1+Γ

1-Γ

) -

θ

2|

Epochs

the beginning of the training process. For the first plot, the y-axis denotes the cosine similarity between the parameters θ 1 ∈ R d learned by the MLP H ( L ) in Architecture 1 and the ansatz µ that realizes the optimal classifier; while in the second plot, the y-axis denotes the absolute difference between the clip parameter θ 2 ∈ R and the ansatz value log( 1+Γ 1 -Γ ) . These experiments are performed in the same setting as Fig. 1b with fixed Γ = 0 . 2 . We see that the parameters converge as the number of training iterations increases. The reported metrics are averaged over 10 trials, and the standard deviation is shown at each iteration using the translucent blue region.

B.2 Comparison with AMP-BP

We now perform a comparison of our architecture with ℓ = 5 against the AMP-BP algorithm from [Deshpande et al., 2018] in two different settings. We set n = 1000 and work with two values of d ∈ { 5 , 500 } . The case d = 5 simulates the low-dimensional case where asymptotically d/n → 0 , while d = 500 represents the high-dimensional case where d/n → c for a constant c . For the AMP-BP algorithm, we choose two values for the number of iterations, t ∈ { 5 , 20 } .

Tables 1 and 2 show the results for γ = 1 with varying values of Γ ∈ { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } . We observe that the classifier obtained after training Architecture 1 almost always outperforms AMP-BP for both low-dimensional and high-dimensional cases.

Tables 3 and 4 show the results for γ = 0 . 2 , i.e., for a weaker feature signal in the data. Here, we observe that although Architecture 1 outperforms AMP-BP in the low-dimensional regime, it exhibits worse performance than AMP-BP in the high-dimensional regime.

It is important to note, however, that this is an apples-to-oranges comparison because AMP-BP is not a local algorithm, i.e., the whole graph is visible to the algorithm and all nodes contribute to the classification of every other node. This is not true for Architecture 1, where only the nodes in the ℓ -hop neighbourhood contribute to this decision. Our notion of optimality is among the class of local algorithms. Furthermore, we observe that AMP-BP with 5 iterations does not converge and obtains a much lower accuracy compared to 20 iterations.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a specific graph convolutional architecture (and the resultant estimators from an architecture with appropriate learned values) \nbased in the setting of a contextual stochastic block model: one where a graph with latent clustering also has node features available that \nare informative for discovering the clustering. The paper studies this in the \"constant average degree and constant feature dimension\" regime\nand makes the following contributions:\n\n1. Identifies the Bayes-optimal classifier for node clustering restricted to a constant size neighborhood in the graph (using local\nweak convergence theory) and shows this matches the GCN architecture proposed.\n2. Specializes the classifier to Gaussian features and 2 latent clusters, proving a formula for the bayes optimal classifier\n3. Shows that the local classifier is nearly optimal even for GCN architectures of $c \\log n$ (for a small enough constant c)\nusing the locally-tree-like nature of the graph.\n\nSTRENGTHS:\nThe main strengths are:\n\n1. Identifies a setting in which specific GCN architectures are provably optimal\n2. Uses local weak convergence theory to show some simple but illustrative examples of how (local) Bayes-optimal classifiers\naggregate information across features/graph structure.\n\nWEAKNESSES:\nIt is hard to get excited about this paper. The main weakness to me is that the Bayes optimal classifier for the tree model is essentially obvious, given Bayes rule. It is far from clear that GCN architectures uninformatively initialized would converge to this, and the paper's experiments do not show this (or in fact consider it from what I could tell). They instead rely on the fact that the architecture they propose can approximate (with some setting of weights) the Bayes optimal classifier. At this point, it is well-known that the success of NN architectures cannot be explained by such existence/approximation theorems.\n\nIf the experiments carried out show that naive initializations of the architectures converge to the optimal ones, then that is a priori surprising, and should be discussed more. I might even pivot the paper to start with that as an experimental finding, since the theoretical results are relatively unsurprising.\n\nQUESTIONS:\nSome questions, roughly in order of the paper reading:\n1. Why is the notion of SNR $\\frac{(a-b)}{a+b}$ when in prior notions of SBM in the constant degree regime it is usually $\\frac{(a-b)^2}{2(a+b)}$ which makes sense in terms of the difference of mean / sd of poisson degrees (which is correct in the limit)? If not, how does this reconcile with prior work when features are not meaningful?\n2. What happens to Corollary 1.1. when $\\Gamma > 1$ (particularly the formula for c_k should be modified somewhat). Naively (using the same formula but with $\\Gamma^k -1$), this means that c_k converges to a constant as $k$ diverges. Intuitively, it should be the case that the features should matter less and less as the neighborhood diverges in this regime, irrespective of the snr, which is the behavior for $\\Gamma <1$. This suggests that the formula needs more than a naive change for $\\Gamma >1$.\n3. Same question as 1 on formula (3) and Theorem 3 (the specialization to the Gaussian case)\n4. Theorem 3: the statement $\\xi_\\ell \\ge 1$ looks like it needs to be probabilistic statement (almost surely?, with high probability?) or perhaps needs vanishing slack? E.g. take $a$ large, $\\ell = 1$ and $b=1$, then there's a constant (but small as a diverges) probability that $\\alpha_1$ is 0 and $\\beta_1 \\leq 2$ is not large so that $\\xi_1 <1$. \n5. Does the Bayes optimal classifier know the probabilities $Q$ and the densiites? Presumably this is necessary. \n6. What about the GCN in Fig 1? Does it learn it from scratch, or initialized at true values, etc?\n7. In figure 1, presumably the (roughly) 0.75 accuracy is basically $\\Phi(-1)$ at $\\Gamma = 0$?\n\nLIMITATIONS:\nThe most important weakness is highlighted in the weaknesses.\n\nSecond, it is also not obvious how one would change this for dense graphs. Analogous results (though with *very* different proof techniques) might be expected\nto hold when graphs have degrees of order $n$ but with within/out cluster probabilities differing by order $1/\\sqrt{n}$. How would one identify GCN architectures that might work there?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the inference problem of node classification in feature-decorated locally tree-like sparse graphs. The authors introduce and motivate a notion of asymptotic local Bayes optimality for local estimators. The first main result, Theorem 1, concerns the characterisation of the Bayes optimal (in the former sense) classifier in the setting considered. This theorem implies that this estimator is implementable in a message passing graph neural network architecture. The authors then consider the case in which the features are Gaussian. In this context, Theorem 2 establishes the asymptotic local Bayes optimal error while Theorem 3 compares this optimal classifier with other standard classifiers in some extreme signal-to-noise settings. These results are contrasted with numerical experiments. Finally, Theorem 4 gives a finite size bound for the difference between the misclassification error of the estimator proposed and the minimum misclassification error of local classifiers.\n\nSTRENGTHS:\n- The questions addressed in the paper are relevant, interesting, and are part of an active research field.\n- The results are interesting, rigorous, and thorough and the proofs are clear.\n- The inclusion of finite size bounds for the difference between the performance of the classifier proposed and the optimal performance is of particular interest and renders the analysis presented very complete.\n\nWEAKNESSES:\n- Maybe the most relevant weakness of the work is, in my opinion, that it solely focuses in locally tree-like graphs. Although this kind of models are very extended in the literature, the topologies present in many of the applications mentioned are not expected to be of this kind. Indeed, in many social settings, for example, the kind of networks that are observed have diverse clustering coefficient values. Though the paper undoubtedly has theoretical value, this will probably limit the impact it could have in more applied research communities.\n\nQUESTIONS:\nGeneral comments:\n- The analytical tools used require the graphs to be locally tree-like and extending the analysis to non locally tree-like graphs is clearly beyond the scope of the work. But it is my opinion that it would make the case for the paper much stronger if the authors could add some numerical simulations comparing the classifier proposed and other estimators in some non locally tree-like settings. This would address to some extent the comment on the Weaknesses' section above.\n- The analysis presented focuses mainly in the case of two classes. This is understandable as it renders the presentation more clear. However, it would be good if the authors could add a brief discussion on how the complexity of the estimator depends with respect to c. Is it feasible to compute it for reasonable graph sizes and a moderately large number of classes?\n\nSome particular comments:\n- In line 270 after the dot the next word should be capitalised.\n- In Figure 1(a), is the value of $\\Gamma$ used above the transition? I guess it is but it would be good if this is explicitly stated.\n- The convergence rate of Theorem 4 is rather slow. It would thus be specially interesting to have some estimation of the constant for this bound. Do the authors think that this would be possible by some adaptations of the proof?\n\nLIMITATIONS:\nAlthough the limitations of the work are not explicitly stated, the context and reach of the results are clear in the presentation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a local analysis of the CSBM. They define and derive a local Bayes-optimal classifier and show it can be achieved by a message-passing architecture. They give an interpretation of it in the limiting cases when the graph caries no or all information and support it with numerical experiments.\n\nSTRENGTHS:\nThe topic of this article is of great interest. CSBM has been widely used as an artificial dataset and deriving an optimal GNN for it would be good. For this reason I think this article should be published.\n\nWEAKNESSES:\n1 In the CSBM usually one considers the high-dimensional limit where the dimension d of the features proportional to n, to model what happens in ML. Here the authors consider d=O(1), which seems limited. I do not see if this is just for convenience or if it is a stronger limitation to this work. For the high-dimensional Gaussian mixture Bayes-optimal classifiers are known and the MLP the authors use works. Could the authors better explain this point?\n\n2 A weakness of this article is a lack of clarity:\n\n– the analysis the authors give is local, the graph is tree-like. For l ~ O(log n) it will not hold. I was a bit confused; maybe the authors should emphasize that l is fixed compared to n, that there are no loops and many things are independent;\n\n– in the definition of architecture 1, the authors should precise that the l and L of the first line (for H^(l)) have nothing to do with the calligraphic l of the second line (the size of the neighborhood). L. 121 \"as a simple MLP\" –> \"as the output of a simple L-layer MLP\";\n\n– maybe introducing the model before theorem 1 is counter-intuitive. The authors could explain that the model is a way to realize theorem 1, that the NN just learns rho and Q. This would ease the introduction of tilde A;\n\n– the experiments of fig. 1 and 2 are not clear. What are the training procedure, l, L, training nodes, number of runs, ...\n\n3 It is a pity that the numerical experiments deal with the interpolation and not the major point of the article: the optimality. The authors could compare against other GNNs. Also one has access to the conjecturally-optimal performances on CSBM (in Deshpande '18 for instance); how far is this l-neighborhood model? does the limit large l converge to these or is there a gap (due to the cycles)?\n\n4 How does this model deal with train labels? In architecture 1 the authors assume there is no train node in the l-neighborhood of u, no? otherwise the Bayes-optimal classification would take the labels in account, M_{u,i} = (H_{u,j=label of u} + log Q_{i,j=label of u}). The prior distribution of node labels is not iid uniform in the semi-supervised setting.\n\nI would be pleased to give a higher rating to this article if the authors improve or comment on these points. If the authors develop them in their revised manuscript maybe they could summarize part 3.5.\n\nQUESTIONS:\nSome more general questions:\n\n1 As to the training: can the performances of the trained model be compared to theorem 2? do the learned Q and rho match the ones of the binary CSBM?\n\n2 Does this neural network generalize well to other datasets?\n\n3 Would the authors have an idea how to generalize their results to non-local large l estimators?\n\nTypos :\n\nl. 150 \"for a class of estimators this general,\" ?\n\nref. l. 409 Andrea Montanari is missing.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe focus of this research paper was the investigation of node classification problem on sparsely populated graphs that exhibit local tree-like structures. Thy introduced asymptotic local Bayes optimality to define the ideal performance standard for node classification tasks. By utilizing this criterion, the paper derived the optimal classifier for a wide range of statistical data models with diverse distributions of node features and edge connectivity. The research paper further assessed the generalization error of this classifier and conducted a theoretical comparison of its performance with existing learning methods. This evaluation was carried out on a model that inherently possesses identifiable signal-to-noise ratios (SNRs) in the data.\n\nSTRENGTHS:\n* Existing works explored conventional message-passing graph neural network architectures. However, their analyses heavily depend on two key assumptions: firstly, the graph is not excessively sparse, and secondly, the node features are represented as a Gaussian mixture. In contrast, this study investigates the realm of highly sparse graphs, where the expected degree of a node is on the order of O(1). Moreover, it considers nodes that extend beyond the immediate neighbors, encompassing nodes at fixed distances.\n\n* Their result holds for a general multi-class statistical model with arbitrary continuous or discrete feature distributions and arbitrary edge-connectivity probabilities between all pairs of classes.\n\n* They showed that in scenarios where the graph signal-to-noise ratio (SNR) is extremely low, the architecture simplifies to a basic MLP that disregards the underlying graph structure. Conversely, when the SNR is significantly high, their architecture transforms into a standard convolutional network that aggregates information from all nodes within the local neighborhood. However, in the intermediate SNR regime, it exhibits interpolation behavior and outperforms both the simple MLP and the typical graph convolutional network (GCN).\n\nWEAKNESSES:\n* The analysis in the paper seems interesting but more experimental results are required to support the claims. For example, comaprison with the baselines on existing benchmarks in GCN literature as well as more synthetic types of graphs (e.g scale-free, random etc) with controllable degree distribution to enforce different level of sparsity.\n\n\n* The proposed method relies on the pre-processing step to calculator A^\\~(k) that models a non-backtracking walk of length k that considers new nodes in the distance-k neighborhood that were not discovered. Specifically A\\~(k)uv=1 If and only if v is present in the distance k neighborhood of u but not within the distance (k-1) neighborhood. Such a calculation seems to be expensive and might not be scalable. Would you please elaborate on that? \n\n\n* Q^k models the probability of observing a distance k path between a pair of nodes in two classes. I am wondering why the log (Q^k) is considered in calculation of M. What is the intuition? What would happen if the Q^k has been used? Having abolition study would help in understanding.\n\n* How is the performance of proposed approach in compare to the traditional GCN baselines in real datasets? The papers just considered synthetic graphs with controllable degree but more experimental results are needed on exiting benchmarks. Also for the synthetic graph what type of graph is it and what is the underlying distribution?\n\nQUESTIONS:\n* Analysis of the cost associated with the preprocessing step and discussion on scalability of the proposed approach\n\n* What is impact of log(Q^k) vs Q^k on the performance?\n\nPlease look at the weakness for more detailed question.\n\nLIMITATIONS:\nThe paper discussed the limitation of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}