SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let us briefly discuss the meaning of Theorem 1. It states that universally among all  $\ell$ -local classifiers,  $h_\ell^*$  is asymptotically Bayes optimal for the sparse CSBM data. We view  $\mathcal{M}_{ik}(\mathbf{X}_v)$  as the message gathered from node  $v$  that is distance  $k$  away from node  $u$ . In particular,  $\mathcal{M}_{ik}(\mathbf{X}_v)$  naturally maximizes the likelihood of observing node  $v$  in class  $j$  at distance  $k$  from node  $u$  in class  $i$ , over all  $j \in [C]$ . Furthermore, this optimal classifier is realizable using Architecture 1 (see for example, Lu et al. [2017, Theorem 1], where it is shown that any Lebesgue measurable function can be approximated arbitrarily closely by standard neural networks). Consequently, this result shows that in the sparse setting, the message-passing paradigm can realize the optimal node classification scheme irrespective of the distributions of the node features or the inter-class edge probabilities.
For an intuitive understanding of Theorem 1, it helps to consider two extreme cases. First, if  $\mathbf{Q} = p\mathbf{I}$  for some  $p \in [0,1]$ , then the classifier reduces to a simple convolution,  $h_{\ell}^{*}(u) = \operatorname{argmax}_{i \in [C]} \left\{\sum_{v \in \eta_k(u)} \log \rho_i(\mathbf{X}_v)\right\}$ . Second, if  $\mathbf{Q} = p\mathbf{1}\mathbf{1}^{\top}$ , then  $q_{ij} = p$  for all  $i,j \in [C]$ , meaning that the graph component of the data is Erdős-Rényi, and hence, completely uninformative for the purposes of node classification. In this case, the classifier reduces to  $h_{\ell}^{*}(u) = \operatorname{argmax}_{i \in [C]} \left\{\log \rho_i(\mathbf{X}_u)\right\}$ , i.e., it is optimal to look at only the features of node  $u$  to predict its label since the neighbourhood does not provide any meaningful information. We formalize this intuition later for a simpler case (see Theorem 3).
3.4 Comparative Study
In this section, we perform a theoretical analysis of the classifier in Theorem 1 using a well-studied specialization of the CSBM data model described in Section 3.2. For ease of discussion, let us restrict ourselves to the setting where there are two classes. Formally, we have  $C = 2$ , and without loss of generality, the class labels  $y_{u} \in \{\pm 1\}$  for all  $u \in [n]$ . The distributions of the node features are given by  $\mathbf{X}_u \sim \mathbb{P}_{y_u}$  with corresponding density  $\rho_{y_u}$ . Furthermore,  $\mathbf{Q} = \{q_{ij}\}$  is a  $2 \times 2$  matrix with  $q_{ii} = p = a / n$  and  $q_{ij} = q = b / n$  with constants  $a > 1, b \geq 0$  for classes  $i \neq j$ . For a data sample  $G = (\mathbf{A}, \mathbf{X})$  from this model, we write  $G \sim \mathrm{CSBM}(n, d, \{\mathbb{P}_{\pm}\}, \mathbf{Q})$  or  $G \sim \mathrm{CSBM}(n, d, \{\mathbb{P}_{\pm}\}, \frac{a}{n}, \frac{b}{n})$ . We also recognize the quantity associated with the signal-to-noise ratio (SNR) in the graph structure for this case, which is given by
$$
\Gamma = \frac {| p - q |}{p + q} = \frac {| a - b |}{a + b}. \tag {1}
$$
Note that the quantity  $\Gamma$  has been recognized as the meaningful SNR in several related works where the underlying random graph model is the binary symmetric stochastic block model, for example, Baranwal et al. [2021], Fountoulakis et al. [2022b], Wei et al. [2022], Baranwal et al. [2023].
Let us now state Theorem 1 in the case of two classes. For given input  $x \in \mathbb{R}$  and  $c > 0$ , let  $\varphi(x, c) = \min(\max(x, -c), c)$  denote the value of  $x$  clipped between the range  $[-c, c]$ .
Corollary 1.1 (Optimal classifier for binary symmetric CSBM). For any  $\ell \geq 1$ , the asymptotically  $\ell$ -locally Bayes optimal classifier of the root for the sequence  $(G_{n},u_{n})\sim \mathrm{CSBM}(n,d,\mathbb{P},\frac{a}{n},\frac{b}{n})$  is
$$
h _ {\ell} ^ {*} (u, \{\mathbf {X} _ {v} \} _ {v \in \eta_ {\ell} (u)}) = \operatorname {s g n} \left(\psi (\mathbf {X} _ {u}) + \sum_ {v \in \eta_ {\ell} (u) \backslash \{u \}} \mathcal {M} _ {d (u, v)} (\mathbf {X} _ {v})\right),
$$
where  $\mathcal{M}_k(\mathbf{x}) = \mathrm{sgn}(a - b)\cdot \varphi (\psi (\mathbf{x}),c_k)$  with  $c_{k} = \log \left(\frac{1 + \Gamma^{k}}{1 - \Gamma^{k}}\right)$ , and  $\psi (\mathbf{x}) = \log \frac{\rho_+(\mathbf{x})}{\rho_-(\mathbf{x})}$ .
In this simplified setting, we note that the messages propagated from nodes in the  $\ell$ -local neighbourhood of node  $u$  are clipped proportional to a function of their distance  $k$  from node  $u$ . In particular, the clip threshold  $c_{k}$  can be expressed in terms of the graph SNR  $\Gamma$  from (1). It is interesting to observe in Corollary 1.1 that  $c_{k}$  decreases rapidly as  $k$  increases. Since  $\Gamma < 1$ , this means that to predict the label of node  $u$ , the value of the message propagated from node  $v$  at distance  $k$  from  $u$  decreases exponentially in  $k$ .
The above simplification helps us interpret the classifier in terms of the graph SNR  $\Gamma$ . We will now impose an assumption on the distribution of node features. This will help us analyze the generalization error in terms of the SNR in both the features and the graph, and enable us to compare the performance with other learning methods that are well-studied in the same statistical settings. We will resort to the setting where the features of the CSBM follow a Gaussian mixture. Note that this specialized statistical model has been studied extensively in previous works for benchmarking existing GNN architectures, see for example, Baranwal et al. [2021], Fountoulakis et al. [2022b], Wei et al. [2022], Baranwal et al. [2023].
In principle, one could compute the generalization error of  $h_{\ell}^{*}$  for arbitrary distributions on the node features (see Appendix A.3.2), however, we report the error for Gaussian features for expository reasons. The generalization error is defined for a classifier  $h$  to be the probability of disagreement between the true label  $y_{u}$  and the output of the classifier  $h_{u}$  for node  $u$ . We characterize the error for  $h_{\ell}^{*}$  in the case where  $\mathbb{P}_{-}, \mathbb{P}_{+}$  correspond to the Gaussian mixture with components  $\mathcal{N}(-\pmb{\mu}, \sigma^2\mathbf{I})$  and  $\mathcal{N}(\pmb{\mu}, \sigma^2\mathbf{I})$  for fixed  $\pmb{\mu} \in \mathbb{R}^d$  and  $\sigma > 0^2$ .
In this case, a notion of the signal-to-noise ratio of the features naturally exists, i.e.,  $\gamma = \| \pmb{\mu}\|_2 / \sigma$ , a quantity proportional to the ratio of the distance between the means of the mixture and the standard deviation. The log-likelihood ratio in this setting is  $\psi(\mathbf{x}) = \log \frac{\rho_{+}(\mathbf{x})}{\rho_{-}(\mathbf{x})} = \frac{2}{\sigma^2}\langle \mathbf{x}, \pmb{\mu}\rangle$ .
Consider a sequence  $\{(G_n, u_n)\}_{n \geq 1}$  with  $G_n = (V(G_n), E(G_n))$  from this model where  $u_n \sim \mathrm{Unif}(V(G_n))$ . In this setting, in the absence of features, it is known that  $(G_n, u_n)$  converges locally weakly to a Poisson Galton-Watson tree (see for example, Mossel et al. [2015, Section 4]). Here, for every node, we additionally have features that are independent of the graph, and hence, as a straightforward consequence of Mossel et al. [2015, Section 4],  $(G_n, u_n)$  in our case converges to a feature-decorated Poisson Galton-Watson tree  $(G, u)$ .
For the root node  $u$ , let  $\alpha_{k}$  and  $\beta_{k}$  denote the number of children at generation  $k$  in class  $y_{u}$  and  $-y_{u}$  respectively, where  $y_{u}$  denotes the label of node  $u$ . Then  $\{\alpha_{k}\}_{k\geq 0}$  and  $\{\beta_k\}_{k\geq 0}$  are characterized by
$$
\alpha_ {0} = 1, \beta_ {0} = 0,
$$
$$
\alpha_ {k} \sim \operatorname {P o i} \left(\frac {a \alpha_ {k - 1} + b \beta_ {k - 1}}{2}\right), \beta_ {k} \sim \operatorname {P o i} \left(\frac {a \beta_ {k - 1} + b \alpha_ {k - 1}}{2}\right) \text {f o r} k \in [ \ell ]. \tag {2}
$$
For a classifier  $h$  acting on  $G$ , let  $\mathcal{E}(h)$  denote the probability of misclassification of the root  $u$  in  $G$ , i.e.,  $\mathcal{E}(h) = \mathbf{Pr}(h_u y_u < 0)$ . Correspondingly, in the case of finite  $n$ , we denote by  $\mathcal{E}_n(h)$  the probability of misclassification of a uniform random node  $u_n$  in  $G_n$ . We are now ready to state the generalization error of  $h_\ell^*$ .
Theorem 2 (Generalization error). For any  $\ell \geq 1$ , the generalization error of the asymptotically  $\ell$ -locally Bayes optimal classifier of the root for the sequence  $(G_{n},u_{n})\sim \mathrm{CSBM}(n,d,\mathbb{P},\mathbf{Q})$  with Gaussian features is given by
$$
\mathcal {E} (h _ {\ell} ^ {*}) = \mathbf {P r} \left[ g + \frac {1}{2 \gamma} \sum_ {k \in [ \ell ]} \left(\sum_ {i \in [ \alpha_ {k} ]} Z _ {k, i} ^ {(a)} + \sum_ {i \in [ \beta_ {k} ]} Z _ {k, i} ^ {(b)}\right) > \gamma \right],
$$
In principle, it is easy to analyze arbitrarily fixed means, say  $\mu, \nu$  instead of keeping them symmetric, i.e.,  $\pm \mu$ , with only minor changes in calculations.
where  $\alpha_{k},\beta_{k}$  are as in (2),  $Z_{k,i}^{(a)} = \varphi (-2\gamma^{2} + 2\gamma g_{k,i},c_{k})$ $Z_{k,i}^{(b)} = \varphi (2\gamma^{2} + 2\gamma g_{k,i},c_{k})$  , and  $g,\{g_{k,i}\}$  are mutually independent standard Gaussian random variables.
Let us now understand how the error described in Theorem 2 behaves in terms of the two SNRs  $\gamma$  (for the features), and  $\Gamma$  (for the graph). Note that  $\mathcal{E}(h_{\ell}^{*}) \to 0$  as  $\gamma \to \infty$ , and  $\mathcal{E}(h_{\ell}^{*}) \to 1/2$  as  $\gamma \to 0$ . This means that if the signal in the features is large, the number of mistakes made by the classifier vanishes, while if the signal is very small, then roughly half of the nodes are misclassified (equivalent to making a uniform random guess for each node).
To see how  $\Gamma$  affects the error, we begin by looking at two extreme settings: first, where the graph is complete noise, i.e.,  $\Gamma = 0$ , and second, where the graph signal is very strong, i.e.,  $\Gamma \rightarrow 1$ , followed by a discussion on how  $h_\ell^*$  interpolates between these extremes. Let  $\phi_{\pm}$  denote the Gaussian density functions with means  $\pm \mu$  and variance  $\sigma^2\mathbf{I}_d$ . Define the random variable
$$
\xi_ {\ell} = \xi_ {\ell} (a, b) = \frac {1 + \sum_ {k = 1} ^ {\ell} \left| \alpha_ {k} - \beta_ {k} \right|}{\sqrt {1 + \sum_ {k = 1} ^ {\ell} \left(\alpha_ {k} + \beta_ {k}\right)}}, \tag {3}
$$
where  $\alpha_{k},\beta_{k}$  follow (2). In the following, we denote the vanilla GCN classifier from Kipf and Welling [2017] by  $h_\mathrm{gcn}$ . We then have the following result.
Theorem 3 (Extreme graph signals). Let  $h_{\ell}^{*}$  be the classifier from Corollary 1.1,  $h_0^* (u) = \operatorname {sgn}(\langle \mathbf{X}_u,\pmb {\mu}\rangle)$  be the Bayes optimal classifier given only the feature information of the root node  $u$ , and  $h_\mathrm{gcn}$  be the one-layer vanilla GCN classifier. Then we have that for any fixed  $\ell$ :
1. If  $\Gamma = 0$  then  $\mathcal{E}(h_{\ell}^{*}) = \mathcal{E}(h_0^*) = \Phi (-\gamma)$ , where  $\Phi$  is the standard Gaussian CDF.
2. If  $\Gamma \to 1$  then  $\xi_{\ell} \geq 1$  a.s. and  $\mathcal{E}(h_{\ell}^{*}) \to \mathbf{Pr}(g > \gamma \xi_{\ell})$ , where  $g \sim \mathcal{N}(0,1)$ .
3.  $\mathcal{E}(h_{\mathrm{gcn}}) = \mathbf{Pr}$ $(g > \gamma \xi_1)$
Theorem 3 shows that in the regime of extremely low graph SNR, the optimal classifier  $h_{\ell}^{*}$  reduces to a linear classifier  $h_0^* (u) = \mathrm{sgn}(\langle \mathbf{X}_u,\pmb {\mu}\rangle)$ , which can be realized by a simple MLP that does not use the graph component of the data at all. On the other hand, in the regime of extremely strong graph SNR,  $h_\ell^*$  reduces to a simple convolution over all nodes in the  $\ell$ -neighbourhood and is comparable to a typical GCN. Furthermore, we note that in the strong graph SNR regime  $\Gamma \rightarrow 1$ ,  $\mathcal{E}(h_{\ell}^{*})\to \mathbf{Pr}(g > \gamma \xi_{\ell})\leq \Phi (-\gamma)$  since  $\xi_{\ell}\geq 1$ . The clip operation during the propagation of messages makes things interesting between these two extremes, where  $h_\ell^*$  interpolates between a simple MLP and an  $\ell$ -hop convolutional network. This interpolation is characterized by the graph signal  $\Gamma$ , since the messages are clipped in the range  $[-c_k,c_k]$ , where  $c_{k} = \log \left(\frac{1 + \Gamma^{k}}{1 - \Gamma^{k}}\right)$ .
In addition, Theorem 3 concludes that if  $\xi_1(a,b) > 1$ , then a GCN can perform better than every classifier that does not see the graph. On the other hand, if  $\xi_1(a,b) < 1$ , then a GCN incurs more errors on the data than the best methods that do not use the graph. Interestingly, but not surprisingly, this result aligns with the Kesten-Stigum weak recovery threshold for the community-detection problem on the sparse stochastic block model [Massoulie, 2014, Mossel et al., 2018], meaning that if weak recovery is possible on the graph component of the data, then a GCN is able to exploit it to perform better than methods that do not use the graph, e.g., a simple MLP.
We now demonstrate our results through experiments using pytorch and pytorch-geometric [Fey and Lenssen, 2019]. The following simulations are for the setting  $n = 10000$  and  $d = 4$  for binary classification on the CSBM. We implement Architecture 1 for the binary case, and perform full-batch training on a graph sampled from the CSBM with certain signals (mentioned in the figures), followed by an evaluation of the architecture on a new graph sampled from the same distribution.
In Fig. 1, we show that the accuracy obtained by the optimal classifier is higher than both a simple MLP and a vanilla GCN [Kipf and Welling, 2017]. We plot the test accuracy of Architecture 1 against the SNR in the node features,  $\gamma = \| \pmb{\mu} \| / \sigma$  in Fig. 1a, and against the graph SNR  $\Gamma = |a - b| / (a + b)$  in Fig. 1b. We fix  $\Gamma = 0.42$  and  $\gamma = 1$  for the two plots, respectively. We chose these specific values because they generate relatively clearer plots where the accuracy metrics for the three architectures are easily visible and distinguished from each other. The results are similar for other values for  $\Gamma$  and  $\gamma$ , i.e., the Bayes optimal architecture is superior to both MLP and GCN.
(a) Varying  $\gamma$  with fixed  $\Gamma = 0.42$ .
(b) Varying  $\Gamma$  with fixed  $\gamma = 1$ .
Figure 1: Comparison of Architecture 1 against an MLP and a vanilla GCN [Kipf and Welling, 2017].
Furthermore, Fig. 2 shows that as claimed in Theorem 3, when the graph signal is at the extremes, i.e.,  $\Gamma = 0$  and  $\Gamma = 1$ , Architecture 1 behaves like a simple MLP and performs a typical convolution (averaging) over all nodes in the  $\ell$ -neighbourhood, respectively. In the regime of poor graph SNR, i.e.,  $\Gamma = 0$ , a GCN is worse than a simple MLP, as inferred from part three of Theorem 3.
(a) Fixed graph signal  $\Gamma = 0$
Figure 2: Demonstration of Theorem 3 for extreme graph signals. In the case where  $\Gamma = 0$ , the architecture reduces to an MLP (Fig. 2a), while if  $\Gamma = 1$ , it behaves the same as a GCN (Fig. 2b).
(b) Fixed graph signal  $\Gamma = 1$
Finally, we observe that for the binary setting when the parameters of the architecture are initialized uniformly at random, gradient descent converges and the neural network learns the right parameters such that Architecture 1 realizes the optimal classifier in Corollary 1.1. This convergence, along with a comparison of our architecture to the Approximate Message-passing Belief Propagation (AMP-BP) algorithm from Deshpande et al. [2018] is presented in Appendix B.
3.5 Non-asymptotic Setting
We now turn to the non-asymptotic regime and argue that for fixed  $n$ , the classifier in Corollary 1.1 is still in a formal sense, Bayes optimal for an overwhelming fraction of nodes. We begin by exploiting the fact that for up to logarithmic depth neighbourhoods, a sparse CSBM graph is tree-like.
Proposition 3.1 (Tree neighbourhoods). Let  $G = (V, E) \sim \mathrm{CSBM}(n, d, \mathbb{P}, \frac{a}{n}, \frac{b}{n})$  for constants  $a, b > 1$ . Then for any  $\ell = c \log n$  such that  $c \log ((a + b) / 2) < 1 / 4$ , with probability  $1 - O(1 / \log^2 n)$ , the number of nodes  $u \in V$  whose  $\ell$ -neighbourhood is cycle-free is  $n(1 - o(\log^4(n) / \sqrt{n}))$ .
In particular, Proposition 3.1 states that for  $\ell = c\log n$  for a suitable constant  $c$ , the  $\ell$ -neighbourhood of an overwhelming fraction of nodes is a tree. This implies that the classifier  $h_\ell^*$  is Bayes optimal for roughly all of the nodes. Moreover, since the diameter of a sparse graph (as in our setting) is  $O(\log n)$  almost surely [Chung and Lu, 2001, Theorem 6], any learning mechanism can only look as far as  $O(\log n)$ -hops away from a node to gather new information. This shows that for such graphs, GNNs that are not very deep and look at only up to logarithmic distance in the neighbourhood are sufficient.
Let us now turn to the misclassification error in the non-asymptotic setting. Recall that for a classifier  $h \in \mathcal{C}_{\ell}$ , we denote by  $\mathcal{E}_n(h)$  and  $\mathcal{E}(h)$  the misclassification error of  $h$  on the data model with  $n$  nodes,
and on the limiting data model with  $n\to \infty$  , respectively. Furthermore, recall from Corollary 1.1 that  $\min_{h\in \mathcal{C}_\ell}\mathcal{E}(h) = \mathcal{E}(h_\ell^*)$  . Our next result shows that the optimal misclassification error in the nonasymptotic setting across all  $\ell$  -local classifiers, i.e.,  $\min_{h\in \mathcal{C}_{\ell}}\mathcal{E}_{n}(h)$  , is close to the misclassification error obtained in the non-asymptotic setting by  $h_\ell^*$  . Moreover,  $\min_{h\in \mathcal{C}_{\ell}}\mathcal{E}_{n}(h)$  is also close to  $\mathcal{E}(h_\ell^*)$  which is explicitly computed in Theorem 2.
Theorem 4 (Misclassification error for fixed  $n$ ). For any  $1 \leq \ell \leq c \log n$  such that the positive constant  $c$  satisfies  $c \log \left( \frac{a + b}{2} \right) < 1 / 4$ , we have that
$$
\left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) \right| = O \left(\frac {1}{\log^ {2} n}\right), \quad \left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} \left(h _ {\ell} ^ {*}\right) \right| = O \left(\frac {1}{\log^ {2} n}\right).
$$
Recall that Corollary 1.1 implies that  $h_{\ell}^{*}$  performs optimally on the limiting data model (asymptotic setting) among the class of  $\ell$ -local classifiers  $\mathcal{C}_{\ell}$ , but it may not be optimal for the non-asymptotic data model where we have a finite feature-decorated graph with  $n$  nodes. However, Theorem 4 helps us conclude that even in the non-asymptotic setting,  $h_{\ell}^{*}$  performs almost as well as the actual optimal classifier among  $\mathcal{C}_{\ell}$  in this case, as long as we compare with classifiers that can only look at moderate logarithmic depths in the local neighbourhood, i.e.,  $\ell \leq c\log n$  for a suitable  $c$ .
4 Conclusion and Future Work
In this work, we present a comprehensive theoretical characterization of the Bayes optimal node classification architecture for sparse feature-decorated graphs and show that it can be realized using the message-passing framework. Utilizing a well-established and well-studied statistical model, we interpret its performance in terms of the SNR in the data and validate our findings through empirical analysis of synthetic data. Additionally, we identify the following limitations as prospects for future work: (1) We consider neighbourhoods up to distance  $\ell = c\log n$  for a small enough  $c$ . Extending  $\ell$  to the graph's diameter (known to be  $O(\log n)$  with high probability) by removing the restriction on  $c$  poses challenges due to the presence of cycles. (2) More insights can be provided through experiments on real data to benchmark the architecture in cases where we have a significant gap between the theoretical assumptions (sparse and locally tree-like graph) and the real-world data.
A.J. acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada Research Chairs programme. Cette recherche a ete enterprise grace, en partie, au soutien financier du Conseil de Recherches en Sciences Naturelles et en Génie du Canada (CRSNG), [RGPIN-2020-04597, DGECR-2020-00199], et du Programme des chaires de recherche du Canada.
K. Fountoulakis would like to acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). Cette recherche a été financée par le Conseil de recherches en sciences naturelles et en génie du Canada (CRSNG), [RGPIN-2019-04067, DGECR-2019-00147].
A. B. Adcock, B. D. Sullivan, and M. W. Mahoney. Tree-like structure in large social and information networks. In 2013 IEEE 13th International Conference on Data Mining, pages 1-10, 2013. doi: 10.1109/ICDM.2013.77.
L. Backstrom and J. Leskovec. Supervised random walks: predicting and recommending links in social networks. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 635-644, 2011.
M. Balcilar, G. Renton, P. Héroux, B. Gaüzère, S. Adam, and P. Honeine. Analyzing the Expressive Power of Graph Neural Networks in a Spectral Perspective. In International Conference on Learning Representations, 2021.
V. Bapst, T. Keck, A. Grabska-Barwińska, C. Donner, E. D. Cubuk, S. S. Schoenholz, A. Obika, A. W. Nelson, T. Back, D. Hassabis, et al. Unveiling the predictive power of static structure in glassy systems. Nature Physics, 16(4):448-454, 2020.
A Supplementary Discussion and Proofs
A.1 Local Weak Convergence
We briefly recall here the notion of local weak convergence of random rooted graphs. The notion of local weak convergence of random, feature-decorated, rooted graphs is defined analogously.
Let us begin first with the case of rooted graphs. A rooted graph  $(G,u) = ((E,V),u)$  is a graph  $G$  with a distinguished vertex  $u$  called the root. We say that two rooted graphs  $(G_{1},u_{1}) = ((E_{1},V_{1}),u_{1})$ ,  $(G_{2},u_{2}) = ((E_{2},V_{2}),u_{2})$  are isomorphic if there is a bijection  $\phi :V_1\to V_2$  such that  $\phi (u) = v$  and such that if  $(x,y)\in E_1$  then  $(\phi (x),\phi (y))\in E_2$ . In this case we write  $(G_{1},u_{1})\cong (G_{2},u_{2})$ . For a rooted graph  $(G,u)$ , we denote its isomorphism class as  $[(G,u)]$ .
Let  $\mathcal{G}_{*}$  denote the set of isomorphism classes of (locally finite) rooted graphs. For a vertex  $v\in G$  we let  $\eta_k(v)$  denote the collection of neighbours of  $v$  of distance at most  $k$  in the canonical edge distance metric, and  $G[\eta_k(v)]$  denote the subgraph induced on this collection of vertices. We then have the notion of local convergence in  $\mathcal{G}_{*}$ .
Definition A.1 (Local convergence on  $\mathcal{G}_*$ ). We say that a sequence  $[(G_n, u_n)] \in \mathcal{G}_*$  converges locally to  $[(G, u)] \in \mathcal{G}_*$  if for each  $k > 0$ , we have that  $G[\eta_k(u_n)] \cong G[\eta_k(u)]$  eventually.
It can be shown [Bordenave, 2016, Lemma 3.4] that  $\mathcal{G}_{*}$  equipped with the topology of local convergence is a Polish space. We are then in the position to define local weak convergence on  $\mathcal{G}_{*}$ . In brief, the topology of local weak convergence of random graphs is the topology of weak convergence of measures on the space of probability measures on  $\mathcal{G}_{*}$ , namely  $\mathcal{M}_1(G_{*})$ .
Definition A.2 (local weak convergence of rooted graphs). A sequence  $\{[G_n, u_n]\}$  of random rooted graphs with corresponding laws  $\{\mu_n\} \subseteq \mathcal{M}_1(\mathcal{G}_*)$  is said to locally weakly converge to a random rooted graph  $[G, u]$  with law  $\mu \in \mathcal{M}_1(\mathcal{G}_*)$  if  $\mu_n \to \mu$  weakly.