SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Consider a random feature-decorated graph  $G_{n} \sim \mathrm{CSBM}(n, d, \{\mathbb{P}_{\pm}\}, a / n, b / n)$ , where  $\mathbb{P}_{\pm}$  correspond to the distributions  $\mathcal{N}(\pm \pmb{\mu}, \sigma^2 \mathbf{I})$  for the node features given by  $\{\mathbf{X}_u\}_{u \in [n]}$ . For a classifier  $h \in C_{\ell}$ , the class of all  $\ell$ -local classifiers, define  $\mathcal{E}_n(h)$  to be the probability of misclassification for a uniform at random node  $u \in [n]$ , i.e.,  $\mathcal{E}_n(h) = \mathbf{Pr}(y_u \cdot h(u, \{\mathbf{X}_v\}_{v \in \eta_\ell(u)}, \eta_\ell(u)) < 0)$ . Since it is known that all classifiers in  $C_{\ell}$  operate on  $u$  given the information in its  $\ell$ -neighbourhood  $\eta_\ell(u)$ , we will omit  $\eta_\ell(u)$  from the notation and say  $h(u)$  instead of  $h(u, \{\mathbf{X}_v\}_{v \in \eta_\ell(u)}, \eta_\ell(u))$  when it is understood. Let  $\mathbf{P}$  be the joint measure of the variables  $\{U_k^{\pm}\}_{k \leq \ell}$  from Eqs. (6) and (7), and  $\mathbf{P}'$  be the joint measure of the variables  $\{\alpha_k, \beta_k\}_{k \leq \ell}$  from Eq. (8). Then Lemma A.5 gives us that  $d_{\mathrm{TV}}(\mathbf{P}, \mathbf{P}') \leq O\left(\frac{\log^3 n}{n^{1/4}}\right) = o_n(1)$ .
Recall  $\mathcal{E}(h_{\ell}^{*})$  computed in Theorem 2 for the limiting data model  $(G,u)$ .
$$
\begin{array}{l} \mathcal {E} \left(h _ {\ell} ^ {*}\right) = \Pr \left[ g + \frac {1}{2 \gamma} \sum_ {k \in [ \ell ]} \left(\sum_ {i = 1} ^ {\alpha_ {k}} Z _ {k, i} ^ {(a)} + \sum_ {i = 1} ^ {\beta_ {k}} Z _ {k, i} ^ {(b)}\right) > \gamma \right] \\ = \int \mathbf {P r} \left[ \right. g + \frac {1}{2 \gamma} \sum_ {k \in [ \ell ]} \left(\sum_ {i = 1} ^ {\alpha_ {k}} Z _ {k, i} ^ {(a)} + \sum_ {i = 1} ^ {\beta_ {k}} Z _ {k, i} ^ {(b)}\right) > \gamma \left| \right. \{\alpha_ {k}, \beta_ {k} \} _ {k \leq \ell} \left. \right] d \mathbf {P} ^ {\prime}. \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) = \Pr \left[ g + \frac {1}{2 \gamma} \sum_ {k \in [ \ell ]} \left(\sum_ {i = 1} ^ {U _ {k} ^ {+}} Z _ {k, i} ^ {(a)} + \sum_ {i = 1} ^ {U _ {k} ^ {-}} Z _ {k, i} ^ {(b)}\right) > \gamma \right] \\ = \int \mathbf {P r} \left[ g + \frac {1}{2 \gamma} \sum_ {k \in [ \ell ]} \left(\sum_ {i = 1} ^ {U _ {k} ^ {+}} Z _ {k, i} ^ {(a)} + \sum_ {i = 1} ^ {U _ {k} ^ {-}} Z _ {k, i} ^ {(b)}\right) > \gamma \left| \{U _ {k} ^ {\pm} \} _ {k \leq \ell} \right. \right] d \mathbf {P}. \\ \end{array}
$$
Thus, we obtain that
$$
\left| \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) - \mathcal {E} \left(h _ {\ell} ^ {*}\right) \right| \leq d _ {\mathrm {T V}} \left(\mathbf {P}, \mathbf {P} ^ {\prime}\right) \leq O \left(\frac {\log^ {3} n}{n ^ {1 / 4}}\right) = o _ {n} (1), \tag {11}
$$
Let us now focus on the case with finite  $n$ . Let  $A$  denote the event from Proposition 3.1 where the number of nodes with cycle-free  $\ell$ -neighbourhoods is  $1 - o\left(\frac{\log^4 n}{\sqrt{n}}\right)$ . For a node  $u \in G_n$ , let  $E_u$  denote the event that the subgraph induced by the  $\ell$ -neighbourhood of  $u$ ,  $\eta_\ell(u)$  is a tree. Then observe that for a uniform random node  $u \in G_n$ ,
$$
\begin{array}{l} \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) = \Pr \left(y _ {u} h _ {\ell , n} ^ {*} (u) <   0\right) \\ = \Pr (E _ {u}) \Pr (y _ {u} h _ {\ell , n} ^ {*} (u) <   0 \mid E _ {u}) + \Pr (E _ {u} ^ {c}) \Pr (y _ {u} h _ {\ell , n} ^ {*} (u) <   0 \mid E _ {u} ^ {c}) \\ = (1 - o _ {n} (1)) \mathbf {P r} \left(y _ {u} h _ {\ell} ^ {*} (u) <   0\right) + o _ {n} (1) \\ = \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) \pm o _ {n} (1). \\ \end{array}
$$
In the above, we used from Proposition 3.1 that  $\mathbf{Pr}(E_u) = \mathbf{Pr}(E_u \cap A) + \mathbf{Pr}(E_u \cap A^{\mathrm{c}}) = 1 - O\left(\frac{1}{\log^2 n}\right)$ , and that  $\mathcal{E}_n(h_\ell^*) = \min_{h \in \mathcal{C}_\ell} \mathbf{Pr}(y_h h(u) < 0 \mid E_u)$ . This establishes the first part:
$$
\left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} _ {n} (h _ {\ell} ^ {*}) \right| = O \left(\frac {1}{\log^ {2} n}\right).
$$
Combining the above display with (11), we obtain the second part, i.e.,
$$
\begin{array}{l} \left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} (h) \right| = \left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} \left(h _ {\ell} ^ {*}\right) \right| \\ = \left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) + \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) - \mathcal {E} \left(h _ {\ell} ^ {*}\right) \right| \\ \leq \left| \min  _ {h \in \mathcal {C} _ {\ell}} \mathcal {E} _ {n} (h) - \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) \right| + \left| \mathcal {E} _ {n} \left(h _ {\ell} ^ {*}\right) - \mathcal {E} \left(h _ {\ell} ^ {*}\right) \right| \\ = O \left(\frac {1}{\log^ {2} n}\right) + O \left(\frac {\log^ {3} n}{n ^ {1 / 4}}\right) = O \left(\frac {1}{\log^ {2} n}\right). \\ \end{array}
$$
B Additional Empirical Observations
B.1 Convergence of Parameters
We observe empirically that gradient descent (SGD and Adam implementations in the pytorch library) converges in the binary setting. In this case, the neural network learns the right parameters corresponding to the parameters of the CSBM, i.e.,  $\rho$  and  $\mathbf{Q}$ , such that Architecture 1 realizes the optimal classifier in Corollary 1.1. In Fig. 3, the x-axis denotes the number of epochs elapsed since
Figure 3: Convergence of parameters of Architecture 1.
the beginning of the training process. For the first plot, the y-axis denotes the cosine similarity between the parameters  $\theta_{1} \in \mathbb{R}^{d}$  learned by the MLP  $\mathbf{H}^{(L)}$  in Architecture 1 and the ansatz  $\mu$  that realizes the optimal classifier; while in the second plot, the y-axis denotes the absolute difference between the clip parameter  $\theta_{2} \in \mathbb{R}$  and the ansatz value  $\log \left(\frac{1 + \Gamma}{1 - \Gamma}\right)$ . These experiments are performed in the same setting as Fig. 1b with fixed  $\Gamma = 0.2$ . We see that the parameters converge as the number of training iterations increases. The reported metrics are averaged over 10 trials, and the standard deviation is shown at each iteration using the translucent blue region.
B.2 Comparison with AMP-BP
We now perform a comparison of our architecture with  $\ell = 5$  against the AMP-BP algorithm from [Deshpande et al., 2018] in two different settings. We set  $n = 1000$  and work with two values of  $d\in \{5,500\}$ . The case  $d = 5$  simulates the low-dimensional case where asymptotically  $d / n\to 0$ , while  $d = 500$  represents the high-dimensional case where  $d / n\rightarrow c$  for a constant  $c$ . For the AMP-BP algorithm, we choose two values for the number of iterations,  $t\in \{5,20\}$ .
Tables 1 and 2 show the results for  $\gamma = 1$  with varying values of  $\Gamma \in \{0.3, 0.4, 0.5, 0.6\}$ . We observe that the classifier obtained after training Architecture 1 almost always outperforms AMP-BP for both low-dimensional and high-dimensional cases.
Table 1: Accuracy metrics for Architecture 1 and AMP-BP for  $\gamma = 1$  and  $d = 5$ .
Graph signal	5-local Bayes Optimal	AMP, t = 5	AMP, t = 20
0.3	0.870	0.753	0.858
0.4	0.954	0.816	0.890
0.5	0.988	0.819	0.916
0.6	0.996	0.892	0.952
Table 2: Accuracy metrics for Architecture 1 and AMP-BP for  $\gamma  = 1$  and  $d = {500}$  .
Graph signal	5-local Bayes Optimal	AMP, t = 5	AMP, t = 20
0.3	0.916	0.554	0.848
0.4	0.995	0.558	0.877
0.5	0.998	0.626	0.920
0.6	0.998	0.657	0.940
Tables 3 and 4 show the results for  $\gamma = 0.2$ , i.e., for a weaker feature signal in the data. Here, we observe that although Architecture 1 outperforms AMP-BP in the low-dimensional regime, it exhibits worse performance than AMP-BP in the high-dimensional regime.
Table 3: Accuracy metrics for Architecture 1 and AMP-BP for  $\gamma = {0.2}$  and  $d = 5$  .
Graph signal	5-local Bayes Optimal	AMP, t = 5	AMP, t = 20
0.3	0.554	0.520	0.579
0.4	0.587	0.528	0.584
0.5	0.823	0.604	0.756
0.6	0.997	0.637	0.880
Table 4: Accuracy metrics for Architecture 1 and AMP-BP for  $\gamma = {0.2}$  and  $d = {500}$  .
Graph signal	5-local Bayes Optimal	AMP, t = 5	AMP, t = 20
0.3	0.584	0.502	0.543
0.4	0.706	0.525	0.694
0.5	0.762	0.568	0.804
0.6	0.788	0.604	0.878
It is important to note, however, that this is an apples-to-oranges comparison because AMP-BP is not a local algorithm, i.e., the whole graph is visible to the algorithm and all nodes contribute to the classification of every other node. This is not true for Architecture 1, where only the nodes in the  $\ell$ -hop neighbourhood contribute to this decision. Our notion of optimality is among the class of local algorithms. Furthermore, we observe that AMP-BP with 5 iterations does not converge and obtains a much lower accuracy compared to 20 iterations.