SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {X} ^ {(l)})\right) = \operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)}) \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)}\right) \\ + \operatorname {T r} \left(\frac {p}{n (p + q)} \boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)}) \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)}\right) \\ + \operatorname {T r} \left(\frac {p}{n (p + q)} \boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)}) \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)}\right) \\ + \operatorname {T r} \left(\frac {\left(p ^ {2} + q ^ {2}\right)}{n (p + q) ^ {2}} \boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {(l - 1)}\right) \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)}\right) \\ = \operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)}) \left[ \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)} + \frac {p}{n (p + q)} \left[ \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)} + \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)} \right] \right. \right. \\ \left. \left. + \frac {\left(p ^ {2} + q ^ {2}\right)}{n (p + q) ^ {2}} \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)} \right]\right) \tag {125} \\ \end{array}
$$
Let  $\mathbf{T}_W = \mathbf{W}_1^{*(l)^\top}\mathbf{W}_1^{*(l)} + \frac{p}{n(p + q)}\left[\mathbf{W}_2^{*(l)^\top}\mathbf{W}_1^{*(l)} + \mathbf{W}_1^{*(l)^\top}\mathbf{W}_2^{*(l)}\right] + \frac{(p^2 + q^2)}{n(p + q)^2}\mathbf{W}_2^{*(l)^\top}\mathbf{W}_2^{*(l)}$ . Then, observe that  $\mathbf{T}_W \in \mathbb{R}^{d_{l-1} \times d_{l-1}}$  is symmetric and positive semi-definite. To this end, the increase/decrease in  $\operatorname{Tr}\left(\boldsymbol{\Sigma}_W(\mathbf{X}^{(l)})\right)$  with respect to  $\operatorname{Tr}\left(\boldsymbol{\Sigma}_W(\mathbf{H}^{(l-1)})\right)$  boils down to:
$$
\frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {d _ {l - 1} - i + 1} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {(l - 1)}\right)\right) \lambda_ {i} \left(\mathbf {T} _ {W}\right)}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {(l - 1)}\right)\right)} \leq \frac {\operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {X} ^ {(l)}\right)\right)}{\operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {(l - 1)}\right)\right)} \leq \frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {(l - 1)}\right)\right) \lambda_ {i} \left(\mathbf {T} _ {W}\right)}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} ^ {l - 1}\right)\right)} \tag {126}
$$
H Addition Experiments
- Infrastructure details: We perform experiments on a virtual machine with 8 Intel(R) Xeon(R) Platinum 8268 CPUs, 32GB of RAM, and 1 Quadro RTX 8000 GPU with 32GB of allocated memory. Our Python package 'gnn_collapse' leverages PyTorch 1.12.1 and PyTorch-Geometric (PyG) 2.1.0 frameworks. For reproducible experiments and consistency with previous research, we extend the SBM generator by Chen et al. [16] and NC metrics by Zhu et al. [74].
H.1 Experiments with GNNs to track penultimate layer features
- Datasets: We consider a variety of SSBM graph datasets as follows:
$$
\mathbf {D 1}: C = 2, N = 1 0 0 0, p = 0. 0 2 5, q = 0. 0 0 1 7, K = 1 0 0 0
$$
$$
\mathbf {D 2}: C = 4, N = 1 5 0 0, p = 0. 0 7 2, q = 0. 0 0 4 8, K = 1 0 0 0
$$
- GNNs: In our experiments, we empirically track the NC metrics of penultimate layer features during training for both the GNN designs  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$ . The number of layers is set to 32 and the hidden dimension is set to 8 across layers for datasets with  $C = 2$  and set to 16 for datasets with  $C = 4$ .
- Optimization: The GNNs are trained for 8 epochs using stochastic gradient descent (SGD) with momentum 0.9, weight decay  $5 \times 10^{-4}$ , and a learning rate set to 0.004 for D1 and 0.006 for D2.
- Observations: Figures 7, 8 illustrate the training loss, overlap and all the NC metrics that we defined in our setup for  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$  on dataset D1. Note that when  $C = 2$ , the re-centering of the 2 class-means by subtracting the global mean, always leads to separation with maximal angle, irrespective of the configuration of the non-centered class-means. Thus, we skip the corresponding  $\mathcal{NC}_2$  plots for H, H $\hat{\mathbf{A}}$  when  $C = 2$ . Additionally, we can observe similar trends in NC metrics from Figures 9, 10 even after increasing  $N, C$  in dataset D2.
Additionally, in all these experiments, notice that  $\mathcal{NC}_2$ ,  $\mathcal{NC}_3$  metrics do not show a significant reduction. In this context, a reduction indicates that a simplex equiangular tight-frame (simplex ETF) or an orthogonal frame (OF) is the desired configuration for weights and penultimate layer feature (re-centered) class-means. This behaviour can be linked to the presence of  $\widehat{\mathbf{A}}$  in the risk formulation. However, our understanding of the role of  $\widehat{\mathbf{A}}$  in determining these alignments towards simplex ETF or an OF is still unclear and would be a valuable future effort.
Figure 7: GNN  $\psi_{\Theta}^{\mathcal{F}}$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 8: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 9: GNN  $\psi_{\Theta}^{\mathcal{F}}$  on D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 10: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  on D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}\hat{\mathbf{A}}$ .
H.2 Experiments with UFM to model penultimate layer behavior
To improve our understanding of the empirical behavior of GNNs and to validate our theoretical results with the gUFM, we prepare datasets based on two strategies:
- Case  $\mathbf{C}^{+}$ : This case represents a graph that satisfies condition  $\mathbf{C}$ . Without loss of generality, we leverage the expected degrees of nodes and consider  $t_{cc} = \lceil n * p \rceil, c \in [C]$  and  $t_{cc'} = \lceil n * q \rceil, c \neq c' \in [C]$ . Based on our notation, recall that  $\Omega_c, \Omega_{c'}$  represents the set of nodes belonging to classes (communities)  $c, c' \in [C]$  respectively. To this end, observe that the following conditions should be satisfied<sup>12</sup>:
1. The sub-graph formed by nodes  $\Omega_c$  should be  $t_{cc}$ -regular, for all  $c \in [C]$ .
2. The bipartite sub-graph formed by  $\Omega_c, \Omega_{c'}$  should be  $t_{cc'}$ -regular, for all  $c, c' \in [C]$ .
- Case  $\mathbf{C}^{-}$ : This case represents a graph that does not satisfy condition  $\mathbf{C}$ . Since any random graph sampled from  $\mathrm{SSBM}(N, C, p, q)$  satisfies this requirement with a high probability (as per theorem 3.2), we simply use this randomly sampled graph. As a simple sanity check, one can verify if the sampled graph satisfies the condition  $\mathbf{C}$  and re-sample.
Especially, we consider the  $\mathbf{C}^{+}$ ,  $\mathbf{C}^{-}$  variants of SSBM graphs with following parameters:
D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 10$
D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 10$
- Optimization: The gUFMs are trained using plain gradient descent for 50000 epochs with a learning rate of 0.1 and L2 regularization parameters  $\lambda_{W_1} = \lambda_{W_2} = \lambda_H = 5 \times 10^{-3}$ .
- Observations: Figures 12, 14, and 16, 18 illustrate the training loss, overlap and the NC metrics for the gUFM acting on  $\mathbf{C}^{-}$  variants of datasets D1, D2 respectively. Although gUFM is an optimistic mathematical model, we can observe a close resemblance of the values and trends of NC metrics to those of the penultimate layer features of the actual GNNs. This observation is justified as any random SSBM graph fails to satisfy condition C with a high probability. To this end, observe from Figures 11, 13, 15, 17 that when graphs satisfy condition C, the NC1 metrics tend to reduce drastically (for gUFM designs based on  $\psi_{\Theta}^{\mathcal{F}}$ ,  $\psi_{\Theta}^{\mathcal{F}'}$  and  $C = 2, 4$ ). Thus proving our theoretical results. Furthermore,
<sup>12</sup>We utilize NetworkX python libraries to generate SSBM graphs that satisfy these conditions.
${}^{13}\lambda_{W_1}$  is not applicable for gUFM based on  $\psi_{\Theta}^{\mathcal{F}'}$
observe from Figures 15, 17 that the (re-centered) class means for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$  tend to align very closely to a simplex ETF and tend to converge at such a configuration. Based on our previous observations for GNN training, we underscore this observation for gUFM and emphasize that a rigorous theoretical analysis on the role of  $\widehat{\mathbf{A}}$  in determining the structures of  $\mathbf{H}$  can be a crucial future effort.
Figure 11: gUFM  $\psi_{\Theta}^{\mathcal{F}}$  on  $\mathbf{C}^{+}$  variant of D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_1, \mathbf{W}_2, \mathcal{NC}_3$  plots for  $\mathbf{W}_1, \mathbf{H}$  and  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
Figure 12: gUFM  $\psi_{\Theta}^{\mathcal{F}}$  on  $\mathbf{C}^{-}$  variant of D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_1, \mathbf{W}_2, \mathcal{NC}_3$  plots for  $\mathbf{W}_1, \mathbf{H}$  and  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
Figure 13: gUFM  $\psi_{\Theta}^{\mathcal{F}'}$  on  $\mathbf{C}^{+}$  variant of D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
Figure 14: gUFM  $\psi_{\Theta}^{\mathcal{F}'}$  on  $\mathbf{C}^{-}$  variant of D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
Figure 15: gUFM  $\psi_{\Theta}^{\mathcal{F}}$  on  $\mathbf{C}^{+}$  variant of D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathit{SNR}(\mathcal{NC}_1)$  for  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathcal{NC}_3$  plots for  $\mathbf{W}_1, \mathbf{H}$  and  $\mathbf{W}_2, \mathbf{H}\hat{\mathbf{A}}$ .
Figure 16: gUFM  $\psi_{\Theta}^{\mathcal{F}}$  on  $\mathbf{C}^{-}$  variant of D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathit{SNR}(\mathcal{NC}_1)$  for  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathcal{NC}_3$  plots for  $\mathbf{W}_1, \mathbf{H}$  and  $\mathbf{W}_2, \mathbf{H}\hat{\mathbf{A}}$ .
Figure 17: gUFM  $\psi_{\Theta}^{\mathcal{F}'}$  on  $\mathbf{C}^{+}$  variant of D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
Figure 18: gUFM  $\psi_{\Theta}^{\mathcal{F}'}$  on  $\mathbf{C}^{-}$  variant of D2:  $C = 4, N = 1500, p = 0.072, q = 0.0048, K = 10$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\widehat{\mathbf{A}}$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2, \mathbf{H}\widehat{\mathbf{A}}$ .
H.3 Experiments with GNNs and Spectral Methods
The main focus of this section is to emphasize the differences between power iterations-based spectral methods and GNNs. To this end, we consider the following datasets and GNNs as follows:
D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(train) = 1000, K(test) = 100$
D2:  $C = 2, N = 1000, p = 0.0017, q = 0.025, K(\text{train}) = 1000, K(\text{test}) = 100$
- GNNs: In our experiments, we empirically track the NC metrics of penultimate layer features during training for both the GNN designs  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$ . The number of layers is varied based on  $L = 64, 128$  and the hidden dimension is set to 8 across layers.
- Spectral methods: The number of projected power-iterations are varied based on  $L = 64$ , 128 for a fair comparison with GNNs.
- Optimization: The GNNs are trained for 8 epochs using stochastic gradient descent (SGD) with learning rate of 0.004, momentum 0.9 and a weight decay of  $5 \times 10^{-4}$ .
- Observations: For the dataset D1 with homophilic graphs, we first plot the training metrics for GNNs  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$  with  $L = 64$  in Figures 19, 20 respectively and ensure that they reach TPT. Now, from Figures 21, 22 observe that the ratios  $\mathrm{Tr}(\Sigma_B(\mathbf{x}^{(l)})) / \mathrm{Tr}(\Sigma_B(\mathbf{w}^{(l - 1)}))$ ,  $\mathrm{Tr}(\Sigma_W(\mathbf{x}^{(l)})) / \mathrm{Tr}(\Sigma_W(\mathbf{w}^{(l - 1)}))$  tend to be constant throughout the power iterations for spectral methods, whereas, the GNNs behave differently as  $\mathrm{Tr}(\Sigma_B(\mathbf{X}^{(l)})) / \mathrm{Tr}(\Sigma_B(\mathbf{H}^{(l - 1)}))$ ,  $\mathrm{Tr}(\Sigma_W(\mathbf{X}^{(l)})) / \mathrm{Tr}(\Sigma_W(\mathbf{H}^{(l - 1)}))$  tend to decrease across depth. We make similar observations for GNNs  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$  with  $L = 128$  in Figures 23,24,25,26 and note that the behaviour is the same as  $L = 32$  case illustrated in the main text. However, when considering the dataset D2 with heterophilic graphs, even though the GNNs reach TPT during training (Figures 27,28,29,30), the evolution of ratios  $\mathrm{Tr}(\Sigma_B(\mathbf{X}^{(l)})) / \mathrm{Tr}(\Sigma_B(\mathbf{H}^{(l - 1)}))$ ,  $\mathrm{Tr}(\Sigma_W(\mathbf{X}^{(l)})) / \mathrm{Tr}(\Sigma_W(\mathbf{H}^{(l - 1)}))$  tends to differ especially for the GNN  $\psi_{\Theta}^{\mathcal{F}'}$  when  $L = 64$  (Figures 31,32) and  $L = 128$  (Figures 33,34). Thus, highlighting the empirical role of depth in GNN design which requires further investigations in future efforts.
Figure 19: GNN  $\psi_{\Theta}^{\mathcal{F}}$  with  $L = 64$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 20: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  with  $L = 64$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
(d) GNN  $\psi_{\mathcal{O}}^{\mathcal{F}'}$
(a) Normalized Laplacian
Figure 22: Ratio of traces of covariance matrices across 64 projected power iterations for NL and BH (a,b), and across 64 layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d) on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(test) = 100$ .
(c) GNN  $\psi_{\mathcal{O}}^{\mathcal{F}}$
Figure 21:  $\mathcal{NC}_1(\mathbf{H}),\widehat{\mathcal{NC}}_1(\mathbf{H})$  metrics (top) and traces of covariance matrices (bottom) across 64 projected power iterations for NL and BH (a,b), and across 64 layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d) on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(test) = 100.$
(d) GNN  $\psi_{\Theta}^{F^{\prime}}$
Figure 23: GNN  $\psi_{\Theta}^{\mathcal{F}}$  with  $L = 128$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\hat{\mathbf{A}}$ .
Figure 24: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  with  $L = 128$  on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(train) = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2, \mathbf{H}, \mathbf{H}\hat{\mathbf{A}}, \mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2, \mathbf{H}\hat{\mathbf{A}}$ .
Figure 25:  $\mathcal{NC}_1(\mathbf{H}),\widetilde{\mathcal{NC}}_1(\mathbf{H})$  metrics (top) and traces of covariance matrices (bottom) across 128 projected power iterations for NL and BH (a,b), and across 128 layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d) on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(test) = 100$ .
(d) GNN  $\psi_{\mathfrak{O}}^{\mathcal{F}'}$
Figure 26: Ratio of traces of covariance matrices across 128 projected power iterations for NL and BH (a,b), and across 128 layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d) on D1:  $C = 2, N = 1000, p = 0.025, q = 0.0017, K(test) = 100$ .
Figure 27: GNN  $\psi_{\Theta}^{\mathcal{F}}$  with  $L = 64$  on D2:  $C = 2, N = 1000, p = 0.0017, q = 0.025, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\hat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\hat{\mathbf{A}}$ .
Figure 28: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  with  $L = 64$  on D2:  $C = 2, N = 1000, p = 0.0017, q = 0.025, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 29: GNN  $\psi_{\Theta}^{\mathcal{F}}$  with  $L = 128$  on D2:  $C = 2, N = 1000, p = 0.0017, q = 0.025, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_1$ ,  $\mathbf{W}_2$ ,  $\mathcal{NC}_3$  plots for  $\mathbf{W}_1$ ,  $\mathbf{H}$  and  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
Figure 30: GNN  $\psi_{\Theta}^{\mathcal{F}'}$  with  $L = 128$  on D2:  $C = 2, N = 1000, p = 0.0017, q = 0.025, K = 1000$ . Illustration of training loss, training overlap,  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $SNR(\mathcal{NC}_1)$  for  $\mathbf{H}\widehat{\mathbf{A}}$ , Frobenius norms of  $\mathbf{W}_2$ ,  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ ,  $\mathcal{NC}_2$  plots for  $\mathbf{W}_2$ , and  $\mathcal{NC}_3$  plots for  $\mathbf{W}_2$ ,  $\mathbf{H}\widehat{\mathbf{A}}$ .
(a) GNN  $\psi_{\Theta}^{\mathcal{F}}$