SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.1. Consider the gUFM in (11) with  $K = 1$  and denote the fraction of neighbors of node  $v_{c,i}$  that belong to class  $c'$  as  $s_{cc',i} = \frac{|\mathcal{N}_{c'}(v_{c,i})|}{|\mathcal{N}(v_{c,i})|}$ . Let the condition  $\pmb{C}$  based on  $s_{cc',i}$  be given by:
$$
\left(s _ {c 1, 1}, \dots , s _ {c C, 1}\right) = \dots = \left(s _ {c 1, n}, \dots , s _ {c C, n}\right), \quad \forall c \in [ C ]. \tag {C}
$$
If a graph  $\mathcal{G}$  satisfies condition  $\mathbf{C}$ , then there exist minimizers of the gUFM that are collapsed (satisfying (12)). Conversely, when either  $\sqrt{\lambda_H\lambda_{W_2}} = 0$ , or  $\sqrt{\lambda_H\lambda_{W_2}} > 0$  and  $\mathcal{G}$  is regular (so that  $\widehat{\mathbf{A}} = \widehat{\mathbf{A}}^{\top}$ ), if there exists a collapsed non-degenerate minimizer of gUFM, then condition  $\mathbf{C}$  necessarily holds.
Remark: The proof is presented in Appendix D. The symmetry assumption on  $\widehat{\mathbf{A}}$  (which implies that  $\mathcal{G}$  is a regular graph) in the second part of the theorem has been made to pass technical obstacles in the proof rather than due to a true limitation. Thus, together with the results of our experiments (where no symmetry is enforced), we believe that this assumption can be dropped. Accordingly, we state the following conjecture.
Conjecture 3.1. Consider the gUFM in (11) with  $K = 1$  and condition  $C$  as stated in theorem 3.1. The minimizers of the gUFM are collapsed (satisfying (12)) iff the graph  $\mathcal{G}$  satisfies condition  $C$ .
Let us dwell on the implication of Theorem 3.1. The stated condition  $\mathbf{C}$  essentially holds when any node  $i\in [n]$  of a certain class  $c$  obeys  $(s_{c1,i},\dots ,s_{cC,i}) = (s_{c1},\dots ,s_{cC})$  for some  $(s_{c1},\dots ,s_{cC})$ , a tuple of the ratio of neighbors  $(\sum_{c^{\prime} = 1}^{C}s_{cc^{\prime}} = 1)$  independent of  $i$ . That is,  $(s_{c1},\dots ,s_{cC})$  must be the same for nodes within the same class but can be different for nodes belonging to different classes. For example, for a plain UFM this condition trivially holds, as  $\widehat{\mathbf{A}} = \mathbf{I}$ . Under the SSBM distribution, it is also easy to see that  $\mathbb{E}\widehat{\mathbf{A}}$  satisfies this condition. However, for more practical graphs, such as those drawn from SSBM, the probability of having a graph that obeys condition  $\mathbf{C}$  is negligible. This is shown in the following theorem.
Theorem 3.2. Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be drawn from SSBM(N,C,p,q). For  $N >> C$ , we have
$$
\mathbb {P} (\mathcal {G} \text {o b e y s} \boldsymbol {C}) <   \left(\sum_ {t = 0} ^ {n} \left[ \binom {n} {t} q ^ {t} (1 - q) ^ {n - t} \right] ^ {n}\right) ^ {\frac {C (C - 1)}{2}}. \tag {13}
$$
The proof is presented in Appendix E. It is not hard to see that as the number of per-class nodes  $n$  increases, the probability of satisfying condition  $\mathbf{C}$  decreases,[6] as numerically exemplified below.
Numerical example. Let's consider a setting with  $C = 2$ ,  $N = 1000$ ,  $a = 3.75$ ,  $b = 0.25$ . This gives us  $n = N / C = 500$ ,  $p = 0.025$ ,  $q = 0.0017$ , for which  $\mathbb{P}(\mathcal{G}$  obeys  $\mathbf{C}) < 2.18\times 10^{-188}$ .
In Appendix E we further show by exhaustive computation of  $\mathbb{P}(\mathcal{G}$  obeys  $\mathbf{C})$  that its value is negligible even for smaller scale graphs. Thus, the probability of sampling a graph structure for which the gUFM minimizers exhibit exact collapse is practically 0.
(d) NCI: H  $\hat{\mathbf{A}}$
Figure 3: gUFM for  $\psi_{\Theta}^{\mathcal{F}'}$ : Illustration of loss, overlap, and  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$  during training on 10 SSBM graphs satisfying condition  $\mathbf{C}$ .
(c) NC1: H
(d) NC1: H  $\hat{\mathbf{A}}$
Figure 4: gUFM for  $\psi_{\Theta}^{\mathcal{F}'}$ : Illustration of loss, overlap, and  $\mathcal{NC}_1$  plots for  $\mathbf{H}$ ,  $\mathbf{H}\widehat{\mathbf{A}}$  during training on 10 SSBM graphs which do not satisfy condition  $\mathbf{C}$ .
gUFM experiments. For a better understanding of these results, we present small-scale experiments using the gUFM model on graphs that satisfy and do not satisfy condition  $\mathbf{C}$ . By training the gUFM (based on  $\psi_{\Theta}^{\mathcal{F}}$ ) on  $K = 10$  graphs that satisfy condition  $\mathbf{C}$ , we can observe from Figure 3 that NC1 metrics on  $\mathbf{H}, \mathbf{H}\hat{\mathbf{A}}$  reduce significantly. On the other hand, these metrics plateau after sufficient reduction when the graphs fail to satisfy condition  $\mathbf{C}$ , as shown in Figure 4. In both the cases, the SSBM parameters are  $C = 2$ ,  $N = 1000$ ,  $p = 0.025$ ,  $q = 0.0017$ , and the gUFM is trained using plain gradient descent for 50000 epochs with a learning rate of 0.1 and L2 regularization parameters  $\lambda_{W_1} = \lambda_{W_2} = \lambda_H = 5 \times 10^{-3}$ . Extensive experiments with varying choices of  $N, C, p, q$ , feature transformation based on  $\psi_{\Theta}^{\mathcal{F}}$  and additional NC metrics are provided in Appendix H. (The additional NC metrics measure the alignment of the classes' mean features with simplex Equiangular Tight Frame (ETF) and Orthogonal Frame (OF) structures.)
Remark. Note that previous papers consider UFM configurations for which the minimizers possess exact NC, typically without any condition on the number of samples or on the hyperparameters of the settings. As the UFMs are "optimistic" models, in the sense that they ignore all the limitations on modifying the features that exist in the training of practical DNNs, such results can be understood as "zero-order" reasoning for practical NC behavior. On the other hand, here we show that even the optimistic gUFM will not yield perfectly collapsed minimizers for graph structures that are not rare. This provides a purer understanding of the gaps in GNNs' features from exact collapse and why these gaps are larger than for plain DNNs. We also highlight the observation that condition  $C$  applies to homophilic as well as heterophilic graphs, as the constraint on neighborhood ratios is independent of label similarity. Thus providing insights on the effectiveness of GNNs on highly heterophilic graphs as empirically observed by Ma et al. [40].
Gradient flow: By now, we have provided a theory for the distinction between the deepest features of GNNs and plain DNNs. Next, to provide reasoning for the partial collapse in GNNs, which is observed empirically, we turn to study the gradient dynamics of our gUFM.
We consider the  $K = 1$  case and, following the common practice [25, 56], analyze the gradient flow along the "central path" — i.e., when  $\mathbf{W}_2 = \mathbf{W}_2^* (\mathbf{H})$  is the optimal minimizer of  $\widehat{\mathcal{R}}^{\mathcal{F}'}(\mathbf{W}_2,\mathbf{H})$
<sup>5</sup>Non-degenerate minimizers in the sense that  $\mathbf{W}_2^*\mathbf{H}^* \in \mathbb{R}^{C \times N}$  is full-rank. This eliminates degenerate 'zero'-solutions which are obtained when the regularization hyper-parameters are large.
Each term in each of the sums is the  $n^{th}$  power of a number smaller than 1 (a binomial probability).
w.r.t.  $\mathbf{W}_2$ , which has a closed-form expression as a function of  $\mathbf{H}$ . The resulting gradient flow is:
$$
\frac {d \mathbf {H} _ {t}}{d t} = - \nabla \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} \left(\mathbf {W} _ {2} ^ {*} \left(\mathbf {H} _ {t}\right), \mathbf {H} _ {t}\right). \tag {14}
$$
Similarly to [25, 56], we aim to gain insights on the evolution of  $\pmb{\Sigma}_{W}(\mathbf{H}_{t})$  and  $\pmb{\Sigma}_{B}(\mathbf{H}_{t})$  (in particular, their traces) along this flow. Yet, the presence of the structure matrix  $\widehat{\mathbf{A}}$  significantly complicates the analysis compared to existing works (which are essentially restricted to  $\widehat{\mathbf{A}} = \mathbf{I}$ ). Accordingly, we focus on the case of two classes,  $C = 2$ , and adopt a perturbation approach, analyzing the flow for a graph  $\widehat{\mathbf{A}} = \mathbb{E}\widehat{\mathbf{A}} +\mathbf{E}$ , where the expectation is taken with respect to the SSBM distribution and  $\mathbf{E}$  is a sufficiently small perturbation matrix. Our results are stated in the following theorem.
Theorem 3.3. Let  $K = 1$ ,  $C = 2$  and  $\lambda_{W_2} > 0$ . There exist  $\alpha > 0$  and  $E > 0$ , such that for  $0 < \lambda_H < \alpha$  and  $0 < \| \mathbf{E} \| < E$ , along the gradient flow stated in (14) associated with the graph  $\widehat{\mathbf{A}} = \mathbb{E}\widehat{\mathbf{A}} + \mathbf{E}$ , we have that: (1)  $\mathrm{Tr}(\Sigma_W(\mathbf{H}_t))$  decreases, and (2)  $\mathrm{Tr}(\Sigma_B(\mathbf{H}_t))$  increases. Accordingly,  $\widetilde{\mathcal{NC}}_1(\mathbf{H}_t)$  decreases.
The proof is presented in Appendix F. The importance of the theorem comes from showing that even graphs that do not satisfy condition  $\mathbf{C}$  (in the context of the analysis: perturbations around  $\mathbb{E}\hat{\mathbf{A}}$ ) exhibit reduction in the within-class covariance and increase in the between-class covariance of the features. This implies a reduction of NC1 metrics (to some extent), which is aligned with the empirical results in Section 3.1. Additionally, we highlight that since an increase in  $\mathrm{Tr}(\Sigma_B(\mathbf{H}_t))$  and decrease in  $\mathrm{Tr}(\Sigma_W(\mathbf{H}_t))$  is desirable for NC, this behavior of the penultimate layer's features can potentially serve as a remedy for the over-smoothing problem in GNNs (more details in Appendix A).
4 Feature separation across layers during inference
Till now, we have analyzed the feature evolution of the deepest GNN layer during training. In this section, we use these well-trained GNNs to classify nodes in unseen SSBM graphs and explore the depthwise evolution of features. In essence, we take an NC perspective on characterizing the weights of these well-trained networks that facilitate good generalization. To this end, we present empirical results demonstrating a gradual decrease of NC1 metrics along the network's depth. The observations hold a resemblance to the case with plain DNNs (shown empirically in [20, 55] and more recently in [26], and theoretically in [56]). To gain insights into this depthwise behavior we also compare it with the behavior of spectral clustering methods along their projected power iterations.
4.1 Experiments
Setup. We consider the 32-layered networks  $\psi_{\Theta}^{\mathcal{F}}, \psi_{\Theta}^{\mathcal{F}'}$  which have been designed and trained as per the setup in section 3.1 and have reached TPT. These networks are now tested on a dataset of  $K = 100$  unseen random SSBM graphs with  $C = 2, N = 1000, p = 0.025, q = 0.0017$ . Additionally, we perform spectral clustering using projected power iterations on the Normalized Laplacian (NL) and Bethe-Hessian (BH) matrices [51] for each of the test graphs. The motivation behind this approach is to obtain an approximation of the Fiedler vector of NL/BH that sheds light on the hidden community structure [1, 4, 46, 67]. Formally, for a test graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ , the NL and BH matrices are given by:
$$
\mathrm {N L} (\mathcal {G}) = \mathbf {I} - \mathbf {D} ^ {- 1 / 2} \mathbf {A} \mathbf {D} ^ {- 1 / 2}, \tag {15}
$$
$$
\mathrm {B H} (\mathcal {G}, r) = (r ^ {2} - 1) \mathbf {I} - r \mathbf {A} + \mathbf {D}, \tag {16}
$$
where  $r \in \mathbb{R}$  is the BH scaling factor. Now, by treating  $\mathbf{B}$  to be either NL or BH matrix, a projected power iteration to estimate the second largest eigenvector of  $\tilde{\mathbf{B}} = \| \mathbf{B}\| \mathbf{I} - \mathbf{B}$  is given by:
$$
\mathbf {x} ^ {(l)} = \widetilde {\mathbf {B}} \mathbf {w} ^ {(l - 1)}, \quad \text {w h e r e} \quad \mathbf {w} ^ {(l - 1)} = \frac {\mathbf {x} ^ {(l - 1)} - \langle \mathbf {x} ^ {(l - 1)} , \mathbf {v} \rangle \mathbf {v}}{\left\| \mathbf {x} ^ {(l - 1)} - \langle \mathbf {x} ^ {(l - 1)} , \mathbf {v} \rangle \mathbf {v} \right\| _ {2}}, \tag {17}
$$
with the vector  $\mathbf{v} \in \mathbb{R}^N$  denoting the largest eigenvector of  $\widetilde{\mathbf{B}}$ . Thus, we start with a random normal vector  $\mathbf{w}^0 \in \mathbb{R}^N$  and iteratively compute the feature vector  $\mathbf{x}^{(l)} \in \mathbb{R}^N$ , which represents the 1-D feature for each node after  $l$  iterations<sup>7</sup>.
<sup>7</sup>Interestingly, the connection between message passing in GNNs and power iterations without the normalization has been explored in [36]. However, projection and normalization are paramount to our setup (with random features) for approximating the Fiedler vector.
(a) Normalized Laplacian
(b) Bethe Hessian
(c) GNN  $\psi_{\Theta}^{\mathcal{F}}$
(d) GNN  $\psi_{\mathfrak{S}}^{\mathcal{F}'}$
(a) Normalized Laplacian
Figure 6: Ratio of traces of covariance matrices across projected power iterations for NL and BH (a,b), and across layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d).
Figure 5:  $\mathcal{NC}_1(\mathbf{H}),\widetilde{\mathcal{NC}}_1(\mathbf{H})$  metrics (top) and traces of covariance matrices (bottom) across projected power iterations for NL and BH (a,b), and across layers for GNNs  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (c,d).
(d) GNN  $\psi_{\mathfrak{G}}^{\mathcal{F}'}$
4.2 Towards understanding depthwise behavior
From Figure 5, we can observe that the rate of decrease in NC1 metrics is much higher in  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$  (avg test overlap  $= 1$ ) when compared to the baseline spectral approaches (avg test overlap  $\mathrm{NL} = 0.04$ ,  $\mathrm{BH} = 0.15$ ) with random normal feature initialization. For  $\psi_{\Theta}^{\mathcal{F}}$  and  $\psi_{\Theta}^{\mathcal{F}'}$ , the NC1 metrics and traces of covariance matrices are tracked after each of the components of a layer: graph operator, ReLU and instance normalization. For spectral methods, the components are: the operator  $\widetilde{\mathbf{B}}$  and the normalization. Interestingly, this rate seems to be relatively higher in  $\psi_{\Theta}^{\mathcal{F}'}$  than in  $\psi_{\Theta}^{\mathcal{F}}$ , and the variance of metrics tends to reduce significantly across all the test graphs after a certain depth in  $\psi_{\Theta}^{\mathcal{F}'}$  and  $\psi_{\Theta}^{\mathcal{F}}$ . Intuitively, the presence of  $\mathbf{W}_1$  in  $\psi_{\Theta}^{\mathcal{F}}$  seems to delay this reduction across layers. On the other hand, owing to the non-parametric nature of the spectral approaches, observe that the ratios  $\mathrm{Tr}(\pmb{\Sigma}_B(\mathbf{x}^{(l)})) / \mathrm{Tr}(\pmb{\Sigma}_B(\mathbf{w}^{(l - 1)}))$ ,  $\mathrm{Tr}(\pmb{\Sigma}_W(\mathbf{x}^{(l)})) / \mathrm{Tr}(\pmb{\Sigma}_W(\mathbf{w}^{(l - 1)}))$  tend to be constant throughout all iterations. However, the GNNs behave differently as  $\mathrm{Tr}(\pmb{\Sigma}_B(\mathbf{X}^{(l)})) / \mathrm{Tr}(\pmb{\Sigma}_B(\mathbf{H}^{(l - 1)}))$ ,  $\mathrm{Tr}(\pmb{\Sigma}_W(\mathbf{X}^{(l)})) / \mathrm{Tr}(\pmb{\Sigma}_W(\mathbf{H}^{(l - 1)}))$  tend to decrease across depth (Figure 6).
For a better understanding of this phenomenon, we consider the case of  $C = 2$  (without loss of generality) and assume that the  $(l - 1)^{th}$ -layer features  $\mathbf{H}^{(l - 1)}$  of nodes belonging to class  $c = 1, 2$  are drawn from distributions  $\mathcal{D}_1, \mathcal{D}_2$  respectively. We do not make any assumptions on the nature of the distributions and simply consider  $\pmb{\mu}_1^{(l - 1)}, \pmb{\mu}_2^{(l - 1)} \in \mathbb{R}^{d_{l - 1}}$  and  $\pmb{\Sigma}_1^{(l - 1)}, \pmb{\Sigma}_2^{(l - 1)} \in \mathbb{R}^{d_{l - 1} \times d_{l - 1}}$  as their mean vectors and covariance matrices, respectively. In the following theorem, we present bounds on the ratio of traces of feature covariance matrices after the graph operator is applied.
Theorem 4.1. Let  $C = 2$ ,  $\lambda_i(\cdot), \lambda_{-i}(\cdot)$  indicate the  $i^{th}$  largest and smallest eigenvalue of a matrix,  $\beta_1 = \frac{p - q}{p + q}, \beta_2 = \frac{p}{n(p + q)}, \beta_3 = \frac{p^2 + q^2}{n(p + q)^2}$ , and denote
$$
\mathbf {T} _ {W} = \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)} + \beta_ {2} \left[ \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {1} ^ {* (l)} + \mathbf {W} _ {1} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)} \right] + \beta_ {3} \mathbf {W} _ {2} ^ {* (l) \top} \mathbf {W} _ {2} ^ {* (l)},
$$
$$
\mathbf {T} _ {B} = \left(\mathbf {W} _ {1} ^ {* (l)} + \beta_ {1} \mathbf {W} _ {2} ^ {* (l)}\right) ^ {\top} \left(\mathbf {W} _ {1} ^ {* (l)} + \beta_ {1} \mathbf {W} _ {2} ^ {* (l)}\right).
$$
Then, the ratios of traces  $\frac{\operatorname{Tr}(\pmb{\Sigma}_B(\mathbf{X}^{(l)}))}{\operatorname{Tr}(\pmb{\Sigma}_B(\mathbf{H}^{(l-1)}))}, \frac{\operatorname{Tr}(\pmb{\Sigma}_W(\mathbf{X}^{(l)}))}{\operatorname{Tr}(\pmb{\Sigma}_W(\mathbf{H}^{(l-1)}))}$  for layer  $l \in \{2, \dots, L\}$  of a network  $\psi_{\Theta}^{\mathcal{F}}$  are bounded as follows:
$$
\frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {- i} (\pmb {\Sigma} _ {B} (\mathbf {H} ^ {(l - 1)})) \lambda_ {i} (\mathbf {T} _ {B})}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} (\pmb {\Sigma} _ {B} (\mathbf {H} ^ {(l - 1)}))} \leq \frac {\mathrm {T r} (\pmb {\Sigma} _ {B} (\mathbf {X} ^ {(l)}))}{\mathrm {T r} (\pmb {\Sigma} _ {B} (\mathbf {H} ^ {(l - 1)}))} \leq \frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} (\pmb {\Sigma} _ {B} (\mathbf {H} ^ {(l - 1)})) \lambda_ {i} (\mathbf {T} _ {B})}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} (\pmb {\Sigma} _ {B} (\mathbf {H} ^ {(l - 1)}))},
$$
$$
\frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {- i} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)})\right) \lambda_ {i} \left(\mathbf {T} _ {W}\right)}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)})\right)} \leq \frac {\operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {X} ^ {(l)})\right)}{\operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)})\right)} \leq \frac {\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)})\right) \lambda_ {i} \left(\mathbf {T} _ {W}\right)}{\sum_ {i = 1} ^ {d _ {l - 1}} \lambda_ {i} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} ^ {(l - 1)}))\right)}.
$$
The proof is presented in Appendix G. To understand the implications of this result, first observe that by setting  $\mathbf{W}_1^* = \mathbf{0}$  and modifying  $\mathbf{T}_W = \beta_3\mathbf{W}_2^{*(l)\top}\mathbf{W}_2^{*(l)}$ ,  $\mathbf{T}_B = \beta_1^2\mathbf{W}_2^{*(l)\top}\mathbf{W}_2^{*(l)}$ , we can obtain a similar bound formulation for  $\psi_{\Theta}^{F'}$ . To this end, as  $\mathbf{T}_W, \mathbf{T}_B$  depend on the spectrum of  $\mathbf{W}_2^{*(l)}$ , the ratios  $\frac{\mathrm{Tr}(\boldsymbol{\Sigma}_B(\mathbf{X}^{(l)}))}{\mathrm{Tr}(\boldsymbol{\Sigma}_B(\mathbf{H}^{(l-1)}))}$ ,  $\frac{\mathrm{Tr}(\boldsymbol{\Sigma}_W(\mathbf{X}^{(l)}))}{\mathrm{Tr}(\boldsymbol{\Sigma}_W(\mathbf{H}^{(l-1)}))}$  are highly dependent on  $\beta_1, \beta_3$ . Notice that since  $\mathbf{W}_1^{*(l)\top}\mathbf{W}_1^{*(l)}$  in  $\mathbf{T}_W$  is not scaled by any factor that is inversely dependent on  $n$ , it tends to act as a spectrum controlling mechanism and the reduction in within-class variability of features in  $\psi_{\Theta}^{F'}$  is relatively slow when compared to  $\psi_{\Theta}^{F'}$ . Thus, justifying the empirical behavior that we observed in subplots 6c and 6d in Figure 6.
In this work, we studied the feature evolution in GNNs for inductive node classification tasks. Adopting a Neural Collapse (NC) perspective, we analyzed both empirically and theoretically the within- and between-class variability of features along the training epochs and along the layers during inference. We showed that a partial decrease in within-class variability (and NC1 metrics) is present in the GNNs' deepest features and provided theory that indicates that greater collapse is not expected when training GNNs on practical graphs (as it requires strict structural conditions). We also showed a depthwise decrease in variability metrics, which resembles the case with plain DNNs. Especially, by leveraging the analogy of feature transformation across layers in GNNs with spectral clustering along projected power iterations, we provided insights into this GNN behavior and distinctions between two GNN architectures.
Interestingly, the structural conditions on graphs for exact collapse, which we rigorously established in this paper, are aligned with those that have been empirically hypothesized to facilitate GNNs learning in [40] (outside the context of NC). As a direction for future research, one may try to use this connection to link NC behavior with the generalization performance of GNNs. Moreover, note that a reduction in NC1 metrics of the deepest features implies not only that the within-class variability decreases but also that the between-class variability is bounded from below. Therefore, methods that are based on promoting NC (e.g., by utilizing the established structural conditions) can potentially mitigate the over-smoothing problem in GNNs. See Appendix A for a formal statement on the relation of NC and over-smoothing and additional discussions on the potential usage of graph rewiring strategies for this goal.
The authors would like to thank Jonathan Niles-Weed, Soledad Villar, Teresa Huang, Zhengdao Chen, Lei Chen and the anonymous reviewers for informative discussions and feedback. The authors acknowledge the NYU High Performance Computing services for providing the computing resources to run the experiments reported in this manuscript. V.K and J.B are partially supported by NSF DMS 2134216, NSF CAREER CIF 1845360, NSF IIS 1901091, and the Alfred P Sloan Foundation. T.T is partially supported by ISF grant no. 1940/23.
A Discussion on Oversmoothing and Graph Rewiring
In this appendix, we briefly discuss oversmoothing and graph rewiring from a neural collapse perspective along with some limitations and scope for future efforts.
A.1 Oversmoothing
Repeated message-passing operations on the node features across the depth of a GNN lead to the 'oversmoothing' phenomenon, which leads to poor node classification performance [14, 15, 32, 47, 62, 64]. We adopt the formal definition of oversmoothing by Rusch et al. [50] to our setup as follows:
Definition A.1. (Oversmoothing): For an undirected, connected graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  with  $|\mathcal{V}| = N$  and  $l$ -th layer hidden features  $\mathbf{H}^l\in \mathbb{R}^{d_l\times N}$ , a function  $\mu :\mathbb{R}^{d_l\times N}\to \mathbb{R}_{\geq 0}$  is called a node-similarity measure if it satisfies:
1.  $\exists \mathbf{c} \in \mathbb{R}^{d_l}$  with  $\mathbf{H}_i = \mathbf{c}$  for all nodes  $i \in \mathcal{V} \iff \mu(\mathbf{H}) = 0$ , for  $\mathbf{H} \in \mathbb{R}^{d_l \times N}$ .
2.  $\mu (\mathbf{H} + \mathbf{T})\leq \mu (\mathbf{H}) + \mu (\mathbf{T}),$  for all  $\mathbf{H},\mathbf{T}\in \mathbb{R}^{d_l\times N}$
Oversmoothing with respect to  $\mu$  is now defined as the layer-wise exponential convergence of the node-similarity measure  $\mu$  to zero, i.e,
$$
\mu (\mathbf {H} ^ {l}) \leq C _ {1} e ^ {- C _ {2} l}, f o r l = 1, \dots , L w i t h s o m e c o n s t a n t s C _ {1}, C _ {2} > 0.
$$
Observe from the definition that as the depth  $l$  increases, all the node features converge to a single feature vector  $\mathbf{c}$ . This implies that both  $\Sigma_W(\mathbf{H}^{L-1}), \Sigma_B(\mathbf{H}^{L-1}) \to 0$ . In this context, if the penultimate layer features exhibit neural collapse during training, then  $\Sigma_W(\mathbf{H}^{L-1})$  decreases, and  $\Sigma_B(\mathbf{H}^{L-1})$  is bounded from below. Thus, potentially addressing the oversmoothing problem.