Title: A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural collapse

Graph neural networks

Community detection
Abstract: Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on node-wise classification, illustrated with community detection on stochastic block model graphs, and explore the feature evolution through the lens of the "Neural Collapse" (NC) phenomenon. When training instance-wise deep classifiers (e.g. for image classification) beyond the zero training error point, NC demonstrates a reduction in the deepest features' within-class variability and an increased alignment of their class means to certain symmetric structures. We start with an empirical study that shows that a decrease in within-class variability is also prevalent in the node-wise classification setting, however, not to the extent observed in the instance-wise case. Then, we theoretically study this distinction. Specifically, we show that even an "optimistic" mathematical model requires that the graphs obey a strict structural condition in order to possess a minimizer with exact collapse. Furthermore, by studying the gradient dynamics of this model, we provide reasoning for the partial collapse observed empirically. Finally, we present a study on the evolution of within- and between-class feature variability across layers of a well-trained GNN and contrast the behavior with spectral methods.

A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks

Abstract

Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in GNNs is not well understood. In this paper, we focus on nodewise classification, illustrated with community detection on stochastic block model graphs, and explore the feature evolution through the lens of the 'Neural Collapse' (NC) phenomenon. When training instance-wise deep classifiers (e.g. for image classification) beyond the zero training error point, NC demonstrates a reduction in the deepest features' within-class variability and an increased alignment of their class means to certain symmetric structures. We start with an empirical study that shows that a decrease in within-class variability is also prevalent in the node-wise classification setting, however, not to the extent observed in the instance-wise case. Then, we theoretically study this distinction. Specifically, we show that even an 'optimistic' mathematical model requires that the graphs obey a strict structural condition in order to possess a minimizer with exact collapse. Interestingly, this condition is viable also for heterophilic graphs and relates to recent empirical studies on settings with improved GNNs' generalization. Furthermore, by studying the gradient dynamics of the theoretical model, we provide reasoning for the partial collapse observed empirically. Finally, we present a study on the evolution of within- and between-class feature variability across layers of a well-trained GNN and contrast the behavior with spectral methods.

1 Introduction

Graph neural networks [52] employ message-passing mechanisms to capture intricate topological relationships in data and have become de-facto standard architectures to handle data with nonEuclidean geometric structure [11, 12, 22, 24, 33, 60, 63, 66, 70]. However, the influence of topological information on feature learning in GNNs is yet to be fully understood [40, 64, 69, 72].

In this paper, we study the feature evolution in GNNs in a node-wise supervised classification setting. In order to gain insights into the role of topology, we focus on the controlled environment of the prominent stochastic block model (SBM) [1-3, 29, 45]. The SBM provides an effective framework to control the level of sparsity, homophily, and heterophily in the random graphs and facilitates analysis of GNN which relies solely on structural information [8, 16, 32, 40, 42, 49]. While inductive supervised learning on graphs is a relatively more difficult problem than transductive learning, it aligns with practical scenarios where nodes need to be classified in unseen graphs [24], and is also amenable to training GNNs that are deeper than conventional shallow Graph Convolution Network (GCN) models [14, 16, 32, 35, 47, 62, 62, 70].

The empirical and theoretical study of GNNs' feature evolution in this paper employs a 'Neural Collapse' perspective [48]. When training Deep Neural Networks (DNNs) for classification, it

is common to continue optimizing the networks' parameters beyond the zero training error point [9, 28, 39], a stage that was referred to in [48] as the 'terminal phase of training' (TPT). Papyan, Han, and Donoho [25, 48] have empirically shown that a phenomenon, dubbed Neural Collapse (NC), occurs during the TPT of plain DNNs 2 on standard instance-wise classification datasets. NC encompasses several simultaneous properties: (NC1) The within-class variability of the deepest features decreases (i.e., outputs of the penultimate layer for training samples from the same class tend to their mean); (NC2) After subtracting their global mean, the mean features of different classes become closer to a geometrical structure known as a simplex equiangular tight frame; (NC3) The last layer's weights exhibit alignment with the classes' mean features. A consequence of NC1-3 is that the classifier's decision rule becomes similar to the nearest class center in the feature space. We refer to [34] for a review on this topic.

The common approach to theoretically study the NC phenomenon is the 'Unconstrained Features Model' (UFM) [31, 43]. The core idea behind this 'optimistic' mathematical model is that the deepest features are considered to be freely optimizable. This idea has facilitated a recent surge of theoretical works in an effort to understand the global optimality conditions and gradient dynamics of these features and the last layer's weights in DNNs [18, 25, 37, 43, 54-56, 61, 65, 71, 74]. In our work, we extend NC analysis to settings where relational information in data is paramount, and creates a tension with the 'freeness' associated with the UFM model. In essence, we highlight the key differences when analyzing NC in GNNs by identifying structural conditions on the graphs, under which the global minimizers of the training objective exhibit full NC1. Interestingly, the structural conditions that we rigorously establish in this paper are aligned with the neighborhood conditions on heterophilic graphs that have been empirically hypothesized to facilitate learning by Ma et al. [40].

Our main contributions can be summarized as follows:

We conduct an extensive empirical study that shows that a decrease in within-class variability is prevalent also in the deepest features of GNNs trained for node classification on SBMs. However, not to the extent observed in the instance-wise setting.

We propose and analyze a graph-based UFM to understand the role of node neighborhood patterns and their community labels on NC dynamics. We prove that even this optimistic model requires a strict structural condition on the graphs in order to possess a minimizer with exact variability collapse. Then, we show that satisfying this condition is a rare event, which theoretically justifies the distinction between observations for GNNs and plain DNNs.

Nevertheless, by studying the gradient dynamics of the graph-based UFM, we provide theoretical reasoning for the partial collapse during GNNs training.

Finally, we study the evolution of features across the layers of well-trained GNNs and contrast the decrease in NC1 metrics along depth with a NC1 decrease along power iterations in spectral clustering methods.

2 Preliminaries and Problem Setup

We focus on supervised learning on graphs for inductive community detection. Formally, we consider a collection of K undirected graphs {G k = ( V k , E k ) } K k =1 , each with N nodes, C non-overlapping balanced communities and a node labelling ground truth function y k : V k →{ e 1 , . . . , e C } . Here, ∀ c ∈ [ C ] , e c ∈ R C indicates the standard basis vector, where we use the notation [ C ] = { 1 , · · · , C } . The goal is to learn a parameterized GNN model ψ Θ ( . ) which minimizes the empirical risk given by:

where ∥·∥ F represents the Frobenius norm, L is the loss function that is invariant to label permutations [16], and λ > 0 is the penalty parameter. We choose L based on the mean squared error (MSE) as:

where π belongs to the permutation group over C elements. Using the MSE loss for training DNN classifiers has become increasingly popular recently. For example, Hui and Belkin [30] have

performed an extensive empirical study that shows that training with MSE loss yields performance that is similar to (and sometimes even better than) training with CE loss. This choice also facilitates theoretical analyses [25, 55, 71].

2.1 Data model

We employ the Symmetric Stochastic Block Model (SSBM) to generate graphs {G k = ( V k , E k ) } k =1 . Stochastic block models (originated in [29]) are classical random graph models that have been extensively studied in statistics, physics, and computer science. In the SSBM model that is considered in this paper, each graph G k is associated with an adjacency matrix A k ∈ R N × N , degree matrix D k = diag( A k 1 ) ∈ R N × N , and a random node features matrix X k ∈ R d × N , with entries sampled from a normal distribution. Formally, if P ∈ R C × C represents a symmetric matrix with diagonal entries p and off-diagonal entries q , a random graph G k is considered to be drawn from the distribution SSBM ( N,C,p,q ) if an edge between vertices v i , v j is formed with probability ( P ) y k ( v i ) ,y k ( v j ) 3 . We choose the regime of exact recovery [1-3, 45] in sparse graphs where p = a ln( N ) N , q = b ln( N ) N for parameters a, b ≥ 0 such that | √ a -√ b | > √ C . The need for exact recovery (informationtheoretically) stems from the requirement that ψ Θ should be able to reach TPT (Appendix B).

2.2 Graph neural networks

Inspired by the widely studied model of higher-order GNNs by Morris et al. [44], we design ψ Θ based on a family of graph operators F = { I , ̂ A k } , ∀ k ∈ [ K ] , and denote it as ψ F Θ . Formally, for a GNN ψ F Θ with L layers, the node features H ( l ) k ∈ R d l × N at layer l ∈ [ L ] is given by:

where H (0) k = X k , and σ ( · ) represents a point-wise activation function such as ReLU. W ( l ) 1 , W ( l ) 2 ∈ R d l × d l -1 are the weight matrices and ̂ A k = A k D -1 k is the normalized adjacency matrix, also known as the random-walk matrix. We also consider a simpler family without the identity operator F ′ = { ̂ A k } , ∀ k ∈ [ K ] and analyze the GNN ψ F ′ Θ with only graph convolution functionality. Formally, the node features H ( l ) k ∈ R d l × N for ψ F ′ Θ is given by:

Here, the subscript for the weight matrix W ( l ) 2 is retained to highlight that it acts on H ( l -1) k ̂ A k . Finally, we employ the training strategy of Chen et al. [16] and apply instance-normalization [58] on σ ( X ( l ) k ) , ∀ l ∈ { 1 , · · · , L -1 } to prevent training instability.

2.3 Tracking neural collapse in GNNs

In our setup, reaching zero training error (TPT) implies that the network perfectly classifies all the nodes (up to label permutations) in all the training graphs. To this end, we leverage the NC metrics introduced in [48, 55, 56, 74] and extend them to GNNs in an inductive setting. To begin with, let us consider a single graph G k = ( V k , E k ) , k ∈ [ K ] with a normalized adjacency matrix ̂ A k . Additionally, we denote H ( l ) k ∈ R d l × N as the output of layer l ∈ [ L -1] , irrespective of the GNN design. Now, by dropping the subscript and superscript for notational convenience, we define the class means and the global mean of H as follows:

where n = N/C represents the number of nodes in each of the C balanced communities, and h c,i is the feature vector (a column in H ) associated with v c,i ∈ V , i.e., the i th node belonging to class

K

c ∈ [ C ] . Next, let N ( v c,i ) denote all the neighbors of v c,i and let N c ′ ( v c,i ) denote only the neighbors of v c,i that belong to class c ′ ∈ [ C ] . We define the class means and global mean of H ̂ A , which is unique to the GNN setting as follows:

Variability collapse in features H : For a given features matrix H , let us define the within- and between-class covariance matrices, Σ W ( H ) and Σ B ( H ) , as:

To empirically track the within-class variability collapse with respect to the between-class variability, we define two NC1 metrics:

where † denotes the Moore-Penrose pseudo-inverse and Tr( · ) denotes the trace of a matrix. Although NC 1 is the original NC1 metric used by Papyan et al. [48], we consider also ˜ NC 1 , which has been proposed by Tirer et al. [56] as an alternative metric that is more amenable to theoretical analysis.

Variability collapse in neighborhood-aggregated features H ̂ A : Similarly to the above, we track the within- and between-class variability of the 'neighborhood-aggregated' features matrix H ̂ A by Σ W ( H ̂ A ) and Σ B ( H ̂ A ) (computed using h N c and h N G ), as well as NC 1 ( H ̂ A ) and ˜ NC 1 ( H ̂ A ) . (See Appendix C for formal definitions.) Finally, we follow a simple approach and track the mean and variance of NC 1 ( H ) , ˜ NC 1 ( H ) , NC 1 ( H ̂ A ) , ˜ NC 1 ( H ̂ A ) across all K graphs in our experiments.

As the primary focus of our paper is the analysis of feature variability during training and inference, we defer the definition and examination of metrics based on NC2 and NC3 to Appendix C, H.

3 Evolution of penultimate layer features during training

In this section, we explore the evolution of the deepest features of GNNs during training. In Section 3.1, we present empirical results of GNNs in the setup that is detailed in Section 2, showing that a decrease in within-class feature variability is present in GNNs that reach zero training error, but not to the extent observed with plain DNNs. Then, in Section 3.2, we theoretically study a mathematical model that provides reasoning for the empirical observations.

3.1 Experiments

Setup. We focus on the training performance of GNNs ψ F Θ , ψ F ′ Θ on sparse graphs and generate a dataset of K = 1000 random SSBM graphs with C = 2 , N = 1000 , p = 0 . 025 , q = 0 . 0017 . The networks ψ F Θ , ψ F ′ Θ are composed of L = 32 layers with graph operator, ReLU activation, and instance-normalization functionality. The hidden feature dimension is set to 8 across layers. They are trained for 8 epochs using stochastic gradient descent (SGD) with a learning rate 0 . 004 , momentum 0 . 9 , and a weight decay of 5 × 10 -4 . During training, we track the NC1 metrics for the penultimate layer features H ( L -1) k , by computing their mean and standard deviation across k ∈ [ K ] graphs after every epoch. To measure the performance of the GNN, we compute the 'overlap' [16] between predicted communities and ground truth communities (up to permutations):

(a) loss

(b) overlap

(c) NC1:

H

(d) NC1:

H

̂

A

1.0

0.5

Tr(Ew)

0.6

Tr(Eg)

0.5

0.8

0.5

scale)

0.0

Tr(Ew)/Tr(Eg)

Tr(Ew²)/C

0.4

overlap

0.0

0001

-0.5

0.3

0.6

-0.5

(HA)

-1.0

(H）

0.2

0.4

NC

-1.0

Tr(Ew)

NC

-1.5

0.1

0.0

0

2

iter%1000

4

6

8

0.2

0

2

iter%1000

4

6

8

1.5

0

Tr（Zw=²)/C

Tr(Ew)/Tr(∑g)

2

iter%1000

4

6

8

-2.0

0

2

iter%1000

4

6

8

(a) loss

(b) overlap

(c) NC1:

H

(d) NC1:

H

̂

A

0.6

1.00

Tr(Ew)

0.3

0.5

0.2

0.4

overlap

0.97

0.96

0.99

0.98

g10

(H）

NC

scal

[601)

-0.5

1.5

-1.0

0.5

0.0

Tr(Ew)

(H)

scale)

10

601)

-0.5

-1.0

-1.5

0.5

0.0

Tr(Ew)/Tr(Ea)

Tr(Ew²)/C

Tr(Eg)

0.1

0.95

-2.0

Tr(Eg)

Tr(EwEg²)/C

-2.0

0.0

0.94

Tr(Ew)/Tr(Eg)

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

where ˆ y is the node labelling function based on GNN design and 1 N ∑ N i =1 δ ˆ y ( v i ) ,π ( y ( v i )) is the training accuracy ( δ denotes the Kronecker delta). The overlap allows us to measure the improvements in performance over random guessing while retaining the indication that the GNN has reached TPT. Formally, when 1 N ∑ N i =1 δ ˆ y ( v i ) ,π ( y ( v i )) = 1 (zero training error), then overlap(ˆ y, y ) = 1 . We illustrate the empirical results in Figures 1 and 2, and present extensive experiments (showing similar behavior) along with infrastructure details in Appendix H 4 .

Observation: The key takeaway is that NC 1 ( H ( L -1) k ) , ˜ NC 1 ( H ( L -1) k ) tend to reduce and plateau during TPT in ψ F Θ and ψ F ′ Θ . Notice that even though we consider a controlled SSBM-based setting, the NC 1 values observed here are higher than the values observed in the case of plain DNNs on realworld instance-wise datasets [48, 74]. Additionally, we can observe that trends for NC 1 ( H ( L -1) k ̂ A k ) , ˜ NC 1 ( H ( L -1) k ̂ A k ) are similar to those of NC 1 ( H ( L -1) k ) , ˜ NC 1 ( H ( L -1) k ) .

3.2 Theoretical analysis

In this section, we provide a theory for this empirical behavior. Most, if not all, of the theoretical papers on NC, adopt the UFM approach, which treats the features as free optimization variables disconnected from data [18, 25, 43, 55, 56, 74]. Here, we consider a graph-based adaptation of this approach, that we dubbed as gUFM. We consider GNNs of the form of ψ F ′ Θ , which is more tractable for mathematical analysis. Formally, by considering L to be the MSE loss, treating { H ( L -1) k } K k =1 as freely optimizable variables, and representing W ( L ) 2 ∈ R C × d L -1 , H ( L -1) k ∈ R d L -1 × N as W 2 , H k (for notational convenience), the empirical risk based on the gUFM can be formulated as follows:

where Y ∈ R C × N is the target matrix, which is composed of one-hot vectors associated with the different classes, and λ W 2 , λ H k > 0 are regularization hyperparameters. To simplify the analysis, let us assume that Y = I C ⊗ 1 ⊤ n , where ⊗ denotes the Kronecker product. Namely, the training data is balanced (a common assumption in UFM-based analyses in literature) with n = N/C nodes per

class in each graph and (without loss of generality) organized class-by-class. Note that for K = 1 (which allows omitting the graph index k ) and no graphical structure, i.e., ̂ A = I (since A = I ), (11) reduces to the plain UFM that has been studied in [25, 55, 71]. In this case, it has been shown that any minimizer ( W ∗ 2 , H ∗ ) is collapsed , i.e., its features have exactly zero within-class variability:

which implies Σ W ( H ∗ ) = 0 . We will show now that the situation in gUFM is significantly different.

Considering the K = 1 case, we start by showing that, to have minimizers of (11) that possess the property in (12), the graph must obey a strict structural condition. For K > 1 , having a minimizer ( W ∗ 2 , { H ∗ k } ) where, for some j ∈ [ K ] , H ∗ j is collapsed directly follows from having the structural condition satisfied by the j -th graph (as shown in our proof, the sufficiency of the condition does not depend on the shared weights W 2 ). On the other hand, generalizing the necessity of the structural condition to the case of K > 1 is technically challenging (see the appendix for details). For that reason, we state the condition in the following theorem only for K = 1 . Note also that, showing that the condition is unlikely to be satisfied per graph is enough for explaining the plateaus above zero of NC metrics (computed over multiple graphs), which are demonstrated in Section 3.1.

Theorem 3.1. Consider the gUFM in (11) with K = 1 and denote the fraction of neighbors of node v c,i that belong to class c ′ as s cc ′ ,i = |N c ′ ( v c,i ) | |N ( v c,i ) | . Let the condition C based on s cc ′ ,i be given by:

If a graph G satisfies condition C , then there exist minimizers of the gUFM that are collapsed (satisfying (12)). Conversely, when either √ λ H λ W 2 = 0 , or √ λ H λ W 2 > 0 and G is regular (so that ̂ A = ̂ A ⊤ ), if there exists a collapsed non-degenerate minimizer 5 of gUFM, then condition C necessarily holds.

Remark: The proof is presented in Appendix D. The symmetry assumption on ̂ A (which implies that G is a regular graph) in the second part of the theorem has been made to pass technical obstacles in the proof rather than due to a true limitation. Thus, together with the results of our experiments (where no symmetry is enforced), we believe that this assumption can be dropped. Accordingly, we state the following conjecture.

Conjecture 3.1. Consider the gUFM in (11) with K = 1 and condition C as stated in theorem 3.1. The minimizers of the gUFM are collapsed (satisfying (12)) iff the graph G satisfies condition C .

Let us dwell on the implication of Theorem 3.1. The stated condition C essentially holds when any node i ∈ [ n ] of a certain class c obeys ( s c 1 ,i , · · · , s cC,i ) = ( s c 1 , · · · , s cC ) for some ( s c 1 , · · · , s cC ) , a tuple of the ratio of neighbors ( ∑ C c ′ =1 s cc ′ = 1 ) independent of i . That is, ( s c 1 , · · · , s cC ) must be the same for nodes within the same class but can be different for nodes belonging to different classes. For example, for a plain UFM this condition trivially holds, as ̂ A = I . Under the SSBM distribution, it is also easy to see that E ̂ A satisfies this condition. However, for more practical graphs, such as those drawn from SSBM, the probability of having a graph that obeys condition C is negligible. This is shown in the following theorem.

Theorem 3.2. Let G = ( V , E ) be drawn from SSBM ( N,C,p,q ) . For N >> C , we have

The proof is presented in Appendix E. It is not hard to see that as the number of per-class nodes n increases, the probability of satisfying condition C decreases, 6 as numerically exemplified below.

Numerical example. Let's consider a setting with C = 2 , N = 1000 , a = 3 . 75 , b = 0 . 25 . This gives us n = N/C = 500 , p = 0 . 025 , q = 0 . 0017 , for which P ( G obeys C ) < 2 . 18 × 10 -188 .

In Appendix E we further show by exhaustive computation of P ( G obeys C ) that its value is negligible even for smaller scale graphs. Thus, the probability of sampling a graph structure for which the gUFM minimizers exhibit exact collapse is practically 0 .

(a) loss

(b) overlap

(c) NC1:

H

(d) NC1:

H

̂

A

20

1.0

0

0.8

(log10scale)

0

15

-5

10

10

-10

(H)

15

5

0.0

2.5

iter%50000

5.0

7.5

10.0

0.2

0.0

0.0

2.5

iter%50000

5.0

7.5

10.0

NC

-15

-20

0.0

Tr(zw)

Tr(Za)

Tr（zC

Tr(EwT(Ia)

2.5

iter%50000

5.0

7.5

10.0

NC

-20

0.0

Tr(Em)

Tr(za)

Tr（xC

Tr（2)/Tr(xa)

2.5

iter%50000

5.0

7.5

10.0

(a) loss

(b) overlap

(c) NC1:

H

(d) NC1:

H

̂

A

20

1.0

3

Tr(Ew)

Tr(Ea)

2

Tr(Eg)

Tr(Ew)

15

0.8

NC(H)(log10 scale)

2

Tr(Ew)/Tr(Eg)

Tr(Ew²)/C

NC(HA)(log10 scale)

1

Tr(Ewg²)C

Tr(Zw)/Tr(Zg)

overlap

0.6

1

0

0.4

0

1

-1

5

0.2

-2

0.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

-3

0.0

2.5

iter%50000

5.0

7.5

10.0

-4

0.0

2.5

iter%50000

5.0

7.5

10.0

gUFM experiments. For a better understanding of these results, we present small-scale experiments using the gUFM model on graphs that satisfy and do not satisfy condition C . By training the gUFM (based on ψ F ′ Θ ) on K = 10 graphs that satisfy condition C , we can observe from Figure 3 that NC1 metrics on H , H ̂ A reduce significantly. On the other hand, these metrics plateau after sufficient reduction when the graphs fail to satisfy condition C , as shown in Figure 4. In both the cases, the SSBM parameters are C = 2 , N = 1000 , p = 0 . 025 , q = 0 . 0017 , and the gUFM is trained using plain gradient descent for 50000 epochs with a learning rate of 0 . 1 and L2 regularization parameters λ W 1 = λ W 2 = λ H = 5 × 10 -3 . Extensive experiments with varying choices of N,C,p,q , feature transformation based on ψ F Θ and additional NC metrics are provided in Appendix H. (The additional NC metrics measure the alignment of the classes' mean features with simplex Equiangular Tight Frame (ETF) and Orthogonal Frame (OF) structures.)

Remark. Note that previous papers consider UFM configurations for which the minimizers possess exact NC, typically without any condition on the number of samples or on the hyperparameters of the settings. As the UFMs are 'optimistic' models, in the sense that they ignore all the limitations on modifying the features that exist in the training of practical DNNs, such results can be understood as 'zero-order' reasoning for practical NC behavior. On the other hand, here we show that even the optimistic gUFM will not yield perfectly collapsed minimizers for graph structures that are not rare. This provides a purer understanding of the gaps in GNNs' features from exact collapse and why these gaps are larger than for plain DNNs. We also highlight the observation that condition C applies to homophilic as well as heterophilic graphs , as the constraint on neighborhood ratios is independent of label similarity. Thus providing insights on the effectiveness of GNNs on highly heterophilic graphs as empirically observed by Ma et al. [40].

Gradient flow: By now, we have provided a theory for the distinction between the deepest features of GNNs and plain DNNs. Next, to provide reasoning for the partial collapse in GNNs, which is observed empirically, we turn to study the gradient dynamics of our gUFM.

We consider the K = 1 case and, following the common practice [25, 56], analyze the gradient flow along the 'central path' - i.e., when W 2 = W ∗ 2 ( H ) is the optimal minimizer of ̂ R F ′ ( W 2 , H )

w.r.t. W 2 , which has a closed-form expression as a function of H . The resulting gradient flow is:

Similarly to [25, 56], we aim to gain insights on the evolution of Σ W ( H t ) and Σ B ( H t ) (in particular, their traces) along this flow. Yet, the presence of the structure matrix ̂ A significantly complicates the analysis compared to existing works (which are essentially restricted to ̂ A = I ). Accordingly, we focus on the case of two classes, C = 2 , and adopt a perturbation approach, analyzing the flow for a graph ̂ A = E ̂ A + E , where the expectation is taken with respect to the SSBM distribution and E is a sufficiently small perturbation matrix. Our results are stated in the following theorem.

Theorem 3.3. Let K = 1 , C = 2 and λ W 2 > 0 . There exist α > 0 and E > 0 , such that for 0 < λ H < α and 0 < ∥ E ∥ < E , along the gradient flow stated in (14) associated with the graph ̂ A = E ̂ A + E , we have that: (1) Tr( Σ W ( H t )) decreases, and (2) Tr( Σ B ( H t )) increases. Accordingly, ˜ NC 1 ( H t ) decreases.

The proof is presented in Appendix F. The importance of the theorem comes from showing that even graphs that do not satisfy condition C (in the context of the analysis: perturbations around E ̂ A ) exhibit reduction in the within-class covariance and increase in the between-class covariance of the features. This implies a reduction of NC1 metrics (to some extent), which is aligned with the empirical results in Section 3.1. Additionally, we highlight that since an increase in Tr( Σ B ( H t )) and decrease in Tr( Σ W ( H t )) is desirable for NC, this behavior of the penultimate layer's features can potentially serve as a remedy for the over-smoothing problem in GNNs (more details in Appendix A).

4 Feature separation across layers during inference

Till now, we have analyzed the feature evolution of the deepest GNN layer during training. In this section, we use these well-trained GNNs to classify nodes in unseen SSBM graphs and explore the depthwise evolution of features. In essence, we take an NC perspective on characterizing the weights of these well-trained networks that facilitate good generalization. To this end, we present empirical results demonstrating a gradual decrease of NC1 metrics along the network's depth. The observations hold a resemblance to the case with plain DNNs (shown empirically in [20, 55] and more recently in [26], and theoretically in [56]). To gain insights into this depthwise behavior we also compare it with the behavior of spectral clustering methods along their projected power iterations.

4.1 Experiments

Setup. We consider the 32 -layered networks ψ F Θ , ψ F ′ Θ which have been designed and trained as per the setup in section 3.1 and have reached TPT. These networks are now tested on a dataset of K = 100 unseen random SSBM graphs with C = 2 , N = 1000 , p = 0 . 025 , q = 0 . 0017 . Additionally, we perform spectral clustering using projected power iterations on the Normalized Laplacian (NL) and Bethe-Hessian (BH) matrices [51] for each of the test graphs. The motivation behind this approach is to obtain an approximation of the Fiedler vector of NL/BH that sheds light on the hidden community structure [1, 4, 46, 67]. Formally, for a test graph G = ( V , E ) , the NL and BH matrices are given by:

where r ∈ R is the BH scaling factor. Now, by treating B to be either NL or BH matrix, a projected power iteration to estimate the second largest eigenvector of ˜ B = ∥ B ∥ I -B is given by:

with the vector v ∈ R N denoting the largest eigenvector of ˜ B . Thus, we start with a random normal vector w 0 ∈ R N and iteratively compute the feature vector x ( l ) ∈ R N , which represents the 1-D feature for each node after l iterations 7 .

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

4.5

Tr（w²）/C:Op

4.5

Tr(wg²/C:Op

Tr（Ew²)/C:Op

Tr（Ew²/C:Op

Tr（w²/C:normalize

4.0

Tr(Ewg/C:normalize

Tr（Ew²）/C:non-lin

Tr（Ewg²)/C:non-lin

4.0

Tr（∑w)/Tr(∑g）:Op

Tr（Zw)/Tr(zg):normalize

3.5

Tr(Ew)/Tr(Eg):Op

Tr（Ew)/Tr(Eg):normalize

NC（H）(log10 scale)

Tr（w²)/C:normalize

Tr（Ew)/Tr(∑g）:Op

Tr（Ew)/Tr(Eg）:Op

Tr（Ewg/C:normalize

3.5

Tr（Ew)/Tr(Eg):non-lin

Tr（Ew)/Tr(Eg）:non-lin

Tr(Ew)/Tr(Eg) : normalize

。

Tr(Ew)/Tr(Eg): normalize

。

1.5

2.0

1.0

-1

-2

1.5

0

5

10

Piidx

15

20

25

30

0.5

0

5

10

Plidx

15

20

25

30

-2

0

5

10

layeridx

15

20

2530

0

5

10

layer idx

15

20

25

30

)

2

D

Tr（Ew）:Op

Tr（Ew):normalize

Tr（Eg）:Op

Tr（Eg):normalize

2

6

Trace(log10 scale)

Tr(Ew):normalize

Tr(Ew):Op

Tr(Ew):non-lin

Tr(Eg）:Op

Tr(Eg):non-lin

(s

10

6

Tr(∑g):Op

Tr(Ew): non-lin

Tr(Ew):normalize

Tr(Eg):non-lin

Tr（∑w):Op

Tr（Ew):Op

Tr(Eg):normalize

Tr(Eg): normalize

-2

Tr(Eg):Op

Tr(Ew）:normalize

3

4

-4

-2

Tr（∑g）:normalize

2

2

-4

-6

-2

-8

0

5

10

PI idx

15

20

25

30

-8

0

5

10

PI idx

15

20

2530

-1

5

10

layeridx

1520

25

30

0

5

10

layer idx

15

20

25

30

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

3.15

Tr(2Op）/Tr（-1)

7.0

Tr（2Op）/T（zW-1)

Tr（Op）/T（u-1)

D

Tr(2p)/（W-1)

Traceratio(log10 scale)

3.10

Tr(2O)/（2-1)

Trace ratio(log10 scale)

6.8

Tr（2）/T（-1）)

E

Tr（）/T(-1）)

Trace ratio(log10 scale)

4

Tr（0）/T（-1)

3.05

6.6

3.00

2.95

6.4

2.90

6.2

0

5

10

layeridx

15

20

25

30

0

5

10

layeridx

15

20

25

30

5

10

layeridx

15

20

25

30

0

5

10

layeridx

15

20

25

30

4.2 Towards understanding depthwise behavior

From Figure 5, we can observe that the rate of decrease in NC1 metrics is much higher in ψ F Θ and ψ F ′ Θ (avg test overlap = 1 ) when compared to the baseline spectral approaches (avg test overlap NL = 0 . 04 , BH = 0 . 15 ) with random normal feature initialization. For ψ F Θ and ψ F ′ Θ , the NC1 metrics and traces of covariance matrices are tracked after each of the components of a layer: graph operator, ReLU and instance normalization. For spectral methods, the components are: the operator ˜ B and the normalization. Interestingly, this rate seems to be relatively higher in ψ F ′ Θ than in ψ F Θ , and the variance of metrics tends to reduce significantly across all the test graphs after a certain depth in ψ F ′ Θ and ψ F Θ . Intuitively, the presence of W 1 in ψ F Θ seems to delay this reduction across layers. On the other hand, owing to the non-parametric nature of the spectral approaches, observe that the ratios Tr ( Σ B ( x ( l ) )) / Tr ( Σ B ( w ( l -1) )) , Tr ( Σ W ( x ( l ) )) / Tr ( Σ W ( w ( l -1) )) tend to be constant throughout all iterations. However, the GNNs behave differently as Tr ( Σ B ( X ( l ) )) / Tr ( Σ B ( H ( l -1) )) , Tr ( Σ W ( X ( l ) )) / Tr ( Σ W ( H ( l -1) )) tend to decrease across depth (Figure 6).

For a better understanding of this phenomenon, we consider the case of C = 2 (without loss of generality) and assume that the ( l -1) th -layer features H ( l -1) of nodes belonging to class c = 1 , 2 are drawn from distributions D 1 , D 2 respectively. We do not make any assumptions on the nature of the distributions and simply consider µ ( l -1) 1 , µ ( l -1) 2 ∈ R d l -1 and Σ ( l -1) 1 , Σ ( l -1) 2 ∈ R d l -1 × d l -1 as their mean vectors and covariance matrices, respectively. In the following theorem, we present bounds on the ratio of traces of feature covariance matrices after the graph operator is applied.

Theorem 4.1. Let C = 2 , λ i ( · ) , λ -i ( · ) indicate the i th largest and smallest eigenvalue of a matrix, β 1 = p -q p + q , β 2 = p n ( p + q ) , β 3 = p 2 + q 2 n ( p + q ) 2 , and denote

Then, the ratios of traces Tr( Σ B ( X ( l ) )) Tr( Σ B ( H ( l -1) )) , Tr( Σ W ( X ( l ) )) Tr( Σ W ( H ( l -1) )) for layer l ∈ { 2 , · · · , L } of a network ψ F Θ are bounded as follows:

The proof is presented in Appendix G. To understand the implications of this result, first observe that by setting W ∗ 1 = 0 and modifying T W = β 3 W ∗ ( l ) ⊤ 2 W ∗ ( l ) 2 , T B = β 2 1 W ∗ ( l ) ⊤ 2 W ∗ ( l ) 2 , we can obtain a similar bound formulation for ψ F ′ Θ . To this end, as T W , T B depend on the spectrum of W ∗ ( l ) 2 , the ratios Tr( Σ B ( X ( l ) )) Tr( Σ B ( H ( l -1) )) , Tr( Σ W ( X ( l ) )) Tr( Σ W ( H ( l -1) )) are highly dependent on β 1 , β 3 . Notice that since W ∗ ( l ) ⊤ 1 W ∗ ( l ) 1 in T W is not scaled by any factor that is inversely dependent on n , it tends to act as a spectrum controlling mechanism and the reduction in within-class variability of features in ψ F Θ is relatively slow when compared to ψ F ′ Θ . Thus, justifying the empirical behavior that we observed in subplots 6c and 6d in Figure 6.

5 Conclusion

In this work, we studied the feature evolution in GNNs for inductive node classification tasks. Adopting a Neural Collapse (NC) perspective, we analyzed both empirically and theoretically the within- and between-class variability of features along the training epochs and along the layers during inference. We showed that a partial decrease in within-class variability (and NC1 metrics) is present in the GNNs' deepest features and provided theory that indicates that greater collapse is not expected when training GNNs on practical graphs (as it requires strict structural conditions). We also showed a depthwise decrease in variability metrics, which resembles the case with plain DNNs. Especially, by leveraging the analogy of feature transformation across layers in GNNs with spectral clustering along projected power iterations, we provided insights into this GNN behavior and distinctions between two GNN architectures.

Interestingly, the structural conditions on graphs for exact collapse, which we rigorously established in this paper, are aligned with those that have been empirically hypothesized to facilitate GNNs learning in [40] (outside the context of NC). As a direction for future research, one may try to use this connection to link NC behavior with the generalization performance of GNNs. Moreover, note that a reduction in NC1 metrics of the deepest features implies not only that the within-class variability decreases but also that the between-class variability is bounded from below. Therefore, methods that are based on promoting NC (e.g., by utilizing the established structural conditions) can potentially mitigate the over-smoothing problem in GNNs. See Appendix A for a formal statement on the relation of NC and over-smoothing and additional discussions on the potential usage of graph rewiring strategies for this goal.

Acknowledgments and Disclosure of Funding

The authors would like to thank Jonathan Niles-Weed, Soledad Villar, Teresa Huang, Zhengdao Chen, Lei Chen and the anonymous reviewers for informative discussions and feedback. The authors acknowledge the NYU High Performance Computing services for providing the computing resources to run the experiments reported in this manuscript. V.K and J.B are partially supported by NSF DMS 2134216, NSF CAREER CIF 1845360, NSF IIS 1901091, and the Alfred P Sloan Foundation. T.T is partially supported by ISF grant no. 1940/23.

References

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Emmanuel Abbe and Colin Sandon. Community detection in general stochastic block models: Fundamental limits and efficient algorithms for recovery. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 670-688. IEEE, 2015.

Emmanuel Abbe, Afonso S Bandeira, and Georgina Hall. Exact recovery in the stochastic block model. IEEE Transactions on information theory , 62(1):471-487, 2015.

Emmanuel Abbe, Jianqing Fan, Kaizheng Wang, and Yiqiao Zhong. Entrywise eigenvector analysis of random matrices with low expected rank. Annals of statistics , 48(3):1452, 2020.

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations , 2021.

Adrián Arnaiz-Rodrıguez, Ahmed Begga, Francisco Escolano, and Nuria M Oliver. Diffwire: Inductive graph rewiring via the lovász bound. In Learning on Graphs Conference , pages 15-1. PMLR, 2022.

Pradeep Kr Banerjee, Kedar Karhadkar, Yu Guang Wang, Uri Alon, and Guido Montúfar. Oversquashing in gnns through the lens of information contraction and graph expansion. In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-8. IEEE, 2022.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Effects of graph convolutions in deep networks. arXiv preprint arXiv:2204.09297 , 2022.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1611-1619. PMLR, 2019.

Ravi B Boppana. Eigenvalues and graph bisection: An average-case analysis. In 28th Annual Symposium on Foundations of Computer Science (sfcs 1987) , pages 280-285. IEEE, 1987.

Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine , 34 (4):18-42, 2017.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 , 2013.

Thang Bui, S. Chaudhuri, T. Leighton, and M. Sipser. Graph bisection algorithins with good average case behavior. In 25th Annual Symposium onFoundations of Computer Science, 1984. , pages 181-192, 1984. doi: 10.1109/SFCS.1984.715914.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3438-3445, 2020.

Zhengdao Chen, Joan Bruna, and Lisha Li. Supervised community detection with line graph neural networks. In 7th International Conference on Learning Representations (ICLR) , 2019.

Paul Erd˝ os, Alfréd Rényi, et al. On the evolution of random graphs. Publ. math. inst. hung. acad. sci , 5(1):17-60, 1960.

Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences , 118(43), 2021.

Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198 , 2020.

Tomer Galanti. A note on the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028 , 2022.

Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. Advances in neural information processing systems , 32, 2019.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Benjamin Gutteridge, Xiaowen Dong, Michael M Bronstein, and Francesco Di Giovanni. Drew: dynamically rewired message passing with delay. In International Conference on Machine Learning , pages 12252-12267. PMLR, 2023.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. arXiv preprint arXiv:2106.02073 , 2021.

Hangfeng He and Weijie J Su. A law of data separation in deep learning. arXiv preprint arXiv:2210.17020 , 2022.

Nicholas J Higham and Hyun-Min Kim. Numerical analysis of a quadratic matrix equation. IMA Journal of Numerical Analysis , 20(4):499-519, 2000.

Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. Advances in neural information processing systems , 30, 2017.

Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks , 5(2):109-137, 1983.

Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In The Ninth International Conference on Learning Representations (ICLR) , 2021.

Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:2110.02796 , 2021.

Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over) smoothing. arXiv preprint arXiv:2205.12156 , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=QTXocpAP9p .

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Xue Li and Yuanzhi Cheng. Understanding the message passing in graph neural networks via power iteration clustering. Neural Networks , 140:130-135, 2021.

Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis , 2022.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. Advances in neural information processing systems , 35:1362-1375, 2022.

Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning , pages 3325-3334. PMLR, 2018.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In International Conference on Learning Representations (ICLR) , 2022.

Albert W Marshall, Ingram Olkin, and Barry C Arnold. Inequalities: theory of majorization and its applications. 1979.

Nikhil Mehta, Lawrence Carin Duke, and Piyush Rai. Stochastic blockmodels meet graph neural networks. In International Conference on Machine Learning , pages 4466-4474. PMLR, 2019.

Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619 , 2020.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4602-4609, 2019.

Elchanan Mossel, Joe Neeman, and Allan Sly. Consistency thresholds for binary symmetric block models. arXiv preprint arXiv:1407.1591 , 3(5), 2014.

Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences , 103(23):8577-8582, 2006.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947 , 2019.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117 (40):24652-24663, 2020.

Luana Ruiz, Soledad Villar, et al. Graph neural networks for community detection on sparse graphs. arXiv preprint arXiv:2211.03231 , 2022.

T Konstantin Rusch, Michael Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. SAM Research Report , 2023, 2023.

Alaa Saade, Florent Krzakala, and Lenka Zdeborová. Spectral clustering of graphs with the bethe hessian. Advances in Neural Information Processing Systems , 27, 2014.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

Kathrin Schacke. On the kronecker product. Master's thesis, University of Waterloo , 2004.

Christos Thrampoulidis, Ganesh R Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. arXiv preprint arXiv:2208.05512 , 2022.

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning , pages 21478-21505. PMLR, 2022.

Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. arXiv preprint arXiv:2210.16658 , 2022.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations , 2022.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 , 2016.

Charles F Van Loan. The ubiquitous kronecker product. Journal of computational and applied mathematics , 123(1-2):85-100, 2000.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Stephan Wojtowytsch et al. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. Proceedings of Machine Learning Research , 145:1-21, 2021.

Xinyi Wu, Zhengdao Chen, William Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. arXiv preprint arXiv:2212.10701 , 2022.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 1287-1292. IEEE, 2022.

Yibo Yang, Liang Xie, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dacheng Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:2203.09081 , 2022.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33:5812-5823, 2020.

Se-Young Yun and Alexandre Proutiere. Accurate community detection in the stochastic block model via spectral algorithms. arXiv preprint arXiv:1412.7335 , 2014.

Fuzhen Zhang and Qingling Zhang. Eigenvalue inequalities for matrix product. IEEE Transactions on Automatic Control , 51(9):1506-1509, 2006.

Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 , 2022.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open , 1:57-81, 2020.

Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning , pages 27179-27202. PMLR, 2022.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11168-11176, 2021.

Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems , 34:29820-29834, 2021.

A Discussion on Oversmoothing and Graph Rewiring

In this appendix, we briefly discuss oversmoothing and graph rewiring from a neural collapse perspective along with some limitations and scope for future efforts.

A.1 Oversmoothing

Repeated message-passing operations on the node features across the depth of a GNN lead to the 'oversmoothing' phenomenon, which leads to poor node classification performance [14, 15, 32, 47, 62, 64]. We adopt the formal definition of oversmoothing by Rusch et al. [50] to our setup as follows:

Definition A.1. (Oversmoothing): For an undirected, connected graph G = ( V , E ) with |V| = N and l -th layer hidden features H l ∈ R d l × N , a function µ : R d l × N → R ≥ 0 is called a node-similarity measure if it satisfies:

Oversmoothing with respect to µ is now defined as the layer-wise exponential convergence of the node-similarity measure µ to zero, i.e,

Observe from the definition that as the depth l increases, all the node features converge to a single feature vector c . This implies that both Σ W ( H L -1 ) , Σ B ( H L -1 ) → 0 . In this context, if the penultimate layer features exhibit neural collapse during training, then Σ W ( H L -1 ) decreases, and Σ B ( H L -1 ) is bounded from below. Thus, potentially addressing the oversmoothing problem.

To this end, in the unconstrained features setting, Theorem 3.1 indicates that the graph must satisfy a strict structural condition for having an exact neural collapse solution as a minimizer of the empirical risk. Additionally, we obtained conditions on the amount of regularization needed for Σ B ( H L -1 ) to increase along the gradient flow in Theorem 3.3 (also see Appendix F). However, since we cannot expect graphs to satisfy this condition (Theorem 3.2), a potential alternative is to explore graph rewiring techniques, which are discussed below.

A.2 Graph Rewiring

Graph rewiring techniques aim to propagate messages between nodes via computational graphs that are suitable for the task [5-7, 19, 21, 23, 24, 57]. One of the popular examples is the Personalized PageRank (PPR) and Heat kernel (HK) based diffusion on graphs by Gasteiger et al. [21]. This approach leverages the diffusion matrix to facilitate message-passing between nodes many hops apart in the actual graph. However, the graph resulting from the diffusion operations tends to be quite dense. On the other hand, recent works on addressing over-squashing in long-range dependent tasks leverage the 'curvature' information for rewiring [57].

In the context of our paper, one can aim to design a rewiring technique that can modify the input graph to satisfy condition C . In previous work, Ma et al. [40] generate synthetic graphs based on CORA by ignoring the original graph and adding edges between nodes to satisfy certain uniform neighborhood distributions. Alternatively, exploring an edge re-weighting scheme as proposed by Yan et al. [64] can also be an interesting research direction. From a scalability perspective, a neighborhood sampling technique based on condition C can also aid in better representation learning. Additionally, note that condition C is not limited to homophilic graphs and can be extended to heterophilic settings as well [38, 73], provided the sum of slices of the columns in ˆ A ∈ R N × N are the same for all nodes in a class/community (refer to the necessity condition in the proof of Theorem 3.1 in Appendix D for more details). We believe that further research in this direction can shed some light on Conjecture 3.1 and improve our understanding of the nature of global minimizers.

A.2.1 Measuring neighborhood similarity

A key question that pops up when rewiring a graph to achieve condition C is a trackable metric for node neighborhoods. The 'Cross-Class Neighborhood Similarity (CCNS)' metric by Ma et al. [40] serves as a good starting point to numerically track the cosine similarity in intra-class neighborhoods and dissimilarity of inter-class neighborhoods of nodes. The definition from Ma et al. [40] is given as:

Definition A.2. Cross-Class Neighborhood Similarity (CCNS): Given a graph G and labels y for all nodes, the CCNS between classes c, c ′ is s ( c, c ′ ) = 1 |V c ||V c ′ | ∑ i ∈V c ,j ∈V c ′ cos ( d ( i ) , d ( j )) where V c indicates the set of nodes in class c and d ( i ) indicates the empirical histogram (over C classes) of node i ′ s neighbors' labels, and the function cos ( . ; . ) measures the cosine similarity.

A limitation of this metric is that the CCNS between two classes is measured as an average of the cosine similarity of node neighborhood histograms while failing to incorporate the variance of these neighborhood similarities. Now, note that when condition C in Theorem 3.1 is combined with CCNS, we can ensure that the variance of the cosine similarities is zero for any pair of classes c, c ′ ∈ [ C ] . O verall, better metrics based on different similarity measures and the condition C , along with efficient rewiring techniques to maximize/minimize such metrics can be a valuable future effort in the community.

B A Brief Note on Exact Recovery of Planted Communities

Phase transitions in recovering the planted communities of the Stochastic Block Model (SBM) graphs have been extensively studied in the literature. In this context, 'exact recovery' indicates a perfect assignment of nodes to their respective communities/clusters. For the bi-clustering problem (i.e., C = 2 ), one can date back to the works of Bui et al. [13] for min-cut based clustering, and Boppana [10] for a spectral clustering based approach (Refer to [1, 2] for a historical perspective and additional references on this topic). Along these series of developments to find thresholds in terms of p, q for the exact recovery of communities, the seminal work of Abbe et al. [3] leveraged an information theoretic perspective to identify sharp thresholds in the logarithmic degree settings. The requirement for logarithmic degree can be understood from the following observation. If we consider p = q , then the SBM is essentially an Erdos-Renyi (ER) random graph model with edge-connection probability p . Based on the results by Erd˝ os et al. [17], if p = c log N N , then a randomly sampled ER graph from ER ( N, c log N N ) is connected with high probability if and only if c > 1 (see section 2.5 in [1]). Thus, in order to achieve exact recovery, one must ensure that the SBM graph is connected along with some over-sampling of edges. In the Symmetric SBM case, when p = a log N N , q = b log N N this condition can be represented using the | √ a -√ b | > √ C inequality (see Theorem 13 and the following remarks in [1]). Finally, observe that from a Neural Collapse perspective, we sample SSBM graphs in this regime to ensure that a GNN can achieve zero node-classification error and reach TPT.

C Additional neural collapse metrics

In this appendix, we define additional NC metrics pertaining to NC1-3 for our problem setup.

Variability collapse in neighborhood-aggregated features H ̂ A : We track the within- and between-class variability of the 'neighborhood-aggregated' features matrix H ̂ A by defining the covariance matrices Σ W ( H ̂ A ) , Σ B ( H ̂ A ) as:

To this end, we define the NC 1 ( H ̂ A ) , ˜ NC 1 ( H ̂ A ) metrics as follows:

Their primary purpose is to track the within-class variability of neighborhood aggregated features H ̂ A relative to their between-class variability, both now dependent on the topological structure of graph G . The motivation to track these features arises from the P2 condition in the proof of theorem 3.1 in Appendix B. Essentially, this condition states that the neighborhood aggregated features should collapse to their respective class means for the minimizer to satisfy NC.

SNRfor variance collapse: We track the following 'Signal-to-Noise' ratios pertaining to variability collapse of H and H ̂ A :

These SNR metrics provide an alternate perspective for us to empirically analyze the desirability of variance collapse. Here H := [ h 1 · · · h C ] ∈ R d l × C and H N := [ h N 1 · · · h N C ] ∈ R d l × C are the class-mean matrices without and with neighborhood aggregation respectively.

Convergence of weights to a simplex ETF: To track the convergence of the weights W 1 , W 2 to a simplex ETF structure, we define NC ETF 2 ( W 1 ) , NC ETF 2 ( W 2 ) as:

Convergence of weights to an Orthogonal Frame (OF): To track the convergence of the weights W 1 , W 2 to an orthogonal frame structure, we define NC OF 2 ( W 1 ) , NC OF 2 ( W 2 ) as:

Convergence of features to a simplex ETF: To track the convergence of the features H , H ̂ A to a simplex ETF structure, we define NC ETF 2 ( H ) , NC ETF 2 ( H ̂ A ) as:

where ˜ H := [ h 1 -h G · · · h C -h G ] ∈ R d l × C and ˜ H N := [ h N 1 -h N G · · · h N C -h N G ] ∈ R d l × C are the re-centered class-means without and with neighborhood aggregation respectively. · Convergence of features to an OF: To track the convergence of the features H , H ̂ A to an OF structure, we define NC OF 2 ( H ) , NC OF 2 ( H ̂ A ) as:

Generic alignment of weights and features: To track the alignment of W 1 with its dual H , we define NC 3 ( W 1 , H ) as:

Similarly, to track the alignment of W 2 with its dual H ̂ A , we define NC 3 ( W 2 , H ̂ A ) as:

Alignment of weights and features with respect to simplex ETF: To track the alignment of W 1 and its dual H with respect to a simplex ETF, we define NC ETF 3 ( W 1 , H ) as:

Similarly, we track the alignment of W 2 and its dual H ̂ A with respect to a simplex ETF using:

Alignment of weights and features with respect to OF: To track the alignment of W 1 and its dual H with respect to an OF, we define NC OF 3 ( W 1 , H ) as:

Similarly, we track the alignment of W 2 and its dual H ̂ A with respect to an OF using:

D Proof of Theorem 3.1

In this appendix, we present the proof for Theorem 3.1 by analyzing the sufficiency and necessity conditions of the graph structure given by:

where the fraction of neighbors of node v c,i that belong to class c ′ as s cc ′ ,i = |N c ′ ( v c,i ) | |N ( v c,i ) | .

Proof sketch: Our aim is to identify the structural properties of a graph G (especially ̂ A ), such that the features H , which exhibit neural collapse are indeed the minimizers of the risk. First, we obtain a lower bound for the risk using Jensen's inequality and show that, for a 'collapsed' H to be the minimizer, it is sufficient if the graph G satisfies condition C . However, the inequality is applied on a convex function of { e c,i } (standard basis vectors) that is not strictly convex, and so, this analysis does not imply necessity. Thus, we show the necessity of condition C for a 'collapsed' H to be a minimizer of the risk by analyzing the optimality conditions of the stationary points. Additional details of the sketch for the 'necessity' argument are also presented.

Sufficiency: We begin by revisiting the risk ̂ R F ′ for K = 1 . For simplicity, we drop the superscript F ′ , subscript k , and treat the unconstrained features of the corresponding graph G = ( V , E ) as H , and denote ̂ A = AD -1 . The risk ̂ R ( W 2 , H ) is now given by:

Now, by denoting e c,i ∈ R N as the one-hot vector associated with the index of the feature column h c,i (among the N feature columns), we lower bound ̂ L ( W 2 , H ) as follows 8 :

Note that Jensen's inequality, which we used above, is applied on a convex function of e c,i 's that is not strictly convex. Therefore, the lower-bound in equation 36 can be attained despite having different e c,i 's, e.g., when the properties P1 and P2 , which are stated below, hold ∀ c ∈ [ C ] :

P1: h c, 1 = · · · = h c,n = h c

P2: h N c, 1 = · · · = h N c,n = h N c

The first property P1 indicates zero intra-class variability of H , i.e., Σ W ( H ) → 0 , and the second property P2 indicates zero intra-class variability of H ̂ A i.e, Σ W ( H ̂ A ) → 0 .

Recall condition C :

where ( s c 1 , · · · , s cC ) (shared by any node i ∈ [ n ] in class c ) represents any suitable tuple of the ratio of neighbors per class 9 .

We just need to show that if C is satisfied then H with both P1 and P2 exists. Equivalently, we can assume P1 (which is in the 'feasible set' of the optimization) and show that then C implies P2 . Indeed, in this case

Therefore P2 holds: h N c, 1 = · · · = h N c,n = h N c . Accordingly, C is sufficient for having H that obeys P1 and P2 , and thus minimizes the risk.

Sketch for 'Necessity': The goal of this analysis is to nullify the possibility of having a minimizer that exhibits collapse (i.e., H which satisfies P1 ) for a graph that does not satisfy condition C . We do so by analyzing the optimality conditions of the stationary points satisfying P1 , and obtaining conditions on the ̂ A matrix. Specifically, we obtain a system of linear equations that is shared by all n nodes within a class, which in turn leads to condition C . Thus, proving its necessity.

Necessity: Analysing the necessity of condition C is relatively more complicated than the sufficiency case. Nonetheless, we prove the necessity by considering K = 1 , and by leveraging the idea of a tight-convex alternative for ̂ R ( W 2 , H ) as been used in [55, 74] for the conventional UFMs 10 . Formally, assuming d L -1 ≥ C , we minimize:

where λ Z = √ λ H λ W 2 , Z ∈ R C × N , and ∥ · ∥ ∗ denotes the nuclear norm. Namely, if W 2 and H minimize the former, then Z = W 2 H minimizes the latter, which follows from:

We start with providing necessity analysis for the case λ Z = 0 . Later, we generalize this analysis to address the case λ Z > 0 .

Analysis for λ Z = 0 : When λ Z = 0 , observe that Z ̂ A = Y = [ y 1 , . . . , y C ] ⊗ 1 ⊤ n = I C ⊗ 1 ⊤ n gives us the minimum value for ̂ R ( Z ) . Here y c ∈ R C represent the one-hot label vectors corresponding to nodes of class c ∈ [ C ] .

Now, note that NC1 implies H = H ⊗ 1 ⊤ n ⇐⇒ Z = Z ⊗ 1 ⊤ n (where H ∈ R d L -1 × C and Z = W 2 H ∈ R C × C ). Thus, by leveraging this Kronecker structure of Z = Z ⊗ 1 ⊤ n = [ z 1 , . . . , z C ] ⊗ 1 ⊤ n and Y = [ y 1 , . . . , y C ] ⊗ 1 ⊤ n , we formulate:

where a i c,c ′ ∈ R n represents the slice of columns in ̂ A corresponding to node i ∈ [ n ] belonging to class c ∈ [ C ] and forming edges with nodes from class c ′ ∈ [ C ] . Additionally, since ̂ A is the

normalized adjacency matrix, we have s cc ′ ,i = 1 ⊤ n a i c,c ′ ∈ R , which represents the sum of elements in a i c,c ′ . This gives us:

because ̂ A = AD -1 is the degree-normalized adjacency matrix. Now, multiplying the matrix Z = Z ⊗ 1 ⊤ n with the first column of ̂ A gives us:

Similarly, due to the block structure of Z , Y , we get for all i ∈ [ n ] and c = 1 :

where s 11 ,i z 1 + · · · + s 1 C,i z C = y 1 itself can be written as C linear equations (one for each of the vector components) as formulated below:

Now, by treating { s 11 ,i , · · · , s 1 C,i } as the C unknowns which satisfy equation 42, observe that the solution to this linear system remains the same for all nodes belonging to class c = 1 . This implies:

The generalization of this result for all C classes essentially indicates that:

which exactly represents condition C as stated above.

Analysis for λ Z > 0 : When H exhibits neural collapse, we have the optimality condition for the minimizer of equation 39 based on the sub-differential of the nuclear norm as follows:

Where Z ∈ R C × C is the matrix of 'collapsed' columns of Z , and U Z ∈ R C × C , V Z ∈ R N × C represent the left and right singular vectors of Z . Additionally, since Z holds a 'block' structure, we represent V ⊤ Z = V ⊤ Z ⊗ 1 √ n 1 ⊤ n , where V Z ∈ R C × C .

Matrix Quadratic Form: By considering -Nλ Z U Z V ⊤ Z ⊗ 1 √ n 1 ⊤ n = B , we get:

as our optimality condition. Analyzing this condition along the same lines as λ Z = 0 is non-trivial due to the outer product of ̂ A ̂ A ⊤ . To address this complication, we leverage the fact that SSBM graphs tend to be regular with a high probability as N increases. Thus, by assuming that ̂ A is symmetric, we obtain the following matrix quadratic form:

As Z is rectangular, we can take the pseudo-inverse and obtain:

Since Z † Y , Z † B ∈ R N × N , we can treat ̂ A as the variable matrix and leverage the results on quadratic matrix equations by Higham and Kim [27]. Especially, we leverage theorem 3 in Higham and Kim [27] and employ a generalized Schur decomposition technique to obtain a condition on ̂ A (as shown in the following lemma). This condition on ̂ A allows us to obtain a system of linear equations that are shared by all n nodes within a class (similar to the λ Z = 0 case). Thus, establishing the necessity of condition C .

Lemma D.1. Let F = [ 0 I Z † B Z † Y ] , G = [ I 0 0 I ] = I ∈ R 2 N × 2 N , and the generalized Schur decomposition of F , G be given by:

where Q , K are unitary and T , E are upper triangular. K = [ K 11 K 12 K 21 K 22 ] ∈ R 2 N × 2 N is a 2 × 2 block matrix with blocks of size N × N . Then ̂ A satisfies: K 11 ̂ A = K 21 .

Proof. The proof is relatively straightforward once we observe that ̂ A satisfies:

Now, the condition K 11 ̂ A = K 21 is a direct consequence of theorem 3 in [27], which leverages the QR decomposition of [ I ̂ A ] using K as the orthogonal matrix.

· Kronecker structure of F : To leverage the relationship between ̂ A and K as per Lemma D.1, a closer look at F is required. Observe that:

By expanding the Kronecker structures of Z , B , Y , we get:

Observe that the pseudo-inverse Z † can be represented by:

which also holds a 'block' structure (but with respect to rows, instead of columns):

Where v j ∈ R C , ∀ j ∈ [ C ] and v ⊤ j is the j th row of V Z ∈ R C × C . With this formulation, a matrix-vector product term in F , for instance Z † b i , i ∈ [ C ] , can be given as:

By considering the following notational simplifications:

we can represent F as:

Schur decomposition of F : For notational simplicity, let :

Since G = I , the diagonal entries of S must equal the eigenvalues of I . This condition is satisfied when Q = K . This also simplifies the generalized Schur decomposition for square matrices F , G to the standard Schur decomposition of F . Now, let the schur decomposition of F ∈ R 2 N × 2 N , ˜ F ∈ R 2 C × 2 C be given as:

Here K , T ∈ R 2 N × 2 N and ˜ K , ˜ T ∈ R 2 C × 2 C . To find a relation between K , T , ˜ K , ˜ T , we can leverage the Schur decomposition properties of Kronecker products [53, 59] and obtain:

Where J , O ∈ R n × n are unitary and upper triangular respectively, and are the Schur decomposition factors of 1 √ n 1 n 1 ⊤ n ∈ R n × n .

Linear systems: In matrix form, K = ˜ K ⊗ J can be represented as:

Where ˜ K 11 , ˜ K 12 , ˜ K 21 , ˜ K 22 ∈ R C × C . Now, observe that K 11 ̂ A = K 21 (based on Lemma D.1) can be reformulated as:

Now, as per equation 41, we leverage the same line of analysis that we followed for the λ Z = 0 case. For notational simplicity, we represent the unitary matrix J ∈ R n × n in column format as follows:

where j i ∈ R n , i ∈ [ n ] are linearly independent vectors. Now, by multiplying the first row of ˜ K 11 ⊗ J and the first column of ̂ A , we get:

This translates to the following linear equation:

Due to linear independence of vectors j i , i ∈ [ n ] , we obtain the following n equations pertaining to the coefficients of j i :

By adding all these equations, we get:

By following the same approach for the other rows of ˜ K 11 ⊗ K 1 n and the first column of ̂ A , we get the following system of equations for node v 1 , 1 :

The same line of analysis can be applied for all the rows of ˜ K 11 ⊗ K 1 n and the second column of ̂ A , to get the following system of equations for node v 1 , 2 :

Thus, it is straightforward that the systems of equations are the same for all n nodes belonging to class c = 1 , as the procedure of row and column multiplication remains the same. Thus, the C unknowns in these linear systems have the same solution for all n nodes belonging to class c = 1 . It is straightforward to extend this to any class c ∈ [ C ] and obtain:

which exactly represents the condition C as per the theorem.

E Proof of Theorem 3.2

In this appendix, we derive the upper bound on the probability of sampling the desired neighborhood for condition C .

to be the same for a given c ∈ [ C ] . To this end, we are primarily concerned with the probabilities of edges between nodes v i , v j , 1 ≤ i ≤ j ≤ N . Thus, as per preliminaries, the probability matrix P can be given in block form as:

Where we are only concerned with the diagonal and upper triangular values 11 . Now, for a pair of classes c, c ′ ∈ [ C ] , we are concerned with the block probability matrix p 1 n 1 ⊤ n when c = c ′ and q 1 n 1 ⊤ n when c = c ′ for sampling edges between nodes. Observe that sampling edges within a community based on diagonal block matrix p 1 n 1 ⊤ n is the same as sampling an Erdos-Renyi graph with edge probability p . Similarly, sampling edges between communities based on off-diagonal block matrix q 1 n 1 ⊤ n is the same as sampling a bipartite graph with edge probability q .

̸

̸

Concentration of pairwise neighbor ratios: To begin with, consider the set of nodes belonging to class c ∈ [ C ] as Ω c = { v c, 1 , · · · , v c,n } . Now, to satisfy condition C , the fraction of neighbors of nodes v c,i , v c,j , i = j ∈ [ n ] that belong to class c ′ ∈ [ C ] should be equal, i.e s cc ′ ,i = s cc ′ ,j . Formally, this leads to:

Without loss of generality, observe that |N c ′ ( v c,i ) | is the sum of n independent Bernoulli random variables γ c ′ ,l c,i , ∀ l ∈ [ n ] with P ( γ c ′ ,l c,i = 1) = q . This implies that E |N c ′ ( v c,i ) | = nq . Now, we apply the Chernoff bound to obtain:

Where δ ∈ [0 , 1] and t > 0 is a constant. By choosing δ = √ ( r +1)ln n tnq for sufficiently large r > 0 , N >> C , we get δ = O (1) as q = b ln N N . Now, by taking a union bound over all the nodes in the class, we get:

Thus, with a probability at-least 1 -2 n -r , we get:

By applying the same line of argument to |N c ′ ( v c,j ) | and assuming a sufficiently large value of n , we get with a probability at-least 1 -4 n -r that:

To this end, we assume that |N c ′ ( v c,i ) | = |N c ′ ( v c,j ) | , ∀ c ∈ [ C ] , i = j ∈ [ n ] with a high probability for the rest of the analysis. The consequence of this assumption is that all the nodes belonging to the same class have the same degree. However, it is not necessary that the graph itself is regular.

̸

̸

Off-diagonal blocks: Without loss of generality, consider the set of nodes belonging to a pair of classes c = c ′ ∈ [ C ] as Ω c = { v c, 1 , · · · , v c,n } , Ω c ′ = { v c ′ , 1 , · · · , v c ′ ,n } respectively. Now, we need

to ensure that every node v c,i ∈ Ω c is connected to (say) exactly t cc ′ ∈ R , 0 ≤ t cc ′ ≤ n , nodes in Ω c ′ . If E c ′ c,i ( t cc ′ ) indicates that such an event occurs for node v c,i ∈ Ω c with respect to Ω c ′ , we can formally represent it as the sum of n independent Bernoulli random variables γ c ′ ,j c,i , i ∈ [ n ] , ∀ j ∈ { 1 , . . . , n } with P ( γ c ′ ,j c,i = 1) = q sum to t cc ′ as follows:

Now, we are concerned with an intersection of events pertaining to all nodes in Ω c , which ensures that each node has exactly t cc ′ neighbors in Ω c ′ . By considering the event E c ′ c ( t cc ′ ) = ⋂ n i =1 E c ′ c,i ( t cc ′ ) and leveraging the fact that edges are sampled independently, we obtain:

Now, to account for all possible values of 0 ≤ t cc ′ ≤ n , we compute the probability of the union of events ˜ E c ′ c = ⋃ n t cc ′ =0 E c ′ c ( t cc ′ ) as:

̸

Note that this result can be applied to any distinct pair of classes c = c ′ ∈ [ C ] based on the characteristics of the SSBM. Since we have ( C 2 ) = C ( C -1) 2 combinations of distinct communities, the probability of occurrence of all the corresponding events is given by:

Observe that the binomial expansion of ( q +1 -q ) n is given by:

where each term, say f ( t cc ′ ) = ( n t cc ′ ) q t cc ′ (1 -q ) n -t cc ′ in the sum is strictly less than 1. Additionally,

This gives us:

Observe that there are n +1 terms in ∑ n t cc ′ =0 f ( t cc ′ ) = 1 and the maximum value that f ( t cc ′ ) can take is < 1 . Thus, each of the f ( t cc ′ ) terms tend to zero after taking the n th power, as n → ∞ , leading to ∑ n t cc ′ =0 f ( t cc ′ ) n → 0 .

Diagonal blocks: Handling the exact event probabilities when c = c ′ ∈ [ C ] is not so straightforward due to symmetry constraints. To begin with, observe that the sum of n independent Bernoulli random variables γ c,j c,i , i ∈ [ n ] , ∀ j ∈ { 1 , . . . , n } with P ( γ c,j c,i = 1) = p sum to t cc is given by:

Since P ( E c c,j ( t cc )) for j > i ∈ [ n ] is conditional on P ( E c c,i ( t cc )) , the desired probability for the event E c c ( t cc ) = ⋂ n i =1 E c c,i ( t cc ) can be formulated as:

Obtaining a clean expression for the diagonal case is complicated by the fact that the events E c c, 1 ( t cc ) , E c c, 2 ( t cc ) , · · · , E c c,n ( t cc ) are not independent. Since we have already shown in the offdiagonal case that the P ( ⋂ C -1 c =1 ⋂ C c ′ = c +1 ˜ E c ′ c ) → 0 as n → ∞ , we proceed with the simpler inequality P ( E c c ( t cc )) < 1 to convey the message of this theorem. Thus, we estimate the probability of the intersection of all events pertaining to c, c ′ ∈ [ C ] as:

E.1 Illustration with exhaustive combinations for a small graph

A key assumption in our theoretical analysis has been N >> C , which allowed us to assume that nodes belonging to the same class have the same degree. However, for an intuitive understanding of condition C , let us consider an extremely simple graph with N = 4 , C = 2 . This leads to the following adjacency matrix formulation:

Where γ ij represents a Bernoulli random variable depending on p, q . We defer assigning values to p, q until the end as we are interested in the 'realizations' of A that satisfy condition C . Observe that there are only 10 random variables in A due to symmetry (4 on the diagonal and 6 on the upper-triangular part). Since each can either take a value of 0 , 1 , there are 1024 unique realizations of A . To this end, condition C can be represented as:

Based on our simulations for graphs with self-edges, less than 1 / 10 of the 1024 realizations satisfy this property. A few are illustrated below:







































A

=





















0

1

1

1

0

0

0

0

1

1

0

1

0

0

0

0

1

0

1

1

0

0

0

0

1

1

1

0

0

0

0

0

0

0

0

0

0

1

1

0

0

0

0

0

1

1

1

1

0

0

0

0

1

1

1

1

0

0

0

0

0

1

1

0





















,

A

=





















1

1

1

1

0

1

0

0

1

1

1

1

0

0

1

0

1

1

1

1

1

0

0

0

1

1

1

1

0

0

0

1

0

0

1

0

0

1

1

1

1

0

0

0

1

1

0

1

0

1

0

0

1

0

1

1

0

0

0

1

1

1

1

0

A

=





















0

1

1

0

0

1

0

1

1

1

0

1

0

1

1

1

1

0

1

1

1

1

1

0

0

1

1

1

1

0

1

1

0

0

1

1

0

1

0

1

1

1

1

0

1

0

1

1

0

1

1

1

0

1

1

1

1

1

0

1

1

1

1

0





















,

A

=





















1

1

1

0

1

1

0

0

1

0

1

1

1

0

1

0

1

1

0

1

0

1

0

1

0

1

1

1

0

0

1

1

1

1

0

0

1

1

1

0

1

0

1

0

1

0

1

1

0

1

0

1

1

1

0

1

0

0

1

1

0

1

1

1



Thus, the takeaway is that, even for small-medium scale homophilic and heterophilic SSBM graphs, random graphs that satisfy condition C are rarely sampled.

F Proof of Theorem 3.3

In this appendix, we analyse the gradient flow stated in (14) and establish the results stated in Theorem 3.3. Our analysis is inspired by the one in [56], but significantly differs from it, as we need to overcome the complexity introduced by the graph structure matrix.

Based on the setup of gUFM for GNN ψ F ′ , we analyze the risk ̂ R F ′ given as follows:

By taking the derivatives of ̂ R F ′ with respect to ( W 2 , H k ) , we get:

Now, by setting ∂ ̂ R F ′ ∂ W 2 = 0 we get the closed form representation of W 2 in terms of H .

Thus, the ideal value of W ∗ 2 is given by:

Now, under the assumption that K = 1 , C = 2 , we drop the subscript for H k , ̂ A k to get:

To this end, we return to our risk minimization formulation for a single graph with these optimal values as follows:

F.1 ̂ A as a perturbation of E ̂ A

Since we are dealing with SSBM graphs, we can formulate ̂ A as the perturbed version of its expected value. Formally, ̂ A = E ̂ A + E where E ̂ A ∈ R N × N is the expected normalized adjacency matrix and E ∈ R N × N is the perturbation matrix. E ̂ A can be written in block matrix form as:

Where the E ̂ A has eigenvalues 1 , p -q p + q associated with eigenvectors 1 √ N 1 and c = 1 √ N [ 1 n -1 n ] N respectively. Thus, we can represent E ̂ A using its spectral information as follows:

Where α 1 = 1 N , α 2 = p -q ( p + q ) N , and Q = [ 1 N c ] [ √ α 1 0 0 √ α 2 ] ∈ R N × 2 is the factor matrix.

F.2 Preliminary results

Let us recall the definitions of the within- and between-class covariance matrices, Σ W ( H ) and Σ B ( H ) , for computing the preliminary results:

Here, the class means h c , ∀ c ∈ [ C ] and the global mean h G of H are given as follows:

F.2.1 Relating ˜ Σ B ( H ) , Σ B ( H )

Since C = 2 in our analysis and h G = h 1 + h 2 2 due to balanced communities, note that:

Thus, we get the following relation between ˜ Σ B ( H ) , Σ B ( H ) :

Additionally, we can extend this result to the following:

Where Σ G = h G h ⊤ G . This gives us:

F.2.2 Expanding H ̂ AH ⊤

Based on the perturbed representation of ̂ A = E ̂ A + E , we can modify H ̂ AH ⊤ and H ̂ A ⊤ H ⊤ as:

Observe that HQ can be broken down in terms of h c,i , c ∈ [ C ] , i ∈ [ n ] as follows:

This leads to the following expansion of [ HQ ] [ HQ ] ⊤ :

Since h G = h 1 + h 2 2 (due to balanced classes). This also implies that:

Thus, by taking the average of values in equation 88, 89, we get:

Finally, H ̂ AH ⊤ can be simplified to:

Where ∆ 1 = HEH ⊤ corresponds to the first order perturbation term. Similarly,

F.2.3 Expanding H ̂ A ̂ A ⊤ H ⊤

Expanding ̂ A ̂ A ⊤ can be done along the same lines:

Based on the formulations above, H [ E ̂ A ] 2 H ⊤ can be given by:

Since h G = h 1 + h 2 2 (due to balanced classes). This also implies that:

Thus, based on taking the average of values in equation 92, 93, we get:

Finally, H ̂ A ̂ A ⊤ H ⊤ can be simplified to:

Where ∆ 2 = H [ EE ⊤ + E [ E ̂ A ] + [ E ̂ A ] E ⊤ ] H ⊤ is a symmetric matrix corresponding to the first and second order perturbation terms.

F.2.4 Expanding Y ̂ A ⊤ H ⊤ , YH ⊤

We follow similar line of expansions for Y ̂ A ⊤ H ⊤ to get:

Where ∆ 3 = YE ⊤ H ⊤ is the first order perturbation term. Next, observe that:

F.2.5 Expanding H ̂ AY ⊤ Y ̂ A ⊤ H ⊤

Where ˜ ∆ 3 = H ( EY ⊤ YE ⊤ + [ E ̂ A ] Y ⊤ YE ⊤ + EY ⊤ Y [ E ̂ A ]) H ⊤ and the equality ( a ) is based on equation 86.

F.3 Trace formulation of risk

Now, note that the risk can be formulated in terms of matrix traces as follows:

Where the term ( W ∗ 2 H ̂ A -Y )( W ∗ 2 H ̂ A -Y ) ⊤ can be expanded as follows:

Since W ∗ 2 [ H ̂ A ̂ A ⊤ H ⊤ + λ W 2 N I ] = Y ̂ A ⊤ H ⊤ , we can multiply W ∗⊤ 2 on both sides and get: W ∗ 2 H ̂ A ̂ A ⊤ H ⊤ W ∗⊤ 2 = Y ̂ A ⊤ H ⊤ W ∗⊤ 2 -λ W 2 N W ∗ 2 W ∗⊤ 2

Using these simplifications and matrix trace properties, the risk can be modified as:

Where the covariance matrix formulations for H ̂ A ̂ A ⊤ H ⊤ , H ̂ AY ⊤ Y ̂ A ⊤ H ⊤ can be leveraged to formulate the risk as:

F.4 Trace evolution of covariance matrices

Now, we analyze the traces of d Σ W dt , d Σ B dt along the gradient flow:

Let ∂ kjl represent the derivative of l th entry of h k,j . For notational simplicity, we also consider Σ W = Σ W ( H ) , Σ B = Σ B ( H ) , ˜ Σ B = ˜ Σ B ( H ) , ˜ Σ T = ˜ Σ T ( H ) . This leads to:

Now, considering J = 2 n ( ˜ Σ B -4 pq ( p + q ) 2 Σ B ) , we formulate ∂ kjl ̂ R F ′ ( H ) as:

Since C = 2 in our analysis, the derivative expands into:

Where the second term can be expanded as:

Now, by expanding ∂ kjl ( J ) in terms of covariance matrix derivatives, we get:

This leads to the following formulation for ∂ kjl ̂ R F ′ ( H ) :

Where P kjl represents the remaining trace terms pertaining to the partial derivatives of ∆ 2 , ˜ ∆ 3 :

We now denote M := [ J +∆ 2 + λ W 2 N I ] -1 to obtain:

Without loss of generality, since P kjl ∈ R , we consider P kjl = p ⊤ k,j e l where p k,j ∈ R d L -1 is a random vector which represents the overall perturbation effect of P kjl . Note that the randomness is associated with the E matrix in ∆ 2 , ˜ ∆ 3 . We can now represent ∂ kjl ̂ R F ′ ( H ) = ⟨ R k,j , e l ⟩ , where:

· Derivative of Σ B : By denoting Σ B ( a, b ) = e ⊤ a Σ B e b as the ( a, b ) -element of Σ B , we use the above result for ∂ kjl ̂ R F ′ ( H ) and the chain rule to compute d Σ B dt along the flow (14), as follows:

For further simplification, let's consider the following term:

Next, we proceed with the simplification of ∑ k,j -λ H h k,j ( h k -h G ) ⊤ -λ H ( h k -h G ) h ⊤ k,j :

These results now simplify d Σ B dt as:

Derivative of Σ W : By denoting Σ W ( a, b ) = e ⊤ a Σ W e b as the ( a, b ) -element of Σ W , we use a similar line of analysis as above to compute d Σ W dt as follows:

For further simplification, let's consider the following term:

Where the last inequality is based on the fact that ˜ Σ T = Σ W + ˜ Σ B . These results simplify d Σ W dt as:

Where ˜ P W = 2 ∑ k,j p k,j ( h k,j -h k ) ⊤ +( h k,j -h k ) p ⊤ k,j .

Trace of covariance matrices along the flow: Taking the derivative of Tr ( Σ W ) gives us:

Similarly, the derivative of Tr ( Σ B ) gives us:

Observe that for small enough perturbation matrix E , formally for ∥ E ∥ < E for sufficiently small E , the sign of d Tr ( Σ W ) dt and d Tr ( Σ B ) dt depends on the first term in each of them, and not on the second term that depends on E . Therefore, since λ H > 0 , we have that Tr ( Σ W ) decreases. It is left to show that Tr ( Σ B ) increases in this regime.

Since the trace of the product of a positive definite matrix and a non-zero positive semidefinite matrix is positive by V on-Neumann trace inequality, we aim for conditions that allow [ ( p -q p + q ) 2 ˜ M -2 λ H I ] to be positive definite. First, observe that ˜ M is symmetric:

Since ˜ Σ B , Σ B , I , ∆ 2 and ˜ ∆ 3 are symmetric.

Observe that 4 pq ( p + q ) 2 ≤ 1 , because

Thus,

Thus also M > 0 for small ∆ 2 . Note also that [ J + λ W 2 N I ] -1 J < I . Therefore, for small enough E (and thus small ∆ 2 , ˜ ∆ 3 ), we have that

Thus, [ ( p -q p + q ) 2 ˜ M -2 λ H I ] is positive definite when:

Here λ min ( ˜ M ) represents the smallest eigenvalue of ˜ M .

G Proof of Theorem 4.1

Let's begin by calculating the expected value and covariance of features X ( l ) , which are obtained after the graph convolution operation based on equation 3.

· Case c = 1 :

We begin by considering the features of a node belonging to class c = 1 as follows:

By expanding the H ( l -1) ̂ A t term based on neighbors from all classes, we get:

Now, by taking expectations on both sides with respect to features h ( l -1) and structure A t

The expectation of W ∗ ( l ) 1 ( h ( l -1) 1 ,i -µ ( l -1) 1 ) ( H ( l -1) ̂ A t e 1 ,i -p µ ( l -1) 1 + q µ ( l -1) 2 p + q ) ⊤ W ∗ ( l ) ⊤ 2 is given by the following:

̸

Since the features are independent draws from their normal distributions, we can simplify the expectation as follows:

Similarly, the expectation of W ∗ ( l ) 2 ( H ( l -1) ̂ A t e 1 ,i -p µ ( l -1) 1 + q µ ( l -1) 2 p + q ) ( h ( l -1) 1 ,i -µ ( l -1) 1 ) ⊤ W ∗ ( l ) ⊤ 1 is given by the following:

Next, the expectation of: W ∗ ( l ) 2 ( H ( l -1) ̂ A t e 1 ,i -p µ ( l -1) 1 + q µ ( l -1) 2 p + q )( H ( l -1) ̂ A t e 1 ,i -p µ ( l -1) 1 + q µ ( l -1) 2 p + q ) ⊤ W ∗ ( l ) ⊤ 2 can be computed as follows:

Where the second term reduces to 0 . On expanding the first term, we get:

̸

̸

Due to the independence of the features, we can simplify the expectation as follows:

To summarize, the means and covariance matrices for X ( l ) can be given by:

· Case c = 2 :

The analysis presented above can be extended for c = 2 in a straightforward fashion to get ˜ µ ( l ) 2 , ˜ Σ ( l ) 2 as follows:

G.1 Modelling an increase/decrease in between-class variability

Let Σ ( l -1) B = ( µ ( l -1) 1 -µ ( l -1) 2 )( µ ( l -1) 1 -µ ( l -1) 2 ) ⊤ indicate the between-class covariance matrix for features at layer l -1 . Based on the equations 118, 119, we analyze Σ B ( X ( l ) ) to understand the effect of the convolution layer on feature separation. First, observe that:

By taking the trace on both sides, we get:

Where ( W ∗ ( l ) 1 + p -q p + q W ∗ ( l ) 2 ) ⊤ ( W ∗ ( l ) 1 + p -q p + q W ∗ ( l ) 2 ) ∈ R d l -1 × d l -1 is a symmetric and positive semi-definite matrix. This matrix product formulation allows us to leverage the eigenvalue-based trace inequalities [41, 68]. Formally, let T B = ( W ∗ ( l ) 1 + p -q p + q W ∗ ( l ) 2 ) ⊤ ( W ∗ ( l ) 1 + p -q p + q W ∗ ( l ) 2 ) . We now leverage Corollary.6 in Zhang and Zhang [68], and get the following inequality based on the eigenvalues of Σ B ( H ( l -1) ) , T B as:

Where λ i ( . ) represents the i th largest eigenvalue of a matrix. Additionally, based on the standard trace equality: Tr ( Σ B ( H ( l -1) ) ) = ∑ d l -1 i =1 λ i ( Σ B ( H ( l -1) ) ) , the increase/decrease in Tr ( Σ B ( X ( l ) ) ) with respect to Tr ( Σ B ( H ( l -1) ) ) boils down to:

G.2 Modelling an increase/decrease in within-class variability

Let Σ W ( H ( l -1) ) = 1 2 ( Σ ( l -1) 1 + Σ ( l -1) 2 ) represent the within-class covariance matrix for features H ( l -1) in our balanced class setting. Similar to the previous analysis, we leverage the results in

equations 118,119 to model Σ W ( X ( l ) ) as follows:

By taking trace on both sides, we get:

Let T W = W ∗ ( l ) ⊤ 1 W ∗ ( l ) 1 + p n ( p + q ) [ W ∗ ( l ) ⊤ 2 W ∗ ( l ) 1 + W ∗ ( l ) ⊤ 1 W ∗ ( l ) 2 ] + ( p 2 + q 2 ) n ( p + q ) 2 W ∗ ( l ) ⊤ 2 W ∗ ( l ) 2 . Then, observe that T W ∈ R d l -1 × d l -1 is symmetric and positive semi-definite. To this end, the increase/decrease in Tr ( Σ W ( X ( l ) ) ) with respect to Tr ( Σ W ( H ( l -1) ) ) boils down to:

H Addition Experiments

Infrastructure details: We perform experiments on a virtual machine with 8 Intel(R) Xeon(R) Platinum 8268 CPUs, 32GB of RAM, and 1 Quadro RTX 8000 GPU with 32GB of allocated memory. Our Python package 'gnn_collapse' leverages PyTorch 1.12.1 and PyTorch-Geometric (PyG) 2.1.0 frameworks. For reproducible experiments and consistency with previous research, we extend the SBM generator by Chen et al. [16] and NC metrics by Zhu et al. [74].

H.1 Experiments with GNNs to track penultimate layer features

Datasets: We consider a variety of SSBM graph datasets as follows:

GNNs: In our experiments, we empirically track the NC metrics of penultimate layer features during training for both the GNN designs ψ F Θ , ψ F ′ Θ . The number of layers is set to 32 and the hidden dimension is set to 8 across layers for datasets with C = 2 and set to 16 for datasets with C = 4 .

Optimization: The GNNs are trained for 8 epochs using stochastic gradient descent (SGD) with momentum 0 . 9 , weight decay 5 × 10 -4 , and a learning rate set to 0 . 004 for D1 and 0 . 006 for D2 .

Observations: Figures 7, 8 illustrate the training loss, overlap and all the NC metrics that we defined in our setup for ψ F Θ , ψ F ′ Θ on dataset D1 . Note that when C = 2 , the re-centering of the 2 classmeans by subtracting the global mean, always leads to separation with maximal angle, irrespective of the configuration of the non-centered class-means. Thus, we skip the corresponding NC 2 plots for H , H ̂ A when C = 2 . Additionally, we can observe similar trends in NC metrics from Figures 9, 10 even after increasing N,C in dataset D2 .

Additionally, in all these experiments, notice that NC 2 , NC 3 metrics do not show a significant reduction. In this context, a reduction indicates that a simplex equiangular tight-frame (simplex ETF) or an orthogonal frame (OF) is the desired configuration for weights and penultimate layer feature (re-centered) class-means. This behaviour can be linked to the presence of ̂ A in the risk formulation. However, our understanding of the role of ̂ A in determining these alignments towards simplex ETF or an OF is still unclear and would be a valuable future effort.

0.6

1.0

0.5

Tr(Z8)

Tr(Ew)

0.5

0.9

0.8

NC(H)(log10 scale)

0.5

NC(HA)(log10 scale)

0.0

Tr(EwE²)/C

0.4

0.0

-0.5

0.5

-1.0

0.5

0.2

0.4

-1.0

Tr(2w)

-1.5

0.1

0.0

0.3

0.2

-1.5

Tr(EWg)C

Tr(Eg)

Tr(Ew)/Tr(Ea)

-2.0

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.5

WH

2.0

0.1

(W.H)

SNR(log10 scale)

1.0

1.5

-0.1

0.0

0.1

0.2

=(WHA.OF)

（WH,OF)

(W2.HA)

0.5

1.0

IIW-ll

IIW2ll

-0.2

0.0

IIHII

-0.3

IIHAllI

0.0

-0.4

-0.1

-0.5

0.0

-0.5

-0.6

W(OF)

W(OF)

W(ETF)

W2(ETF)

-0.2

-0.3

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

o

0.6

1.00

Tr(Ew)

0.5

0.4

660

0.98

NC(H)(log10 scale)

0.5

0.0

NC(HA)(log10 scale)

0.5

0.0

Tr(Eg)

Tr(Zw)/Tr(EB)

Tr(Ew²)/C

-0.5

-0.5

-1.0

-1.0

0.2

0.96

1.5

)

1.5

0.1

0.95

Tr(EB)

-2.0

Tr(EwEg)/C

-2.0

0.0

0.94

Tr(Ew)/Tr(Ea)

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

2.00

0.0

1.2

1.75

0.0

-0.1

scale)

-0.5

1.0

-0.2

一

Wll

—IIHAIl

IIHI

-0.3

W2(OF)

W2(ETF)

1.5

-1.0

NC3

-2.0

0.6

0.25

-0.5

(W2.HA)

0.4

0

2

4

6

8

0.00

0

2

4

6

8

-0.6

2

4

6

8

2.5

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

0.4

1.0

1.0

Tr(EB)

Tr(Ew)

1.0

0.3

0.9

0.8

NC(H)(log10 scale)

0.0

0.5

Tr(Ew)/Tr(EB)

NC(HA)(log10 scale)

0.5

0.0

0.2

-0.5

-0.5

Tr(Ew)

Tr(Eg)

-1.0

Tr(Ew²/C

0.5

-1.5

-1.0

Tr(Ew)/Tr(E)

0.1

-2.0

-1.5

0.4

0.3

2.5

-2.0

0.0

-3.0L

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.50

—WH

0.0

W(ETF)

0.30

1.25

W2HA

2.0

W2(ETF)

H(ETF)

0.25

I

1.00

0.75

1.5

lIW2llr

scale)

-0.1

-0.2

=·W(OF)

W（OF)

HA(ETF)

scale)

0.20

0.15

0.50

IIHAIIF

IIHI

-0.3

0.10

(W.H)

UAIC

0.25

NC3

0.05

(W2.HA)

0.00

-0.4

0.00

(WH,ETF)

-0.25

0.0

-0.5

-0.05

(WHA.OF)

(WH,OF)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.00

Tr(Ew)

1.0

1.0

Tr(Z8)

0.20

0.95

0.5

Tr(Ewg)C

Tr(Ew)/Tr(EB)

scale)

0.5

0.15

0.90

0.0

00601)

0.0

Tr(Ew)

0.10

0.80

-0.5

-1.0

-1.0

0.5

Tr(EB)

Tr(EwEg²C

Tr(Zw)/Tr(Eg)

0.05

0.75

-1.5

-1.5

0.00

0.70

-2.0

-2.0

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

W2HA

(W2.HA)

1.04

2.0

-0.1

0.4

(W2HA.OF)

(

（log10scale)

-0.2

（log10scale)

1.02

一

lWellr

-0.3

0.2

一

IIHII

0.0

1.00

IHAII

-0.4

SNR

NC3

0.98

0.5

-0.5

H(ETF)

W(ETF)

-0.2

-0.6

HA (ETF)

-0.4

0.96

0.0

W(OF)

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

6

8

H.2 Experiments with UFM to model penultimate layer behavior

To improve our understanding of the empirical behavior of GNNs and to validate our theoretical results with the gUFM, we prepare datasets based on two strategies:

̸

Case C + : This case represents a graph that satisfies condition C . Without loss of generality, we leverage the expected degrees of nodes and consider t cc = ⌈ n ∗ p ⌉ , c ∈ [ C ] and t cc ′ = ⌈ n ∗ q ⌉ , c = c ′ ∈ [ C ] . Based on our notation, recall that Ω c , Ω c ′ represents the set of nodes belonging to classes(communities) c, c ′ ∈ [ C ] respectively. To this end, observe that the following conditions should be satisfied 12 :

The sub-graph formed by nodes Ω c should be t cc -regular, for all c ∈ [ C ] .

The bipartite sub-graph formed by Ω c , Ω c ′ should be t cc ′ -regular, for all c, c ′ ∈ [ C ] .

Case C -: This case represents a graph that does not satisfy condition C . Since any random graph sampled from SSBM ( N,C,p,q ) satisfies this requirement with a high probability (as per theorem 3.2), we simply use this randomly sampled graph. As a simple sanity check, one can verify if the sampled graph satisfies the condition C and re-sample.

Especially, we consider the C + , C -variants of SSBM graphs with following parameters:

Optimization: The gUFMs are trained using plain gradient descent for 50000 epochs with a learning rate of 0 . 1 and L2 regularization parameters λ W 1 = λ W 2 = λ H = 5 × 10 -3 . 13

Observations: Figures 12, 14, and 16, 18 illustrate the training loss, overlap and the NC metrics for the gUFM acting on C -variants of datasets D1, D2 respectively. Although gUFM is an optimistic mathematical model, we can observe a close resemblance of the values and trends of NC metrics to those of the penultimate layer features of the actual GNNs. This observation is justified as any random SSBM graph fails to satisfy condition C with a high probability. To this end, observe from Figures 11, 13, 15, 17 that when graphs satisfy condition C , the NC1 metrics tend to reduce drastically (for gUFM designs based on ψ F Θ , ψ F ′ Θ and C = 2 , 4 ). Thus proving our theoretical results. Furthermore,

observe from Figures 15, 17 that the (re-centered) class means for H , H ̂ A tend to align very closely to a simplex ETF and tend to converge at such a configuration. Based on our previous observations for GNN training, we underscore this observation for gUFM and emphasize that a rigorous theoretical analysis on the role of ̂ A in determining the structures of H can be a crucial future effort.

1.0

25

0

0

0.8

NC(H)(log10 scale)

-5

NC(HA)(log10 scale)

-5

20

-10

-10

-15

-15

10

-20

5

0.2

-25

一

Tr(Ew)

Tr(Za)

-25

Tr(Ew)

Tr(E8)

一

一

0.0

-30

Tr(Ew)/Tr(∑g)

-30

一

Tr(Ew)/Tr(EB)

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

14

12

一

WH

W2HA

2.0

1.8

lWall

IIW2llI

IHIlr

-0.2

0.0

0.25

0.00

——(W.H)

(W2.HA)

(WH,OF)

10

IIHAIIF

-0.4

-0.25

(WHA.OF)

8642

-0.6

W(ETF)

W2 (ETF)

-0.50

-0.8

--·W(OF)

--W(OF)

-0.75

-1.0

NC3

-1.00

1.2

0

0.6

-1.4

-1.25

-2

0.0

2.5

iter%50000

5.0

7.5

10.0

0.4

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

0007.5

10.0

-1.50

0.0

2.5

iter%50000

5.0

7.5

10.0

1.0

Tr(Ew)

N

Tr(Ew)

25

Tr(Ee)

Tr(Za)

0.8

NC(H)(log10 scale)

Tr（Ew²)/C

Tr(Ew)/Tr(EB)

1

Tr(Ew)/Tr(EB)

20

1

0

0

-1

10

-2

-2

5

0.2

-3

-3

0

0.0

-4

-4

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

2.0

1.0

1.8

一

IIWll

Wllr

IIHII

-0.2

0.0

0.25

0.00

(W2.HA)

(WH,OF)

SNR(log10 scale)

0.5

1.6

IIHAlIr

（log10scale)

-0.4

scale)

-0.25

(W2HA.OF)

0.0

-0.6

W2(ETF)

W(ETF)

-0.50

-0.8

W(OF)

-0.75

-0.5

1.0

--W2(OF)

NC3

-1.00

=0.8

-1.0

-1.2

-1.25

1.5

WH

W2HA

0.6

0.4

-1.4

-1.50

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

1.0

20

0.8

NC(H)(log10 scale)

D

)

D

15

-5

-5

10

-10

-10

-15

5

0.2

-15

Tr(Ew)

Tr(EB)

-20

Tr(Ew)

Tr(Zs)

0

0.0

-20

Tr(Ew²/C

Tr(Ewg²)/C

Tr(Ew)/Tr(EB)

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0

2.5

iter%50000

5.0

7.5

10.0

12

一

2.0

lWell

0.0

W(ETF)

0.25

(W2.HA)

10

1.8

一

IIHAlLE

-0.2

W2(OF)

0.00

(W2HA,OF)

()

8

NC2(log10 scale)

-0.4

-0.25

6

-0.6

-0.50

-0.75

4

-0.8

NC3

-1.00

2

-1.0

-1.25

0

0.8

-1.2

-1.50

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

20.0

1.0

Tr(Ew)

Tr(Ew)

Tr(EB)

Tr(ZB)

17.5

0.8

NC(H)(log10 scale)

2

Tr(Ew²C

Tr(Ew)/Tr(Ee)

NC(HA)(log10 scale)

1

Tr(Ew)/Tr(Eg)

Tr(EwE²)/C

15.0

0

12.5

0

-1

7.5

-1

-2

5.0

2.5

0.2

-2

-3

0.0

0.0

2.5

iter%50000

5.0

7.5

10.0

0.0L

0.0

2.5

iter%50000

5.0

10.0

-3

0.0

2.5

iter%50000

5.0

7.5

10.0

-4

0.0

2.5

iter%50000

5.0

7.5

10.0

1.00

0.75

1.8

2.0

lIWell

IIHIlr

IIHAlIr

-0.2

0.0

0.25

0.00

(W2.HA)

0.50

(log10scale)

-0.4

scale)

-0.25

-0.50

0.25

-0.6

W2 (ETF)

0.00

-0.8

--W(OF)

-0.75

SNR

-0.25

1.0

=

NC2

-1.0

NC3

-1.00

-0.50

0.8

-1.2

-1.25

-0.75

W2HA

0.6

-1.4

-1.50

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

1.0

80

0

0

0.8

NC(H)(log10 scale)

-5

NC(HA)(log10 scale)

-5

60

-10

-10

15

-15

-20

-20

20

0.2

-25

Tr(Ew)

Tr(E8)

-25

Tr(Ew)

Tr(EB)

一

Tr(Ewg²)/C

一

Tr(Ewg²)/C

0

0.0

-30

一

Tr(Ew)/Tr(Ea)

-30

Tr(Ew)/Tr(Ea)

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

14

12

WH

W2HA

2.25

2.00

IIHI

Iwll

W2llr

-1

0

-1

0

SNR(log10 scale)

10

IIHAllr

)

-2

W(ETF)

NC3（log10 scale)

-2

(W.H)

1.50

-3

W(ETF)

E-

(W2.HA)

H(ETF)

(WH,ETF)

-4

HA(ETF)

-4

4

-5

=·W(OF)

W（OF)

-5

(WH,OF)

2

0.75

-6

-6

0

-7

-7

-2

0.0

2.5

5.0

7.5

10.0

0.50

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

8-

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

1.0

3

Tr(Ew)

Tr(Ew)

80

2

Tr(Ea)

Tr(ZB)

0.8

NC(H)(log10scale)

1

NC(HA)(log10 scale)

0

Tr(Ew)/Tr(EB)

60

-1

-1

-2

-2

20

0.2

-3

-3

-4

0

0.0

00g-

-4L

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

1.5

2.25

—Wll

0.0

)

1.0

0.5

2.00

IHAlL

IHII

W2ll

-1.0

-0.5

W (ETF)

W2(ETF)

scale)

-1.0

-0.5

0.0

1.5

H(ETF)

060

0.0

HA (ETF)

-1.5

-0.5

1.0

-2.0

W(OF)

W2(OF)

NC3

-2.0

(W2.HA)

(W.H)

=

2.5

(WH,ETF)

-1.0

WH

0.75

-2.5

(WH.OF)

0.50

-3.0

-3.0

-

(WHA.OF)

-1.5

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

60

1.0

50

0.8

NC(H)(log10 scale)

40

-5

30

-10

-10

20

-15

0.2

-15

Tr(Ew)

Tr(Ew)

10

Tr(ZB)

Tr(Za)

Tr(（w²)/C

-20

Tr(Ewg²)/C

0.0

-20

Tr(Ew)/Tr(EB)

Tr(Ew)/Tr(E)

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

2.2

lIW2ll

0

0

10

一

IIHII

2.0

一

IIHAIl

-1

1

8

-2

-2

6

1.6

-4

-3

W(ETF)

H(ETF)

HA(ETF)

-3

-4

(W2.HA)

·W(OF)

-5

-

2

1.0

-6

-6

-7

0

0.8

-7

8-

0.0

2.5

5.0

7.5

10.0

0:0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

60

1.0

3

Tr(Ew)

Tr(Zw)

Tr(Eg)

Tr(EB)

50

0.8

NC(H)(log10 scale)

2

Tr(Ew)/Tr(Es)

Tr(EwEg²)C

40

0

-1

20

-1

-2

10

0.2

-2

-3

0.0

-3

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

-4

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

(log10scale)

1.0

0.8

0.6

2.2

2.0

一

lHI

lIWllr

IIHAII

（log10scale)

-1.0

-0.5

-0.5

0.0

(W.HA)

0.4

1.6

-1.5

-1.0

0.2

0.0

NC3

1.5

-0.2

1.0

-2.5

H(ETF)

W2(ETF)

-2.0

-0.4

W2HA

0.8

-3.0

--.

HA(ETF)

W(OF)

2.5

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

0.0

2.5

5.0

7.5

10.0

iter%50000

iter%50000

iter%50000

iter%50000

H.3 Experiments with GNNs and Spectral Methods

The main focus of this section is to emphasize the differences between power iterations-based spectral methods and GNNs. To this end, we consider the following datasets and GNNs as follows:

GNNs: In our experiments, we empirically track the NC metrics of penultimate layer features during training for both the GNN designs ψ F Θ , ψ F ′ Θ . The number of layers is varied based on L = 64 , 128 and the hidden dimension is set to 8 across layers.

Spectral methods: The number of projected power-iterations are varied based on L = 64 , 128 for a fair comparison with GNNs.

Optimization: The GNNs are trained for 8 epochs using stochastic gradient descent (SGD) with learning rate of 0 . 004 , momentum 0 . 9 and a weight decay of 5 × 10 -4 .

Observations: For the dataset D 1 with homophilic graphs, we first plot the training metrics for GNNs ψ F Θ , ψ F ′ Θ with L = 64 in Figures 19, 20 respectively and ensure that they reach TPT. Now, from Figures 21, 22 observe that the ratios Tr ( Σ B ( x ( l ) )) / Tr ( Σ B ( w ( l -1) )) , Tr ( Σ W ( x ( l ) )) / Tr ( Σ W ( w ( l -1) )) tend to be constant throughout the power iterations for spectral methods, whereas, the GNNs behave differently as Tr ( Σ B ( X ( l ) )) / Tr ( Σ B ( H ( l -1) )) , Tr ( Σ W ( X ( l ) )) / Tr ( Σ W ( H ( l -1) )) tend to decrease across depth. We make similar observations for GNNs ψ F Θ , ψ F ′ Θ with L = 128 in Figures 23,24,25,26 and note that the behaviour is the same as L = 32 case illustrated in the main text. However, when considering the dataset D 2 with heterophilic graphs, even though the GNNs reach TPT during training (Figures 27,28,29,30 ), the evolution of ratios Tr ( Σ B ( X ( l ) )) / Tr ( Σ B ( H ( l -1) )) , Tr ( Σ W ( X ( l ) )) / Tr ( Σ W ( H ( l -1) )) tends to differ especially for the GNN ψ F ′ Θ when L = 64 (Figures 31,32) and L = 128 (Figures 33,34). Thus, highlighting the empirical role of depth in GNN design which requires further investigations in future efforts.

0.5

1.0

0.5

0.5

Tr(Ew)

0.4

0.9

NC(H)(log10 scale)

0.0

0.0

Tr(EwEg/C

Tr(Ew)/Tr(Eg)

0.3

-0.5

-0.5

SSOI

0.2

-1.0

-1.0

0.1

0.7

-1.5

Tr(Ew)

-1.5

-2.0

()

Tr(EwE²)/C

-2.0

0.0

0.6

-2.5

Tr(Ew)/Tr(Zg)

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

1.2

1.4

WH

W2HA

2.0

0.1

0.2

0.0

SNR(log10 scale)

1.0

（log10scale)

1.5

scale)

-0.1

0.1

0.8

0.6

1.0

IIWll

W2ll

-0.2

0.0

0.4

IIHIl

-0.3

-0.1

0.2

0.5

—IIHAIl

NC3

-0.2

-0.2

0.0

0.0

-0.5

-0.6

W(OF)

W(OF)

W (ETF)

W2 (ETF)

-0.3

-0.4

(W2.HA)

(WH,OF)

(W.H)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

0.40

1.00

1.0

Tr(Ew)

0.35

0.30

0.99

NC(H)（log10 scale)

0.5

0.0

Tr(Ewg²/C

Tr(Ew)/Tr(Eg)

0.0

0.5

0.20

0.25

-0.5

-0.5

Tr(Ew)

Tr(E8)

Tr(Ew)C

0.15

0.97

-1.0

-1.0

Tr(Ew)/Tr(Eg)

0.10

-1.5

-1.5

0.05

0.96

-2.0

-2.0

0.00

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

2.0

0.0

0.0

1.0

)

1.5

-0.2

-0.5

1.0

—IW2ll

IHIl

-0.4

-1.0

IHAIl

0.6

NC3

-1.5

0.4

0.0

-0.8

W2(ETF)

W(OF)

-2.0

(W2.HA)

2.5

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

4.5

Tr（Ewg)/C:Op

Tr（Ew²)/C:Op

Tr（Ew²)/C:Op

N

Tr（Ewg²)/C:Op

3.5

4.0

Tr(Ew)/Tr(Eg):Op

Tr（wg²)/C:normalize

Tr（Ew)/Tr(Za):normalize

m

Tr（Ewg²)/C:normalize

Tr（Ew)/Tr(Eg):Op

Tr（Ew)/Tr(Eg):normalize

2

Tr（Ew)/Tr(∑g）:Op

Tr（Ew)/Tr(Eg）:non-lin

Tr（Ew²）/C:non-lin

Tr（Ew²/C:normalize

1

Tr（Ewg²/C:normalize

N

1

Tr(Ew)/Tr(∑g):normalize

Tr（Ew)/Tr(Eg）:normalize

1

-1

1.5

1.0

0

-2

-2

0

10

20

Plidx

30

40

50

60

-1

0

10

20

Plidx

30

40

50

60

0

10

20

layer idx

30

40

50

60

0

10

20

layeridx

30

40

50

60

12

N

Tr(Ew）:Op

Tr(Ew):normalize

6

10

Tr（Ew）:Op

Tr(Ew):non-lin

10

Tr(Ew）:Op

Tr(Ew):non-lin

)

Tr(Ea):normalize

Tr(Eg）:Op

2

6

8

Tr（Ew):normalize

Tr（∑g）:Op

Tr(Eg):normalize

Tr(Eg):non-lin

Trace(log10 scale)

Tr（Eg）:normalize

Tr(Ew):normalize

Tr(∑g):non-lin

Tr(Eg）:Op

-2

。

-2

4

一

-4

Tr（Ew）:Op

2

-6

-6

Tr（∑g）:Op

Tr(Ew）:normalize

0

-8

0

10

20

Pl idx

30

40

50

60

8-

0

Tr(Eg):normalize

10

20

xp!d

30

40

50

60

0

10

20

layer idx

30

40

50

60

-2

0

10

20

layeridx

40

50

09

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

scale)

3.20

3.15

Tr（O）/T（-1)

Traceratio(log10 scale)

7.0

6.8

Tr（O）)T（-1)

Tr（2Op）/T（W-1）)

Trace ratio (log10 scale)

Tr（zo）/r（-)

Tr（zO）/T（W-11)

Trace ratio (log10 scale)

D

Tr(20）/T（1)

3.10

3.05

6.6

m

Traceratio（

3.00

2

2

2.95

6.4

1

0

2.90

6.2

0

2.85

0

10

20

layeridx

30

40

50

60

6.0

0

10

20

layeridx

30

40

50

60

0

10

20

layeridx

30

40

50

60

0

10

20

layeridx

30

40

50

60

0.6

1.0

0.5

0.5

Tr(Ew)

Tr(Z8)

0.5

0.9

NC(H)(log10 scale)

0.0

NC(HA)(log10 scale)

0.0

Tr(2wEg/C

0.4

0.8

0.5

-0.5

.3

-1.0

1.0

-1.5

0.2

0.6

-2.0

Tr(Ew)

1.5

0.1

0.5

-2.5

Tr(EwEg)/C

Tr(EB)

-2.0

0.0

Tr(Ew)/Tr(Ea)

0

2

4

6

8

0

2

4

6

8

-3.0L

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.25

2.0

1.00

0.0

0.2

0.75

())

1.5

)

-0.2

(log10scale)

0.1

0.50

1.0

IIWll

0.0

lIW2llr

0.25

IIHIl

-0.4

0.1

0.00

0.5

IIHAIIF

NC3

-0.2

-0.25

0.0

-0.6

W(ETF)

W(ETF)

-0.3

(W2.HA)

(W.H)

-0.50

0

2

4

6

WH

W2HA

8

-0.5

0

2

4

6

8

-0.8

0

2

4

6

W(OF)

W(OF)

8

-0.4

0

--

(W2HA.OF)

(WH.OF)

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.00

1.0

Tr(Ew)

0.25

0.30

660

NC(H)(log10 scale)

0.5

0.0

NC:(HA)(log10 scale)

0.0

0.5

Tr(Eg)

Tr(Ew)/Tr(Eg)

Tr(Ew²/C

0.20

-0.5

-0.5

-1.0

-1.0

0.10

0.96

1.5

Tr(Ew)

-1.5

0.05

Tr(Eg)

-2.0

Tr(EwE/C

-2.0

0.00

0.95

Tr(Ew)/Tr(Eg)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.2

—WHA

2.0

0.0

0.0

1.5

scale)

-0.1

scale)

-0.5

一

IIHI

IIW2ll

01601

-0.2

W2(ETF)

-1.0

一

IIHAlIF

NC2

-0.3

W2(OF)

NC3

1.5

-0.4

-2.0

(W2.HA)

0.4

0.0

-0.5

-2.5

(W2HA.OF)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

Tr（Ewg)/C:Op

Tr（EwE/C:Op

Tr(wg/C:normalize

Tr(Ewg²)/C:non-lin

Tr（Ew²）/C:non-lin

m

NC（H)(log10 scale)

m

Tr（Ew/Tr(Ea）:Op

Tr(Ew)/Tr(Eg):normalize

2

m

Tr（∑w²)/C:normalize

Tr(Ew)/Tr(Eg):Op

)

1

Tr（w/C:normalize

Tr（Ew）/Tr(Ee）:Op

Tr（Ew）/Tr(∑g）:non-lin

Tr（∑w)/Tr（Eg）:non-lin

Tr（Ew)/Tr（Eg）:normalize

Tr（Ew)/Tr（ze）:normalize

。

Tr（EwE²)C:Op

。

-1

-1

0

0

Tr（Ewg²/C:normalize

Tr（Ew）/Tr(Eg）:Op

Tr(Ew)/Tr(Eg):normalize

20

40

Plidx

60

80100120

-1

-2

0

20

40

60

Plidx

80

100120

-3

-2

0

20

40

layeridx

6080100120

-2

0

20

40

layer idx

60

80

100120

()

2

(s0

2

Tr（Ew）:Op

20

35

30

Tr(Ew):non-lin

Tr(Ew）:Op

Tr(Ew):normalize

Tr(Eg）:Op

Tr(Eg):normalize

Trace （log10 scale)

12

10

Tr（w）:Op

Tr（Ew):normalize

Tr(Ea):Op

Tr（Zg）:normalize

-2

。

Tr（Eg):Op

Tr(Ew):normalize

-2

Tr（Ea):normalize

-4

Tr（Ew）:Op

-6

Tr（Ew):normalize

Tr(Eg）:Op

-6

。

---

Tr(Eg):normalize

-2

0

20

40

PI idx

60

80100120

0

20

40

PIidx

60

80

100120

0

20

40

6080

layeridx

100120

0

20

40

layeridx

80

100120

(a) Normalized Laplacian

(b) Bethe Hessian

(c) GNN

ψ

F

Θ

(d) GNN

ψ

F

′

Θ

6.8

17.5

Tr（O）/T（W-2）)

3.2

Tr（2O）/T（-1）)

6.7

Tr（）/T（-1)

15.0

Tr（zOn）)T（2-1)

b

12.5

3.1

中

6.6

10.0

3.0

6.5

7.5

2

5.0

2.9

6.4

2.5

0

Tr（20p）)/Tr（r-1)

2.8

0.0

Tr（x）/T（-1)

0

20

40

layeridx

60

80

100

120

6.3

0

20

40

layeridx

60

80

100

120

0

20

40

layeridx

6080

100120

0

20

40

layeridx

60

80

100

120

1.0

1.0

0.5

0.5

0.9

0.5

NC(HA)(log10 scale)

0.0

0.4

0.0

0.5

0.5

-1.0

0.2

0.7

-1.0

Tr(Ew)

1.5

Tr(Ew)

0.1

Tr(Zg)

Tr(Eg)

0.0

0.6

-1.5

Tr(Ewg)C

Tr(Ew)/Tr(Eg)

-2.0

Tr(Ew)/Tr(Eg)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.25

2.0

0.1

(W.H)

1.00

0.2

(W2,HA)

)

0.75

1.5

0.0

0.1

.(W2HA.OF)

-(WH,OF)

0.50

1.0

IIW-ll

IIWIllr

-0.1

0.0

0.25

IIHIIr

-0.2

IIHAlIr

0.1

-0.25

0.00

-0.3

W(ETF)

NC3

-0.2

-0.50

WH

0.0

-0.4

W(OF)

W(OF)

W2(ETF)

-0.3

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

1.00

1.0

Tr(Ew)

0.5

0.4

0.98

NC(H)（log10 scale)

0.5

0.0

Tr(Ew)/Tr(∑g)

()

0.5

0.0

0.96

-0.5

-0.5

Tr(Ew)

Tr(E8)

Tr(wg²/C

0.2

0.92

-1.0

-1.0

Tr(Ew)/Tr(∑g)

0.1

0.90

-1.5

1.5

-2.0

-2.0

0.0

0.88

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

2.00

1.1

1.75

0.0

0.0

1.0

-0.2

（log10scale)

-0.5

—Wll

一

IHAlI

IIHILF

-0.4

-1.0

0.6

0.50

NC2

-0.6

NC3

-1.5

0.25

0.5

W2(ETF)

-2.0

一

(W2.HA)

0.4

0.00

-0.8

W(OF)

(W2HA,OF)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

0.6

1.00

0.5

0.5

Tr(Ew)

Tr(E8)

0.5

0.95

0.90

NC(H)(log10 scale)

0.0

NC(HA)(log10 scale)

0.0

Tr(Ew²/C

Tr(Ew)/Tr(Eg)

0.4

0.5

-0.5

1.0

-1.0

0.2

0.75

1.5

Tr(Ew)

-1.5

0.1

0.70

Tr(EB)

-2.0

-2.0

0.0

0.65

Tr(Ew)/Tr(Ea)

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

1.0

2.0

0.0

0.25

(W.H)

(W.HA)

0.8

-0.2

0.20

-.(WH,OF)

0.6

1.5

(log10scale)

-0.4

NC3(log10 scale)

0.15

.(WHA,OF)

IWll

0.4

1.0

lIWellr

-0.6

0.10

IIHIl

0.2

IIHAIlF

NC2

-0.8

0.05

0.0

-1.0

W(ETF)

0.00

-0.2

WH

W2HA

0.0

-1.2

1.4

W(OF)

W(OF)

W(ETF)

-0.05

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0

2

iter%1000

4

6

8

0.30

1.00

1.0

Tr(Ew)

0.25

0.99

0.0

0.5

NC(HA)(log10 scale)

0.5

0.0

Tr(Ew²)/C

Tr(Eg)

Tr(Ew)/Tr(EB)

0.20

0.15

0.5

-1.0

0.10

0.97

1.5

1.5

Tr(Ew)

0.05

Tr(EB)

0.96

Tr(EwEg/C

-2.0

0.00

-2.0

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

iter%1000

iter%1000

iter%1000

iter%1000

2.00

1.2

1.75

0.0

0.0

1.1

NC2(log10 scale)

-0.2

-0.5

-IWllr

一

IIHAlI

IIHIlI

-0.4

-1.0

0.8

-1.5

0.50

-0.6

0.6

0.25

W(ETF)

-2.0

(W2.HA)

0.5

0

2

4

6

8

0.00

0

2

4

6

8

-0.8

0

2

4

6

W(OF)

8

-2.5

0

(W2HA,OF)

2

6

8

iter%1000

iter%1000

iter%1000

iter%1000

(a) GNN

ψ

F

Θ

(b) GNN

ψ

F

′

Θ

Tr（Ew²)/C:Op

1.5

T(Ew²C:Op

Tr(Ew)/C:non-lin

1.0

Tr（w）/C:normalize

NC（H)(log10 scale)

Tr（w)/C:normalize

2

Tr(Ew）/Tr(EB):Op

Tr(∑w)/Tr(B）:Op

0.0

Tr（Ew)/Tr(Ee):normalize

.5

Tr(Ew）/Tr(Zg）:normalize

1.0

-1.5

1

-2.0

-2

-2.5

0

10

20

30

layeridx

40

50

60

0

10

20

3040

layeridx

50

60

12

Tr(Ew):Op

do:（

10

Tr(Ew):normalize

Tr(Ew):non-lin

6

Tr(Ew）:normalize

Trace(log10 scale)

8

Tr（Eg):Op

Tr（Zg):non-lin

Trace(log10 scale)

Tr（Eg):Op

Tr（Eg):non-lin

6

Tr(Eg):normalize

Tr（Es):normalize

4

0

0

0

10

20

30

layer idx

40

50

60

0

10

20

3040

layeridx

50

60

(a) GNN

ψ

F

Θ

(b) GNN

ψ

F

′

Θ

b

Trace ratio(log10 scale)

5

Traceratio(log10 scale)

4

3

2

0

0

Tr(）/(-11)

-1

0

10

20

layeridx

30

40

50

60

0

10

20

layeridx

30

40

50

60

(a) GNN

ψ

F

Θ

(b) GNN

ψ

F

′

Θ

Tr（Ewg)/C:Op

Tr（Ew²）/C:Op

Tr(Ew²）/C:non-lin

Tr(Ew）/C:non-lin

Tr（w)/C:normalize

Tr（Ew）/C:normalize

2

Tr(Ew)/Tr(Eg）:Op

Tr(Ew)/Tr(Ea）:Op

1

Tr(Ew)/Tr(Eg):non-lin

)

Tr（Ew）/Tr(∑s）:normalize

0

Tr（Ew)/Tr（∑s）:normalize

-1

-2

-2

-3

0

20

40

100120

0

20

40

6080

100120

layeridx

layeridx

OE

Tr（Ew):Op

一

Tr(Ew）:Op

Tr(Ew):non-lin

25

25

Tr（Eg）:Op

Tr（Ew):normalize

Tr(Ew）:normalize

Tr（(Eg):Op

20

Tr（Es):non-lin

Tr(Eg):non-lin

Tr（Ea):normalize

15

Tr(∑g):normalize

10

M

5

0

20

40

layer idx

60

80

100

120

0

20

40

layeridx

60

80

100120

(a) GNN

ψ

F

Θ

(b) GNN

ψ

F

′

Θ

Tr(2p)/T(W-1)

14

Tr（(2Op/T（W1）)

15.0

Tr（0）T（2-11)

12

Tr(2）/T（1)

Traceratio(log10scale)

12.5

Traceratio（log10 scale)

10

10.0

8

7.5

6

5.0

4

2.5

2

0

0.0

-2

0

20

40

layeridx

60

80

100

120

0

20

40

layeridx

60

80

100

120

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the relationship between graph topology and feature evolution in Graph Neural Networks (GNNs). The paper starts by discussing the phenomenon of Neural Collapse (NC) in instance-wise deep classifiers, where within-class variability decreases and class means align to specific symmetric structures. The study is then extended to node-wise classification using Stochastic Block Model (SBM) graphs. An empirical study reveals a decrease in within-class variability in GNNs trained for node classification on SBMs, although not as pronounced as in the instance-wise setting. The authors propose a graph-based mathematical model to understand the influence of node neighborhood patterns and community labels on NC dynamics. The model requires strict structural conditions on the graphs to exhibit exact variability collapse, highlighting the distinction between GNNs and plain deep neural networks. Gradient dynamics analysis of the graph-based model provides theoretical explanations for the observed partial collapse in GNNs. The paper also explores the evolution of features in well-trained GNNs and compares it to spectral clustering methods. Overall, the paper investigates feature evolution in GNNs and provides insights into the impact of graph topology on this process.\n\nSTRENGTHS:\n1. This paper focuses on an interesting topic: Neural Collapse during the training of graph neural networks.\n\n2. The paper concretely shows the decrease of within-class variability.\n\nWEAKNESSES:\n1. Analyzing only intra-class (within-class) variability without discussing inter-class variability is meaningless.\n\n2. The study of over-smoothing [1] phenomena has also discussed the decrease of within-class variability. A discussion about the difference between within-class variability caused by over-smoothing and neural collapse can clarify the contribution of this work.\n\n3. The theoretical analysis presented in this work has limited practical utility.\n\n\n[1] Keriven, Nicolas. \"Not too little, not too much: a theoretical analysis of graph (over) smoothing.\" *arXiv preprint arXiv:2205.12156* (2022).\n\nQUESTIONS:\n1. Can the theoretical analysis in this work lead to any practical design improvements?\n\n2. The term \"instance-wise case\" (line 11) should be defined.\n\n3. The notation of class is used in line 155 but is not defined in the 2.1 data model section.\n\n4. A more detailed discussion should be given about the regime of exact recovery (line 94).\n\n5. The optimizer is defined in line 147, but the loss or objective function to be optimized is not specified.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the feature evolution in Graph Neural Networks (GNNs) via the lens of Neural Collapse. They conduct an empirical study that reveals a decrease in within-class variability in the deepest features of GNNs, but not to the extent observed in instance-wise classification settings. By proposing and analyzing a graph-based Unconstrained Features Model (UFM), the authors show that a strict structural condition on the graphs is necessary for exact variability collapse, which is a rare event. They provide theoretical reasoning for the partial collapse observed during GNNs training and study the evolution of features across layers in well-trained GNNs, comparing the decrease in NC metrics with power iterations in spectral clustering methods.\n\nSTRENGTHS:\nThis paper presents a pioneering study on Neural Collapse (NC) in Graph Convolutional Networks (GCNs) and the forward process of a neural network, which has the potential to inspire future work. The authors provide unique and valuable theoretical results, such as the partial collapse in Theorem 3.1. Additionally, the paper is well-written, the methodology is clearly explained, and the proposed approaches appear to be solid.\n\nWEAKNESSES:\nThis paper has a few areas that could be improved upon.\n\n1. Some experimental findings could benefit from stronger support through theoretical results (Please refer to Questions 1-3).\n\n2. The applications and implications of certain theorems could be further elaborated (Please refer to Questions 4-5).\n\nQUESTIONS:\nTo enhance the manuscript, the authors may consider addressing the following points:\n\n1. In addition to the collapse of variability (NC1), the preference towards a simplex ETF (NC2) is an important observation in the original NC paper [34]. From Figure 1(c), it seems that some variant of NC2 might also hold in GCNs. To provide a more comprehensive understanding of NC, it would be helpful if the authors could include a theorem or at least a discussion on NC2.\n\n2. While Theorem 3.1 and Theorem 3.2 demonstrate that the within-class variability is non-zero with high probability, the authors have not provided a lower bound for the variability concerning p and q. Analyzing this lower bound could strengthen the paper's contribution. For example, the authors might consider analyzing the expected variability under SSBM.\n\n3. Theorem 3.3 appears to be somewhat weaker compared to existing studies on NC (e.g., [54]). It would be beneficial to provide a local or global convergence.\n\n4. In Lines 270 - 276, the authors show that the ratios in GCN behave differently compared to those in power iterations (i.e., simplified graph convolutional networks). It would be insightful if the authors could explain how this difference benefits GCNs. Additionally, the evolutions in $\\mathcal{F}$ and $\\mathcal{F}'$ are different, and understanding how this difference impacts the performance of $\\mathcal{F}$ and $\\mathcal{F}'$ would be valuable.\n\n5. Relating Theorem 4.1 to classic results in GNNs (e.g., over-smoothing) could provide further context and strengthen the paper's overall contribution to the field.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tries to investigate the feature evolution in GNNs in inductive setting. In particular, it focuses on the Neural Collapse problem using the SSBM data model to ensure the existence of this phenomenon. Moreover, it proposes to verify that the extent of NC in GNNs is not as severe as in instance-wise classifications.\n\nSTRENGTHS:\n1. It is interesting to study the feature evolution with GNNs and especially the correlation between the topological information.\n2. The data model is clearly presented and the experiments are close to the theoretical results.\n\nWEAKNESSES:\n1. The study of feature evolution with GNNs and especially the correlation between the topological information is a great topic, however, the data model and the concentration on the NC condition slightly retreat the original claim into a theoretically friendly but less practical direction.\n2. Most concerns are raised about the practical value of this study, which lies in two aspects:\n    1. the motivation of this work needs to be further clarified, e.g. why the NC phenomenon is important, what about other stages of trait development.\n    2. data model: SSBM is a good model to study the topological features.\n\nMy concerns are mainly about the practical significance of this paper. If I have missed anything important in the proofs and arguments, please let me know.\n\nQUESTIONS:\n1. Major:\n    1. A more comprehensive motivation for studying feature evolution in GNNs is highly recommended. And there is another smoothing problem: If the paper focuses on the NC view, the pre-convergence stage is completely missing. Since for most of the case or GNN practitioners are doing with pre-convergence stage, which should be more worthy to study. It is just to make sure that the problem and the perspective are really practical.\n    2. For the inductive setting, which this paper concentrates on, it is not necessary that the test nodes are in a different graph than the training nodes. It only requires that the test nodes do not appear in the training phase.\n    3. The experiments are performed only on the graphs simulated by the SSBM assumptions.\n2. Minor:\n    1. Line 53-54: Essentially, we highlight the main differences in the analysis of NC in GNNs by identifying structural conditions on the graphs under which the global minimizers of the training objective exhibit full NC1. Hard to understand, you can split it - since it is one of the main findings of the paper, which needs to be clear enough.\n    2. Line 79-80, the MLE loss is more and more popular to train DNN classifiers might be misleading somehow. However, MLE is a well-known alternative for more complex loss functions in machine learning theories.\n\nLIMITATIONS:\nLimitations are not included by the authors. As I presented questions above, my concerns are mainly about the practical significance of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEDIT: I am changing my score based on the revision. I think this is a very interesting paper. In particular, it helps us understand why GNNs work well, but not super-duper well. CNNs get full NC, but GNNs don't\n\nThis work discusses neural collapse of GNNs. Much of the work on neural collapse focuses on unconstrained features which is somewhat in conflict with the constraints imposed by graph structure. The main contributions of this work are.\n\n\n1. Empirical study showing that some decrease in within class variablility, but not to the same extent as in vanilla DNN\n\n2. Graph based UFM. Prove that with optimistic model, need strict structural condition to get full NC.\n\n3. Study gradient dynamics of graph-UFM that partially explains theoretical reaonsing for partial NC\n\n4. Compare NC1 metrics of GNNs vs spectral clustering\n\nOverall, this is a very good paper. It shows that GNNs exhibit partial, but not total neural collapse. In some sense, this may help explain why GNNs are near state of the art for many tasks, but nowhere near \"perfect\". However, I believe there is an error in the proof of theorem 3.2. I believe this error is fixable, but that the statement of the theorem will have to change. Therefore, I am strongly opposed to acceptance before this issue is fixed.\n\nI should also mention that I did not have time to check the last two proofs as thoroughly as I would like. The authors should make sure to check them very carefully before resubmission since there was already one mistake. (I will check aggressively then.)\n\nIf these issues are addressed, I would likely be in favor of acceptance.\n\nSTRENGTHS:\nThis paper contributes many new ideas to understand NC in the graph setting. Much of which are highly-nontrivial extensions of the original NC such as providing a characterization of when NC occurs and showing that the loss function will decrease, but level off before zero.\n\nWEAKNESSES:\nMajor Issues\n\nIn the proof of Theorem 3.1, it is not clear what $y_c$ is. This needs to be made more clear.\n\nIn Section D.2, you should recall the definitions of the different $\\Sigma$s or at least reference where they are defined. It is difficult to keep track of the many definitions.\n\nI don't understand the inequality of the form $P(A|B)\\leq P(A)$ given in 444. I also don't think you need it to obtain equation (50) which I think is correct. Could you please revise or clarify? (I am not overly concerned since the downstream equation appears to me to be correct. However, there is a similar issue for the diagonal terms which I think is more serious.)\n\nThe bound obtained in Theorem 3.2 is hard to parse an it is not obvious to me whether or not this quanity tends to zero. Perhaps you can approximate via sterlings inequality or something? Footnote 2 is unconvnincing since the terms (and the number of terms) change as n increases. I think you can argue (informally) that for large $n$ the probability that a Binomial RC is exactly equal to its mean, is equivalent to the probability that a normal RV with std $n^{-1/2}$ is within $1/n$ of its mean. Interpretation the standard deviation as average difference between the sample and the mean, this means that the probability of this occuring is on the order of $(1/n)/(1/\\sqrt{n})$ which then tends to zero as $n\\rightarrow \\infty$.\n\n(THIS IS THE BIG ONE): I don't think the argument used for the diagonal blocks is the proof of Theorem 3.2 regarding the upper bound for conditional probabilities is correct. For example, Set $t_{cc}=n-1$ and consider the probability of $E^c_{c,i}$ occurring given that $E^c_{c,i}$ for all $i<n$. Given that every other vertex is connected to $n-1$ other vertices of class $c$, this means that the set of all vertices in class $c$ form a clique. Therefore the probability of $E^c_{c,n}$ occurring is one. THIS ISSUE MUST BE FIXED IN ORDER FOR ME TO RECOMMEND ACCEPTANCE. Possible remedies include (i) convince me that I am mistaken (which is possible) (ii) remove the terms corresponding to the diagonal from the statement of the Lemma (in this case you should verify that this still converges to zero) (iii) add the assumptions that we are in the sparse regime to the statement of the lemma. Argue that cliques (or dense subgraphs in general) are exceptionally unlikely in this regime and that for reasonable values of $t$ the events are ``close to being independent.\" This is likely the best solution, but it would also require a bunch of non-trivial computations in order to make it precise (and the statement of the theorem would still need to change to reflect what you gt in these estimates) (iv) the event that the diagonal blocks satisfy the condition you need is equivalent to an erdos renyi random graph being regular. There is probably a bunch of existing work on this topic.\n\nMinor Issues:\n\nIn the summation on line 121, adding a subscript j to the summation would make things more clear, i.e., $j \\in \\mathcal{N}_c(v_c,i)$\n\nThe capitlizatin is off in some of the refreeneds, in Euclidean shoudl be capitalized in ref [7]. Please fix.\n\nLine 344 in the proof of Theorem 3.1: ``Theorem 3.1\" should be capitlalized. \n\nIt might be a good idea to first explain the idea of the proof of theorem 3.2, i.e. that first you show that in the limit, every vertex in class c has the same degree. Therefore, the stated condition that all of the vertices in $v_i$ have the same \\textit{fraction} or neighbors in each class $c'$ is equivalent to them having the same \\textit{number} or neighbors in $c'$ (which can then be estimated more easily using formulae for binomial RVs) \n\nLine 419 in the supplement. I think you should remove the word essentially, right? The number of neigbors IS a binomial RV.\n\nQUESTIONS:\n\\subsection{Questions}\n1. Line 44, when you say that the rules become similar to the nearest class center in the feature space do you mean the classifier is effectively an NN-classifier?\n\n2. Why is there a $y_k$ in (1), but $y_k(V_k)$ in (2)? This seems inconsistent. \n\n3. In the ``simpler\" model (4), is the definition of $\\hat{A}$ the same as in (3)? In this setting it would make sense to add self loops as in Kipf and Welling, i.e., $\\hat{A}=(A+I)(D+I)^{-1}$. Otherwise, nodes don't communicate with themselves at all. Am I missing something?\n\n4. For the sake of self-containedness, could you please briefly explain the instance-normalizaiton?\n\n5. How important is the assumption of balanced communities in your theoretical analysis and how does this affect the applicablity of your theory to real-world imbalanced datasets\n\n6. Hidden feature is the same size in all layers. How does this affect the results? What happens if you first increase and then decrease (or something) is there still partial NC1?\n\n7. In the Graph UFM, if the $H_k$ are freely optimizable, why do you still need seperate matrices $W_2$ and $H_k$?v (This may be a standard thing. I am not very familiar with UFMs)\n\n8. For the stochasitc block models, shouldn't there be some sort of concentration of measure result where for large $n$ we have $A\\approx EA$? This should allow you to apply theorem 3.3 to it, right? \n\n9. In Theorem 3.3, you obtain that $\\tilde{NC}_1$ decreases which means it converges to some non-zero limit. Is there a good way to think about what this limit is? Does it, for example go to zero in the case that $E\\rightarrow 0$? How does it depend on $\\alpha?$\n\n10. In line 427 of the proof of theorem 3.2, I think you need probability $1-4n^{-r}$ because of the union bound. (This is unimportant but should be fixed.)\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}