SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To this end, in the unconstrained features setting, Theorem 3.1 indicates that the graph must satisfy a strict structural condition for having an exact neural collapse solution as a minimizer of the empirical risk. Additionally, we obtained conditions on the amount of regularization needed for  $\Sigma_B(\mathbf{H}^{L - 1})$  to increase along the gradient flow in Theorem 3.3 (also see Appendix F). However, since we cannot expect graphs to satisfy this condition (Theorem 3.2), a potential alternative is to explore graph rewiring techniques, which are discussed below.
A.2 Graph Rewiring
Graph rewiring techniques aim to propagate messages between nodes via computational graphs that are suitable for the task [5-7, 19, 21, 23, 24, 57]. One of the popular examples is the Personalized PageRank (PPR) and Heat kernel (HK) based diffusion on graphs by Gasteiger et al. [21]. This approach leverages the diffusion matrix to facilitate message-passing between nodes many hops apart in the actual graph. However, the graph resulting from the diffusion operations tends to be quite dense. On the other hand, recent works on addressing over-squashing in long-range dependent tasks leverage the "curvature" information for rewiring [57].
In the context of our paper, one can aim to design a rewiring technique that can modify the input graph to satisfy condition C. In previous work, Ma et al. [40] generate synthetic graphs based on CORA by ignoring the original graph and adding edges between nodes to satisfy certain uniform neighborhood distributions. Alternatively, exploring an edge re-weighting scheme as proposed by Yan et al. [64] can also be an interesting research direction. From a scalability perspective, a neighborhood sampling technique based on condition C can also aid in better representation learning. Additionally, note that condition C is not limited to homophilic graphs and can be extended to heterophilic settings as well [38, 73], provided the sum of slices of the columns in  $\hat{\mathbf{A}}\in \mathbb{R}^{N\times N}$  are the same for all nodes in a class/community (refer to the necessity condition in the proof of Theorem 3.1 in Appendix D for more details). We believe that further research in this direction can shed some light on Conjecture 3.1 and improve our understanding of the nature of global minimizers.
A.2.1 Measuring neighborhood similarity
A key question that pops up when rewiring a graph to achieve condition  $\mathbf{C}$  is a trackable metric for node neighborhoods. The 'Cross-Class Neighborhood Similarity (CCNS)' metric by Ma et al. [40] serves as a good starting point to numerically track the cosine similarity in intra-class neighborhoods and dissimilarity of inter-class neighborhoods of nodes. The definition from Ma et al. [40] is given as:
Definition A.2. Cross-Class Neighborhood Similarity (CCNS): Given a graph  $\mathcal{G}$  and labels  $\mathbf{y}$  for all nodes, the CCNS between classes  $c, c'$  is  $s(c, c') = \frac{1}{|\mathcal{V}_c| |\mathcal{V}_{c'}|} \sum_{i \in \mathcal{V}_c, j \in \mathcal{V}_{c'}} \cos(d(i), d(j))$  where  $\mathcal{V}_c$  indicates the set of nodes in class  $c$  and  $d(i)$  indicates the empirical histogram (over  $C$  classes) of node  $i$ 's neighbors' labels, and the function  $\cos(.,.)$  measures the cosine similarity.
A limitation of this metric is that the CCNS between two classes is measured as an average of the cosine similarity of node neighborhood histograms while failing to incorporate the variance of these neighborhood similarities. Now, note that when condition  $\mathbf{C}$  in Theorem 3.1 is combined with CCNS, we can ensure that the variance of the cosine similarities is zero for any pair of classes  $c, c' \in [C]$ . Overall, better metrics based on different similarity measures and the condition  $\mathbf{C}$ , along with efficient rewiring techniques to maximize/minimize such metrics can be a valuable future effort in the community.
B A Brief Note on Exact Recovery of Planted Communities
Phase transitions in recovering the planted communities of the Stochastic Block Model (SBM) graphs have been extensively studied in the literature. In this context, 'exact recovery' indicates a perfect assignment of nodes to their respective communities/clusters. For the bi-clustering problem (i.e.,  $C = 2$ ), one can date back to the works of Bui et al. [13] for min-cut based clustering, and Boppana [10] for a spectral clustering based approach (Refer to [1, 2] for a historical perspective and additional references on this topic). Along these series of developments to find thresholds in terms of  $p, q$  for the exact recovery of communities, the seminal work of Abbe et al. [3] leveraged an information theoretic perspective to identify sharp thresholds in the logarithmic degree settings. The requirement for logarithmic degree can be understood from the following observation. If we consider  $p = q$ , then the SBM is essentially an Erdos-Renyi (ER) random graph model with edge-connection probability  $p$ . Based on the results by Erdős et al. [17], if  $p = \frac{c\log N}{N}$ , then a randomly sampled ER graph from  $ER(N, \frac{c\log N}{N})$  is connected with high probability if and only if  $c > 1$  (see section 2.5 in [1]). Thus, in order to achieve exact recovery, one must ensure that the SBM graph is connected along with some over-sampling of edges. In the Symmetric SBM case, when  $p = \frac{a\log N}{N}, q = \frac{b\log N}{N}$  this condition can be represented using the  $|\sqrt{a} - \sqrt{b}| > \sqrt{C}$  inequality (see Theorem 13 and the following remarks in [1]). Finally, observe that from a Neural Collapse perspective, we sample SSBM graphs in this regime to ensure that a GNN can achieve zero node-classification error and reach TPT.
C Additional neural collapse metrics
In this appendix, we define additional NC metrics pertaining to NC1-3 for our problem setup.
- Variability collapse in neighborhood-aggregated features  $\mathbf{H}\widehat{\mathbf{A}}$ : We track the within- and between-class variability of the "neighborhood-aggregated" features matrix  $\mathbf{H}\widehat{\mathbf{A}}$  by defining the covariance matrices  $\boldsymbol{\Sigma}_{W}\left(\mathbf{H}\widehat{\mathbf{A}}\right), \boldsymbol{\Sigma}_{B}\left(\mathbf{H}\widehat{\mathbf{A}}\right)$  as:
$$
\boldsymbol {\Sigma} _ {W} \left(\mathbf {H} \widehat {\mathbf {A}}\right) := \frac {1}{C n} \sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {n} \left(\mathbf {h} _ {c, i} ^ {\mathcal {N}} - \overline {{\mathbf {h}}} _ {c} ^ {\mathcal {N}}\right) \left(\mathbf {h} _ {c, i} ^ {\mathcal {N}} - \overline {{\mathbf {h}}} _ {c} ^ {\mathcal {N}}\right) ^ {\top}
$$
$$
\boldsymbol {\Sigma} _ {B} \left(\mathbf {H} \widehat {\mathbf {A}}\right) := \frac {1}{C} \sum_ {c = 1} ^ {C} \left(\overline {{\mathbf {h}}} _ {c} ^ {\mathcal {N}} - \overline {{\mathbf {h}}} _ {G} ^ {\mathcal {N}}\right) \left(\overline {{\mathbf {h}}} _ {c} ^ {\mathcal {N}} - \overline {{\mathbf {h}}} _ {G} ^ {\mathcal {N}}\right) ^ {\top}
$$
To this end, we define the  $\mathcal{NC}_1$ $\left(\mathbf{H}\widehat{\mathbf{A}}\right)$ $\widetilde{\mathcal{NC}_1}$ $\left(\mathbf{H}\widehat{\mathbf{A}}\right)$  metrics as follows:
$$
\mathcal {N C} _ {1} (\mathbf {H} \widehat {\mathbf {A}}) = \frac {1}{C} \operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} \widehat {\mathbf {A}}) \boldsymbol {\Sigma} _ {B} ^ {\dagger} (\mathbf {H} \widehat {\mathbf {A}})\right), \quad \widetilde {\mathcal {N C}} _ {1} (\mathbf {H} \widehat {\mathbf {A}}) = \frac {\operatorname {T r} \left(\boldsymbol {\Sigma} _ {W} (\mathbf {H} \widehat {\mathbf {A}})\right)}{\operatorname {T r} \left(\boldsymbol {\Sigma} _ {B} (\mathbf {H} \widehat {\mathbf {A}})\right)}. \tag {18}
$$
Their primary purpose is to track the within-class variability of neighborhood aggregated features  $\widehat{\mathbf{H}\mathbf{A}}$  relative to their between-class variability, both now dependent on the topological structure of graph  $\mathcal{G}$ . The motivation to track these features arises from the P2 condition in the proof of theorem 3.1 in Appendix B. Essentially, this condition states that the neighborhood aggregated features should collapse to their respective class means for the minimizer to satisfy NC.
- SNR for variance collapse: We track the following 'Signal-to-Noise' ratios pertaining to variability collapse of  $\mathbf{H}$  and  $\mathbf{H}^{\hat{\mathbf{A}}}$ :
$$
S N R \left(\mathcal {N} \mathcal {C} _ {1}\right) := \frac {\left\| \mathbf {W} _ {1} \left(\overline {{\mathbf {H}}} \otimes \mathbf {1} _ {n} ^ {\top}\right) \right\| _ {F}}{\left\| \mathbf {W} _ {1} \left(\mathbf {H} - \overline {{\mathbf {H}}} \otimes \mathbf {1} _ {n} ^ {\top}\right) \right\| _ {F}} \tag {19}
$$
$$
S N R \left(\mathcal {N C} _ {1} ^ {\mathcal {N}}\right) := \frac {\left\| \mathbf {W} _ {2} \left(\overline {{\mathbf {H}}} ^ {\mathcal {N}} \otimes \mathbf {1} _ {n} ^ {\top}\right) \right\| _ {F}}{\left\| \mathbf {W} _ {2} \left(\mathbf {H} \widehat {\mathbf {A}} - \overline {{\mathbf {H}}} ^ {\mathcal {N}} \otimes \mathbf {1} _ {n} ^ {\top}\right) \right\| _ {F}} \tag {20}
$$
These SNR metrics provide an alternate perspective for us to empirically analyze the desirability of variance collapse. Here  $\overline{\mathbf{H}} := \left[\overline{\mathbf{h}}_1 \cdots \overline{\mathbf{h}}_C\right] \in \mathbb{R}^{d_l \times C}$  and  $\overline{\mathbf{H}}^{\mathcal{N}} := \left[\overline{\mathbf{h}}_1^{\mathcal{N}} \cdots \overline{\mathbf{h}}_C^{\mathcal{N}}\right] \in \mathbb{R}^{d_l \times C}$  are the class-mean matrices without and with neighborhood aggregation respectively.
- Convergence of weights to a simplex ETF: To track the convergence of the weights  $\mathbf{W}_1, \mathbf{W}_2$  to a simplex ETF structure, we define  $\mathcal{NC}_2^{ETF}(\mathbf{W}_1), \mathcal{NC}_2^{ETF}(\mathbf{W}_2)$  as:
$$
\mathcal {N C} _ {2} ^ {E T F} \left(\mathbf {W} _ {1}\right) := \left\| \frac {\mathbf {W} _ {1} \mathbf {W} _ {1} ^ {\top}}{\left\| \mathbf {W} _ {1} \mathbf {W} _ {1} ^ {\top} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {21}
$$
$$
\mathcal {N C} _ {2} ^ {E T F} \left(\mathbf {W} _ {2}\right) := \left\| \frac {\mathbf {W} _ {2} \mathbf {W} _ {2} ^ {\top}}{\left\| \mathbf {W} _ {2} \mathbf {W} _ {2} ^ {\top} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {22}
$$
- Convergence of weights to an Orthogonal Frame (OF): To track the convergence of the weights  $\mathbf{W}_1, \mathbf{W}_2$  to an orthogonal frame structure, we define  $\mathcal{NC}_2^{OF}(\mathbf{W}_1), \mathcal{NC}_2^{OF}(\mathbf{W}_2)$  as:
$$
\mathcal {N C} _ {2} ^ {O F} \left(\mathbf {W} _ {1}\right) := \left\| \frac {\mathbf {W} _ {1} \mathbf {W} _ {1} ^ {\top}}{\left\| \mathbf {W} _ {1} \mathbf {W} _ {1} ^ {\top} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {23}
$$
$$
\mathcal {N C} _ {2} ^ {O F} \left(\mathbf {W} _ {2}\right) := \left\| \frac {\mathbf {W} _ {2} \mathbf {W} _ {2} ^ {\top}}{\left\| \mathbf {W} _ {2} \mathbf {W} _ {2} ^ {\top} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {24}
$$
- Convergence of features to a simplex ETF: To track the convergence of the features  $\mathbf{H}$ ,  $\mathbf{H}^{\widehat{\mathbf{A}}}$  to a simplex ETF structure, we define  $\mathcal{NC}_2^{ETF}(\mathbf{H}), \mathcal{NC}_2^{ETF}(\mathbf{H}^{\widehat{\mathbf{A}}})$  as:
$$
\mathcal {N C} _ {2} ^ {E T F} (\mathbf {H}) := \left\| \frac {\tilde {\mathbf {H}} ^ {\top} \tilde {\mathbf {H}}}{\left\| \tilde {\mathbf {H}} ^ {\top} \tilde {\mathbf {H}} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {25}
$$
$$
\mathcal {N C} _ {2} ^ {E T F} (\mathbf {H} \widehat {\mathbf {A}}) := \left\| \frac {\widetilde {\mathbf {H}} ^ {\mathcal {N} \top} \widetilde {\mathbf {H}} ^ {\mathcal {N}}}{\left\| \widetilde {\mathbf {H}} ^ {\mathcal {N} \top} \widetilde {\mathbf {H}} ^ {\mathcal {N}} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {26}
$$
where  $\widetilde{\mathbf{H}} := [\overline{\mathbf{h}}_1 - \overline{\mathbf{h}}_G \dots \overline{\mathbf{h}}_C - \overline{\mathbf{h}}_G] \in \mathbb{R}^{d_l \times C}$  and  $\widetilde{\mathbf{H}}^{\mathcal{N}} := [\overline{\mathbf{h}}_1^{\mathcal{N}} - \overline{\mathbf{h}}_G^{\mathcal{N}} \dots \overline{\mathbf{h}}_C^{\mathcal{N}} - \overline{\mathbf{h}}_G^{\mathcal{N}}] \in \mathbb{R}^{d_l \times C}$  are the re-centered class-means without and with neighborhood aggregation respectively.
- Convergence of features to an OF: To track the convergence of the features  $\mathbf{H}, \mathbf{H}^{\widehat{\mathbf{A}}}$  to an OF structure, we define  $\mathcal{NC}_2^{OF}(\mathbf{H}), \mathcal{NC}_2^{OF}(\mathbf{H}^{\widehat{\mathbf{A}}})$  as:
$$
\mathcal {N C} _ {2} ^ {O F} (\mathbf {H}) := \left\| \frac {\overline {{\mathbf {H}}} ^ {\top} \overline {{\mathbf {H}}}}{\left\| \overline {{\mathbf {H}}} ^ {\top} \overline {{\mathbf {H}}} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {27}
$$
$$
\mathcal {N C} _ {2} ^ {O F} (\mathbf {H} \widehat {\mathbf {A}}) := \left\| \frac {\overline {{\mathbf {H}}} ^ {\mathcal {N} \top} \overline {{\mathbf {H}}} ^ {\mathcal {N}}}{\left\| \overline {{\mathbf {H}}} ^ {\mathcal {N} \top} \overline {{\mathbf {H}}} ^ {\mathcal {N}} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {28}
$$
- Generic alignment of weights and features: To track the alignment of  $\mathbf{W}_1$  with its dual  $\mathbf{H}$ , we define  $\mathcal{NC}_3(\mathbf{W}_1, \mathbf{H})$  as:
$$
\mathcal {N C} _ {3} \left(\mathbf {W} _ {1}, \mathbf {H}\right) := \left\| \frac {\mathbf {W} _ {1}}{\left\| \mathbf {W} _ {1} \right\| _ {F}} - \frac {\overline {{\mathbf {H}}} ^ {\top}}{\left\| \overline {{\mathbf {H}}} \right\| _ {F}} \right\| _ {F} \tag {29}
$$
Similarly, to track the alignment of  $\mathbf{W}_2$  with its dual  $\mathbf{H}\hat{\mathbf{A}}$ , we define  $\mathcal{NC}_3(\mathbf{W}_2,\mathbf{H}\hat{\mathbf{A}})$  as:
$$
\mathcal {N C} _ {3} \left(\mathbf {W} _ {2}, \mathbf {H} \widehat {\mathbf {A}}\right) := \left\| \frac {\mathbf {W} _ {2}}{\left\| \mathbf {W} _ {2} \right\| _ {F}} - \frac {\overline {{\mathbf {H}}} ^ {\mathcal {N} \top}}{\left\| \overline {{\mathbf {H}}} ^ {\mathcal {N}} \right\| _ {F}} \right\| _ {F} \tag {30}
$$
- Alignment of weights and features with respect to simplex ETF: To track the alignment of  $\mathbf{W}_1$  and its dual  $\mathbf{H}$  with respect to a simplex ETF, we define  $\mathcal{NC}_3^{ETF}(\mathbf{W}_1,\mathbf{H})$  as:
$$
\mathcal {N C} _ {3} ^ {E T F} \left(\mathbf {W} _ {1}, \mathbf {H}\right) := \left\| \frac {\mathbf {W} _ {1} \tilde {\mathbf {H}}}{\left\| \mathbf {W} _ {1} \tilde {\mathbf {H}} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {31}
$$
Similarly, we track the alignment of  $\mathbf{W}_2$  and its dual  $\mathbf{H}\hat{\mathbf{A}}$  with respect to a simplex ETF using:
$$
\mathcal {N C} _ {3} ^ {E T F} \left(\mathbf {W} _ {2}, \mathbf {H} \widehat {\mathbf {A}}\right) := \left\| \frac {\mathbf {W} _ {2} \widetilde {\mathbf {H}} ^ {\mathcal {N}}}{\left\| \mathbf {W} _ {2} \widetilde {\mathbf {H}} ^ {\mathcal {N}} \right\| _ {F}} - \frac {1}{\sqrt {C - 1}} \left(\mathbf {I} _ {C} - \frac {1}{C} \mathbf {1} _ {C} \mathbf {1} _ {C} ^ {\top}\right) \right\| _ {F} \tag {32}
$$
- Alignment of weights and features with respect to OF: To track the alignment of  $\mathbf{W}_1$  and its dual  $\mathbf{H}$  with respect to an OF, we define  $\mathcal{NC}_3^{OF}(\mathbf{W}_1,\mathbf{H})$  as:
$$
\mathcal {N C} _ {3} ^ {O F} \left(\mathbf {W} _ {1}, \mathbf {H}\right) := \left\| \frac {\mathbf {W} _ {1} \overline {{\mathbf {H}}}}{\left\| \mathbf {W} _ {1} \overline {{\mathbf {H}}} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {33}
$$
Similarly, we track the alignment of  $\mathbf{W}_2$  and its dual  $\mathbf{H}\hat{\mathbf{A}}$  with respect to an OF using:
$$
\mathcal {N C} _ {3} ^ {O F} \left(\mathbf {W} _ {2}, \mathbf {H} \widehat {\mathbf {A}}\right) := \left\| \frac {\mathbf {W} _ {2} \overline {{\mathbf {H}}} ^ {\mathcal {N}}}{\left\| \mathbf {W} _ {2} \overline {{\mathbf {H}}} ^ {\mathcal {N}} \right\| _ {F}} - \frac {\mathbf {I} _ {C}}{\sqrt {C}} \right\| _ {F} \tag {34}
$$
In this appendix, we present the proof for Theorem 3.1 by analyzing the sufficiency and necessity conditions of the graph structure given by:
$$
\left(s _ {c 1, 1}, \dots , s _ {c C, 1}\right) = \dots = \left(s _ {c 1, n}, \dots , s _ {c C, n}\right), \quad \forall c \in [ C ], \tag {C}
$$
where the fraction of neighbors of node  $v_{c,i}$  that belong to class  $c'$  as  $s_{cc',i} = \frac{|\mathcal{N}_{c'}(v_{c,i})|}{|\mathcal{N}(v_{c,i})|}$ .
Proof sketch: Our aim is to identify the structural properties of a graph  $\mathcal{G}$  (especially  $\widehat{\mathbf{A}}$ ), such that the features  $\mathbf{H}$ , which exhibit neural collapse are indeed the minimizers of the risk. First, we obtain a lower bound for the risk using Jensen's inequality and show that, for a 'collapsed'  $\mathbf{H}$  to be the minimizer, it is sufficient if the graph  $\mathcal{G}$  satisfies condition  $\mathbf{C}$ . However, the inequality is applied on a convex function of  $\{\mathbf{e}_{c,i}\}$  (standard basis vectors) that is not strictly convex, and so, this analysis does not imply necessity. Thus, we show the necessity of condition  $\mathbf{C}$  for a 'collapsed'  $\mathbf{H}$  to be a minimizer of the risk by analyzing the optimality conditions of the stationary points. Additional details of the sketch for the 'necessity' argument are also presented.
Sufficiency: We begin by revisiting the risk  $\widehat{\mathcal{R}}^{\mathcal{F}'}$  for  $K = 1$ . For simplicity, we drop the superscript  $\mathcal{F}'$ , subscript  $k$ , and treat the unconstrained features of the corresponding graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$  as  $\mathbf{H}$  and denote  $\widehat{\mathbf{A}} = \mathbf{A}\mathbf{D}^{-1}$ . The risk  $\widehat{\mathcal{R}}(\mathbf{W}_2, \mathbf{H})$  is now given by:
$$
\begin{array}{l} \widehat {\mathcal {R}} \left(\mathbf {W} _ {2}, \mathbf {H}\right) := \frac {1}{2 N} \left\| \mathbf {W} _ {2} \mathbf {H} \widehat {\mathbf {A}} - \mathbf {Y} \right\| _ {F} ^ {2} + \frac {\lambda_ {\mathbf {H}}}{2} \| \mathbf {H} \| _ {F} ^ {2} + \frac {\lambda_ {\mathbf {W} _ {2}}}{2} \| \mathbf {W} _ {2} \| _ {F} ^ {2} \tag {35} \\ =: \widehat {\mathcal {L}} \left(\mathbf {W} _ {2}, \mathbf {H}\right) + \frac {\lambda_ {\mathbf {W} _ {2}}}{2} \left\| \mathbf {W} _ {2} \right\| _ {F} ^ {2}. \\ \end{array}
$$
Now, by denoting  $\mathbf{e}_{c,i} \in \mathbb{R}^N$  as the one-hot vector associated with the index of the feature column  $\mathbf{h}_{c,i}$  (among the  $N$  feature columns), we lower bound  $\widehat{\mathcal{L}}(\mathbf{W}_2, \mathbf{H})$  as follows:
$$
\begin{array}{l} \widehat {\mathcal {L}} (\mathbf {W} _ {2}, \mathbf {H}) = \frac {1}{2 N} \left\| \mathbf {W} _ {2} \mathbf {H} \widehat {\mathbf {A}} - \mathbf {Y} \right\| _ {F} ^ {2} + \frac {\lambda_ {H}}{2} \| \mathbf {H} \| _ {F} ^ {2} \\ = \frac {1}{2 N} \sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {n} \left\| \mathbf {W} _ {2} \mathbf {H} \widehat {\mathbf {A}} \mathbf {e} _ {c, i} - \mathbf {y} _ {c} \right\| _ {F} ^ {2} + \frac {\lambda_ {H}}{2} \sum_ {c = 1} ^ {C} \sum_ {i = 1} ^ {n} \| \mathbf {H} \mathbf {e} _ {c, i} \| _ {2} ^ {2} \\ = \frac {1}{2 N} \sum_ {c = 1} ^ {C} \frac {n}{n} \sum_ {i = 1} ^ {n} \left\| \mathbf {W} _ {2} \mathbf {H} \widehat {\mathbf {A}} \mathbf {e} _ {c, i} - \mathbf {y} _ {c} \right\| _ {F} ^ {2} + \frac {\lambda_ {H}}{2} \sum_ {c = 1} ^ {C} \frac {n}{n} \sum_ {i = 1} ^ {n} \| \mathbf {H} \mathbf {e} _ {c, i} \| _ {2} ^ {2} \tag {36} \\ \geq \frac {1}{2 N} \sum_ {c = 1} ^ {C} n \left\| \mathbf {W} _ {2} \mathbf {H} \widehat {\mathbf {A}} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {e} _ {c, i} - \mathbf {y} _ {c} \right\| _ {F} ^ {2} + \frac {\lambda_ {H}}{2} \sum_ {c = 1} ^ {C} n \left\| \mathbf {H} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {e} _ {c, i} \right\| _ {2} ^ {2} \\ = \frac {1}{2 N} \sum_ {c = 1} ^ {C} n \left\| \mathbf {W} _ {2} \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {h} _ {c, i} ^ {\mathcal {N}} - \mathbf {y} _ {c} \right\| _ {F} ^ {2} + \frac {\lambda_ {H}}{2} \sum_ {c = 1} ^ {C} n \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {h} _ {c, i} \right\| _ {2} ^ {2}. \\ \end{array}
$$
Note that Jensen's inequality, which we used above, is applied on a convex function of  $\mathbf{e}_{c,i}$ 's that is not strictly convex. Therefore, the lower-bound in equation 36 can be attained despite having different  $\mathbf{e}_{c,i}$ 's, e.g., when the properties  $\mathbf{P1}$  and  $\mathbf{P2}$ , which are stated below, hold  $\forall c\in [C]$ :
- P1:  $\mathbf{h}_{c,1} = \dots = \mathbf{h}_{c,n} = \overline{\mathbf{h}_c}$
P2:  $\mathbf{h}_{c,1}^{\mathcal{N}} = \dots = \mathbf{h}_{c,n}^{\mathcal{N}} = \overline{\mathbf{h}}_c^{\mathcal{N}}$
The first property  $\mathbf{P1}$  indicates zero intra-class variability of  $\mathbf{H}$ , i.e.,  $\Sigma_W(\mathbf{H})\to \mathbf{0}$ , and the second property  $\mathbf{P2}$  indicates zero intra-class variability of  $\mathbf{H}\hat{\mathbf{A}}$  i.e.,  $\Sigma_W(\mathbf{H}\hat{\mathbf{A}})\to \mathbf{0}$ .
Recall condition C:
When  $K > 1$ , observe that  $\widehat{\mathcal{R}} (\mathbf{W}_2,\{\mathbf{H}\}_{k = 1}^K) = \sum_{k = 1}^K\widehat{\mathcal{L}} (\mathbf{W}_2,\mathbf{H}_k) + \frac{\lambda\mathbf{W}_2}{2}\| \mathbf{W}_2\| _F^2$ . Thus, each of the  $\widehat{\mathcal{L}} (\mathbf{W}_2,\mathbf{H}_k)$  terms can be lower-bounded independently, resulting in a lower-bound for  $\widehat{\mathcal{R}} (\mathbf{W}_2,\{\mathbf{H}\}_{k = 1}^K)$  itself.
$$
\bullet \mathbf {C}: \quad (s _ {c 1, 1}, \dots , s _ {c C, 1}) = \dots = (s _ {c 1, n}, \dots , s _ {c C, n}) = (s _ {c 1}, \dots , s _ {c C}),
$$
where  $(s_{c1},\dots ,s_{cC})$  (shared by any node  $i\in [n]$  in class  $c$ ) represents any suitable tuple of the ratio of neighbors per class<sup>9</sup>.
We just need to show that if  $\mathbf{C}$  is satisfied then  $\mathbf{H}$  with both  $\mathbf{P1}$  and  $\mathbf{P2}$  exists. Equivalently, we can assume  $\mathbf{P1}$  (which is in the "feasible set" of the optimization) and show that then  $\mathbf{C}$  implies  $\mathbf{P2}$ . Indeed, in this case
$$
\begin{array}{l} \mathbf {h} _ {c, i} ^ {\mathcal {N}} = \frac {\sum_ {\mathcal {N} _ {c} \left(v _ {c , i}\right)} \mathbf {h} _ {c , j} + \sum_ {\mathcal {N} _ {c ^ {\prime} \neq c} \left(v _ {c , i}\right)} \mathbf {h} _ {c ^ {\prime} , j}}{| \mathcal {N} \left(v _ {c , i}\right) |} (37) \\ = \frac {\sum_ {\mathcal {N} _ {c} \left(v _ {c , i}\right)} \overline {{\mathbf {h}}} _ {c} + \sum_ {\mathcal {N} _ {c ^ {\prime}} \neq c \left(v _ {c , i}\right)} \overline {{\mathbf {h}}} _ {c ^ {\prime}}}{| \mathcal {N} \left(v _ {c , i}\right) |} = \sum_ {c ^ {\prime} = 1} ^ {C} s _ {c c ^ {\prime}} \overline {{\mathbf {h}}} _ {c ^ {\prime}}. (38) \\ \end{array}
$$
Therefore  $\mathbf{P2}$  holds:  $\mathbf{h}_{c,1}^{\mathcal{N}} = \dots = \mathbf{h}_{c,n}^{\mathcal{N}} = \overline{\mathbf{h}}_c^{\mathcal{N}}$ . Accordingly,  $\mathbf{C}$  is sufficient for having  $\mathbf{H}$  that obeys  $\mathbf{P1}$  and  $\mathbf{P2}$ , and thus minimizes the risk.