SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3More generally, for  $\Psi (\cdot ,\cdot)$  that outputs bounded attention scores for bounded inputs.
We can use the existing results on the ergodicity of infinite products of inhomogeneous stochastic matrices [14, 33] to show that any sequence of matrices in  $\mathcal{P}_{\mathcal{G},\epsilon}$  is ergodic.
Lemma 3. Fix  $\epsilon >0$ . Consider a sequence of matrices  $\{P^{(t)}\}_{t = 0}^{\infty}$  in  $\mathcal{P}_{\mathcal{G},\epsilon}$ . That is,  $P^{(t)}\in \mathcal{P}_{\mathcal{G},\epsilon}$  for all  $t\in \mathbb{N}_{\geq 0}$ . Then  $\{P^{(t)}\}_{t = 0}^{\infty}$  is ergodic.
The main proof strategy for Lemma 3 is to make use of the Hilbert projective metric and the Birkhoff contraction coefficient. These are standard mathematical tools to prove that an infinite product of inhomogeneous stochastic matrices is ergodic. We refer interested readers to the textbooks [14, 33] for a comprehensive study of these subjects.
Despite the nonlinearity of  $\sigma(\cdot)$ , the formulation (7) enables us to express the evolution of the feature vector trajectories as a weighted sum of products of matrices of the form  $DP$  where  $D \in \mathcal{D}$  and  $P \in \mathcal{P}_{\mathcal{G},\epsilon}$ . We define the set of such matrices as
$$
\mathcal {M} _ {\mathcal {G}, \epsilon} := \left\{D P: D \in \mathcal {D}, P \in \mathcal {P} _ {\mathcal {G}, \epsilon} \right\}.
$$
A key step in proving oversmoothing for attention-based GNNs under our assumptions is to show the ergodicity of the infinite products of matrices in  $\mathcal{M}_{\mathcal{G},\epsilon}$ . In what follows, we lay out the main ideas of the proof, and refer readers to Appendix F for the details.
Consider a sequence  $\{D^{(t)}P^{(t)}\}_{t = 0}^{\infty}$  in  $\mathcal{M}_{\mathcal{G},\epsilon}$ , that is,  $D^{(t)}P^{(t)}\in \mathcal{M}_{\mathcal{G},\epsilon}$  for all  $t\in \mathbb{N}_{\geq 0}$ . For  $t_0\leq t_1$  define
$$
Q _ {t _ {0}, t _ {1}} := D ^ {\left(t _ {1}\right)} P ^ {\left(t _ {1}\right)} \dots D ^ {\left(t _ {0}\right)} P ^ {\left(t _ {0}\right)}, \quad \delta_ {t} = \| D ^ {(t)} - I _ {N} \| _ {\infty},
$$
where  $I_N$  denotes the  $N \times N$  identity matrix. The common connectivity structure among  $P^{(t)}$ 's established in Section 4.2 allows us to show that long products of matrices  $DP$  from  $\mathcal{M}_{\mathcal{G},\epsilon}$  will eventually become a contraction in  $\infty$ -norm. More precisely, we can show that there exists  $T \in \mathbb{N}$  and  $0 < c < 1$  such that for all  $t \in \mathbb{N}_{\geq 0}$ ,
$$
\| Q _ {t, t + T} \| _ {\infty} \leq 1 - c \delta_ {t}.
$$
Next, define  $\beta_{k} := \prod_{t=0}^{k}(1 - c\delta_{t})$  and let  $\beta := \lim_{k \to \infty} \beta_{k}$ . Note that  $\beta$  is well-defined because the partial product is non-increasing and bounded from below. We can use the above contraction property to show the following key lemma.
Lemma 4. Let  $\beta_{k} := \prod_{t=0}^{k}(1 - c\delta_{t})$  and  $\beta := \lim_{k \to \infty} \beta_{k}$ .
1. If  $\beta = 0$ , then  $\lim_{k\to \infty}Q_{0,k} = 0$ ;
2. If  $\beta > 0$ , then  $\lim_{k \to \infty} BQ_{0,k} = 0$ .
The ergodicity of sequences of matrices in  $\mathcal{M}_{\mathcal{G},\epsilon}$  immediately follows from Lemma 4, which in turn implies oversmoothing as defined in (2).
Lemma 5. Any sequence  $\{D^{(t)}P^{(t)}\}_{t = 0}^{\infty}$  in  $\mathcal{M}_{\mathcal{G},\epsilon}$  is ergodic.
Remark The proof techniques developed in [5, 26] are restricted to symmetric matrices hence cannot be extended to more general family of GNNs, as they primarily rely on matrix norms for convergence analysis. Analyses solely using matrix norms are often too coarse to get meaningful results when it comes to asymmetric matrices. For instance, while the matrix 2-norm and matrix eigenvalues are directly related for symmetric matrices, the same does not generally hold for asymmetric matrices. Our analysis, on the other hand, exploits the inherently common connectivity structure among these matrices in deriving the ergodicity results in Lemma 3-5.
4.4 Joint spectral radius
Using the ergodicity results in the previous section, we can establish that oversmoothing happens in attention-based GNNs. To show that oversmoothing happens at an exponential rate, we introduce the notion of joint spectral radius, which is a generalization of the classical notion of spectral radius of a single matrix to a set of matrices [7, 30]. We refer interested readers to the textbook [17] for a comprehensive study of the subject.
Definition 4 (Joint Spectral Radius). For a collection of matrices  $\mathcal{A}$ , the joint spectral radius  $\mathrm{JSR}(\mathcal{A})$  is defined to be
$$
\operatorname {J S R} (\mathcal {A}) = \lim  _ {k \to \infty} \sup  _ {A _ {1}, A _ {2}, \dots , A _ {k} \in \mathcal {M}} \| A _ {1} A _ {2} \dots A _ {k} \| ^ {\frac {1}{k}},
$$
and it is independent of the norm used.
In plain words, the joint spectral radius measures the maximal asymptotic growth rate that can be obtained by forming long products of matrices taken from the set  $\mathcal{A}$ . To analyze the convergence rate of products of matrices in  $\mathcal{M}_{\mathcal{G},\epsilon}$  to a rank-one matrix with identical rows, we treat the two cases of linear and nonlinear activation functions, separately.
For the linear case, where  $\sigma(\cdot)$  is the identity map, we investigate the dynamics induced by  $P^{(t)}$ 's on the subspace orthogonal to  $\operatorname{span}\{\mathbf{1}\}$  and use the third property of the orthogonal projection  $B$  established in Section 4.3 to write  $BP_{1}P_{2}\ldots P_{k} = \tilde{P}_{1}\tilde{P}_{2}\ldots \tilde{P}_{k}B$ , where each  $\tilde{P}_i$  is the unique matrix in  $\mathbb{R}^{(N-1)\times (N-1)}$  that satisfies  $BP_{i} = \tilde{P}_{i}B$ . Let us define
$$
\tilde {\mathcal {P}} _ {\mathcal {G}, \epsilon} := \left\{\tilde {P}: B P = \tilde {P} B, P \in \mathcal {P} _ {\mathcal {G}, \epsilon} \right\}.
$$
We can use Lemma 3 to show that the joint spectral radius of  $\tilde{\mathcal{P}}_{\mathcal{G},\epsilon}$  is strictly less than 1.
Lemma 6. Let  $0 < \epsilon < 1$ . Under assumptions A1-A4,  $\mathrm{JSR}(\mathcal{P}_{\mathcal{G},\epsilon}) < 1$ .
For the nonlinear case, let  $0 < \delta < 1$  and define
$$
\mathcal {D} _ {\delta} := \left\{\operatorname {d i a g} (\mathbf {d}): \mathbf {d} \in \mathbb {R} ^ {N}, 0 \le_ {\mathrm {e w}} \mathbf {d} \le_ {\mathrm {e w}} \delta \right\}, \quad \mathcal {M} _ {\mathcal {G}, \epsilon , \delta} := \left\{\mathrm {D P}: \mathrm {D} \in \mathcal {D} _ {\delta}, \mathrm {P} \in \mathcal {P} _ {\mathcal {G}, \epsilon} \right\}.
$$
Then again, using the ergodicity result from the previous section, we establish that the joint spectral radius of  $\mathcal{M}_{\mathcal{G},\epsilon ,\delta}$  is also less than 1.
Lemma 7. Let  $0 < \epsilon, \delta < 1$ . Under assumptions A1-A4,  $\mathrm{JSR}(\mathcal{M}_{\mathcal{G},\epsilon,\delta}) < 1$ .
The above lemma is specifically useful in establishing exponential rate for oversmoothing when dealing with nonlinearities for which Assumption 4 holds in the strict sense, i.e.  $0 \leq \frac{\sigma(x)}{x} < 1$  (e.g., GELU and SiLU nonlinearities). Exponential convergence, however, can still be established under a weaker requirement, making it applicable to ReLU and Leaky ReLU, as we will see in Theorem 1.
It follows from the definition of the joint spectral radius that if  $\mathrm{JSR}(\mathcal{A}) < 1$ , for any  $\mathrm{JSR}(\mathcal{A}) < q < 1$ , there exists a  $C$  for which
$$
\left\| A _ {1} A _ {2} \dots A _ {k} y \right\| \leq C q ^ {k} \| y \| \tag {8}
$$
for all  $y\in \mathbb{R}^{N - 1}$  and  $A_{1},A_{2},\ldots ,A_{k}\in \mathcal{A}$
4.5 Main Theorem
Applying (8) to the recursive expansion of  $X_{i}^{(t + 1)}$  in (7) using the 2-norm, we can prove the exponential convergence of  $\mu(X^{(t)})$  to zero for the similarity measure  $\mu(\cdot)$  defined in (4), which in turn implies the convergence of node representations to a common representation at an exponential rate. This completes the proof of the main result of this paper, which states that oversmoothing defined in (2) is unavoidable for attention-based GNNs, and that an exponential convergence rate can be attained under general conditions.
Theorem 1. Under assumptions A1-A4,
$$
\lim  _ {t \rightarrow \infty} \mu (X ^ {(t)}) = 0,
$$
indicating oversmoothing happens asymptotically in attention-based GNNs with general nonlinearities. In addition, if
- (linear)  $\sigma(\cdot)$  is the identity map, or
- (nonlinear) there exists  $K \in \mathbb{N}$  and  $0 < \delta < 1$  for which the following holds: For all  $m \in \mathbb{N}_{\geq 0}$ , there is  $n_m \in \{0\} \cup [K - 1]$  such that for any  $c \in [d]$ ,  $\sigma(X_{r_c c}^{(mK + n_m)}) / X_{r_c c}^{(mK + n_m)} \leq \delta$  for some  $r_c \in [N]$ ,
then there exists  $q < 1$  and  $C_1(q) > 0$  such that
$$
\mu (X ^ {(t)}) \leq C _ {1} q ^ {t}, \forall t \geq 0.
$$
As a result, node representations  $X^{(t)}$  exponentially converge to the same value as the model depth  $t \to \infty$ .
Theorem 1 establishes that oversmoothing is asymptotically inevitable for attention-based GNNs with general nonlinearities. Despite similarity-based importance assigned to different nodes via the aggregation operator  $P^{(t)}$ , such attention-based mechanisms are yet unable to fundamentally change the connectivity structure of  $P^{(t)}$ , resulting in node representations converging to a common vector. Our results hence indirectly support the emergence of alternative ideas for changing the graph connectivity structure such as edge-dropping [15, 29] or graph-rewiring [21], in an effort to mitigate oversmoothing.
Remark 1. For nonlinearities such as SiLU or GELU, the condition  $(\star)$  is automatically satisfied under A3-A4. For ReLU and LeakyReLU, this is equivalent to requiring that there exists  $K \in \mathbb{N}$  such that for all  $m \in \mathbb{N}_{\geq 0}$ , there exists  $n_m \in \{0\} \cup [K - 1]$  where for any  $c \in [d]$ ,  $X_{r_c c}^{(mK + n_m)} < 0$  for some  $r_c \in [d]$ .
Remark 2. We note that our results are not specific to the choice of node similarity measure  $\mu(X) = \|X - \mathbf{1}_{\gamma_X}\|_F$  considered in our analysis. In fact, exponential convergence of any other Lipschitz node similarity measure  $\mu'$  to 0 is a direct corollary of Theorem 1. To see this, observe that for a node similarity measure  $\mu'$  with a Lipschitz constant  $L$ , it holds that
$$
\mu^ {\prime} (X) = \left| \mu^ {\prime} (X) - \mu^ {\prime} \left(\mathbf {1} _ {\gamma_ {X}}\right) \right| \leq L \| X - \mathbf {1} _ {\gamma_ {X}} \| _ {F} = L \mu (X).
$$
In particular, the Dirichlet energy is Lipschitz given that the input  $X$  has a compact domain, established in Lemma 1. Hence our theory directly implies the exponential convergence of Dirichlet energy.
4.6 Comparison with the GCN
Computing or approximating the joint spectral radius for a given set of matrices is known to be hard in general [36], yet it is straightforward to lower bound  $\mathrm{JSR}(\bar{\mathcal{P}}_{\mathcal{G},\epsilon})$  as stated in the next proposition.
Proposition 2. Let  $\lambda$  be the second largest eigenvalue of  $D_{\mathrm{deg}}^{-1/2} AD_{\mathrm{deg}}^{-1/2}$ . Then under assumptions A1-A4, it holds that  $\lambda \leq \mathrm{JSR}(\tilde{\mathcal{P}}_{\mathcal{G},\epsilon})$ .
In the linear case, the upper bound  $q$  on the convergence rate that we get for graph attention in Theorem 1 is lower bounded by  $\mathrm{JSR}(\tilde{\mathcal{P}}_{\mathcal{G},\epsilon})$ . A direct consequence of the above result is that  $q$  is at least as large as  $\lambda$ . On the other hand, previous work has already established that in the graph convolution case, the convergence rate of  $\mu (X^{(t)})$  is  $O(\lambda^t)$  [5, 26]. It is thus natural to expect attention-based GNNs to potentially have better expressive power at finite depth than GCNs, even though they both inevitably suffer from oversmoothing. This is also evident from the numerical experiments that we present in the next section.
In this section, we validate our theoretical findings via numerical experiments using the three commonly used homophilic benchmark datasets: Cora, CiteSeer, and PubMed [44] and the three commonly used heterophilic benchmark datasets: Cornell, Texas, and Wisconsin [28]. We note that our theoretical results are developed for generic graphs and thus hold for datasets exhibiting either homophily or heterophily and even those that are not necessarily either of the two. More details about the experiments are provided in Appendix K.
For each dataset, we trained a 128-layer single-head GAT and a 128-layer GCN with the random walk graph convolution  $D_{\mathrm{deg}}^{-1}A$ , each having 32 hidden dimensions and trained using the standard features and splits. The GCN with the random walk graph convolution is a special type of attention-based GNNs where the attention function is constant. For each GNN model, we considered various nonlinear activation functions: ReLU, LeakyReLU (with three different negative slope values: 0.01, 0.4 and 0.8) and GELU. Here, we chose GELU as an illustration of the generality of our assumption on nonlinearities, covering even non-monotone activation functions such as GELU. We ran each experiment 10 times. Figure 1 shows the evolution of  $\mu(X^{(t)})$  in log-log scale on the largest connected component of each graph as we forward pass the input  $X$  into a trained model. The solid curve is the average over 10 runs and the band indicates one standard deviation around the average.
Figure 1: Evolution of  $\mu(X^{(t)})$  (in log-log scale) on the largest connected component of each dataset (top 2 rows: homophilic graphs; bottom 2 rows: heterophilic graphs). Oversmoothing happens exponentially in both GCNs and GATs with the rates varying depending on the choice of activation function. Notably, GCNs demonstrate faster rates of oversmoothing compared to GATs.
number of layers
We observe that, as predicted by our theory, oversmoothing happens at an exponential rate for both GATs and GCNs, regardless of the choice of nonlinear activation functions in the GNN architectures. Notably, GCNs exhibit a significantly faster rate of oversmoothing compared to GATs. This aligns the observation made in Section 4.6, expecting a potentially better expressive power for GATs than GCNs at finite depth. Furthermore, the exponential convergence rate of oversmoothing varies among GNNs with different nonlinear activation functions. From a theory perspective, as different activation functions constitute different subsets of  $\mathcal{M}_{\mathcal{G},\epsilon}$  and different sets of matrices have different joint spectral radii, it is not surprising that the choice of nonlinear activation function would affect the convergence rate. In particular, among the nonlinearities we considered, ReLU in fact magnifies oversmoothing the second most. As a result, although ReLU is often the default choice for the standard implementation of many GNN architectures [10, 19], one might wish to consider switching to other nonlinearities to better mitigate oversmoothing.
Oversmoothing is one of the central challenges in developing more powerful GNNs. In this work, we reveal new insights on oversmoothing in attention-based GNNs by rigorously providing a negative answer to the open question of whether graph attention can implicitly prevent oversmoothing. By analyzing the graph attention mechanism within the context of nonlinear time-varying dynamical systems, we establish that attention-based GNNs lose expressive power exponentially as model depth increases.
We upper bound the convergence rate for oversmoothing under very general assumptions on the nonlinear activation functions. One may try to tighten the bounds by refining the analysis separately for each of the commonly used activation functions. Future research should also aim to improve the design of graph attention mechanisms based on our theoretical insights and utilize our analysis techniques to study other aspects of multi-layer graph attention.
Xinyi Wu would like to thank Jennifer Tang and William Wang for helpful discussions. The authors are grateful to Zhijian Zhuo and Yifei Wang for identifying an error in an earlier draft of the paper, thank the anonymous reviewers for providing valuable feedback, and acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing computing resources that have contributed to the research results reported within this paper.
This research has been supported in part by ARO MURI W911NF-19-0217, ONR N00014-20-1-2394, and the MIT-IBM Watson AI Lab.
[1] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NeurIPS, 2016.
[2] Vincent D. Blondel, Julien M. Hendrickx, Alexander Olshevsky, and John N. Tsitsiklis. Convergence in multiagent coordination, consensus, and flocking. Proceedings of the 44th IEEE Conference on Decision and Control, pages 2996-3000, 2005.
[3] Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In ICLR, 2022.
[4] Joan Bruna, Wojciech Zaremba, Arthur D. Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In ICLR, 2014.
[5] Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. In ICML Graph Representation Learning and Beyond (GRL+) Workshop, 2020.
[6] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020.
[7] Ingrid Daubechies and Jeffrey C. Lagarias. Sets of matrices all infinite products of which converge. Linear Algebra and its Applications, 161:227-263, 1992.
[8] Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS, 2016.
[9] David Kristjanson Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gomez-Bombarelli, Timothy D. Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In NeurIPS, 2015.
[10] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
[11] Kimon Fountoulakis, Amit Levi, Shenghao Yang, Aseem Baranwal, and Aukosh Jagannath. Graph attention retrospective. *ArXiv*, abs/2202.13060, 2022.
[12] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017.
[13] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In IJCNN, 2005.
[14] Darald J. Hartfiel. Nonhomogeneous Matrix Products. 2002.
[15] Arman Hasanzadeh, Ehsan Hajiramezanali, Shahin Boluki, Mingyuan Zhou, Nick G. Duffield, Krishna R. Narayanan, and Xiaoning Qian. Bayesian graph neural networks with adaptive connection sampling. In ICML, 2020.
[16] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Raphaël M. Jungers. The Joint Spectral Radius: Theory and Applications. 2009.
[40] Shiwen Wu, Wentao Zhang, Fei Sun, and Bin Cui. Graph neural networks in recommender systems: A survey. ACM Computing Surveys, 55:1 - 37, 2020.
[41] Xinyi Wu, Zhengdao Chen, William Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. In ICLR, 2023.
[42] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32:4-24, 2019.
[43] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In ICML, 2018.
[44] Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In ICML, 2016.
[45] Jiaxuan You, Bowen Liu, Rex Ying, Vijay S. Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In NeurIPS, 2018.
[46] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In ICLR, 2020.
[47] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In ICLR, 2020.
A Basic Facts about Matrix Norms
In this section, we list some basic facts about matrix norms that will be helpful in comprehending the subsequent proofs.
A.1 Matrix norms induced by vector norms
Suppose a vector norm  $\| \cdot \|_{\alpha}$  on  $\mathbb{R}^n$  and a vector norm  $\| \cdot \|_{\beta}$  on  $\mathbb{R}^m$  are given. Any matrix  $M \in \mathbb{R}^{m \times n}$  induces a linear operator from  $\mathbb{R}^n$  to  $\mathbb{R}^m$  with respect to the standard basis, and one defines the corresponding induced norm or operator norm by
$$
\| M \| _ {\alpha , \beta} = \sup  \left\{\frac {\| M v \| _ {\beta}}{\| v \| _ {\alpha}}, v \in \mathbb {R} ^ {n}, v \neq \mathbf {0} \right\}.
$$
If the  $p$ -norm for vectors  $(1 \leq p \leq \infty)$  is used for both spaces  $\mathbb{R}^n$  and  $\mathbb{R}^m$ , then the corresponding operator norm is
$$
\| M \| _ {p} = \sup  _ {v \neq \mathbf {0}} \frac {\| M v \| _ {p}}{\| v \| _ {p}}.
$$
The matrix 1-norm and  $\infty$ -norm can be computed by
$$
\| M \| _ {1} = \max  _ {1 \leq j} \sum_ {i = 1} ^ {m} | M _ {i j} |,
$$
that is, the maximum absolute column sum of the matrix  $M$ ;
$$
\| M \| _ {\infty} = \max  _ {1 \leq m} \sum_ {j = 1} ^ {n} | M _ {i j} |,
$$
that is, the maximum absolute row sum of the matrix  $M$ .
Remark In the special case of  $p = 2$ , the induced matrix norm  $\| \cdot \|_2$  is called the spectral norm, and is equal to the largest singular value of the matrix.
For square matrices, we note that the name "spectral norm" does not imply the quantity is directly related to the spectrum of a matrix, unless the matrix is symmetric.
Example We give the following example of a stochastic matrix  $P$ , whose spectral radius is 1, but its spectral norm is greater than 1.
$$
P = \left[ \begin{array}{c c} 0. 9 & 0. 1 \\ 0. 2 5 & 0. 7 5 \end{array} \right] \qquad \| P \| _ {2} \approx 1. 0 1 8 8
$$
A.2 Matrix  $(p,q)$ -norms
The Frobenius norm of a matrix  $M\in \mathbb{R}^{m\times n}$  is defined as
$$
\| M \| _ {F} = \sqrt {\sum_ {j = 1} ^ {n} \sum_ {i = 1} ^ {m} | M _ {i j} | ^ {2}},
$$
and it belongs to a family of entry-wise matrix norms: for  $1 \leq p, q \leq \infty$ , the matrix  $(p, q)$ -norm is defined as
$$
\| M \| _ {p, q} = \left(\sum_ {j = 1} ^ {n} \left(\sum_ {i = 1} ^ {m} | M _ {i j} | ^ {p}\right) ^ {q / p}\right) ^ {1 / q}.
$$
The special case  $p = q = 2$  is the Frobenius norm  $\| \cdot \|_F$ , and  $p = q = \infty$  yields the max norm  $\| \cdot \|_{\max}$ .
A.3 Equivalence of norms
For any two matrix norms  $\| \cdot \|_{\alpha}$  and  $\| \cdot \|_{\beta}$ , we have that for all matrices  $M \in \mathbb{R}^{m \times n}$ ,
$$
r \| M \| _ {\alpha} \leq \| M \| _ {\beta} \leq s \| M \| _ {\alpha}
$$
for some positive numbers  $r$  and  $s$ . In particular, the following inequality holds for the 2-norm  $\| \cdot \|_2$  and the  $\infty$ -norm  $\| \cdot \|_{\infty}$ :
$$
\frac {1}{\sqrt {n}} \| M \| _ {\infty} \leq \| M \| _ {2} \leq \sqrt {m} \| M \| _ {\infty}.
$$
It is straightforward to check that  $\| X - \mathbf{1}\gamma_{X}\|_{F}$  satisfies the two axioms of a node similarity measure:
1.  $\| X - \mathbf{1}\gamma_{X}\|_{F} = 0\Longleftrightarrow X = \mathbf{1}\gamma_{X}\Longleftrightarrow X_{i} = \gamma_{X}$  for all node  $i$
2. Let  $\gamma_{X} = \frac{\mathbf{1}^{\top}X}{N}$  and  $\gamma_{Y} = \frac{\mathbf{1}^{\top}Y}{N}$ , then  $\gamma_{X} + \gamma_{Y} = \frac{\mathbf{1}^{\top}(X + Y)}{N} = \gamma_{X + Y}$ . So
$$
\begin{array}{l} \mu (X + Y) = \| (X + Y) - \mathbf {1} (\gamma_ {X} + \gamma_ {Y}) \| _ {F} = \| X - \mathbf {1} \gamma_ {X} + Y - \mathbf {1} \gamma_ {Y} \| _ {F} \\ \leq \| X - \mathbf {1} \gamma_ {X} \| _ {F} + \| Y - \mathbf {1} \gamma_ {Y} \| _ {F} \\ = \mu (X) + \mu (Y). \\ \end{array}
$$