SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Deyu Bo $^{1*}$ , Yuan Fang $^{2}$ , Yang Liu $^{1}$ , Chuan Shi $^{1\dagger}$
<sup>1</sup>Beijing University of Posts and Telecommunications, China
$^{2}$ Singapore Management University, Singapore
{bodeyu, liuyangjanet, shichuan}@bupt.edu.cn, yfang@smu.edu.sg
Graph contrastive learning (GCL) aims to learn representations by capturing the agreements between different graph views. Traditional GCL methods generate views in the spatial domain, but it has been recently discovered that the spectral domain also plays a vital role in complementing spatial views. However, existing spectral-based graph views either ignore the eigenvectors that encode valuable positional information, or suffer from high complexity when trying to address the instability of spectral features. To tackle these challenges, we first design an informative, stable, and scalable spectral encoder, termed EigenMLP, to learn effective representations from the spectral features. Theoretically, EigenMLP is invariant to the rotation and reflection transformations on eigenvectors and robust against perturbations. Then, we propose a spatial-spectral contrastive framework  $(\mathrm{Sp}^2\mathrm{GCL})$  to capture the consistency between the spatial information encoded by graph neural networks and the spectral information learned by EigenMLP, thus effectively fusing these two graph views. Experiments on the node- and graph-level datasets show that our method not only learns effective graph representations but also achieves a 2–10x speedup over other spectral-based methods.
Graph neural networks (GNNs) have become the de facto framework to encode graph-structured data [14, 35, 39]. However, training high-quality GNNs usually requires a large number of domain-specific labels, which is not feasible in many real-world applications. Therefore, as a paradigm of self-supervised learning, graph contrastive learning (GCL) is proposed to learn node or graph representations without using labels [20, 19, 12, 38].
Typically, GCL methods first generate different views of a graph, and then contrast the positive views against the negative ones. By minimizing a contrastive loss, GCL methods can learn invariant information from different views for various downstream tasks. Therefore, how to generate ideal graph views is crucial to GCL. Most graph views are obtained by augmenting graphs in the spatial domain, such as dropping nodes and edges, heuristically [49, 44, 8] or adversarially [27, 40, 41]. Nevertheless, recent studies [18, 17] argue that spatial perturbations ignore the structural properties, and propose to perturb graph spectrum in the spectral domain.
Generally, the spatial and spectral views represent different information of the graph. The spatial domain captures the feature information and learns local graph representations by propagating the node features along local topology, i.e.,  $k$ -hop subgraphs. In contrast, the spectral domain covers the global structural information. The eigenvalues and eigenvectors encode the global shapes [13] and
*This work was done when the first author was a visiting student at Singapore Management University
node absolute positions [25]. Leveraging the agreements between the spatial and spectral views can significantly improve the expressive power and generalization ability of GNNs [37, 47]. However, several reasons hinder the study of spectral view: First, the spectral features are unstable. Previous work [24] shows that randomly flipping the signs or rotating the coordinates of eigenvectors also satisfies the eigenvalue decomposition, a.k.a., the sign and basis ambiguity issues, implying that spectral features are not unique and hard to transfer [15, 37]. Besides, perturbing spectral features is time-consuming. Existing spectral-based methods [18, 17] need to decompose and reconstruct the adjacency matrix, in which the complexity is cubic and quadratic in the number of nodes, respectively.
Therefore, to realize the spatial-spectral contrast, it is natural to ask: How to encode the spectral view of a graph effectively and efficiently? To answer this question, we first propose a spectral encoder, named EigenMLP, which not only inherits the scalability advantage of multilayer perceptrons (MLP), but also adheres to two key design principles. First, EigenMLP resolves the sign ambiguity issue by taking both positive and negative eigenvectors as input. Second, to address the basis ambiguity issue, the weights of EigenMLP are generated by a learnable mapping of the eigenvalues, making them equivariant to the coordinates of eigenvectors. To integrate the representations learned from the spatial and spectral views, we propose a Spatial-Spectral GCL framework  $(\mathrm{Sp}^2\mathrm{GCL})$  to maximize the agreements between views and learn effective representations for downstream tasks.
The contributions of our paper are as follows. (1) We propose EigenMLP, a novel spectral encoder to effectively and efficiently learn sign- and basis-invariant representations from spectral features. (2) We theoretically prove that EigenMLP is permutation-equivariant, rotation- and reflection-invariant, and can learn more stable representations against structural perturbations. (3) We propose  $\mathrm{Sp}^2\mathrm{GCL}$ , a spatial-spectral GCL framework that utilizes the cross-domain contrasts to effectively fuse the feature and structural information learned by spatial GNNs and EigenMLP. (4) Extensive experiments on both node-level and graph-level tasks demonstrate the effectiveness of the proposed framework  $\mathrm{Sp}^2\mathrm{GCL}$ , and verify the scalability and stability aspects of EigenMLP.
Graph Contrastive Learning. Most GCL methods aim to learn invariant information by maximizing the mutual information between different graph views [1, 36]. There are many ways to generate graph views and we broadly categorize them into the spatial and spectral approaches. In the spatial domain, graph views are usually generated by augmenting the original graphs. For example, GRACE [48], GCA [49] and CCA-SSG [44] propose to augment graphs by randomly dropping edges and nodes, AD-GCL [27] leverages adversarial training to filter unimportant edges, and JOAO [41] combines different augmentation strategies automatically. While in the spectral domain, the views are generated by perturbing graph spectrum. For example, MVGRL [8] heuristically uses a graph diffusion as augmentation, which acts as a low-pass filter, SpCo [18] proposes to preserve low-frequency components and perturbs high-frequency ones, and SPAN [17] generates augmentations by maximizing the spectral change. Besides, SFA [46] analyzes the spectrum of node representations, which is out of the scope of this work. In general, spectral views of graphs may have better performance and interpretability than spatial views but also suffer from high complexity.
Spectral Encoder. Perturbing the graph spectrum is time-consuming. Another approach is to encode the eigenvalues and eigenvectors instead of the eigenspaces. However, eigenvectors suffer from sign and basis ambiguity issues, and using these features directly will affect the stability of the model. Therefore, some spectral encoders are proposed to learn invariant representations from the non-unique spectral features. For example, SAN [15] uses a Transformer-based [34] encoder, which is invariant to the order of the bases. BasisNet [16] uses IGN [21] to learn permutation-invariant representations from the eigenspaces. PEG [37] leverages the distance between eigenvectors to reweigh the graph structure and avoids sign and basis ambiguity. However, the complexity of SAN and BasisNet is quadratic, which is difficult to scale to large graphs.
Assume that  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  is a graph, where  $\mathcal{V}$  is the node set with  $|\mathcal{V}| = N$  and  $\mathcal{E}$  is the edge set with  $|\mathcal{E}| = E$ . Let  $\mathbf{A} \in \{0,1\}^{N\times N}$  be the adjacency matrix, and  $\mathbf{X} \in \mathbb{R}^{N\times d}$  be the node feature matrix on  $\mathcal{G}$ . The normalized graph Laplacian  $\mathbf{L}$  of  $\mathcal{G}$  is defined as  $\mathbf{L} = \mathbf{I}_n - \mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}$ , where  $\mathbf{I}_n$  is the  $N\times N$  identity matrix and  $\mathbf{D}$  is the degree matrix with  $D_{ii} = \sum_{j}A_{ij}$  for  $i \in \mathcal{V}$  and  $D_{ij} = 0$  for  $i \neq j$ . The eigenvalue decomposition (EVD) of graph Laplacian is defined as  $\mathbf{L} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{\top}$ , where  $\boldsymbol{\Lambda}$
is a diagonal matrix whose diagonal entries  $0 \leq \lambda_1 \leq \dots \leq \lambda_N \leq 2$  are the eigenvalues of  $\mathbf{L}$ , and  $\mathbf{U} = [\mathbf{u}_1, \dots, \mathbf{u}_N]$  are the corresponding eigenvectors.
It is worth noting that in some cases, randomly flipping the signs and rotating the coordinates of eigenvectors may also satisfy EVD [24], which we refer to as sign and basis ambiguity.
Sign ambiguity. Given a pair of eigenvalue and eigenvector  $(\lambda_i, \mathbf{u}_i)$ , it satisfies  $\mathbf{L}\mathbf{u}_i = \lambda_i\mathbf{u}_i$ , and  $\lambda_i = \mathbf{u}_i^\top \mathbf{L}\mathbf{u}_i = \sum_{(v,v') \in \mathcal{E}} (u_{iv} - u_{iv'})^2$ . Therefore, if  $\mathbf{u}_i$  is an eigenvector of  $\mathbf{L}$ , then  $-\mathbf{u}_i$  also satisfies EVD, i.e.,  $\mathbf{u}_i^\top \mathbf{L}\mathbf{u}_i = (-\mathbf{u}_i)^\top \mathbf{L}(-\mathbf{u}_i)$ .
Basis ambiguity. If there is high multiplicity in the eigenvalues, i.e.,  $\lambda_{p+1} = \cdots = \lambda_{p+q}$  for some  $q > 1$ , then the corresponding eigenvectors  $[\mathbf{u}_{p+1}, \cdots, \mathbf{u}_{p+q}]$  lie in an orthogonal group  $\mathrm{O}(q) = \{\mathbf{Q} \in \mathbb{R}^{q \times q} | \mathbf{Q}^\top \mathbf{Q} = \mathbf{QQ}^\top = \mathbf{I}_q\}$ . Therefore, for any  $\mathbf{Q} \in \mathrm{O}(q)$ , replacing  $[\mathbf{u}_{p+1}, \cdots, \mathbf{u}_{p+q}]$  with  $[\mathbf{u}_{p+1}, \cdots, \mathbf{u}_{p+q}]\mathbf{Q}$  also satisfies EVD, i.e.,  $\mathbf{L}\mathbf{u}_i = \lambda_j\mathbf{u}_i$ ,  $p + 1 \leq i, j \leq p + q$ .
The above two facts state that the eigenvectors of graph Laplacian are not unique. Therefore, the model should consider how to learn sign- and basis-invariant representations from spectral features for better stability and generalization [37, 16, 15].
4 Proposed Framework:  $\mathbf{Sp}^2\mathbf{GCL}$
In this section, we present the proposed spatial-spectral GCL framework called  $\mathrm{Sp}^2\mathrm{GCL}$ . We first give a high-level overview on how to represent and contrast the spatial and spectral views of a graph. We then introduce the proposed invariant and equivariant spectral view encoder in detail. Finally, we briefly describe the preprocessing, training, and inference processes.
4.1 Overview of  $\mathbf{Sp}^2\mathbf{GCL}$
Views refer to different perspectives of the same data [30]. Unlike previous GCL methods that generate graph views in a single domain, we propose to model the spatial and spectral views separately and further utilize cross-domain contrasts to capture invariant information. Here we first describe how to represent the spatial and spectral views of graphs.
Spatial View represents the explicit connectivity of nodes, which can be denoted as  $\mathbb{V}_a = (\mathbf{A},\mathbf{X})$ . Through propagating node features along graph substructures, the spatial view can naturally fuse the topology and content information and learn local smooth representations of a graph.
Spectral View indicates the implicit relationships between nodes, which is expressed as  $\mathbb{V}_e = (\Lambda ,\mathbf{U})$  The eigenvalues and eigenvectors encode the geometric information and node positions of the graph topology, which can be seen as global structural information of a graph.
Since graphs are non-Euclidean data, it is difficult to directly contrast the spatial and spectral views. Therefore, we need to design suitable encoders to learn different view representations for GCL:
$$
\mathbf {H} _ {a} = f (\mathbf {A}, \mathbf {X}), \quad \mathbf {H} _ {e} = g (\boldsymbol {\Lambda}, \mathbf {U}), \tag {1}
$$
where  $f$  and  $g$  are the encoders of the spatial and spectral views, respectively, and  $\mathbf{H}_a,\mathbf{H}_e\in \mathbb{R}^{N\times d}$  are the spatial and spectral representation matrices, respectively. These representations can then be used in contrastive learning to learn invariant information across both domains.
The fundamental idea of contrastive learning is to define the positive and negative pairs, from which the model can capture the self-supervised signals. In our framework, we define the spatial and spectral representations of the same node or graph as positive pairs, and those of different nodes or graphs as negative pairs. For graph-level contrasts, we additionally use a readout function to learn the graph-level representations. Then, two projection heads  $\varphi_{a},\varphi_{e}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  are used to transform the representations into the contrastive space:
$$
\mathbf {Z} _ {a} = \varphi_ {a} (\mathbf {H} _ {a}), \quad \mathbf {Z} _ {e} = \varphi_ {e} (\mathbf {H} _ {e}). \tag {2}
$$
Subsequently, we employ InfoNCE [33], a classical contrastive objective function, to maximize the agreements between spatial and spectral representations:
$$
\mathcal {L} = - \frac {1}{2} \sum_ {i = 1} ^ {N} \left(\log \frac {e ^ {\langle \mathbf {z} _ {a} ^ {i} , \mathbf {z} _ {e} ^ {i} \rangle}}{e ^ {\langle \mathbf {z} _ {a} ^ {i} , \mathbf {z} _ {e} ^ {i} \rangle} + \sum_ {j \neq i} e ^ {\langle \mathbf {z} _ {a} ^ {j} , \mathbf {z} _ {e} ^ {i} \rangle}} + \log \frac {e ^ {\langle \mathbf {z} _ {e} ^ {i} , \mathbf {z} _ {a} ^ {i} \rangle}}{e ^ {\langle \mathbf {z} _ {e} ^ {i} , \mathbf {z} _ {a} ^ {i} \rangle} + \sum_ {j \neq i} e ^ {\langle \mathbf {z} _ {e} ^ {j} , \mathbf {z} _ {a} ^ {i} \rangle}}\right), \tag {3}
$$
Table 1: Comparison between different spectral encoders.
Methods	Information	Stability	Scalability
EigVal	EigVec	Sign	Basis	Inductive	Complexity
MLP [3, 4]	✓	✓	O(Nkd)
SAN [15]	✓	✓	✓	✓	O(Nk2d + Nkd)
BasisNet [16]	✓	✓	✓	O(Nk2d + Nkd)
PEG [37]	✓	✓	✓	✓	O(Ekd)
EigenMLP	✓	✓	✓	✓	✓	O(NTd)
Note: Here  $k$  is the number of eigenvectors,  $T$  is the period of polynomial, and  $d$  is the hidden dimension.
EigenMLP
MLP
Figure 1: Schematic illustration of EigenMLP and MLP.
where  $\langle \cdot ,\cdot \rangle$  represents the cosine similarity and  $i$  is the index of nodes or graphs.
Finally, to materialize our contrastive framework  $\mathrm{Sp}^2\mathrm{GCL}$ , we must choose or design the encoders  $f$  and  $g$  for the two views. For the spatial encoder  $f$ , we directly employ a standard message-passing neural network (MPNN), e.g., GraphSAGE [7] or GIN [39], which is widely adopted in previous GCL methods. For the spectral encoder  $g$ , which is the emphasis of this work, we propose EigenMLP in the next part (Section 4.2) based on several key properties. Note that, on the one hand, although MPNNs can learn useful representations, their expressive power is bounded by the 1-Weisfeiler-Lehman (WL) test [39]. On the other hand, the spectral view encodes the global information, which can help overcome the limitation of the 1-WL test [5]. Hence, from the perspective of expressive power, the spectral view is also complementary to the spatial view.
4.2 Proposed Spectral Encoder: EigenMLP
The spectral encoder is designed to learn stable representations from spectral features. A desirable spectral encoder should have three properties: (1) It can encode both the information of eigenvalues and eigenvectors, which represent different structural information [15]; (2) It is free of the sign and basis ambiguity issues in spectral features, toward learning stable representations [37]; (3) It is scalable to large graphs and has linear or sublinear time complexity. These three properties pose great challenges to the design of spectral encoders. Here we propose EigenMLP, which adopts an MLP-based architecture for scalability and addresses the sign and basis ambiguity problems for stability. Table 1 summarizes the differences between the proposed EigenMLP and existing spectral encoders. Specifically, EigenMLP satisfies all of the three properties above, as we elaborate below.
Scalability. In general, MLPs have demonstrated good scalability and have been widely used in learning graph representations [45, 11]. Since EigenMLP employs an MLP-based architecture, it inherits the low complexity and is linear to the number of nodes  $N$ .
Stability. MLPs are sensitive to input data, making the outputs vary with respect to the flipping of signs and rotation of coordinates of eigenvectors. This motivates us to increase the stability of MLP while maintaining high efficiency. We adopt two design principles that enable MLPs to learn sign- and basis-invariant representations. Figure 1 illustrates the difference between EigenMLP and MLP.
To address the sign ambiguity issue, we follow the design of SignNet [16], taking both positive and negative eigenvectors as inputs:
$$
\tilde {\mathbf {U}} = \left[ \psi \left(\phi \left(\mathbf {u} _ {i}\right) + \phi (- \mathbf {u} _ {i})\right) \right] _ {i = 1} ^ {N}, \tag {4}
$$
where  $\psi$  and  $\phi$  are neural networks,  $[\cdot ]$  is the concatenation operator, and  $\tilde{\mathbf{U}}$  is the sign-invariant eigenvectors. In practice, the sign-invariant neural networks may slow down model convergence. In this case, we can resort to some heuristics to determine the sign of the eigenvectors [47].
To solve the basis ambiguity issue, we first observe that each eigenvector has a corresponding eigenvalue, and when the coordinates of the eigenvectors are rotated, the positions of the eigenvalues are also displaced. Therefore, if we replace the weights of MLP with eigenvalues  $\lambda = [\lambda_1,\lambda_2,\dots ,\lambda_N]$ , the model will be invariant to the rotation of eigenvectors, i.e.,  $\mathbf{UQ}(\boldsymbol {\lambda}\mathbf{Q})^{\top} = \mathbf{UQQ}^{\top}\boldsymbol{\lambda}^{\top} = \mathbf{U}\boldsymbol{\lambda}^{\top}$ . However, directly replacing the learnable weights with fixed eigenvalues is trivial and will greatly
<sup>3</sup>Some details, such as the neural networks  $\phi$  and  $\psi$ , are omitted for brevity.
limit the expressive power of the model. Therefore, we first extend the scalar eigenvalues to their high-dimensional Fourier features [28]. The weights of EigenMLP are then decoded from the Fourier features using a learnable matrix:
$$
\rho (\lambda) = \left[ \sin (\lambda), \cos (\lambda), \sin (2 \lambda), \cos (2 \lambda), \dots \sin (T \lambda), \cos (T \lambda) \right] \cdot \mathbf {W} _ {\rho}, \tag {5}
$$
where  $T$  is the period and  $\mathbf{W}_{\rho} \in \mathbb{R}^{2T \times d}$  is a learnable matrix. Here  $\rho(\lambda)$  can be seen as a graph filter and the filtered eigenvalues are equivariant to the coordinates of eigenvectors [2], which can be used to learn powerful and basis-invariant spectral representations:
$$
g (\boldsymbol {\Lambda}, \mathbf {U}) = \tilde {\mathbf {U}} \cdot \rho (\boldsymbol {\lambda}), \tag {6}
$$
where  $\rho (\lambda) = [\rho (\lambda_1),\rho (\lambda_2),\dots ,\rho (\lambda_N)]^\top \in \mathbb{R}^{N\times d}$  represents all the filtered eigenvalues. The detailed matrix form of Equation (6) is provided in Appendix C. Note that the learnable matrix  $\mathbf{W}_{\rho}$  is shared between different eigenvalues. Therefore, the size of  $\mathbf{W}_{\rho}$  is independent of the number of eigenvalues but depends on the period  $T$ , which reduces the number of parameters in EigenMLP. More discussions can be found in Section 5.2.
Information. Notably, the invariant layer, i.e., Equation (6), in EigenMLP incorporates both eigenvalues and eigenvectors, which can capture both geometric and positional information.
4.3 Preprocessing, Training, and Inference
It is worth noting that running EVD for full eigenvectors has the complexity  $\mathcal{O}(N^3)$ , which is unacceptable for large graphs. Therefore, in the preprocessing, we use the eigenvectors with smallest- $k$  eigenvalues as a substitute and reduce the complexity to  $\mathcal{O}(N^2 k)$ . Besides, we can pre-calculate the rotation-invariant spectral features  $\mathbf{V} \cdot \rho(\lambda_k)$ . Therefore, in the training and inference, EigenMLP has the implementation of MLP. Note that the decomposition is only computed once in the training. Therefore, the overhead of the preprocessing should be amortized by each training epoch.
In training, traditional GCL methods need to compute the message-passing twice for different graph views, whose complexity is  $\mathcal{O}(E)$ . While in our framework, only the spatial view needs to calculate the message-passing, and the complexity of spectral view is  $\mathcal{O}(N)$ . Therefore, our framework is faster than others in the training processing. In inference, because we need to calculate the spatial and spectral representations separately, the inference speed is slightly lower than traditional methods. The overheads of training and inference are provided in Section 6.6.
5 Deeper Insights
In this section, we provide deeper insights into EigenMLP to understand its effectiveness. Specifically, we prove that EigenMLP is invariant, equivariant, and stable, and can generalize existing spectral augmentations. Besides, we also comment on the connections to previous work.
5.1 Theoretical Results
Theorem 1. EigenMLP is equivariant to permutation, and invariant to rotation, and reflection.
Proof. (Permutation) Assume that there are two matrices  $\mathbf{L}^{(1)},\mathbf{L}^{(2)}\in \mathbb{R}^{N\times N}$ , and  $\mathbf{L}^{(1)} = \mathbf{P}\mathbf{L}^{(2)}\mathbf{P}^{\top}$ . We have  $\mathbf{L}^{(1)} = \mathbf{P}\mathbf{L}^{(2)}\mathbf{P}^{\top} = (\mathbf{P}\mathbf{U}^{(2)})\boldsymbol {\Lambda}(\mathbf{P}\mathbf{U}^{(2)})^{\top}$  such that  $\mathbf{U}^{(1)}\rho (\pmb {\lambda}) = \mathbf{P}\mathbf{U}^{(2)}\rho (\pmb {\lambda})$
(Rotation) For any rotation matrix  $\mathbf{Q} \in \mathbf{O}(N)$ , the eigenvectors are rotated as  $\mathbf{U}\mathbf{Q}$ , and the corresponding Fourier matrix are permuted as  $\mathbf{Q}^\top \rho(\lambda)$ . Therefore, we have  $\mathbf{U}\mathbf{Q} \cdot \mathbf{Q}^\top \rho(\lambda) = \mathbf{U}\rho(\lambda)$ .
(Reflection) Following Proposition 1 in [16], a continuous function is sign-invariant iff it satisfies  $f(v) = h(v) + h(-v)$ . Hence, EigenMLP is invariant to reflection on the signs of eigenvectors.
Lemma 1. (Lemma 3.4 in [37]) For any positive semidefinite  $\mathbf{L}$  without multiple eigenvalues, set  $\mathbf{V}$  as the eigenvectors corresponding to the smallest  $k$  eigenvalues and sorted as  $0 < \lambda_1 < \dots < \lambda_k$ . For any sufficiently small  $\epsilon > 0$ , there exists a perturbation  $\Delta \mathbf{L}$ ,  $\| \Delta \mathbf{L} \|_F < \epsilon$  such that
$$
\min  _ {\mathbf {Q} \in \mathbf {O} (k)} | | (\mathbf {V} + \Delta \mathbf {V}) - \mathbf {V Q} | | _ {F} \geq 0. 9 9 \max  _ {1 \leq i \leq k} | \lambda_ {i + 1} - \lambda_ {i} | ^ {- 1} | | \Delta \mathbf {L} | | _ {F} + o (\epsilon). \tag {7}
$$
Theorem 2. For any sufficiently small  $\epsilon >0$ , there exists a perturbation  $\Delta \mathbf{L}$ $\| \Delta \mathbf{L}\| _F < \epsilon$  such that
$$
\forall_ {\mathbf {Q} \in \mathbf {O} (k)} \| (\mathbf {V} + \Delta \mathbf {V}) \rho (\boldsymbol {\lambda} _ {k}) - \mathbf {V} \rho (\boldsymbol {\lambda} _ {k}) \| _ {F} \leq 0. 9 9 T \max  _ {1 \leq i \leq k} | \lambda_ {i + 1} - \lambda_ {i} | ^ {- 1} \| \Delta \mathbf {L} \| _ {F} + o (\epsilon). \tag {8}
$$
Lemma 1 states that a small structural perturbation will produce changes unbounded from above in non-equivariant spectral features  $\mathbf{V}$  if there is a small spectral gap. Theorem 2 shows that there is a finite upper bound on the changes of equivariant spectral features  $\mathbf{V}\rho(\lambda_k)$ . Therefore, EigenMLP can learn more stable representations against structural perturbations. We present the proof of Theorem 2 in Appendix A. Experiments in Section 6.5 further give an empirical justification of Theorem 2.
Proposition 1. EigenMLP generalizes existing spectral augmentations.