SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[35] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
[36] Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR (Poster). OpenReview.net, 2019.
[37] Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. In *ICLR*. OpenReview.net, 2022.
[38] Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z. Li. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[39] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR. OpenReview.net, 2019.
[40] Longqi Yang, Liangliang Zhang, and Wenjing Yang. Graph adversarial self-supervised learning. In NeurIPS, pages 14887-14899, 2021.
[41] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 12121-12132. PMLR, 2021.
[42] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, 2020.
[43] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In ICLR. OpenReview.net, 2020.
[44] Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. From canonical correlation analysis to self-supervised graph neural networks. In NeurIPS, pages 76-89, 2021.
[45] Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In ICLR. OpenReview.net, 2022.
[46] Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In AAAI, 2023.
[47] Ziwei Zhang, Peng Cui, Jian Pei, Xin Wang, and Wenwu Zhu. Eigen-gnn: A graph structure preserving plug-in for gnuNs. IEEE Trans. Knowl. Data Eng., 35(3):2544-2555, 2023.
[48] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, abs/2006.04131, 2020.
[49] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pages 2069-2080. ACM / IW3C2, 2021.
We use  $i$  to indicate the index of  $\lambda^{-1} = \max_{i\in N}\left|\lambda_{i + 1} - \lambda_i\right|^{-1}$  and perturb graph Laplacian  $\mathbf{L}$  by perturbing the eigenvectors. Specifically, we set
$$
\mathbf {u} _ {i} ^ {\prime} = \sqrt {1 - \epsilon^ {2}} \mathbf {u} _ {i} + \epsilon \mathbf {u} _ {i + 1},
$$
$$
\mathbf {u} _ {i + 1} ^ {\prime} = - \epsilon \mathbf {u} _ {i} + \sqrt {1 - \epsilon^ {2}} \mathbf {u} _ {i + 1}. \tag {1}
$$
Note that  $||\mathbf{u}_i^{\prime}|| = ||\mathbf{u}_{i + 1}^{\prime}|| = 1$  and  $\mathbf{u}_i^\prime^\top \mathbf{u}_{i + 1}^\prime = 0$ . Therefore, replacing  $\mathbf{u}_i$  and  $\mathbf{u}_{i + 1}$  with  $\mathbf{u}_i^\prime$  and  $\mathbf{u}_{i + 1}^{\prime}$  still satisfies EVD. We denote  $\mathbf{V}' = [\mathbf{u}_1,\dots ,\mathbf{u}_i',\mathbf{u}_{i + 1}',\dots ,\mathbf{u}_k]$ . Then the perturbation can be represented as  $\Delta \mathbf{L} = \mathbf{V}'\boldsymbol {\Lambda}\mathbf{V}'^\top -\mathbf{V}\boldsymbol {\Lambda}\mathbf{V}^\top = \lambda_i(\mathbf{u}_i'\mathbf{u}_i'^\top -\mathbf{u}_i\mathbf{u}_i^\top) + \lambda_{i + 1}(\mathbf{u}_{i + 1}'\mathbf{u}_{i + 1}^\top -\mathbf{u}_{i + 1}\mathbf{u}_{i + 1}^\top)$ .
For sufficient small  $\epsilon > 0$ , we have:
$$
\begin{array}{l} \min  _ {\mathbf {Q} \in \mathbf {O} (k)} | | (\mathbf {V} + \Delta \mathbf {V}) - \mathbf {V Q} | | _ {F} \\ = \left\| \mathbf {u} _ {i} ^ {\prime} - \mathbf {u} _ {i}, \mathbf {u} _ {i + 1} ^ {\prime} - \mathbf {u} _ {i + 1} \right\| _ {F} \\ = \left\| \left(\sqrt {1 - \epsilon^ {2}} - 1\right) \mathbf {u} _ {i} + \epsilon \mathbf {u} _ {i + 1} \right\| _ {F} + \left\| \left(\sqrt {1 - \epsilon^ {2}} - 1\right) \mathbf {u} _ {i + 1} - \epsilon \mathbf {u} _ {i} \right\| _ {F} \tag {2} \\ = 4 \left(1 - \sqrt {1 - \epsilon^ {2}}\right) \\ = 2 \epsilon^ {2} + o (\epsilon^ {2}). \\ \end{array}
$$
For Fourier features with period  $\frac{T}{2}$ , we have:
$$
\begin{array}{l} \forall_ {\mathbf {Q} \in \mathbf {O} (k)} | | (\mathbf {V} + \Delta \mathbf {V}) \rho (\boldsymbol {\lambda} _ {k}) - \mathbf {V} \rho (\boldsymbol {\lambda} _ {k}) | | _ {F} \\ = \left\| \left[ \mathbf {u} _ {i} ^ {\prime} - \mathbf {u} _ {i}, \mathbf {u} _ {i + 1} ^ {\prime} - \mathbf {u} _ {i + 1} \right] \left[ \rho (\lambda_ {i}), \rho (\lambda_ {i + 1}) \right] ^ {\top} \right\| _ {F} \\ = \sum_ {t = 1} ^ {T / 2} | | \sin (\lambda_ {i}) \left(\mathbf {u} _ {i} ^ {\prime} - \mathbf {u} _ {i}\right) + \sin (\lambda_ {i + 1}) \left(\mathbf {u} _ {i + 1} ^ {\prime} - \mathbf {u} _ {i + 1}\right) | | _ {F} \\ + \sum_ {t = 1} ^ {T / 2} | | \cos (\lambda_ {i}) \left(\mathbf {u} _ {i} ^ {\prime} - \mathbf {u} _ {i}\right) + \cos (\lambda_ {i + 1}) \left(\mathbf {u} _ {i + 1} ^ {\prime} - \mathbf {u} _ {i + 1}\right) | | _ {F} \tag {3} \\ \leq T \left(\left| \left| \mathbf {u} _ {i} ^ {\prime} - \mathbf {u} _ {i} \right| \right| _ {F} + \left| \left| \mathbf {u} _ {i + 1} ^ {\prime} - \mathbf {u} _ {i + 1} \right| \right| _ {F}\right) \\ \leq T \left(2 \epsilon^ {2} + o \left(\epsilon^ {2}\right)\right), \\ \end{array}
$$
Next, we characterize  $||\Delta \mathbf{L}||_F$ :
$$
\begin{array}{l} \left| \left| \Delta \mathbf {L} \right| \right| _ {F} \\ = \left\| \lambda_ {i} \left(\mathbf {u} _ {i} ^ {\prime} \mathbf {u} _ {i} ^ {\prime} ^ {\top} - \mathbf {u} _ {i} \mathbf {u} _ {i} ^ {\top}\right) + \lambda_ {i + 1} \left(\mathbf {u} _ {i + 1} ^ {\prime} \mathbf {u} _ {i + 1} ^ {\prime} ^ {\top} - \mathbf {u} _ {i + 1} \mathbf {u} _ {i + 1} ^ {\top}\right) \right\| _ {F} \\ = \left\| \left(\lambda_ {i + 1} - \lambda_ {i}\right) \left[ - \epsilon^ {2} \left(\mathbf {u} _ {i} \mathbf {u} _ {i} ^ {\top} - \mathbf {u} _ {i + 1} \mathbf {u} _ {i + 1} ^ {\top}\right) + \epsilon \sqrt {1 - \epsilon^ {2}} \left(\mathbf {u} _ {i} \mathbf {u} _ {i + 1} ^ {\top} + \mathbf {u} _ {i + 1} \mathbf {u} _ {i} ^ {\top}\right) \right] \right\| _ {\mathrm {F}} ^ {2} \tag {4} \\ = \left(\lambda_ {k + 1} - \lambda_ {k}\right) ^ {2} \left(\epsilon^ {2} \left\| \mathbf {u} _ {k} \mathbf {u} _ {k + 1} ^ {\top} + \mathbf {u} _ {k + 1} \mathbf {u} _ {k} ^ {\top} \right\| _ {\mathrm {F}} ^ {2} + o \left(\epsilon^ {2}\right)\right) \\ = 2 \left(\lambda_ {k + 1} - \lambda_ {k}\right) ^ {2} \left(\epsilon^ {2} + o \left(\epsilon^ {2}\right)\right) \\ \end{array}
$$
Combining Equations (2) and (4), we have the lower bound of the changes of non-equivariant spectral features under small perturbations:
$$
\min  _ {\mathbf {Q} \in \mathbf {O} (k)} | | (\mathbf {V} + \Delta \mathbf {V}) - \mathbf {V Q} | | _ {F} \geq 0. 9 9 \max  _ {1 \leq i \leq k} | \lambda_ {i + 1} - \lambda_ {i} | ^ {- 1} | | \Delta \mathbf {L} | | _ {F} + o (\epsilon), \tag {5}
$$
which concludes the Lemma 1, i.e., Lemma 3.4 in [37].
Combining Equations (3) and (4), we have the upper bound of the changes of equivariant spectral features under small perturbations:
$$
\forall_ {\mathbf {Q} \in \mathbf {O} (k)} \| (\mathbf {V} + \Delta \mathbf {V}) \rho (\boldsymbol {\lambda} _ {k}) - \mathbf {V} \rho (\boldsymbol {\lambda} _ {k}) \| _ {F} \leq 0. 9 9 T \max  _ {1 \leq i \leq k} | \lambda_ {i + 1} - \lambda_ {i} | ^ {- 1} \| \Delta \mathbf {L} \| _ {F} + o (\epsilon), \tag {6}
$$
which concludes the Theorem 2.
B Detailed Experimental Setup
In this section, we report the details of our experiments. Specifically, we first introduce some general settings in all experiments. Then we introduce the detailed setup of each experiment one by one.
B.1 General Settings
Optimizer. For all experiments, we use the Adam optimizer.
Environment. The environment in which we run experiments is:
Linux version: 5.19.0-38-generic
- Operating system: Ubuntu 22.04.2
- CPU information: AMD EPYC 7313P 16-Core Processor
- GPU information: GeForce RTX 3090 (24 GB)
Resources. The addresses and licenses of all datasets are as follows:
- PubMed: https://github.com/tkipf/pygcn (MIT License)
- Wiki-CS: https://github.com/pmernyei/wiki-cs-dataset (MIT License)
- Facebook: https://github.com/benedekrozemberczki/MUSAE (GPL-3.0 license)
- arXiv: https://github.com/snap-stanford/ogb (MIT license)
- Flickr: https://github.com/GraphSAINT/GraphSAINT (MIT license)
- PPI: https://github.com/mims-harvard/ohmnet (MIT license)
- OGB-graph: https://github.com/snap-stanford/ogb (MIT license)
- ZINC-2M: https://github.com/snap-stanford/pretrain-gnns (MIT license)
Reproducibility. Our code is attached in the supplementary material.
B.2 Unsupervised Node Classification
Evaluation protocol. In the unsupervised node classification task, all methods are first trained with the corresponding self-supervised learning objectives. Then the learned representations are evaluated with a Logistic classifier with  $l_{2}$  normalization. We evaluate the method every 10 epochs and the maximum epoch is set to 1000. For the mini-batch training, we set the batch size to 1024. The detailed statistics are shown in Table 1 and the hyperparameters are shown in Table 2.
Table 1: Statistics of unsupervised node classification datasets.
Graphs	Nodes	Edges	Features	Classes
PubMed	1	19,717	88,648	500	3
Wiki-CS	1	11,701	216,123	300	10
Facebook	1	22,470	342,004	128	4
arXiv	1	169,343	1,116,243	128	40
Flickr	1	89,250	899,756	500	7
PPI	24	56,928	1,226,368	50	121
B.3 Unsupervised Graph Prediction
Evaluation protocol. In the unsupervised graph prediction task, we use the stand encoder, provided by OGB  $^{4}$ , as the spatial encoder of  $\mathrm{Sp}^2\mathrm{GCL}$ , which is a 5-layer GIN with hidden dimension  $d = 300$ . We use add pooling to learn graph-level representations and set the batch size to 32. For the spectral
$^{4}$ https://github.com/snap-stanford/ogb/blob/master/ogb/graphproppred/molEncoder.py
Table 2: Statistics of unsupervised node classification datasets.
# Eigenvectors (k)	Period (T)	lr	wd	Dropout
PubMed	30	20	1e-3	0	0
Wiki-CS	100	20	1e-3	0	0
Facebook	100	20	1e-3	0	0
arXiv	200	20	1e-3	0	0
Flickr	100	20	1e-3	0	0
PPI	50	20	1e-3	0	0
encoder, due to the relatively small sizes of the molecular graphs, we use all eigenvectors as the spectral features. We set the learning rate to 0.001 and the period to 10 for all datasets, and the number of training epochs is chosen among  $\{20, 50, 80, 100, 150\}$  using the validation set, as suggested by AD-GCL [27]. For the downstream evaluator, we use a Riger regressor for the regression tasks and a Logistic classifier for the binary classification tasks. The strength of  $l_{2}$  normalization is grid searched among  $\{0.001, 0.01, 0.1, 1, 10, 100, 1000\}$  on the validation set for each dataset. The detailed statistics of the datasets are shown in Table 3.
Table 3: Statistics of unsupervised graph prediction datasets.
Graphs	Avg. Nodes	Avg. Edges	Classes	Task	Metric
ogbg-molesol	1,128	13.3	13.7	1	Regression	RMSE
ogbg-mollipo	4,200	27.0	29.5	1	Regression	RMSE
ogbg-molfreesolv	642	8.7	8.4	1	Regression	RMSE
ogbg-molbace	1,513	34.1	36.9	1	Binary Class.	ROC-AUC
ogbg-molbbbpl	2,039	24.1	26.0	1	Binary Class.	ROC-AUC
ogbg-molclintosh	1,477	26.2	27.9	2	Binary Class.	ROC-AUC
ogbg-moltox21	7,831	18.6	19.3	12	Binary Class.	ROC-AUC
ogbg-molsider	1,427	33.6	35.4	27	Binary Class.	ROC-AUC
B.4 Transfer Learning
Evaluation protocol. For the transfer learning task, we use the same GIN encoder as [10]. In the pre-training stage, the learning rate is set to 0.001 and the number of training epochs is chosen from  $\{20, 50, 80, 100\}$  based on the validation set. Similarly, we use all eigenvalues and eigenvectors as the spectral features, and the period is set to 10. In the fine-tuning stage, we remove the self-supervised learning objective, and an additional linear projection layer is used on the output of the encoder for classification. The hyperparameters are the same as in the pre-training stage. The detailed statistics of the datasets are shown in Table 4.
Table 4: Statistics of transfer learning datasets.
Graphs	Utilization	Avg. Nodes	Avg. Edges
ZINC-2M	Pre-Training	2,000,000	26.62	57.72
BBBP	Finetuning	2,039	24.06	51.90
Tox21	Finetuning	7,831	18.57	38.58
SIDER	Finetuning	1,427	33.64	70.71
ClinTox	Finetuning	1,477	26.15	55.76
BACE	Finetuning	1,513	34.08	73.71
HIV	Finetuning	41,127	25.51	54.93
MUV	Finetuning	93,087	24.23	52.55
ToxCast	Finetuning	8,576	18.78	38.52
B.5 Stability Experiment
We use the PyTorch-style pseudo code to explain how we generate the synthetic perturbations (Figure 2) and practical perturbations (Figure 3).
e, u = torch.linalg.eigh(L) # EVD  
random_sign = 2*torch.randint(0, 2, (N,))-1  
signFlip = torch.diag(random_sign).float()  
coorFlip = torch.randint(N)  
u_sign = torch.mm(u, signFlip)  
u BASIS = u.clone()[ :, coorFlip]
Figure 2: Synthetic perturbations
e3，u3  $=$  scipysparse.linalg.eigsh( L,k=100,which  $\equiv$  'SM'tol  $\coloneqq$  1e-3)   
e4，u4  $=$  scipysparse.linalg.eigsh( L,k=100,which  $\equiv$  'SM'tol  $\equiv$  1e-4)   
e5，u5  $=$  scipysparse.linalg.eigsh( L,k=100,which  $\equiv$  'SM'tol  $\equiv$  1e-5)
Figure 3: Practical perturbations
C Matrix Form of EigenMLP
We give a detailed matrix form of Equation 6, from which we can see that the Fourier features of eigenvalues give different weights to the eigenvectors, thus making the model invariant to the rotation of coordinates and preserving good fitting ability.
$\begin{array}{r}\underbrace{\left[ \begin{array}{cccc}u_1^1 & u_2^1 & \cdots & u_k^1\\ u_1^2 & u_2^2 & \cdots & u_k^2\\ \vdots & \vdots & \ddots & \vdots \\ u_1^N & u_2^N & \cdots & u_k^N \end{array} \right]}_{\mathrm{Eigenvectors},N\times k}\\ \times \underbrace{\left[ \begin{array}{ccccccc} \cos(\lambda_1) & \sin(\lambda_1) & \cdots & \cos(T\lambda_1) & \sin(T\lambda_1)\\ \cos(\lambda_2) & \sin(\lambda_2) & \cdots & \cos(T\lambda_2) & \sin(T\lambda_2)\\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \cos(\lambda_k) & \sin(\lambda_k) & \cdots & \cos(T\lambda_k) & \sin(T\lambda_k) \end{array} \right]}_{\mathrm{Fourier~features~of~eigenvalues,}k\times 2T}\end{array}$  (7) Parameters of learnable matrix,  $2t\times d$
D Pseudo Algorithm
In order to better demonstrate our algorithm, here we provide the pseudo algorithms of EigenMLP (Figure 4) and  $\mathrm{Sp}^2\mathrm{GCL}$  (Figure 5).
Figure 4: Pseudo Algorithm of EigenMLP
class EigenMLP(nnModule)   
def__init__(self,k,d,T): #k: number of Eigenvectors #d: hidden dimension #T: period self.phi  $\equiv$  nnSEQUENTIAL(nn.Linear(1,d),nn.ReLU(),nn.Linear(d,d)) self.psi  $\equiv$  nnSEQUENTIAL(nn.Linear(d,d),nn.ReLU(),nn.Linear(d,1)) self.mlp  $\equiv$  nnSEQUENTIAL(nn.Linear(2\*T,d),nn.ReLU(),nn.Linear(d,d))   
defforward(e,u): u  $=$  u unsqueeze(-1) u  $=$  self.psi(self.phi(u)+ self.phi(-u)).squeeze(-1) #[N,k] T_term  $\equiv$  torch.arange(0,T).float() T_e  $\equiv$  e.unsqueeze(1)\*T F_e  $\equiv$  torch.cat([torch.sin(T_e)，torch.cos(T_e)],dim=-1)#[k,2T] return self.mlp(torch.mm(u,F_e))
Figure 5: Pseudo Algorithm of  $\mathbf{Sp}^2\mathbf{GCL}$
def Sp2GCL(g, x, e, u):  
    # g: graph structures  
    # x: node features  
    # e: eigenvalues  
    # u: eigenvectors  
    x_a = GNN(g, x)  
    x_e = EigenMLP(e, u)  
    # For graph-level tasks  
    # x_a = add_pool(g, x_a)  
    # x_e = add_pool(g, x_e)  
    h_a = spa_projection_head(x_a)  
    h_e = speprojection_head(x_e)  
    h_a = F.normalize(h_a, dim=-1, p=2)  
    h_e = F.normalize(h_e, dim=-1, p=2)  
    logits = torch.mm(h_a, h_e.t())  
    labels = torch.arange(h_a.size(0), dtype=torch.long)  
    return 0.5 * F CROSS_entropy(logits, labels) + 0.5 * F CROSS_entropy(logits.t(), labels)