Title: Uncovering the Redundancy in Graph Self-supervised Learning Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph self-supervised learning

Model redundancy

Pre-training and fine-tuning paradigm
Abstract: Graph self-supervised learning, as a powerful pre-training paradigm for Graph Neural Networks (GNNs) without labels, has received considerable attention. We have witnessed the success of graph self-supervised learning on pre-training the parameters of GNNs, leading many not to doubt that whether the learned GNNs parameters are all useful. In this paper, by presenting the experimental evidence and analysis, we surprisingly discover that the graph self-supervised learning models are highly redundant at both of neuron and layer levels, e.g., even randomly removing 51.6\% of parameters, the performance of graph self-supervised learning models still retains at least 96.2\%. This discovery implies that the parameters of graph self-supervised models can be largely reduced, making simultaneously fine-tuning both graph self-supervised learning models and prediction layers more feasible. Therefore, we further design a novel graph pre-training and fine-tuning paradigm called SLImming DE-correlation Fine-tuning (SLIDE). The effectiveness of SLIDE is verified through extensive experiments on various benchmarks, and the performance can be even improved with fewer parameters of models in most cases. For example, in comparison with full fine-tuning GraphMAE on Amazon-Computers dataset, even randomly reducing 40\% of parameters, we can still achieve the improvement of 0.24\% and 0.27\% for Micro-F1 and Macro-F1 scores respectively.

Uncovering the Redundancy in Graph Self-supervised Learning Models

Abstract

Graph self-supervised learning, as a powerful pre-training paradigm for Graph Neural Networks (GNNs) without labels, has received considerable attention. We have witnessed the success of graph self-supervised learning on pre-training the parameters of GNNs, leading many not to doubt that whether the learned GNNs parameters are all useful. In this paper, by presenting the experimental evidence and analysis, we surprisingly discover that the graph self-supervised learning models are highly redundant at both of neuron and layer levels, e.g., even randomly removing 51 . 6% of parameters, the performance of graph self-supervised learning models still retains at least 96 . 2% . This discovery implies that the parameters of graph self-supervised models can be largely reduced, making simultaneously fine-tuning both graph self-supervised learning models and prediction layers more feasible. Therefore, we further design a novel graph pre-training and fine-tuning paradigm called SLImming DE-correlation Fine-tuning (SLIDE 2 ). The effectiveness of SLIDE is verified through extensive experiments on various benchmarks, and the performance can be even improved with fewer parameters of models in most cases. For example, in comparison with full fine-tuning GraphMAE on Amazon-Computers dataset, even randomly reducing 40% of parameters, we can still achieve the improvement of 0 . 24% and 0 . 27% for Micro-F1 and Macro-F1 scores respectively.

1 Introduction

Graph self-supervised learning, aiming at learning the parameters of Graph Neural Networks (GNNs) without labels, has been a popular graph pre-training paradigm [1-3]. Usually, graph self-supervised learning is naturally divided into two learning methods, i.e., graph contrastive learning [4, 5] and graph generative self-supervised learning [6, 7]. After pre-training the parameters of self-supervised GNNs, graph self-supervised learning achieves state-of-the-art performance on a variety of tasks by fine-tuning an additional prediction layer [8, 7].

Various researches have attempted to improve and understand graph self-supervised learning from different perspectives. For example, the graph augmentation techniques [9, 10], graph spectrum feature of graph self-supervised learning [11, 12], and others [7, 13]. Despite their remarkable achievements, little efforts have been made to understand the behavior of the learned model parameters by graph self-supervised learning. Here, we ask: Are the model parameters all always useful? Or

what part of model parameters is useful? Whether there is the model redundancy in self-supervised GNNs? Understanding the model property in graph self-supervised learning, particularly the model redundancy, can provide valuable guidelines and insights for the development of advanced graph self-supervised learning.

For this purpose, we conduct experiments and surprisingly find out that the graph self-supervised learning models are actually highly redundant. We start with the experiments (Section 2) to investigate the downstream performance of different graph self-supervised learning models with randomly removed parameters at both neuron and layer levels. Our findings indicate that graph self-supervised learning models with even half of the neurons randomly removed have virtually no impact on the performance of the node classification task. Moreover, by closely examining the learned representations, we find out that the representations in each layer with a substantial number of neurons removed are quite similar to the representations obtained by the full set of neurons. Meanwhile, there is also a strong similarity between the representations of each layer and its adjacent layer. All results demonstrate that graph self-supervised models exhibit high model redundancy at both neuron level and layer levels.

The above findings hold great potential to improve current graph self-supervised learning models. On the one hand, it may provide valuable guideline for the pruning or the sparsity of GNNs [14, 15]. On the other hand, the phenomenon of model redundancy provides a new opportunity for designing a full fine-tuning graph self-supervised learning model. Previously, considering that the number of parameters to be fine-tuned is excessive when we directly fine-tune both GNNs and prediction layers, we usually only have to fine-tune the attached prediction layer, i.e., linear probing [3, 5, 7]. Here, if the parameters can be greatly reduced, we can simultaneously fine-tune both graph selfsupervised learning models and prediction layers. Besides, the findings imply that de-correlating the learned representations is also necessary. Therefore, we propose a novel pre-training and finetuning paradigm called SLIDE by obtaining SLI mGNNsfrom the self-supervised GNNs and using the DE -correlation strategy to reduce the correlation between features during the fine-tuning phase. Extensive experiments on various benchmark datasets validate the effectiveness of SLIDE.

In summary, our contributions are three-fold:

To the best of our knowledge, we are the first to uncover that graph self-supervised models exhibit high model redundancy at both neuron and layer levels. The model redundancy allows for the removal of a majority of model parameters with almost no impact on the performance of the downstream task. This discovery can significantly enhance model efficiency while maintaining acceptable task performance.

This discovery provides two key guidelines for the subsequent graph pre-training and finetuning framework: one is that the parameters can be reduced, and the other is the representations should be de-correlated. These motivate us to propose a novel method, SLIDE, to achieve a pre-training and fine-tuning paradigm with fewer parameters and better performance on the downstream task.

Comprehensive experiments demonstrate that our SLIDE outperforms baselines across multiple benchmark datasets. For example, using the Amazon-Computers dataset [16] with GRACE [1], compared to full fine-tuning, we achieve improvements of 0 . 37% and 0 . 16% for Macro-F1 and Micro-F1 scores respectively with a random reduction of 30% of parameters.

2 The Model Redundancy in Graph Self-supervised Learning

In this section, we investigate the model redundancy in two representative graph self-supervised learning models (GraphMAE [3] and GRACE [1]) on node classification tasks. Specifically, we pretrain these graph self-supervised models on Cora, Citeseer, Pubmed [17], Amazon-Photo, AmazonComputers [16], Ogbn-arxiv [18], and then we evaluate the performance again after removing neurons in various ways. If the performance gap between the original GNNs and the slim GNNs with fewer neurons is small, it indicates great model redundancy in self-supervised GNNs. In other words, even with reduced neurons, the performance remains largely unaffected, suggesting that many neurons in self-supervised GNNs are dispensable. Notably, model redundancy is not limited to node classification tasks. We conduct experiments in Appendix A demonstrating that model redundancy also persists in link prediction and graph classification tasks.

half GNNs

original GNNs

2-half GNNs

inputneurons

hidden neurons

outputneurons

drop neurons

Neuron removal We explore two different ways to reduce the number of neurons in selfsupervised GNNs, i.e., the original GNNs, which are illustrated in Figure 1. Specifically, we consider two approaches: (1) neuron level: randomly retaining the number of neurons by 50% and 25% in each layer named "half" and "quarter", and (2) layer level: retaining only the first two layers if the layer number is greater than two and randomly retaining the number of neurons by 100%, 50% and 25% in the second layer named "2-original", "2-half" and "2-quarter". In this way, we can obtain five types of slim GNNs: "half GNNs", "quarter GNNs", "2-original GNNs", "2-half GNNs" and "2-quarter GNNs".

↓

↓

↓

↓

Experimental results Wefirst pre-train the original GNNs through GraphMAE and GRACE, then we can obtain five kinds of slim GNNs by using the neuron removal methods mentioned above. For

both the slim GNNs and the original GNNs, we individually attach an additional prediction layer which is trained while keeping the original GNNs frozen on the node classification task to evaluate the Micro-F1 (F1-Mi) and Macro-F1 (F1-Ma) scores, as shown in Table 1 and Table 2, where "-" means that there are only two layers in the original GNNs (i.e.,"2-Original" and "Original" have the same performance) and "Change-Param" means the percentage change of the number of parameters compared to "Original". Surprisingly, we have following observations:

In all cases, "half GNNs" retain at least 96.2% of the performance of the original GNNs while the numbers of parameters are reduced by at least 51.6%. In most of the cases, "quarter GNNs" still retain at least 90% of the performance of the original GNNs while the number of parameters are reduced by at least 76.2%. As can be seen, "quarter GNNs" of GRACE on Computers retains 95.5% of the Micro-F1 score and 94.7% of the Macro-F1 score, while the number of parameters is reduced by 82.5%.

The removal of layers after the second layer has a negligible impact on the performance. The performance of "2-original GNNs" with GraphMAE on Ogbn-arxiv dataset demonstrates that even when layers after the second layer are removed, both the Micro-F1 score and the Macro-F1 score retain 99.4% of the performance, while the number of parameters is reduced by 47.0% .

The observations indicate that the graph self-supervised learning models are highly redundant, both at the neuron level and at the layer level. More details about the hyperparameters and the number of parameters can be found in Appendix B.

(b) Citeseer

(c) Pubmed

(d) Photo

(e) Computers

(f) Ogbn-arxiv

1.0

1.0

1.0

0.8

0.8

0.8

K

A

0.6

0.4

GraphMAE_all_I1

GraphMAE_all_12

CKA

0.6

0.4

GraphMAE_all_I1

GraphMAE_all_12

CKA

0.6

0.4

GraphMAE_all_12

GraphMAE_all_I1

0.2

0.2

GRACE_all_I1

0.2

GRACE_all_11

0.0

0.0

0.0

0%

25%

50%

75%

100%

0%

25%

50%

75%

100%

0%

25%

50%

75%

100%

neuronremovalrate

neuronremovalrate

neuronremovalrate

1.0

1.0

1.0

0.8

0.8

0.8

CKA

0.6

CKA

0.6

CKA

0.6

0.4

GraphMAE_all_12

GraphMAE_all_I1

0.4

GraphMAE_all_|1

GraphMAE_all_12

0.4

GraphMAE_all_I1

0.2

0.2

GRACE_all_I1

0.2

GraphMAE_all_12

GraphMAE_all_I3

0.0

0.0

0.0

0%

25%

50%

75%

100%

0%

25%

50%

75%

100%

0%

25%

50%

75%

100%

neuronremovalrate

neuronremovalrate

neuronremovalrate

Redundancy analysis on neuron level Here, we further analyze the model redundancy by closely examining the learned representations in order to explore why the slim GNNs achieve similar performance to the original GNNs. Specifically, we get the slim GNNs by dropping the neurons of each layer in 25%, 50% and 75% to analyze the model redundancy at the neuron level. Given the learned representations of the i -th layer of the slim GNNs and the original GNNs, Centered Kernel Alignment (CKA) is adopted to calculate their similarity [19]. A high CKA score implies a strong similarity. As shown in Figure 2, compared to the original GNNs, the CKA scores are over 85% when 50% neurons are dropped. Even if the neurons are removed by 75% , the CKA scores are still almost over 80% . This indicates that removing a significant number of neurons from each layer has a minimal impact on the significance of representations of each layer and task performance consequently.

Redundancy analysis on layer level

As for the model redundancy at the layer level, given the learned representations of the i -th layer and the ( i +1) -th layer of the original GNNs, we also adopt CKA to calculate their similarity, where the 0 -th layer represents the original features of the nodes from the datasets. As shown in Figure 3, we report the CKA scores between the representations of each layer and its adjacent layer of the original GNNs for GraphMAE and GRACE."data-layer1" means that we calculate CKA scores between the original features of the nodes from the datasets and the representations of

(a) GraphMAE

(b) GRACE

1.0

1.0

0.8

0.8

0.6

0.6

Citeseer

Ogbn-arxiv

0.4

0.4

0.2

Citeseer

0.2

Computer

Pubmed

Computers

Pubmed

Phioto

data-layer1

Photo

data-layer1

the first layer, and so on. As can be seen, CKA scores between the original features and the representations of the first layer are relatively low, while CKA scores between the representations of the layers after the first layer and their next layer are much closer to 1 in most of the cases, indicating the model redundancy at the layer level.

3 Our Proposed Tuning Approach: Slimming De-correlation Fine-tuning

RFF Extract

LossReweight

Model Slimming

Pre-trained GNN

Slim GNN

Prediction

Losspreliminary

Node

Embeddings

Slim GNN

Update

Classifer

…

…

…

RFF Maps

…

…

Loss

Wreights

Reweight Loss

Weights

Minimize Correlation

…

Label

…

Value

>Dim.

p

Ep²p

0

Ot

03

>Freq.

In

u2u3

Probability

c1c2c3

Class

Label

→Class

12c3

In general, for the self-supervised pre-trained GNNs, we attach an additional prediction layer which is fine-tuned while keeping the GNNs frozen to conduct the downstream tasks. Ideally, if we are able to tune the parameters of both the GNNs and the linear layer, it is possible to achieve the best performance. However, the number of parameters to be fine-tuned is excessive. Here, since we identify the model redundancy in the self-supervised pre-trained GNNs, this motivates us that we actually only need to tune the additional classifier and the slim GNNs, so as to obtain a better trade-off between the model performance and the number of fine-tunable parameters. Therefore, we propose a novel pre-training and fine-tuning paradigm called SLI mming DE -correlation Fine-tuning (SLIDE), as shown in Figure 4. Specifically, firstly it reduces model redundancy in self-supervised pre-trained GNNs by randomly removing redundant neurons to obtain the slim GNNs. Then, we input the graph data into the slim GNNs to obtain the embeddings of the nodes, combined with an additional prediction layer, we can predict the label of each node. Meanwhile, we design another model de-correlation module based the squared Frobenius norm [20, 21] (an analogue corresponding to the Hilbert-Schmidt Independence Criterion, i.e., HSIC [22] in Euclidean space). The module learns the de-correlation weights for the classification loss, so as to reduce the redundancy among embeddings and make the embeddings more informative.

Specifically, let G = ( A,V,X ) denote the input graph, where V is the node set, N = | V | is the number of nodes, N tr is the number of nodes in the training set and d is the dimension of the node features, A ∈ { 0 , 1 } N × N is the adjacency matrix, and X ∈ R N × d is the input node feature matrix. For model slimming, we first pre-train the original GNNs through the existing pretraining frameworks, e.g., the graph contrastive learning [1, 4, 5] or graph generative self-supervised learning [3, 7, 2]. Then the slim GNNs f S can be obtained by randomly reducing both the neurons and the layers. Furthermore, given G , we can get the node embeddings H ∈ R N × d h as H = f S ( A,X ) , where d h is the dimension of the node embeddings.

Motivated by Section 2 that the neurons, as well as the learned embeddings, in self-supervised GNNs are highly redundant, we aim to de-correlate the learned embeddings H in the fine-tuning phase, making models with fewer parameters more informative. In particular, in inspiration of the de-correlation methods [21, 23], given the i-th dimension and j-th dimension of the node embeddings H ∗ ,i and H ∗ ,j , we obtain Random Fourier Features (RFF) [24, 25] as

where N RFF is the number of functions in the random fourier space, u q and v q denote the functions from the space of Random Fourier Features. Then, we elaborate on reweighting of the weights, which encourages the independence of the node embeddings. Define the weights of the nodes in the training set as W = { w n } N tr n =1 , where w n is the learnable weight for the n -th node of the training set in G . Consequently, the reweighted partial cross-covariance matrix can be calculated as:

The learnable weights W participate in the process of optimization to eliminate as much as possible the correlation between the dimensions of the node embeddings by minimizing the partial crosscovariance matrix in Eq. 2. Specifically, for the process of optimization, given the labels of the nodes Y n ∈ R N tr , we iteratively optimize the weights of the nodes W , the slim GNNs f S , and the additional prediction layer R :

where ℓ denotes the cross-entropy loss for the node classification task. The optimization of the weights W encourages the slim GNNs f S to generate the node embeddings H , and eliminates the correlations between embeddings. The optimization of the slim GNNs f S and the additional classifier R will lead to good performance on the node classification task.

To put it in a nutshell, SLIDE is a general pre-training and fine-tuning framework in graph, which balances the number of parameters and the performance of self-supervised pre-trained GNNs. Therefore, SLIDE can be implemented using different ideas of reducing parameters and de-correlation methods. We use the methods mentioned above as examples and conduct some experiments to demonstrate the feasibility of SLIDE.

4 Experiments

Datasets. For a comprehensive comparison, we use six real-world datasets to evaluate the performance of node classification (i.e., Cora, Citeseer, Pubmed, Amazon-Photo, Amazon-Computers and Ogbn-arxiv). More details about the datasets are in Appendix C.1.

Baselines. Our proposed SLIDE is a general paradigm which removes neurons from the selfsupervised pre-trained GNNs randomly and introduces model de-correlation methods during the fine-tuning phase. We choose three representative graph pre-training frameworks for evaluation in our SLIDE: generative graph self-supervised learning (GraphMAE [3] and MaskGAE [7]), and graph contrastive learning (GRACE [1]). For each framework, we choose two classical fine-tuning methods as baselines: linear probing, and full fine-tuning. Notably, our proposed SLIDE is orthogonal to other fine-tuning methods. We provide additional experiments with SLIDE in Appendix C.2 as an example.

Experimental setup. For GraphMAE and MaskGAE, we use the implementations of their official codes [3, 7]. As for GRACE, we use the implementation of training an additional prediction layer for the node classification task, instead of using a LIBSVM classifier [26], in order to facilitate the comparison of the model's performance. For our proposed SLIDE, we use "2-half GNNs" as the slim GNNs for all datasets and pre-training frameworks. In all tables and datasets, we report averaged results along with the standard deviation computed over 5 different runs. All experiments are conducted on Linux servers equipped with NVIDIA RTX A5000 GPUs (22729 MB). We refer readers of interest to Appendix C.3 for more details on the experiments.

4.1 Effectiveness of SLIDE

To evaluate our proposed SLIDE more comprehensively, we use two common evaluation metrics, Macro-F1 and Micro-F1 scores, and show their difference between baselines and SLIDE. The results

are shown in Table 3 - 5, where "LP" means "linear probe", "FT" means "full fine-tune", and "OOM" means "out of memory". We have the following observations: (1) In general, SLIDE improves the performance compared to "LP" because SLIDE is able to fine-tune both "2-half GNNs" and the additional prediction layer. For example, in comparison with "LP" on Computers with these three competitive pre-training frameworks, SLIDE achieves an average improvement of 0 . 75% and 0 . 69% for Micro-F1 and Macro-F1 scores, respectively. (2) Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than "FT". Especially on large-scale graphs like Ogbn-arxiv with GraphMAE, SLIDE is able to fine-tune both the pre-trained GNNs and the additional prediction layer.

4.2 Model Analysis

(b) Citeseer

(c) Pubmed

(d) Photo

(e) Computers

(f) Ogbn-arxiv

SLIDEMi

w/o de Mi

SLIDE Mi

w/o de Mi

SLIDE Mi

w/o de Mi

SLIDEMa

SLIDE Ma

SLIDE Ma

86

75

84

Evaluation

84

Evaluation

70

Evaluation

82

82

65

80

80

60

78

GraphMAE

GRACE

MaskGAE

GraphMAE

GRACE

MaskGAE

GraphMAE

GRACE

MaskGAE

SLIDEMi

w/o de Mi

SLIDEMi

w/o de Mi

SLIDE Mi

w/ode Mi

SLIDEMa

SLIDEMa

SLIDEMa

w/odeMa

94

92

72

Evaluation

93

Evaluation

90

Evaluation

64

92

88

56

91

86

48

GraphMAE

GRACE

MaskGAE

GraphMAE

GRACE

MaskGAE

GraphMAE

GRACE

MaskGAE

Ablation study Here, we test the performance of the slim GNNs with and without model decorrelation on the node classification task. The results are shown in Figure 5, where "w/o dec" means that the slim GNNs are directly fine-tuned without model de-correlation. We find that the slim GNNs with de-correlation perform much better than the GNNs without de-correlation, proving that correlation is still present when self-supervised GNNs are directly fine-tuned.

Parameter analysis In order to quantify the number of parameters of the self-supervised GNN reduced by SLIDE, taking GraphMAE and GRACE as an example, we report the number of parameters of our proposed SLIDE and "FT" for fine-tuning. As can be seen in Figure 6, we observe that the parameters of "2half" GNNs are significantly reduced. In particular, on Ogbn-arxiv with GraphMAE, the number of parameters for fine-tuning is reduced by 70.1%. More details about the number of parameters are provided in B.1.

(a) GraphMAE

(b) GRACE

SL1DE

FT

SL1DE

FT

2M

Parameter

Parameter

1M

1.5M

1M

0.5M

PubMed

Photo Computers arXiv

PubMed

Photo

Computers

5 Related Work

Graph self-supervised learning Self-supervised methods on graphs can be naturally divided into contrastive and generative domains according to objective designs and model architectures [9, 2729]. Graph Contrastive Learning (GCL) has shown its outstanding ability in unsupervised setting, and many studies have been proposed [1, 4, 5, 8]. On the other hand, Generative self-supervised learning [6, 2] aims to recover missing parts of the input data. Among them, methods which have emerged in the last two years [3, 7] have significantly enhanced the performance of generative methods, resulting in competitive performance on downstream tasks and attracting much attention. Despite the remarkable achievements of these methods, the issue of model redundancy in these selfsupervised GNNs remains unexplored in the current research landscape.

Model redundancy In recent years, researchers have investigated redundancy in several pretrained model architectures for different domains. Among them, in [30], researchers dissect two pre-trained models, BERT [31] and XLNet [32], studying how much redundancy they exhibit by using feature selection to choose the subset of neurons. In [33], researchers dissected several pretrained visual models and randomly removed neurons of the penultimate layer in proportion, proving that redundancy exists in the penultimate layer. In [34], researchers find that many layers of LLMs exhibit high similarity. By removing some of the layers of large language models (LLMs), LLMs can still maintain good performance, proving that model redundancy exists in LLMs. Graph Neural Networks (GNNs) [35-37] have been widely applied in recent years and there are some studies focusing on graph sparsification and graph lottery ticket [14, 15]. Graph sparsification approximates a graph to a sparse graph by reducing the number of edges instead of parameters. And graph lottery ticket reduces parameters in networks systematically, not randomly. However, the study of model redundancy in self-supervised GNNs remains largely unexplored.

Pre-training and fine-tuning Traditional pre-training and fine-tuning paradigms mainly include "linear probe" and "full fine-tune". The former faces the challenge of insufficient performance, while the latter requires high computational cost and memory. In recent years, several Parameter-Efficient Fine-Tuning (PEFT) methods have been introduced to address these issues. Among them, Low Rank Adaptation (LoRA) [38] alters the fine-tuning phase by keeping the original model parameters frozen and introducing modifications to a separate, smaller set of parameters. These changes are then incorporated into the original parameters. On the other hand, Adapter Tuning [39] adds new modules, called adapters, between the layers of a pre-trained model. The parameters from the pretraining phase are frozen, and a smaller set of additional parameters is introduced for the new task. Acommon feature of these methods is the addition of a small number of additional parameters to the complete model for fine-tuning. The focus of this paper is orthogonal to these methods, as it aims to fine-tune the model under the condition of reduced parameters. In this paper, we provide a unique perspective on the pre-training and fine-tuning paradigm and contribute to the ongoing exploration of effective fine-tuning strategies.

6 Conclusion and Broader Impacts

In this paper, we make an exploration of model redundancy in self-supervised pre-trained GNNs. We find out that model redundancy in self-supervised GNNs exists at both neuron level and layer level, which deepens our understanding of self-supervised GNNs. We then propose a novel pretraining and fine-tuning paradigm, SLIDE, which achieves better performance with fewer number of parameters for fine-tuning. Our experiments validate the effectiveness of SLIDE.

Limitations and broader impact Although we discover that the graph self-supervised learning models are highly redundant at neuron and layer levels and deepen our understanding of selfsupervised GNNs, a potential limitation is that some theoretical foundations are still lacking. Our findings hold great potential to improve current graph self-supervised learning models and may provide valuable guideline for the pruning or the sparsity of GNNs. In the future, we will further understand self-supervised GNNs from the perspective of model redundancy by theoretical analysis. Beyond that, we do not expect any immediate negative impact on society.

Acknowledgments and Disclosure of Funding

This work is supported in part by the National Natural Science Foundation of China (No. 62322203, 62172052).

References

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning. In ICML Workshop on Graph Representation Learning and Beyond , 2020. URL http://arxiv.org/abs/2006.04131 .

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 594-604, 2022.

Petar Veliˇ ckovi´ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv:1809.10341 , 2018.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International conference on machine learning , pages 4116-4126. PMLR, 2020.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pretraining of graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1857-1867, 2020.

Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. What's behind the mask: Understanding masked graph modeling for graph autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1268-1279, 2023.

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems , 34:76-89, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33:5812-5823, 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021 , pages 2069-2080, 2021.

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11289-11297, 2023.

Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. Advances in Neural Information Processing Systems , 35:2972-2983, 2022.

Ruijia Wang, Xiao Wang, Chuan Shi, and Le Song. Uncovering the structural fairness in graph contrastive learning. Advances in neural information processing systems , 35:32465-32473, 2022.

Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International conference on machine learning , pages 1695-1706. PMLR, 2021.

Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference on Machine Learning , pages 11458-11468. PMLR, 2020.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519-3529. PMLR, 2019.

Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning , pages 528-539. PMLR, 2020.

Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5372-5382, 2021.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77. Springer, 2005.

Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering , 2022.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems , 20, 2007.

Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards a unified analysis of random fourier features. In International conference on machine learning , pages 3905-3914. PMLR, 2019.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1150-1160, 2020.

Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems , 31, 2018.

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010 , 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , 32, 2019.

Vedant Nanda, Till Speicher, John Dickerson, Krishna Gummadi, Soheil Feizi, and Adrian Weller. Diffused redundancy in pre-trained representations. Advances in Neural Information Processing Systems , 36, 2024.

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853 , 2024.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Zhongying Zhao, Zhan Yang, Chao Li, Qingtian Zeng, Weili Guan, and MengChu Zhou. Dual feature interaction-based graph convolutional network. IEEE Transactions on Knowledge and Data Engineering , 35(9):9019-9030, 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1365-1374, 2015.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

A More Experiences about Model Redundancy

↓

↓

↓

↓

↓

↓

To demonstrate that model redundancy exists across a broader spectrum of graph learning tasks, we conduct experiments on GraphMAE, effective for graph classification, and MaskGAE which excels in link prediction. For graph classification, we conduct experiments on 4 benchmarks: MUTAG, IMDB-B, IMDB-M, REDDIT-B [40]. The results in Table 6 - 7 indicate that model redundancy exists across a wide range of tasks.

B More Details about Model Redundancy

B.1 GNN Parameters and Linear Parameters

Tables 8 and Table 9 record the number of GNN parameters and Linear parameters for different ways of removing neurons. We find out that proper removal of neurons still maintains decent performance, as we show in Section 2, which means that there is a lot of model redundancy in the original GNNs. Note that we train the same number of epochs for the different ways of reducing neurons to save model training time, and the gap in model performance will be smaller, i.e. there will still be more model redundancy, if training time is more.

B.2 Hyper-parameters for GraphMAE and GRACE

For datasets where the original pre-training framework has been tested, we use the hyper-parameters from the official code, while for the other datasets, we obtain the hyper-parameters ourselves by testing on these datasets.

For GraphMAE, we obtain the hyper-parameters of pre-training on Amazon-Photo and AmazonComputers by ourselves. For both datasets, linear probes are trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.0005 while GNNs are pre-trained with a learning rate of 0.001, weight decay of 0, hidden number of 1024, head number of 4, layer number of 2, mask rate of 0.5, drop edge rate of 0.5 and epoch number of 1000.

For GRACE, we obtain the hyper-parameters of all linear probes on all datasets by ourselves because it trains a LIBSVM classifier in the official code while we obtain the the hyper-parameters of pretraining on Amazon-Photo and Amazon-Computers by ourselves. Here we list the hyper-parameters for pre-trained models and linear probes used in our experiments:

For Cora, the linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9, epoch number of 1000 and weight decay of 0.

For Citeseer, the linear probe is trained using Adam with a learning rate of 0.5, momentum of 0.9, epoch number of 1000 and weight decay of 0.

For Pubmed, the linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9, epoch number of 500 and weight decay of 0.

For Photo, the linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9, epoch number of 500 and weight decay of 0 whlie GNN is pre-trained with a learning rate of 0.001, weight decay of 0, hidden number of 512, layer number of 2, and epoch number of 200.

For Computers, the linear probe is trained using Adam with a learning rate of 0.5, momentum of 0.9, epoch number of 500 and weight decay of 0 whlie GNN is pre-trained with a learning rate of 0.001, weight decay of 0, hidden number of 256, layer number of 2, and epoch number of 200.

C Experimental Details

C.1 Datasets and Pre-training Frameworks

Here, we give some details about datasets we choose to evaluate the performance of SLIDE. As we have mentioned in Section 4, we use several citation networks and two social networks and Ogbn-arxiv datasets. Among these, the edges in citation networks (i.e. Cora, Citeseer, and Pubmed) represent the citation relationship between two papers (undirected), the node features are the bag-ofwords vector of the papers, and the labels are the fields of the papers. The nodes in social networks (i.e. Amazon-Photo and Amazon-Computers) represent the products, the edges represent whether the two products are frequently purchased together, the features represent the product reviews encoded in bag-of-words, and the labels are the predefined product categories. Ogbn-arxiv captures citation relationships between computer science papers on arxiv. Nodes denote papers, edges denote citation relationships of papers, and each paper has a dimensional feature vector obtained by averaging the embeddings of words in the title and abstract. The embeddings are obtained using Word2Vec [41]. The test is to predict 40 domains over CS.

For the implementations of three pre-training frameworks, we use their original code. The sources are listed as follows:

GraphMAE: https://github.com/THUDM/GraphMAE

GRACE: https://github.com/CRIPAC-DIG/GRACE

MaskGAE: https://github.com/EdisonLeeeee/MaskGAE

C.2 Additional Experiments and Analysis of SLIDE with Fine-Tuning Methods

Here we examine the orthogonality of SLIDE in relation to traditional fine-tuning methods. We start with the classic LoRA method [38] applied to GraphMAE, where SLIDE randomly prunes a subset of neurons to create Slim GNNs. We then introduce an additional LoRA module designed for finetuning. The results are shown in Table 11. Notably, "SLIDE-LoRA" can only adjust the parameters of the LoRA modules, as the Slim GNNs remain fixed. Despite this limitation, "SLIDE-LoRA" enhances performance by reducing correlations among final representations, achieving slightly better results compared to using LoRA directly on Original GNNs. This supports the efficacy of our method in improving model capabilities.

C.3 Experimental Settings

Here, we provide more experimental settings about the experience about SLIDE. We obtain the hyper-parameters ourselves by testing on these datasets with three frameworks except linear probing (we obtain the hyper-parameters from the official code).

Here we list the hyper-parameters for full fine-tuning:

GraphMAE:

-Cora: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of 1e-4 whlie GNN is tuned with a learning rate of 1e-7, weight decay of 0.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 1e-1 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 1e-3.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 0.

-Photo: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-6, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-5, weight decay of 0.

-Ogbn-arxiv: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-4, weight decay of 1e-3.

• GRACE:

-Cora: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-7, weight decay of 0.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 1e-8, weight decay of 0.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 0.

-Photo: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-5, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 0.1, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-4, weight decay of 0.

MaskGAE:

-Cora: The linear probe is trained using Adam with a learning rate of 5e-3, momentum of 0.9 and weight decay of 1e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 1e-3.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 1e-4.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.015, momentum of 0.9 and weight decay of 5e-4 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 0.

-Photo: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 5e-3, momentum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of 2e-4, weight decay of 0.

-Ogbn-arxiv: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 0.

Here we list the hyper-parameters for SLIDE:

GraphMAE:

-Cora: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of 1e-4 whlie GNN is tuned with a learning rate of 1e-7, weight decay of 0.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 1e-1 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 1e-3.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 0.

-Photo: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-6, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-5, weight decay of 0.

-Ogbn-arxiv: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-4, weight decay of 1e-3.

GRACE:

-Cora: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-7, weight decay of 0.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 1e-8, weight decay of 0.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of 0.

-Photo: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-5, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 0.1, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 5e-4, weight decay of 0.

MaskGAE:

-Cora: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of 1e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 5e-3.

-Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 5e-4, weight decay of 1e-4.

-Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 1e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 1e-2.

-Photo: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 6e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 0.

-Computers: The linear probe is trained using Adam with a learning rate of 5e-3, momentum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of 1e-4, weight decay of 0.

-Ogbn-arxiv: The linear probe is trained using Adam with a learning rate of 3e-3, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 2e-4, weight decay of 0.

And the hyper-parameters of linear probing is the same as the config file of these pre-training frameworks. The hyper-parameters are different sometimes because their model structures are different, the parameters used to achieve optimal performance are sometimes different. And the hyperparameters of the slim GNNs without model de-correlation are the same with SLIDE.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we mention that the graph self-supervised learning models are highly redundant at neuron and layer levels and propose a new model to fine-tune both GNNs and predictive layers.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We provide some experience about CKA scores of presentations in Section 2, and to our best knowledge, many researches about model redundancy do not provide theory assumptions and proofs either. So I think it's difficult to provide the proofs.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix C.3.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Section 1 and Section 4.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4 and Appendix C.3.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Table 3 - 5 in 4.1 and Figure 5 in Section 4.2.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 4.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I have read the NeurIPS Code of Ethics https://neurips.cc/ public/EthicsGuidelines , and I think the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section 6.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We test self-supervised GNNs on the node classification task. There is not a high risk for misuse in it.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [No]

Justification: We were unable to find the license for the assets we used.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This paper presents new insights on graph self-supervised learning models. Namely, the parameters, as well as the learned representations, of graph self-supervised learning models are highly redundant. The paper also proposes a novel pre-training and fine-tuning paradigm, SLIDE, which achieves better performance with fewer number of parameters for fine-tuning. The experimental results are also remarkable, e.g., the improvements can be also achieved even with a random reduction of parameters.

STRENGTHS:
- Deepening our understanding of graph self-supervised learning models is an important endeavor, given the popularity of these models. This paper is aligned with research lines and proposes some important discovers on redundancy in terms of both neuron and layer levels.
- The paper is overall well-organized and clearly written. I enjoy reading it.
- The experiments are clearly designed and well-executed, further demonstrating the value of this work.

Overall, this paper is a valuable contribution to the field and deserves a wide audience for its discovers and insights.

WEAKNESSES:
I think it would be valuable to explain more on the empirical results. I in particular wonder why “Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than FT".
Is there any (mathematical) definition on the model redundancy? It is better to verify that the proposed model does reduce the model redundancy.

QUESTIONS:
see weakness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the redundancy in graph self-supervised learning models. The authors discover that even randomly removing a number of parameters, the performance of graph self-supervised learning models is still comparable, revealing the redundancy problem. Then the authors propose to simultaneously fine-tune the graph self-supervised learning models and the prediction layers, and the effectiveness are well demonstrated on various benchmarks.\n\nSTRENGTHS:\nDiscovery of Redundancy: The paper identifies an intriguing phenomenon of redundancy within graph self-supervised learning models. This finding provides valuable insights for future research directions, particularly in the full fine-tuning of these models. Clear Organization: The paper is well-organized and logical, featuring high-quality tables and figures that effectively support the presented data and analyses. Strong Performance: The performance of the proposed model is robust, showing significant improvements when compared to baseline models.\n\nWEAKNESSES:\nSome technique details in the paper are not clearly introduced, leaving several critical aspects insufficiently explained. For example, the rationale behind the use of Random Fourier Features (RFF) is not adequately addressed. It would be beneficial to provide more in-depth explanations on why RFF was chosen for this context and how it contributes to the model's performance. Additionally, the meaning and significance of Equation 2 are not thoroughly discussed, leaving readers uncertain about its role and impact within the overall framework. The paper also lacks an analysis of the learned weights 𝑊 which is crucial for understanding the model's learning dynamics and interpretability. Including such an analysis could offer valuable insights into how the model processes and prioritizes different features. Moreover, the inclusion of more experimental results would enhance the robustness of the findings. By providing a more comprehensive set of experiments, the paper could demonstrate the consistency and generalizability of the proposed approach across different scenarios and datasets.\n\nQUESTIONS:\nPlease refer to the weakness above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper is the first to uncover that graph self-supervised models exhibit high model redundancy at both neuron and layer levels, providing two key perspectives for graph pre-training and fine-tuning framework.
This paper proposes SLIDE to achieve a pre-training and fine-tuning paradigm with fewer parameters and better performance on the downstream task.
The authors conduct comprehensive experiments, showing superior performance.

STRENGTHS:
Overall, this paper is in general of good quality that it is well organized and in general clearly written. The motivation is clear and strong.
The problem under investigation is an interesting problem and the work offers some important discovers and results for this problem. 
The proposed method is simple, effective, and well-motivated with excellent performance. Some results are particularly impressive (e.g., even randomly reducing 40% of parameters, the improvement is still 0.24% and 0.27%)

WEAKNESSES:
The task evaluated in this paper is only node classification. I’m curious what about other tasks? It might be important to check the correlation between the model redundancy problem and node classification, or the problem is task-independent?
 The authors point out that the proposed model is orthogonal to other fine tuning methods. A detailed discussion or more experimental results would be a plus.

QUESTIONS:
See weakness section

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the redundancy issue of GNNs that are pre-trained in the self-supervised manners. Examples of these methods include GraphMAE and GRACE. The first part of this paper shows that the numbers of parameters of these models can be reduced by half, while their performance wouldn’t change much (only drops to 96.2% of the original GNNs). This is a very  interesting finding. \nThe second part of the paper presents a pre-training and fine-tuning paradigm called SLIDE. It aims to fine-tune a slim GNN (a half-sampled pre-trained GNN), along with a prediction layer and weights of the nodes, so the node classification performance could be improved (comparing to other fine-tuning method).\n\nSTRENGTHS:\n1.\tThe finding presented in the first part of the paper is very interesting. The redundancy found in pre-trained weights of GNN via self-supervised learning could help simplification of GNNs and their deployment in low-resource computing environments. \n2.\tThe empirical study of the redundancy issue in the first part is reasonable, and covers two representative graph self-supervised learning models (GraphMAE  and GRACE). Therefore, the results are meaningful and insightful. \n3.\tThe paper overall is well written, and easy to follow.\n\nWEAKNESSES:\n1.\tThe  study of the redundancy issue in the first part is  solely on the node classification task. Although node classification is the most common problem in graph learning, GNN pre-trained models are also widely used for link prediction and the entire graph classification. The further study of the redundancy issue for other tasks would make this work more solid.\n2.\tThe second part of the work (the SLIDE model) is a less valuable contribution. It is unclear about the motivation for “de-correlating the learned embeddings H in the fine-tuning phase, making models with fewer parameters more informative”. The “Slim GNN” preserved most of the useful weights already. Encouraging the independence of the node embeddings or not doesn’t change much the performance in downstream tasks. In Fig. 5, models without “de-correlation” only have a slight performance decrease. However, including “de-correlation” can level up the computational complexity.\n3.\tThe comparison in Fig.6 with “full fine-tune” is unfair. Why not including the  “linear probing” for comparison?  fine-tuning only the additional classifier and freezing the original GNNs). The number of parameters in “linear probing” is much less, while the performance of “linear probing” is not that bad (as shown in Table 3-5).\n\nQUESTIONS:\nWhile I really like the first part of the work, I have the following questions:\n1.\tWhat datasets were used for pre-training these GNNs before getting the slim GNNS? \n2.\tAny possibility to provide theoretical analysis for the redundancy issue?\n3.\tDid authors run statistical tests for comparing the performance of LP, FT and SLIDE in Table 3-5?  The performance drop indicated in red color is often smaller than the std. So it is unknown if the performance difference is statistically significant.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}