Uncovering the Redundancy in Graph Self-supervised Learning Models
Authors: Zhibiao Wang
Xiao Wang
Haoyue Deng
Nian Liu
Shirui Pan
Chunming Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph self-supervised learning
Model redundancy
Pre-training and fine-tuning paradigm
Primary area: graph_neural_networks

Abstract: Graph self-supervised learning, as a powerful pre-training paradigm for Graph Neural Networks (GNNs) without labels, has received considerable attention. We have witnessed the success of graph self-supervised learning on pre-training the parameters of GNNs, leading many not to doubt that whether the learned GNNs parameters are all useful. In this paper, by presenting the experimental evidence and analysis, we surprisingly discover that the graph self-supervised learning models are highly redundant at both of neuron and layer levels, e.g., even randomly removing 51.6\% of parameters, the performance of graph self-supervised learning models still retains at least 96.2\%. This discovery implies that the parameters of graph self-supervised models can be largely reduced, making simultaneously fine-tuning both graph self-supervised learning models and prediction layers more feasible. Therefore, we further design a novel graph pre-training and fine-tuning paradigm called SLImming DE-correlation Fine-tuning (SLIDE). The effectiveness of SLIDE is verified through extensive experiments on various benchmarks, and the performance can be even improved with fewer parameters of models in most cases. For example, in comparison with full fine-tuning GraphMAE on Amazon-Computers dataset, even randomly reducing 40\% of parameters, we can still achieve the improvement of 0.24\% and 0.27\% for Micro-F1 and Macro-F1 scores respectively.

Uncovering the Redundancy in Graph Self-supervised Learning Models
Zhibiao Wang', Xiao Wang"*, Haoyue Deng', Nian Liu', Shirui Pan?, Chunming Hul
Beihang University, China
Beijing University of Posts and Telecommunications China Griffith University, Australia
(wzb2321, _ang, hacyue _engg, hucm) @buaa edu. haaniuubuupt.duu en, s. waangriiniiitete edu. au
Abstract
Graph self-supervised learning, as powerful pre-training paradigm for Graph Neural Networks (GNNs) without labels, has received considerable attention. We have witnessed the success of graph self-supervised learning on pre-training the parameters of GNNs, leading many not doubt that whether the learned GNNs parameters are all useful. In this paper, by presenting the experimental evidence and analysis, we surprisingly discover that the graph self supervisdd learning mod- els are highly redundant at both of neuron and layer levels, e.g., even randomly removing 51.6% of parameters, the performance of graph self-supervised learn- ing models still retains at least 96. 9%. This discovery implies that the parameters of graph self-supervised models can be largely reduced, making simultaneously mne-tuning both graph self-supervised learning models and prediction layers more feasible. Therefore, we further design a novel graph pre-training and fine-tuning paradigm called SLImming DE-correlation Fine-tuning (SLIDE*). The effective- ness of SLIDE is verified through extensive experiments on various benchmarks, and the performance most cases. For example, in comparison with full fine-tuning GraphMAE on Amazon-Computee dataset, even randomly reducing 40% of parameters, still achieve the improvement of 0.24% and 0.27% for Micro-F1 and Macro-F1 scores respectively.
be even improved with fewer parameters of models in
Introduction
Graph self-supervised learning, aiming at learning the parameters of Graph Neural Networks (GNNs) without labels, has been a popular graph pre-training paradigm [1-3]. Usually, graph self-supervised learning is naturally divided into two learning methods, i.e., graph contrastive leam- ing [4, 5] and graph generative self-supervised learning [6. 7]. After pre-training the parameters of self-supervised GNNs, graph self-supervised learning achieves state-of-the-art performance variety of tasks by fine-tuning an additional prediction layer [8, 7].
Various researches have attempted to improve and understand graph self-supervised learning from different perspectives. For example, the graph augmentation techniques [9, 10), graph spectrum fea- ture of graph self-supervised learning [11. 12], and others [7, 13]. Despite their remarkable achieve- ments, little efforts have been made to understand the behavior of the learned model parameters by graph self self-supervised learning. Here, we ask: Are the model parameters all always useful? Or
"Corresponding authors.
"Code available at https//githab.com//hggg/LLDD
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

what part of model parameters is useful? Whether there is the model redundancy in self-supervised GNNs? Understanding the model property in graph self-supervised learning, particularly the model redundancy, can provide valuable guidelines and insights for the development of advanced graph self-supervised learning.
For this purpose, we conduct experiments and surprisingly find out that the graph self-supervised learning models are actually highly redundant. We start with the experiments (Section 2) investi- gate the downstream performance of different graph self-supervised learning models with randomly removed parameters at both neuron and layer levels. Our findings indicate that graph supervuper learning models with even half of the neurons randomly removed have virtually no impact on the performance of the node classification task. Moreover, by closely examining the learned represen- tations, we find out that the representations in each layer with a substantial number of neurons removed are quite similar to the representations obtained by the full set of neurons. Meanwhile, there is also strong similarity between the representations of each layer and its adjacent layer. All results demonstrate that graph self-supervised models exhibit high model redundancy at both neuron level and layer levels.
The above findings hold great potential to improve current graph self-supervised learning models. On the one hand, it may provide valuable guideline for the pruning or the sparsity of GNNs [14, 15]. On the other hand, the phenomenon of model redundancy provides a new opportunity for designing a full fine-tuning graph self-supervised learning model. Previously, considering that the number of parameters be fine-tuned is excessive when we directly fine-tune both GNNs and prediction layers, we usually only have to fine-tune the attached prediction layer, i.e., linear probing [3. 5, 7]. Here, if the parameters can be greatly reduced, we can simultaneously fine-tune both graph self- supervised learning models and prediction layers. Besides, the findings imply that de-correlating the learned representations is also necessary. Therefore, we propose novel pre-training and fine- tuning paradigm called SLIDE by obtaining SLIm GNNs from the self-supervised GNNs and using the DE-correlatin strategy to reduce the correlation between features during the fine-tuning phase. Extensive experiments on various benchmark datasets validate the effectiveness of SLIDE.
In summary, our contributions are three-fold:
To the best of our knowledge, we are the first to uncover that graph self-supervised models exhibit high model redundancy at both neuron and layer levels. The model redundancy allows for the removal of majority of model parameters with almost no impact on the performance of the downstream task. This discovery can significantly enhance model effi- ciency while maintaining acceptable task performance.
This discovery provides two key guidelines for the subsequent graph pre-training and fine- tuning framework: one is that the parameters can be reduced, and the other is the represen- tations should be de-correlated. These motivate to propose a novel method, SLIDE, to achieve pre- pre-training and fine-tuning paradigm with fewer parameters and better perfor- mance on the downstream task.
Comprehensive experiments demonstrate that our SLIDE outperforms baselines across multiple benchmark datasets. For example, using the Amazon-Compute dataset [16] with GRACE [1], compared to full fine-tuning, we achieve improvements of 0.37% and 0.16% for Macro-Fl and Micro-Fl scores respectively with a random reduction of 30% of parameters.
2 The Model Redundancy in Graph Self-supervised Learning
In this section, we investigate the model redundancy in two representative graph self-supervised learning models (GraphMAE [3] and GRACE [1]) on node classification tasks. Specifically, we pre- train these graph self-supervised models on Cora, Citeseer, Pubmed [17], Amazon-Photu Amazon- Computers [16], Ogbn-arxiv [18), and then we evaluate the performance again after removing neu- rons in various ways. If the performance gap between the original GNNs and the slim GNNs with fewer neurons is small, it indicates great model redundancy in self-supervised GNNs. In other words, even with reduced neurons, the performance remains largely unaffected, suggesting that many neu- rons in self-supervised GNNs are dispensable. Notably, model redundancy is not limited to node classification tasks. We conduct experiments in Appendix demonstrating that model redundancy also persists in link prediction and graph classification tasks.

---

holf GNNs
original GNNs
2-half GNNNs
imput neurond
hidden neurons
output neurond
drop neurons
Figure 1: Neuron dropout. To initialize a smaller variant of the self-supervised pre-trained GNNs, we select parameters from self-supervised GNNs in different ways. From left to right: randomly reduce the number of neurons each layer proportionally. the original GNNs, retain only the first two layers while randomly reducing the number of neurons in the second layer proportionally.
Neuron removal We explore two different ways to reduce the number of neurons in self- supervised GNNs, i.e., the original GNNs, which are illustrated in Figure 1. Specifically, we consider two approaches: (I) neuron level: randomly retaining the number of neurons by 50% and 25% in each layer named "half" and "quarter", and (2) layer level: retaining only the first two layers if the layer number is greater than two and randomly retaining the number of neurons by 100%, 50% and 25% in the second layer named "2-original", "2-half" and "2-quarter". In this way, we obtain five types of slim GNNs: "half GNNs", "quarter GNNs", "2-original GNNs", "2-half GNNs" and "2-quarter GNNs".
Table
The performance of different neuron removal methods on six datasets with GraphMAE.
Dataset
Metric
Original
Half
Quarter
2-Original
2-Half
2-Quarter
F1-Mi
83,92
80.36, 5.50
74.88, 9.04
82.84, 1.08
81.14, 2.78
Cora
Fl-Ma
79.85, 3.36
73.94, 9.07
82.06, ous
80.31, 2.70
Change Puram
156556
J. 79.92
13:20
191.80
FI-Mi
73.26
72. 10
69.34,
73. aues
72.28,
CitcSeer
FI-Ma
67.31
64,80,
62.18,
66.86, ouss
66.32,
Change-Param
1 53.03
1 17.277
6.10
1 1.915
FI.Mi
80.62
77.94, 9:68
74.86, 5.76
79.74 o.ss
77.62, 3.00
PuhMed
FI-Ma
79.97
77.40 9:59
74.31,
79.06 0.91
77.17; 9:50
Change-Param
166667
4. 87.55
J. 1.33366
1.50.31
FI-Mi
93.11
92.75, 0.36
92,26,
92.94 0.17
92.95,
Photo
F1-Ma
91.11
91.42,
90.81; 1.10
91.65, 0.00
91.63;
Change
64.39
J. 1.8488
1. 2.28.22
444.38
F1-Mi
89.49, 0.36
87.87, 2.57
90.14, 0.30
89.75; 0.69
Computers
F1-Ma
89.24
88.30, 0.94
86.25 2.00
88.92, 0.02
88.55,
Change Puram
1 64.21
4 15.666
1. 182857
1.42.85
F1-Mi
71.90
70.83
69 69.65, 2.25
71.47
70.64; 1:28
69.90;
arXiv
F1-Ma
51.14
49.16 19.116,11 1.08
46.77_ 4.37
50.81 0.35
49.82, 1.32
48.50,
2.64
Change-Param
73.37
12.55
16166
70.45
1882
Table 2: The performance of different neuron removal methods on five datasets with GRACE Ogbn- arxiv is not included because it is "out of memory" when Ogbn-arxiv is pre-trained with GRACE.
Dataset
Metric
Original
Quarter
2.Quarter
FI-Mi
82,30
80.70,
1:50
77,23, s.0n
81.83
0.47
79,45
2.87
Com
FI-Ma
81.12
79.35
75.11
80.66
78.12,
Change-Param
52.00
4 76.54
141.1
6.19
F1-Mi
69.49
69.47, 0.00
68.01; 0.08
69.75 0.05
69.27, 0.92
CiteSeer
F1-Ma
61.77
68 0:00
61.69, 0.08
62.41: 0.61
62.13-
Change Purum
1..662
1. 76.21
3.224
J. 4.488
F1-Mi
81.14
79.47,
76.29; 6.s9
81.06,
80,42,
PubMed
FI-Ma
81.05
79.52,
75.70, 5.5s
80.98,
80.48, 0.57
Change-Purae Purum
5.54.55
4 181.34
16.93
[ 2,25.0
FI-Mi
91.95
91.40, 0.36
87.90, 0.0s
91.35, 0.00
90.95 1.00
Photo
FI-Ma
90.10
89.40, 0.70
83.76 0.54
89.40 0.70
88,95 1.15
Change-Paree
60.17
1 1.82.63
20.36
3..55
Fi-Mi
87.57
85.98,
83.61 3.00
86
85.82, 1.75
Computers
F1-Ma
85.84
84.31; (
81.33 4.51
85.10
84.15, 1.69
Chango-Param
60.00
J 1..250
20.01
3..002
Experimental results can obtain five kinds of slim GNNs by using the neuron removal methods mentioned above. For
We first pre-train the original GNNs through GraphMAB and GRACE, then
3

---

both the slim GNNs and the original GNNs, we individually attach an additional prediction layer which is trained while keeping the original GNNs frozen on the node classification task evaluate the Micro-F1 (F1-Mi) and Macro-F1 (F1-Ma) scores, as shown in Table 1 and Table 2, where "-" means that there are only two layers in the original GNNs (i.e.," "2-Original" and "Original" have the same performance) and Shaang-PPrramy means the percentage change of the number of parameters compared to "Original". Surprisingly, we have following observations:
In all cases, "half GNNs" retain at least 96.2% % of the performance of the original GNNs while the numbers of parameters are reduced by at least 51.6%. In most of the cases, "quarter GNNs" still retain at least 90% of the performance of the original GNNs while the number of parameters are reduced by at least 76.2%. As can be seen, "quarter GNNs" of GRACE on Computers retains 95.5% of the Micro-F1 score and 94.7% of the Macro-Fl score, while the number of parameters is reduced by 82.5%.
The removal of layers after the second layer has negligible impact on the perfor- mance. The performance of "2-original GNNs" with GraphMAE on Ogbn-arxiv dataset demonstrates that even when layers after the second layer are removed, both the Micro-Fl score and the Macro-F1 1 score retain 99.4% % of the performance, while the number of parameters is reduced by 47.0%
The observations indicate that the graph sulf-supe learning models are highly redundant, both at the neuron level and at the layer level. More details about the hyperparameters and the number of parameters can be found in Appendix B.
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
a
GRACE_1
0%
25%
50%
75%
-00%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
neuron removal rate
neuron removal rate
neuron removal rate
(a) Cora
(b) Citesecr
(c) Pubmed
1.0
0.8
0.8
0.8
g
GraztMAE JUL
GraphIKE
0.4
0.4
0.4
Grapleee 1
GRACT al.I
GRACE 11.13
GraghMet 2l.
D
GRACE $2
GraphMee a
0.0
0.0
0%
25%
50%
75%
200%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
neuron removal rate
neuron removal rate
neuron removal rate
(d) Photo
(e) Computers
(f) Ogbn-arxiv
Figure 2: CKA scores between the representations of the slim GNNs and the same layer in the orig- inal GNNs with GraphMAE and GRACE on several datasets. "all" means we remove the neurons from all layers the same proportion. "11" means that we calculate CKA scores of the representa- tions from the first layer, and "I2" means CKA scores from the second layer, and so on.
Redundancy analysis neuron level Here, we further analyze the model redundancy by closely examining the learned representations in order to explore why the slim GNNs achieve similar per- formance to the original GNNs. Specifically, each layer in 25%, 50% and 75% to analyze the model redundancy at the neuron level. Given the learned representations of the i-th layer of the slim GNNs and the original GNNs, Centered Ker- nel Alignment (CKA) is adopted to calculate their similarity [19]. A high CKA score implies a strong similarity. As shown in Figure 2, compared the original GNNs, the CKA scores are 85% when 50% neurons are dropped. Even if the neurons are removed by 75%, the CKA scores are still almost over 80%. This indicates that removing a significant number of neurons from each layer has a minimal impact on the significance of representations of each layer and task performance consequently.
get the slim GNNs by dropping the neurons of

---

Redundancy analysis on layer level As for the model redundancy at the layer level, given the learned repre- sentations of the i-th layer and the (i + 1--th layer of the original GNNs, we also adopt CKA to calculate their similarity, where the O-th layer rep- resents the original features of nodes from the datasets. As shown in Figure 3, we report the CKA scores between the representations of each layer and its adjacent layer the original GNNs for GraphMAF and GRACE. "data-layerl" means that we calculate CKA scores between the original features of the nodes from the datasets and the representations of the first layer, and so on. As can be seen, CKA scores between the original features and the repre- sentations of the first layer are relatively low, while CKA scores between the representations of the layers after the first layer and their next layer are much closer to 1 in most of the cases, indicating the model redundancy at the layer level.
Cora
Cora
0.8
0.8
Citeseer
Ogbn-arsiv
0.4
0.4
Computer
Pubmed
Computers
Puomes
Pficto
data-layer?
layart-uper2 Photo
layar--upers
byert ayer?
(a) GraphMAE
(b) GRACE
Figure 3: CKA scores between the representations of each layer and its adjacent layer of the original GNNs for Graph- MAE and GRACE on several datasets.
Our Proposed Tuning Approach: Slimming De-correlation Fine-tuning
Model Slimming
De-correlation
Mininize Corclation
Update
Pre-trained GNN
Slim GNN
RFF Extract
RFF Maps
Roweight Weights
Lostsnnipe
Prediction
Input Graph
Slim GNN
Node
Wriights
Classieer
Losspataiaay
Figure The overall framework SLIDE.
In general, for the self supervised pra-trdi GNNs, attach additional prediction layer which fine-tuned while keeping the GNNs frozen to conduct the downstream tasks. Ideally, we are able the parameters of both the GNNs and the linear layer, is possible to achieve the best performance. However, the number of parameters to be fine-tuned is excessive. Here, since identify the model redundancy in the self-supervised pre-trained GNNs, this motivates that we actually only need to tune the additional classifier and the slim GNNs, so as to obtain a better trade-off between the model performance and the number of fine-tunable parameters. Therefore, we propose a novel pre-training and fine-tuning paradigm called Smmmiing DE-correlation Fine-tuning (SLIDE), shown in Figure 4. Specifically, firstly reduces model redundancy in self-supervised pre-trained GNNs by randomly removing redundant neurons to obtain the slim GNNs. Then, we input the graph data into the slim GNNs to obtain the embeddings of the nodes, combined with an additional prediction layer, we can predict the label of each node. Meanwhile, we design another model de-correlation module based the squared Frobenius norm [20, 21] (an analogue corresponding to the Hilbert-Schmidt Independence Criterion, i.e., HSIC [22] in Euclidean space). The module learns the de-correlation weights for the classification loss, so as to reduce the redundancy among embeddings and make the embeddings more informative.

---

Specifically, let G (AA, V, X) denote the imput graph, where V is the node set, N the number of nodes, Nrr the number nodes in the training set and is the dimension of the node features, A [0,1) is the adjacency matrix, and X E R is the input node feature matrix. For model slimming, we first pre-train the original GNNs through the existing pre- training frameworks, e.g., the graph contrastive learning [1. 4, 5] or graph generative self-superviied learning [3, 7, 2]. Then the slim GNNs fs can be obtained by randomly reducing both the neurons and the layers. Furthermore, given G, we can get the node embeddings H E N as H [A, X), where d, the dimension of the node embeddings.
[V is
Motivated by Section 2 that the neurons, as well as the learned embeddings, in self-supervised GNNs are highly redundant, we aim to de-correlate the learned embeddings H in the fine-tuning phase, making models with fewer parameters more informative. In particular, in inspiration of the de-correlation methods [21, 23), given the i-th dimension and dimension of the node embeddings H., and H..j. we obtain Random Fourier Features (RFF) [24, 25] as
2 H. .) = (u1 (H, .). (H, (H. UNner (H,,)), o (H.,)) = (v..H..)) (H.,)) U2 H. ...... UNRFF (H.,)))
where NRFF is the number of functions in the random fourier space, Ug and Ug denote the functions from the space of Random Fourier Features. Then, we elaborate on reweighting of the weights, which encourages the independence of the node embeddings. Define the weights of the nodes in the N.. training set as W { In where wn, is the learnable weight for the n-th node of the training set in G. Consequently, the reweighted partial cross-cos coos-coovr matrix can be calculated as:
1 Ener W H., N.. -
(Hns)
No. Em=i Wm u (Hm.s)
(2)
Wr (,,nHH,,,) m.
1. ExYe Wm v "H (Hmm))
N..
The learnable weights W participate in the process of optimization to eliminate as much as possible the correlation between the dimensions of the node embeddings by minimizing the partial cross- covariance matrix in Eq. 2. Specifically, for the process of optimization, given the labels of the nodes Y, Re we iteratively optimize the weights of the nodes W, the slim GNNs fs, and the additional prediction layer R:
Now
fr. R = argmin fg.R
Wn e (R o fs (Xn). Y,),
(3)
W argminw
CC H.,, 11:
IsicjEda
where denotes the cross-entropy loss for the node classification task. The optimization of the weights W encourages the slim GNNs Is to generate the node embeddings H, and eliminates the correlations between embeddings. The optimization of the slim GNNs Is and the additional classifier R will lead to good performance on the node classification task.
To put in nutshell, SLIDE is a general pre-training and fine-tuning framework in graph, which balances the number of parameters and the performance of self-supervised pre-trained GNNs. There- fore, SLIDE can be implemented using different ideas of reducing parameters and de-correlation methods. We use the methods mentioned above as examples and conduct some experiments to demonstrate the feasibility of SLIDE.
4 Experiments
Datasets. For comprehensive comparison, use six real-world datasets to evaluate the perfor- mance of node classification (i.e., Cora, Citeseer, Pubmed, Amazon-Photo, Amazon- Coopuutrsssserss and Ogbn-arxiv). More details about the datasets are in Appendix C.I.

---

Baselines. Our proposed SLIDE is a general paradigm which removes neurons from the self- supervised pre-trained GNNs randomly and introduces model de-correlation methods during the fine-tuning phase. We choose three representative graph pre-training frameworks for evaluation in our SLIEE generative graph self-supervised learning (GraphMAE. [3] and MaskGAE [7],, and graph contrastive learning (GRACE [1). For each framework, we choose two classical fine-tuning methods as baselines: linear probing, and full fine-tuning. Notably, our proposed SLIDE is orthogo- nal to other fine-tuning methods. We provide additional experiments with SLIDE in Appendix C.2 as an example.
Experimental setup. For GraphMAE and MaskGAE, we use the implementations of their official codes [3, 7J. As for GRACE, we use the implementation of training an additional prediction layer for the node classification task, instead of using a LIBSVM classifier [26], in order to facilitate the comparison of the model's performance. For our proposed SLIDE, we use "2-half GNNs" as the slim GNNs for all datasets and pre-training frameworks. In all tables and datasets, we report averaged results along with the standard deviation computed over 5 different runs. All experiments are conducted on Linux servers equipped with NVIDIA RTX A5000 GPUs (22729 MB). We refer readers of interest Appendix C.3 for more details on the experiments.
Table 3: Node classification accuracy (%.oor on six benchmark datasets with GraphMAE.
Baselines
Metries
Cora
Citescor
Pobmod
Photo
Computers
Ogbu-arxiv
Fi-Mi
4 0.32
2 0.48
80.62 0.10
93.11 2om 4 0.56
90.44 0.41
71.90gam J 0.37
LP
FI-Ma
3 0:28
67.7 2 1.87
79.97
0:15
0.79
89.24 0.64
51.142o2 J 1.89
FI-Mi
102130 0.188
73.62
0.122
80
20.6
0.04
93.61
0:006
90.61=max
J 0.24
00M
FT
F1-Ma
83.08 0.21
65.21_mmm
0.68
80.05
0.06
92.55
4 0.15
89.61_nos 4 0.27
00M
FI-Mi
14
80.72
93.67
90.85
72.27
SLIDE
FI-Ma
23.
25.87
80.1 2200
92.70
89.88
53 53.03
Table 4: Node classification accuracy (o) on five benchmark datasets with GRACE.
Baselines
Metrics
Com
Citeseer
Pubmed
Photo
Computers
FI-Mi
82.30 0.50
69.49auzz 2.09
81.14 0:18
91.95 2nm 0.98
87.57 1.25
FL-Ma
81.12 0.05
61.77 1.86
81.05
0.26
10.000 1.55
85.84 L. 0.89
FI-Mi
82 62.662u24 1 10.14
70.38
1:220
81,44 J 0.18
92. g2.8
0.07
88
28.66ge25
0.16
FI-Ma
4 1.1.
61.73
1,50
81.11=02 J 0.20
91.58 J 0.07
86.36
0.37
FI-Mi
82.80 82.80
71 71.58
81.62aa2s
92, 92,93
88.82201s
SLIDE
F1-Ma
81.17
63.63
81.31
91.65
86.73m034
Table 5: Node classification accuracy (%doo) on six benchmark datasets with MaskGAE.
Baselines
Metrics
Cora
Citeseer
Pubmed
Computees
Oghu-arxiv
FI-Mi
83.08 0.61
72.66 72.66=ma1 0.74
33.463mml 0.02
92.94a01 4 0.15
89.57 0.58
71 71.28 J 0.12
LP
FL-Ma
81.82 0.58
a7 1.03
82. 32.84 0.20
91.72
0.25
18.15
0.55
49.48 J 0.94
FI-Mi
83.34 0.38
72.42
0.988
83. 14s0 L. 0.14
93.05 sun 4 0.00
90.10 J 0.05
71.34 J 0.06
FT
FI Ma
12.15 4 0.25
6
0.39
82. 2933
0.25
91.84
4 0.133
88.58 J 0.12
50.51 1 0.09
F1-Mi
83.72
73.40_a51
83,48
93.09
90.1
71.40g0M
SLIDE
FI-Ma
82.40
19.31
83 222s
91.97
88,70
50.42zus0
4.1 Effectiveness of SLIDE
To evaluate our proposed SLIDE more comprehensively, we use two common evaluation metries, Macro-F1 and Micro-Fl scores, and show their difference between baselines and SLIDE. The results
7

---

are shown in Table 3 5, where "LP" means "linear probe", "FT" means "full fine-tune", and "OOM" means "out of memory". We have the following observations: (1) In general, SLIDE improves the performance compared to "I.P" because SLIEE is able to fine-tune both "2-half GNNs" and the additional prediction layer. For example, in comparison with "I.P" on Computers with these three competitive pre-training frameworks, SLIDE achieves an average improvement of 0.75% and 0.69% for Micro-FI and Macro-FI scores, respectively. (2) Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than "FT". Especially on large-scale graphs like Ogbn-arxiv with GraphMAE, SLIDE is able to fine-tune both the pre-treee tre-trained GNNs and the additional prediction layer.
4.2 Model Analysis
SU
wodde/
BLLDDE MI
We de
SU SUIDE
86
75
Eveliatoo
64
Eviluaiion
70
Eveluction
82
Groph1ANE
GropAAAE
GoopMMA
(a) Cora
(b) Citeseer
(c) Pubmed
Mo
BLIDE MI
wi
St. BDE Ms
Iol tis
94
Evelation
09
Exilafion
00
Exelustion
02
88
GrophtAEE
GRACE
NexA4E
GrapAANE
GRADE
MasRGAE
GrepMMAA
GRAAEE
MexGGE
(d) Photo
(e) Computers
(I) Ogbn-arxiv
Figure 5: Ablation studies of model de-correlation on six benchmark datasets and three pre-training frameworks. "w/o de" means that we fine-tune the slim GNNs without model de-correlation methods. "Mi" means Micro-F1 scores and "Ma" means Macro-F1 scores. GRACE are unseen because of "out of memory".
The results of Ogbn-arxiv with
Ablation study Here, test the performance of the slim GNNs with and without model de- correlation on the node classification task. The results are shown in Figure 5, where "w/o dec" means that the slim GNNs directly fine-tuned without model de-correlation. We find that the slim GNNs with de-correlation perform much better than the GNNs without de-correlation, proving that correlation is still present when self-supervised GNNs are directly fine-tuned.
Parameter analysis In order to quantify the number of parameters of the self-supervised GNN reduced by SLIDE, taking GraphMAE and GRACE as an example, we report the num- ber of parameters of our proposed SLIDE and "FT" for fine-tuning. As can be seen in Fig- ure 6, we observe that the parameters of "2- half" GNNs are significantly reduced. In partic- ular, on Ogbn-arxiv with GraphMAE, the num- ber of parameters for fine-tuning is reduced by 70.1%. 1%. More details about the number of pa- rameters are provided in B.
FT
IDE
(a) GraphMAR
(b) GRACE
Figure 6: The number of parameters several datasets with GraphMAE and GRACE
8

---

5 Related Work
Graph self-supervised learning Self-supervised methods on graphs can be naturally divided into contrastive and generative domains according to objective designs and model architectures [9. 27- 29]. Graph Contrastive Learning (GCL) has shown its outstanding ability in unsupervised setting, and many studies have been proposed [1. 4, 5, 8]. On the other hand, Generative self-supervised learning [6. 2] aims to recover missing parts of the input data. Among them, methods which have emerged in the last two years [3, 7] have significantly enhanced the performance of generative meth- ods, resulting in competitive performance on downstream tasks and attracting much attention. De- spite the remarkable achievements of these methods, the issue of model redundancy in these self- supervised GNNs remains unexplored the current research landscape.
Model redundancy recent years, researchers have investigated redundancy in several pre- trained model architectures for different domains. Among them, in [30], researchers dissect two pre-trained models, BERT [31] and XLNet [32], studying how much redundancy they exhibit by using feature selection to choose the subset of neurons. In [33], researchers dissected several pre- trained visual models and randomly removed neurons of the penultimate layer in proportion, proving that redundancy exists in the penultimate layer. In [34], researchers find that many layers of LLMs exhibit high similarity. By removing some of the layers of large language models (LLMs). LLMs can still maintain good performance, proving that model redundancy exists in LLMs. Graph Neural Networks (GNNs) [35-37] have been widely applied in recent years and there are some studies fo- cusing on graph sparsification and graph lottery ticket [14, 15]. Graph sparsification approximates graph to a sparse graph by reducing the number of edges instead of parameters. And graph lottery ticket reduces parameters in networks systematically, not randomly. However, the study of model redundancy in self-supervised GNNs remains largely unexplored.
Pre-training and fine-tuning Traditional pre-training and fine- fine-tuning paradigms mainly include "linear probe" and "full fine-tune". The former faces the challenge insufficient performance, while the latter requires high computational cost and memory. In recent years, several Parameter-Efficiiee Fine-Tuning (PEFT) methods have been introduced to address these issues. Among them, Low Rank Adaptation (LoRA) [38] alters the fine-tuning phase by keeping the original model parameters frozen and introducing modifications to a separate, smaller set of parameters. These changes are then incorporated into the original parameters. On the other hand, Adapter Tuning [39] adds new modules, called adapters, between the layers of a pre-trained model. The parameters from the pre- training phase are frozen, and smaller set of additional parameters is introduced for the new task. common feature of these methods is the addition a small number of additional parameters to the complete model for fine-tuming. The focus of this paper orthogonal to these methods, it aims to fine-tune the model under the condition of reduced parameters. In this paper, we provide a unique perspective the pre-training and fine-tuning paradigm and contribute to the ongoing exploration of effective fine-tuning strategies.
6 Conclusion and Broader Impacts
In this paper, we make an exploration of model redundancy in self-supervised pre-trained GNNs. We find out that model redundancy in self-supervised GNNs exists both neuron level and layer level, which deepens our understanding of self-supervised GNNs. We then propose a novel pre- training and fine-tuning paradigm, SLIDE, which achieves better performance with fewer number of parameters for fine-tuning. Our experiments validate the effectiveness of SLIDE.
Limitations and broader impact Although we discover that the graph self-supervised learning models are highly redundant neuron and layer levels and deepen our understanding of self- supervised GNNs, a potential limitation is that some theoretical foundations are still lacking. Our findings hold great potential to improve current graph self-supervised learning models and may provide valuable guideline for the pruning or the sparsity of GNNs. In the future, we will further understand self-supervised GNNs from the perspective of model redundancy by theoretical analysis. Beyond that, we do not expect any immediate negative impact on society.

---

Acknowledgments and Disclosure of Funding
This work is supported in part by the National Natural Science Foundation of China (No. 62322203.
62172052).
References
[1] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning- In ICML Workshop Graph Representation Learning and Beyond, 2020. URL http::/taaaee httppp///aa.wv...//aab//////////
04131.
[2] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv: 1021661110200 1.0300, 2016.
[3] Zhenyu Hou, Xiao Liu, Yukuo Cen. Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self supervisd masked graph autoencoders. In Proceedings of the 28th ACM SICKDD Conference on Knowledge Discovery and Data Mining, pages 594 604, 2022.
[4] Petar Veličkovié, William Fodus, William L Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv: 180, 2018.
[5] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International conference on machine learning, pages 4116-4126. PMLR, 2020.
[6] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre- training of graph neural networks. In Proceedings the 26th ACM SIGKDD international conference knowledge discovery & data mining, pages 1857-1867, 2020.
[7] Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. What's behind the mask: Understanding masked graph modeling for graph autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1268 1279, 2023.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems, 34:76-89
2021.
[9] Yuning You. Tianlong Chen, Yongduo Sui. Ting Chen, Zhangyang Wang. and Yang Shen. Graph con- trastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823.. 2020.
[10] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2069-2080, 2021.
[11] Yifei Zhang, Hao Zhu, Zixing Song. Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11289-11 297, 2023.
[12] Nian Liu, Xiao Wang, Deyu Bo, Chuan Shi, and Jian Pei. Revisiting graph contrastive learning from the perspective of graph spectrum. Advances in Neural Information Processing Systems, 35:2972-2983, 2022.
[13] Ruijia Wang, Xiao Wang, Chuan Shi, and Le Song. Uncovering the structural fairness in graph contrastive learning. Advances in neural information processing systems, 35:32465-3222 2022.
[14] Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International conference machine learning, pages 1695- 1706. PMLR, 2021.
[15] Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International Conference Machine Learning, pages 11 1458-11468 PMLR, 2020.
[16] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski. and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:18J 1.00688 2018.
[17] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi Rad. Col- lective classification in network data. Al magazine, 29(3):3399, 2008.
10

---

[18] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.
[19] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[20] Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jacgul Choo, and Scong Joon Oh. Learning diabdd representations with biased representations. In International Conference on Machine Learning, pages 528-539. PMLR, 2020.
[21] Xingxuan Zhang. Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5372-5382, 2021.
[22] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bemhard Schölkopf. Mcasuring statistical depen- dence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
[23] Haoyang Li, Xin Wang. Ziwei Zhang, and Wenwu Zhu. Ood-gnn: of distribution generalized graph neural network. IEEE Transactions on Knowledge and Data Engineering, 2022.
[24] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[25] Zhu Li, Jean-Francois Ton, Dino Oglic, and Dino Sejdinovic. Towards unified analysis of random fourier features. International conference on machine learning, pages 3905-392 3914. PMLR, 2019.
[26] Chih- Crihh-Chng Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):122,, 2011.
[27] Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint mxXiv:198.000 0100000000 2019.
[28] Jiezhong Qiu, Qihin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1 1150-1160, 2020.
[29] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovee. Graph convolutional policy net- work for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
[30] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov, Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.0400 2020.
[31] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18X1v:1810004805, 2018.
[32] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le.. XI- net: Generalized autoregressive pretraining for language understanding. Advances neural information processing systems, 32, 2019.
[33] Vedant Nanda, Till Speicher, John Dickerson, Krishna Gummadi, Soheil Feizi, and Adrian Weller. Dif- fused redundancy in pre-trained representations. Advances in Neural Information Processing Systems, 36, 2024.
[34] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint wxXiv22403.38552 2024.
[35] Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
[36] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 17 2017.
11

---

[37] Zhongying Zhao, Zhan Yang, Chao Li. Qingtian Zeng, Weili Guan, and MengChu Zhou. Dual feature interaction-based graph convolutional network. IEEE Trunsactions Knowledge and Data Engineering. 35(9):0019-9030, 2022.
[38] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0968: 2021.
[39] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. crrrmmteer-fffccien transfer learning for nlp. In Interna- tional conference on machine learning, pages 2790-2700 2799. PMLR, 2019.
[40] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 3365-1374 2015.
[41] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 37073113,,,7 2013.
12

---

A
More Experiences about Model Redundancy
Table 6: The performance of different neuron removal methods on three datasets with MaskGAE on link prediction tasks.
Dataset
Metric
Original
Half
Quarter
AUC
96.7
96.3
93.8
Cora
AP
96.2
96.2
94.0
Change-Paraie Param
49.9
4 74.9
AUC
97.8
97.1
95.5
CiteScer
AP
98.1
97.4
96.3
Change-Paree Param
50.0
4
AUC
98.8
98.3
97.4
PubMed
AP
98.7
98.2
96.8
Change- Pargge
49.8
4. 74.8
Table The performance of different neuron removal methods on four datasets with GraphMAE on graph classification tasks.
Dataset
Metric
Original
Half
Quarter
2-Original
2-Half
2-Quarter
ACC
87.6
85.4
84,5
85.0
84,9
83.6
MUTAG
Change-Paree Param
72.1
J 91.5
1. 64.6
1 70.0
72.7
ACC
75.3
74.4
73.5
75.3
75.3
IMDB-B
Change-Param
71.1
4 10.88
4 14.2
21.2
ACC
52.1
50.7
48.6
52.1
$2.0
51.9
IMDB-M
Change-Paue Param
73.2
J. 12.2
3.37.4
L 46.77
51.4
ACC
88.2
85.9
82.5
88.0
87.9
REDDIT-B
Change-Param
69.7
J. 89.8
+ 13.2
19.8
To demonstrate that model redundancy exists across a broader spectrum of graph learning tasks, we conduct experiments on GraphMAE, effective for graph classification, and MaskGAE which excels link prediction. For graph classification, we conduct experiments on benchmarks: MUTAG, IMDB-B, IMDB-M, REDDIT-B [40]. The results in Table 6 7 indicate that model redundancy exists across a wide range of tasks.
B
More Details about Model Redundancy
B.1
GNN Parameters and Linear Parameters
Table 8: More details about paramters with different neuron removal methods for GraphMAF, where the parameters GNN is not fine-tunable while the parameters in Linear fine-tunable.
Dataset
Parameters
Original
Half
Quarter
2-Original
2-Half
2-Quarter
GNN
998,914
433,922
200,578
867,074
801,154
Cora
Linear
3,591
1,799
903
1,799
903
GNN
2.161,154
1,015,042
491,138
2,029.314
1,963,394
CiteSeer
Lincar
3,078
1,542
774
1,542
774
GNN
1,566,722
521,218
195,074
1,040,898
777,986
PubMed
Lincar
3,075
1,539
1,539
771
GNN
1,821,698
648,706
258,818
1,294,850
1,031,426
Photo
Linear
8,200
4,104
2,056
4,104
2056
GNN
1,844,226
659,970
264,450
1,317,378
1,053,954
Computers
Lincar
10,250
5,130
2,570
5,130
2,570
GNN
2,243,587
597,507
167,683
1,189,890
663,042
399,618
arXiv
Linear
41,000
20,520
10,280
41,000
20,520
10280
13

---

Table 9: More details about paramters with different neuron removal methods for GRACE
Dataset
Parameters
Original
Half
Quarter
2-Half
2-Quarter
GNN
400,000
191,808
93,856
383,552
375,328
Cora
Linear
903
455
455
231
GNN
2,027,777
981,121
482,369
1,962,113
1,929,281
CiteSeer
Linear
1542
774
390
774
390
GNN
387,840
161,152
72,384
322,176
289,344
PubMed
Linear
771
387
195
387
195
GNN
1,288,705
513,281
223,873
1,026,305
895,105
Photo
Linear
4104
2056
1032
2056
1032
GNN
1,311,232
524,544
229,504
1,048,832
917,632
Computers
Linear
5130
2570
1290
2570
1290
Tables 8 and Table record the number of GNN parameters and Linear parameters for different ways of removing neurons. We find out that proper removal of neurons still maintains decent performance, we show in Section 2, which means that there is lot of model redundancy in the original GNNs. Note that we train the same number of epochs for the different ways of reducing neurons to save model training time, and the gap in model performance will be smaller, i.c. there will still be more model redundancy, if training time is more.
B.2 Hyper-paraaeteer for GraphMAE and GRACE
For datasets where the original pre-training framework has been tested, we use the hyper-parametie from the official code, while for the other datasets, we obtain the hyper-parameters ourselves by testing on these datasets.
For GraphMAE, we obtain the Computers by ourselves. For both datasets, linear probes are trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.0005 while GNNs are pre-trained with a learning rate of 0.001, weight decay of O, hidden number of 1024, head number of 4, layer number of 2, mask rate of 0.5, drop edge rate of 0.5 and epoch number of 1000.
of pre- pre-training on Amazon- Photo and Amazon-
For GRACE, obtain the ppermpprraeteess of all linear probes all datasets by ourselves because it trains a LIBSVM classifier in the official code while we obtain the the nyper-parameter of pre- training Amazon- Photo and Amazon- -ooruuerss by ourselves. Here we list the typer-parametery for tre-tr models and linear probes used in our experiments:
For Cora, the linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9, epoch number of 1000 and weight decay of
For Citeseer, the linear probe is trained using Adam with a learning rate of 0.5, momentum of 0.9, epoch number of 1000 and weight decay of O.
For Pubmed, the linear probe is trained using Adam with learning rate of 0.05, momentum of 0.9, epoch number of 500 and weight decay of O.
For Photo, the linear probe trained using Adam with a learning rate 0.05, momentum of 0.9, epoch number of 500 and weight decay of O whlie GNN is pre-trained with a learning rate of 0.001, weight decay of O, hidden number of 512, layer number of and epoch number of 200.
For Computers, the linear probe is trained using Adam with learning rate 0.5, momen- tum of 0.9, epoch number of 500 and weight decay of 0 whlie GNN is pre-trained with learning rate of 0.001, weight decay of 0, hidden number of 256, layer number of and epoch number of 200.
14

---

Table 10: Dataset Statisties
Datasets
# Nodes
Edges
# Features
# Classes
Split ratio
Cora
2,708
10,556
1,433
7
140/500/1, ,000
Citeseer
3,327
9,104
3,703
120/500/1,000
Pubmed
19,717
88,648
500
60/500/1
Photo 7,650
238,162
745
10%%/1%%/80%
Computers
13,752
491,722
767
10
10%/10%/880
arXiv
16,9343
2,315,598
128
90,941/29,799/4
C Experimental Details
C.1 Datasets and Pre-training Frameworks
Here, we give some details about datasets we choose to evaluate the performance of SLIDE. As we have mentioned in Section 4, we use several citation networks and two social networks and Ogbn-arxiv datasets. Among these, the edges in citation networks (i.e. Cora, Citeseer, and Pubmed) represent the citation relationship between two papers (undirected), the node features are the bag-of- words vector of the papers, and the labels are the fields of the papers. The nodes in social networks (i.e. Amazon-Phott and Amazon (amzzonCommuters) represent the products, the edges represent whether the two products are frequently purchased together, the features represent the product reviews en- coded bag-of-words, and the labels are the predefined product categories. Oghn-arxiv captures citation relationships between computer science papers note citation relationships papers, and each paper has a dimensional feature vector obtained by averaging the embeddings of words in the title and abstract. The embeddings are obtained using Word2Vec [41]. The test is to predict 40 domains over CS.
arxiv. Nodes denote papers, edges de-
For the implementations of three pre-training frameworks, we use their original code. The sources are listed as follows:
1. GraphMAE:
http://gthbbcccm//UUUD//Guuhaaa
2. GRACE: http://gthabboommccccrc..CCCCCCC
3. MaskGAE: http://gihabbcomm/iiiioooccceccs
C.2 Additional Experiments and Analysis of SLIDE with Fine-Tuning Methods
Table 11: Orthogonality experiment of our proposed SLIDE and traditional fine-tuning methods, using LoRA as an example.
Dataset
Metric
Linear-probing
LoRA
Slim-LoRA
SLIDE-LORA
Cora
ACC
83.961am2
84.18
83.6220.0
84.26
CiteSeer
ACC
73.26+024
73.27
72.88
73.37
PubMed
ACC
80.62
80.691061
80.36
80.63
Here we examine the orthogonality of SLIDE in relation traditional fine-tuning methods. We start with the classic LoRA method [38] applied to GraphMAE, where SLIDE randomly prunes a subset of neurons to create Slim GNNs. We then introduce an additional LoRA module designed for fine- tuning. The results are shown in Table 11. Notably, "SLIDE-LoRA" can only adjust the parameters of the LoRA modules, as the Slim GNNs remain fixed. Despite this limitation, "SLIDE-LORA" enhances performance by reducing correlations among final representations, achieving slightly bet- ter results compared to using LoRA directly on Original GNNs. This supports the efficacy of our method in improving model capabilities.
15

---

C.3 Experimental Settings
Here, we provide more experimental settings about the experience about SLIDE. We obtain the saper--arameters ourselves by testing on these datasets with three frameworks except linear probing (we obtain the hyper-parameter from the official code).
Here we list the hyper- prper-parameters for full fine-tuning:
GraphMAE:
Cora:
The linear probe is trained using Adam with learning rate of 0.05, momentum of 0.9 and weight decay of le-4 whlie GNN is tuned with a learning rate of 1e-7, weight decay of O.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.02, momen- tum of 0.9 and weight decay of le-1 whlie GNN is tuned with a learning rate of 1e-6, weight decay of le-3.
Pubmed: The linear probe is trained using Adam with a learning rate of 0.05, momen- tum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay 0.
-
Photo: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-6, weight decay of O.
Computers: The linear probe is trained using Adam with a learning rate of 0.01, mo- mentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-5, weight decay of O.
Ogbn-arxiv: The linear probe is trained using Adam with learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of Se-4, weight decay of le-3.
GRACE:
Cora: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of O whlie GNN is tuned with a learning rate of 1c-7, weight decay of O.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momen- tum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 1e-8, weight decay of O.
Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momen- tum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of O.
Photo: The linear probe is trained using Adam with a learning rate of 0.02. momentum of 0.9 and weight decay of whlie GNN is tuned with a learning rate of 5e-5, weight decay of O.
Computers: The linear probe is trained using Adam with a learning rate of 0.1, mo- mentum of 0.9 and weight decay of 0 whlie GNN is tuned with learning rate of 5e-4, weight decay of O.
MaskGAE:
Cora: The linear probe is trained using Adam with learning rate of 5e-3, momentum of 0.9 and weight decay of 1e-3 whlie GNN is tuned with a learning rate of le-4, weight decay of 1e-3.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momen- tum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of le-4, weight decay of le-4.
- Pubmed: The linear probe is trained using Adam with a learning rate of 0.015, mo- mentum of 0.9 and weight decay of 5e-4 whlie GNN is tuned with learning rate of le-4, weight decay of
Photo: The linear probe is trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.01 whlie GNN tuned with a learning rate of 1e-4, weight decay of O.
16

---

Computers: The linear probe is trained using Adam with a learning rate of 5e-3, mo- mentum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of 2e-4, weight decay of O.
Ogbn-arxiv: The linear probe is trained using Adam with learning rate of 0.01, momentum of 0.9 and weight decay of whlie GNN is tuned with a learning rate of le-4, weight decay of O.
Here we list the asperpparametrs for SLIDE:
GraphMAE:
Cora: The linear probe is trained using Adam with a learning rate of 0.05, momentum of 0.9 and weight decay of le-4 whlie GNN is tuned with learning rate of 1e-7, weight decay of O.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.02, momen- tum of 0.9 and weight decay of lc-1 whlie GNN is tuned with a learning rate of 1e-6, weight decay of le-3.
Pubmed: The linear probe is trained using Adam with a learning rate of 0.05, momen- tum of 0.9 and weight decay of 0 whlie GNN is tuned with learning rate of 1e-6, weight decay of O.
-
Photo: The linear probe trained using Adam with a learning rate of 0.01, momentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with learning rate of 5e-6, weight decay of O.
-
Computers: The linear probe is trained using Adam with a learning rate of 0.01, mo- mentum of 0.9 and weight decay of 0.05 whlie GNN is tuned with a learning rate of 5e-5, weight decay of O.
- Ogbn-arxiv: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of Se-4, weight decay of le-3.
GRACE:
Cora: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of whlie GNN is tuned with a learning rate of 1e-7, weight decay of O.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momen- tum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 1e-8, weight decay of O.
Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momen- tum of 0.9 and weight decay of 0 whlie GNN is tuned with a learning rate of 1e-6, weight decay of O.
Photo: The linear probe is trained using Adam with a learning rate of 0.02. momentum of 0.9 and weight decay of whlie GNN is tuned with a learning rate of 5e-5, weight decay of O.
Computers: The linear probe is trained using Adam with a learning rate of 0.1, mo- mentum of 0.9 and weight decay of 0 whlie GNN is tuned with learning rate of 5e-4, weight decay of O.
MaskGAE:
Cora: The linear probe is trained using Adam with learning rate of 0.05, momentum of 0.9 and weight decay of le-3 whlie GNN is tuned with a learning rate of 1c-4, weight decay of 5e-3.
Citeseer: The linear probe is trained using Adam with a learning rate of 0.01, momen- tum of 0.9 and weight decay of 0.01 whlie GNN is tuned with a learning rate of 5e-4, weight decay of lc-4.
Pubmed: The linear probe is trained using Adam with a learning rate of 0.02, momen- tum of 0.9 and weight decay of 1e-3 whlie GNN is tuned with a learning rate of Ie-4, weight decay of le-2.
Photo: The linear probe is trained using Adam with a learning rate of 0.02, momentum of 0.9 and weight decay of 6e-3 whlie GNN tuned with a learning rate of 1e-4, weight decay of O.
17

---

Computers: The linear probe is trained using Adam with a learning rate of 5e-3, mo- mentum of 0.9 and weight decay of 5e-3 whlie GNN is tuned with a learning rate of le-4, weight decay of O.
Ogbn-arxiv: The linear probe is trained using Adam with learning rate of 3e-3, momentum 0.9 and weight decay of whlie GNN is tuned with a learning rate of 2e-4, weight decay of O.
And the hyper-parameters of linear probing is the same as the config file of these pre-training frame- works. The hyper-parameters ent, the parameters used to achieve optimal performance are sometimes different. And the hyper- parameters of the slim GNNs without model de-correlation are the same with SLIDE.
different sometimes because their model structures differ-
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we mention that graph self-supervised learning models are highly redundant neuron and layer levels and propose new model to fine-tune both GNNs and predictive layers.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: See Section 6.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We provide some experience about CKA scores of presentations in Section and to our best knowledge, many researches about model redundancy do not provide theory assumptions and proofs either. So I think it's difficult provide the proofs.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: See Appendix C.3.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: See Section and Section 4.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendix C.3.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Table 3 5 in 4.1 and Figure 5 in Section 4.2.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
19

---

Justification: See Section 4.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettp:://neurippe https//ccccc///tatcccccee.......
Answer: [Yes]
Justification: have read the NeurIPS Code of Ethics https: /et::://neurppppp cc/ public/Ethie esCuide lines, and I think the research conducted in the paper form, in every respect, with the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We test self-supervised GNNs on the node classification task. There is not high risk for misuse in it.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: We were unable to find the license for the assets we used.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
20


Reviews:
[{"review_id": "9gCD9QjgSp", "reviewer": "Reviewer_WSVe", "summary": "This paper studies the redundancy issue of GNNs that are pre-trained in the self-supervised manners. Examples of these methods include GraphMAE and GRACE. The first part of this paper shows that the numbers of parameters of these models can be reduced by half, while their performance wouldn’t change much (only drops to 96.2% of the original GNNs). This is a very  interesting finding. \nThe second part of the paper presents a pre-training and fine-tuning paradigm called SLIDE. It aims to fine-tune a slim GNN (a half-sampled pre-trained GNN), along with a prediction layer and weights of the nodes, so the node classification performance could be improved (comparing to other fine-tuning method).", "questions": "While I really like the first part of the work, I have the following questions:\n1.\tWhat datasets were used for pre-training these GNNs before getting the slim GNNS? \n2.\tAny possibility to provide theoretical analysis for the redundancy issue?\n3.\tDid authors run statistical tests for comparing the performance of LP, FT and SLIDE in Table 3-5?  The performance drop indicated in red color is often smaller than the std. So it is unknown if the performance difference is statistically significant.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe finding presented in the first part of the paper is very interesting. The redundancy found in pre-trained weights of GNN via self-supervised learning could help simplification of GNNs and their deployment in low-resource computing environments. \n2.\tThe empirical study of the redundancy issue in the first part is reasonable, and covers two representative graph self-supervised learning models (GraphMAE  and GRACE). Therefore, the results are meaningful and insightful. \n3.\tThe paper overall is well written, and easy to follow.", "weaknesses": "1.\tThe  study of the redundancy issue in the first part is  solely on the node classification task. Although node classification is the most common problem in graph learning, GNN pre-trained models are also widely used for link prediction and the entire graph classification. The further study of the redundancy issue for other tasks would make this work more solid.\n2.\tThe second part of the work (the SLIDE model) is a less valuable contribution. It is unclear about the motivation for “de-correlating the learned embeddings H in the fine-tuning phase, making models with fewer parameters more informative”. The “Slim GNN” preserved most of the useful weights already. Encouraging the independence of the node embeddings or not doesn’t change much the performance in downstream tasks. In Fig. 5, models without “de-correlation” only have a slight performance decrease. However, including “de-correlation” can level up the computational complexity.\n3.\tThe comparison in Fig.6 with “full fine-tune” is unfair. Why not including the  “linear probing” for comparison?  fine-tuning only the additional classifier and freezing the original GNNs). The number of parameters in “linear probing” is much less, while the performance of “linear probing” is not that bad (as shown in Table 3-5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for valuable opinions and concerns about our work. It is our obligation to describe more details and give more explanations. We really hope that these further efforts can alleviate your confusion.\n\n>[W\\#1] About the issue of redundancy\n\n**Response:** To demonstrate that redundancy exists across a broader spectrum of graph learning tasks, we conduct experiments on GraphMAE, effective for graph classification, and MaskGAE which excels in link prediction. The results indicate that model redundancy exists across a wide range of tasks.\n\nLink prediction:\n\n|Dataset|Metric|Original|Half|Quarter|\n|:------:|:----------:|:--------:|:--------:|:--------:|\n||AUC|96.7|96.3|93.8|\n|Cora|AP|96.2|96.2|94.0|\n||Change-Param|\\-|49.9|74.9|\n||AUC|97.8|97.1|95.5|\n|CiteSeer|AP|98.1|97.4|96.3|\n||Change-Param|\\-|50.0|75.0|\n||AUC|98.8|98.3|97.4|\n|PubMed|AP|98.7|98.2|96.8|\n||Change-Param|\\-|49.8|74.8|\n\nGraph classification:\n\n|Dataset|Metric|Original|Half|Quarter|2-Original|2-Half|2-Quarter|\n|:-----------:|:----------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|MUTAG|ACC|87.6|85.4|84.5|85.0|84.9|83.6|\n||Change-Param|\\-|72.1|91.5|64.6|70.0|72.7|\n|IMDB-B|ACC|75.3|74.4|73.5|\\-|75.3|75.3|\n||Change-Param|\\-|71.1|90.8|\\-|14.2|21.2|\n|IMDB-M|ACC|52.1|50.7|48.6|52.1|52.0|51.9|\n||Change-Param|\\-|73.2|92.4|37.4|46.7|51.4|\n|REDDIT-B|ACC|88.2|85.9|82.5|\\-|88.0|87.9|\n||Change-Param|\\-|69.7|89.8|\\-|13.2|19.8|\n\n>[W\\#2] About SLIDE\n\n**Response:** Thanks. We appreciate the opportunity to clarify the contributions and the motivations behind our approach.\n\nOur research has demonstrated the existence of model redundancy in graph self-supervised learning models. Even though we remove a portion of parameters, model redundancy may still occur during fine-tuning. Therefore, based on our findings, we believe that it is necessary to introduce a de-correlation module during fine-tuning. Our experiments have also shown that integrating such a module improves the performance. We observe that models without de-correlation show a noticeable decrease in performance compared to models with de-correlation by conducting statistical significance tests. Specifically, we present the p-values from experiments conducted across 6 datasets using MaskGAE. As can be seen, p-values are less than 0.01 on most of them. These results underscore the statistical significance of the performance differences between models with and without de-correlation.\n\n|Cora|CiteSeer|PubMed|Photo|Computers|arxiv|\n|:------:|:------:|:------:|:----------:|:--------------:|:--------:|\n|*0.00026|*0.013|*0.0032|*0.0013|*0.027|*0.0095|\n\nRegarding computational complexity, the de-correlation method we introduced maintains linear complexity relative to the number of nodes. This ensures that the additional computational cost is minimal compared to fine-tuning Slim GNNs without de-correlation, preserving the efficiency of the models.\n\n>[W\\#3] About comparison\n\n**Response:** Thanks.\n\nThe motivation behind Fig. 6 is to show that despite reducing the parameters, SLIDE maintains comparable performance. We show that our approach effectively reduces the parameter tuning required compared to full fine-tuning of Original GNNs.\n\nActually, in Section 2 we have already compared the performance and parameter reduction achieved by \"linear probing\" between Original GNNs and Slim GNNs. Only classifiers are fine-tuned while GNNs are frozen. This results of the comparisons are explicitly detailed in Table 1-2, which also implicitly indicate the proportion of parameters attributed to linear layers (50% and 25%). Furthermore, our experimental results indicate that despite significant parameter reduction, the performance of Slim GNNs does not exhibit a noticeable degradation. \n\n>[Q\\#1] About datasets\n\n**Response:** As mentioned in Section 2, we pretrain Original GNNs on six datasets (Cora, CiteSeer, PubMed, Amazon-Photo, Amazon-Computers and Ogbn-arxiv), and obtain several kinds of Slim GNNs.\n\n>[Q\\#2] About theoretical analysis\n\n**Response:** Thanks.\n\nAs highlighted in the conclusion, we agree with you and recognize the importance of theoretical analysis in enhancing our understanding. Theoretical analysis of model redundancy represents a significant avenue for future research. Enhancing the theoretical foundations of model redundancy will be crucial for advancing our methodologies. \n\nWe could potentially explore theoretical models from several angles. For instance, we could theoretically analyze model redundancy from the perspectives of feature decorrelation. Investigating causal mechanisms within GNN architectures could also offer formal insights into how redundant features affect model predictions. Additionally, developing mathematical formulations to quantify redundant features could offer intuitive insights and further optimize GNN efficiency.\n\nWe appreciate your insightful question and recognize theoretical analysis as a critical next step in advancing our understanding and methodologies for addressing redundancy within graph self-supervised learning models.\n\n>[Q\\#3] About statistical tests\n\n**Response:** We have performed significance tests and calculated p-values to evaluate the performance differences between our method and previous approaches. The p-values for most of comparisons are less than 0.05, indicating that the performance differences are statistically significant. As an example, we present significance test results for MaskGAE in the table below. \n\n||Cora|CiteSeer|PubMed|Photo|Computers|Arxiv|\n|----|:------:|:------:|:------:|:----------:|:--------------:|:-------:|\n|LP|*0.00003|*0.00043|0.42|*0.00002|*0.0000003|*0.011|\n|FT|*0.0035|*0.0077|*0.035|*0.046|*0.0086|0.91|\n\nObserving the results, out of 12 comparisons, 10 pass the significance test with p-values less than 0.05. One exception is observed in the comparison between SLIDE and FT on Ogbn-arxiv, where SLIDE achieves comparable performance to FT despite using parameters that are 67% fewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QQrBbEaQWA", "reviewer": "Reviewer_nfbW", "summary": "This paper is the first to uncover that graph self-supervised models exhibit high model redundancy at both neuron and layer levels, providing two key perspectives for graph pre-training and fine-tuning framework.\nThis paper proposes SLIDE to achieve a pre-training and fine-tuning paradigm with fewer parameters and better performance on the downstream task.\nThe authors conduct comprehensive experiments, showing superior performance.", "questions": "See weakness section", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, this paper is in general of good quality that it is well organized and in general clearly written. The motivation is clear and strong.\nThe problem under investigation is an interesting problem and the work offers some important discovers and results for this problem. \nThe proposed method is simple, effective, and well-motivated with excellent performance. Some results are particularly impressive (e.g., even randomly reducing 40% of parameters, the improvement is still 0.24% and 0.27%)", "weaknesses": "The task evaluated in this paper is only node classification. I’m curious what about other tasks? It might be important to check the correlation between the model redundancy problem and node classification, or the problem is task-independent?\n The authors point out that the proposed model is orthogonal to other fine tuning methods. A detailed discussion or more experimental results would be a plus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude for the thoughtful review and constructive feedback. Your thorough evaluation has undoubtedly contributed to enhancing the robustness of our findings.\n\n>[W\\#1]  The task evaluated in this paper is only node classification. I’m curious what about other tasks? It might be important to check the correlation between the model redundancy problem and node classification, or the problem is task-independent? \n\n**Response:** Thanks for your question.\n\nWe choose node classification as a representative task for evaluating model redundancy in graph self-supervised learning due to its classic and widely studied nature in graph learning. However, the issue of model redundancy is not limited to node classification. We further conduct similar experiments on other graph learning tasks, i.e., graph classification and link prediction, using GraphMAE and MaskGAE, respectively. The results are consistent with those observed in node classification, demonstrating that the problem of model redundancy is indeed pervasive across different graph self-supervised learning tasks. \n\nWe appreciate your interest in this aspect and hope that these additional results provide a comprehensive understanding of the issue across various tasks.\n\n**Link prediction:**\n\n| Dataset  |    Metric    |  Original  |    Half    |  Quarter   |\n| :------: | :----------: | :--------: | :--------: | :--------: |\n|          |     AUC      | 96.66±0.08 | 96.27±0.06 | 93.80±0.20 |\n|   Cora   |      AP      | 96.21±0.17 | 96.21±0.06 | 93.99±0.22 |\n|          | Change-Param |     \\-     |   49.94    |   74.90    |\n|          |     AUC      | 97.84±0.12 | 97.12±0.11 | 95.52±0.06 |\n| CiteSeer |      AP      | 98.06±0.10 | 97.37±0.07 | 96.25±0.04 |\n|          | Change-Param |     \\-     |   49.97    |   74.96    |\n|          |     AUC      | 98.79±0.02 | 98.31±0.03 | 97.38±0.04 |\n|  PubMed  |      AP      | 98.71±0.03 | 98.15±0.06 | 96.81±0.06 |\n|          | Change-Param |     \\-     |   49.84    |   74.76    |\n\n**Graph classification:**\n\n|    Dataset    |    Metric    |  Original  |    Half    |  Quarter   | 2-Original |   2-Half   | 2-Quarter  |\n| :-----------: | :----------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n|     MUTAG     |     ACC      | 87.56±0.43 | 85.36±0.42 | 84.51±0.61 | 85.02±0.92 | 84.92±1.00 | 83.64±1.13 |\n|               | Change-Param |     \\-     |   72.08    |   91.53    |   64.61    |   69.99    |   72.68    |\n|  IMDB-BINARY  |     ACC      | 75.32±0.56 | 74.40±0.64 | 73.54±0.36 |     \\-     | 75.34±0.54 | 75.28±0.65 |\n|               | Change-Param |     \\-     |   71.11    |   90.83    |     \\-     |   14.16    |   21.23    |\n|  IMDB-MULTI   |     ACC      | 52.12±0.42 | 50.71±0.55 | 48.55±0.60 | 52.05±0.53 | 52.01±0.69 | 51.91±0.72 |\n|               | Change-Param |     \\-     |   73.23    |   92.42    |   37.38    |   46.73    |   51.41    |\n| REDDIT-BINARY |     ACC      | 88.24±0.12 | 85.93±0.61 | 82.47±0.98 |     \\-     | 88.03±0.19 | 87.93±0.21 |\n|               | Change-Param |     \\-     |   69.70    |   89.78    |     \\-     |   13.21    |   19.82    |\n\n>[W\\#2]  The authors point out that the proposed model is orthogonal to other fine tuning methods. A detailed discussion or more experimental results would be a plus. \n\n**Response:** Thanks for your thoughtful feedback and interest in our research. \n\nIn our study, we have explored how SLIDE differs from previous fine-tuning approaches. Specifically, to demonstrate the orthogonality of our method with traditional fine-tuning approaches, we employ the following approach: we started with the classic fine-tuning method LoRA on GraphMAE, where our SLIDE model randomly prunes a subset of neurons to create frozen Slim GNNs. Subsequently, we introduce an additional LoRA module specifically designed for fine-tuning. To further validate our approach, we apply the de-correlation methods to adjust the LoRA module, aiming to reduce the correlation between embeddings within the Slim GNNs integrated with LoRA modules. The results are shown in the table, where \"Slim-LoRA\" denotes direct fine-tuning of Slim GNNs using LoRA and \"SLIDE-LoRA\" is our method. \n\n| Dataset  | Metric | Linear-probing |       LoRA        | Slim-LoRA  |    SLIDE-LoRA     |\n| :------: | :----: | :------------: | :---------------: | :--------: | :---------------: |\n|   Cora   |  ACC   |   83.96±0.12   | 84.18±0.34 | 83.62±0.29 |  **84.26±0.43**   |\n| CiteSeer |  ACC   |   73.26±0.24   | 73.27±0.36 | 72.88±0.51 |  **73.37±0.57**   |\n|  PubMed  |  ACC   |   80.62±0.17   |  **80.69±0.61**   | 80.36±0.63 | 80.63±0.65 |\n\nSLIDE's reduction of model parameters enables fine-tuning of the entire model, highlighting a distinct advantage. However, in SLIDE-LoRA which applies SLIDE based on LoRA, the parameters of Slim GNNs cannot be adjusted; only the parameters of LoRA modules can be modified. This limitation impacts SLIDE-LoRA's performance. Nevertheless, LoRA serves as a coarse method for adjusting model parameters, enabling SLIDE-LoRA to enhance model performance by reducing correlations among final representations. We achieve slightly superior results compared to using LoRA directly on Original GNNs, underscoring the efficacy of SLIDE-LoRA in enhancing model capabilities. The experiment provides empirical evidence supporting the feasibility and efficacy of our method in enhancing model performance and the orthogonality of our method with traditional fine-tuning approaches.\n\nOnce again, we sincerely appreciate your valuable feedback and hope that these insights clarify the distinctiveness and effectiveness of SLIDE in the context of fine-tuning methodologies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4hrCiJ7eT", "reviewer": "Reviewer_oLUe", "summary": "This paper studies the redundancy in graph self-supervised learning models. The authors discover that even randomly removing a number of parameters, the performance of graph self-supervised learning models is still comparable, revealing the redundancy problem. Then the authors propose to simultaneously fine-tune the graph self-supervised learning models and the prediction layers, and the effectiveness are well demonstrated on various benchmarks.", "questions": "Please refer to the weakness above.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Discovery of Redundancy: The paper identifies an intriguing phenomenon of redundancy within graph self-supervised learning models. This finding provides valuable insights for future research directions, particularly in the full fine-tuning of these models. Clear Organization: The paper is well-organized and logical, featuring high-quality tables and figures that effectively support the presented data and analyses. Strong Performance: The performance of the proposed model is robust, showing significant improvements when compared to baseline models.", "weaknesses": "Some technique details in the paper are not clearly introduced, leaving several critical aspects insufficiently explained. For example, the rationale behind the use of Random Fourier Features (RFF) is not adequately addressed. It would be beneficial to provide more in-depth explanations on why RFF was chosen for this context and how it contributes to the model's performance. Additionally, the meaning and significance of Equation 2 are not thoroughly discussed, leaving readers uncertain about its role and impact within the overall framework. The paper also lacks an analysis of the learned weights 𝑊 which is crucial for understanding the model's learning dynamics and interpretability. Including such an analysis could offer valuable insights into how the model processes and prioritizes different features. Moreover, the inclusion of more experimental results would enhance the robustness of the findings. By providing a more comprehensive set of experiments, the paper could demonstrate the consistency and generalizability of the proposed approach across different scenarios and datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your insightful feedback and constructive suggestions. Your thorough review has significantly strengthened the quality of our manuscript.\n\n>[W\\#1] About RFF\n\n**Response:** Thanks for your good question about Random Fourier Features (RFF).\n\nDeep neural networks exhibit complex dependencies between their features, where simply removing linear correlations is insufficient. A direct approach to address this challenge involves mapping features to a high-dimensional space using kernel methods. However, kernel mapping expands the original feature map to an infinite dimension, rendering it impractical to compute correlations between dimensions. Recognizing the advantageous properties of Random Fourier Features (RFF) in approximating kernel functions and measuring feature independence, SLIDE employs RFF to project original features into the function space of Random Fourier Features. Then, we eliminate the linear correlations among new features, thereby removing both linear and non-linear correlations among the original features.\n\nWithout RFF, as shown in Equation 4, the regularization mechanism would degrade, focusing only on linear correlations. This highlights the critical role of RFF in enabling the model to effectively capture and model non-linear relationships. This capability significantly enhances the model's capacity to learn intricate data patterns, which is particularly advantageous in tasks such as node classification.\n\n>[W\\#2] About Equation 2\n\n**Response:** \n$\\hat{C}\\_{H\\_{\\*,i},H\\_{\\*,j}}^W=\\frac{1}{N_{tr}-1}\\sum_{n=1}^{N_{tr}}[(w_nu(H_{n,i})-\\frac{1}{N_{tr}}\\sum_{m=1}^{N_{tr}}w_mu(H_{m,i}))^T\\\\ \\cdot (w_nv(H_{n,j})-\\frac{1}{N_{tr}}\\sum_{m=1}^{N_{tr}}w_mv(H_{m,j}))]$\n\nFor the meaning of Equation 2, $H_{\\*, i}$ and $H_{\\*, j}$ mean the i-th and j-th dimension of node embeddings. $W$ means the learnable weights of nodes in the input graph data, and $w_n$ means the weight of the n-th nodes in the training set which has $N_{tr}$ nodes. $\\hat{C}\\_{H\\_{\\*,i},H\\_{\\*,j}}^{W}$ is the partial cross-covariance matrix. What's more, $u$ and $v$ are the random Fourier features function space. Overall, Equation 2 aims to obtain the partial cross-covariance matrix between different dimensions of embeddings through node weighting.\nAnd for the significance of Equation 2, it facilitates the minimization of correlations across different dimensions of embeddings by learning the weights W associated with nodes. This approach enables the framework to effectively manage and optimize relationships among different dimensions of embeddings, thereby reducing the model redundancy and enhancing the model's performance. \n\n>[W\\#3] About Weights W\n\n**Response:** Thanks for your valuable suggestions. \n\nUsing MaskGAE on the Cora dataset, we analyze the learned weights W with respect to node degrees. We observe that nodes with smaller degrees tend to have larger weights. Across multiple epochs and runs, we identify the top five nodes with the highest weights for each epoch (referred to as \"big answer\"). The 8th node consistently appears prominently, representing about 10% of occurrences, while over half of the nodes rarely appear in the \"big answer\". These findings highlight important nodes in the dataset, suggesting further investigation into their significance.\n\nOur hypothesis links these observations to the graph's underlying structure. We examine the relationship between node degrees and weights by comparing the average degree of nodes in the training set with those of the top ten nodes most and least frequent in the \"big answer\". Nodes with degrees ≤ 4 average 125.0222 occurrences, while those ≥ 10 average 37.8723 occurrences, reinforcing our hypothesis. We sincerely thank for this insightful question, which has prompted further exploration into an intriguing aspect of our analysis.\n\n>[W\\#4] About the consistency and generalizability of the proposed approach\n\n**Response:** Thank you for your valuable feedback and suggestion.\n\nIn our study, we conduct experiments to investigate model redundancy in graph self-supervised learning models on the classical node classification tasks. These preliminary experiments lay the foundation for our findings. We acknowledge the importance of expanding our analysis to include more diverse graph learning tasks, and we have conducted initial experiments about model redundancy in graph self-supervised learning models on graph classification and link prediction tasks. The results are shown below, illustrating that model redundancy in graph self-supervised learning extends across a broad spectrum of graph learning tasks. This will allow us to further validate and generalize the effectiveness of our approach across different scenarios and datasets. We appreciate your insights and look forward to incorporating them into our future work.\n\n**Link prediction:**\n\n|Dataset|Metric|Original|Half|Quarter|\n|:------:|:----------:|:--------:|:--------:|:--------:|\n||AUC| 96.66 | 96.27 | 93.80 |\n|Cora|AP|96.21|96.21|93.99|\n||Change-Param |\\-|49.94|74.90|\n||AUC|97.84|97.12|95.52|\n|CiteSeer|AP|98.06|97.37|96.25|\n||Change-Param |\\-|49.97|74.96|\n||AUC|98.79|98.31|97.38|\n|PubMed|AP|98.71|98.15|96.81|\n||Change-Param|\\-|49.84|74.76|\n\n**Graph classification:**\n\n|Dataset|Metric|Original|Half|Quarter|2-Original|2-Half|2-Quarter|\n|:-----------:|:----------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|MUTAG|ACC|87.56|85.36|84.51|85.02|84.92|83.64|\n||Change-Param|\\-|72.08|91.53|64.61|69.99|72.68|\n|IMDB-BINARY|ACC|75.32|74.40|73.54|\\-|75.34|75.28|\n||Change-Param|\\-|71.11|90.83|\\-|14.16|21.23|\n|IMDB-MULTI|ACC|52.12|50.71|48.55|52.05|52.01|51.91|\n||Change-Param|\\-|73.23|92.42|37.38|46.73|51.41|\n|REDDIT-BINARY|ACC|88.24|85.93|82.47|\\-|88.03|87.93|\n||Change-Param|\\-|69.70|89.78|\\-|13.21|19.82|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqV3hopETI", "reviewer": "Reviewer_1uus", "summary": "This paper presents new insights on graph self-supervised learning models. Namely, the parameters, as well as the learned representations, of graph self-supervised learning models are highly redundant. The paper also proposes a novel pre-training and fine-tuning paradigm, SLIDE, which achieves better performance with fewer number of parameters for fine-tuning. The experimental results are also remarkable, e.g., the improvements can be also achieved even with a random reduction of parameters.", "questions": "see weakness", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Deepening our understanding of graph self-supervised learning models is an important endeavor, given the popularity of these models. This paper is aligned with research lines and proposes some important discovers on redundancy in terms of both neuron and layer levels.\n- The paper is overall well-organized and clearly written. I enjoy reading it.\n- The experiments are clearly designed and well-executed, further demonstrating the value of this work.\n\nOverall, this paper is a valuable contribution to the field and deserves a wide audience for its discovers and insights.", "weaknesses": "I think it would be valuable to explain more on the empirical results. I in particular wonder why “Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than FT\".\nIs there any (mathematical) definition on the model redundancy? It is better to verify that the proposed model does reduce the model redundancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for all the comments and it is a great honor for us for your enjoying our paper. We have addressed all your questions below and hope they have clarified all confusion you had about our work.\n\n> [W\\#1] I in particular wonder why “Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than FT\". \n\n**Response:** \n\nIndeed, reducing the number of parameters can lead to a slight performance decrease in self-supervised GNNs, specifically when applying FT to Slim GNNs compared to Original GNNs. However, as highlighted in Section 2, we identify strong correlations among different dimensions of embeddings, emphasizing the necessity of introducing de-correlation techniques to enhance the informativeness of graph self-supervised learning models. Therefore, the improved performance of SLIDE despite having fewer parameters is attributed to the incorporation of decorrelation methods. This underscores the effectiveness of our approach in mitigating model redundancy and enhancing model performance. Moreover, in Section 4.2 of our paper we conduct ablation experiments on the de-correlation method, validating our findings in Section 2 and demonstrating its critical importance within SLIDE. \n\n> [W\\#2] Is there any (mathematical) definition on the model redundancy? It is better to verify that the proposed model does reduce the model redundancy.\n\n**Response:** We appreciate and thank you for your insightful question. \n\nThe redundancy of the model currently lacks a mathematical definition, but can be roughly measured using some alternative indicators. Typically, we evaluate the model redundancy by assessing the performance of the models and the correlations within the representations. In our paper, we highlight the beneficial performance of the SLIDE models, which suggests a reduction in model redundancy. As for correlations, we conduct layer-level and neuron-level correlation analyses on embeddings from both Original GNN and the SLIDE models, and these results are presented in the table below. To simplify the evaluation process, we analyze representations of randomly sampled nodes and we use MaskGAE as an example. At the layer level, we assess the correlations between representations of adjacent layers. At the neuron level, we calculate correlations between representations represented by different subsets of neurons within a single layer. All these results are computed using CKA scores. Furthermore, as mentioned in the Conclusion, we recognize the importance of supplementing theoretical analyses for model redundancy, which is a focus of our future work. We aim to provide a detailed mathematical definition of model redundancy in our future endeavors. Once again, thank you for your insightful questions and your appreciation of our work!\n\n**Layer Level:**\n\n|    Module     |    Metric     |  Cora  | CiteSeer | PubMed |\n| :-----------: | :-----------: | :----: | :------: | :----: |\n|               |  data-layer1  | 0.3939 |  0.3131  | 0.5141 |\n| Original GNNs | layer1-layer2 | 0.9471 |  0.9509  | 0.9405 |\n|               |    average    | 0.6705 |  0.6320  | 0.7273 |\n|               |  data-layer1  | 0.3834 |  0.2680  | 0.4881 |\n|     SLIDE     | layer1-layer2 | 0.8622 |  0.8413  | 0.8945 |\n|               |    average    | 0.6228 |  0.5546  | 0.6963 |\n\n**Neuron Level:**\n\n|    Module     | Metric  |  Cora  | CiteSeer | PubMed |\n| :-----------: | :-----: | :----: | :------: | :----: |\n|               | layer1  | 0.6957 |  0.7567  | 0.7596 |\n| Original GNNs | layer2  | 0.7566 |  0.7772  | 0.7652 |\n|               | average | 0.7262 |  0.7670  | 0.7624 |\n|               | layer1  | 0.6768 |  0.7462  | 0.7394 |\n|     SLIDE     | layer2  | 0.5641 |  0.5637  | 0.7056 |\n|               | average | 0.6204 |  0.6550  | 0.7225 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All four reviewers agreed this paper should be accepted: it identifies an important problem, the paper is clearly written, and the proposed method is simple and effective. The paper will likely provide useful insights for future work in graph self-supervised learning methods. This is a clear accept. Authors: you've already responded to the detailed reviewer feedback, please make any changes to your manuscript to further improve the paper based on the feedback from reviewers. The paper will make a great contribution to the conference!
Decision: Accept (poster)