Uncovering Capabilities of Model Pruning in Graph Contrastive  Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 07:18:30

Abstract: Graph contrastive learning has achieved great success in pre-training graph neural networks without ground-truth labels. 
Leading graph contrastive learning follows the classical scheme of contrastive learning, forcing model to identify the essential information from augmented views.
However, general augmented views are produced via random corruption or learning, which inevitably leads to semantics alteration. 
Although domain knowledge guided augmentations alleviate this issue, the generated views are domain specific and undermine the generalization.
In this work, motivated by the firm representation ability of sparse model from pruning, we reformulate the problem of graph contrastive learning via contrasting different model versions rather than augmented views.
We first theoretically reveal the superiority of model pruning in contrast to data augmentations.
In practice, we take original graph as input and dynamically generate a perturbed graph encoder to contrast with the original encoder by pruning its transformation weights. 
Furthermore, considering the integrity of node embedding in our method, we are capable of developing a local contrastive loss to tackle the hard negative samples that disturb the model training.
We extensively validate our method on various benchmarks regarding graph classification via unsupervised and transfer learning. Compared to the state-of-the-art (SOTA) works, better performance can always be obtained by the proposed method.

Main paper: 
 related works: 
 Graph contrastive learning has been broadly adopted for tasks without ground-truth labels, such as graph classification [24, 51] , node classification [8] , and link prediction [54] . Here, we devote our attention to contrastive learning for graph classification, the most relevant topic of this work.Great success has been achieved by graph contrastive learning in self-supervised graph representation learning. Among various research, the study of the contrastive view is a key issue in graph contrastive learning. Recently, based on the data augmentation on images, numerous works have explored the feasibility of augmentations on graphs [24, 36, [50] [51] [52] . While gratifying, the proposed data augmentations via random perturbation or learning suffer from structural damage and noisy information [36, 51] . To tackle this issue, several works have attempted to preserve the graph semantic structure by resolving profound domain knowledge into augmentations, such as MoCL [35] , KCL [7] , GraphMVP [25] and 3D-Infomax [32] However, this domain knowledge only exists on molecules, which significantly limits the generality. Moreover, beyond the general contrastive learning framework, DGCL [23] disentangles the graph encoder, and OEPG [47] explores the semantic structure of datasets. Although they present excellent performance, they still rely on the graph augmentations and thus are orthogonal to these works that keep the general contrastive learning framework; put differently, other models can work with the framework of DGCL and OEPG to produce superior performance. Among recent works, SimGRACE [44] preserves semantics by disturbing the model weight with Gaussian noise. However, the introduced noise is data-agnostic, which could degenerate the model performance when the actual data distribution goes beyond the Gaussian distribution and explain the sub-optimal performance of SimGRACE in the experiment section.In the early stage, pruning is generally a technique to improve model efficiency, which aims to shrink model size at surprisingly little sacrifice of model performance [20] , and various pruning techniques have been proposed and effective for that goal [12, 15, 22] . Recently, besides the inherent function of model compression, several works explored its deeper connection with model generalization [9, 53] . Moreover, the capability of pruning on model memorization has also been validated with long-tail distribution dataset [18] . In this paper, we particularly investigate contrastive learning via model pruning for graph representation learning. Note that, model pruning is first employed in this work to address the issue of semantics alteration caused by data augmentation. 
notations and preliminaries: 
 Before the elaboration of LAMP, here, some preliminary concepts and notations are given. Let E and V be the sets of edges and vertices, a simple graph can be formally written as 𝐺 = (V, E). Graph representation learning. Generally, GNNs based on a message-passing scheme serve as the graph encoder [10] . A GNN learns an embedding ℎ 𝑣 ∈ R for each node and a vector ℎ 𝐺 ∈ R is produced by a READOUT function for graph 𝐺. For an 𝐿-layer GNN, each node vector is decorated with the 𝐿-hop information from its neighbors. The hidden vector of node 𝑣 in layer 𝑐 can be obtained by:EQUATIONwhere𝑓 (𝑐 )𝑈 aims to update each node vector in current layer, 𝑓 (𝑐 ) 𝑀 is the designed function for message-passing on graphs, the firstorder neighbor nodes of 𝑣 is represented as 𝑁 (𝑣), and ℎ(𝑐 )𝑣 denotes the hidden feature of 𝑣 in the 𝑐-th layer. After 𝐿 iterations, the entire graph representation can be written asEQUATIONwhere 𝑓 𝑅 pools the final set of node representations and is generally a summation or averaging function.Graph contrastive learning. A typical unsupervised model via contrastive learning takes two views from one graph as input, and the two views are produced by two data augmentation operators and serve as a positive pair. At the phase for pre-training, a GNNbased encoder is used for structural information modeling of the input views, and a projection head is further employed to embed the two views into the same feature space for contrast. Output feature vectors ℎ 1 𝑖 and ℎ 2 𝑖 from the same graph are expected to identify themselves from the others. Thus, the NT-Xent loss [4] is adopted to achieve this goal via maximizing the consensus of a positive pair:EQUATION)where 𝑁 is the batch size, 𝜏 controls the temperature parameter, and 𝑠𝑖𝑚(ℎ 1 , ℎ 2 ) generally refers to a cosine similarity functionℎ 1⊤ ℎ 2 | |ℎ 1 | | • | |ℎ 2 | | .In particular, there are two types of negative pairs; put differently, ℎ 1 𝑖 can pair with all ℎ 2 𝑗 , and ℎ 2 𝑖 can pair with all ℎ 1 𝑗 . The mutual information maximization principle. Graph contrastive learning leverages the principle of mutual information maximization (InfoMax) to enhance the correspondence between a graph representation and its corresponding views from various augmentations. The graph representation ℎ 𝐺 is supposed to contain the feature underlying 𝐺, because the representation is expected to distinguish the graph 𝐺 from others within the same batch. The principle of mutual information maximization can formally beEQUATIONwhere P 𝐺 denotes the distribution defined over the graph 𝐺 and 𝐼 (•) refers to the mutual information. 
methodology: 
 In this section, by turning the attention from data augmentation to model pruning, we bring about the proposed graph contrastive learning framework, termed LAMP. Before the detailed description, we first give the motivation from the quantification of structural damage caused by general graph augmentation.In previous works regarding graph contrastive learning, data augmentation is a general technique to help the graph encoder identify the essential information of input graphs [8, 49, 51] . Despite various forms, they are mostly built upon the concept of topology corruption, such as node dropping, edge perturbation, and learnable graph generation. Although these works have shown some extent of effectiveness on the actual tasks, the underlying structural damage remains, which inevitably leads to semantics alteration and undermines the model performance. Now, based on structural entropy [21] , a metric for structural information, we are the first to give the quantitative illustration of structural damage caused by data augmentations. Give a simple graph 𝐺 = (V, E), the structural information underlying 𝐺 can be measured by:EQUATIONin which 𝑣𝑜𝑙 (V) denotes the sum of all node degrees and 𝑔 𝑣 is the node degree of 𝑣. Among various forms of data augmentation proposed in previous works, here, we give the quantitative illustration of structural damage by three ubiquitous rules from GraphCL [51] , including subgraph, edge perturbation, and node dropping. In particular, the augmentation strength is consistent with the setting in GraphCL, that is, 20%. The structural damage of graph 𝐺 is measured by the percent change of its structural entropy. Formally, given any data augmentation function 𝑡, the percent change of structural entropy will beEQUATIONThe quantitative illustrations of structural damage caused by three data augmentation rules on a social network dataset (i.e., REDDIT-BINARY) and a bioinformatic dataset (i.e., MUTAG) are shown in Figure 2 . 1 The effect of structural damage varies with the augmentation rules. Specifically, node dropping and subgraph lead to different degrees of structural damage, and the information loss caused by subgraph is the largest and generally over 50%. Besides the simple information loss, the structure damage composition of edge perturbation is more complex; put differently, in light of the distribution of REDDIT-BINARY, edge perturbation even introduces external data noise with the additional edges, which further interferes with the model from learning the actual structural information. Therefore, we naturally wonder: Can we design a more advanced model with effective contrastive pairs without structural damage? Next, to tackle our expectations, we present the superiority of model pruning in contrast to data augmentation.Through theoretical analysis, in this subsection, we present the property of graph encoder trained from model pruning. Statement 1 in Theorem 4.1 guarantees a lower bound of the mutual information between the learned representations and the labels of the downstream task; put differently, the learned essential information via the sparse encoder is more than views from augmentations.Statement 2 in Theorem 4.1 suggests that the training performance of LAMP is better than the models based on augmentations in the architecture of graph contrastive learning.Proof. Suppose G is a set of graphs. According to the definition, 𝑓 * = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑓 𝐼 (𝑓 (𝐺); 𝐺), 𝑓 * should be injective. Given some graph 𝐺 ∈ G, 𝐺 ⇒ 𝑓 * (𝐺) is an injective deterministic mapping. Thus, for any random variable 𝑄, 𝐼 (𝑓 * (𝐺); 𝑄) = 𝐼 (𝐺; 𝑄).(EQUATIONWhen there is 𝑄 = 𝑌 , we will have,EQUATION1 The quantitative illustrations of other six datasets in unsupervised learning are shown in Appendix A.Moreover, in light of theoretically proof in [27] , a depth-two network can be approximated by pruning a random-weighted subnetwork 𝑓 * 𝑝 as follows:supEQUATIONAccordingly, we have the following proof:EQUATIONwhere (𝑎) is because of the arbitrariness of 𝜖 and the continuity of the entropy 𝐻 (•). Meanwhile, because 𝜖 can be arbitrarily small, we can achieveEQUATIONNow, introducing the data processing inequality [37] for data augmentation,EQUATIONCombining above equations, we have the statement 1: EQUATIONThen, according to the data processing inequality [37] , we move forward to which concludes the proof of the statement 2. □ vs.(a) Same structure but different node features.vs.(b) Different structure and different node features.Figure 3 : Illustration of hard negative samples. Via contrasting the graph embeddings, the pre-trained model is hard to distinguish this two kinds of graphs.With the superiority of pruning in graph contrastive learning, now, we move from theory to practice. Instead of producing contrastive pairs via data augmentation, here, we give our model design via model pruning. Figure 1 general pictures the workflow of LAMP.As can be seen, LAMP sticks to the twin-tower architecture of GraphCL [51] , while gets rid of its trail-and-error data augmentation rules. Especially, LAMP feeds the original graph into the dense graph encoder and its pruned version to contrast their embeddings. Formally, let 𝐺 = (A, X) be the input graph, A is the adjacency matrix, and X is the initial node features. For the 𝑙-th layer of graph encoder, the node representations of 𝐺 via Equation 1 will be:EQUATIONEQUATIONwhere 𝑊 (𝑙 ) is the weight matrix of update function 𝑓 In practice, we prune the dense graph encoder according to a predefined ratio 𝛾 with the given pruning strategy, that is, the weight values of𝑊 𝑙 will be masked if they are ranked below 𝛾. In particular, the sparse degree of the graph encoder can be controlled by tuning the pruning ratio 𝛾, which meets various demands of different datasets. Detailed discussion about the pruning ratio is conducted in the ablation study. Moreover, to avoid drastic gradient changes and save computations, the wight matrix will be pruned at the beginning of each epoch. Since the sparse encoder is always obtained and updated from the latest dense version, the two branches will coevolve during training. Remark. To avoid possible confusion, we emphasize that the adoption of pruning is not aimed at enhancing model efficiency, but rather at boosting the performance of contrastive learning. Furthermore, our approach in this study is not tied to a specific pruning strategy, but is compatible with any given pruning methods. Specifically, we have implemented our method using two distinct pruning techniques: magnitude pruning [12] and soft filter pruning [15] , which are referred to as LAMP-Mag and LAMP-Soft, respectively. Projection head. After obtaining the graph representations via global pooling, a projection head 𝑔(•) is employed to cast the representations to another feature space for contrasting. In the framework of LAMP, a two-layer perceptron is also employed to produce the graph final representations 𝑧 1 𝐺 and 𝑧 2 𝐺 , for each mini-batch do 5:EQUATIONObtain the node representations H 1 V and H 2 V through the two graph encoders 𝑓 (•) and 𝑓 𝑝 (•);Obtain the graph representations Z 1 and Z 2 through Equation 2 and projection head 𝑔(•); Update weights:W ′ 𝑓 ← W 𝑓 -𝑟 ▽ W 𝑓 L W ′ 𝑔 ← W 𝑔 -𝑟 ▽ W 𝑔 L 9:end for 10: end forAmong the research on contrastive learning, hard negative samples are quite ubiquitous and have great potential to improve model performance [29] . However, little attention has been drawn to the hard negative samples within current contrastive learning for graph classification. As shown in Figure 3 , there are two kinds of hard negative samples. In Figure 3a , this negative pair has the same topology but different features. Let ℎ 𝑐𝑜𝑙𝑜𝑟 (𝑔 for green, 𝑏 for blue) denote the node features, the graph representations would be similar after pooling as 𝑓 𝑅 (4 × ℎ 𝑔 + 𝑓 𝑏 ) = 𝑓 𝑅 (4 × ℎ 𝑔 + 𝑓 𝑏 ). In Figure 3b , this negative pair has different topology and node features. Let ℎ 𝑏 = 2ℎ 𝑔 , the graph representations would be also similar through summation or maximization pooling function. Given the common design of current contrastive learning for graph classification, the model is encouraged to enlarge the distance of negative pairs via graph representations, which may fail with the two kinds of negative samples in Figure 3 and deteriorate model performance.Despite the hardness of distinguishing the two kinds of negative samples from the graph representations, we may be able to spot some opportunities from the local features. For example, the green nodes could be effortlessly identified from the blue nodes in Figure 3a , and the read nodes also have obvious differences compared to the green and blue nodes in Figure 3b . Therefore, in order to cultivate the ability of contrastive learning to tackle hard negative samples, we propose a local contrastive loss with the node embeddings. Local Contrastive Loss. Because each node embedding matrix after graph encoder contains the full set of nodes in original graph, here, we propose a local contrastive loss to enhance graph learning from the node level. Our critical insight is that current models lack the ability to separate the hard negative samples via graph representations, while a contrastive angle based on node representations would be helpful in this scene. Specifically, as the general NT-Xent loss enforces the dissimilarity among different graphs, we move forward to generalizing this dissimilarity to nodes. Formally, the EQUATIONwhere G refers to a training batch. Note that the local contrastive loss also has two kinds of negative pairs. In particular, since the node batch size varies among different datasets, LAMP will randomly sample a sub-node set of size 𝑁 𝑠 for large dataset to avoid high computation costs. Considering the batch node size in actual model training, 𝑁 𝑠 is fixed to 5,000 in the experiment setting.Remark. Although the local contrastive loss for hard negative samples is relatively simple in its presentation, it is not easy to implement when the two graphs of a positive pair have different structures due to corruption. This limitation may account for why previous graph contrastive learning models have overlooked this approach and consequently produced sub-optimal performance. Currently, we have presented the main components of the proposed LAMP that aims to help graph contrastive learning free from structural damage caused by data augmentation and enhance model training facing hard negative samples. The final objective function of LAMP for pre-training isEQUATIONwhere L 𝐺 is the NT-Xent loss, 𝛼 controls the loss weight. Algorithm 1 reveals the pre-training procedure of LAMP. 
experiments: 
 In this section, we are devoted to evaluating LAMP with extensive experiments 2 . Under the setting of unsupervised and transfer learning, LAMP empirically shows its superiority compared to the SOTA competitors. Ablation studies regarding hyper-parameters are further conducted to make an in-depth analysis.Datasets. For unsupervised learning, various benchmarks are adopted from TUDataset [28] , including COLLAB, REDDIT-BINARY, REDDIT-MULTI-5K, IMDB-BINARY, NCI1, MUTAG, PROTEINS and DD. For transfer learning, ZINC15 [33] dataset is adopted for pre-training.In particular, a subset with two million unlabeled molecular graphs are sampled from the ZINC15. We employ the eight ubiquitous benchmarks from the MoleculeNet dataset [42] Then, we finetune and evaluate the model on MoleculeNet dataset using the scaffold split scheme [3] . Configuration. To keep in line with GraphCL [51] , the same GNN architectures are employed with their original hyper-parameters under individual experiment settings. Specifically, in unsupervised learning, GIN [45] with 32 hidden units and 3 layers is set up. In transfer learning, GIN is used with 5 layers and 300 hidden dimensions. The pruning ratio for sparse encoder is selected from 5% to 95% with a step of 5%. For local contrastive loss balance, 𝛼 is tuned among {0.01, 0.1, 1, 10, 100}. Hyper-parameters are selected by the grid search on the validation sets. Additional details are shown in the Appendix C. Pruning strategy. To demonstrate the compatibility of our framework, LAMP, with a variety of pruning methodologies, in this study, we adopt two distinct pruning techniques for the implementation of our method: magnitude pruning [12] and soft filter pruning [15] , denoted as LAMP-Mag and LAMP-Soft, respectively.Unsupervised learning. The baselines in unsupervised learning have three categories. The first set is three SOTA kernel-based methods that include GL [31] , WL [30] , and DGK [46] . The second set is four heuristic self-supervised methods, including node2vec [11] , sub2vec [1], graph2vec [2] , and InfoGraph [34] . The final category comes from the graph contrastive learning domain, including MV-GRL [13] , GraphCL [51] , AD-GCL [36] , JOAO [50] , AutoGCL [49] , RGCL [24] , SimGRACE [44] , SEGA [41] and GCS [40] .The classification accuracies of LAMP against the SOTA competitors are shown in Table 1 , and a significant performance improvement from the disappearance of the data augmentation can be witnessed as opposed to the baselines. Before the specific performance description of LAMP, here, we first glance at SimGRACE, which first attempts to transfer the contrastive attention from data augmentation to model perturbation via introducing Gaussian noise to model weights. Although SimGRACE reveals its effectiveness on various datasets, the introduced Gaussian noise still degenerates the model performance; put differently, SimGRACE does not defeat the models with data augmentations.We now present a comprehensive analysis of the superior performance of LAMP. As indicated by the final column for average rank, LAMP-Mag and LAMP-Soft secure the top two positions and exhibit the highest average accuracies among all baseline models. Notably, as evidenced by the column for average accuracy, LAMP-Soft surpasses the second-best model (i.e., GCS) with an accuracy improvement of 1.43%. Specifically, LAMP-Soft achieves the best performance on six out of eight benchmarks, while still maintaining the second-best performance on the remaining two datasets. Although LAMP-Mag is not superior as LAMP-Soft, it achieves the best performance on the NCI1 dataset and the second-best on three other datasets. Thus, we can conclude that model pruning may be a more promising direction for graph contrastive learning. Transfer learning. Baseline methods in transfer learning include EdgePred, AttrMsking, ContexPred [17] , Infomax [39] , JOAO [50] , GraphCL [51] , AD-GCL [36] , LP-Info [52] , GraphMAE [16] , Auto-GCL [49] , GraphMVP [25] , RGCL [24] , D-SLA [19] , SimGRACE [44] , SEGA [41] and GCS [40] . A model without pre-train, termed 'No Pre-Train', is also adopted for comparison.The results of LAMP, along with baselines under the setting of transfer learning on eight benchmarks, are shown in Table 2 . In summary, the proposed models, namely LAMP-Mag and LAMP-Soft, demonstrate superior efficacy in comparison to preceding studies, as evidenced by the average ROC-AUC and ranking. Specifically, LAMP-Soft outperforms all other models on six out of the eight benchmarks, while securing second place on the remaining two datasets. LAMP-Mag exhibits optimal performance on the SIDER benchmark and obtains the second highest performance on five out of the eight benchmarks. In particular, LAMP-Soft obtains a 2.27% performance gain in terms of average ROC-AUC compared to the best baselines (i.e., GCS with a 74.72% average ROC-AUC) and over 10% performance gain compared to the model trained from scratch.Here, we make an in-depth analysis about the performance of LAMP under the setting of unsupervised learning. In particular, the magnitude pruning is adopted for ablation study. Effectiveness of local contrastive loss. Besides the contrastive angle from the ubiquitous NT-Xent loss for unsupervised learning, we take another step to help model be capable of handling hard negative samples from the perspective of nodes. As shown in Table 3 , LAMP-Mag obtains better performance with a 1.16% average accuracy gain when decorated with the proposed local contrastive loss, which suggests the effectiveness of the proposed local loss in addressing hard negative samples and improving model performance.In particular, LAMP-Mag w/o L 𝐿𝑜𝑐𝑎𝑙𝐶 has an average accuracy of 76.83% on eight benchmarks, which not only outperforms current SOTAs for view generation but also defeats SimGRACE which disturbs the model weight with Gaussian noise. Thus, we can reaffirm the effectiveness of pruning on graph contrastive learning.Sensitivity regarding pruning ratio. The pruning ratio controls the information that the sparse encoder captures; thus, a proper ratio would help the model identify the essential structure of input graphs. As shown in Figure 4 , the datasets in the first row prefer a lower pruning ratio, while the other datasets would like a sparser encoder. Moreover, regardless of the optimal pruning ratio, the performances of all datasets quickly deteriorate when the sparsity goes above 75%, due to limited capacity. Sensitivity regarding loss balance. As we have validated the effectiveness of the proposed local contrastive loss, we further inspect the influence of its hyper-parameter (i.e., 𝛼) on model performance.The unsupervised results of LAMP-Mag with candidate 𝛼 on eight benchmarks are shown in Figure 5 . As can be seen, DD and NCI1 show a positive correlation with 𝛼, while MUTAG is not sensitive to the choice of 𝛼, which is consistent with the stable performance of MUTAG w/o L 𝐿𝑜𝑐𝑎𝑙𝐶 in Table 3 . The other datasets show a trade-off within the given selections, and generally obtain the best performance with 𝛼 around 1. 
conclusion: 
 In this work, we reformulate the problem of graph contrastive learning from the angle of model compression. To avoid the loss of semantics caused by data augmentation, we present a novel method based on model pruning, termed LAMP, rather than relying on profound domain knowledge. Before the empirical validation, we theoretically explain the superiority of model pruning compared to data augmentation. Extensive experiments under unsupervised and transfer learning show that LAMP suppresses the current SOTA methods based on data augmentations. An automatic pruning ratio and more advanced pruning strategies shed light on the future research direction. 
: 
 graph contrastive learning, such as GraphCL [51] , AD-GCL [36] and RGCL [24] .In general, graph contrastive learning sticks to the twin-tower architecture of contrastive learning [4, 43] , in which two augmented views are generated from the input graph, and the model loss (i.e., NT-Xent loss [4] ) maximizes the mutual information between the two output embeddings of the two augmented views. With this design, the trained model is capable of capturing the essential information of graphs [26] . Moreover, researchers have found that the quality of views influences the performance of contrastive learning models [38] . Therefore, plenty of research efforts have been devoted to the generation of effective views that lead to better performance for graphs through the data augmentation [36, 50] .Contrastive data augmentation on graphs presents a significant challenge compared to images due to the complex structural information and diverse contexts inherent in graph data [8] . Inspecting prior studies on graph contrastive learning, we can systematize the two common paradigms for view generation. The first category is the random or learnable data corruption, such as the four types of general augmentations (node dropping, edge perturbation, attribute masking, and subgraph) in GraphCL [51] and learnable edge dropping in AD-GCL [36] . Despite the effectiveness of these graph views on various tasks, the proposed data augmentations via random corruption or learning suffer from structural damage and artificially introduced noise, which could alter the fundamental property of input graphs. Based on predefined sub-structure substitution rules [35] or contrasted with 3-dimensional geometric views [25, 32] , the second way is to integrate the domain knowledge to alleviate the issue of semantic alteration with data corruption. However, the fusion of domain knowledge inevitably undermines the generalization of the pre-trained model to other domains [24] .To fully leverage the potential of contrastive learning in the graph domain, it is desirable to develop a graph contrastive learning model that can preserve semantic information while remaining independent of domain-specific knowledge. To address this challenge, we shift our focus from contrastive views to the graph encoder within the contrastive learning framework. Drawing inspiration from model compression techniques, we note that the performance of sparse sub-networks could be comparable to their complete versions [5, 12] , suggesting that pruning may be a viable approach for graph contrastive learning. Accordingly, we introduce a novel framework called LAMP that enables Graph Contrastive Learning viA Model Pruning to remedy these issues in previous works.The framework of LAMP is shown in Figure 1 . LAMP takes the original graph as model input to prevent semantic information alteration from graph corruption. While fostering the model with the ability to identify the essential information, we employ pruning [12, 15] for perturbation. In particular, since the pruned graph  
 
 Appendix: The code of LAMP will be public after acceptance. 


Decision: The authors have adequately addressed the concerns raised by most reviewers and received three positive ratings and one negative rating. As reviewer WcYq did not provide a final rating after the rebuttal, most reviewers agree that this paper is ready for publication. Therefore, the paper is recommended for acceptance.

Reviews: review: paper_summary: paper_summary: This work presents a view of graph pruning based pre-training by graph contrastive learning via contrasting different model versions rather than augmented views. The authors provide interesting theoretical proof to show the effectiveness of model pruning. Then, they use the original graph as input and dynamically generate a perturbed graph encoder to contrast with the original encoder. In addition, a local contrastive loss is proposed to handle the hard negative samples during the training. The proposed method achieves SOTA performance on various graph benchmarks.
strength_weakness: strength_weakness: 1, The idea of using pruning-based method for graph contrastive learning  is interesting and novel to me. 

2, The overall writing is good and easy to follow. 

3, I check the proof part and find it interesting and convincing, which motivates the research using model pruning. 

4, The proposed methods achieve stronger performance .
limitations: limitations: 1, The comparison with previous augmentation-based graph contrastive pre-training is week. I think more detailed comparisons are needed.

2, The scope problems. I wonder whether the submission meets the requirements of MM since there is only graph data and no multi-modal data involved.  

3, Several presentation issues:
	Missing error analysis.
	Missing training and inference cost analysis.
paper_summary: paper_summary: The paper presents a framework for enhancing graph contrastive learning by focusing on model pruning rather than traditional data augmentation techniques. The authors argue that typical data augmentations can alter the semantic integrity of the data and propose using model pruning as an alternative to preserve semantic information and improve model generalization.

The proposed methods operate by dynamically pruning a graph neural network to create contrasting versions of the model, which are then used to generate embeddings for contrastive learning. This approach not only preserves the original graph's semantic information but also enhances the learning process by addressing the issue of hard negative samples through a local contrastive loss.

The paper provides theoretical insights and conducts extensive experiments to demonstrate LAMP's strength over the other baselines on the graph classification tasks.
strength_weakness: strength_weakness: The shift from traditional data augmentation to model pruning is novel in the context of graph contrastive learning.

Extensive experiments are conducted to show the proposed methods' superior performance over the baselines across numerous benchmarks on the graph classification tasks for both unsupervised/transfer learning, making the generalizability of LAMP convincing.

The paper provides theoretical insights for its proposed method.

The introduction of the local contrastive loss has a reasonable performance boost.
limitations: limitations: The proof of theorem 4.1 does not seem to be rigorous enough. For example, Equation 9 relies on the conclusion given by https://arxiv.org/pdf/2002.00585, but in that paper, they made many assumptions such as over-parameterization, has the paper checked those assumptions carefully?

The performance gain looks over-rely on the local contrastive loss. A direct comparison between Table 3 and Table 1 shows that if we remove the local contrastive loss, the performance gain will be very marginal.

Combining Figures 4 and 5, it looks like a good pruning ratio is relatively important for datasets like PROTEIN, RED-M5K, and IMDB-B, whereas datasets like NCI1 and DD rely on local contrastive loss more. However, the paper does not give any qualitative or qualitative analysis about the nature of those datasets to explain why that is the case.

It would be great to include some hard-negative examples like those in Figure 3 from the experiment datasets as an ablation study.

The related work section is missing some citations for other graph contrastive learning works.
paper_summary: paper_summary: This paper introduces LAMP (Learning via Model Pruning), a novel framework that applies model pruning to graph contrastive learning, deviating from traditional data augmentation methods to preserve semantic integrity. By dynamically generating a perturbed graph encoder and contrasting it with the original through the pruning of transformation weights, LAMP avoids domain-specific dependencies and improves handling of hard negative samples with a local contrastive loss. Extensive testing demonstrates LAMP's superiority over state-of-the-art models in both unsupervised and transfer learning for graph classification.
strength_weakness: strength_weakness: 1. The paper benefits from clear, concise, and well-structured writing. The authors effectively communicate complex ideas and methodologies in a manner that is accessible to readers, which enhances the paper's overall impact and readability. This clarity extends to the presentation of experimental results and theoretical concepts, making the novel approach understandable and the results compelling.
2. The framework is not only theoretically grounded with detailed analyses of its superiority over data augmentation but is also rigorously tested in practical scenarios. This dual approach strengthens the credibility and reliability of the proposed method.
limitations: limitations: 1. While the paper successfully demonstrates the advantages of LAMP over many baselines, it does not extensively explore or compare against other novel augmentation alternatives in the same experimental setting that could potentially offer similar benefits. This limits the scope of the study and may overlook other viable methods in graph contrastive learning.
2. The paper focuses primarily on graph classification tasks, which, while providing a robust test bed for the proposed methodology, does not discuss the potential implications or adaptability of the LAMP framework to other types of graph-related tasks (like node classification and link prediction) or broader application domains. This could be seen as a missed opportunity to highlight the versatility of the approach.
paper_summary: paper_summary: This paper introduces LAMP, a novel graph contrastive learning framework that uses model pruning instead of data augmentation to enhance generalization and avoid semantic alteration. This approach theoretically and practically outperforms existing methods on various graph classification benchmarks, both in unsupervised and transfer learning scenarios. Additionally, LAMP incorporates a local contrastive loss to effectively handle hard negative samples, demonstrating significant improvements over state-of-the-art models in terms of accuracy and robustness.
strength_weakness: strength_weakness: Strengths:

1.	The idea of using model pruning rather than traditional data augmentation for graph contrastive learning is novel. This concept not only preserves the semantic integrity of the data but also broadens the applicability of graph neural networks without relying on domain-specific augmentations.

2.	The paper provides a strong theoretical basis for the proposed method. It includes a theoretical analysis demonstrating the advantages of model pruning over data augmentation in maintaining the structural and semantic properties of graphs, which adds depth and credibility to the research.

3.	The experiments are extensive and well-structured, covering a variety of benchmarks and scenarios. The results clearly demonstrate the superiority of the LAMP framework over state-of-the-art methods, offering empirical evidence that supports the theoretical claims.
limitations: limitations: 1.	This paper lacks a discussion on the scalability issues, the computational overhead of pruning strategies, and the impact of pruning on very large graphs, which might help in setting realistic expectations for its applicability.

2.	The method's success is heavily dependent on the choice of pruning techniques. Although this allows flexibility, it might also lead to inconsistencies in performance across different applications or datasets, depending on how the pruning is implemented.

3.  While the paper claims superiority over data augmentation methods, it could provide a more direct comparative analysis, demonstrating how model pruning performs in scenarios where data augmentation has traditionally excelled.