Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization
Authors: Dingshuo Chen
Zhixun Li
Yuyan Ni
Guibin Zhang
Ding Wang
Qiang Liu
Shu Wu
Jeffrey Xu Yu
Liang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic Data Pruning
Training Acceleration
Molecular Representation Learning
Primary area: other

Abstract: With the emergence of various molecular tasks and massive datasets, how to perform efficient training has become an urgent yet under-explored issue in the area. Data pruning (DP), as an oft-stated approach to saving training burdens, filters out less influential samples to form a coreset for training. However, the increasing reliance on pretrained models for molecular tasks renders traditional in-domain DP methods incompatible. Therefore, we propose a **Mol**ecular data **P**runing framework for **e**nhanced **G**eneralization (**MolPeg**), which focuses on the source-free data pruning scenario, where data pruning is applied with pretrained models. By maintaining two models with different updating paces during training, we introduce a novel scoring function to measure the informativeness of samples based on the loss discrepancy. As a plug-and-play framework, MolPeg realizes the perception of both source and target domain and consistently outperforms existing DP methods across four downstream tasks. Remarkably, it can surpass the performance obtained from full-dataset training, even when pruning up to 60-70% of the data on HIV and PCBA dataset. Our work suggests that the discovery of effective data-pruning metrics could provide a viable path to both enhanced efficiency and superior generalization in transfer learning.

Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization
Dingshuo C2 Zhixun Li Yuyan N
Guibin Zhang Ding Wang!2
Qiang Liu Shu Wul Jeffrey Xu Yu² Liang Wang!
'New Laboratory of Pattern Recognition
State Key Laboratory of Multimodal Artificial Intelligence Systems
Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences
'Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong
Academy of Mathematics and Systems Science, Chinese Academy of Sciences 'Tongji University
Primary contact: dingshuo wwwn@cripac.
Abstract
With the emergence of various molecular tasks and massive datasets, how perform efficient training has become an urgent yet under-explored issue the area. Data pruning (DP), filters out less influential samples to form a coreset for training. However, the increasing reliance pretrained models for molecular tasks renders traditional in-domain DP methods incompatible. Therefore, we propose a Molecular data Pruning framework for enhanced Generalization (MolPeg), which focuses on the source-free data pruning scenario, where data pruning is applied with pretrained models. By maintaining two models with different updating paces during training, we introduce novel scoring function to measure the informativeness of samples based on the loss discrepancy. As plug-ant aad---adyylyy framework, MolPeg realizes the perception of both source and target domain and consistently outperforms existing DP methods across four downstream tasks. Remarkably, it can surpass the performance obtained from full-dataset training, even when pruning up to 60-70% of the data on HIV and PCBA dataset. Our work suggests that the discovery of effective data-pruning metrics could provide a viable path to both enhanced efficiency and superior generalization in transfer learning.
oft-stated approach to saving training burdens,
Introduction
The research enthusiasm for developing molecular foundation models is steadily increasing [1-5], attributed to its foreseeable performance gains with ever-larger model and amounts of data, as observed neural scaling laws [6, 7] and emergence ability [8] in other domains. However, the computational and storage burdens are daunting in model training [9], hyperparameter tuning, and model architecture search [10-12]. It is therefore urgent to ask for training-efficiin molecular learning in the community.
Data pruning (DP), in a natural and simple manner, involves the selection of the most influential samples from the entire training dataset to form coreset as paragons for model training. The primary goal is to alleviate training costs by striking balance point between efficiency and performance compromise, trend in this field developing data influence functions [13-15], training dynamic metrics [16-19], and coreset selection [20-22] for lossless although typically compromised model
"Corresponinng author: Shu Wu (shu. wu@nlpr ia.ac. cn).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Data Pruning Pretrained Model
Distribution Patterns
36
MW
HIV PCOMAM2
ROC-AUC (%)
82
TPSA
80
78
-O- Dyyaaix Random
InfsBatch
50
100
150
200
253
1.0
350
403
76
Static Random
Forgetting
QED
GraNd
Kmeans
72
MolPeg (Ours)
N honds
No Prune
70
201
50%
40%
60%
Data Ratio (1 p)
Figure (Left) The performance comparison of different data pruning methods in HIV dataset under source-free data pruning setting, (Right) Distribution patterns of four important molecular features molecular weight (MW), topological polar surface area (TPSA), Quantitative Estimate of Drug-likeness (QED) and number of bonds- in PCQM4Mv2 [33] and HIV [34] dataset, which are used for pretraining and finetuning, respectively.
generalization. When it comes to molecular tasks, transfer learning, particularly the pretrain-finetune paradigm, has been regarded as the de-facto standard for enhanced training stability and superior performance [23-27]. However, existing DP methods are purposed for train-from- footfrrr---c-ccchh setting, ie., the model is randomly initialized and trained on the selected coreset. natural question arises as to whether not current DP methods remain esfective when applied with pre-truined models. Experimental analysis, as illustrated in Figure (Left), suggests a negative answer. Most existing pruning strategies exhibit inferior results relative to the performance achieved with the full dataset, even falling short of simple random pruning.
In contrast to the existing DP approaches, which focus solely on a single target domain, the incorpo- ration of pretrained model introduces an additional source domain, thereby inevitably exposing us to the challenge of distribution shift [28, 291. Unfortunately, this is especially severe in molecular tasks, owing to the limited diversity of large-scale pretraining datasets compared to the varied nature of downstream tasks. As illustrated in Figure (Right), we investigate the distribution patterns of several important molecular properties across the upstream and downstream datasets following Beaini et al. [30]. The observed disparities impede the model generalization, thus making DP with pretrained models a highly non- non-trivial task. We define this out-of-domain DP setting as source-free data pruning. It entails removing data from downstream tasks leveraging pre-traincs models while remaining agnostic to the specifics of the pre-training data.
Of particular relevance to this work are approaches that propose DP methods for transfer learning [31, 32], which also target cross-domain scenarios. Despite the promising results they achieved, these methods select pretraining samples based on downstream data distribution, which necessitates reevaluation of previously selected samples and retraining heavy models as new samples involving, undermining the goal of achieving generalization and universality in pretraining. To this end, we take a step towards designing a DP method under the source-free data pruning setting to achieve efficient and effective model training, which aligns better with practical deployment for molecular tasks.
In this work, we propose a Molecular data Pruning framework for enhanced Generalization, which we term MolPeg for brevity. The core idea of MolPeg is to achieve cross-domain perception via maintaining an online model and a reference model during training, which places emphasis on the target and source domain, respectively. Besides, we design a novel scoring function to simultaneously select easy (representative) and hard (challenging) samples by comparing the absolute discrepancy between model losses. We further take deep dive into the theoretical understanding and glean insight on its connection with the previous DP strategies. Note that our proposed MolPeg framework is generic, allowing for seamless integration of off-the-shelf pretrained models and architectures. To the best of our knowledge, this is the first work that studies how to perform data pruning for molecular learning from transfer learning perspective. Our contributions can be summarized as follows:
We analyze the challenges of efficient training in the molecular domain and formulate a tailored DP problem for transfer learning. which better aligns with the practical requirements of molecular pre-trained models.
2

---

Overall workflow
Theoretical Perspective
model
[xx, 6)
gradient
Target
score
cosiest
hardest
EMA update
[L 6) -xxx,)
Coreset
- (00 + (1 SOI
projetion
gradient
pruned area
Hard examples
Source
reference model
{xx,t)
difficulty
ranking and selection
EMA g Sidene projection
The overall framework of MolPeg. (Left) We maintain an online model and reference model with different updating paces, which focus on the target and source domain, respectively. After model inference, the samples are scored by the absolute loss discrepancy and selected in ascending order. The easiest and hardest samples are given the largest score and selected form the coreset. (Right) The selection process of Mel can be interpreted from gradient projection perspective. Samples with low projection norms (grey) are discarded, while those with high norms are kept.
Figure 2:
We propose efficient data pruning framework that can perceive both the source and target domains. can achieve lightweight and effective DP without the need for retraining, facilitating easy adaptation to varied downstream tasks. We also provide theoretical understanding of MolPeg and build its connections with existing DP strategies.
We conduct extensive experiments on 4 downstream tasks, spanning different modalities, pretrain- ing strategies, and task settings. Our method can surpass the full-dataset performance when up 60%-70% of the data is pruned, which validates the effectiveness of our approach and unlocks a door enhancing model generalization with fewer samples.
2 Preliminaries
In this section, we take a detour to revisit the traditional data pruning setting and pretrain-finctune paradigm before introducing the problem formulation of source-free data pruning.
Problem statement of traditional data pruning. Consider a learning scenario where we have a large training set denoted as D = ((24,,)) [=) consisting of input-output pairs (æ;,y), where a; E X represents the input and y; y denotes the ground-truth label corresponding to 2;- Here, X and y refer to the input and output spaces, respectively. The objective of traditional data pruning is to identify a subset D C D. that captures the most informative instances. The model trained on this subset D should yield a slightly inferior or comparative performance to the model trained on the entire training set D. Thus they need to strike balance between efficiency and performance.
Revisit on transfer learning. Given source and target domain datasets Ds and Dy. the goal of pretraining is to obtain a high-quality feature extractor ff in a supervised or unsupervised manner. While in the finetuning phase, aim to adapt the pretrained ff in conjunction with output head g to the target dataset Dy.
Considering the proficiency of molecular pre-trained models in capturing meaningful chemical spaces, their widespread usage enhancing performance across diverse molecular tasks has become commonplace. This necessitates reassessment of the conventional approach to DP within the molecular domain and, more broadly, within the field of transfer learning. Previous attempts [31, in data pruning for transfer learning primarily focus trimming upstream data, selecting samples that closely match the distribution of downstream tasks to align domain knowledge. However, this necessitates retraining the model from scratch, which is notably ill-suited for the molecular domain, where the continual influx of new molecules introduces novel functionalities and structures. To this end, propose a tailored DP problem for molecular transfer learning:
Problem formulation (Source-free data pruning). Given a target domain dataset Dyr and a pretrained feature extractor parameterized by As, we aim identify subset Dr c Dr for training, while being agnostic of the source domain dataset Ds. to maximize the model generalization.

---

3 Methodology
As with generic data pruning pipelines, the MolPeg framework is divided into stages, scoring and selection. In the first stage, we define a scoring function to measure the informativeness of samples and apply it to the training set. In the subsequent stage, given the sample scores, we rank them in ascending order and maintain the high-ranking samples for training. Note that our pruning method is dynamically performed during the training process, rather than conducted before training.
We next introduce the MolPeg framework in detail. We track the training dynamics of two models with different update paces. For each training sample, we measure the difference in loss between the two models to quantify its importance, and then make the final selection based on this metric. In the following parts, we first intuitively introduce our design of the scoring function. Then, we further explore the theoretical support behind the effectiveness of the MolPeg. The overall framework is illustrated in Figure 2.
3.1 The MolPeg framework
The design of the scoring function addresses two key issues, (1) how to achieve the perception of source and target domain and (2) how to measure the informativeness of the samples.
Cross-domain perception. Since serves as the only entry point of the source domain. During the finetuning stage, apart from online encoder undergoing gradient optimization via back propaaatagg,,n,n we further maintain reference encoder updated with exponential moving average (EMA) to perceive the cross-domain knowledge. Note that both encoders initialized by pretrained model 6g- Eo O where 0, and & denotes the parameters of online and reference model at batch step 1, respectively. They are updated as follows:
unable access the upstream dataset, the pre- pre-trained model
0,+1 = 0, - aVoL(D, 0,) & =0, + (1 --
(1)
where is the learning rate and B E [0, 1) is the pace coefficient that controls the degree of history preservation. Here D, is the selected finetuning dataset for epoch and V,L(D,, 0,) denotes the average gradient ib. Eneo, VoL(xx 0,) for short. Intuitively, We control the influence of target domain on the reference encoder via EMA. With small update pace B, the online encoder prioritizes target domain, while the reference encoder emphasizes source domain.
Informativeness mcasurement and selection. By far we explicitly represent the inaccessible source do- main knowledge with the help of the reference model, facilitating us to further quantify the informativeness of each sample in the cross- context. Our motivation for measuring the sample informativeness comes from recent work that improves the neural scaling laws [35]. They suggest that the best pruning strategy depends on the amount of initial data. When the data volume is large, retaining the hardest samples yields better pruning results than retaining the easiest ones; the conclusion is the opposite when the data volume is small. This contrasts with the conclusion that only the hardest samples should be selected [16]. From intuitive perspective, simple samples are more representative, allowing the model to adapt to downstream tasks more quickly, while hard samples are crucial for model generalization since they are considered supporting vectors near the decision boundaries. This debate highlights that in data pruning. how to perform a mixture of easy and hard samples is a critical factor. As shown in Figure 3, when 60% samples in the HIV dataset pruned, simply selecting the easiest or hardest samples leads to a performance drop in later epochs.
Comparison of Selection Criteria
85.0
82.5
ROC-AUC (%)
77.5
75.0
72.5
70.0
Easy
Hard
67.5
Both
No Prune
68.0
20
40
60
80
Epochs
Figure 3: Performance comparison of selection cri- teria on HIV dataset when pruning 40% samples.
Therefore, opt retain both easy and hard samples instead of singularly removing type. To measure the information gap between domains, we adopt both online and reference encoder to infer each sample and calculate the absolute loss discrepancy between them:
D, (x D.IL[(a, 0,) L(x, ((x.E)) 5).
(2)

---

Algorithm 1: Molecular Data Pruning for Enhanced Generalization (MolPeg)
Inputs:
D (a yr. s dataset with the score for each example (s; 1, Vs; E D);
a: learning rate; 3: EMA update pace;
data pruning ratio (p 1): T: total number of training epochs;
fo pretrained encoder parameterized by
0:
3 while 1 - 7 do
/ Get the number of remaining samples */ / Rank and Select the top-K samples for training " -(fg)))))) Vaa, yr, si) s D;; 0 A aVo((D,.0)
p p. | D. - ;
/* Scoring the samples */ / Gradient update for online model "/ 1" EMA update for reference model "/
E 00 + (1 -BE;
/ - 1+1
10 return
where D, E DT comprises the target domain data sampled for batch step r and D) E D, comprises the data selected by MolPeg. ō is not constant, but threshold determined by the pruning ratio. Specifically, the rank of 6 in the absolute loss discrepancy sequence {l_(z; 0,) L(x; 12) is 1D), ie. pruning ratiox|D,j. is easy to infer that a positive loss discrepancy, i.e. L(x. 0,) - L(x,E) &i) > O, indicates the model struggles accurately distinguish the sample, identifying it as hard one. Conversely, a negative loss discrepancy indicates that the model can easily improve its accuracy, marking it easy sample. Therefore, intuitively, of samples during the training process. By measuring the absolute value of the loss discrepancy, we keep the simplest (most representative) and the hardest (most challenging) samples, which are integrated as the most informative ones (Orange line in Figure 3). We also provide the pscudo-code of MolPeg in Algorithm 1.
dynamically assess the learning difficulty
3.2 Theoretical Understanding
In this section, we explore the theoretical underpinnings of the data selection process in MolPeg. Recall that our scoring function is defined by loss discrepancy, we further make use of Taylor expansion on the designed scoring function. Then, from the gradient perspective, i.e., the first-order expansion term, we derived the following propositions and the complete proof is provided in the Appendix E.
Assumption 1 (Slow parameter updating). Assume the learning rate is small enough, so that the parameter update A6, 0..1 6, is small for every time step, ||A0,1| s e, VI constant.
is small
Proposition 1 (Interpretation loss discrepancy). With Assumption 1, the loss discrepancy can be approximately expressed by the dot product between the data gradient and the "EMA gradient":
[(x,E, 0,) VVol(e of O(E2)
where EMA denotes E'=1 BVVV (r-)). i.e. the weighted sum of the historical gradients, which termed "EMA gradient".
It indicates that the scoring function is essentially influenced by the magnitude of the dot product between the data gradient and the EMA gradient, illustrated in Figure 2 (right). Given the EMA gradient, the size of the dot product is influenced by two factors: the norm of voL(x, 0,) and the angle between the two vectors. (i) larger norm of the current data gradient is more likely be selected, which resembles the criteria of GraNd score [19]. More connections to several well-known scoring functions are provided the appendix F. (ii) If the current gradient direction closely aligns with the (opposite) EMA gradient direction, it often indicates easy (hard) optimization of the sample, corresponding to the goal of selecting simple and hard samples in the previous analysis. Conversely, samples with gradient directions orthogonal to the EMA gradient are discarded.
5

---

In the following proposition, we examine the gradient of the selected samples and analyze simple and hard samples separately. Since the selection performed each fixed batch time step, we focus on one step of selection and omit the common time subscript 1. Note that this result involves certain simplifications and approximations, and formal version provided in the appendix.
Proposition 2 (Gradient projection interpretation of MolPeg, informal). Let D C D and D' S D denote the sets of samples for which the dot products between the data gradients and the "EMA gradient" are positive. Then, the gradient of the selected "simple" samples can be expressed as:
VoL(D .0 =Vo(("* .0) av a
EMA
a
Similarly, define D D and D D samples that have negative dol products, then
VoL(d 0) VoL(D .0) bu EMA b O.
(5)
a = O and b holds if and only if the loss discrepancy across D' and D is uniform respectively, which are uncommon scenarios.
Therefore, our data selection strategy essentially increases the weight of the (opposite) EMA gradient direction in the data gradient for easy (hard) samples. When D" predominates, indicating a majority of simple samples in the dataset, this simplified model is akin to the momentum optimization strategy. which utilizes the sum of the current data gradient and the weighted EMA gradient update the model parameters. This suggests that retaining simple samples may enhance optimization stability, allowing the model overcome saddle points and local minima [36]. However, our method differs from the momentum optimization strategy in two key aspects. Firstly, we preserve directions opposite to the EMA gradient to target hard and forgettable samples. Secondly, our EMA gradient, which records the gradient of the coreset rather than the entire set, can retain more historical information under the same update pace.
4 Experimental Settings
4.1 Datasets and tasks
To comprehensively validate the effectiveness of our proposed MolPeg, we conduct experiments on three datasets, i.e., HIV [34], PCBA [37], XUV [38] and QM9 [39], covering four types of molecular tasks. These tasks span two molccular modalities- 2D graph and 3D geometry as well as two types of supervised tasks, i.e.. classification and regression.
Given the potential issues of over-fitting and spurious correlations that may arise with limited samples when a large pruning ratio is adopted, we focus on relatively large-scale datasets containing at least 40K molecules. Below, we briefly summarize the information of the datasets. For a more detailed description and statistics of the dataset, please refer to Appendix A.
4.2 Implementation details
In this section, we provide a succinct overview of the implementation details for our experiments, including backbone models for different modalities, training details and evaluation protocols.
Backbone models. Given the two modalities involved in our experiment, we need corresponding backbone models for data modeling. Below a concise introduction to the backbone models. For more comprehensive understanding of the model architecture, please refer to the Appendix D.
For 2D graphs, we utilize the Graph Isomorphism Network (GIN) [40] as the encoder. To ensure the generalizability of our research findings, we adopt the commonly recognized experimental settings proposed by Hu et al. [41], with 300 hidden units in each layer, and a 50% dropout ratio. The number of layers is set to 5.
For 3D geometries, we employ two widely used backbone models, PaiNN [42] and SchNet [43]. as the encoders for different datasets. For SchNet, we set the hidden dimension and the number of filters in continuous-file filter convolution to 128. The interatomic distances are measured with 50 radial basis functions, and we stack interaction layers. For PaiNN, we adopt the setting with 128 hidden dimensions, 384 filters, 20 radial basis functions, and stack 3 interaction layers.

---

Table 1: The performance comparison -fate-of-the-art methods HIV and PCBA in terms of ROC- AUC (%,, 1) and Average Precision (%, T). We highlight the best-performing results in boldface. The performance difference with whole dataset training is highlighted with blue and orange, respectively.
Dataset
HIV
PCBA
Pruning Ratio %
90
80
70
60
40
20
90
80
70
60
40
20
Hard Random
77.7.24
81.1
82.4ja
84.0
85.0
14.6
18.7
21.1
23.2
25.3
26.2
CD
77.5,t.
80.9,z
81.5
p2.7
83,4
84.9 pm
14.7.no
18.0
20.8
21.9/u
25.1g12
26.0gm
Herding
63.6
72.0,ia
73.9
76.9 82.2
84.7
10.6
11.7p46
13.7,12
17.2pu
22.6jan
K-Means
61.8 68.5
74.9,ma
jm 78.5
81.4
83.7
12.8 1s8
16.7
19.6
21.4us
p4.1pa
25.8,o.
Least Confidence
79.2. 81.0ja.
22.4
82.8
83.2
85.1
14.4, 11.9
19.2
21.6
23.2,1.
p5.7,ou
26.0ga3
Entropy
78.7,6a
81.1
8
82.4
84.3
85.2
14.6.
18.4
21.4
23.2
25.5
26.7
Forgetting
80.0,s1 81.6
82.9/a2
83.8
84.7
84.9 po.3
15.3,
18.9
21:3
22.3
25.3
26.1
GraNd-4
77.5,7o
81.2
81.7
82.8
83,7
84.5 pm
14.7.m
18.4.1
21.1
22.6117
25,5jux 26.2,u.
Statie
GraNd-20
80.1 .so p2.5
83.0
83.9
84.7
84.9
15.8 jos
19.4
22.0
p3.1
25.7
26.0
DeepFool
76.8 xs 80.9
81.5
82.0
83.1p
84.6
13.9, 1P4 17.5
20.9,sa
22.2
24.9
25.9 jma
Craig
76.5
80,8
p2.5
83.8
85.0
14.5
18.7
21.3
22.9/14
25.1
26.0
Glister
80.9 42 62.3ps
83.4
84.0
84.9 joo
p5.2
15.5 ins 18.8
21.6/47
23.2pp
25.3
26.1,ao
Influence
76.5 86 80.5
81.7
p2.5
83,4
84.2
13.7 126
17.9,x
20.5
22.1
24.5
25.4
EL22N-20
79.8 s.a. p2.0
13.5
g4.0
p5.4
85.1 poio
14.7 11.
19.1
21.7 jan
22.5
25.5
26.1
DP
77.9_na
80.1
12.5
83.7
84.6
85.0ga
14.1 1122
18.2,k1
20.9
22.8
25.1 25.9 mm
Soft Random
82.3 aw
83.7 am
p4.
84.6
85.0
85.
16.1 16.1
19:2,7
21.0
22.3
24.2,z1
25.4juu
Dynamic
é-greedy
82.5 26 83.2
83.7
84.1
84.
84
85. 1o.
16.5
19.8
20.3
21.5
23.8
25.2
UCB
82.6
p3.0
13.5
83.9
84.5
84.7 po.
16.7
20.2
22.0
23.5
24.9
26.1
InfoBatch!
82.9 22 83,5 11
84.4
84.9
85.4
85.2 your
19.9 je.4
22.8,s.
24.5 j18
25.5 joa
26.8 ju.
27.0
NolPeg
83.7
84.8 jaa
85.3
85.5
86.0
85.6
20.7
23.9
25.6 25.6
26.4
26.8 n.s 27.0
Whole Dataset
85.1
26.3++0.1
To make fair comparison, remove the annealing operation in the InfoBatch, since it uses the full dataset for training at later epochs.
Training details. We adhere to the settings proposed by [41] for our experiments. In classification tasks, the dataset is randomly split, with an 90%%/0%/10% partition for training, validation and testing, respectively. In regression tasks, the QM9 dataset is divided into 10K molecules for training, 10K for validation, and another 10K for testing. The Adam optimizer [44] is employed for training with a batch size of 256. For classification tasks, the learning rate is set at 0.001 and we opt against using a scheduler. For regression tasks, we align with the original experimental settings of PaiNN and SchNet, setting the learning rate to 5 x 10-1 and incorporating a cosine annealing scheduler.
Evaluation protocols. We conduct series of experiments between model performance and varied data quantities. Specifically, we divide the pruning ratio into six proportional subsets: [20%, 40%, 60%, 70%, 80%, 90%]. and for each configuration, mean performance. For HIV datasets, performance is measured using the Area Under the ROC-Curve (ROC-AUC), while reporting the performance PCBA in terms of Average Precision (AP)-hights values in both metrics indicate better performance. When assessing quantum property predictions in the QM9 dataset, the Mean Absolute Error (MAE) is used as the performance metric, with lower values indicating better accuracy. The additional results on MUV dataset are demonstrated in Table 5.
randomly select five seeds and report the
5 Empirical Studies
5.1 Empirical analysis on classification tasks
Our empirical studies for classification tasks utilize the 2D graph modality as the input. We employ GIN as the backbone model and adopt GraphMAE [45] for model pre- tre-training on the PCQM4Mv2 dataset. For a comprehensive comparison, we select the following two groups of DP methods as
primary baselines in our experiments: static DP and dynamic DP, following [46]. The majority
of previous methods fall into the former group, from which we select 14 competitive and classic DP methods as baselines, i.c., hard random pruning, CD [47], Herding [18], K-means [35], Least
7

---

Table 2: The performance comparison state-of-the-art methods on QM9 dataset in terms of MAE (L). We highlight the best- and the -ecoon- results in boldface and underlined, respectively.
Dataset
QM9-U0 (meV)
QM9-7pve (meV)
Pruning Ratio %
90
80
70
60
40
20
90
80
70
60
40
20
Random
85.0
45.7
34.2
30.9
19.2
15.7
4.94
3.09
2.53
2.26
1.93
1.65
DP
136.0
68,5
39.8
32.3
20.8
16.1
8.56
6.29
3.62
2.36
2.05
1.68
InfoBatch MolPeg
116.0
57.0
36.4
30.1
20.4
6.26
4.61
3.22
2.34
1.91
1.64
92.4
48.2
32.4
26.1
17.7
14.3
5.40
3.18
2.51
2.24
1.86
1.62
Confidence [48], Entropy [48], Forgetting [16]. GraNd [19]. EL2N [19]. DeepFool [49], Craig [50], Glister [51], Influence [14] and DP [15]. Since dynamic pruning remains niche topic, we identify four methods, the best of our knowledge, greedy [52], UCB [52] and InfoBatch [46], with MolPeg also falling into this category. Please refer Appendix C for more detailed introduction to the baselines.
serve baselines, i.e., soft random pruning,
Performance comparison. Empirical results for DP methods are presented in Table Our systematic study suggests the following trends: (i) Dynamic DP strategies significantly oulperform static DP strategies. Soft random, as a fundamental baseline in dynamic DP, consistently outperforms the baselines of static groups across almost all pruning ratios, even surpassing some strong competitors such as Glister and GraNd. We also observe that the performance advantage of dynamic DP becomes more pronounced when the pruning ratio is relatively large. Intuitively, compared to fixing a subset for training, dynamic pruning can perceive the full dataset during training, thereby possessing larger receptive field and naturally yielding better performance. As more data samples are retained, the ability of both groups to perceive the full training set converges, leading to smaller performance differences between them. (ii) Mol achieves the state-of-the-ar performance across all proportions. On the HIV dataset, we can achieve nearly lossless pruning by removing 80% of the samples, surpassing other baseline methods significantly. Similarly, on the larger-scale PCBA dataset, we can still achieve lossless pruning by removing 60% of the data. (iii) MolPeg brings superior generalization performance compared to fine-tuning on the full dataset. For example, on the HIV dataset, we achieve an ROC-AUC performance of 86 when pruning 40% of the data, surpassing the 85.1 achieved with training on the full dataset. This indicates that appropriate data pruning can better aid model generalization given pre-trained model. However, as more downstream data is introduced, the improvement brought by our method diminishes, as shown by the 20% pruning proportion, due to introducing data samples that hinder model generalization. More empirical results on MUV dataset
Efficiency comparison. In addition to performance, time efficiency is another crucial indicator for DP. We conduct performance-efficieen
Performance-Effccienyy Comparison
comparison of var- ious DP methods on the HIV dataset at a 60% prun- ing ratio, as shown in Figure 4. We define time effi- ciency as the reciprocal of the runtime multiplied by 1000. A higher value of this metric indicates greater efficiency. We can observe that despite MolPeg ex- periencing slight efficiency loss compared to random pruning. it demonstrates superior pruning performance. Compared to the current SOTA baseline model, Info- Batch, our method achieves better model generalization with comparable efficiency. Conversely, static pruning methods incur 1.6x to 2. greater time costs than ran- dom pruning, with model performance stagnating or declining. This underscores that Mo MolPeg achieves supe- rior performance with minimal efficiency costs. Despite increased memory usage introduced by the reference model, EMA is commonly used to stabilize molecular training, which allows our method to utilize EMA-saved models without added memory overhead.
ROC-AUC (%)
Mildatth
Grati
Herding
Uecerianty
MiiPigg
Static fandon
0.150 0.175 0,200 0.225 0,250 0,275 0.300 0.325
Time Efficiency (100/sec)
Figure 4: Performance and efficiency compari- between different DP methods. Pretrained models are fine-tuned the HIV dataset at 60% pruming ratio.
8

---

HDV GraphMAE
PCEA-Graph PCBA-GriphMME
PCBA-GraphCL
ANNEE Freehium
ROC -CC. (%0
12
RDC CLALCC
movin 12
(1-iootl-p
Dem Batio (1-P)
Dhtta Rabio (1-P)
Rate P)
Figure 5: Data pruning trajectory given by downstream performance (%). Here the source models pretrained on the PCQM4Mv2 dataset with GraphMAE and GraphCL. strategies, respectively.
5.2 Results QM9 dataset
Since regression is another common type of downstream molecular task, we also present the empirical results of MolPeg on two properties using the QM9 dataset, alongside comparisons with state-of-the- art methods. To ensure a fair comparison of experimental results, we employ the commonly used 3D geometry modality for modeling. We adopt GeoSSL [53] as the pretraining strategy and PaiNN as the backbone model, following the settings outlined by Liu et al [53]. Empirical results presented in Table 2. It can be observed that MolPeg consistently outperforms other DP methods. However, all DP methods unexpectedly demonstrate inferior performance than random pruning in certain pruning ratios (80% and 90%). We speculate this phenomenon is attributed to the PCQM4Mv2 dataset used for pre-training and the QM9 dataset having close match in the distribution patterns of molecular features. Thus, any non-uniform sampling methods would lead to biased data pruning which exacerbates distribution shift and hinders domain generalization.
5.3 Sensitivity Analysis
We further conduct extensive sensitivity analysis to validate the robustness of MolPeg across different pre-training strategies, molecular modalities, pre-training datasets and hyperparameter choices. All experiments below are conducted on the HIV dataset.
Robustness evaluation across pretraining strategies. Given that MolPeg primarily targets scenarios involving pre-trained models, it is necessary to compare its robustness when applied with different pre-training strategies. Without loss of generality, we select two representative pre- pre-training strategies: generative self seff-supervised learning (SSL) and contrastive self-superviser learning, both of which dominate the field of molecular pre-training. Specifically, in addition to the results based on Graph- MAE [45] (generative SSL) presented in Table 1, we also conduct experiments based on GraphCL (contrastive SSL) [54] whose results are shown in Figure 5. We can observe that MolPeg achieves optimal performance both pre- rre-training methods across different pruning ratios. Promisingly, it demonstrates better model generalization than training on the full dataset, indicating insensitivity to pre-training strategies of our proposed framework, thus allowing for convenient plug-in application to other pre-trained models in different molecular tasks.
Robustness evaluation across modali- ties. The selection of molecular modal- ity has long been contentious issue in the field. To validate the effective- ness of MolPeg across different molecu-
Table 3: Performance with 3D modality on HIV dataset.
Pruning Ratio %
60
40
20
Random Pruning MolPeg
80. 81.9
80.8 82.3ruo
81.2 82.2 mo.s
lar modalitics, we present a comparison of pruning results using 3D geometry in the HIV dataset as shown in Table 3. We pretrain the SchNet [43] on the PCQM4Mv2 dataset, and keep other settings the same as in Section 4.2. It is evident from the results that the MolPeg framework, consistent with the conclusions drawn in Section 5.1, continues to outperform dynamic random pruning and enhance the model generalization ability. At 40% pruning ratio, MolPeg also surpasses the performance achieved with training on the full dataset. This demonstrates the robustness of our proposed DP method across molecular modalities.
Whole Dataset
81.4+1.7
Robustness evaluation across pretaining datasets. In source-free transfer learning, pretrained model is hard-encoded module, and their variations naturally lead to performance changes. Therefore,

---

it is necessary to evaluate the robustness of MolPeg when pretrained with different pre-training datasets. We conduct additional experiments on the HIV dataset using two pretrained models of varying quality, obtained from the ZINC15 [55] and QM9 datasets, respectively. Compared to the PCQM4Mv2 dataset used in the Section 5.1, these two datasets are smaller in scale and exhibit more pronounced distribution shifts, resulting in poorer pretraining quality. We observe the following trends from Table 5: (i) In the case fine-tuning on the entire dataset, models pretrained on ZINC15 and QM9 show significantly inferior performance, even worse than training from scratch, indicating poor quality of pretrained models. (ii) MolPeg still achieves the best performance with these two pretrained models. This demonstrates the robustness of MolPeg across pretraining datasets.
How choose B. Since EMA is cru- cial component of our framework, it is necessary evaluate how to choose a
Evaluation of B
F-
0.002
2-05
ROC-AUC (%)
proper B. We conduct empirical anal- ysis on the HIV dataset across three prun- ing ratios, i.e., [0.1. 0.4, 0.8], and con- sider a candidate list covering the value ranges of B: [0.001, 0.01, 0.1, 0.5, 0.9]. Intuitively, a smaller & implies a slower parameter update pace in the reference model. When 3 = 0, it signifies using a frozen pre-trained model the reference. The experimental results corresponding to the variation of are illustrated in Fig- ure Empirical results indicate that the overall performance shows only moderate sensitivity parameter change. However, typically, when B = 0.5, the model tends to achieve better performance and smaller standard deviation. Hence, for our primary experiments, we opt default 0.5.
16
2
0.1
0.4
0.5
Pruning Ratio
Figure 6: Performance bar chart different choices of hyper- parameter f on HIV dataset. The error bar is measured in stan- dard deviation and plotted in grey color.
6 Conclusion
In this work, we propose MolPeg, a novel molecular data pruning framework designed to enhance generalization without need for source domain data, thereby addressing the limitations of existing in-domain data pruning (DP) methods. Our approach leverages two models with different update paces to measure the informativeness of samples. Through extensive experiments across four downstream tasks involving both classification and regression tasks, we demonstrate that MolPeg not only achieves lossless pruning but also outperforms full dataset training in certain scenarios. This underscores the potential of MolPeg optimize training efficiency and improve the generalization of pre-trained models in the molecular domain. Our contributions highlight the importance of considering source domain information in DP methods and pave the way for more efficient and scalable training paradigms in molecular machine learning.
Broader impacts. Given that our application tasks fall within the molecular domain, improper use of methods for tasks such as molecular property prediction may result significant deviations. This could impact subsequent applications of the molecules in drug development materials design, especially in predicting properties like toxicity and stability. We recommend further experimental validation of key molecules after using the model to ensure the reliability of the results. We provide further discussions in Appendix H.
7 Acknowledgements
This work is jointly supported by National Science and Technology Major Project (20232D0120001) and National Natural Science Foundation of China (62372454).
10

---

References
[1] Oscar Méndez-Lucio, Christos Nicolaou, and Berton Earnshaw. Mole: a molecular foundation model for drug discovery. arXiv preprint arXiv:2211.0200 02657,, 2022. 1
[2] Yizhen Luo, Kai Yang, Massimo Hong, Xingyi Liu, and Zaiqing Nic. Molfm: multimodal molecular foundation model. arXiv preprint 230iv:2300.00444,, 2023.
[3] Jinho Chang and Jong Chul Ye. Bidirectional generation of structure and properties through a single molecular foundation model, Nature Communications, 1,5())2323,, 2024.
[4] Kerstin Kläser, Blažej Banaszewski, Samuel Maddrell-Mander Callum McLean, Luis Müller, Ali Parviz, Shenyang Huang, and Andrew Fitzgibbon. Minimol: A parameter-effiiiet foundation model for molecular learning. arXiv preprint arXiv:2404.1499
2024.
[5] Nathan C Frey, Ryan Soklaski, Simon Axclrod, Siddharth Samsi, Rafacl Cromez-Bombarellyy Connor W Coley, and Vijay Gadepally. Neural scaling of deep chemical models. Nature Machine Intelligence, 5(11):1297-1305 2023.
[6] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020. 1
[7] Dingshuo Chen, Yanqiao Zhu, Jieyu Zhang, Yuanqi Du. Zhixun Li. Qiang Liu, Shu Wu, and Liang Wang- Uncovering neural scaling laws in molecular representation learning. Advances Neural Information Processing Systems, 36, 2023.
[8] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Schastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[9] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[10] Babatounde Moctard Oloulade, Jianliang Gao, Jiamin Chen, Raeed Al-Sabri, and Zhenpeng Wu. Cancer drug response prediction with surrogate modeling- graph neural architecture search. Bioinformatics, 2023.
[11] Yijian Qin, Xin Wang, Ziwei Zhang, Pengtao Xie, and Wenwu Zhu. Graph neural architecture search under distribution shifts. In International Conference on Machine Learning, pages 18083-18095, PMLR, 2022.
[12] Shengli Jiang, Shiyi Qin, Reid C Van Lehn, Prasanna Balaprakash, and Victor M Zavala. Uncertainty guantification for molecular property predictions with graph neural architecture search. arXiv preprint arXiv:2307.10433 2023.
[13] Zakán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33: 14879-14890, 2020. 1, 15
[14] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017. 8
[15] Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.0933 2022. 1,
[16] Mariya Toneva, Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint wtXiv:18/..0511211 2018. 4, 8, 15, 19
Alessandro Sordoni. Remi Tachet des Combes. Adam Trischler, Yoshua Bengio, and
[17] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint /.0i::770000444 2017. 15
[18] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th annual international conference on machine learning, pages 1 1121-1128, 2009. 7
[19] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596- 20607, 2021. 5, 8, 15, 18
11

---

[20] Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. Advances neural information processing systems, 29, 2016. 1
[21] Trevor Camphell and Tamara Broderick. Automated scalable bayesian inference via hilbert coresets. Journal of Machine Learning Research, 20(15):1-38, 2019.
[22] Sungnyun Kim, Sangmin Bac, and Se- Young Yun. Coresct sampling from open-set for fine-grained self-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattem Recognifion, pages 7537-7547, 2023. 1, 16
[23] Xiaomin Fang, Lihang Liu, Jieqiong Lei. Donglong He, Shanzhuo Zhang. Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry- enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-134, 2022. 2
[24] Jinhua Zhu, Yingee Xia, Lijun Wu, Shufang Xie, Tao Qin. Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2626-2636, 2022.
[25] Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):779-2 1777992277777 2022.
[26] Haisong Gong, Qiang Liu, Shu Wu, Liang Wang. Text-guided molecule generation with difffusion language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 109-117.20 2024.
[27] Yanqiao Zhu, Dingshuo Chen, Yuanqi Du, Yingze Wang, Qiang Liu, and Shu Wu. Molecular contrastive pretraining with collaborative featurizations. Journal of Chemical Information and Modeling, 64(4):1 1122, 2024.
[28] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: graph out-of ofttt--disttibution benchmark. Advances in Neural Information Processing Systems, 20720,9-2073, 2022. 2
[29] Xin Sun, Liang Wang, Qiang Liu, Shu Wu, Zilci Wang, and Liang Wang. Dive: Subgraph disagreement for graph dut-of-dittrbbution generalization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2794-2805, 2024.
[30] Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabricla Moisescu-Parejp Oleksandr Dymov, Samuel Madddell-Manderr Callum Mcl.aan, Frederik Wenkel, Luis Müller, et al. Towards foundational models for molecular lcarning on large-scale multi-task datascts. In The Twelfth International Conference on Learning Representations, 2023. 2
[31] Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jiancheng Liu, Gaowen Liu, Mingyi Hong, Shiyu Chang, Sijia Liu, al. Selectivity drives productivity: Efficient dataset pruning for enhanced transfer learning. Advances in Neural Information Processing Systems, 36, 2023. 2,3, 16
[32] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201-34227, 2023.
2, 3, 16
[33] Maho Nakata and Tomomi Shimazaki. Pubchemqe project: large-scale first fritccpincsples electronic structure database for data-driven chemistry. Journal of chemical information and modeling. 57(6):1300 1308, 2017. 2, 15
[34] AIDS Antiviral Screen Data. 2, 6, 15
[35] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523 19536, 2022. 4,7
[36] Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep learning. MIT press, 2016. 6
[37] Yanli Wang, Jewen Xiao, Tugba O Suzek, Jian Zhang, Jiyao Wang, Zhigang Zhou, Lianyi Han, Karen Karapetyan, Svetlana Dracheva, Benjamin A Shoemaker, et al. Pubchem's 's bioassay database. Nucleic acids research, 40(D1):D400-D412 2012. 6, 15
[38] Sebastian G. Rohrer and Knut Baumann. Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based PubChem Bioactivity Data. J. Chem. Inf. Model., 19(2):169-188 2009. 6, 15
12

---

[39] Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. 17. Journal of chemical information and modeling, (2(11):2864-255) 2875. 2012. 6, 15
[40] Keyulu Xu. Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00822. 2018. 16
[41] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for pre-training graph neural networks. arXiv preprint 19X.v:1905.12200 2019, 6. 7
[42] Kristof Schütt. Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377-9388. PMLR, 2021. 6, 17
[43] Kristof Schüitt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller, Schnet: A continuous- folttutuuuu-tuuerrrere convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017. 6, 9, 16
[44] Diederik P Kingma and Jimmy Ba. Adam:
method for stochastic optimization. arXiv preprint
arXiv: 14226694,,,,90,, 2014. 7
[45] Zhenyu Hou, Xiao Liu, Yukuo Con, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Duta Mining, pages 594-604,77 2022. 7.9
[46] Ziheng Qin, Kai Wang, 7angwei Zheng, Jianyang Gu, Xiangyu Peng, Zhaopan Xu, Daquan Zhou, Lei Shang. Baigui Sun, Xuansong Xie, et al. Infobatch: Lossless training speed up by unbiased dynamic data pruning. arXiv preprint arXiv:2303.0494 049477 2023. 7, 8. 15, 16, 18
[47] Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020. Proceedings, Part XVI 16. pages
Springer, 2020. 15
[48] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovee, and Matci Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint or X9v: / 1829, 2019. 8, 15, 16
[49] Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach. arXiv preprint 1002.08882209441 2018. 8, 15
[50] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960. PMLR, 2020. 8, 15
[51] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings the AAAI Conference on Artificial Intelligence, 2021. 8, 15
[52] Ravi S Raju, Kyle Daruwalla, and Mikko I Lppasti. Accelcrating deep learning with dynamic data pruning. arXiv preprint arXiv:2111. 126211, 2021. 8, 16
[53] Shengehao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206. 13602, 2022. 9
[54] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing syxtems, 33:5812-5823, 2020. 9
[55] Teague Sterling and John J Irwin. Zine Iigand discovery for everyone. Journal of chemical information and modeling, 15112)2332 2337, 2015. 10
[56] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 22133, 2020. 15
[57] Raghunathan Ramakrishnan. Pavlo O Dral, Matthias Rupp, and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Duta. 1, 2014. 15
[58] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019. 15
13

---

[59] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv
preprini arXiv: 024282288 2019. 15
[60] Greg Landrum et al. Rdkit: Open-source cheminformatics software. 2016. URL http://www. rdkit. org/ htpps:/github. com/rdkitrdkitt 149(150):650, 2016. 15
[61] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications, pages 181-195. Springer, 2022. 15
[62] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint vrXiv:1203.3472 2012. 15
[63] Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, and Augustus Odena. Small-gan: Speeding up gan training using core-sets. In International Conference on Machine Learning. pages 9005-9015, PMLR, 2020. 15
[64] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for rorl-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022. 15
[65] Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. arXiv preprint www.i:220090376000 2021. 15
[66] Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages
PMLR, 2021. 15
[67] Krishnateja Killamsetty. Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-super learning. Advances in neural information processing systems, 34: 14488 14501, 2021. 15
[68] Vishal Kaushal, Suraj Kothawade, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A unified framework of parameterized submodular information measures for targeted data subset selection and summarization. arXiv preprint 1rXiv:2103.00/22 2021. 16
[69] Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular infor- mation measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34: 18685-18677, 2021.
[70] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pages 1954-1963 PMLR, 2015. 16
[71] Noveen Sachdeva, Carole-Jean Wu, and Julian McAuley. Svp-cf: Selection via proxy for collaborative filtering data. arXiv preprint 1xXiv:2007.0499 2021. 16
[72] Jiarong Xu, Renhong Huang. Xin Jiang, Yuxuan Cao, Carl Yang, Chunping Wang, and Yang Yang. Better with less: A data active perspective on pre- pre-trinnng graph neural networks. Advances in Neural Information Processing Systems, 36:56946 56978, 2023. 16
[73] Boris Weisfeiler and Andrei Leman. A Reduction of a Graph to a Canonical Form and an Algebra Arising During This Reduction.
Informatsia, 2(9):12-16, 1968. 16
[74] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 1771 2024. 20
[75] Liang Wang, Xiang Tao, Qiang Liu, and Shu Wu. Rethinking graph masked autoencoders through alignment and uniformity. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15528-15536, 2024.
[76] Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399 2023.
[77] Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Li. Zerog: Investigating cross-dataset zero-shot transferability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1725-1735, 2024. 20
14

---

A Datasets and Tasks
In the following, we will elaborate on the adopted datasets and the statistics are summarized in Table 4.
Table 4: Statistics of datasets used in experiments.
Dataset
Data Type
#Molecules
Avg. #atoms
Avg. #honds
#Tasks
Avg. degree
Pre-training PCQM4Mv2
SMILES
3,746,620
14.14
14.56
2.06
HIV
SMILES
41,127
25.51
27.47
2.15
PCBA
SMILES
437,929
25.96
28.09
92
2.16
Finetuning
QM9-U0
SMILES, 3D
130,831
18.03
18.65
2.07
QM9-7PVE
SMILES, 3D
130,831
18.03
18.65
2.07
PCQM4Mv2 is a quantum chemistry dataset curated by Hu al. [56] based on the PubChemQC project [33]. It comprises 3,746,620 molecules and is extensively utilized in molecular pretraining tasks. We also adopt this widely recognized dataset for our molecular pretraining endeavors.
HIV dataset is designed evaluate the ability of molecular compounds inhibit HIV replica- tion [34] in a binary classification setting, consisting of 41,127 organic molecules.
PCBA is a dataset consisting of biological activities of small molecules generated by high- throughput screening [37]. It contains 437,929 molecules with annotations of 92 classification tasks.
QM9 is a comprehensive dataset, structured for regression tasks, that provides geometric, energetic, electronic and thermodynamic properties for subset of GDB-17 database, comprising 134 thousand stable organic molecules with up to nine heavy atoms [39]. In our experiments, we delete 3,054 uncharacterized molecules which failed the geometry consistency check [57]. We include the UO and ZPVE in our experiment, which cover properties related to stability, and thermodynamics. These properties collectively capture important aspects of molecular behavior and can effectively represent various energetic and structural characteristics within the QM9 dataset.
MUV (Maximum Unbiased Validation) group was selected from PubChem BioAssay via a refined nearest neighbor analysis approach, which is specifically designed for validation of virtual screening techniques [38].
B Computing infrastructures
Software infrastructures. All the experiments are implemented in Python 3.7, with the following supporting libraries: PyTorch 1.10.2 [58], PyG 2.0.3 [59], RDKit 2022.03.1 [60].
Hardware infrastructures. We conduct all experiments on a computer server with 8 NVIDIA GeForce RTX 3090 GPUs (with 24GB memory each) and 256 AMD EPYC 7742 CPUs.
C Related work
Data pruning (DP) has been ongoing research topic since the rise of deep learning. Traditional data pruning strategies often focus solely on the task dataset, exploring ways to represent the distribution of the entire dataset with fewer data points, thereby reducing training costs. However, with the recent advancements in transfer learning, focusing solely on the task dataset has become insufficient. Consequently, some data pruning strategies have been developed for transfer learning scenarios. We classify these strategics into in-domain data pruning and cross-domain data pruning.
In-domain data pruning. Most existing data pruning methods fall into this category. We further divide them into two groups: static data pruning and dynamic data pruning following [46]. Static data pruning aims to select subset of data that remains unchanged throughout the training process, while dynamic data pruning methods consider that the optimal data subset evolves dynamically during training. Guo et al. [61] classify existing static data pruning methods based on their scoring function into the following categories: geometry [47, 62, 17, 63, 64], uncertainty [48], loss [16, 19, 46), decision boundary [49, 65], gradient matching [66, 50], bilevel optimization [13, 51, 67].
15

---

submodularity [68-70], and proxy [71, 48). Despite dynamic data pruning is still in its early stages, has demonstrated superior performance. Raju et [52] propose two dynamic pruning methods called UCB and e-greedy. These methods define an uncertainty value and calculate the estimated moving average. During each pruning period, e-greedy or UCB used to select a fraction of the samples with the highest scores, and training is then conducted on these selected samples for that period. Recently, InfoBatch [46] achieves lossless pruning based on loss distribution and rescales the gradients of the remaining samples to approximate the original gradient. However, all of these methods place much emphasis on the target domain while ignoring the widespread use of transfer learning.
Cross-domain data pruning. We observe that with the use of pretraining, there is an additional source domain alongside the target domain. The key issue now is how to effectively utilize the information from both domains for data pruning in the context of transfer learning. To effectively address downstream tasks, straightforward approach is to measure the distribution shift between the upstream and downstream data, and then prune the pretraining dataset to align its distribution with that of the downstream dataset [32, 31, 72, 22]. However, this method requires retraining the pretrained model for each different downstream task, which contradicts the intended pretrain-fineture paradigm. Therefore, we propose the problem of source-free data pruning which is aligned with practical usage of transfer learning.
D Backbone Model
D.1 Embedding 2D graphs
Graph Isomorphism Network (GIN) [40] is simple and effective model learn discriminative graph representations, which is proved to have the same representational power as the Weiseeiler-Lehman test [73]. Recall that each molecule represented as G = (A, X, E), where A the adjacency matrix, X and E are features for atoms and bonds respectively. The layer-wise propagation rule of GIN can be written as:
h (++1) plk+1)
h
Σ JENU)
(n
(k+1) hood
E())
(6)
where the input features h (0) = 2;, N(1) is the neighborhood set of atom Vi, and fatom- foood are two MultiLayer Perceptron (MLP) layers for transforming atoms and bonds features, respectively. By stacking K layers, we can incorporate K- K-hop neighborhood information into each center atom the molecular graph. Then, take the output of the last layer as the atom representations and further use the mean pooling to get the graph-level molecular representation:
2D
h
(7)
N
JVV
D.2 Embedding 3D geometries
SchNet [43]. We use the SchNet [43] as the encoder for the 3D geometries in HIV dataset. SchNet models message passing in the 3D space continuous-file filter convolutions, which is composed of series of hidden layers, given as follows:
h (K+ = SMLP Σ fralh....... (r.r) n/"
where the input h, a; is an embedding dependent on the type of atom vi, fra(.) denotes the filter-generating network. To ensure rotational invariance of a predicted property, the message passing function restricted to depend only on rotationally invariant inputs such as distances, which satisfying the energy properties of rotational equivariance by construction. Moreover, SchNet adopts radial basis functions to avoid highly correlated filters. The fiter-generating network is defined as follow:
fro( T..T) x, ex(r) T)) æ, exp(-ylllrr r/l22 -m1l)).
(9)
16

---

Similarly, for non- quantumm properties prediction concerned in this work, we take the average of the node representations as the 3D molecular embedding:
JD
h (K)
(10)
LEV
where K is the number of hidden layers.
PaiNN [42]. We the PaiNN [42] the encoder for the 31 geometries in QM9 dataset. PaiNN identify limitations of invariant representations in SchNet and extend the message passing formulation rotationally equivariant representations, attaining more expressive SE(3)-cqiivarian neural network model.
E Proof of Theoretical Analyses
Assumption 1 (Slow parameter updating) Assume the learning rate small enough, so that the parameter update A0, 0,11 - 0, is small for every time step, i.e. ||A0,II s c, VI E N, e is small constant.
Lemma 1. With the assumption of slow parameter update, we can prove that ll,r 0.1l
1-8
Proof.
& - 0, = (1 -B);;1 -1
(1 -B)(E1 (1-1) -B)AA
(11)
- (1- (1-BAA0-
For the first equations, we respectively the definition of EMA parameter update in equation 1 and the definition of A0. For the third equation, we iteratively employed the results from the previous two steps, along with the initial condition Eo Og. With Assumption we have
lK,; 0,1 (1 (1-B)es
1 1-B
(12)
For the following results, the default setting in experiment 0.5, i.e. lE., 0.ll e.
Proposition (Interpretation of loss discrepancy) With Assumption the loss discrepancy can be approximately expressed by the dot product between the data gradient and the "EMA gradient":
L(x, E,) L(x, ofoo 0, EMA + +(Ee).
(13)
where MAA denotes El -VV, O-,)), i.e. the weighted sum of the historical gradients, which we termed as "EMA gradient".
Proof. From Lemma 1, since [|,, 6,1l small, we can use Taylor expansion of the loss function at 0,:
C(x.ft) L(x, 0,) voL(x, 0, - 0.) O - 0,112)
(14)
VoL(a, 0.) E (1- V O(leel?),
where we use equation 11 and the definition of online parameter update equation 1.
Proposition 2 (Gradient projection interpretation of MolPeg) In the context of neglecting higher- order small quantities, we define D* E D and f++ E Đ as samples that have positive dot products between the data gradient and the "EMA gradient", then
V voL 0) V' 0) av EMA
cul .a 0,
EMA
(15)
17

---

Similarly, we define D D and D' E D as samples that have negative dot products, then
VoD 0) = VoL(D 0) + bv EMA + d
EMA b = 0, d E R.
(16)
Equality holds if and only if the absolute loss discrepancy (_(x, L(x, 0,) across DD and D uniform. This a rare situation, and such a case, our data selection stralegy degenerates to random selection on D and D
Discussion about c and d is provided after the proof.
Proof. In the context of neglecting higher-order small quantities, MolPeg selects data with large loss discrepancies, meaning large dot products by using Proposition 1. That is Vx D and Vx' D D we have
VoL(x, 0) v F.M IVoL(x' 0) a F.
(17)
Then for samples in 2D and Dt, we have
Σ
() v EMA
Σ D
Vol(x,))
FMA >0
D
D
(18)
That is voL(D* 0) a VoL(D", O) on ULMA, the coefficient a is (VaL(D), 0) VoL(D",, EMA 2 0.
EMA 2 VoL(D* .0) EMA 0 for short. Thus when projecting VaL(D*
Similarly,
VoLL 0) EMA VoL(D) .0) v FMA <0
(19)
Then (VoL(i) .0) VoL(D), .)) EMA $ O.
The condition for a = 0 (c 0) is that the equality in equation 17 holds for samples in D (D)).
Since selection strategy does not constrain the direction perpendicular to the EMA gradient, consider a simplified model where and are treated as random variables with an expectation of zero. Consequently, in the sense of expectation, equation 4 and equation 5 hold. The feasibility of this simplified model is demonstrated as follows. Assume that VgL(x,, ()) D for all sam- ples independent and identically distributed random variables with expectation μ and variance 2 When the sample sizes |D* and [D* are sufficiently large, the central limit theorem implies that 1DT EXEDY VoL(x, () DfMA is approximately a Gaussian distribution N (u. 12
and similarly,
ior ExED VoL(x, 0) EMA is approximately a Gaussian distribution N [1. 1 The expectation of their difference Ec EVal(*+ 8) EMA EVoL(D* A) EMA Similarly, we can prove
Ed = O.
F
Connections to Existing DP Methods
F1 MolPeg & GraNd [19]
In the pretraining scenario, where the initialization is fixed, the GraNd score is defined as the norm of the gradient [[,, 0,)l. With Assumption 1, we deduce Ikg, 0,1 s shown in equation 12,
then the data selected by MolPeg satisfies 0 < [[xx 0,) - L(x, & = Vv (0, - E,) + 0(2) eL,x, 0,)l O))2). The data select has lower bound on the GraNd score [V 0,) 2
O(9), making it more likely to be chosen by the GraNd score.
F.2 MolPeg & Infobatch [46]
Our strategy employs relative loss scales rather than absolute values, enabling more flexible adaptation for transfer scenarios. For simple downstream samples for pretraining model, where L(x, E,) is small, both Infobatch and MolPeg eliminate samples with small online loss which are regarded as redundant for finetuning. However, for difficult samples for pretraining model, where L(x, E,) large, our method diverges from Infobatch by preserving the crucial samples for transfer learning.
18

---

F.3 MolPeg & Forgetting [16]
If we consider classification tasks and use accuracy loss, our method tends to select samples near the classification boundary. This can be related to the forgetting method, which aims to select samples that have been forgotten (i.e., initially classified correctly and then incorrectly) multiple times. For simplicity, let's explain this in the context of binary classification under Assumption 1. Further assume the class prediction probability f I-Lipschitz continuous with respect to the parameters 8. where f = ( f()) and ( (() = 1, we have (z, 8) f(x.g) N le. The loss function L(x, 8) = argmax, (a ()(0) yl, E [0, 1] not continuous the classification boundary where (ar, 6) = (n, 8) = 0.5. Consequently, only when the sample is located near the classification boundary (x, 0) (0.5 le/ v2, 0.5 le/ v2), exhibit non-zero loss discrepancy.
G Additional Empirical Results
G.1
The frequency of sample usage
Figure 7: Statics of frequency of sample usage in HIV datasets. The x-axis represents the number of times samples are used throughout the entire training process
Ratio 0.9
Ratio 0.8
Ratio = 0.7
Ratio 0.6
Ratio 0..4
Ratio 0..2
Frequenny
12599
yx8-
6637
15689
1293
4890
2000
2008
3890
15
100
90
169
1/f
Sample usage
Sample usage
Sample usage
Sample usage
Sample usage
Sample usage
Unlike static DP, although we use fixed proportion of samples in each iteration, we monitor the training dynamics of the entire dataset. Our method naturally avoids the issue of completely ignoring crucial samples, as almost all samples are used to varying degrees. Coordinating the use of samples in each iteration to achieve better generalization the main contribution of MolPeg. In Figure we visualize the frequency of sample usage in the HIV datasets, showing that almost all samples are used for training even with an aggressive pruning ratio adopted, with crucial samples being used more frequently.
G.2 Robustness evaluation of different pretraining datasets
Table 5: The performance comparison on HIV with different pre-taining datasets of varying quality in terms of ROC-AUC (%. T)
Pre-train Dataset
QM9 (Low quality)
ZINC (medium quality)
Pruning Ratio $
90
70
90
70
40
Hard Random
74.8,sa
78. 78.1
80.5
78.2
80.9
83. 83.1
Statie
Forgetting
76.5,52
79.4
80.9
79.6
81.5
83.7 jo.
GraNd
77.3
80.2
80.8119
80.2
81.7
83.9
Glister
75.9
78.6,41
81.1
78.6
81.2pa
83.5
Dynumic
Soft Random
79.1
81.3
82.5
81.8
83.2
84.0
LCB
79.5
81.8 m
82.5
82.2
83.1
83.9 p3
InfoBatch
80.5
81.7
83.1 ym
82.5
83.8
84,5
MolPeg
81.5
82.9
83.9
83.2
84.5
85.6
Whole Dataset
82.740.5
84.2=0.2
Train from Scratch
83.8+0.2
G.3 Results on MUV dataset
In Table 5, we have supplemented with additional experiments the MUV dataset, following the same experimental setup described Section 5.1. We observe that Mol still achieves state-of-the- art performance the MUV dataset, further validating the effecttivnnes of our method.
19

---

Table 6:
The performance comparison state-of-the-art methods MUV in terms of ROC-AUC
(%, 1).
MUV
Static Pruning
Dynamic Pruning
Pruning Ratio
Random
Forgetting
GraNd-20
Glister
Random
UCB
InfoBatch
MolPeg
90
73.8
75.3
75.6
75.5,47
76.3
76.8,s
78.0
79.8
70
75.7 (us
76.8 p4
77.2
77.6
77.9
78.2
79.1
80.7 m0.7.us
40
78.4
78.5
77.9
78.3
79.0
79.5jn7
80.5
81.2
Whole
80.2+0.2
H
Diseussions
Limitations and future works. Our data pruning strategy is specifically designed for molecular downstream tasks, but source-frre data pruning is task setting with broad applications in other fields well. For example, in large language models (LLMs) and heavy-weight models, pretraining data is often difficult for users to obtain or even kept confidential [74-77]. However, we have not validated method in these more general scenarios. Therefore, verifying the effectiveness of MolPeg in more general tasks is one our future research directions, e.g., graph, natural language and vision scenarios. Additionally. the first work designed for the source-free free data pruning setting, have only made simple attempts at perceiving upstream and downstream knowledge via loss discrepancy. In the future, will explore how to better utilize knowledge from both the source and target domains to achieve data pruning, which leaves significant potential to be explored.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines: It accurately reflect our contribution and scope. The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix H.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach only tested on a few datasets or with few runs, In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-tee system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the complete proof in Appendix E.
Guidelines:
21

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide our codes and README file in supplementary materials ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others replicate the model with the same dataset, access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that appropriate the research performed.
provide
also be provided via detailed instructions for how to replicate the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. example
If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models. it may be that access the model is limited in some way (e.g.. to registered users), but should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:
[Yes]
Justification:
We provide our codes and README file in supplementary materials to ensure reproducibility.
Guidelines:
22

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines Mides/Cooeeuubi cernmisssiooooooo ssionolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open- spen-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://oe /ttsss////iipeeee http//issscccpcpppceppppp cctpccccpulc cc/publ cc/ ///gui gui/ / ssi iisonnoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the error bars in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by eπror bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, trainjtest split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
23

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
We provide the computational resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimen- tal runs as well as estimate the total compute.
The paper should disclose whether the full research project required compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpg//tttte////eo......///////a cc/ppbblcc/tthiccuuueeeenee
Question:
Answer: [Yes]
Justification: The research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the discussion of broader impacts in Appendix H.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
legitimate to point out that
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
24

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly respect and credit the existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ com/datasee datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We provide asset with CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
25

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information in the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
26


Reviews:
[{"review_id": "vWYISh9gmZ", "reviewer": "Reviewer_zkte", "summary": "The paper introduces MolPeg, a molecular data pruning framework designed to enhance generalization when applying data pruning to pretrained models for molecular tasks. MolPeg uses two models with different updating rates to develop a new scoring function that assesses the informativeness of data based on loss discrepancies.", "questions": "None", "limitations": "It is suggested that the discussion of limitations be moved to the main text.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. By maintaining dual models that focus on both source and target domains and introducing a novel scoring function that selects both easy and hard samples, MolPeg achieves efficient, lightweight data pruning without the need for retraining.\n2. The paper provides the code, ensuring the reproducibility of the method. I will attempt to run the code in the coming weeks and may modify my review comments as necessary.", "weaknesses": "1. Although the paper claims to be pioneering in applying data pruning to pretrained models, the motivation may require further exploration. Specifically, how can it ensure that OOD samples crucial for each task are not pruned, which could potentially undermine the very purpose of the pretrained model?\n2. The definition of what constitutes a 'hard case' is unclear. Are these cases task-specific? If so, there's a risk that pruning might eliminate hard cases essential for certain tasks, affecting the model's comprehensiveness and utility.\n3. The approach might exacerbate the 'molecular cliff' problem, where slight changes in molecular structure lead to significant changes in activity, and such nuances could be lost with aggressive data pruning in pre-training.\n4. The experimental section is limited to only three commonly used molecular datasets. Other equally important datasets, such as MUV, were not included in the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **1. Although the paper claims to be pioneering in applying data pruning to pretrained models, the motivation may require further exploration. Specifically, how can it ensure that OOD samples crucial for each task are not pruned, which could potentially undermine the very purpose of the pretrained model?**\n\nThanks for your valuable feedback and insightful concerns. Preserving crucial samples is indeed a challenge for static DP methods in OOD scenarios. However, we would like to clarify that we employ a **dynamic DP strategy**. We invite the reviewer to refer to the `General Response` describing our pruning setup and the pseudo-code in the `Appendix G` for a better understanding. Below, we explain in detail why OOD samples crucial for each task are not pruned:\n\n- **Broader Receptive Field of Dynamic Pruning Strategy.** Unlike static DP, although we use a fixed proportion of samples in each iteration, we monitor the training dynamics of the entire dataset. Our method naturally avoids the issue of completely ignoring crucial samples, as almost all samples are used to varying degrees. Coordinating the use of samples in each iteration to achieve better generalization is the main contribution of MolPeg. In `Figure 1` of additional PDF file, we visualize the frequency of sample usage in the HIV datasets, showing that almost all samples are used for training even with aggressive pruning ratio, with crucial samples being used more frequently.\n- **Crucial OOD samples are essentially our hard samples.** Note that it is a contentious issue to determine whether or not a sample is crucial. Even in supervised training with the full dataset, the importance of different samples vary across epochs. In our scenario, crucial OOD samples can be considered as the ones that the training struggles with, corresponding to those with gradients opposite to the EMA gradient, as theoretically analyzed in `Proposition 2`. Therefore, we actually regard crucial OOD samples as an important category (hard cases) to preserve, rather than filtering them out.\n\n> **2. The definition of what constitutes a 'hard case' is unclear. Are these cases task-specific? If so, there's a risk that pruning might eliminate hard cases essential for certain tasks, affecting the model's comprehensiveness and utility.**\n\n\nWe apologize for any lack of clarity in our descriptions. Below, we provide clearer explanations for hard and easy cases to address reviewer's concern, and we will also include these in the revised main text for better understanding.\n\n- **Hard cases** refer to samples that the model struggles with during optimization. These can also be understood as samples near the decision boundary of downstream tasks. As analyzed in lines 156-160 of our manuscript, these samples satisfy $\\mathcal{L}(x,\\theta_t)-\\mathcal{L}(x,\\xi_t)>0$，indicating that this epoch's optimization gave negative feedback compared to historical optimization. \n- Conversely, **easy cases** are samples that can be optimized very smoothly, leading to a continuous loss reduction, satisfying $\\mathcal{L}(x,\\theta_t)-\\mathcal{L}(x,\\xi_t)<0$.\n\nMoreover, we want to emphasize that neither hard nor easy cases are pre-defined; they are identified in real-time based on the loss discrepancy reflecting training dynamics, making them definitely task-specific since the loss value is closely related to the specific task. Regarding the reviewer's concern about the risk that eliminating hard cases, we believe this is similar to concern in `Weakness 1`. In our previous response, hard samples are actually crucial OOD samples. Therefore, **our method does not eliminate these cases but rather preserves them as informative ones.**\n\n\n\n> **3. The approach might exacerbate the 'molecular cliff' problem, where slight changes in molecular structure lead to significant changes in activity, and such nuances could be lost with aggressive data pruning in pre-training.**\n\n\nThank you for your constructive feedback. We believe there might be some misunderstanding regarding our data pruning setup. As elaborated in the `General Response`, our data pruning is conducted on the downstream dataset, not during the pretraining stage.\n\nFurthermore, we would like to address the reviewer's concerns from an experimental perspective. If our method exacerbated the molecular cliff phenomenon in the pruned training set, the test performance in a random split setup would be poor. This is because retaining only these special cases during training would lead to a significant distribution shift between the training and test sets. However, **the experimental results in `Section 5` demonstrate SOTA data pruning performance**, and these **experiments were all conducted under the random split as mentioned in line 232 of our manuscript**. Therefore, we believe that the issue the reviewer is concerned about does not arise with our method.\n\n\n\n> **4. The experimental section is limited to only three commonly used molecular datasets. Other equally important datasets, such as MUV, were not included in the comparisons.**\n\nThanks for your valuable comments to further enrich our empirical analysis. We have supplemented our work with additional experiments on the MUV dataset, following the same experimental setup described in the manuscript. Please refer to `Table 1` in the PDF file in the `General Response` for the results. We observe that MolPeg still achieves state-of-the-art performance on the MUV dataset, further validating the effectiveness of our method. In the revised version, we will include these additional experimental results in the appendix to enrich the experimental validation.\n\n\n\n> **5. It is suggested that the discussion of limitations be moved to the main text.**\n\nThank you for pointing out this problem. In the revised version, we will follow the reviewer's suggestion and move the limitation to the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihWwexr6ZQ", "reviewer": "Reviewer_mAK4", "summary": "By utilizing the pre-trained models, this paper presents a plug-and-play framework (MolPeg) to prune target data without source dataset. By maintaining two models with different updating paces during training, this paper introduces a novel scoring function to measure the informativeness of samples based on the loss discrepancy. The experimental results on 3 datasets shows the enhanced efficiency and superior generalization in transfer learning.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper is well-written. Each component of the design space is carefully explained and well-presented. \n- This method is easy to follow can be adapted to other tasks. \n- Extensive experiments are conducted to verify the effectiveness of this method.", "weaknesses": "(Minor) Missing recent works: 1) static data pruning [1,2], 2) dynamic data pruning [3]\n\n1. Active learning is a strong baseline for data subset selection. NeurIPS workshop, 2022\n\n2. CCS: Coverage-centric Coreset Selection for High Pruning Rates. ICLR, 2023\n3. Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. ICML, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **(Minor) Missing recent works: 1) static data pruning [1,2], 2) dynamic data pruning [3]**\n\nThanks for your recognition of our work and constructive suggestions. We apologize for omitting any relevant works. For the related works provided by the reviewer, we have added experimental results on the HIV and PCBA datasets for the first two works as additional baselines (AL and CCS). The experimental results are shown below:\n\n| HIV Pruning ratio (%) | 90       | 80       | 70       | 60       | 40       | 20       |\n| --------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| AL                    | 80.7     | 81.1     | 82.9     | 84.0     | 84.8     | 85.1     |\n| CCS                   | 81.5     | 82.3     | 83.8     | 84.2     | 85.0     | 85.2     |\n| MolPeG                | **83.7** | **84.8** | **85.3** | **85.5** | **86.0** | **85.6** |\n\n| PCBA Pruning ratio (%) | 90       | 80       | 70       | 60       | 40       | 20       |\n| ---------------------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| AL                     | 15.2     | 19.2     | 20.9     | 22.5     | 25.2     | 26.2     |\n| CCS                    | 15.5     | 19.9     | 21.5     | 23.5     | 25.9     | 26.3     |\n| MolPeG                 | **20.7** | **23.9** | **25.6** | **26.4** | **26.8** | **27.0** |\n\nIt can be observed that MolPeG still achieves the best performance with significant improvements.  We will add the above experimental results and empirical analysis to the appendix of the revised manuscript to enrich our research.\n\nFor the last related work, due to the time constraint during the rebuttal period, we are unable to reproduce their results. However, we will also include it in the discussion of related works in the revised version. Finally, we appreciate your helpful comments. If you have any other questions, please feel free to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DFWFZmfKza", "reviewer": "Reviewer_uU9Z", "summary": "The paper presents a new framework MolPeg aimed at improving the efficiency and generalization of training models in molecular tasks using pretrained models. MolPeg introduces a novel DP technique that maintains two models with different update paces during training, leveraging the loss discrepancy between these models to score and select the most informative samples. This approach is applied in a source-free data pruning scenario, which does not require access to the original pretraining data. Extensive experiments across four molecular tasks demonstrate the effectiveness of MolPeg method.", "questions": "Please refer to weakness 3, why not validate the effectiveness of MolPeg with mainstream datasets? Molecular datasets belong to a more specialized field and seem less persuasive compared to datasets like ImageNet.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach to data pruning by leveraging pretrained models and a loss discrepancy scoring mechanism.\n2. The paper is well-organized, with a clear presentation of the problem, the proposed solution, and the experimental results.\n3. The experiments are comprehensive, covering multiple datasets and tasks, and provide empirical evidence supporting the efficacy of MolPeg.", "weaknesses": "1. The effectiveness of MolPeg is highly dependent on the quality and suitability of the pretrained models utilized. If appropriate pretrained models are unavailable, the benefits of MolPeg may not be fully realized.\n2. The proposed framework requires the maintenance and simultaneous updating of two models, which could increase both implementation complexity and computational overhead compared to simpler pruning methods.\n3. Most DP methods demonstrate their effectiveness on widely-used datasets such as ImageNet. However, molecular datasets represent a more specialized field. It is crucial to test MolPeg with mainstream validation datasets to further demonstrate its effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**1. The effectiveness of MolPeg is highly dependent on the quality and suitability of the pretrained models utilized.**\n\nThanks for your valuable comments. We agree with the reviewer’s point that the quality of pre-training can influence the effectiveness of our method. In source-free transfer learning, pretrained model is a hard-encoded module, and their variations naturally lead to performance changes. To address the reviewer's concern more thoroughly, we have conducted additional experiments on the HIV dataset using two pretrained models of different quality, obtained from the ZINC-100K and QM9 datasets, respectively. Compared to the PCQM4Mv2 dataset used in the main text, these two datasets are smaller in scale and exhibit more pronounced distribution shifts, resulting in poorer pretraining quality. The experimental results are shown in `Table 2` of the PDF file in the `General Response`.\n\nWe observe the following trends: MolPeg still achieves the best performance with these two pretrained models, demonstrating that while pretraining quality is a key factor affecting performance, **MolPeg remains the most robust and effective method compared to existing DP strategies.** We again thank the reviewer for pointing out this issue and hope this explanation addresses your concerns.\n\n\n\n> **2. The proposed framework requires the maintenance and simultaneous updating of two models, which could increase both implementation complexity and computational overhead compared to simpler pruning methods.**\n\nThanks for your valuable comments. The complexity is undoubtedly crucial for efficient learning, but we respectfully disagree with the reviewer regarding our method being more costly compared to existing pruning methods. It is worth noting that most existing DP methods are static and require scoring and ranking **on the full dataset before training**, which involves a high cost and computational overhead. We address the reviewer's concerns from two perspectives: complexity analysis and experimental validation.\n\n- **Complexity Analysis**: To compare the computational complexity of our method, MolPeg, with other data pruning methods, we use the following notations: $ N $ is the total number of data points, $ \\delta $ is the pruning ratio, $ T $ is the total number of training epochs, $ t $ is the number of pre-scoring epochs required by other methods to determine scores, and $H_{fw}, H_{bw}$ denotes complexity of forward and backward operation. Note that compared to the TopK operation, the majority of the time consumption comes from the process of scoring the samples, as this process typically involves computing the loss, gradients, or even more complex metrics. Therefore, we have disregarded the complexity of the sorting process and focus on the forward and backward complexity.\n\n  |                     | MolPeg                              | Other methods                            |\n  | ------------------- | ----------------------------------- | ---------------------------------------- |\n  | **Forward passes**  | $ \\mathcal{O}(2H_{fw}(N)) $         | $  \\mathcal{O}(H_{fw}((t + T\\delta)N)) $ |\n  | **Backward passes** | $  \\mathcal{O}(H_{bw}(T\\delta N)) $ | $  \\mathcal{O}(H_{bw}((t + T\\delta)N)) $ |\n\n  While MolPeg performs more forward passes due to processing two models as noticed by the reviewer, the computational bottleneck is the backward pass, which typically takes more than two times the effort of a forward pass [1]. Other methods require $ tN $ extra backward passes for pre-scoring, which outweighs their advantage in forward passes.\n\n  Therefore, despite the added forward pass complexity, **MolPeg is overall more efficient due to the significantly reduced number of backward passes**.\n\n  [1] PyTorch Distributed: Experiences on Accelerating Data Parallel Training. VLDB 2020.\n\n- **Experimental Validation**: In `section 5.1` of the manuscript, we have provided an experimental analysis of efficiency comparison. As seen in `Figure 4` of the manuscript, our method demonstrates significantly shorter runtime compared to previous DP methods and achieves better model performance. Compared to the latest dynamic pruning methods, although our runtime efficiency is slightly behind, we achieve better generalization performance, which is crucial for transfer learning scenarios. This minor efficiency compromise is acceptable.\n\n> **3. Molecular datasets represent a more specialized field. It is crucial to test MolPeg with mainstream validation datasets to further demonstrate its effectiveness.**\n\nThanks for your constructive suggestion and helpful feedback. To further demonstrate MolPeg's effectiveness, we have provided the experimental results on CIFAR-10 and CIFAR-100 in `Table 3` of the rebuttal PDF file. For the experimental setup, we follow the settings of InfoBatch. However, since our method requires a pretrained model, we adopt MoCo strategy to pretrain the ResNet-18 on ImageNet, and then fine-tune it on the CIFAR dataset. As seen from the experimental results, **MolPeg still achieves the best results on most pruning ratios, further validating the effectiveness of our method across different domains.** \n\nRegarding the reviewer's concern about the used datasets, we would like to reclaim our research motivation. As mentioned in `General Response`, our research is problem-driven and targets on ubiquitous and unsolved problem in AI for Chemistry community. Moreover, molecular tasks represent a complex and comprehensive task system. Unlike traditional DP tasks in CV, which focus on image classification, we validate MolPeg on **both classification and regression tasks**, involving **various molecular modalities** and **diverse task types**, which makes our evaluation even more comprehensive than traditional DP approaches. We will add the above experimental results to the appendix of the revised manuscript and we hope this response addresses your concern.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General Response**

We would like to thank all reviewers very much for their extensive reviews and constructive critiques. We are encouraged that reviewers find that our approach is efficient and lightweight (Reviewer zkte), that the experiments are comprehensive and verify the effectiveness (Reviewer uU9Z and mAK4), that the paper is well-organized with a clear presentation (Reviewers mAK4 and uU9Z).

However, we notice that the research motivation and data pruning setup have not been fully captured, leading to potential misunderstandings of some reviewers. Therefore, we would like to **restate our motivation and pruning setup** to address related concerns:

- **Motivation**: We point out the current need for efficient training in molecular modeling and attempt to improve training efficiency from the perspective of data pruning. Our exploration in the `Introduction` shows that **traditional DP methods fail** in the molecular domain due to significant distribution shifts, damaging the model generalization. Moreover, such distribution shift is inevitably due to the continual influx of novel molecular structures and functionalities in downstream tasks. In response to this phenomenon, we propose **the first source-free DP setup tailored for molecular domain**, which targets the main practical challenge in the field and represents the core motivation of our research.
- **Pruning Setup**: We want to emphasize that our approach involves a **dynamic data pruning on downstream datasets**, rather than static pruning on pretraining dataset. Unlike static pruning, which selects a fixed subset before training, our method dynamically selects the samples in each iteration. This means our approach **adapts in real-time based on training dynamics**, avoiding selection biases inherent in static DP methods and better catering to the specific task requirements.

### **Contents of PDF File**
Moreover, we have provided an extra **PDF file containing results and figures supporting our rebuttal arguments**. It should be noted that due to the time and page constraints during the rebuttal period, we were unable to supplement comparative experiments for all pruning ratios and all baselines. However, we selected 3 challenging aggressive pruning ratios and 8 competitive DP baselines for comparison. We believe these results could sufficiently validate the effectiveness of our method in the additional experiments. Below, we provide **a brief summary** of the content in the PDF for the reviewers' reference:

- **Figure 1** : We present the usage frequency statistics of samples when MolPeg is applied on the HIV dataset. The x-axis represents the number of times samples are used throughout the entire training process, and the y-axis represents the corresponding sample amounts. (for Reviewer `zkte` Weakness 1)
- **Table 1** : We have supplemented the pruning effectiveness of our method on the MUV dataset. (for Reviewer `zkte` Weakness 4)
- **Table 2** : We have supplemented the robustness of MolPeg's performance when using lower-quality pretrained models. These lower-quality pretrained models were obtained from the ZINC and QM9 datasets, which have more limited diversity and a smaller scale compared to the PCQM4Mv2 dataset used in the main text. (for Reviewer `uU9Z` Weakness 1)
- **Table 3** : We have supplemented MolPeg's performance on the classic image classification datasets—— CIFAR-10 and CIFAR-100. (for Reviewer `uU9Z` Weakness 3)

Finally, we appreciate all your helpful comments that strengthen the quality and clarity of our work. we hope the following responses address your concerns and we look forward to engaging in an active and productive discussion with the reviewers. If you have any other questions, please feel free to let us know.

Meta-review: After discussion, all reviewers provide positive recommendations. In particular, all reviewer recognized that the proposed method is novel with comprehensive experiments. They also suggested several aspects where the paper can benefit from, including analysis of various pretrained models and experiments on mainstream validation sets. The authors have addressed most of these concerns properly during rebuttal, and  should include these results in the final version. Therefore, I recommend acceptance for this work.
Decision: Accept (poster)