Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning
Authors: Yihua Zhang
Yimeng Zhang
Aochuan Chen
Jinghan Jia
Jiancheng Liu
Gaowen Liu
Mingyi Hong
Shiyu Chang
Sijia Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset pruning
transfer learning
TL;DR: We propose two effective and scalable dataset pruning methods for transfer learning.

Abstract: Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged as an effective way to improve data efficiency by identifying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e., how to prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy on downstream target tasks. To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning as separate problems. By contrast, we establish a unified viewpoint to integrate DP with transfer learning and find that existing DP methods are not suitable for the transfer learning paradigm. We then propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-target domain mapping. Furthermore, we demonstrate the effectiveness of our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to $40\%\sim 80\%$ without sacrificing the downstream performance, resulting in a significant $2\sim 5\times$ speed-up during the pretraining stage. Besides, our proposal exhibits broad applicability and can improve other computationally intensive transfer learning techniques, such as adversarial pretraining.

Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning
Yihua Zhang
Yimeng Zhang!*
Aochuan Chen
Jinghan Jia!
Jiancheng Liu'
Gaowen Liu
Mingyi Hong
Shiyu Chang
Sijia Liu!
Michigan State University. 2 Cisco Research, University of Minnesota, Twin City, UC Santa Barbara Equal Contribution
Abstract
Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged an effective way to improve data efficiency by identi- fying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e., how prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy downstream largel lasks. To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning separate problems. By contrast, we establish unified viewpoint to integrate DP with transfer learning and find that existing DP methods not suitable for the transfer learning paradigm. We then propose methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-targer domain mapping. Furthermore, we demonstrate the effectiveness our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to 40% ~ 80% without sacrificing downstream performance, resulting in significant our proposal exhibits broad applicability and can improve other computationalyy intensive transfer learning techniques, such adversarial pretraining. Codes are available at https: // thub. com/ C0//OTTL- /ttp///PPTTO/OODOOOAP Group/DP4TL.
new DP
5x speed-up during the pretraining stage. Besides,
1 Introduction
The abundance of the training data has long been regarded as the key driver of the contemporary machine learning (ML) algorithms [1-3]. However, the untrimmed, ever-growing training dataset could not only introduce training biases that compromise
model performance [4-6], but also poses an almost insurmountable obstacle to high training efficiency [1, 2]. Therefore, understanding the impact of the training data and selecting the most critical samples has emerged as an important goal, collectively referred as the problem of dataset pruning (DP).
Although the feasibility and the promise of DP have been unveiled in numerous applications, such as noisy data cleansing [7-9], continue learning [10-13] and active learning [14], greater emphasis is placed on the in-domain training setting, i.e., the pruned training set share the similar distribution as the evaluation set. Examples of methods to condense the training dataset with lossless generalization (on the in-distribution testing dataset) include data influence functions [10, 15-23], model training dynamics [11 [11,14, 24-28], and coreset selection [29-36]. In contrast, our work investigates the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Source Dutaset Pruning
Efficient Transfer Learning
ImageNet OxfordPets
Stage I: Pretrain (5x Speed Up
Turget Test Accuracy (%)
90
IMAGENET
Surrogate Mudel
(Small)
Stage II: Finetune
Source Model (L.arge)
80
85
GraNd
Moderate
Random
75
LM
Source Target
70
No Prume
FM
Domain Mapping
0%
20%
40%
50%
80%
Pruning Ratio
B
ImageNet StanfordCars
Target Test Accurcy (%)
Source Source/Target Data Features from Surrugate Mudel
LMFFM Data Pruning
Target Data
Source Data
Retained Data
+
Pruned Duta
30
American
Abyssinian
Shiba Inu
Black
Indigo
English
Tennis
Bulldog
Cat
Swan
Bunting
Springer
Ball
0%
20%
40%
60%
305
Pruming Ratio
Figure I: Left: An illustration of the proposed dataset pruning methods (I.M and FM) and their performance overview. Large scale source dataset is pruned by LM and FM through small surrogate model (ResNet- 18). Large foundation models can achieve up to 5x speed-up on pretraining without no downstream performance drop. Right: Dowastream performance overview at different pruning ratios when ResNet- 101 [54] is used as the source model (trained the pruned souree dataset) with ImageNet [55] as the source datasct transferring to OxfordPets [56] and StanfordCars [57]. The feature extractor fixed during finctuning.
problem of DP in the transfer learning paradigm [37-39], which has emerged as a popular way to leverage the knowledge of foundation model learned source dataset (referring to "pretraining' stage) to further enhance the performance on a droms-domain targer dataset (referring to "finetuning' stage). Recent evidence [40-42] has shown that some source data points could make harmful influence in the downstream performance. In particular, the previous study [41] showed that removing specific source data classes improve the transfer learning accuracy of a pretrained model.
Yet, efficiently identifying those harmful source data classes for improved transfer learning is a highly non-trivial task. Firstly, the unique pretrain- finretun-- paradigm complicates the analysis of the influence of source data on downstream performance, making it an indirect and very challenging process. Consequently, existing gradient and data influence-based in-domain DP methods [10, 19-23] cannot be naïvely adapted to the transfer setting. Secondly, transfer learning encompasses a wide range of source training methods other than supervised learning. such sepfrsuper learning (SSL) [43-52]. Therefore, a generic DP framework is desired given various pretraining scenarios. Thirdly, from efficiency standpoint, the design of efficient DP algorithms is non-trivial. Even in the non-transfer learning paradigm, a majority of current in-domain DP methods [11, 25, 26] introduce substantial computational overheads. For instance, influence function-based DP methods [10, 15-22] necessitate the calculation of the inverse Hessian matrix of model parameters, which is highly demanding and computationally intensive process. Moreover, training cynamics-basee methods [1 1-1,, 24-28, 53], such as GraNd-Score [26] and Forgetting
[11], necessitate model training on the entire dataset multiple times. As result, the development of an efficient and effective DP framework tailored for transfer learning remains a significant challenge in the field.
The most relevant work to ours is [41], which proposed brute-force method to evaluate the influence of every source class in the downstream task following a leave-one-out principle. While this method effectively selects the most influential source classes, it is prohibitively slow, the data selection process demands significantly higher computational resources than performing transfer learning itself. This brings the central question addressed in this paper:
(Q) How can we extend DP to transfer learning with minimal computation overhead. broad
applicability, and improved target performance?
To address (Q), we formally define the task of DP (dataset pruning) for transfer learning. We start by uncovering the limitations of conventional in-domain DP methods when applied to transfer learning tasks, and establish an intrinsic connection between source dataset pruning for transfer learning and source-target domain mapping. In the supervised pretraining paradigm, we propose an effective and scalable label mapping bbsdd framework capable of pinpointing the most beneficial source

---

data labels for a cross cross-domain target dataset. Furthermore, we extend the idea of LM to feature mapping (FM) for the SSL pretraining protocol, where data labels are unavailable. To achieve greater practical impact, we demonstrate that our proposed methods (LM and FM) can facilitate effective DP over a small, simple surrogate source model, and further translate the positive impact of the pruned source dataset to other larger, more complex source models. We provide a schematic overview and result highlights in Fig. Our contributions can be summarized as follows:
O We connect the concept of DP (dataset pruning) to transfer learning for the first time and formalize the problem of DP for transfer learning.
We develop a highly efficient and effective framework for DP in transfer learning, leveraging the source- target domain mapping with customized LM (label mapping) and FM (feature mapping) for both supervised and SSL. (self-supervised learning) settings. Our approach is principled and achieves significant speedups compared to the ofate-of-the-art methods.
We empirically show the effectiveness of our proposals (LP and FP) on downstream tasks. We find that the source dataset (ImageNet) can be pruned up to 40% 80% without sacrificing the downstream performance, together with a 2x ~ 5x speed-up in the pretraining stage. Our proposal also unlocks a door to prune dataset using simple surrogate source model (e.g., ResNet-18) and then reuse the pruned dataset to improve transfer learning on a larger source model (e.g.. ResNet-101).
4 Lastly. we show that our proposed DP framework can benefit other computationally intenssivv transfer learning techniques. For example, DP for adversarial pretraining [58] leads to a 1% ~ 2% improvement in downstream performance with time consumption similar to standard pretraining.
2 Related Work
Dataset pruning. DP (dataset pruning) is emerging technique to improve the data efficiency of model training by selecting the most representative training samples or removing the less influential ones [10-28]. Thus, the problem of coreset selection [29-36] can also be considered a form of DP. methods [59-61] to the more recent score-based
Prior studies on DP range from clustering- pruning methods [10-28]. In the latter approach, an importance score is assigned to each training data point to quantify its influence on particular permanence metric during model learning. Specifically, score-based DP methods can be broadly categorized into two main groups: influence function- based approaches [10, 15-23] and training dynamics- based approaches [11-14, 24-28, 53]. The first category measures data influence by examining the effeet of data removal the learning algorithm used during training [15-18] and the model's prediction [10, 19-23]. However, influence function- based approaches typically require high computational costs due to the need for high-order derivatives and complex optimization methods, such as bi-level optimization, as used in [22]. Although an approximate influence score be obtained efficiently, it may result in a large estimation error [62]. In the second category, training dynamics-bast based approaches find statistical indicators for pruned data from the training trajectory. Examples of DP metrics include data loss/error [25-27], prediction confidence [24], model gradient norm [26], forgetting event [11], and compactness [12-14, 28]. However, these methods typically require repeated training to ensure the representativeness of the collected statistics [11, 25, 26]. In addition to improving data efficiency, DP has been applied in variety of contexts, such as noisy label cleansing [7-9], continue learning [10-13, 63], active learning
[14], and reducing annotation cost of finetuner [64].
Data valuation & attribution. Data valuation [19, 41, 65-70] and attribution [71-73] are research streams related to dataset pruning that aim to quantify the influence of training data points on model's performance. Unlike dataset pruning, these approaches do not focus primarily on training efficiency but instead aim to enhance interpretability [74]. adversarial robustness [68, 70J, generative model design [75]. and data acquisition quality [72]. Representative methods include Shapley value-based
methods [65, 66]. datamodels [73], sampling- based methods [65, 68], and proxy-based based methods
[67, 72]. Recently, Kim et al. [35] proposed to select data samples from large public dataset (Open-Set) for self-supervisee
learning given a specific target dataset through distribution mismatch. However, it fails to make a general framework in both supervised and self-supervised scenarios, leaving the former under-explored. The most relevant work to ours is [41], which leverages leave- one-out analysis to quantify the influence of source data on downstream tasks in the context of transfer learning. With this inspiration, we propose to connect DP to transfer learning in this work.
3

---

Recent advancements in fransfer learning. Transfer learning has been prominent area over the past decade [37-39]. Significant strides have been made in understanding, analyzing, and improving various aspects of this technique [58, 76-84]. Recent studies [77, 78] have rigorously analyzed the relationship between source model performance and downstream task effectiveness, arguing that a narrow focus on minimizing source training loss may not lead to improved transfer learning results. To improve transfer learning, adversarial training has been shown to benefit the transferability of source pretraining on target downstream tasks [58, 76]. There also exist studies to identify and understand the failure cases in transfer learning [79-84]. Other research [77, 78] has examined model transferability from the perspective of model sharpness and argues that good pretrained model should be situated in a flat basin in the downstream loss landscape. Last but not the least, transfer leamnigg has progressed in several other directions, such self- self-supervised learning [43-52], model weight pruning [85-89], and visual prompting [90-99].
3 Problem Formulation
In this section, we introduce some essential preliminaries on transfer learning and DP (dataset pruning), and elucidate the design challenge of DP for transfer learning.
Preliminaries on transfer learning and connection to DP. Let g o f denote deep neural network that consists of a feature extractor f and a classification head g. where o denotes the function composition. Given source and target datasets D3 and Dy, we study transfer learning in the "pretrain- finetune" paradigm. The primary goal of pretraining is to obtain a high-quality feature extractor Is X' + Z, which draws a mapping from the input space (X) the deep representation space (2) in a data-rich source domain (Ds). Popular pertaining recipes include supervised learning (SL) [84, 100] and self-supervised learning (SSL.) [43-46] depending whether the source labels (ys) are available or not in Ds. In the finetuming stage, the pretrained model is further trained on specific downstream task under the target dataset DT. Transfer learning expects improved downstream performance over training on Dy from scratch. In this work, we consider two finetuning protocols, linear probe (LP) and full-finetune (FF), with Dy being a labeled dataset. LP finetunes the linear classification head with a fixed feature extractor fs. acquired from pretraining. In contrast, FF finetunes the entire model gof from the initialization fs. FF typically yields better transfer learning accuracy than LP, but the former takes a higher computation cost.
Our motivation for connecting transfer learning with DP comes from recent data-based perspective on transfer learning [41]. The study shows that removing certain source data classes from Ds can potentially improve the accuracy of finetuned model Dy. However, the task of evaluating the transfer effects of source data and removing their influence from a pre-trained source model has not been addressed efficiently. The approach developed in [41] involves leave-one-out analysis to estimate the influence of source class c on a target example t, which is computed as the prediction discrepancy of the finetuned source model at when the class is either included or excluded from Ds. During this process, one must train multiple source models (over 7000 models on ImageNet in [41]) from scratch over different subsets of Dg for a given target task. This approach becomes computationally unaffordable when dealing with large source datasets like ImageNet given limited computing resources. To address this challenge, we propose a DP perspective on transfer learning.
Problem of interest: DP for transfer learning. Next, we introduce the background of DP and the problem we focus on. Let D = {Zz, Zz,.... ZN| denote dataset consisting of n samples, where each sample 2, is represented as pair (x;, V/), with x; denoting the input feature vector and yı denoting the corresponding label. DP aims to generate pruned dataset D {2), 22.... ... _m} } D with M < N. which can reduce the training cost without a significant decrease in model generalization performance when trained on D. In the context of [41], instead of individual source data sample (2.). the entire source classes are evaluated and selected in terms of transfer influences. Based on the above, we define the problem of our interest below.
(DP for transfer learning) How to prune source data classes to obtain Dss (a subset of Ds), with lossless improved transfer learning accuracy of the source model (fs) on target task Dy?
DP for transfer learning has two key distinctions 1 (0-@ from the vanilla DP setup. First (0), DP must be performed in the source domain (Ds), while its effectiveness is evaluated based on the target

---

domain (Dy). This cross-doman challenge makes the design of efficient and effective DP highly non-trivial. For example, prior work [41] utilizes a computationally inmplttttionallyymmeniiivee intensive leave-one-out analysis. Classical influence function-based methods [10, 19-23, 101], which trace the eventual model's prediction through the learning algorithm and back to its training data, are also computationally infeasible due to the complex bi-level optimizer and the calculation of high-order derivatives. Second (9), the pre-trained source model (fss) in today's transfer learning regime is typically of a large scale. This motivates to develop DP method that can be independent of the source model, while the pruned source dataset Ds remains effective in the original transfer learning setup. Given this challenge, we will design DP methods for transfer learning using a simple surrogate source model to avoid the computation the large source model fs (see Fig. 1 for an illustration).
Conventional DP methods lack ef- fectiveness on transfer learning.
Flowers 02-LP
OxfordPets LP
Target Test Test Accuracy (%)
94
.00
Given the challenges (0-0) posed
788
13
86
by DP for transfer learning, we fur-
ther conduct a preliminary study to investigate the effectiveness of exist-
90
Randon
Forget
84
Moderate
-1.N
82
g g
GraNd
In.Sam
80
ing 8 DP methods, including SP [20],
SSP
SP
Max
Prune
78
SSP [20], GRAND [26], EL2N [26]. MODER ATE [3], FORGET [53], INF-
0%
20% 40%
60% 80%
76
0%
20%
40% 60% 80%
Source Data Pruning Ratio
Source Data Pruning Ratio
MAX [53], and INFSUM [53]. Our
Figure 2: Transfer learning accuracy of existing DP methods on ImageNet different pruning ratios, where ResNet-101 101 is the source model, and linear probing (I.P) is used for downstream finetuning on the target datasets Flowers 102 (Left) and OxfordPets (Right).
results show that these methods are unable yield significant improve- ments over random source data prun- ing. Fig. 2 shows the transfer learning performance of the ResNet-101 [54] model trained on different pruned versions of the ImageNet dataset when LP-based finetuning is conducted on the downstream Flowers 102 [102] and OxfordPets [56] datasets. As we can see, in transfer learning, random prun- ing is a solid baseline for various sfate-of-the-rr DP methods, which have demonstrated superior performance to the former in the non-tranc non-transfer learning regime. Therefore, it crucial develop an efficient and effective DP method specifically tailored for transfer learning.
4 Label/Feature Mapping-based DP for Transfer Learning
In this section, first introduce a simple yet powerful DP method called label mapping (LM) by leveraging the class-discimmnative capability of a supervised source model. We then extend the concept of LM to feature mapping (FM), suitable for self-supervised pretraining.
LM-based DP for supervised pretraining. Following the notations used in Sec. 3, we represent the model obtained through supervised learning on the source dataset (Ds) as gs o fs. This model predicts a source label given an input example (x). In DP for transfer learning, the source data classes serve as the variables to be pruned. Thus, we express Ds as Ds = {c( : ...., ) for N source classes, cla) denotes the set of data points belonging to the source class i. Additionally, the pruner has access to the target dataset DD= = [t,,..., t,. which consists of n target data points. Our objective is to utilize the information provided by 9s o fss Ds. and Df to devise a computationally efficient DP criterion. Importantly, our criterion should not involve model training, distinguishing it from the non-scalable approach in [41]. An important observation by [78] in transfer learning is that transferred knowledge improves transferability. This insight is supported by the loss landscape analysis in transfer learning: The finetuned weights may remain within the flat basin of the pretrained weights for enhanced transfer learning performance.
We next propose to extract the 'transferred knowledge' by leveraging the class discriminatiiee ca- pability of the source model gs fs on the target data samples in Dy. Specifically, we focus on monitoring the responsiveness of the source label predictions made by gs o ʃs when using target samples (t: E D alignment with source data. For the ith source data class cl.o) we then define its pruning score below:
input data. Here we resize these target samples to ensure their resolution
BLM SLM(C()) (1)
Σ
1 (gs fs(t;) 1), for 1,2,.... .., N.
(LM)
5

---

where 1()) represents an indicator function that evaluates to when the condition is satisfied, and 0 otherwise. We refer to the aforementioned formula as LM (Label Mapping) since the condition gs fs(t,) = i establishes a mapping between the predicted labels of the target samples {t, } by the source model and the corresponding source labels i. The larger SLM in (LM) signifies that a specific source class is more frequently utilized to interpret the target data. Consequently, this indicates a more tightly connected relationship between the source class and the transferred knowledge. Therefore, we prune (or select) source data classes with low (or high) SLM values.
Although LM is simple, it offers several advantages. Firstly, it is highly efficient in computation. Given pre-trai source model gs o fs, the calculation of LM only requires forward passes through the model. cases where obtaining the pretrained model in the source domain is challenging, our
proposal supports an alternative ap- proach: training a smaller and sim- pler surrogate model to carry out
95.0
Time Consumittio (h)
Mudel
94.5
Model Pretrining
L.M. This surrogate model can effec- tively replace the complex pretrained model Is o fs and facilitate the ef- ficient execution of the pruning pro- cess. As we show in Fig. 3a, employ- ing ResNet-19 18 [54] is sufficient to successfully prune the source dataset (ImageNet [55]). The resulting DP scheme remains effective to improve transfer learning utilizing other larger source models, such as ResNet- 101 [54]. In addition, Fig. 3b shows that the computational overhead incurred by training the small surrogate model (ResNet-18) for DP is insignificant, compared to the time saved during the pretraining phase of a larger source model (ResNet- on the pruned dataset for transfer learning. Lastly, pretraining on the subset found by LM can guide the source model towards a flatter region in the downstream loss landscape, (see results in Fig. A4). The source model trained on the LM-pruned dataset achieves a higher flatness score than baselines, which aligns with the understanding of transfer learning in [78].
93.5
94.0
t 4-
3.5
93.0
RN-18
2.4
92.5
RN-101
92.0
No Prune
91.5
0%
20%
40%
60% 80%
No Prane 40%
60%
Source Data Pruning Ratio
Praning Ratio (%)
(a) Downstream performance.
(b) Time consumption.
Figure 3: Preliminary studies on the usage of surrogate models for
LM. (a) The downstream performance (on Flowers 102) of using the source model ResNet- 101 trained on the pruned ImageNet delivered hy LM at different pruning ratios. Here LM is conducted using either RN-101 or smaller surrogate model R (b) Computation time decomposition analysis for ohtaining the pretrained model using 18 18 as surrogate model with different pruning ratios.
FM-based DP framework for self-supervised pretraining. The requirement for labeled source data in LM may pose limitations on the application of DP methods, particularly in the context of self-supervised learning (SSL)-based pertaining. To address this limitation, we introduce new method called FM (Feature Mapping). Unlike LM, FM determines the DP scheme using only the feature extractor network fs. the unlabeled source dataset Ds. and the target dataset Dy. This allows us to overcome the dependence on labeled source data, making FM applicable in SSL scenarios. The inspiration for FM is derived from the deep clustering technique [103-105] operating in the representation space, which can generate pseudo source labels using cluster indices provided by e.g., K K-means. With the assistance deep clustering, we can represent the unlabeled source dataset Ds as Dg = - c ,...,,cA,,, c(K)), }, where c& denotes the set of source data samples within cluster k. Building upon the similar spirit as LM, we propose ranking the importance of pseudo source classes for DP by evaluating the source fcature extractor's 's responsiveness to target data samples. To achieve this objective, we quantify the responsiveness of fs for a target sample as follows:
r(t) = arg min
[s(()) E
Exeego [fs(x))]]
(I)
kE[K]
where E Execso [fs(x)] is the centroid of source data within the cluster k, and r(t) is the nearest pseudo label as the responsiveness of fs against t. The FM score then integrates (1) with (LM):
(C Σ 1(((t))= = f), for = 1,2....,
(FM)
where different from (LM), K represents the number of pseudo source classes produced by deep clustering, and r(t,) corresponds to the source feature extractor's prediction on ty. It is important to
note that the value of K is free parameter of the deep clustering process. Our empirical study in Fig. A2 shows that FM is quite robust to the choice of K without sacrificing the benefit of DP for

---

transfer learning compared to using the unpruned source dataset. Lastly, it is worth mentioning that FM can also be applied in the context of supervised pretraining by specifying data labels as clusters.
Experiments
In this section, we provide comprehensive set of experiments and analyses to showcase the effectiveness of our proposed methods (LM and FM) in diverse transfer learning scenarios.
5.1 Experiment setup
Datasets and models. In line with existing transfer learning benchmarks [41, 106], we utilize ImageNet-IK [55] for pretraining and 8 datasets as downstream tasks. These datasets include DTD [107], Flowers102 [102], UCF101 [108], Food101 [109], SUN397 [110], OxfordPets [56], StanfordCars [57], and CIFARIO [111]. Please refer to Tab, Al for more details about the datasets. As discussed in Sec. 4, we utilize ResNet-18 (RN-18) [54] as the surrogate source model for pruning source classes. This method significantly reduces the computational cost associated with DP, making the process more efficient. Subsequently, range of larger models, e.g., ResNet-101 (RN-101) and ViT-B/16 [112], are trained on the (pruned) ImageNet and then finetuned on downstream tasks.
Baselines, training, and evaluation. By examining the performance of the existing 8 DP baselines as shown in Fig. 2, our experiments focus on two of the most effective methods: ① GRAND [26] and @ MODER ATE [3], together with @ RANDOM (the random pruning strategy). In Fig. AI, show more results of the rest DP baselines. Unfortunately, we are unable to include the existing data attribution method [41] as our baseline, it does not release its pruning results, and are unable to repeat its experiments due to the need for intensive computations. Unless specified otherwise, we focus on the supervised pretraining setting in the experiments. For self-supe pretraining, we follow the implementation of MoCov2 [45]. The finetuning strategies employed include LP (linear probing), which finetunes the classification head with fixed feature extractor, and FF (full finetuning), which finetunes the entire source model. For FM-based DP method, we utilize K-means clustering to group the ImageNet training data points into K 2000 clusters for the computation of (1).
In accordance with the terminology convention in model pruning [87, 113-115), we refer the term 'winning subsets' to the obtained source subsets that do not compromise downstream performance. Among these winning subsets, we identify the one with the highest pruning ratio as the 'best winning subset'. We then evaluate the performance of DP methods from the following aspects: the downstream performance of the model pretrained on the pruned source dataset obtained by various DP methods, and the pruning ratio of the best winning subset achieved by DP methods, accompanied by the corresponding time saved in the pretraining phase.
5.2 Experiment results
LM/FM improves transfer learning accuracy by identifying 'winning subsets'. We first showease the significant improvements achieved by our proposed DP methods (LM and FM) compared to baselines. Our methods successfully identify winning subsets of the ImageNet, yielding transfer accuracy on par or even better than scenarios without pruning.
Fig. 4 presents the downstream accuracy of transfer learning vs. different pruning ratios. Here DP is performed using the surrogate model (RN-18) on ImageNet for 8 downstream tasks. The source model (RN-101) is then trained on the pruned ImageNet and the transfer learning accuracy is assessed through I.P (linear probing) and FF (full finetuning). We also present the downstream performance without pruning the source dataset (No Prune) reference for winning subsets. As we can see, both LM and FM significantly outperform the baselines by a substantial margin. Notably, LM and FM consistently identify winning subsets with significantly larger pruning ratios in all settings. This highlights the effectiveness of our proposed methods in achieving substantial dataset pruning without hurting downstream performance. Furthermore, we observe that the downstream performance of LM and FM initially improves and then declines as the pruning ratio increases. This is not surprising, since the initial increase in performance corresponds to the scenario where harmful source classes are removed, consistent with [41]. When the source dataset continues to shrink, the performance inevitably decreases, as a natural consequence of reducing the size of the source dataset. Moreover, in some datasets of small sizes (e.g., OxfordPets), LP achieves performance comparable to FF. This
7

---

3,90
OxfordPets ecs LP
SUN3VILP
64
Po
2.68
DID:LP
Darget Test Acturs
96
94
GmNa
Tirge Test Acturace
50
Agaraey
F62
60
58
Target Test 1e009
3.66
64
660
62
(Ouns)
49
56
58
54
56
47
0%
20% 40%
60% 30%
0%
20%
40%
60% 80%
05
20%
40%
60%
80%
0%
20%
40% 60% 80%
Source Data Proming Rabio
Source Data Proning Ratio
Source Data Prining Ratio
Source Data Pruning Ralio
Torpet Accuracy (16)
SrantvnlCars.LP
UCF101-LP
3.90
CIFARJO-LP
lagel Test Accurary (g)
Flo
72.5
2.89
7000
67.5
40
65,0
92
60.0
%
30
57.5
K5
55.0
82
60% 80%%
0%
20%
40% 60%
20%
61%
80%
20% 45% 60% 80%
Pruning
Pruning Ratio
Source
Pruning Ratio
Source Data Pruning Ratio
92
OxfordPass FF
25
SUN397-FF FF
Tonpe Tog Acuuray (33)
Food
Hager lost Accuracy (%)
DTD-FF
96
Random
Moderate
52
50
80
2.
84
48
2
(Chrr)
46
59
E
(Ouns)
No Prunc
Taape
44
76 1%%
20%
45% 60% 80%
01%
20%
40%
60% SUP
78
0%
20%
40%
60%
56
55% 20% 41f 60% 80%
Source Data Pruning Ratio
Source Data Prasing Kanio
Source Data Prining Ratio
Source Data Pruning Ratio
larget Jost Aceuncy (4)
StanfordCars-FF
UCFI01-FF
Acurncy (P6)
CIPARIO.FF
Juget Test Ascuray (8)
.99
Plowers 102-FF FF
97.0
96.5
66
Torgel Tea
644
62
95.5
660
95.0
0% 200 40%
60%
50%
0%
20%
40%
60%
80%
20%
40% 60% 80%
20% 40% 60% 80%
Source Data Pruming Ratio
Sounce Data Pruning Raão
Source Data Pruning Ratio
Source Data Pruning Ratio
Figure 4: Source datasct pruning trajectory given by downstream testing accuracy (%) vs. source dataset pruning ratio (%) in the supervised pretraining setting. Here the source model RN- 101 is trained on each full/pruned source dataset (ImageNet) and finetuned on different downstream tasks through LP and FF. The downstream performance without pruning (No Prune) marked with the black dashed line. Results are averaged over three independent trials (see exact numbers and variances in Tab. A2).
attributed to the fact that the performance gap between LP and FF using the large-scale RN-101 model on small-scale datasets tends to diminish.
Table I:
The pruning ratio of the best winning subsets obtained by each method for different downstream tasks. The other experiment setup is consistent with Fig. 4. The achieved largest pruning ratio is highlighted in bold in each column. And N/A indicates the scenario where no winning subsets are found.
Datrert
OxtoedPass LP
SUN197
Food101
17113
Stanford Ces
UCPI0I
CBFARIO
Flowers I0!
Finetuse methed
FF
LP
FF
L.
FF
LP
FF
LP
FF
LP
FF
LP
BF
LP
FF
RANDONM
2005
2183
185
N/A
N1A
3035
105
203
N/A
10%
2055
1068
N/A
1838
4035
35%
Moossstt STE
1006
200
15%5
20%
N7A
207
15%
30%
N/A
536
10%
NIA
35%
GR
2576
1075
356
15%
N/A
5%
15%
41%6
LM (ouss)
5005
65%
65%
55%
65%
55%
6075
Tab. 1 provides a summary of the pruning ratios achieved by the best winning subsets identified using different DP methods for all downstream datasets. Both LM and FM methods successfully remove more than 45% the source classes without downstream performance drop. In contrast, all the baselines experience significant performance degradation when the pruning ratio exceeds 40%.
Table 2: The downstream performance with different source data pruning ratios the SSL, pretraining setting. randomly initialized RN-101 is self-supervised pretrained using MoCo v2 on each full/pruned source dataset and finetuned the downstream task through I.P. The hest result in each pruning ratio is marked in bold and the performance surpassing the unpruned setting (pruning ratio 0%) is highlighted in eyan
Dasasel
OxfordPets
SUN397
Flowers 102
Pouning Ratio
0%
50%
60%
70%
80%
0%
50%
00%
70%
80%
0%
50%
60%
70%
80%
RANDOM
62.32
61.27
59.09
53.75
45.63
45.08
43.54
39.81
82.23
82.
81.03
80.02
GRAND
26
62.45
63.31
56.12
$7.42
44.23
FM (ours)
69.92
69.99
70.29
70.21
48.46
48.38
47.90
46.00
85.22
85.42
84.37
84.61
8

---

Tab. 2 highlights the effectiveness of FM-basee DP in the self-supervised pretraining setup for three representative downstream tasks. As we can see, the transfer learning accuracy achieved by using FM consistently outperforms baselines in the self-supervised pretraining paradigm. FM can identify winning subsets for transfer learning even in the challenging regime of large pruning ratios, ranging from 50% to 80%. For instance, in the case of SUN397, FM-based winning subsets achieves a pruning ratio up to 70%, and for Flowers 102 the maximum pruning ratio is 60%. These pruning merits align with the findings for FM in the supervised pretraining setup, as illustrated in Tab. 1.
DP enhances the efficiency of source pretrain- ing. Tab. 3 displays the computation time re- quired to obtain the pretrained source model us- ing LM at different pruming ratios. The reported time consumption includes the entire pipeline, encompassing surrogate model training (RN-
Table 3: Time consumption of LM/FM in Fig.4 to ob- tain the pretrained model. The reported time consump- tion covers surrogate model (RN-18) training, LM/FM dataset pruning, and source model pretraining (RN-101).
Pruning Ratio
0%
20%
40%
60%
80%
Time Consumption (h)
5.4
4.6
3.5
2.4
1.3
18), DP process, and source model training (RN-
(15%g)
(35%
(56%)L
(76%)
101) on the pruned ImageNet dataset. The runtime cost of the conventional transfer learning on the full ImageNet dataset for RN-101 is also listed as a reference. As we can see, DP enjoys high efficiency merit of source training. Taking the 5.4 hours required for source training on the full ImageNet dataset as a reference, LM-enabled 20% pruned ImageNet achieves a 15% reduction in training time. Moreover, the efficiency advantage increases to 76% when the pruning ratio reaches 80% and these computational benefits do not sacrifice transfer learning accuracy at all.
Next, we compare the efficiency of our methods with the state-of-the-art dataset pruning base- lines in Tab. 4, including GRAND, MODERATE, and the brute-force method proposed in [41]. We showcase the efficiency comparison with prun- ing ratio of 60%. As we can see, our method (even using surrogate model) achieves substantial computation efficiency improvement over other methods. In particular, [41] involves training thousands of source models. Thus, its computational cost is typically unaffortable in experiments.
Table 4: Time consumption comparison with a pruning ratio of 60% of different datasct pruning methods. Other settings follow Fig. 4,
Mothol
MODERATE
GRAND BRUTE-FORCE [-1] ours
Time
Consumption
DP enables efficient adversarial pretraining, Tab. 5 showcases the improvement in transfer learn- ing accuracy and efficiency achieved by our proposed LP-based DP method when incorporating adversar- ial training (AT) [116] on either the full or pruned ImageNet dataset. This transfer learning setup is motivated by the findings in [58], showing that en- hancing the robustness of the source model against adversarial attacks through AT can improve transfer learning accuracy in both LP and FF-based finetun-
Table 5: Downstream performance of models pre- trained on full/pruned source dataset (DENSE/LM) w/wo adversarial pretraining (Adv). For DENSE- ADV and LM-ADV, 3-step adversarial training [116] is used for pretraining. Pruning ratio, down- stream test accuracy (Acc.), and time consump- tions for obtaining pretrained models reported.
SUN39
DTD
Methal
Pruzing
Acc:/o)
Tims
Prasing
Asc.%)
Time
FP
DeNSE:
Nix
51.45
54.21
54
Nix
65.1
67.21
54
Dexan Apv
NIA
53.90
33.07
13.
NAA
6723
6192
13.7
ing scenarios. We then report downstream accuracies
LMM-AN
70
55.49
67.00
68.54
67
using LP and FF chosen due to the large room for improvement in transfer learning accuracy, as shown in Fig. We also determine the pruning ratios for LM by selecting those that led the best winning subsets. Our experimental results demonstrate that employing AT on both the unpruned and pruned source datasets can improve transfer learning accuracy. Specifically, we refer to AT on the original unpruned Ima- geNet dataset as DENSE-AT; and AT on the L. ImageNet dataset as LM-AT. One notable advantage of integrating LM into AT is the significant improvement in computation efficiency. A key highlight of this approach is that LM achieves a similar computation time as the standard source training on ImageNet (DENse), while exhibiting almost no accuracy drop compared to DENSE-AT. This observation demonstrates the potential to accelerate AT through DP.
specific downstream datasets:
SUN397 and DTTD, which are intentionally
Robustness against the surrogate model size. To explore the sensitivity of our proposed method to the surrogate model's s size and accuracy, we show the transfer learning performance on the task Oxford- Pets against different surrogate model sizes in Fig. 5. It is evident that even though the performance of the surrogate model on the source dataset (ImageNet) decreases, the downstream performance of RN-101 pretrained on the L.M-baedd pruned source subsets remains relatively stable. Further, Tab. A6 compares the class indices selected by the corresponding surrogate models. Interestingly. the most rele- vant source class selections exhibit high level of agreement across surrogate models of differing sizes.

---

LM shows robust behavior with different surro- gate models, even they yield different perfor- mance on the source dataset due to their differ- ent model capacities. For example, RN-32s only achieves test accuracy of 40.77% on the source dataset, but manages to achieve the same max- imum pruning ratio of the winning subsets as RN-18. This underscores that LM can accom- modate small surrogate model of 1% the size of the pretrained model (Rn-32s vs. RN-101). Moreover, it reveals that the surrogate model does not need to attain exceptional performance in the source domain for dataset pruning in transfer learning.
91
Tanger The Aesuaay
Rangen Teat Accmmcy
89
RexNet-200
89
RosNett32s
VGG-2
88
ResNet-4
VGG4
87
ResNet-Sta
87
VGGS
ResNet-18
Tamps
ResNet-18
No Prune
Prune
85
65 0%
0% 20% 40%
60% 50% Source Data Pruning Ratio
20% 40% 60% 80%
Source Data Pruning Ratio
(a) ResNets family
(b) VGG family
Figure 5: Source dataset pruning trajectory given the downstream task OxfordPets using different surrogate models. Other settings follow Fig.
Table 6: Downstream performance of different DP methods using ViT-B/16 as the source model and FF as the finetuning method. Other settings are consistent with Fig. 4. The best performance at each pruning ratio is marked in bold, and performance exceeding the unpruned setting (0% pruning ratio) is highlighted in cyan
Dataset
OxfordPets
SUN39)
Flower1022
Pruning Ratio
0%
20%
40%
60%
80%
0%
20%
40%
60%
80%
0%
20%
40%
60%
80%
RANDOM
85.63
83.72
75.94
69.83
43.74
41.18
36.72
29.41
90.25
89
87.47 47
$4.44
MODERATE
GRAND
85.75
85.39
83.88
83.97
82.35
82,39
79.12
78.32
43.97
43.97
40.93
41.82
36.52
38.89
31.00
31.15
90.86
89.77
88.34
86.69
86.15
85.72
84.77
$0.64
44.24
39.59
32.35
$9.07
87.23
DP for VIT. In Tab. 6, we present the consistent transfer learning improvement achieved by LM for Vision Transformer (ViT) pretraining. Specifically, utilize ViT-B/16 as the source model, RN-18 as the surrogate model for DP, and use FF on the source model for the downstream tasks studied in see, LM yields better transfer learning accuracy than baselines at different pruning
Tab. 2. As ratios, consistent with the previous results on RN-101. This observation highlights the model-agnostic advantage of our proposed method in transfer learning, We also observe the pruning ratio of the best winning subset achieved for VIT is smaller compared to ResNet. This discrepancy is possibly due to the higher data requirements of VIT during pretraining [112].
Additional results. We conduct ablation studies on the cluster number for FM in Fig. A2 and the results of selecting data proposed by LM in reverse order in Fig. A3. We shows FM can be applied to more advanced SSL framework in Tab. A3. We also investigate the performance of our proposed method in the multi-task setting in Fig. A5, and the scenarios with data biases in Tab. A4. We further examine our method on the few-shot benchmark in Tab. A5. We also present feature distribution analysis in Fig. A6 and image examples for the top classes selected by LM/FM in Fig. A7.
Limitations. Despite the remarkable empirical results above, we admit LM/FM mainly caters to exploring the source data influence in specific downstream task, and the largest pruning ratio is reduced as more downstream tasks are considered (see Fig. A5), which we consider as a limitation. Meanwhile, we acknowledge that the promising performance achieved by our method still lacks rigorous theoretical analysis, despite the feature distribution and model flatness analysis, which we believe will be challenging but exciting future work DP for transfer learning.
6 Conclusion
In this paper, we first formally define and investigate the problem of DP (dataset pruning) for transfer learning. Recognizing the ineffectiveness of conventional DP methods in the transfer learning setting, we proposed two novel and efficient DP techniques, label mapping and feature mapping. Extensive experiments demonstrate that both methods can effectively prune large ratio of source dataset and substantially reduce the pretraining computational costs without sacrificing the downstream performance in various transfer learning settings.
Acknowledgement
The work of Y. Zhang, Y. Zhang. A. Chen, J. Jia, J. Liu, S. Chang, and S. Liu was supported by the Cisco Research Award and partially supported by the NSF Grant 2IS-2207052. The work of M. Hong was supported by NSF grants 191-19 and EPCN-231 1007.
10

---

References
[1] B. Zhao, K. R. Mopuri, and H. Bilen, "Dataset condensation with gradient matching," arXiv preprint arXiv:2006 250,::0006.05929, 2020.
[2] B. Zhao and H. Bilen, "Dataset condensation with differentiable siamese augmentation," in International Conference on Machine Learning. PMLR, 2021, pp. 6265-12665. 1
[3] Xia, Liu, J. Yu, X. Shen, B. Han, and Liu, "Moderate coreset: A universal method of data selection for weal-world data-efficient deep learning," in The Eleventh International Conference on Learning Representations, 2023. 1, 7,
[4] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, A. Wichmann, and W. Brendel, "Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robust- ness," arXiv preprint 1. 12231,, 2018.
[5] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, "Adversarial examples are not bugs, they are features," Advances in neural information processing systems, vol. 32, 2019.
[6] T. Gu, B. Dolan-Gavitt, and S. Garg, "Badnets: Identifying vulnerabilities in the machine learning model supply chain," arXiv preprint arXiv: 17Xx::/708.06733 2017.
[7] H. Wei, L. Feng, X. Chen, and B. An, "Combating noisy labels by agreement: A joint training method with co-regularization." in Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2020, pp. 13 736-133 735. 1,3
[8] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, "Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels," in International conference on machine learning. PMLR, 2018, pp. 2304-2313.
[9] C. Guo, B. Zhao, and Y. Bai, "Deepcore: A comprehensive library for coreset selection in deep learning," in Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I. Springer, 2022, pp. 181-195. 3,
[10] Z. Borsos, M. Mutny, and A. Krause, "Coresets via bilevel optimization for continual learning and streaming," Advances in Neural Information Processing Systems, vol. 33, pp. 14 879- 14 890, 2020. 3,
[11] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon, "An empirical study of example forgetting during deep neural network learning," arXiv preprint /rX2v:/8/2.000 02/29, 2018. 1, 2,3
[12] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, "End-to-en incremen- tal learning," " in Proceedings of the European conference on computer vision (ECCV), 2018,
pp. 233-248. 3
[13] R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, "Gradient based sample selection for online continual learning," Advances neural information processing systems, vol. 32, 2019. 1,3
[14] O. Sener and S. Savarese, "Active learning for convolutional neural networks: core-set approach," arXiv preprint arXiv: /708. 00489, 2017. 2,
[15] S. Chatterjee and A. S. Hadi, "Influential observations, high leverage points, and outliers in linear regression," Statistical science, pp. 379-393, 1986. 1, 2,
[16] D. Cook, "Assessment of local influence." " Journal of the Royal Statistical Society: Series (Methodological), vol. 48, no. 2, pp. 133-155, 1986.
[17] W. Thomas and R. D. Cook, "Assessing influence on predictions from generalized linear models," Technometrics, vol. 32, no. 1, pp. 59-65, 1990.
[18] B.-C. Wei, Y.-Q. Hu, and W.-K. Fung, "Generalized leverage and its applications," Scandina- vian Journal of statistics, vol. 25, no. pp. 25-37, 1998.

---

[19] P. W. Koh and P. Liang, "Understanding black-box predictions via influence functions," in International conference on machine learning. PMLR, 2017. pp. 1885-1894. 2, 3,5
[20] A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov, "Scaling up influence functions," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, 2022, pp. 8179-8186.
[21] H. Guo, N. F. Rajani, P. Hase, M. Bansal, and C. Xiong, "Fastif: Scalable influence functions for efficient model interpretation and debugging," arXiv preprint arXh::2012.1578: 2020.
[22] S. Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and Li, "Dataset pruning: Reducing training data by examining generalization influence," arXiv preprint arXiv:2205.0983 093.29999, 2022. 2,
[23] S. Kong, Shen, and L. Huang, "Resolving training biases via influence-bain data relabeling," in International Conference on Learning Representations, 2022. [Online]. Available: http://ppnneeeeeeeee//ooonnndddd
1, 2.5.
[24] G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger, "Identifying mislabeled data using the area under the margin ranking," Advances in Neural Information Processing Systems, vol. 33, pp. 17 174417700 056, 2020. 1, 3
[25] M. Welling, "Herding dynamical weights learn," " in Proceedings of the 26th Annual Interna- tional Conference on Machine Learning, 2009, pp. 1121-1128. 2,3
[26] M. Paul, S. Ganguli, and G. K. Dziugaite, "Deep learning data diet: Finding important examples early in training." Advances in Neural Information Processing Systems, vol. 34, pp. 20 1996-20607 2021. 2, 5,
[27] G. Pruthi, F. Liu, Kale, and M. Sundararajan, "Estimating training data influence by tracing gradient descent," Advances in Neural Information Processing Systems, vol. 33, pp. 19 920- 930 2020. 3
[28] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "icarl: Incremental classifier and representation learning," in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001-2010. 1,2,3
[29] D. Feldman and M. Langberg. "A unified framework for approximating and clustering data," in Proceedings of the forty-third annual ACM symposium on Theory of computing, 2011, pp. 569-578 1,3
[30] J. Ju, H. Jung, Y. Oh, and J. Kim, "Extending contrastive learning to unsupervised coreset selection," IEEE Access, vol. 10, pp. 7704-7715, 2022.
[31] J. Huggins, T. Campbell, and T. Broderick, "Coresets for scalable bayesian logistic regression," " Advances in Neural Information Processing Systems, vol. 29, 2016.
[32] T. Campbell and T. Broderick, "Automated scalable bayesian inference via hilbert coresets," The Journal of Machine Learning Research, vol. 20, no. I, pp. 551-588, 2019.
[33] Nguyen, Y. Li, T. D. Bui, and R. E. Turner, "Variational continual learning," arXiv preprint arXiv: 17 10628, 2017.
[34] S. Farquhar and Y. Gal, "Towards robust evaluations of continual learning," arXiv preprint 1rXiv:1805.099) 2018.
[35] S. Kim, S. Bae, and S.-Y. Yun, "Coreset sampling from open-set for fine-grained self-supervised learning." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7537-7547. 3
[36] Y. Yang, H. Kang, and B. Mirzasoleiman, "Towards sustainable learning: Coresets for data- efficient deep learning." arXiv preprint arXiv:2306.01244, 2023. 1,3
[37] S. Pan and Q Yang, "A survey on transfer learning." " IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345-1359, 2010. 2, 4
12

---

[38] L. Torrey and J. Shavlik, "Transfer learning," in Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. IGI global, 2010, pp. 242-264.
[39] L. Yang, S. Hanneke, and J. Carbonell, "A theory of transfer learning with applications to active learning," Machine learning, vol. 90, pp. 161-189, 2013. 2,4
[40] B. Sorscher, R. Geirhos, S. Shekhar, Ganguli, and A. S. Morcos, "Beyond neural scaling laws: beating power law scaling via data pruning," in Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: http:///peenrreieeeeeefooommttap
2
[41] S. Jain, Salman, A. Khaddaj, E. Wong, S. M. Park, and A. Madry, "A data-based perspective on transfer learning," arXiv preprint wrxiv:220..057335 2022. 2, 3, 4, 7, 7,9, 18
[42] P. Sattigeri, S. Ghosh, I. Padhi, P. Dognin, and K. R. Varshney, "Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting." " arXiv preprint arXiv:2212.0688 2022. 2
[43] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations"" in International conference machine learning. PMLR, 2020, pp. 1597-1607. 2, 4
[44] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1729-9778..
[45] X. Chen, H. Fan, R. Girshick, and K. He, "Improved baselines with momentum contrastive learning," arXiv preprint arXiv:2003.04297 2020. 7
[46] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, "Big self-supervised models are strong semi-super learners," Advances neural information processing systems, vol. 33, pp. 22 243 22 255, 2020. 4
[47] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3733-3742.
[48] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and Isola, "What makes for good views for contrastive learning?" Advances in neural information processing systems, vol. 33, pp. 6827-6839, 2020.
[49] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, Avila Pires, Guo, M. Gheshlaghi Azar al., "Bootstrap your own latent-a new approach to self-super learning," Advances in neural information processing systems, vol. 33, pp. 21 271-21 284, 2020.
[50] Y. M. Asano, C. Rupprecht, and A. Vedaldi, "Self-labelling via simultaneous clustering and representation learning." arXiv preprint 1rXiv:1911.05337,, 2019.
[51] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments." Advances in neural information processing systems, vol. 33, pp. 9912-9924, 2020.
[52] J. Li, P. Zhou, C. Xiong, and S. C. Hoi, "Prototypical contrastive learning of unsupervised representations." arXiv preprint vrXiv:2055.04966. 2020. 2, 4
[53] V. Feldman and C. Zhang, "What neural networks memorize and why: Discovering the long tail via influence estimation." Advances in Neural Information Processing Systems, vol. 33, pp. 2881-2891, 2020. 3,
[54] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp- 770-778. 2. 6,
13

---

[55] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., "Imagenet large scale visual recognition challenge," Interna- tional journal of computer vision, vol. 115, pp. 211-252, 2015. 2, 6,
[56] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, "Cats and dogs." in 2012 IEEE conference on computer vision and pattern recognition. IEEE, 2012, pp. 3498-3505. 2, 5, 22
[57] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, "3d object representations for fine-grained categorization." in Proceedings of the IEEE international conference on computer vision workshops, 2013, pp.
2,7
[58] H. Salman, A. Iyyss, L. Engstrom, A. Kapoor, and A. Madry, "Do adversarially robust imagenet models transfer better?" Advances in Neural Information Processing Systems, vol. 33, pp. 3533-3545, 2020. 3,4.
[59] P. K. Agarwal, S. Har-Peled, and K. Varadarajan, "Approximating extent measures of points," Journal of the ACM (JACM), vol. 51, no. 4, pp. 606-635, 2004. 3
[60] S. Har-Peled and S. Mazumdar, "On coresets for k-means and k-median clustering," in Proceedings of the shirty-sixth annual ACM symposium on Theory of computing, 2004, pp. 291-300.
[61] D. Feldman, M. Schmidt, and C. Sohler, "Turning big data into tiny data: Constant-size coresets for k-means, pea, and projective clustering," SIAM Journal on Computing, vol. 49, no. 3, pp. 601-657, 2020. 3
[62] S. Basu, P. Pope, and S. Feizi, "Influence functions in deep learning are fragile," arXiv preprint arXiv:2006. 14655,,,,,4,,,,, 2020.
[63] J. Jia, Y. Zhang, D. Song, S. Liu, and A. Hero, presserrnn lifelong learning via dataset condensation," in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5.
[64] Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan, "Active finetuning: Exploiting annotation budget in the pverraniigg-finetuiiin paradigm," arXiv preprint 2rXiv:2303.14382 2023. 3
[65] A. Ghorbani and J. Zou, "Data shapley: Equitable valuation of data for machine learning," in International Conference on Machine Learning. PMLR, 2019, pp. 2242-2251.
[66] R. Jia, D. Dao, B. Wang, F. A. Hubis, Hynes, N. M. Gürel, B. Li, C. Zhang, Song, and C. J. Spanos, "Towards efficient data valuation based on the shapley value," in The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019, pp. 1167-1176.-
[67] R. Jia, D. Dao, B. Wang. F. A. Hubis, N. M. Gurel, B. Li, C. Zhang, C. Spanos, and D. Song, "Efficient task-specific data valuation for nearest neighbor algorithms," arXiv preprins arXiv:19088 08619, 2019. 3
[68] J. Lin, A. Zhang. M. Lécuyer, J. Li, A. Panda, and S. Sen, "Measuring the effect of training data on deep learning predictions via randomized experiments," in International Conference on Machine Learning. PMLR, 2022, pp. 13 13468-13 504.
[69] K. Guu, A. Webson, E. Pavlick, L. Dixon, Tenney, and T. Bolukbasi, "Simfluence: Modeling the influence of individual training examples by simulating training runs," arXiv preprint arXiv:2303.08/ 14, 2023.
[70] Z. Hammoudeh and Lowd, "Identifying a training-set attack's 's target using renormalized influence estimation," in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2022, pp. 1367-1381.
[71] S. M. Park, K. Georgiev, A. Ilyas, G. Leclere, and A. Madry, "Trak: Attributing model behavior at scale," arXiv preprint arXiv:2303. 2023,
14

---

[72] H. A. Just, F. Kang, J. T. Wang, Y. Zeng, M. Ko, M. Jin, and R. Jia, "Lava: Data valuation without pre-specified learning algorithms," arXiv preprint arXiv:2305.00033 2023,
[73] A. Ilyas, S. M. Park, L. Engstrom, G. Leclere, and A. Madry, "Datamodels: Predicting predictions from training data," arXiv preprini urXiv:2202.0062) 2022. 3
[74] C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, "Representer point selection for explaining deep neural networks," Advances in neural information processing systems, vol. 31, 2018. 3
[75] Y. Song and Ermon, "Generative modeling by estimating gradients of the data distribution." Advances in neural information processing systems, vol. 32, 2019. 3
[76] 7.. Deng, I.. Zhang, K. Vodrahalli, K. Kawaguchi, and J. Y. Zou, "Adversarial training helps transfer learning via better representations," Advances Neural Information Processing Systems, vol. 34, pp. 25 179-25 191, 2021. 4
[77] H. Liu, S. M. Xie, Z. Li, and T. Ma, "Same pre-training loss, better downstream: Implicit bias matters for language models," arXiv preprint arXiv:2210.14199, 2022. 4
[78] H. Liu, M. Long, J. Wang, and M. I. Jordan, "Towards understanding the transferability of deep representations" arXiv preprint arXiv: 12001.2031, 2019. 4, 5,
[79] M. Rosenstein, Z. Marx, L. P. Kaelbling, and G. Dietterich, "To transfer or not transfer," in NIPS 2005 workshop on transfer learning, vol. 898, no. 3, 2005. 4
[80] L. Duan, D. Xu, and S.-F. Chang. "Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach," " in 2012 IEEE Conference on computer vision and pattern recognition. IEEE, 2012, pp. 1338-1345.
[81] L. Ge, J. Gao, H. Ngo, K. Li, and A. Zhang, "On handling negative transfer and imbalanced distributions in multiple source transfer learning," Statistical Analysis and Data Mining: The ASA Data Science Journal, vol.
4, pp. 254-271, 2014.
[82] J. Yosinski, Clune, Y. Bengio, and H. Lipson. "How transferable are features deep neural networks?" Advances in neural information processing systems, vol. 27, 2014.
[83] Z. Cao. M. Long, J. Wang, and M. I. Jordan, "Partial transfer learning with selective adversarial networks," " Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2724-2732.
[84] Z. Wang, Z. Dai, B. Póezos, and J. Carbonell, "Characterizing and avoiding negative transfer." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11 293-11 302.
[85] D. Guo, A. M. Rush, and Y. Kim, *Parameter-efficient transfer learning with diff pruning," arXiv preprint arXiv:2012.07466 2020. 4
[86] B. Liu, Y. Cai, Y. Guo, and X. Chen, "Transtailor: Pruning the pre-trained model for improved transfer learning," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 10, 2021, pp. 8627-8634.
[87] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, M. Carbin, and Z. Wang, "The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021, pp. 16 13306-16336...
[88] T. Chen, Y. Cheng, Z. Gan, L. Yuan, L. Zhang, and Z. Wang. "Chasing sparsity in vision transformers: An end-to-end exploration," Advances in Neural Information Processing Systems, vol. 34, pp. 19 974- 988, 2021.
[89] S. Myung, I. Huh, W. Jang, J. M. Choe, J. Ryu, D. Kim, K.-F. Kim, and C. Jeong, "Pac-net: A model pruning approach to inductive transfer learning," International Conference on Machine Learning. PMLR. 2022, pp. 16 250-16222. 4
15

---

[90] H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola, "Exploring visual prompts for adapting large-scale models," arXiv preprint arXiv:2203.17274, vol. 1, no. 3, p. 4, 2022.
[91] G. F. Elsayed, I. Goodfellow, and Sohl-Dickstein, "Adversarial reprogramming of neural networks," arXiv preprint /rXiv:/86.1114400 2018.
[92] P.Y. Chen, "Model reprogramming: Resource-efficient cross-domain machine learning," arXiv preprint arXiv:2202.10629 2022.
[93] Y. Zheng, X. Feng, Z. Xia, X. Jiang, A. Demontis, M. Pintor, B. Biggio, and F. Roli, "Why adversarial reprogramning works, when it fails, and how to tell the difference," arXiv preprint arXiv:2108. /673, 2021.
[94] P. Neekhara, S. Hussain, S. Dubnov, and F. Koushanfar, "Adversarial reprogramming of text classification neural networks," arXiv preprint arXiv: 829, 2018.
[95] P. Neekhara, S. Hussain, J. Du, S. Dubnov, F. Koushanfar, and J. McAuley, (Cross-modat ad- versarial reprogramming." in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 2427-2435.
[96] L. Chen, Y. Fan, and Y. Ye, "Adversarial reprogramming of pretrained neural networks for fraud detection," in Proceedings of the 30th ACM International Conference Information & Knowledge Management, 2021. pp. 1935-2939)
[97] A. Chen, Y. Yao, P.-Y. Chen, Y. Zhang, and Liu, "Understanding and improving visual prompting: A label-mapping perspective." arXiv preprini urXiv:2211.11633. 2022.
[98] A. Chen, P. Lorenz, Y. Yao, P.-Y. Chen, and S. Liu, "Visual prompting for adversarial robust- ness," " ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1-5. 18
[99] G. Zhang, Y. Zhang, Y. Zhang, W. Fan, Q. Li, S. Liu, and S. Chang, "Fairness reprogramming." arXiv preprint 220i:22009.0222,, 2022.
[100] G. Zhang, H. Zhao, Y. Yu, and P. Poupart, "Quantifying and improving transferability in domain generalization." Advances in Neural Information Processing Systems, vol. 34, pp. 10957-10 2021.
[101] Y. Zhang, P. Khanduri, Tsaknakis, Y. Yao, M. Hong, and S. Liu, "An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning," arXiv preprint arXiv:2308. 00788, 2023.
[102] M.-E. Nilsback and A. Zisserman, "Automated flower classification a large number of classes," in 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing. IEEE, 2008, pp. 722-729. 5,7
[103] M. Caron, P. Bojanowski. A. Joulin, and M. Douze, "Deep clustering for unsupervised learning of visual features," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 132-149. 6
[104] X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan, "Clusterfit: Improving general- ization of visual representations," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6509-6516
[105] L. Fan, S. Liu, P.-.. Chen, G. Zhang, and C. Gan, "When does contrastive learning preserve adversarial robustness from pretraining to finetuning?" Advances in Neural Information
Processing Systems, vol. 34, pp. 21 480-21 492, 2021. 6
[106] U. Evci, V. Dumoulin, H. Larochelle, and M. C. Mozer, "Head2toe: representations for better transfer learning," in International Conference on Machine Learning. PMLR, 2022, pp. 6009-6033.
Utilizing intermediate
[107] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures in the wild," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 3606-3613,
16

---

[108] K. Soomro, A. R. Zamir, and M. Shah, "Ucf101: A dataset of 101 human actions classes from videos in the wild," arXiv preprint arXiv:1212.0402, 2012.
[109] L. Bossard, M. Guillaumin, and L. V. Gool, "Food-101-mining discriminative components with random forests," in European conference computer vision. Springer, 2014, pp. 446 166-661.7
[110] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "Sun database: Large-scale scene recognition from abbey zoo," in 2010 IEEE computer society conference computer vision and pattern recognition. IEEE, 2010, pp. 3485-3492. 7
[111] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," cS.utoronto.co 2009. 7
[112] Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x 16 words: Transform- ers for image recognition scale," arXiv preprint / 1929, 2020. 7, 10, 18
[113] J. Frankle and M. Carbin, "The lottery ticket hypothesis: Finding sparse, trainable neural networks," arXiv preprint arXiv: 00635,, 2018. 7
[114] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin, "The lottery ticket hypothesis for pre-trained bert networks," Advances in neural information processing systems, vol. 33, pp. 15 834-15 846, 2020.
[115] Y. Zhang, Y. Yao, P. Ram, Zhao, Chen, M. Hong, Y. Wang, and S. Liu, "Advancing model pruning via bi-level optimization," arXiv preprint arXiv:2210.04092, 2022. 7
[116] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint arXiv: 06083, 2017. 9
[117] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, Malik, and C. Feichtenhofer, "Multiscale vision transformers," in Proceedings of the IEEE/CVF international conference computer vision, 2021, pp. 6824-6835. 19
[118] D. Hendryeks and T. Dietterich, "Benchmarking neural network robustness to common corrup- tions and perturbations," arXiv preprint arXiv: 1903. 12261, 2019. 21
[119] X. Zhai, Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. Pinto, M. Neumann, A. Dosovitskiy et al., "A large-scale study of representation learning with the visual task adaptation benchmark." arXiv preprint (xX.v:1910.4440
2019. 22
[120] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, "On large-batch train- ing for deep learning: Generalization gap and sharp minima," arXiv preprint arXiv: (1166699944444 2016. 22
[121] D. Bisla, J. Wang, and A. Choromanska, "Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape." in International Conference on Artificial Intelli- gence and Statistics. PMLR, 2022, pp. 8299-8339. 22
[122] W. Maddox, G. Benton, and A. G. Wilson, "Rethinking parameter counting in deep models: Effective dimensionality revisited," arXiv preprint wXXv:2003.0213:: 2020. 22
[123] T. Liang, T. Poggio, A. Rakhlin, and J. Stokes, "Fisher-rao metric, geometry, and complexity of neural networks." in The 22nd international conference on artificial intelligence and statistics. PMLR, 2019, pp. 088-896.7 22
[124] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." Journal of machine learning research, vol. 9, no. 11, 2008. 23
17

---

Appendix
A Experiment Settings
A.1 Training Settings
General setups for DP (dataset pruning). Regardless of the choice of the downstream tasks and DP methods, each DP process will go through a three-stage paradigm, namely ! source dataset pruning, pretraining based on the pruned source dataset, and finetuning on the target dataset using either LP (linear probing) FF (full finetuning). Regarding baseline methods for source dataset pruning, we strictly follow the baselines' settings provided in the literature. As stated before, utilize small surrogate model (ResNet- 18) pretrained on the full ImageNet to conduct DP. The proportion of pruned class numbers to the total source class numbers determines pruning ratios for methods: LM (label mapping) and FM (feature mapping). To ensure fair comparison with other non-class-win wiss baselines (RANDOM, GRAND, MODERATE, ATF), the same pruning ratio is applied to the total number of training data points.
Pretraining setups. We keep our pretraining procedure consistent for all models and strictly follow the settings released by the existing work [41]. All the models are trained from scratch using Stochastic Gradient Descent (SGD) to minimize the standard cross-entropy loss in a multi-class classification setting. We use batch size of 1024, momentum of 0.9, and weight decay of 5 x 10 The training utilizes a cyclic learning rate schedule, which begins with an initial learning rate of 0.5 and peaks at the second epoch. During training, we incorporate data augmentations such as random resized cropping and random horizontal flipping.
Downstream task finetuning settings. Our approach to finetuning the pretrained model on down- stream tasks involves using LP and FF. Details of the downstream datasets and the training configura- tions are presented Tab. A1, following [98]. For LP, we employ the Adam optimizer, a multi-step decaying scheduler, and initial learning rate of 0.1 across 50 total training epochs. As for FF, utilize the Adam optimizer over 200 epochs with a cosine-annealing scheduler, an initial learning rate of 0.01, and weight decay of 5 10 All finetuning experiments employ batch size of 256 and standard data augmentations, such as random resized cropping and horizontal flipping.
Training settings for SSL (self-supervised learning) and VIT. Our SSL training settings follow the configurations provided by MoCov2. Details of the pretraining and finetuning stages can be accessed at https: //gi /gi thub. cww/cooo//ooooo wcc.oookeeeeaarccccooooooo cm/oooo. For the training of ViTs, we rely on the setting released in the original ViT paper [112] (see ViT/B in Table 3).
Table Al: Dataset attributes and training configurations of downstream image classification datasets considered this work.
Dataset
Train Size
Test Size
Class Number
Batch Size
Rescaled Resolution
Flowers 102
4093
2463
102
224x:244
DTD
2820
1692
47
256
224x224
UCF101
7639
3783
101
256
224x224
Food101
50500
30300
101
256
224x224
OxfordPets
29444
3669
37
256
224x224
StanfordCars
6509
8041
196
256
224x224
SUN397
15888
19850
397
256
224x224
CIFAR1O
50000
10000
10
256
160x160
B Additional Results
Expanded performance evaluation of in-domain DP methods across all downstream datasets. Fig. A1 expands on the performance comparisons in Fig. 2, providing more thorough evaluation of various in-domain DP methods on eight downstream datasets. The trends observed are consistent with those in Fig. 2: Random pruning shows strong baseline method in DP for transfer learning compared to other state-of-the-art DP methods designed for non-transfer learning. This observation prompts us to explore more effective dataset pruning strategies for transfer learning, MODERATE and
18

---

GRAND are also demonstrating strong baselines, motivating us to choose them as the default DP baselines in Section 5.
Targel Toa Aserracy M5
Flowers
SUND97 LP
Food
Tacse Tost Aonuaay (N)
DTD
94
-2
Targe Tes Agrzaoo
50
Aveuracy
555
-0
Rasden
Midraas
30
98
Guidd
3F
-6
---3o Proce
20%
50%
20% 40%
80%
0%
25%
40% 69%
80%
Data Priming
Source Duta Pruning
Pruning Ratio
Source Pruning
Tirget Accuracy (50)
5--
StanfordCss-LP
Tnnget Test Accurmyy (%)
UCFI0I-LP
(96)
15
CIFARIO-LP
2000
OxhardPet-LLP
15
86
10
64
R2
8n
16
0%
20%
40%
60% B0h/
0%
80% Source Data Prunting Ralio
20%
40%
60%
0%
201
40%
60% 80%
016
20%
40%
60%
60%
Source Data Pruning Ratio
Source Data Pruning Kalio
Source Data Pruning Ratio
Figure A1: Extended performance comparison of different io-domain DP methods for transfer learning across all downstream tasks. Experiment settings are consistent with Fig.4.
The main results reported in Fig. 4 are stable: analysis with detailed numerical results and standard deviations. In Tab. A2, we provide the exact numerical results used to generate Fig, 4. These numbers give a more granular view of the performance comparisons. The results, calculated over three independent trials, show that the magnitude of the standard deviations is quite small relative to the mean values. This indicates that the trends and conclusions drawn from Fig. 4 are generally valid and not significantly affected by trial variations. To maintain readability and clarity, we choose not include these minor standard deviations in Fig.
OxfordPets.
1lowers102-LP 102-LP
Target Test Accuracy (5%
85
Target Test Accuracy
60
84
50
Moderate
83
GraNd
40
(2000 Cluster)
(1000 Clustur) Cluster)
30
81
No Pruno
2030%
60%
70%
80%
50%
60%
70%
80%
Source Data Pruning Ratio
Source Data Pruning Ratio
Figure A2: Ablation study on the sensitivity FM against the choice of cluster numbers. For FM, the original dataset is pre-clustered into K clusters, with K = 2000 as the default setting in this study, Here, various K values (500, 1000, and 2000) studied in this experiment. Other settings are aligned with Tab. 2.
FM performance is robust to the choice of cluster number. The results presented in Fig. A2 provide insights into the sensitivity of FM performance to variations of the cluster number (K). This extended study from Tab. shows that FM performance remains robust as the cluster number varies from 500 2000, preserving the benefits of DP for transfer learning without much perfor- mance degradation. This implies that FM can provide reliable and stable results even when the hyperparameter settings are not optimal.
LM identifies the most influential source classes. To validate that the classes with the highest LM scores are the most influential, we present the pruning trajectory in Fig. A3, where pruning is executed in the reverse order of the class scores, different from the proposed DP implementation. That is, classes with the smallest scores are retained, while the ones with the highest scores are pruned. Remarkably, even a slight pruning ratio (such as 10%) leads to a significant degradation in downstream performance. This evidence underscores the benefits of source classes with high LM scores in promoting the downstream performance.
FM can be smoothly applied MoCov3. We conducted experiments to illustrate the effectiveness of our proposed method (FM) when applied to the more recent SSL framework MoCov3 [117] using
19

---

76.294N22
88855118
90102100
489771935
$1,211022
St 51.004000
525449200
54261n24
55.074032
60015-622
32.76411.2
29.97440330
42877003
42.00:
58.0660.4
58,70LI
0495H155
K2451003
82.2 82,721032
33.02543311
09.28--.39
900
500
90
82.313002
28.3-0018
10
09.0220.33
10.05.0111
49 49.3--0-
516:10.0
5443:0:
53
6111500:2
57.03-0-33
28.22.0.28
55000002
00
00.2-0018
16-0-00.1
05
45:45:00244
61
63 68-00-33
57.4610338
35:00:3
84.741033
M662-0022
05
91 $1.52000000
90 00--0-55
91.99 91.96-0-22
05
49.65
40
00
61
84
85.95
520
03
31
90 38
90 50.32003
10110000
50,16-033
49.72-0018
51.0000
5522-005
55.5 10
61.51 0151:20300
61.38-0211
5049-035
58.2-00 -032-033
5727-50
62.17-0000
62.07-03
46300-0018
58.9 10:6
0877-033
60.3000
20.2
85:6-1022 20
87 57:0--01
86 00.30-0330
11,51=0
03.3 38
93. 93.50
75%6
57.81
4647
6463--0
84.75-50
49.11
65.18
83
84.610114
901 01
80/7880010
49.880022
09.441011
51.8
57178021
021441030
61.7150332
90577031
9975505
63.8800335
613009033
38000.00
464500038
040720330
6601
617810000
0044012
MK 26
06 7210-33
47-3--200
67.3910335
92.7740322
0261+00-28
02.45-0388
8054101111
97 34.34422
7006
5%
85
83.50
59-001
00-
89-0-11
00.02:003
0.07
80.3--0.35
50.02
51168-018
51.57.0225
55.15
38.000300
58 19-00-2
62.78-0.200
01.02-0335
62.00-02
025
60 20
6 64240032
63.
41.00-0355
11.09-0:21
37-71-0.28
47. 07.56-0:20
06.930031
65.98-0.20
6000300.2
63.2800311
70:9-0022
36 38. 0.24
86 06
88 05
M8 03-33-032
07.79-0.227
03-26-022
0322
93 02
05 42-00331
60
50.41
89
90
50.
99
86992002
56-11 6131-0337
0.17
50.28-011
51.42.030
0.30
59.00 0:30
5838-033
59.12-033 0:33
6308 0:36
62.55-16
6133 0224
0141=030
05:00
10
0.20
43,61 -02
4100000
4733-0000
0700-0024
66388 0
62.20 62.1000300 0:30
71:000038
69766663
6331
0:30
57.30
0
0.30
93.900031
93.34 9334-0-31
9618-022 02
9000
50.33
S1000
51.42
64.5
4633
56.1
57.11
13
88
93.99
Table A2: Exact numbers and standard deviations for Fig. 4.
5516
29
009220111
55.94201111
50.0000.30
36
50N7511-22
49:8110-3
51.2210227
601910::1
50 224226124
50.22-0.25
62.091112
6300600222
0.955-929
630010-32
47
65 0010.2
6678-1122
43.74811115
57-210-30
47.08200:00
67.78-0331
41-2111235
717164113
7.001022
26.7710035
67.751022 21
5661
20
93 262600 28
9462100-0
94325022
88
86,
88
87 27
87 87.03-0000
89 25
09.72
50.31 0.27
50,28_00:3
01.610022
80.50_...1
6011::022
60 60.5-024
63-47003
02.35-033
63.21 0:38
62 0245-016
66 25
66. 66.77-033
22:00-022
43,030033
40002030
07227-029
022
69 002
66, 00331
71 11:0-033
11.35
87. 87.38 0.24
86 22
87 27
88 04:74-0300
88 09 0.36
03 03.300021
03.71-0277
03
233
300%
60
89
50
48
94
Priming Raiio 45%
58.220005
87.79-0-55
$899
89-46-0-10
00.36-00066
0..0
5146-933
51.5
60.1
1.020
6160-0.2
6324-022
62.73:032
61:00-036
6310
66.39-0000
66.22-017 -0,2
4326-n226
4547:0030
41.40:124
4763-0228
6900 00:20
7031--039
62.14.026
7251-022
72162:028
SK .0:30
88337-0:32
5881-033
8941
0000
30
04:00000000
SUNS97 LP
50.00
SHLAM
50.8
Food101 LP
6035
DTD LP
LP
11P
1PP
Oxforileas-
SmaffrdCann
UCF 101
CIARRO
Flowess022
83 42.4-1
59 21
06.2240.22
27
41.32.1.. 28
16.668011
50.6810-32
61 12.12
39
61 61665922
61 01.2330.51
63 29.0910.29
63 63.3480-22
08
64 61.8100
64 61
22
66 66.301021 21
46 18
46 46.3310-52
46002--228
47. 17.361922
42.41200
28
27
70 00-7---.20
68. 28.22222222
72-8840-20
7L.970033
38.41 24
29
27
240-0
98.221/333
24
400
05
88
80,
51
20,
59
800
89
94
94
92:033
88.22 74.22004
89 72:0015
89 59.895013
51.110034
51 34.441122
311740113
51.74.024
61 01331031
61 112660300
03:27-031
63:912225
03.20:0335
04.42-0222
6442:016
07433022
6655-023
45, 35
45:34:1222
48 78
47. 30
20.5 22.54-022
70 18-0220
72.000011
26
89 33
19 14-0000
892330300
93 93.962999
044-0-38
04.200277
94 34-633
35%
89
04
$863-00335
10
598330037
50.8 10.08
51.00 11.00-10000
51.7910:000
62.550.27
0200-0.24
61.00 1100:0034
6350
633420227
662550.38
6613-121
4663-1-2
46.770-00
45.00:0-00
47.40:00222
48.1230.18
000020:31
70.08 00.00-0.22
0833
72-58-0-3
22
8817110:31
88.2251020
26
$89610020
10.31 -0
592253331
93.9900-33
9423310-2
38
94 24020-0030
301
2506
20-2666333
11
89 39
50.02:0022
22
51 11.06.0011
51 11-45801200
11.2241322
317010057
62.3340177
03455:3366
63 60.25:0227
63.301111
05,77-3300
03.401222
65 23
330210333
66.1--022
46, 00:00:0000
06.0360222
45. 45.5646166
46.113200
47:2520332
20.37:3124
70.1 ML13-3.25
69.211044
72,09950220
718266..8
31
38 08.025122
83. 10
59 40-003
940346033
0468--3330
04 01
58.27
89.
58
04
88.91 98.900001 14
20
51, 51.161032
51.73-0077
31.420011
51 50
62 52.53
02 22.221016
02
63.32033
05.11038
65666026
68 5110:2
25
06. 66.22-0116
46 36.3111199
1210033
21:2122099
22.00:00300
00.68-033
70 00.980029
72 72-200300
02.500032
NX 922-0-20
05 08.550006
28
00.79-0227
W94710020
29
94 222320020
94, 221000000
71.71-0039
32
20%
94.
593620115
39.781222
70
51.23100277
11.368n.00
31.8100188
62.2200.8
6249901:2
62 62.301053
630011111
63
65.21101
65962011
664120022
66679130
46.2200211
163510:30
461220
37.2210:30
45.09200:33
20.5360 25 21.2200:3
71.5880.38
1100010:00
7277200-5
23144:3:0
38.661027
2874
88 20
9942H133
39
34
24
94 40-00
15%
89
10PS
29
59:50
09.7860110
01
51,23.0.3
51, 311756035
S1
62 21
21
23.343022
2.
62. 20:00:
27
63.295021
17
(3.
22
22
21
06
67.8-0332
32
40
48 60100000
47.98220.3
10
71 11-56-0.20 20
71.92.0
73.5-00220
72.8360
38-5550. 22
55 03:530002 21
31
89,32 09.32.0.00
31
94. 24 94.25
24
04 08.998008
33
39.78
89
89,
89 67200 56
89 20
60.72-0111
90 30
51.2510.14
S1 88.18 26
62 10,2210.30
30
02 19.40.111
62.70 08.00:0.18
63. 17.1116
06 18
06 21
06,7310 17
20
20
26.4761-221
47 871001000 28
47 09520122222
22
71 31.6600331
72.72 72.72-0.24 24
21.8211222
73 12,52-00-0
72-72600322
589.0024
HP 17-17511:00 00
89 35.5510000 20
80 10.5::00.0
89 $3
28
94.1-10-31
26
94 36600
Method
RASDOM
METEERATE
GRAND
LM
FM
RANDOM
GRAND
LM FM
RANDOM
MEDDRAAT
GRAND
LM
RANNOM
MODERATE
GRAND
LM
FM
RANDOM
GRAND
IM
FM
RANDOM
MODERAATE
GRAND
LM
RANDOM
MODERATT
GRAND
LM
FM
RANDOM
MODERAAT
ORAND
LM FM
20

---

Oxt LP
OxfordPets FF
990
Target Test Accuracy (%)
Target Accuracy
80
85
70
80
60
Ordered
50
Reversed
75
No Prune
40
40% 60% 80% Source Datu Pruning Ratio
20%
0%
60% Source Data Pruning Ratio
20%
Figure A3: Performance comparison the downstream performance of LM with the "Ordered" and the "Reversed" pruning order. Here, the "Ordered" strategy retains source classes with highest scores, while "Reversed" order prunes these classes first. Other settings are aligned with Fig. 4.
E-Flatness
100%
75%
Ama((H)
FRN
%
25%
LPF
|H|
Dense
Moderate
LM
Random
GraNd
models pretrained with pruned source data using different methods. The models are first pretrained and then finetuned using LP. The flatness are evaluated with respect to the down- stream training loss and are quantified by the reversed sharpness evaluated through five widely acknowledged sharpness metries. The results normalized to 0% ~ 100%, with 100% denoting the highest (best) flatness across all the methods given one specific flatness metric.
Figure A4: Flatness evaluations
the ViT structure. In line with our SSL experiment plan detailed in Tab. 2 of the paper, Tab. A3 tested FM on three downstream datasets, specifically OxfordPets, Flowers102, and SUN397, with source data class pruning ratio ranging from 50% to 80%. The results affirm that the principal conclusions drawn from MoCov2 remain consistent with MoCov3 on ViT. Our method, FM, successfully identifies data subsets within the source dataset that can be pruned at high ratios without compromising downstream performance (termed as "winning subsets").. For instance, in one particular case, the FM-based winning subsets achieve a pruning ratio of up to 70% on OxfordPets.
Explore LM in the multi-task setting, A limitation of our method is its task-specific nature in data influence analysis. As our methods assess source data influence for specific downstream tasks, developing a universal DP solution for multiple tasks simultaneously is challenging. As preliminary study, we examine the performance source dataset pruning for multiple downstream tasks simultaneously in Fig. AS. While LM can still identify winning subsets, the maximum pruning ratio diminishes as more tasks are considered. We will include this limitation in the Conclusion section.
LM helps remove data biases through DP for a downstream task. To investigate the scenarios with data biases, we conducted experiments on CIFAR-10C (the out-of-distribution scenario) [1 1188. We first pruned ImageNet given CIFAR-10 10 as downstream task and evaluated the model on CIFAR- 10C with different corruption types. Tab. A4 shows LM results with different pruning ratios for strong perturbation types in CIFAR-10C. Impressively, LM can achieve winning subsets with pruning up to 80%, even better than on CIFAR- 10, confirming our method's effectiveness filter biased data points to some degree.
21

---

Table A3: Experiments with MoCov3 VIT. Other settings follow Tab. 2 of the original submission. The performance surpassing unpruned setting (pruning ratio 0%) is highlighted in cyan The result in each setting is marked in bold. FM consistently outperforms other baselines and can find winning subsets with pruning ratios more than 50%.
Dataset
OxfordPets
Pruning Ratio
0%
50%
60%
70%
80%
RANDOM MODERATE
82,13
80.27
75.42
68.34
86.17
85,29
84.01
81.32
87.34
GRAND
87.21
86.11
83.19
80.78
FM (ours)
87.68
87.51
87.39
84.14
Dataset Pruning Ratio
SUN397
0%
50%
60%
70%
80%
RANDOM
59.22
58.45
56.73
54.29
MODERATE
60.13
59.61
58.21
56.44
60.36
GRAND
60.39
59.27 39.27
58.95
57.15
(ours)
60.49
60.55
60.42
59.88
Dataset Pruning Ratio
Flowers102
0%
50%
60%
70%
80%
RANDOM
92.41
91.65
90.17
88.41
MODERATE
93.75
92.41
91.42
90.11
GRAND FM (ours)
93.88
93.21
91.77
90.45
94.11
94.28
93.97
91.42
Dataset: OxfordPels
Dalaset: DTD
Target Accuracy (%)
91
Target Test Accuracy (%
-90
89
88
87
No Prune Single-Task
$5
No Prune Single-Task
86
Multi- Task
50
Multi-Task
0%
20%
40% 60%
80%
0%
20% 40% 60% 80%
Source Pruning
Source Data Pruning Ratio
Target Accuracy (%)
Dalaset: SunfordCars
Target Test Accuracy (%)
56
Dataset: SUN397
89
88
52
50
48
86
No Prune
No Prune Single-Task
Single-Task Task
85
Multi- Task
Multi-Task
42
0% 20%
40% 60% 80%
0%
20%
40% 60% 80%
Source Data Pruning Ratio
Source Data Pruning Ratio
Figure AS: DP achieved by the multi-task setting given 4 downstream tasks. This expands Fig. 4, i.e.
the single-task setting, where source data is pruned based on an individual task.
Experiments on the few-sho benchmark VTAB [119]. We extend our experiments include the VTAB benchmark [1 19]. As shown in Tab. A5, even in the few-shot setting, our major conclusions remain valid. LM can effectively pinpoint the most influential source subsets and climinate the less valuable ones. For most tasks considered, winning subsets with pruning ratios up to 40% are successfully found.
Subsets obtained by LM improve the flatness of loss landscape on downstream tasks. Fig. A4 evaluates the flatness of the loss landscape of the model pretrained on 80%-pruned source dataset and subsequently finetuned on the downstream OxfordPets dataset [56]. Better flatness in the
loss landscape is typically associated with better transferability. We quantify this flatness using the measure reversed sharpness calculated via five widely accepted metries: e-sharpness [120], low pass filter-based measure (LPF) [121] J. the max eigenvalue of the Hessian (Amax (H)) [122], the Frobenius norm of the Hessian ( (H(r) r) [122], and Fisher Rao Norm (FRN) [123]. The results suggest that pretraining on the subset selected by LM leads the model towards flatter region in
22

---

Table Experiments on CIPAR-1OC. --sadddd source dataset pruning on ImageNet (given CIFAR- as the downstream task) applies to transfer learning against CIFAR 10C. 5 out of the 19 corruption types are tested.
Pruning Ratio
Dataset
0%
20%
40%
60%
80%
CIFARIO
96.83
96.88
97.03
96.57
95.41
Gaussian Noise
82.13
82.67
82.89
82.60
81.19
Defocus Blur
84.73
85,22
85.36
84.92
82.75
Impulse Noise
84.62
85,21
84.78
85.93
85.11
Shot Noise
83.18
83,25
83.49
83.24
Speckle Noise
83.11
83,59
83.29
83.57
82.27
Table A5: Experiments on the few-shot transfer learning benchmark VTAB. Seven tasks the NATURAL set are studied following the setting of Fig. 4. Each task contains 800 training and 200 testing samples.
Caatechtor
CIFAR- 100
#llowersto
OcfordPeta
SUN
DTD
Mean
80.23
46.39
90.39
33.32
87.24
Pruning
46.65
62.88
90.78
88.57
34.22
87.45
Pruning
LM
Pruning 60%
78.65
45.51
62.57
90.12
87.35
33.57
86.38
69.16
Pruning 80%
73.31
41.33
61.18
89.42
85.11
31.49
84.22
66.58
the downstream loss landscape, potentially contributing to the superior transferability of LM when compared to the baseline methods.
Target Class
OxfordPets (Target Class)
Retained Source Class Pruned Source Class
American Abyssinian Hulldog
Shiba Inu
Retained Source Class
Black Swan
Indigo Bunting
English Springer
Source Class
Pruned
J0pe
Tennis
Loafer
Cab
Figure A6: (Left) Interpretation merit of the data pruning strategy by L.M. (Right) Feature distribution visualiza- tion using t-SNE source class selection by LM for OxfordPets with 90% pruning ratio.
Feature distribution analysis. Fig. A6 provides visual explanations of DP both deep repre- sentation (feature) and data levels. Here we focus on the LM method with pruning ratio 90% given the downstream dataset OxfordPets. The other settings are consistent with Fig, In Fig. A6 (right), we visualize the source ImageNet classes (including pruned ones and retrained ones) and target OxfordPets classes in terms of their class-wise feature centroids in a 2D space achieved by 1-SNE[124]. The class feature centroid is obtained by averaging the features of data points within class, extracted by the pretrained source model on the full ImageNet dataset. retained souree classes are grouped together and are close to the target classes. This indicates that the source classes that the pruning process share the most resemblance with the target data. contrast, the pruned source classes are more dispersed and located further away from the target data classes. Furthermore, Fig. A6 (left) exhibits image examples of target classes as well as pruned and retrained source classes. We observe that image examples in the retained source classes (e.g., relating to
we can see, all
23

---

Table A6: Performance on surrogate model size. Experiments follow the setting of Fig. 4: RN-101 is first pretrained on the pruned source dataset (ImageNet) based on the surrogate model, and then finetuned on the downstream task OxfordPets. Under different surrogate models, the source class selection overlapping ratio with the used surrogate model RN-18 in the submission reported under 50% pruning ratio.
Surrogate Model
RN-11 (Defank)
RN-20s
VGG-2
RN-32s
VGG-4
RN-44s
RN-56s
VGG-8
Architecture
Param. #
0.236
0.417
0.516
0.698
0.706
0.896
5.53
11.69
Source Acc. (%)
36.25
22.56
40.77
29.44
43.74
45.72
58.45
68.73
Largest Pruning Ratio
of Winning Subsets (%%)
80
70
Source Class
Selection Overlap (%)
89.3
84.4
90.7
87.2
93.5
94.8
97.7
100
animals) are semantically closer to the target data points (relating to pets) than the pruned ones. This highlights the ability of LM effectively identify and retain the most relevant classes for the downstream tasks. We provide more examples for FM in Fig. A7.
Examining the top selected classes by FM and their image examples. In Fig. A7, showcase the top-10 source classes chosen by FM, as determined by the endowed scores. These selected classes closely correspond to the downstream datasets' subjects, demonstrating FM's effectiveness in identifying relevant classes for transfer learning. This finding also aligns with our observations in Fig, A6, showing that FM identifies source classes resembling downstream data.
24

---

Window
Honey Comb
Shower
Chainlink
Jigsaw Puzzle
Stone Wall
Prayer
Screen
Velvet
Curtain
Fence
Dishrag
Wool
Mat
DTD.
Cypripedium
Head
Cabbage Butterfly
Sulphur
Daisy
Bee
Pinwheel
Rosehip
Cardoon
Calccolus
Cabbage
Acron
Butterfly
Flowers
Chocolate Sauce
Mashed
Soup Bowl
Meatloaf
Burrito
Ice Cream
Potato
Hotpot
Hot Dog
Consomme
(c) Food
Egyptian Cat
Persian
Siamese Cat
Tabby
Cat
Chihuahua
Pomeranian
Pug
Samoyed
Basset
(d) OxfordPets.
Sports Car
Pickup Truck
Racing
Minivan
Wagon
Convertible
Grille
Jeep
Limousine
Taxi
Car
(e) Stanfordears.
Mobile Home
Worm
Restaurant
Lakeshore
Valley
Fence
Paluce
Desk
Terrace
Cinema
Prison
Sun397.
Balunce
Neck
Bars
Streicher
Stage
Barbershop Ballplayer
Beam
Barbell
Racket
Lakeshore
Brace
(g) UCF101.
Hair Spray
Fox
Croquet Ball
Band
Nail
Syringe
Whistle
Squirrel
Weasel
Aid
Lakeshore
(b) CIFAR-I4 10.
Figure A7: The source classes with top- 10 selected FM for the downstream tasks studied in this work. For each class, the class label (name) well representative image example is presented.
25


Reviews:
[{"review_id": "8DelxsPeC1", "reviewer": "Reviewer_RSvb", "summary": "The following paper tackles the problem of dataset pruning from the lens of transfer learning, where the main goal is to achieve lossless or improved transfer learning accuracy of the source model on a target task using less amount of source data. To achieve this, this paper proposes two dataset pruning algorithms, Label Mapping (LM) for supervised pretraining and Feature Mapping (FM) which is capable of finding winning subsets within source dataset for self-supervised pre-training by monitoring the model's responsiveness toward target data samples. Experiments show that incorporating LP and FM on the source dataset by 40-80% pruning ratio does not sacrifice the classification performance on downstream tasks while speeding up the pretraining process by a factor of 2-5. Furthermore, additional experiments on adversarial pretraining tasks show classification performance improvement of 1-2% on downstream tasks.", "questions": "As pointed in Weakness, would the transfer learning results based on self-supervised learning pretraining holds for Feature Mapping (FM) on other self-supervised learning methods, i.e. MoCov3 [1] or Barlow Twins [2]?\n\n[1] https://arxiv.org/abs/2104.02057\n\n[2] https://arxiv.org/abs/2103.03230", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(+) The Paper is well-written and easy to understand. \n\n(+) The methods are highly intuitive, simple, and efficient, making them accessible for general-purpose usage.\n\n(+) Experiments show promising results on various benchmarks with regards to various metric of interest not limited to downstream task performance, and pre-training time, and leads to a more efficient adversarial learning", "weaknesses": "(-) For self-supervised learning experiments, I think the experiment is quite limited, where only one self-supervised method and one network architecture is used for evaluation. Perhaps adding more relevant self-supervised learning architectures such as MoCov3 [1], which uses ViT, or information-maximization self-supervised learning methods such as Barlow Twins [2] would strengthen the paper.\n\n(-) Minor typos: \n\na) In pg. 2, finetue -> fine-tune\n\nb) In pg. 7, savings -> saved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments and a precise understanding of our contributions. **We use [W] and [Q] to refer to the specific points listed in the “Weakness” and “Question” columns in your official review correspondingly.** Please see our response below.\n\n**Q1: [Q] [W] More experiments are suggested on more recent self-supervised learning architectures, such as MoCov3 with ViT or Barlow Twins to strengthen the paper.**\n\nA1: Thank you for your valuable suggestions. In response, we have conducted new experiments to illustrate the effectiveness of our proposed method (FM) when applied to the more recent SSL framework MoCov3 using the ViT structure. Due to the limited time available during the rebuttal phase, we were only able to conduct experiments with MoCov3. However, we recognize the importance of extending our work to include other recent frameworks like Barlow Twins, and we will continue to run these additional experiments upon receiving the reviewer's follow-up response. Detailed results of the experiments conducted so far can be found in **Tab. R1** in **the attached PDF** of the general response.\n\nIn line with our SSL experiment plan detailed in Tab. 2 of the paper, **Tab. R1** tested FM on three downstream datasets, specifically OxfordPets, Flowers102, and SUN397, with a source data class pruning ratio ranging from 50% to 80%. The results affirm that the principal conclusions drawn from MoCov2 remain consistent with MoCov3 on ViT. Our method, FM, successfully identifies data subsets within the source dataset that can be pruned at high ratios without compromising downstream performance (termed as \"winning subsets\"). For instance, in one particular case, the FM-based winning subsets achieve a pruning ratio of up to 70% on OxfordPets.\n \n&nbsp;\n\n**Q2: [W] There are some minor typos.**\n\nA2: We appreciate your meticulous attention to detail. Rest assured, we have corrected these minor typos in our revision. Thank you for bringing them to our attention."}, {"review_id": "wL1AZesPWC", "reviewer": "Reviewer_BXTc", "summary": "This paper presents the problem of dataset pruning for transfer learning. The idea is to remove redundant samples in the source dataset to improve the source pre-training efficiency while maintaining the downstream fine-tuning accuracy. The label and feature mapping method proposed for the problem utilizes a surrogate model to compute a relatedness score for each source class. Source classes with lower scores are then pruned afterward. The proposed methods are compared to existing dataset pruning methods and demonstrate better performance on 8 downstream datasets.", "questions": "Please see the above weakness section and also the following questions. \n\n1) By assuming the availability of the pre-training dataset during fine-tuning, what would be the practical use cases for the proposed problem? What could be the problem’s additional cost and benefits, compared to the existing pre-training and fine-tuning paradigm, where we just need the pre-trained model, without the pre-training dataset? \n\n2) Table 3 only compares the time consumption for one downstream task. However, the scenario might become different if we consider multiple downstream tasks at once because we only need to pre-train once for the unpruned dataset. Given some downstream tasks, like 8 tasks in the paper, what would be the downstream accuracy if we use the same time budget for pre-training on pruned or unpruned datasets? \n\n3) How accurate does the surrogate model need to be for the proposed methods? For example, if we use a surrogate model even smaller than ResNet-18 and get a lower pre-training accuracy, how would the fine-tuning accuracy change?", "limitations": "As mentioned in the conclusion of the paper, the paper mainly focuses on the experimental results of the proposed methods and lacks theoretical viewpoints/insights. Besides, the use of a surrogate model and the assumption of an accessible pre-training dataset might be some other potential limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. The proposed problem is interesting and the proposed approaches seem to be reasonable.  \n- The experimental study on adversarial pre-training is inspiring and shows the potential for a wider extension of the problem to other areas.", "weaknesses": "- The problem assumes that the source pre-training dataset is available during downstream fine-tuning. However, this might not be realistic because the pre-training dataset is sometimes private and inaccessible, like medical domains. \n- The efficiency comparison with pre-training on the unpruned dataset is a little bit unfair to me (Table 3). In the standard pre-training and fine-tuning paradigm, we only need to pre-train once, and then we can fine-tune the pre-trained model on multiple tasks. However, in the proposed scenario, we need to pre-train on the pruned dataset for each downstream task every time. Pre-training on the unpruned dataset yields a generic model but requires more time. Although pre-training on the pruned dataset is faster, it outputs a task-specific model. It would be important to distinguish the trade-off to make a fair comparison. \n- The need for training a surrogate model might be a huge limitation because it cannot avoid training on the whole source dataset, which could be extremely large. Although the authors try to argue that the surrogate model can be small, it remains unclear how small it can be.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. We present a point-to-point response below, where **we use [W] and [Q] to refer to the specific points listed in the “Weakness” and “Question” columns in your official review.** All the additional results use the index starting with “R” (e.g., Tab. R2, Fig. R1) to differentiate from the submission (e.g., Fig. 4) and are listed in the attached pdf.\n\n**Q1: [W] [Q] The assumption of accessible pretraining dataset might sometimes be unavailable, because the source dataset could be private. [Q] What would be the practical use cases for the proposed problem and what are the benefits of the proposed method over using the off-the-shelf pretrained model?**\n\nA1: Thanks for raising this valuable question. We apologize for any confusion introduced by our previous presentation. However, it appears there may be a misunderstanding regarding our method's motivation and setting. Please refer to the [general response (GR1.1)](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH) for detailed clarifications.\n\nAs a highlight, since our primary motivation is to study the source data influence in transfer learning, we need access to the pre-training dataset to be pruned and knowledge of the downstream task. Although this differs from some transfer learning scenarios where only the pretrained model is available, it provides a fair setting for dataset pruning, consistent with [R1].\n\nMoreover, instead of using a once-for-all pre-trained source model, there exist practical use cases that the off-the-shelf pre-trained source model needs to be customized for user-specific challenges, e.g., distribution shifts (see **Tab. R3**) or other data regulation, security and privacy issues as suggested in [R1], or the need for more advanced but computationally-intensive training protocol, such as adversarial training as we showed in the experiment section (see Line 347 ~ Line 367).\n\nLast but not the least, our approach is not positioned in opposition to the trend of using off-the-shelf source models. Rather, it seeks to bridge the gap between the capabilities of these open-source models and the computationally-intensive source model modifications for particular tasks, industries, or organizations. We strongly believe that our method makes a novel contribution to data-model influence in transfer learning. \n\n> [R1] A data-based perspective on transfer learning, Jain et al., CVPR 2023.\n\n&nbsp;\n\n**Q2: [W] [Q] Unfair efficiency comparison in Tab. 3. Model pretrained on the full source dataset can be used for all downstream tasks, while model pretrained on pruned source dataset can only be applied to one task only. How about comparing the training time for several downstream tasks?**\n\nA2: This is an important point! We acknowledge that a model pretrained on a pruned source dataset might not be as versatile as one pretrained on the full dataset. However, the context of our approach caters to exploring the source data influence in a specific downstream task, which we have elaborated on in [General Response 1 (GR1.3)](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH).\n\nYour comment prompted us to further investigate the applicability of our method across multiple downstream tasks. In **Fig. R2**, we demonstrate LM's performance when the source dataset is pruned collectively for four downstream datasets. Interestingly, LM maintains its efficacy in this expanded setting, successfully discovering source subsets without compromising downstream performance (termed \"winning subsets\") with a maximum pruning ratio ranging from 30% ~ 50%. Compared to the results in Fig. 4, the largest pruning ratio is reduced as more downstream tasks are considered in LM. This also suggests that the task-specific customization of the source model allows for a larger margin for dataset pruning.\n\nWe appreciate this insight, recognizing it as a limitation, and will address and discuss it appropriately in our revised manuscript. \n \n&nbsp;\n\n**Q3: [W] [Q] It is not clear how small the surrogate model can be and how accurate the surrogate model needs to be? The usage of the surrogate model could be a limitation of this work.**\n\nA3: Thank you for this constructive and insightful question. To explore the sensitivity of our methods to the surrogate model's size and accuracy, we have conducted additional experiments using various surrogate models. These results are presented in **Tab. R2** and **Fig. R1**. In accordance with Fig. 4 in our paper, we replaced ResNet-18 with smaller neural networks, specifically ResNet20s, ResNet32s, ResNet44s, ResNet56s, VGG-2, VGG-4, and VGG-8. **Tab. R2** shows the transfer learning performance on the task OxfordPets against different surrogate model sizes. It is evident that even though the performance of the surrogate model on the source dataset (ImageNet) decreases, the downstream performance of RN-101 pretrained on the LM-based pruned source subsets remains relatively stable.\n\nTo further elucidate this observation, **Tab. R2** compares the class indices selected by the corresponding surrogate models. Interestingly, the most relevant source class selections exhibit a high level of agreement across surrogate models of differing sizes. Our label mapping method  shows robust behavior with different surrogate models, even if they yield different performance on the source dataset due to their different model capacities. **For example, ResNet32s only achieves a test accuracy of 40.77% on the source dataset, but it manages to achieve the same maximum pruning ratio of the winning subsets as ResNet-18 on the downstream task OxfordPets.** This experiment underscores that our method can accommodate a small surrogate model of 1% the size of the pretrained model (Rn-32s vs. RN-101). Moreover, it reveals that the surrogate model does not need to attain exceptional performance in the source domain for dataset pruning in transfer learning."}, {"review_id": "igCllyik42", "reviewer": "Reviewer_omyc", "summary": "This paper addresses the problem of dataset pruning, which serves to smaller the dataset for faster computation and learning. The authors claim that some existing methods require overly much computation for selecting the pruned dataset and thus propose a method with less computational requirements. However, in line 163 the authors seem to suggest pruning dataclasses from the source dataset, which requires an additional training of the foundation model, which is usually not possible in practice. ", "questions": "Have any experiments been done on benchmarks for transfer learning, such as [1] or [2]?\n\n[1] Zhai, Xiaohua, et al. \"The visual task adaptation benchmark.\" (2019).\n\n[2] Triantafillou, Eleni, et al. \"Meta-dataset: A dataset of datasets for learning to learn from few examples.\" arXiv (2019).\n", "limitations": "The paper focuses on computational efficiency and thus could reduce power consumption of learning algorithms at scale. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The proposed method does not require training or heavy computation when selecting the source classes. Moreover, the results in Figure 4 suggest that better test accuracy is achieved for higher pruning rates in a wide range of datasets. \n\n* Line 282-292 provides a clear overview of the training setup, which makes the experiments reproducible. \n", "weaknesses": "* The results in Figure 4 indicate that over a wide range of datasets about 70% of classes can be pruned. However, in Table 3 one can read that such pruning only saves about 3 hours of computation time. In what scenario is such saving critical? I would say that dataset pruning is complex and introduces many unknown biases, and I am not sure that this is outweighed by saving only three hours of computation. \n\n* Removing any datapoint from the dataset introduces, by definition, a bias in the dataset. What are the insights on such biases and has any analysis been done on class imbalances? Alternatively, has any analysis on datasets such as CIFAR-C and ImageNet-C been run to assess robustness [1,2]? \n\n* I would say the pruning of classes is specific to the target dataset. And usually, training of foundation models is not possible as the foundation models are usually trained by large institutions on large datasets and thus only fine tuning is an option. This paper seems to address the problem of pre-training models on a new dataset. What benefits does such an approach have over established foundation models? There have been papers to suggest that actually the pre-training on large datasets yields benefits for downstream performance [3].\n\n[1] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" arXiv preprint arXiv:1903.12261 (2019).\n\n[2] https://github.com/hendrycks/robustness/blob/master/ImageNet-C/create_c/make_cifar_c.py\n\n[3] Kolesnikov, Alexander, et al. \"Big transfer (bit): General visual representation learning.\" ECCV, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. **We use [W] and [Q] to refer to the specific points listed in the “Weakness” and “Question” columns in your official review respectively.** All the additional results use the index starting with “R” (e.g., Tab. R2, Fig. R1) to differentiate from the submission (e.g., Fig. 4) and are listed in the one-page pdf.\n\n**Q1: [W] Even though in Fig. 4 about 70% of the source classes can be pruned, the time saving (reduction) of 3 hours in Tab. 3 does not seem to be significant. In what scenario is such saving critical?**\n\nA1: Thank you for drawing attention to this aspect. We are afraid that our previous presentation might have led to a potential misunderstanding. Our experiments on supervised training (Fig. 2, 3, 4) followed the literature [R1], where we utilized the fast-forward accelerating framework (FFCV [R2]) to accelerate training on ImageNet (sorry for making this detail unclear). As shown in Tab. 3, FFCV allows us to train a ResNet101 on ImageNet within 6 hours. Aided by FFCV, the absolute time saving thus does not seem significant. However, it is crucial to consider the relative improvement. In this context, achieving a 56% computation time saving for a pruning ratio of 60% without losing downstream performance is indeed a significant accomplishment from our perspective.\n\nFurther, we remark that FFCV is not a generic accelerated framework applicable to diverse settings. For example, it is not applicable in the self-supervised training (SSL) setting. In such a case, the absolute improvement in time saving provided by our proposal also becomes significant. In **Tab. R5** of the attached PDF, we showcase the time comparison related to Tab. 2 (SSL experiments), revealing that our method can save up to 168 hours (7 days) per single experiment trial by pruning 60% data without hurting the downstream performance for all the datasets in Tab. 2.\n\nThus, the time savings from our method are indeed significant, especially when FFCV is not an option. Your observation has helped us recognize the need for a more detailed demonstration, and we will ensure this is addressed in our revision. Thank you!\n\n&nbsp;\n\n**Q2: [W] Dataset pruning is complex and introduces many unknown biases. Does the removal of data classes introduce additional biases? How should we analyze this? Are there any analyses on datasets such as CIFAR-C and ImageNet-C?**\n\nA2:  This is an insightful question. As detailed in [General Response 1 (GR1)](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH), our work focuses on selecting the most influential source data points without sacrificing downstream performance. As [R1] shows, source data pruning can effectively remove the harmful influence of biased data classes relevant to the targeted downstream task, rather than introducing an additional data bias. This phenomenon is also confirmed in our work, where LM/FM successfully find the most useful points (Fig. 4), resulting in higher downstream performance than using the full data.\n\nTo further justify this point, we followed the reviewer’s suggestion and investigated the applicability of our method on CIFAR-10C [R3]. The experiment setting is consistent with Fig. 4 based on the CIFAR-10 task and evaluated it on the CIFAR-10C with different corruptions. Due to space limit, in **Tab. R3** we report the results of LM on the 5 out of the 19 perturbation types using the largest perturbation strength defined in CIFAR-10C. As we can see, when facing distribution shifts, LM can obtain winning subsets with a pruning ratio up to 80%, even higher than DP for the CIFAR-10 task.\n\nIn conclusion, our work may help removing data biases through dataset pruning for a downstream task, rather than inadvertently introducing them. The additional analyses on CIFAR-C reinforce our method's validity in handling these complexities. Your insightful query has greatly assisted us in this explanation, and we will include these discussions in our revision. Thank you!\n\n&nbsp;\n\n**Q3: [W] This paper seems to address the problem of pre-training models on a new dataset. What benefits does training a foundation model have over using the established foundation models?**\n\nA3: Thanks for this question. Our objective is NOT to address the problem of pretraining models on a new dataset. Instead, our primary objective is to study the influence of source data in transfer learning and to explore if an efficient dataset pruning algorithm can be designed for achieving lossless transfer learning on a given downstream task, as opposed to the computationally-intensive source data attribution method in [R1]. We kindly refer to [GR1.1](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH) for the doubts about our setup, and [GR1.3](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH) for practical significance and benefits of our proposal. \n\n&nbsp;\n\n**Q4: [Q] Have any experiments been done on benchmarks for transfer learning, such as [R4] and [R5]?**\n\nA4: Thank you for this insightful question! We have extended our experiments to include the VTAB benchmark [R4]. As shown in **Tab. R4**, even in the few-shot setting, our major conclusions remain valid. LM can effectively pinpoint the most influential source subsets and eliminate the less valuable ones. For most tasks considered, winning subsets with pruning ratios up to 40% are successfully found. Due to the time and space constraints, we did not report the results on [R5] and we will continue to run these additional experiments upon receiving the reviewer's follow-up response.\n\n&nbsp;\n\n> [R1] A data-based perspective on transfer learning, Jain et al., CVPR 2023.\n> \n> [R2] FFCV: Accelerating Training by Removing Data Bottlenecks, Leclerc et al., CVPR 2023\n> \n> [R3] Benchmarking neural network robustness to ..., ICLR 2019\n> \n> [R4] A Large ... Visual Task Adaptation Benchmark\n> \n> [R5] Meta-dataset: A dataset of datasets for learning to learn from few examples.\n"}, {"review_id": "40Cn9mvdRC", "reviewer": "Reviewer_vmFL", "summary": "The paper proposes two new dataset pruning (DP) methods, label mapping and feature mapping, for transfer learning. The aim is to improve pretraining efficiency and fine-tuning accuracy on downstream target tasks without sacrificing performance. The proposed methods are evaluated on numerous transfer learning tasks and show significant speed-up during the pretraining stage.", "questions": "Please see the weakness above, I would like the authors to address these limitations.", "limitations": "This article examines the use of transfer learning for data pruning, but questions whether this topic is meaningful. Firstly, the dataset used in the article is very small, with a maximum size of no more than 100,000, making it too small for practical application. Additionally, the current trend is to train very large models using large amounts of data. Furthermore, is the method used in the article simply a combination of two fields, namely data compression and transfer learning? It is important to research truly meaningful topics.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper addresses an important problem of dataset pruning for transfer learning, which has not been explored much in previous studies.\n\n+ The proposed methods show significant improvement in pretraining efficiency and finetuning accuracy on downstream target tasks.\n\n+ The paper provides a unified viewpoint to integrate DP with transfer learning.\n\n+ The proposed methods are evaluated on numerous transfer learning tasks, demonstrating their broad applicability.", "weaknesses": "+ The paper does not consider other types of transfer learning scenarios, such as domain adaptation, multi-task learning, or meta-learning, which may have different challenges and requirements for dataset pruning.\n\n+ The paper does not conduct a thorough ablation study or sensitivity analysis to understand the impact of different design choices, such as the surrogate model, the cluster number, and the reverse order selection, on the performance and efficiency of the proposed methods.\n\n+ The paper does not provide any discussion of the limitations or failure cases of its method. This could limit the understanding of when and how to apply its method in practice. \n\n+ The paper does not provide a detailed analysis of the impact of the proposed methods on the size of the pruned dataset, which could be an important factor in some applications.\n\n+ The paper does not specifically address the question of whether it is necessary to compress data with a relatively small amount of data, but rather focuses on improving the efficiency of using the available data. It is important to note that while larger amounts of data can lead to better performance\n\n+ The proposed DP methods require access to a surrogate source model (e.g., ResNet-18), which may not be available or practical in all settings.\n\n+ The effectiveness of the proposed DP methods may depend on the quality and diversity of the source dataset, and it is unclear how well they would perform with low-quality or biased data.\n\n+ The proposed DP methods (label mapping and feature mapping) were evaluated on a limited set of downstream tasks, and it is unclear how well they would generalize to other tasks or domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We provide response to **the eight points you listed in the weakness column (W1 to W8) and the limitations.** Additional results are indexed with \"R\" (e.g., Fig. R1) to distinguish them from the submission (e.g., Fig. 4), and are detailed in the one-page PDF.\n\n**Response to W1 & W8.**\n\nThanks for the suggestions. We expanded our experiments to test our method's robustness, with details in **Fig. R1 and Tab. R2, R3, R4**. **Tab. R3** shows even with out-of-distribution biases, the LM method can prune over 40% of training data without affecting downstream performance, implying LM can enhance performance by removing harmful data. Tab. R4 highlights our method's success on the few-shot VTAB benchmark [R1], finding winning subsets with winning ratios up to 40%.\n\n Besides, we’d also clarify that our paper's main focus is exploring data influence and DP for transfer learning. Our submission, covering two mainstream pretraining paradigms and eight datasets, have effectively demonstrated the efficacy and relevance of our proposed approach. \n\n&nbsp;\n\n**Response to W2.**\n\nThanks. We'd like to clarify that our original submission **already explicitly includes sensitivity studies on cluster number and reverse order selection** (Lines 376-380, Fig. A2, A3 in Appendix).\n\nIn addition, following your suggestion, we conducted sensitivity studies on surrogate model size (**Tab. R2 and Fig. R1**). Fig. R1 shows that the downstream performance of the original source pre-trained model RN-101 remains quite stable using diverse surrogate models (0.516M-11M in size), despite a reduction in the surrogate model's performance on the source dataset (Tab. R2). **Besides**, there is a strong agreement in source class choices across surrogate models of varying sizes. These results confirm our methods can adapt to models as small as about 1% the size of the original, emphasizing that superb performance with the surrogate model is not vital for obtaining source dataset pruning schemes with lossless downstream performance.\n\n&nbsp;\n\n**Response to W3.**\n\nThanks for the insightful comments. A limitation of our method is its task-specific nature in data influence analysis. As our methods assess source data influence for specific downstream tasks, developing a universal DP solution for multiple tasks simultaneously is challenging. \nAs a preliminary study, we examine the performance of source dataset pruning for multiple downstream tasks simultaneously (**Fig. R2**). While LM can still identify winning subsets, the maximum pruning ratio diminishes as more tasks are considered. We will include this limitation in the Conclusion section. \n\n&nbsp;\n\n**Response to W4 & W5**\n\nThere might exist a misunderstanding of the motivation of our work. Please see general response ([GR1.1](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH)) for a detailed clarification. (1) As shown in Fig. 4 and Tab. 2, DP can in fact enhance downstream performance, which is in line with the existing work [R2]. (2) DP could enable us to train the source model using an advanced but costly training protocol, such as adversarial training as we demonstrated in Line 347-367. (3) There also exist other strong reasons for the necessity of pruning the source dataset when public released models need to be debugged or updated to handle user-specific challenges, e.g., data regulation, security and privacy issues as suggested in [R2].\n\n&nbsp;\n\n**Response to W6**\n\nA5: Please see [GR1.3](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH). As another interesting point, **Tab. R2 and Fig. R1** showed a remarkable level of agreement in the source dataset pruning among surrogate models of different sizes. See details in our **response to W2**. \n\n&nbsp;\n\n**Response to W7**\n\nA6: This question is insightful. To investigate the scenarios with data biases, we conducted experiments on CIFAR-10C (the out-of-distribution scenario) [R3]. We first pruned ImageNet given CIFAR-10 as a downstream task and evaluated the model on CIFAR-10C with different corruption types. Due to space limitations, Tab. R3 shows LM results with different pruning ratios for 5 strong perturbation types in CIFAR-10C. Impressively, LM can achieve winning subsets with pruning up to 80%, even better than on CIFAR-10, confirming our method's effectiveness to filter biased data points to some degree. Thank you for your valuable input!\n\n&nbsp;\n\n**Response to Limitation “This article examines the use of transfer learning for data pruning, but questions are whether this topic is meaningful. Is the method simply a combination of two fields, namely data compression and transfer learning?”**\n\nAs shown in [GR1](./forum?id=D0MII7rP3R&noteId=JygqCsKyMH) (GR1.3), our study is practically meaningful and significant. We have made a notable contribution by tackling efficient source data pruning for transfer learning, a challenge previously unresolved. Our approach goes beyond a mere combination of dataset pruning and transfer learning. Line 176-194 and Fig. 2 demonstrate that a simple combination does not solve our studied problem.\n\n&nbsp;\n\n**Response to Limitation “The paper lacks analysis of the impact on the size of the pruned dataset. The dataset in the article is very small, with a max size of no more than 100,000, making it too small for practical application.”**\n\nWe respectfully disagree that \"the pruned dataset used is small, with a size of no more than 100,000.\" In our work, we adopt ImageNet as the source dataset to be pruned, with more than 1.3 million high-resolution images. ImageNet has been the most commonly used benchmark for pretraining and is the largest dataset considered in the literature [R2, R4]. \n\n&nbsp;\n\n> [R1] A Large ... Visual Task Adaptation Benchmark\n>\n> [R2] A data-based perspective on transfer learning, CVPR 2023.\n> \n> [R3] Benchmarking neural network robustness to ..., ICLR 2019\n> \n> [R4] Coreset sampling from open-set for ..., CVPR 2023."}]

Author rebuttal:
## Highlighted General Response

We thank all the reviewers for their valuable comments. First, we would like to clarify some possible misunderstandings about our work based on the comments.

### GR1 - Possible misunderstandings of the motivation and the setting of this work. (Reviewer BXTC, vmFL, omyc)


A few concerns were raised by reviewers on the availability of the source dataset (**@Reviewer BXTc**) or the surrogate model (**@Reviewer vmFL**), as well as the necessity of pretraining own models instead of using off-the-shelf pretrained foundation models (**@Reviewer omyc, vmFL**). It seems that the motivation and the primary objective of our work was not very clearly delivered. We apologize for that! **This work aims to study the influence of source data in transfer learning and to explore an efficient data pruning algorithm designed for lossless transfer learning given a downstream task**, as opposed to the computationally-intensive source data attribution method for transfer learning in [R1]. We highlight our motivation and setting below.

* **GR1.1: A data-based perspective of transfer learning, but in a much more efficient manner (@Reviewer BXTc, omyc, vmFL).** Our work follows the existing research in data influence analysis for transfer learning [R1, R2], where the most influential source data is pinpointed given a downstream task. As our paper discussed (Line 54 - 58), [R1] pioneered this line of research and proposed a brute-force approach to evaluate the influence of every source class. Such a manner is not affordable when the problem scales up. This motivates a more efficient source data influence analysis method, which is scalable to diverse downstream tasks and to different pre-training paradigms (as we showed in supervised learning, adversarial pre-training, and SSL scenarios). **However, this is also a non-trivial problem as the existing dataset pruning (DP) methods become ineffective in transfer learning (see Fig. 2 vs. Fig. 4).** As our primary motivation is to study source data influence in transfer learning, we naturally require access to the pre-training dataset to be pruned and knowledge of the downstream task. This provides a fair setting for DP, consistent with [R1].

* **GR1.2: The use of a smaller source surrogate model is not a limitation but an interesting and important aspect from the perspective of DP (@Reviewer vmFL).** Previous DP studies assumed no mismatch between DP and model training on the pruned dataset. However, our work demonstrates for the first time that DP can be conducted with a much smaller surrogate model without sacrificing downstream task performance. **This brings a significant learning efficiency benefit, as selecting the most influential source data can now be achieved in an ultra-efficient manner, unlike the super-costly approach in [R1].** Moreover, we find that the assumption of having access to a surrogate small model is not restrictive. The results show that **our approach is NOT sensitive to the surrogate model size**, accommodating a small surrogate model of nearly 1% the size of the original pretrained model without downstream performance loss (see **Tab. R2** and **Fig. R1**).

* **GR1.3: The study of source dataset pruning for transfer learning has practical significance and is a meaningful and prosperous field. (@Reviewer vmFL, BXTc, omyc).** In response to why we should conduct source dataset pruning and retrain our own model instead of using public ones, there are compelling reasons. For example, public released models may not meet the needs for user-specific challenges, e.g., distribution shifts (see **Tab. R3**) or other data regulation, security and privacy issues as suggested in [R1], or need to be improved using another more advanced but computationally-intensive training protocol, such as adversarial training as we demonstrated in the experiment section (Line 347-367). Thus, our proposal can speed up the customized source training process and reduce the computational costs. Furthermore, investigating source data influence to predict and manage the behavior of models, including data-models [R3], is a promising field that enhances understanding of data biases and optimizes resource allocation. This work resonates with existing studies [R1, R2], continuing an established line of inquiry acknowledged for its value in the community .

In summary, our work is not opposed to the current transfer learning paradigm, but a focused and efficient solution tailored to specific needs with broader implications, offering valuable insights into data influence and source training customization.

> [R1] A data-based perspective on transfer learning, CVPR 2023.
>
> [R2] Coreset sampling from open-set for fine-grained self-supervised learning, CVPR 2023.
>
> [R3] Datamodels: Predicting Predictions from Training Data, ICML 2022

---

### GR2 - A summary of additional experiments in the one-page PDF.

We have made a substantial effort to enrich our experiments (see the attached PDF). Below is a summary of them, where Q$i$ represents the $i$th question in our response).

* Reviewer RSvb
    * Q1 - Experiments on advanced SSL frameworks, MoCov3, with ViT (**Tab. R1**).


* Reviewer BXTc
    * Q2 - Experiments in the multi-task setting (**Fig. R2**)
    * Q3 - Sensitivity study of the surrogate model size (**Fig. R1**, **Tab. R2**).


* Reviewer omyc
    * Q1 - Time saving report for SSL (**Tab. R5**).
    * Q2 - Experiments on CIFAR-10C (**Tab. R3**).
    * Q4 - Experiments on transfer learning benchmark VTAB (**Tab. R4**).


* Reviewer vmFL
    * Q1, Q6 - Experiments on CIFAR-10C (**Tab. R3**).
    * Q1 - Experiments in the multi-task setting (**Fig. R2**).
    * Q1 - Experiments on transfer learning benchmark VTAB (**Tab. R4**).
    * Q2, Q5 - Sensitivity study of the surrogate model size (**Fig. R1**, **Tab. R2**).

Meta-review: I am recommending to **accept** this paper "Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning".

Four reviewers provided feedback on this paper. The authors provided a response to the reviews and I appreciate the authors' comments and clarifications, specifically I think the authors were successful at discussing their work with the reviewers and at clarifying reviewers' concerns. 

The reviewers did not fully agree on the assessmetn with scores of 4/5/6/7. Based on the reviews and the rebuttal discussions, the reviews with the slightly higher scores (5/6/7) carry somewhat more weight in my opinion.

This paper looks at an interesting combination of problems - dataset pruning and transfer learning. I share a concern about the practical relevance of the approach that has been voiced by reviewers, though. 

Overall, I believe the paper is of high enough quality to be accepted at NeurIPS, hence my recommendation to accept.
Decision: Accept (poster)