Title: Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset pruning

transfer learning
Abstract: Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged as an effective way to improve data efficiency by identifying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e., how to prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy on downstream target tasks. To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning as separate problems. By contrast, we establish a unified viewpoint to integrate DP with transfer learning and find that existing DP methods are not suitable for the transfer learning paradigm. We then propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-target domain mapping. Furthermore, we demonstrate the effectiveness of our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to $40\%\sim 80\%$ without sacrificing the downstream performance, resulting in a significant $2\sim 5\times$ speed-up during the pretraining stage. Besides, our proposal exhibits broad applicability and can improve other computationally intensive transfer learning techniques, such as adversarial pretraining.

Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning

Abstract

Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged as an effective way to improve data efficiency by identifying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e. , how to prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy on downstream target tasks . To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning as separate problems. By contrast, we establish a unified viewpoint to integrate DP with transfer learning and find that existing DP methods are not suitable for the transfer learning paradigm. We then propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-target domain mapping. Furthermore, we demonstrate the effectiveness of our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to 40% ∼ 80% without sacrificing downstream performance, resulting in a significant 2 ∼ 5 × speed-up during the pretraining stage. Besides, our proposal exhibits broad applicability and can improve other computationally intensive transfer learning techniques, such as adversarial pretraining. Codes are available at https://github.com/OPTML-Group/DP4TL .

1 Introduction

The abundance of the training data has long been regarded as the key driver of the contemporary machine learning (ML) algorithms [1-3]. However, the untrimmed, ever-growing training dataset could not only introduce training biases that compromise the model performance [4-6], but also poses an almost insurmountable obstacle to high training efficiency [1, 2]. Therefore, understanding the impact of the training data and selecting the most critical samples has emerged as an important goal, collectively referred to as the problem of dataset pruning ( DP ).

Although the feasibility and the promise of DP have been unveiled in numerous applications, such as noisy data cleansing [7-9], continue learning [10-13] and active learning [14], greater emphasis is placed on the in-domain training setting, i.e. , the pruned training set share the similar distribution as the evaluation set. Examples of methods to condense the training dataset with lossless generalization (on the in-distribution testing dataset) include data influence functions [10, 15-23], model training dynamics [11-14, 24-28], and coreset selection [29-36]. In contrast, our work investigates the

Abyssinian

Cat

American

Bulldog

Shiba

Inu

Black

English

Springer

Indigo

Bunting

Tennis

Cab

Loafer

Source - Target

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

Source/Target Data Features from Surrogate Model

LM/FM Data Pruning

❌

Target Data

Source Data

Retained Data

Pruned Data

ImageNet

→

OxfordPets

ImageNet

→

StanfordCars

Stage I: Pretrain (5

×

Speed Up

🚀

)

Source Dataset Pruning

Surrogate Model

(Small)

Stage II: Finetune

Source Model (Large)

IMGENET

IM GENET

Random

Moderate

GraNd

LM

O

△

O

A

70

FM

NoPrune

O

oo

O

O

A

O

A

O

O

△

A

O

oo

O

O

△

O

65

0%

20%

Pruning Ratio

40%

60%

80%

(%)

45

A

40

Denn

30

25

0%

20%

Pruning Ratio

40%

60%

80%

problem of DP in the transfer learning paradigm [37-39], which has emerged as a popular way to leverage the knowledge of a foundation model learned on a source dataset (referring to ' pretraining ' stage) to further enhance the performance on a cross-domain target dataset (referring to ' finetuning ' stage). Recent evidence [40-42] has shown that some source data points could make a harmful influence in the downstream performance. In particular, the previous study [41] showed that removing specific source data classes can improve the transfer learning accuracy of a pretrained model.

Yet, efficiently identifying those harmful source data classes for improved transfer learning is a highly non-trivial task. Firstly, the unique pretrain-finetune paradigm complicates the analysis of the influence of source data on downstream performance, making it an indirect and very challenging process. Consequently, existing gradient and data influence-based in-domain DP methods [10, 19-23] cannot be naïvely adapted to the transfer setting. Secondly, transfer learning encompasses a wide range of source training methods other than supervised learning, such as self-supervised learning ( SSL ) [43-52]. Therefore, a generic DP framework is desired given various pretraining scenarios. Thirdly, from an efficiency standpoint, the design of efficient DP algorithms is non-trivial. Even in the non-transfer learning paradigm, a majority of current in-domain DP methods [11, 25, 26] introduce substantial computational overheads. For instance, influence function-based DP methods [10, 15-22] necessitate the calculation of the inverse Hessian matrix of model parameters, which is a highly demanding and computationally intensive process. Moreover, training dynamics-based methods [11-14, 24-28, 53], such as GraNd-Score [26] and Forgetting-Score [11], necessitate model training on the entire dataset multiple times. As a result, the development of an efficient and effective DP framework tailored for transfer learning remains a significant challenge in the field.

The most relevant work to ours is [41], which proposed a brute-force method to evaluate the influence of every source class in the downstream task following a leave-one-out principle. While this method effectively selects the most influential source classes, it is prohibitively slow, as the data selection process demands significantly higher computational resources than performing transfer learning itself. This brings us to the central question addressed in this paper:

(Q) How can we extend DP to transfer learning with minimal computation overhead, broad applicability, and improved target performance?

To address (Q) , we formally define the task of DP (dataset pruning) for transfer learning. We start by uncovering the limitations of conventional in-domain DP methods when applied to transfer learning tasks, and establish an intrinsic connection between source dataset pruning for transfer learning and source-target domain mapping. In the supervised pretraining paradigm, we propose an effective and scalable label mapping ( LM )-based framework capable of pinpointing the most beneficial source

data labels for a cross-domain target dataset. Furthermore, we extend the idea of LM to feature mapping ( FM ) for the SSL pretraining protocol, where data labels are unavailable. To achieve a greater practical impact, we demonstrate that our proposed methods (LM and FM) can facilitate effective DP over a small, simple surrogate source model, and further translate the positive impact of the pruned source dataset to other larger, more complex source models. We provide a schematic overview and result highlights in Fig. 1 . Our contributions can be summarized as follows:

❶ We connect the concept of DP (dataset pruning) to transfer learning for the first time and formalize the problem of DP for transfer learning.

❷ We develop a highly efficient and effective framework for DP in transfer learning, leveraging the source-target domain mapping with customized LM (label mapping) and FM (feature mapping) for both supervised and SSL (self-supervised learning) settings. Our approach is principled and achieves significant speedups compared to the state-of-the-art methods.

❸ We empirically show the effectiveness of our proposals (LP and FP) on 8 downstream tasks. We find that the source dataset (ImageNet) can be pruned up to 40% ∼ 80% without sacrificing the downstream performance, together with a 2 × ∼ 5 × speed-up in the pretraining stage. Our proposal also unlocks a door to prune a dataset using a simple surrogate source model ( e.g. , ResNet-18) and then reuse the pruned dataset to improve transfer learning on a larger source model ( e.g. , ResNet-101).

❹ Lastly, we show that our proposed DP framework can benefit other computationally-intensive transfer learning techniques. For example, DP for adversarial pretraining [58] leads to a 1% ∼ 2% improvement in downstream performance with time consumption similar to standard pretraining.

2 Related Work

Dataset pruning. DP (dataset pruning) is an emerging technique to improve the data efficiency of model training by selecting the most representative training samples or removing the less influential ones [10-28]. Thus, the problem of coreset selection [29-36] can also be considered a form of DP. Prior studies on DP range from clustering-based methods [59-61] to the more recent score-based pruning methods [10-28]. In the latter approach, an importance score is assigned to each training data point to quantify its influence on a particular permanence metric during model learning. Specifically, score-based DP methods can be broadly categorized into two main groups: influence function-based approaches [10, 15-23] and training dynamics-based approaches [11-14, 24-28, 53]. The first category measures data influence by examining the effect of data removal on the learning algorithm used during training [15-18] and the model's prediction [10, 19-23]. However, influence functionbased approaches typically require high computational costs due to the need for high-order derivatives and complex optimization methods, such as bi-level optimization, as used in [22]. Although an approximate influence score can be obtained efficiently, it may result in a large estimation error [62]. In the second category, training dynamics-based approaches find statistical indicators for pruned data from the training trajectory. Examples of DP metrics include data loss/error [25-27], prediction confidence [24], model gradient norm [26], forgetting event [11], and compactness [12-14, 28]. However, these methods typically require repeated training to ensure the representativeness of the collected statistics [11, 25, 26]. In addition to improving data efficiency, DP has been applied in a variety of contexts, such as noisy label cleansing [7-9], continue learning [10-13, 63], active learning [14], and reducing annotation cost of a finetuner [64].

Data valuation & attribution. Data valuation [19, 41, 65-70] and attribution [71-73] are research streams related to dataset pruning that aim to quantify the influence of training data points on model's performance. Unlike dataset pruning, these approaches do not focus primarily on training efficiency but instead aim to enhance interpretability [74], adversarial robustness [68, 70], generative model design [75], and data acquisition quality [72]. Representative methods include Shapley value-based methods [65, 66], datamodels [73], sampling-based methods [65, 68], and proxy-based methods [67, 72]. Recently, Kim et al. [35] proposed to select data samples from a large public dataset (Open-Set) for self-supervised learning given a specific target dataset through distribution mismatch. However, it fails to make a general framework in both supervised and self-supervised scenarios, leaving the former under-explored. The most relevant work to ours is [41], which leverages a leaveone-out analysis to quantify the influence of source data on downstream tasks in the context of transfer learning. With this inspiration, we propose to connect DP to transfer learning in this work.

Recent advancements in transfer learning. Transfer learning has been a prominent area over the past decade [37-39]. Significant strides have been made in understanding, analyzing, and improving various aspects of this technique [58, 76-84]. Recent studies [77, 78] have rigorously analyzed the relationship between source model performance and downstream task effectiveness, arguing that a narrow focus on minimizing source training loss may not lead to improved transfer learning results. To improve transfer learning, adversarial training has been shown to benefit the transferability of source pretraining on target downstream tasks [58, 76]. There also exist studies to identify and understand the failure cases in transfer learning [79-84]. Other research [77, 78] has examined model transferability from the perspective of model sharpness and argues that a good pretrained model should be situated in a flat basin in the downstream loss landscape. Last but not the least, transfer learning has progressed in several other directions, such as self-supervised learning [43-52], model weight pruning [85-89], and visual prompting [90-99].

3 Problem Formulation

In this section, we introduce some essential preliminaries on transfer learning and DP (dataset pruning), and elucidate the design challenge of DP for transfer learning.

Preliminaries on transfer learning and connection to DP. Let g ◦ f denote a deep neural network that consists of a feature extractor f and a classification head g , where ◦ denotes the function composition. Given source and target datasets D S and D T , we study transfer learning in the ' pretrainfinetune ' paradigm. The primary goal of pretraining is to obtain a high-quality feature extractor f S : X → Z , which draws a mapping from the input space ( X ) to the deep representation space ( Z ) in a data-rich source domain ( D S ). Popular pertaining recipes include supervised learning ( SL ) [84, 100] and self-supervised learning ( SSL ) [43-46] depending on whether the source labels ( y S ) are available or not in D S . In the finetuning stage, the pretrained model is further trained on a specific downstream task under the target dataset D T . Transfer learning expects improved downstream performance over training on D T from scratch. In this work, we consider two finetuning protocols, linear probe ( LP ) and full-finetune ( FF ), with D T being a labeled dataset. LP finetunes the linear classification head g with a fixed feature extractor f S , acquired from pretraining. In contrast, FF finetunes the entire model g ◦ f from the initialization f S . FF typically yields a better transfer learning accuracy than LP, but the former takes a higher computation cost.

Our motivation for connecting transfer learning with DP comes from a recent data-based perspective on transfer learning [41]. The study shows that removing certain source data classes from D S can potentially improve the accuracy of a finetuned model on D T . However, the task of evaluating the transfer effects of source data and removing their influence from a pre-trained source model has not been addressed efficiently. The approach developed in [41] involves a leave-one-out analysis to estimate the influence of a source class c on a target example t , which is computed as the prediction discrepancy of the finetuned source model at t when the class c is either included or excluded from D S . During this process, one must train multiple source models (over 7000 models on ImageNet in [41]) from scratch over different subsets of D S for a given target task. This approach becomes computationally unaffordable when dealing with large source datasets like ImageNet given limited computing resources. To address this challenge, we propose a DP perspective on transfer learning.

Problem of interest: DP for transfer learning. Next, we introduce the background of DP and the problem we focus on. Let D = { z 1 , z 2 , . . . , z N } denote a dataset consisting of n samples, where each sample z i is represented as a pair ( x i , y i ) , with x i denoting the input feature vector and y i denoting the corresponding label. DP aims to generate a pruned dataset ˆ D = { ˆ z 1 , ˆ z 2 , . . . , ˆ z M } ⊂ D with M < N , which can reduce the training cost without a significant decrease in model generalization performance when trained on ˆ D . In the context of [41], instead of individual source data sample { ˆ z i } , the entire source classes are evaluated and selected in terms of transfer influences. Based on the above, we define the problem of our interest below.

(DP for transfer learning) How to prune source data classes to obtain ˆ D S (a subset of D S ), with lossless or improved transfer learning accuracy of the source model ( f S ) on a target task D T ?

DP for transfer learning has two key distinctions ( ❶ -❷ ) from the vanilla DP setup. First ( ❶ ), DP must be performed in the source domain ( D S ), while its effectiveness is evaluated based on the target

domain ( D T ). This 'cross-domain' challenge makes the design of efficient and effective DP highly non-trivial. For example, prior work [41] utilizes a computationally-intensive leave-one-out analysis. Classical influence function-based methods [10, 19-23, 101], which trace the eventual model's prediction through the learning algorithm and back to its training data, are also computationally infeasible due to the complex bi-level optimizer and the calculation of high-order derivatives. Second ( ❷ ), the pre-trained source model ( f S ) in today's transfer learning regime is typically of a large scale. This motivates us to develop a DP method that can be independent of the source model, while the pruned source dataset ˆ D S remains effective in the original transfer learning setup. Given this challenge, we will design DP methods for transfer learning using a simple surrogate source model to avoid the computation on the large source model f S (see Fig. 1 for an illustration).

Conventional DP methods lack effectiveness on transfer learning. Given the challenges ( ❶ -❷ ) posed by DP for transfer learning, we further conduct a preliminary study to investigate the effectiveness of existing 8 DP methods, including SP [20], SSP [20], GRAND [26], EL2N [26], MODERATE [3], FORGET [53], INFMAX [53], and INFSUM [53]. Our results show that these methods are unable to yield significant improvements over random source data prun-

0%

20%

40%

60%

80%

Source Data Pruning Ratio

86

88

90

92

94

Target Test Accuracy (%)

Flowers102-

LP

Random

Moderate

GraNd

SP

SSP

Forget

EL2N

InfSum

InfMax

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

76

78

80

82

84

86

88

90

Target Test Accuracy (%)

OxfordPets-

LP

ing. Fig. 2 shows the transfer learning performance of the ResNet-101 [54] model trained on different pruned versions of the ImageNet dataset when LP-based finetuning is conducted on the downstream Flowers102 [102] and OxfordPets [56] datasets. As we can see, in transfer learning, random pruning is a solid baseline for various state-of-the-art DP methods, which have demonstrated superior performance to the former in the non-transfer learning regime. Therefore, it is crucial to develop an efficient and effective DP method specifically tailored for transfer learning.

4 Label/Feature Mapping-based DP for Transfer Learning

In this section, we first introduce a simple yet powerful DP method called label mapping ( LM ) by leveraging the class-discriminative capability of a supervised source model. We then extend the concept of LM to feature mapping ( FM ), suitable for self-supervised pretraining.

LM-based DP for supervised pretraining. Following the notations used in Sec. 3, we represent the model obtained through supervised learning on the source dataset ( D S ) as g S ◦ f S . This model predicts a source label given an input example ( x ). In DP for transfer learning, the source data classes serve as the variables to be pruned. Thus, we express D S as D S = {C (1) S , . . . , C ( N ) S } for N source classes, C ( i ) S denotes the set of data points belonging to the source class i . Additionally, the pruner has access to the target dataset D T = { t 1 , . . . , t n } , which consists of n target data points. Our objective is to utilize the information provided by g S ◦ f S , D S , and D T to devise a computationally efficient DP criterion. Importantly, our criterion should not involve model training, distinguishing it from the non-scalable approach in [41]. An important observation by [78] in transfer learning is that transferred knowledge improves transferability. This insight is supported by the loss landscape analysis in transfer learning: The finetuned weights may remain within the flat basin of the pretrained weights for enhanced transfer learning performance.

We next propose to extract the 'transferred knowledge' by leveraging the class-discriminative capability of the source model g S ◦ f S on the target data samples in D T . Specifically, we focus on monitoring the responsiveness of the source label predictions made by g S ◦ f S when using target samples { t i ∈ D T } n i =1 as input data. Here we resize these target samples to ensure their resolution alignment with source data. For the i th source data class C ( i ) S , we then define its pruning score below:

where ✶ ( · ) represents an indicator function that evaluates to 1 when the condition · is satisfied, and 0 otherwise. We refer to the aforementioned formula as LM (Label Mapping) since the condition g S ◦ f S ( t j ) = i establishes a mapping between the predicted labels of the target samples { t j } n j =1 by the source model and the corresponding source labels i . The larger s LM in (LM) signifies that a specific source class is more frequently utilized to interpret the target data. Consequently, this indicates a more tightly connected relationship between the source class and the transferred knowledge. Therefore, we prune (or select) source data classes with low (or high) s LM values.

Although LM is simple, it offers several advantages. Firstly , it is highly efficient in computation. Given a pre-trained source model g S ◦ f S , the calculation of LM only requires forward passes through the model. In cases where obtaining the pretrained model in the source domain is challenging, our proposal supports an alternative approach: training a smaller and simpler surrogate model to carry out LM. This surrogate model can effectively replace the complex pretrained model g S ◦ f S and facilitate the efficient execution of the pruning process. As we show in Fig. 3a , employing ResNet-18 [54] is sufficient to successfully prune the source dataset (ImageNet [55]). The resulting DP scheme remains effective to improve transfer learning utilizing other larger source models, such as ResNet-101 [54]. In addition , Fig. 3b shows that the computational overhead incurred

0%

20%

40%

60%

80%

Source Data Pruning Ratio

91.5

92.0

92.5

93.0

93.5

94.0

94.5

95.0

Target Test Accuracy (%)

RN-18

RN-101

No Prune

No Prune

40%

60%

Pruning Ratio (%)

0

1

2

3

4

5

6

Time Consumption (h)

5.4

3.5

2.4

Source Model Pretraining

(a) Downstream performance.

(b) Time consumption.

by training the small surrogate model (ResNet-18) for DP is insignificant, compared to the time saved during the pretraining phase of a larger source model (ResNet-101) on the pruned dataset for transfer learning. Lastly , pretraining on the subset found by LM can guide the source model towards a flatter region in the downstream loss landscape, (see results in Fig. A4 ). The source model trained on the LM-pruned dataset achieves a higher flatness score than baselines, which aligns with the understanding of transfer learning in [78].

FM-based DP framework for self-supervised pretraining. The requirement for labeled source data in LM may pose limitations on the application of DP methods, particularly in the context of self-supervised learning ( SSL )-based pertaining. To address this limitation, we introduce a new method called FM (Feature Mapping). Unlike LM, FM determines the DP scheme using only the feature extractor network f S , the unlabeled source dataset D S , and the target dataset D T . This allows us to overcome the dependence on labeled source data, making FM applicable in SSL scenarios. The inspiration for FM is derived from the deep clustering technique [103-105] operating in the representation space, which can generate pseudo source labels using cluster indices provided by e.g. , K -means. With the assistance of deep clustering, we can represent the unlabeled source dataset D S as D S = {C (1) S , C (2) S , . . . , C ( K ) S } , where C ( k ) S denotes the set of source data samples within cluster k . Building upon the similar spirit as LM, we propose ranking the importance of pseudo source classes for DP by evaluating the source feature extractor's responsiveness to target data samples. To achieve this objective, we quantify the responsiveness of f S for a target sample t as follows:

where E x ∈C ( k ) S [ f S ( x )] is the centroid of source data within the cluster k , and r ( t ) is the nearest pseudo label as the responsiveness of f S against t . The FM score then integrates (1) with (LM):

where different from (LM), K represents the number of pseudo source classes produced by deep clustering, and r ( t j ) corresponds to the source feature extractor's prediction on t j . It is important to note that the value of K is a free parameter of the deep clustering process. Our empirical study in Fig. A2 shows that FM is quite robust to the choice of K without sacrificing the benefit of DP for

transfer learning compared to using the unpruned source dataset. Lastly, it is worth mentioning that FM can also be applied in the context of supervised pretraining by specifying data labels as clusters.

5 Experiments

In this section, we provide a comprehensive set of experiments and analyses to showcase the effectiveness of our proposed methods (LM and FM) in diverse transfer learning scenarios.

5.1 Experiment setup

Datasets and models. In line with existing transfer learning benchmarks [41, 106], we utilize ImageNet-1K [55] for pretraining and 8 datasets as downstream tasks. These datasets include DTD [107], Flowers102 [102], UCF101 [108], Food101 [109], SUN397 [110], OxfordPets [56], StanfordCars [57], and CIFAR10 [111]. Please refer to Tab. A1 for more details about the datasets. As discussed in Sec. 4, we utilize ResNet-18 (RN-18) [54] as the surrogate source model for pruning source classes. This method significantly reduces the computational cost associated with DP, making the process more efficient. Subsequently, a range of larger models, e.g., ResNet-101 (RN-101) and ViT-B/16 [112], are trained on the (pruned) ImageNet and then finetuned on downstream tasks.

Baselines, training, and evaluation. By examining the performance of the existing 8 DP baselines as shown in Fig. 2, our experiments focus on two of the most effective methods: ① GRAND [26] and ② MODERATE [3], together with ③ RANDOM (the random pruning strategy). In Fig. A1, we show more results of the rest DP baselines. Unfortunately, we are unable to include the existing data attribution method [41] as our baseline, as it does not release its pruning results, and we are unable to repeat its experiments due to the need for intensive computations. Unless specified otherwise, we focus on the supervised pretraining setting in the experiments. For self-supervised pretraining, we follow the implementation of MOCOV2 [45]. The finetuning strategies employed include LP (linear probing), which finetunes the classification head with fixed feature extractor, and FF (full finetuning), which finetunes the entire source model. For FM-based DP method, we utilize K-means clustering to group the ImageNet training data points into K = 2000 clusters for the computation of (1).

In accordance with the terminology convention in model pruning [87, 113-115], we refer the term ' winning subsets ' to the obtained source subsets that do not compromise downstream performance. Among these winning subsets, we identify the one with the highest pruning ratio as the ' best winning subset '. We then evaluate the performance of DP methods from the following aspects: the downstream performance of the model pretrained on the pruned source dataset obtained by various DP methods, and the pruning ratio of the best winning subset achieved by DP methods, accompanied by the corresponding time saved in the pretraining phase.

5.2 Experiment results

LM/FM improves transfer learning accuracy by identifying 'winning subsets'. We first showcase the significant improvements achieved by our proposed DP methods (LM and FM) compared to baselines. Our methods successfully identify winning subsets of the ImageNet, yielding transfer accuracy on par or even better than scenarios without pruning.

Fig. 4 presents the downstream accuracy of transfer learning vs. different pruning ratios. Here DP is performed using the surrogate model (RN-18) on ImageNet for 8 downstream tasks. The source model (RN-101) is then trained on the pruned ImageNet and the transfer learning accuracy is assessed through LP (linear probing) and FF (full finetuning). We also present the downstream performance without pruning the source dataset (No Prune) as a reference for winning subsets. As we can see, both LM and FM significantly outperform the baselines by a substantial margin. Notably, LM and FM consistently identify winning subsets with significantly larger pruning ratios in all settings. This highlights the effectiveness of our proposed methods in achieving substantial dataset pruning without hurting downstream performance. Furthermore, we observe that the downstream performance of LM and FM initially improves and then declines as the pruning ratio increases. This is not surprising, since the initial increase in performance corresponds to the scenario where harmful source classes are removed, consistent with [41]. When the source dataset continues to shrink, the performance inevitably decreases, as a natural consequence of reducing the size of the source dataset. Moreover, in some datasets of small sizes ( e.g. , OxfordPets), LP achieves performance comparable to FF. This

0%

20%

40%

60%

80%

Source Data Pruning Ratio

76

78

80

82

84

86

88

90

Target Test Accuracy (%)

OxfordPets-

LP

Random

Moderate

GraNd

LM (Ours)

FM (Ours)

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

47

48

49

50

51

52

Target Test Accuracy (%)

SUN397-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

52

54

56

58

60

62

64

Target Test Accuracy (%)

Food101-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

52

54

56

58

60

62

64

66

68

Target Test Accuracy (%)

DTD-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

30

35

40

45

Target Test Accuracy (%)

StanfordCars-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

55.0

57.5

60.0

62.5

65.0

67.5

70.0

72.5

Target Test Accuracy (%)

UCF101-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

82

83

84

85

86

87

88

89

90

Target Test Accuracy (%)

CIFAR10-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

89

90

91

92

93

94

95

Target Test Accuracy (%)

Flowers102-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

76

78

80

82

84

86

88

90

92

Target Test Accuracy (%)

OxfordPets-

FF

Random

Moderate

GraNd

LM (Ours)

FM (Ours)

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

42

44

46

48

50

52

54

56

Target Test Accuracy (%)

SUN397-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

78

79

80

81

Target Test Accuracy (%)

Food101-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

56

58

60

62

64

66

68

Target Test Accuracy (%)

DTD-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

85

86

87

88

89

90

Target Test Accuracy (%)

StanfordCars-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

58

60

62

64

66

68

70

72

74

Target Test Accuracy (%)

UCF101-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

95.0

95.5

96.0

96.5

97.0

Target Test Accuracy (%)

CIFAR10-

FF

0%

20%

40%

60%

80%

Source Data Pruning Ratio

96

97

98

99

Target Test Accuracy (%)

Flowers102-

FF

is attributed to the fact that the performance gap between LP and FF using the large-scale RN-101 model on small-scale datasets tends to diminish.

Tab. 2 highlights the effectiveness of FM-based DP in the self-supervised pretraining setup for three representative downstream tasks. As we can see, the transfer learning accuracy achieved by using FM consistently outperforms baselines in the self-supervised pretraining paradigm. FM can identify winning subsets for transfer learning even in the challenging regime of large pruning ratios, ranging from 50% to 80% . For instance, in the case of SUN397, FM-based winning subsets achieves a pruning ratio up to 70% , and for Flowers102 the maximum pruning ratio is 60% . These pruning merits align with the findings for FM in the supervised pretraining setup, as illustrated in Tab. 1.

DP enhances the efficiency of source pretraining. Tab. 3 displays the computation time required to obtain the pretrained source model using LM at different pruning ratios. The reported time consumption includes the entire pipeline, encompassing surrogate model training (RN18), DP process, and source model training (RN-

101) on the pruned ImageNet dataset. The runtime cost of the conventional transfer learning on the full ImageNet dataset for RN-101 is also listed as a reference. As we can see, DP enjoys high efficiency merit of source training. Taking the 5 . 4 hours required for source training on the full ImageNet dataset as a reference, LM-enabled 20% pruned ImageNet achieves a 15% reduction in training time. Moreover, the efficiency advantage increases to 76% when the pruning ratio reaches 80% and these computational benefits do not sacrifice transfer learning accuracy at all.

Next, we compare the efficiency of our methods with the state-of-the-art dataset pruning baselines in Tab. 4, including GRAND, MODERATE, and the brute-force method proposed in [41]. We showcase the efficiency comparison with a pruning ratio of 60% . As we can see, our method

settings follow Fig. 4.

(even using a surrogate model) achieves a substantial computation efficiency improvement over other methods. In particular, [41] involves training thousands of source models. Thus, its computational cost is typically unaffortable in experiments.

DP enables efficient adversarial pretraining. Tab. 5 showcases the improvement in transfer learning accuracy and efficiency achieved by our proposed LP-based DP method when incorporating adversarial training ( AT ) [116] on either the full or pruned ImageNet dataset. This transfer learning setup is motivated by the findings in [58], showing that enhancing the robustness of the source model against adversarial attacks through AT can improve transfer learning accuracy in both LP and FF-based finetuning scenarios. We then report downstream accuracies

using LP and FF on two specific downstream datasets: SUN397 and DTD, which are intentionally chosen due to the large room for improvement in transfer learning accuracy, as shown in Fig. 4. We also determine the pruning ratios for LM by selecting those that led to the best winning subsets. Our experimental results demonstrate that employing AT on both the unpruned and pruned source datasets can improve transfer learning accuracy. Specifically, we refer to AT on the original unpruned ImageNet dataset as DENSE-AT, and AT on the LM-pruned ImageNet dataset as LM-AT. One notable advantage of integrating LM into AT is the significant improvement in computation efficiency. A key highlight of this approach is that LM-AT achieves a similar computation time as the standard source training on ImageNet (DENSE), while exhibiting almost no accuracy drop compared to DENSE-AT. This observation demonstrates the potential to accelerate AT through DP.

Robustness against the surrogate model size. To explore the sensitivity of our proposed method to the surrogate model's size and accuracy, we show the transfer learning performance on the task OxfordPets against different surrogate model sizes in Fig. 5 . It is evident that even though the performance of the surrogate model on the source dataset (ImageNet) decreases, the downstream performance of RN-101 pretrained on the LM-based pruned source subsets remains relatively stable. Further, Tab. A6 compares the class indices selected by the corresponding surrogate models. Interestingly, the most relevant source class selections exhibit a high level of agreement across surrogate models of differing sizes.

LM shows robust behavior with different surrogate models, even if they yield different performance on the source dataset due to their different model capacities. For example, RN-32s only achieves a test accuracy of 40 . 77% on the source dataset, but it manages to achieve the same maximum pruning ratio of the winning subsets as RN-18. This underscores that LM can accommodate a small surrogate model of 1% the size of the pretrained model (Rn-32s vs. RN-101). Moreover, it reveals that the surrogate model does not need to attain exceptional performance

0%

20%

40%

60%

80%

Source Data Pruning Ratio

85

86

87

88

89

90

91

Target Test Accuracy (%)

ResNet-20s

ResNet-32s

ResNet-44s

ResNet-56s

ResNet-18

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

85

86

87

88

89

90

91

Target Test Accuracy (%)

VGG-2

VGG-4

ResNet-18

No Prune

(a) ResNets family

(b) VGG family

in the source domain for dataset pruning in transfer learning.

DP for ViT. In Tab. 6 , we present the consistent transfer learning improvement achieved by LM for Vision Transformer (ViT) pretraining. Specifically, we utilize ViT-B/16 as the source model, RN-18 as the surrogate model for DP, and use FF on the source model for the downstream tasks studied in Tab. 2. As we can see, LM yields a better transfer learning accuracy than baselines at different pruning ratios, consistent with the previous results on RN-101. This observation highlights the model-agnostic advantage of our proposed method in transfer learning. We also observe the pruning ratio of the best winning subset achieved for ViT is smaller compared to ResNet. This discrepancy is possibly due to the higher data requirements of ViT during pretraining [112].

Additional results. We conduct ablation studies on the cluster number for FM in Fig. A2 and the results of selecting data proposed by LM in reverse order in Fig. A3. We shows FM can be applied to more advanced SSL framework in Tab. A3. We also investigate the performance of our proposed method in the multi-task setting in Fig. A5, and the scenarios with data biases in Tab. A4. We further examine our method on the few-shot benchmark in Tab. A5. We also present feature distribution analysis in Fig. A6 and image examples for the top classes selected by LM/FM in Fig. A7.

Limitations. Despite the remarkable empirical results above, we admit LM/FM mainly caters to exploring the source data influence in a specific downstream task, and the largest pruning ratio is reduced as more downstream tasks are considered (see Fig. A5), which we consider as a limitation. Meanwhile, we acknowledge that the promising performance achieved by our method still lacks rigorous theoretical analysis, despite the feature distribution and model flatness analysis, which we believe will be a challenging but exciting future work on DP for transfer learning.

6 Conclusion

In this paper, we first formally define and investigate the problem of DP (dataset pruning) for transfer learning. Recognizing the ineffectiveness of conventional DP methods in the transfer learning setting, we proposed two novel and efficient DP techniques, label mapping and feature mapping. Extensive experiments demonstrate that both methods can effectively prune a large ratio of source dataset and substantially reduce the pretraining computational costs without sacrificing the downstream performance in various transfer learning settings.

References

B. Zhao, K. R. Mopuri, and H. Bilen, 'Dataset condensation with gradient matching,' arXiv preprint arXiv:2006.05929 , 2020. 1

B. Zhao and H. Bilen, 'Dataset condensation with differentiable siamese augmentation,' in International Conference on Machine Learning . PMLR, 2021, pp. 12 674-12 685. 1

X. Xia, J. Liu, J. Yu, X. Shen, B. Han, and T. Liu, 'Moderate coreset: A universal method of data selection for real-world data-efficient deep learning,' in The Eleventh International Conference on Learning Representations , 2023. 1, 5, 7

R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, 'Imagenettrained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,' arXiv preprint arXiv:1811.12231 , 2018. 1

A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, 'Adversarial examples are not bugs, they are features,' Advances in neural information processing systems , vol. 32, 2019.

T. Gu, B. Dolan-Gavitt, and S. Garg, 'Badnets: Identifying vulnerabilities in the machine learning model supply chain,' arXiv preprint arXiv:1708.06733 , 2017. 1

H. Wei, L. Feng, X. Chen, and B. An, 'Combating noisy labels by agreement: A joint training method with co-regularization,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 13 726-13 735. 1, 3

L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei, 'Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels,' in International conference on machine learning . PMLR, 2018, pp. 2304-2313.

C. Guo, B. Zhao, and Y. Bai, 'Deepcore: A comprehensive library for coreset selection in deep learning,' in Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I . Springer, 2022, pp. 181-195. 1, 3

Z. Borsos, M. Mutny, and A. Krause, 'Coresets via bilevel optimization for continual learning and streaming,' Advances in Neural Information Processing Systems , vol. 33, pp. 14 87914 890, 2020. 1, 2, 3, 5

M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon, 'An empirical study of example forgetting during deep neural network learning,' arXiv preprint arXiv:1812.05159 , 2018. 1, 2, 3

F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, 'End-to-end incremental learning,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 233-248. 3

R. Aljundi, M. Lin, B. Goujaud, and Y. Bengio, 'Gradient based sample selection for online continual learning,' Advances in neural information processing systems , vol. 32, 2019. 1, 3

O. Sener and S. Savarese, 'Active learning for convolutional neural networks: A core-set approach,' arXiv preprint arXiv:1708.00489 , 2017. 1, 2, 3

S. Chatterjee and A. S. Hadi, 'Influential observations, high leverage points, and outliers in linear regression,' Statistical science , pp. 379-393, 1986. 1, 2, 3

R. D. Cook, 'Assessment of local influence,' Journal of the Royal Statistical Society: Series B (Methodological) , vol. 48, no. 2, pp. 133-155, 1986.

W. Thomas and R. D. Cook, 'Assessing influence on predictions from generalized linear models,' Technometrics , vol. 32, no. 1, pp. 59-65, 1990.

B.-C. Wei, Y.-Q. Hu, and W.-K. Fung, 'Generalized leverage and its applications,' Scandinavian Journal of statistics , vol. 25, no. 1, pp. 25-37, 1998. 3

P. W. Koh and P. Liang, 'Understanding black-box predictions via influence functions,' in International conference on machine learning . PMLR, 2017, pp. 1885-1894. 2, 3, 5

A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov, 'Scaling up influence functions,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, 2022, pp. 8179-8186. 5

H. Guo, N. F. Rajani, P. Hase, M. Bansal, and C. Xiong, 'Fastif: Scalable influence functions for efficient model interpretation and debugging,' arXiv preprint arXiv:2012.15781 , 2020.

S. Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and P. Li, 'Dataset pruning: Reducing training data by examining generalization influence,' arXiv preprint arXiv:2205.09329 , 2022. 2, 3

S. Kong, Y. Shen, and L. Huang, 'Resolving training biases via influence-based data relabeling,' in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=EskfH0bwNVn 1, 2, 3, 5

G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger, 'Identifying mislabeled data using the area under the margin ranking,' Advances in Neural Information Processing Systems , vol. 33, pp. 17 044-17 056, 2020. 1, 2, 3

M. Welling, 'Herding dynamical weights to learn,' in Proceedings of the 26th Annual International Conference on Machine Learning , 2009, pp. 1121-1128. 2, 3

M. Paul, S. Ganguli, and G. K. Dziugaite, 'Deep learning on a data diet: Finding important examples early in training,' Advances in Neural Information Processing Systems , vol. 34, pp. 20 596-20 607, 2021. 2, 3, 5, 7

G. Pruthi, F. Liu, S. Kale, and M. Sundararajan, 'Estimating training data influence by tracing gradient descent,' Advances in Neural Information Processing Systems , vol. 33, pp. 19 92019 930, 2020. 3

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, 'icarl: Incremental classifier and representation learning,' in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2017, pp. 2001-2010. 1, 2, 3

D. Feldman and M. Langberg, 'A unified framework for approximating and clustering data,' in Proceedings of the forty-third annual ACM symposium on Theory of computing , 2011, pp. 569-578. 1, 3

J. Ju, H. Jung, Y. Oh, and J. Kim, 'Extending contrastive learning to unsupervised coreset selection,' IEEE Access , vol. 10, pp. 7704-7715, 2022.

J. Huggins, T. Campbell, and T. Broderick, 'Coresets for scalable bayesian logistic regression,' Advances in Neural Information Processing Systems , vol. 29, 2016.

T. Campbell and T. Broderick, 'Automated scalable bayesian inference via hilbert coresets,' The Journal of Machine Learning Research , vol. 20, no. 1, pp. 551-588, 2019.

C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, 'Variational continual learning,' arXiv preprint arXiv:1710.10628 , 2017.

S. Farquhar and Y. Gal, 'Towards robust evaluations of continual learning,' arXiv preprint arXiv:1805.09733 , 2018.

S. Kim, S. Bae, and S.-Y. Yun, 'Coreset sampling from open-set for fine-grained self-supervised learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 7537-7547. 3

Y. Yang, H. Kang, and B. Mirzasoleiman, 'Towards sustainable learning: Coresets for dataefficient deep learning,' arXiv preprint arXiv:2306.01244 , 2023. 1, 3

S. J. Pan and Q. Yang, 'A survey on transfer learning,' IEEE Transactions on knowledge and data engineering , vol. 22, no. 10, pp. 1345-1359, 2010. 2, 4

L. Torrey and J. Shavlik, 'Transfer learning,' in Handbook of research on machine learning applications and trends: algorithms, methods, and techniques . IGI global, 2010, pp. 242-264.

L. Yang, S. Hanneke, and J. Carbonell, 'A theory of transfer learning with applications to active learning,' Machine learning , vol. 90, pp. 161-189, 2013. 2, 4

B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos, 'Beyond neural scaling laws: beating power law scaling via data pruning,' in Advances in Neural Information Processing Systems , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https://openreview.net/forum?id=UmvSlP-PyV 2

S. Jain, H. Salman, A. Khaddaj, E. Wong, S. M. Park, and A. Madry, 'A data-based perspective on transfer learning,' arXiv preprint arXiv:2207.05739 , 2022. 2, 3, 4, 5, 7, 9, 18

P. Sattigeri, S. Ghosh, I. Padhi, P. Dognin, and K. R. Varshney, 'Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting,' arXiv preprint arXiv:2212.06803 , 2022. 2

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, 'A simple framework for contrastive learning of visual representations,' in International conference on machine learning . PMLR, 2020, pp. 1597-1607. 2, 4

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, 'Momentum contrast for unsupervised visual representation learning,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 9729-9738.

X. Chen, H. Fan, R. Girshick, and K. He, 'Improved baselines with momentum contrastive learning,' arXiv preprint arXiv:2003.04297 , 2020. 7

T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, 'Big self-supervised models are strong semi-supervised learners,' Advances in neural information processing systems , vol. 33, pp. 22 243-22 255, 2020. 4

Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, 'Unsupervised feature learning via non-parametric instance discrimination,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3733-3742.

Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, 'What makes for good views for contrastive learning?' Advances in neural information processing systems , vol. 33, pp. 6827-6839, 2020.

J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , 'Bootstrap your own latent-a new approach to self-supervised learning,' Advances in neural information processing systems , vol. 33, pp. 21 271-21 284, 2020.

Y. M. Asano, C. Rupprecht, and A. Vedaldi, 'Self-labelling via simultaneous clustering and representation learning,' arXiv preprint arXiv:1911.05371 , 2019.

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, 'Unsupervised learning of visual features by contrasting cluster assignments,' Advances in neural information processing systems , vol. 33, pp. 9912-9924, 2020.

J. Li, P. Zhou, C. Xiong, and S. C. Hoi, 'Prototypical contrastive learning of unsupervised representations,' arXiv preprint arXiv:2005.04966 , 2020. 2, 4

V. Feldman and C. Zhang, 'What neural networks memorize and why: Discovering the long tail via influence estimation,' Advances in Neural Information Processing Systems , vol. 33, pp. 2881-2891, 2020. 2, 3, 5

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778. 2, 5, 6, 7

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , 'Imagenet large scale visual recognition challenge,' International journal of computer vision , vol. 115, pp. 211-252, 2015. 2, 6, 7

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, 'Cats and dogs,' in 2012 IEEE conference on computer vision and pattern recognition . IEEE, 2012, pp. 3498-3505. 2, 5, 7, 22

J. Krause, M. Stark, J. Deng, and L. Fei-Fei, '3d object representations for fine-grained categorization,' in Proceedings of the IEEE international conference on computer vision workshops , 2013, pp. 554-561. 2, 7

H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry, 'Do adversarially robust imagenet models transfer better?' Advances in Neural Information Processing Systems , vol. 33, pp. 3533-3545, 2020. 3, 4, 9

P. K. Agarwal, S. Har-Peled, and K. R. Varadarajan, 'Approximating extent measures of points,' Journal of the ACM (JACM) , vol. 51, no. 4, pp. 606-635, 2004. 3

S. Har-Peled and S. Mazumdar, 'On coresets for k-means and k-median clustering,' in Proceedings of the thirty-sixth annual ACM symposium on Theory of computing , 2004, pp. 291-300.

D. Feldman, M. Schmidt, and C. Sohler, 'Turning big data into tiny data: Constant-size coresets for k-means, pca, and projective clustering,' SIAM Journal on Computing , vol. 49, no. 3, pp. 601-657, 2020. 3

S. Basu, P. Pope, and S. Feizi, 'Influence functions in deep learning are fragile,' arXiv preprint arXiv:2006.14651 , 2020. 3

J. Jia, Y . Zhang, D. Song, S. Liu, and A. Hero, 'Robustness-preserving lifelong learning via dataset condensation,' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1-5. 3

Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan, 'Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm,' arXiv preprint arXiv:2303.14382 , 2023. 3

A. Ghorbani and J. Zou, 'Data shapley: Equitable valuation of data for machine learning,' in International Conference on Machine Learning . PMLR, 2019, pp. 2242-2251. 3

R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. Song, and C. J. Spanos, 'Towards efficient data valuation based on the shapley value,' in The 22nd International Conference on Artificial Intelligence and Statistics . PMLR, 2019, pp. 1167-1176. 3

R. Jia, D. Dao, B. Wang, F. A. Hubis, N. M. Gurel, B. Li, C. Zhang, C. J. Spanos, and D. Song, 'Efficient task-specific data valuation for nearest neighbor algorithms,' arXiv preprint arXiv:1908.08619 , 2019. 3

J. Lin, A. Zhang, M. Lécuyer, J. Li, A. Panda, and S. Sen, 'Measuring the effect of training data on deep learning predictions via randomized experiments,' in International Conference on Machine Learning . PMLR, 2022, pp. 13 468-13 504. 3

K. Guu, A. Webson, E. Pavlick, L. Dixon, I. Tenney, and T. Bolukbasi, 'Simfluence: Modeling the influence of individual training examples by simulating training runs,' arXiv preprint arXiv:2303.08114 , 2023.

Z. Hammoudeh and D. Lowd, 'Identifying a training-set attack's target using renormalized influence estimation,' in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , 2022, pp. 1367-1381. 3

S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry, 'Trak: Attributing model behavior at scale,' arXiv preprint arXiv:2303.14186 , 2023. 3

H. A. Just, F. Kang, J. T. Wang, Y. Zeng, M. Ko, M. Jin, and R. Jia, 'Lava: Data valuation without pre-specified learning algorithms,' arXiv preprint arXiv:2305.00054 , 2023. 3

A. Ilyas, S. M. Park, L. Engstrom, G. Leclerc, and A. Madry, 'Datamodels: Predicting predictions from training data,' arXiv preprint arXiv:2202.00622 , 2022. 3

C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, 'Representer point selection for explaining deep neural networks,' Advances in neural information processing systems , vol. 31, 2018. 3

Y. Song and S. Ermon, 'Generative modeling by estimating gradients of the data distribution,' Advances in neural information processing systems , vol. 32, 2019. 3

Z. Deng, L. Zhang, K. Vodrahalli, K. Kawaguchi, and J. Y. Zou, 'Adversarial training helps transfer learning via better representations,' Advances in Neural Information Processing Systems , vol. 34, pp. 25 179-25 191, 2021. 4

H. Liu, S. M. Xie, Z. Li, and T. Ma, 'Same pre-training loss, better downstream: Implicit bias matters for language models,' arXiv preprint arXiv:2210.14199 , 2022. 4

H. Liu, M. Long, J. Wang, and M. I. Jordan, 'Towards understanding the transferability of deep representations,' arXiv preprint arXiv:1909.12031 , 2019. 4, 5, 6

M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich, 'To transfer or not to transfer,' in NIPS 2005 workshop on transfer learning , vol. 898, no. 3, 2005. 4

L. Duan, D. Xu, and S.-F. Chang, 'Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach,' in 2012 IEEE Conference on computer vision and pattern recognition . IEEE, 2012, pp. 1338-1345.

L. Ge, J. Gao, H. Ngo, K. Li, and A. Zhang, 'On handling negative transfer and imbalanced distributions in multiple source transfer learning,' Statistical Analysis and Data Mining: The ASA Data Science Journal , vol. 7, no. 4, pp. 254-271, 2014.

J. Yosinski, J. Clune, Y . Bengio, and H. Lipson, 'How transferable are features in deep neural networks?' Advances in neural information processing systems , vol. 27, 2014.

Z. Cao, M. Long, J. Wang, and M. I. Jordan, 'Partial transfer learning with selective adversarial networks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 2724-2732.

Z. Wang, Z. Dai, B. Póczos, and J. Carbonell, 'Characterizing and avoiding negative transfer,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 11 293-11 302. 4

D. Guo, A. M. Rush, and Y. Kim, 'Parameter-efficient transfer learning with diff pruning,' arXiv preprint arXiv:2012.07463 , 2020. 4

B. Liu, Y. Cai, Y . Guo, and X. Chen, 'Transtailor: Pruning the pre-trained model for improved transfer learning,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 10, 2021, pp. 8627-8634.

T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, M. Carbin, and Z. Wang, 'The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 16 306-16 316. 7

T. Chen, Y. Cheng, Z. Gan, L. Yuan, L. Zhang, and Z. Wang, 'Chasing sparsity in vision transformers: An end-to-end exploration,' Advances in Neural Information Processing Systems , vol. 34, pp. 19 974-19 988, 2021.

S. Myung, I. Huh, W. Jang, J. M. Choe, J. Ryu, D. Kim, K.-E. Kim, and C. Jeong, 'Pac-net: A model pruning approach to inductive transfer learning,' in International Conference on Machine Learning . PMLR, 2022, pp. 16 240-16 252. 4

H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola, 'Exploring visual prompts for adapting large-scale models,' arXiv preprint arXiv:2203.17274 , vol. 1, no. 3, p. 4, 2022. 4

G. F. Elsayed, I. Goodfellow, and J. Sohl-Dickstein, 'Adversarial reprogramming of neural networks,' arXiv preprint arXiv:1806.11146 , 2018.

P.-Y. Chen, 'Model reprogramming: Resource-efficient cross-domain machine learning,' arXiv preprint arXiv:2202.10629 , 2022.

Y. Zheng, X. Feng, Z. Xia, X. Jiang, A. Demontis, M. Pintor, B. Biggio, and F. Roli, 'Why adversarial reprogramming works, when it fails, and how to tell the difference,' arXiv preprint arXiv:2108.11673 , 2021.

P. Neekhara, S. Hussain, S. Dubnov, and F. Koushanfar, 'Adversarial reprogramming of text classification neural networks,' arXiv preprint arXiv:1809.01829 , 2018.

P. Neekhara, S. Hussain, J. Du, S. Dubnov, F. Koushanfar, and J. McAuley, 'Cross-modal adversarial reprogramming,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2022, pp. 2427-2435.

L. Chen, Y. Fan, and Y. Ye, 'Adversarial reprogramming of pretrained neural networks for fraud detection,' in Proceedings of the 30th ACM International Conference on Information & Knowledge Management , 2021, pp. 2935-2939.

A. Chen, Y. Yao, P.-Y. Chen, Y. Zhang, and S. Liu, 'Understanding and improving visual prompting: A label-mapping perspective,' arXiv preprint arXiv:2211.11635 , 2022.

A. Chen, P. Lorenz, Y. Yao, P.-Y. Chen, and S. Liu, 'Visual prompting for adversarial robustness,' in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1-5. 18

G. Zhang, Y. Zhang, Y. Zhang, W. Fan, Q. Li, S. Liu, and S. Chang, 'Fairness reprogramming,' arXiv preprint arXiv:2209.10222 , 2022. 4

G. Zhang, H. Zhao, Y. Yu, and P. Poupart, 'Quantifying and improving transferability in domain generalization,' Advances in Neural Information Processing Systems , vol. 34, pp. 10 957-10 970, 2021. 4

Y. Zhang, P. Khanduri, I. Tsaknakis, Y. Yao, M. Hong, and S. Liu, 'An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning,' arXiv preprint arXiv:2308.00788 , 2023. 5

M.-E. Nilsback and A. Zisserman, 'Automated flower classification over a large number of classes,' in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008, pp. 722-729. 5, 7

M. Caron, P. Bojanowski, A. Joulin, and M. Douze, 'Deep clustering for unsupervised learning of visual features,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 132-149. 6

X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan, 'Clusterfit: Improving generalization of visual representations,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 6509-6518.

L. Fan, S. Liu, P.-Y. Chen, G. Zhang, and C. Gan, 'When does contrastive learning preserve adversarial robustness from pretraining to finetuning?' Advances in Neural Information Processing Systems , vol. 34, pp. 21 480-21 492, 2021. 6

U. Evci, V. Dumoulin, H. Larochelle, and M. C. Mozer, 'Head2toe: Utilizing intermediate representations for better transfer learning,' in International Conference on Machine Learning . PMLR, 2022, pp. 6009-6033. 7

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, 'Describing textures in the wild,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 3606-3613. 7

K. Soomro, A. R. Zamir, and M. Shah, 'Ucf101: A dataset of 101 human actions classes from videos in the wild,' arXiv preprint arXiv:1212.0402 , 2012. 7

L. Bossard, M. Guillaumin, and L. V. Gool, 'Food-101-mining discriminative components with random forests,' in European conference on computer vision . Springer, 2014, pp. 446-461. 7

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, 'Sun database: Large-scale scene recognition from abbey to zoo,' in 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 2010, pp. 3485-3492. 7

A. Krizhevsky, G. Hinton et al. , 'Learning multiple layers of features from tiny images,' cs.utoronto.ca , 2009. 7

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , 'An image is worth 16x16 words: Transformers for image recognition at scale,' arXiv preprint arXiv:2010.11929 , 2020. 7, 10, 18

J. Frankle and M. Carbin, 'The lottery ticket hypothesis: Finding sparse, trainable neural networks,' arXiv preprint arXiv:1803.03635 , 2018. 7

T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin, 'The lottery ticket hypothesis for pre-trained bert networks,' Advances in neural information processing systems , vol. 33, pp. 15 834-15 846, 2020.

Y. Zhang, Y. Yao, P. Ram, P. Zhao, T. Chen, M. Hong, Y. Wang, and S. Liu, 'Advancing model pruning via bi-level optimization,' arXiv preprint arXiv:2210.04092 , 2022. 7

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, 'Towards deep learning models resistant to adversarial attacks,' arXiv preprint arXiv:1706.06083 , 2017. 9

H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer, 'Multiscale vision transformers,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 6824-6835. 19

D. Hendrycks and T. Dietterich, 'Benchmarking neural network robustness to common corruptions and perturbations,' arXiv preprint arXiv:1903.12261 , 2019. 21

X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy et al. , 'A large-scale study of representation learning with the visual task adaptation benchmark,' arXiv preprint arXiv:1910.04867 , 2019. 22

N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, 'On large-batch training for deep learning: Generalization gap and sharp minima,' arXiv preprint arXiv:1609.04836 , 2016. 22

D. Bisla, J. Wang, and A. Choromanska, 'Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2022, pp. 8299-8339. 22

W. J. Maddox, G. Benton, and A. G. Wilson, 'Rethinking parameter counting in deep models: Effective dimensionality revisited,' arXiv preprint arXiv:2003.02139 , 2020. 22

T. Liang, T. Poggio, A. Rakhlin, and J. Stokes, 'Fisher-rao metric, geometry, and complexity of neural networks,' in The 22nd international conference on artificial intelligence and statistics . PMLR, 2019, pp. 888-896. 22

L. Van der Maaten and G. Hinton, 'Visualizing data using t-sne.' Journal of machine learning research , vol. 9, no. 11, 2008. 23

Appendix

A Experiment Settings

A.1 Training Settings

General setups for DP (dataset pruning). Regardless of the choice of the downstream tasks and DP methods, each DP process will go through a three-stage paradigm, namely ❶ source dataset pruning, ❷ pretraining based on the pruned source dataset, and ❸ finetuning on the target dataset using either LP (linear probing) or FF (full finetuning). Regarding baseline methods for source dataset pruning, we strictly follow the baselines' settings provided in the literature. As stated before, we utilize a small surrogate model (ResNet-18) pretrained on the full ImageNet to conduct DP. The proportion of pruned class numbers to the total source class numbers determines pruning ratios for our methods: LM (label mapping) and FM (feature mapping). To ensure a fair comparison with other non-class-wise baselines (RANDOM, GRAND, MODERATE), the same pruning ratio is applied to the total number of training data points.

Pretraining setups. We keep our pretraining procedure consistent for all models and strictly follow the settings released by the existing work [41]. All the models are trained from scratch using Stochastic Gradient Descent (SGD) to minimize the standard cross-entropy loss in a multi-class classification setting. We use a batch size of 1024, momentum of 0.9, and a weight decay of 5 × 10 -4 . The training utilizes a cyclic learning rate schedule, which begins with an initial learning rate of 0.5 and peaks at the second epoch. During training, we incorporate data augmentations such as random resized cropping and random horizontal flipping.

Downstream task finetuning settings. Our approach to finetuning the pretrained model on downstream tasks involves using LP and FF. Details of the downstream datasets and the training configurations are presented in Tab. A1, following [98]. For LP, we employ the Adam optimizer, a multi-step decaying scheduler, and an initial learning rate of 0.1 across 50 total training epochs. As for FF, we utilize the Adam optimizer over 200 epochs with a cosine-annealing scheduler, an initial learning rate of 0 . 01 , and a weight decay of 5 × 10 -4 . All finetuning experiments employ a batch size of 256 and standard data augmentations, such as random resized cropping and horizontal flipping.

Training settings for SSL (self-supervised learning) and ViT. Our SSL training settings follow the configurations provided by MOCOV2. Details of the pretraining and finetuning stages can be accessed at https://github.com/facebookresearch/moco . For the training of ViTs, we rely on the setting released in the original ViT paper [112] (see ViT/B in Table 3).

B Additional Results

Expanded performance evaluation of in-domain DP methods across all downstream datasets. Fig. A1 expands on the performance comparisons in Fig. 2 , providing a more thorough evaluation of various in-domain DP methods on all eight downstream datasets. The trends observed are consistent with those in Fig. 2 : Random pruning shows a strong baseline method in DP for transfer learning compared to other state-of-the-art DP methods designed for non-transfer learning. This observation prompts us to explore more effective dataset pruning strategies for transfer learning. MODERATE and

GRAND are also demonstrating strong baselines, motivating us to choose them as the default DP baselines in Section 5.

0%

20%

40%

60%

80%

Source Data Pruning Ratio

86

88

90

92

94

Target Test Accuracy (%)

Flowers102-

LP

Random

Moderate

GraNd

SP

SSP

Forget

EL2N

InfSum

InfMax

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

35

40

45

50

55

Target Test Accuracy (%)

SUN397-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

40

45

50

55

60

Target Test Accuracy (%)

Food101-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

45

50

55

60

65

Target Test Accuracy (%)

DTD-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

20

25

30

35

40

45

Target Test Accuracy (%)

StanfordCars-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

45

50

55

60

65

70

Target Test Accuracy (%)

UCF101-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

80

82

84

86

88

Target Test Accuracy (%)

CIFAR10-

LP

0%

20%

40%

60%

80%

Source Data Pruning Ratio

76

78

80

82

84

86

88

90

Target Test Accuracy (%)

OxfordPets-

LP

The main results reported in Fig. 4 are stable: analysis with detailed numerical results and standard deviations. In Tab. A2 , we provide the exact numerical results used to generate Fig. 4 . These numbers give a more granular view of the performance comparisons. The results, calculated over three independent trials, show that the magnitude of the standard deviations is quite small relative to the mean values. This indicates that the trends and conclusions drawn from Fig. 4 are generally valid and not significantly affected by trial variations. To maintain readability and clarity, we choose not to include these minor standard deviations in Fig. 4 .

50%

60%

70%

80%

Source Data Pruning Ratio

20

30

40

50

60

70

Target Test Accuracy (%)

OxfordPets-

LP

Random

Moderate

GraNd

FM (2000 Cluster)

FM (1000 Cluster)

FM (500 Cluster)

No Prune

50%

60%

70%

80%

Source Data Pruning Ratio

80

81

82

83

84

85

Target Test Accuracy (%)

Flowers102-

LP

FM performance is robust to the choice of cluster number. The results presented in Fig. A2 provide insights into the sensitivity of FM performance to variations of the cluster number ( K ). This extended study from Tab. 2 shows that FM performance remains robust as the cluster number varies from 500 to 2000 , preserving the benefits of DP for transfer learning without much performance degradation. This implies that FM can provide reliable and stable results even when the hyperparameter settings are not optimal.

LMidentifies the most influential source classes. To validate that the classes with the highest LM scores are the most influential, we present the pruning trajectory in Fig. A3 , where pruning is executed in the reverse order of the class scores, different from the proposed DP implementation. That is, classes with the smallest scores are retained, while the ones with the highest scores are pruned. Remarkably, even a slight pruning ratio (such as 10% ) leads to a significant degradation in downstream performance. This evidence underscores the benefits of source classes with high LM scores in promoting the downstream performance.

FMcan be smoothly applied to MoCov3. We conducted experiments to illustrate the effectiveness of our proposed method (FM) when applied to the more recent SSL framework MoCov3 [117] using

0%

20%

40%

60%

80%

Source Data Pruning Ratio

40

50

60

70

80

90

Target Test Accuracy (%)

OxfordPets-

LP

Ordered

Reversed

No Prune

0%

20%

40%

60%

80%

Source Data Pruning Ratio

70

75

80

85

90

Target Test Accuracy (%)

OxfordPets-

FF

25%

50%

75%

100%

H

)

-Flatness -Flatness -Flatness -Flatness -Flatness

FRN

LPF

|

H

|

F

H

)

FRN

LPF

|

H

|

F

H

)

FRN

LPF

|

H

|

F

H

)

FRN

LPF

|

H

|

F

H

)

FRN

LPF

|

H

|

F

Dense

Random

Moderate

GraNd

LM

the ViT structure. In line with our SSL experiment plan detailed in Tab. 2 of the paper, Tab. A3 tested FM on three downstream datasets, specifically OxfordPets, Flowers102, and SUN397, with a source data class pruning ratio ranging from 50% to 80% . The results affirm that the principal conclusions drawn from MoCov2 remain consistent with MoCov3 on ViT. Our method, FM, successfully identifies data subsets within the source dataset that can be pruned at high ratios without compromising downstream performance (termed as 'winning subsets'). For instance, in one particular case, the FM-based winning subsets achieve a pruning ratio of up to 70% on OxfordPets.

Explore LM in the multi-task setting. A limitation of our method is its task-specific nature in data influence analysis. As our methods assess source data influence for specific downstream tasks, developing a universal DP solution for multiple tasks simultaneously is challenging. As a preliminary study, we examine the performance of source dataset pruning for multiple downstream tasks simultaneously in Fig. A5 . While LM can still identify winning subsets, the maximum pruning ratio diminishes as more tasks are considered. We will include this limitation in the Conclusion section.

LMhelps remove data biases through DP for a downstream task. To investigate the scenarios with data biases, we conducted experiments on CIFAR-10C (the out-of-distribution scenario) [118]. We first pruned ImageNet given CIFAR-10 as a downstream task and evaluated the model on CIFAR10C with different corruption types. Tab. A4 shows LM results with different pruning ratios for 5 strong perturbation types in CIFAR-10C. Impressively, LM can achieve winning subsets with pruning up to 80% , even better than on CIFAR-10, confirming our method's effectiveness to filter biased data points to some degree.

0%

20%

40%

60%

80%

Source Data Pruning Ratio

85

86

87

88

89

90

91

Target Test Accuracy (%)

Dataset: OxfordPets

No Prune

Single-Task

Multi-Task

0%

20%

40%

60%

80%

Source Data Pruning Ratio

50

55

60

65

Target Test Accuracy (%)

Dataset: DTD

No Prune

Single-Task

Multi-Task

0%

20%

40%

60%

80%

Source Data Pruning Ratio

85

86

87

88

89

Target Test Accuracy (%)

Dataset: StanfordCars

No Prune

Single-Task

Multi-Task

0%

20%

40%

60%

80%

Source Data Pruning Ratio

42

44

46

48

50

52

54

56

Target Test Accuracy (%)

Dataset: SUN397

No Prune

Single-Task

Multi-Task

Experiments on the few-shot benchmark VTAB [119]. We extend our experiments to include the VTAB benchmark[119]. As shown in Tab. A5, even in the few-shot setting, our major conclusions remain valid. LM can effectively pinpoint the most influential source subsets and eliminate the less valuable ones. For most tasks considered, winning subsets with pruning ratios up to 40% are successfully found.

Subsets obtained by LM improve the flatness of loss landscape on downstream tasks. Fig. A4 evaluates the flatness of the loss landscape of the model pretrained on an 80% -pruned source dataset and subsequently finetuned on the downstream OxfordPets dataset [56]. Better flatness in the loss landscape is typically associated with better transferability. We quantify this flatness using the measure of reversed sharpness calculated via five widely accepted metrics: ϵ -sharpness [120], low pass filter-based measure (LPF) [121], the max eigenvalue of the Hessian ( λ max ( H ) ) [122], the Frobenius norm of the Hessian ( | H | F ) [122], and Fisher Rao Norm (FRN) [123]. The results suggest that pretraining on the subset selected by LM leads the model towards a flatter region in

the downstream loss landscape, potentially contributing to the superior transferability of LM when compared to the baseline methods.

Abyssinian

Cat

American

Bulldog

Shiba

Inu

Black

English

Springer

Indigo

Bunting

Tennis

Cab

Loafer

Target

Class

Retained

Source Class

Pruned

Source Class

2

Feature distribution analysis. Fig. A6 provides visual explanations of DP at both deep representation (feature) and data levels. Here we focus on the LM method with a pruning ratio of 90% given the downstream dataset OxfordPets. The other settings are consistent with Fig. 4. In Fig. A6 (right) , we visualize the source ImageNet classes (including pruned ones and retrained ones) and target OxfordPets classes in terms of their class-wise feature centroids in a 2D space achieved by t-SNE[124]. The class feature centroid is obtained by averaging the features of data points within a class, extracted by the pretrained source model on the full ImageNet dataset. As we can see, all retained source classes are grouped together and are close to the target classes. This indicates that the source classes that the pruning process share the most resemblance with the target data. In contrast, the pruned source classes are more dispersed and located further away from the target data classes. Furthermore, Fig. A6 (left) exhibits image examples of target classes as well as pruned and retrained source classes. We observe that image examples in the retained source classes ( e.g. , relating to

animals) are semantically closer to the target data points (relating to pets) than the pruned ones. This highlights the ability of LM to effectively identify and retain the most relevant classes for the downstream tasks. We provide more examples for FM in Fig. A7.

Examining the top selected classes by FM and their image examples. In Fig. A7 , we showcase the top-10 source classes chosen by FM, as determined by the endowed scores. These selected classes closely correspond to the downstream datasets' subjects, demonstrating FM's effectiveness in identifying relevant classes for transfer learning. This finding also aligns with our observations in Fig. A6, showing that FM identifies source classes resembling downstream data.

Window

Screen

Honey

Comb

Velvet

Shower

Curtain

Chainlink

Fence

Dishrag

Wool

Jigsaw

Puzzle

Stone

Wall

Prayer

Mat

(a) DTD.

Bee

Pinwheel

Rosehip

Cypripedium

Calceolus

Head

Cabbage

Acron

Cabbage

Butterfly

Butterfly

Cardoon

(b) Flowers102.

Plate

Meatloaf

Chocolate

Sauce

Mashed

Potato

Hotpot

Hot Dog

Soup

Bowl

Consomme

Ice Cream

(c) Food101.

Egyptian

Cat

Persian

Cat

Siamese

Cat

Tabby

Cat

Samoyed

Basset

(d) OxfordPets.

Sports

Car

Wagon

Convertible

Pickup

Truck

Grille

Jeep

Limousine

Taxi

Racing

Car

(e) Stanfordcars.

Restaurant

Mobile

Home

Lakeshore

Valley

Worm

Fence

Palace

Desk

Terrace

Cinema

Prison

(f) Sun397.

Bars

Stretcher

Stage

Barbershop Ballplayer

Balance

Beam

Barbell

Racket

Lakeshore

Neck

Brace

(g) UCF101.

Nail

Syringe

Hair

Spray

Fox

Croquet

Weasel

Band

Aid

Muzzle

Lakeshore

be

a

iports

CH

pok

(h) CIFAR-10.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two new dataset pruning (DP) methods, label mapping and feature mapping, for transfer learning. The aim is to improve pretraining efficiency and fine-tuning accuracy on downstream target tasks without sacrificing performance. The proposed methods are evaluated on numerous transfer learning tasks and show significant speed-up during the pretraining stage.\n\nSTRENGTHS:\n+ The paper addresses an important problem of dataset pruning for transfer learning, which has not been explored much in previous studies.\n\n+ The proposed methods show significant improvement in pretraining efficiency and finetuning accuracy on downstream target tasks.\n\n+ The paper provides a unified viewpoint to integrate DP with transfer learning.\n\n+ The proposed methods are evaluated on numerous transfer learning tasks, demonstrating their broad applicability.\n\nWEAKNESSES:\n+ The paper does not consider other types of transfer learning scenarios, such as domain adaptation, multi-task learning, or meta-learning, which may have different challenges and requirements for dataset pruning.\n\n+ The paper does not conduct a thorough ablation study or sensitivity analysis to understand the impact of different design choices, such as the surrogate model, the cluster number, and the reverse order selection, on the performance and efficiency of the proposed methods.\n\n+ The paper does not provide any discussion of the limitations or failure cases of its method. This could limit the understanding of when and how to apply its method in practice. \n\n+ The paper does not provide a detailed analysis of the impact of the proposed methods on the size of the pruned dataset, which could be an important factor in some applications.\n\n+ The paper does not specifically address the question of whether it is necessary to compress data with a relatively small amount of data, but rather focuses on improving the efficiency of using the available data. It is important to note that while larger amounts of data can lead to better performance\n\n+ The proposed DP methods require access to a surrogate source model (e.g., ResNet-18), which may not be available or practical in all settings.\n\n+ The effectiveness of the proposed DP methods may depend on the quality and diversity of the source dataset, and it is unclear how well they would perform with low-quality or biased data.\n\n+ The proposed DP methods (label mapping and feature mapping) were evaluated on a limited set of downstream tasks, and it is unclear how well they would generalize to other tasks or domains.\n\nQUESTIONS:\nPlease see the weakness above, I would like the authors to address these limitations.\n\nLIMITATIONS:\nThis article examines the use of transfer learning for data pruning, but questions whether this topic is meaningful. Firstly, the dataset used in the article is very small, with a maximum size of no more than 100,000, making it too small for practical application. Additionally, the current trend is to train very large models using large amounts of data. Furthermore, is the method used in the article simply a combination of two fields, namely data compression and transfer learning? It is important to research truly meaningful topics.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of dataset pruning, which serves to smaller the dataset for faster computation and learning. The authors claim that some existing methods require overly much computation for selecting the pruned dataset and thus propose a method with less computational requirements. However, in line 163 the authors seem to suggest pruning dataclasses from the source dataset, which requires an additional training of the foundation model, which is usually not possible in practice.\n\nSTRENGTHS:\n* The proposed method does not require training or heavy computation when selecting the source classes. Moreover, the results in Figure 4 suggest that better test accuracy is achieved for higher pruning rates in a wide range of datasets. \n\n* Line 282-292 provides a clear overview of the training setup, which makes the experiments reproducible.\n\nWEAKNESSES:\n* The results in Figure 4 indicate that over a wide range of datasets about 70% of classes can be pruned. However, in Table 3 one can read that such pruning only saves about 3 hours of computation time. In what scenario is such saving critical? I would say that dataset pruning is complex and introduces many unknown biases, and I am not sure that this is outweighed by saving only three hours of computation. \n\n* Removing any datapoint from the dataset introduces, by definition, a bias in the dataset. What are the insights on such biases and has any analysis been done on class imbalances? Alternatively, has any analysis on datasets such as CIFAR-C and ImageNet-C been run to assess robustness [1,2]? \n\n* I would say the pruning of classes is specific to the target dataset. And usually, training of foundation models is not possible as the foundation models are usually trained by large institutions on large datasets and thus only fine tuning is an option. This paper seems to address the problem of pre-training models on a new dataset. What benefits does such an approach have over established foundation models? There have been papers to suggest that actually the pre-training on large datasets yields benefits for downstream performance [3].\n\n[1] Hendrycks, Dan, and Thomas Dietterich. \"Benchmarking neural network robustness to common corruptions and perturbations.\" arXiv preprint arXiv:1903.12261 (2019).\n\n[2] https://github.com/hendrycks/robustness/blob/master/ImageNet-C/create_c/make_cifar_c.py\n\n[3] Kolesnikov, Alexander, et al. \"Big transfer (bit): General visual representation learning.\" ECCV, 2020.\n\nQUESTIONS:\nHave any experiments been done on benchmarks for transfer learning, such as [1] or [2]?\n\n[1] Zhai, Xiaohua, et al. \"The visual task adaptation benchmark.\" (2019).\n\n[2] Triantafillou, Eleni, et al. \"Meta-dataset: A dataset of datasets for learning to learn from few examples.\" arXiv (2019).\n\nLIMITATIONS:\nThe paper focuses on computational efficiency and thus could reduce power consumption of learning algorithms at scale.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the problem of dataset pruning for transfer learning. The idea is to remove redundant samples in the source dataset to improve the source pre-training efficiency while maintaining the downstream fine-tuning accuracy. The label and feature mapping method proposed for the problem utilizes a surrogate model to compute a relatedness score for each source class. Source classes with lower scores are then pruned afterward. The proposed methods are compared to existing dataset pruning methods and demonstrate better performance on 8 downstream datasets.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow. The proposed problem is interesting and the proposed approaches seem to be reasonable.  \n- The experimental study on adversarial pre-training is inspiring and shows the potential for a wider extension of the problem to other areas.\n\nWEAKNESSES:\n- The problem assumes that the source pre-training dataset is available during downstream fine-tuning. However, this might not be realistic because the pre-training dataset is sometimes private and inaccessible, like medical domains. \n- The efficiency comparison with pre-training on the unpruned dataset is a little bit unfair to me (Table 3). In the standard pre-training and fine-tuning paradigm, we only need to pre-train once, and then we can fine-tune the pre-trained model on multiple tasks. However, in the proposed scenario, we need to pre-train on the pruned dataset for each downstream task every time. Pre-training on the unpruned dataset yields a generic model but requires more time. Although pre-training on the pruned dataset is faster, it outputs a task-specific model. It would be important to distinguish the trade-off to make a fair comparison. \n- The need for training a surrogate model might be a huge limitation because it cannot avoid training on the whole source dataset, which could be extremely large. Although the authors try to argue that the surrogate model can be small, it remains unclear how small it can be.\n\nQUESTIONS:\nPlease see the above weakness section and also the following questions. \n\n1) By assuming the availability of the pre-training dataset during fine-tuning, what would be the practical use cases for the proposed problem? What could be the problem’s additional cost and benefits, compared to the existing pre-training and fine-tuning paradigm, where we just need the pre-trained model, without the pre-training dataset? \n\n2) Table 3 only compares the time consumption for one downstream task. However, the scenario might become different if we consider multiple downstream tasks at once because we only need to pre-train once for the unpruned dataset. Given some downstream tasks, like 8 tasks in the paper, what would be the downstream accuracy if we use the same time budget for pre-training on pruned or unpruned datasets? \n\n3) How accurate does the surrogate model need to be for the proposed methods? For example, if we use a surrogate model even smaller than ResNet-18 and get a lower pre-training accuracy, how would the fine-tuning accuracy change?\n\nLIMITATIONS:\nAs mentioned in the conclusion of the paper, the paper mainly focuses on the experimental results of the proposed methods and lacks theoretical viewpoints/insights. Besides, the use of a surrogate model and the assumption of an accessible pre-training dataset might be some other potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe following paper tackles the problem of dataset pruning from the lens of transfer learning, where the main goal is to achieve lossless or improved transfer learning accuracy of the source model on a target task using less amount of source data. To achieve this, this paper proposes two dataset pruning algorithms, Label Mapping (LM) for supervised pretraining and Feature Mapping (FM) which is capable of finding winning subsets within source dataset for self-supervised pre-training by monitoring the model's responsiveness toward target data samples. Experiments show that incorporating LP and FM on the source dataset by 40-80% pruning ratio does not sacrifice the classification performance on downstream tasks while speeding up the pretraining process by a factor of 2-5. Furthermore, additional experiments on adversarial pretraining tasks show classification performance improvement of 1-2% on downstream tasks.\n\nSTRENGTHS:\n(+) The Paper is well-written and easy to understand. \n\n(+) The methods are highly intuitive, simple, and efficient, making them accessible for general-purpose usage.\n\n(+) Experiments show promising results on various benchmarks with regards to various metric of interest not limited to downstream task performance, and pre-training time, and leads to a more efficient adversarial learning\n\nWEAKNESSES:\n(-) For self-supervised learning experiments, I think the experiment is quite limited, where only one self-supervised method and one network architecture is used for evaluation. Perhaps adding more relevant self-supervised learning architectures such as MoCov3 [1], which uses ViT, or information-maximization self-supervised learning methods such as Barlow Twins [2] would strengthen the paper.\n\n(-) Minor typos: \n\na) In pg. 2, finetue -> fine-tune\n\nb) In pg. 7, savings -> saved\n\nQUESTIONS:\nAs pointed in Weakness, would the transfer learning results based on self-supervised learning pretraining holds for Feature Mapping (FM) on other self-supervised learning methods, i.e. MoCov3 [1] or Barlow Twins [2]?\n\n[1] https://arxiv.org/abs/2104.02057\n\n[2] https://arxiv.org/abs/2103.03230\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}