SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) Time consumption.
Figure 3: Preliminary studies on the usage of surrogate models for LM. (a) The downstream performance (on Flowers102) of using the source model ResNet-101 trained on the pruned ImageNet delivered by LM at different pruning ratios. Here LM is conducted using either RN-101 or a smaller surrogate model ResNet-18. (b) Computation time decomposition analysis for obtaining the pretrained model using ResNet-18 as surrogate model with different pruning ratios.
by training the small surrogate model (ResNet-18) for DP is insignificant, compared to the time saved during the pretraining phase of a larger source model (ResNet-101) on the pruned dataset for transfer learning. Lastly, pretraining on the subset found by LM can guide the source model towards a flatter region in the downstream loss landscape, (see results in Fig. A4). The source model trained on the LM-pruned dataset achieves a higher flatness score than baselines, which aligns with the understanding of transfer learning in [78].
FM-based DP framework for self-supervised pretraining. The requirement for labeled source data in LM may pose limitations on the application of DP methods, particularly in the context of self-supervised learning (SSL)-based pertaining. To address this limitation, we introduce a new method called FM (Feature Mapping). Unlike LM, FM determines the DP scheme using only the feature extractor network  $f_{S}$ , the unlabeled source dataset  $\mathcal{D}_S$ , and the target dataset  $\mathcal{D}_{\mathcal{T}}$ . This allows us to overcome the dependence on labeled source data, making FM applicable in SSL scenarios. The inspiration for FM is derived from the deep clustering technique [103-105] operating in the representation space, which can generate pseudo source labels using cluster indices provided by e.g.,  $K$ -means. With the assistance of deep clustering, we can represent the unlabeled source dataset  $\mathcal{D}_S$  as  $\mathcal{D}_S = \{\mathcal{C}_S^{(1)},\mathcal{C}_S^{(2)},\dots,\mathcal{C}_S^{(K)}\}$ , where  $\mathcal{C}_S^{(k)}$  denotes the set of source data samples within cluster  $k$ . Building upon the similar spirit as LM, we propose ranking the importance of pseudo source classes for DP by evaluating the source feature extractor's responsiveness to target data samples. To achieve this objective, we quantify the responsiveness of  $f_{S}$  for a target sample  $\mathbf{t}$  as follows:
$$
r (\mathbf {t}) = \underset {k \in [ K ]} {\arg \min } \left\| f _ {\mathcal {S}} (\mathbf {t}) - \mathbb {E} _ {\mathbf {x} \in \mathcal {C} _ {\mathcal {S}} ^ {(k)} [ f _ {\mathcal {S}} (\mathbf {x}) ]} \right\| _ {2}, \tag {1}
$$
where  $\mathbb{E}_{\mathbf{x} \in \mathcal{C}_S^{(k)}}[f_S(\mathbf{x})]$  is the centroid of source data within the cluster  $k$ , and  $r(\mathbf{t})$  is the nearest pseudo label as the responsiveness of  $f_S$  against  $\mathbf{t}$ . The FM score then integrates (1) with (LM):
$$
\mathbf {s} _ {\mathrm {F M}} \left(\mathcal {C} _ {S} ^ {(i)}\right) = \sum_ {j = 1} ^ {n} \mathbb {1} (r (\mathbf {t} _ {j}) = i), \text {f o r} i = 1, 2, \dots , K, \tag {FM}
$$
where different from (LM),  $K$  represents the number of pseudo source classes produced by deep clustering, and  $r(\mathbf{t}_j)$  corresponds to the source feature extractor's prediction on  $\mathbf{t}_j$ . It is important to note that the value of  $K$  is a free parameter of the deep clustering process. Our empirical study in Fig. A2 shows that FM is quite robust to the choice of  $K$  without sacrificing the benefit of DP for
transfer learning compared to using the unpruned source dataset. Lastly, it is worth mentioning that FM can also be applied in the context of supervised pretraining by specifying data labels as clusters.
In this section, we provide a comprehensive set of experiments and analyses to showcase the effectiveness of our proposed methods (LM and FM) in diverse transfer learning scenarios.
Datasets and models. In line with existing transfer learning benchmarks [41, 106], we utilize ImageNet-1K [55] for pretraining and 8 datasets as downstream tasks. These datasets include DTD [107], Flowers102 [102], UCF101 [108], Food101 [109], SUN397 [110], OxfordPets [56], StanfordCars [57], and CIFAR10 [111]. Please refer to Tab. A1 for more details about the datasets. As discussed in Sec. 4, we utilize ResNet-18 (RN-18) [54] as the surrogate source model for pruning source classes. This method significantly reduces the computational cost associated with DP, making the process more efficient. Subsequently, a range of larger models, e.g., ResNet-101 (RN-101) and ViT-B/16 [112], are trained on the (pruned) ImageNet and then finetuned on downstream tasks.
Baselines, training, and evaluation. By examining the performance of the existing 8 DP baselines as shown in Fig. 2, our experiments focus on two of the most effective methods: ① GRAND [26] and ② MODERATE [3], together with ③ RANDOM (the random pruning strategy). In Fig. A1, we show more results of the rest DP baselines. Unfortunately, we are unable to include the existing data attribution method [41] as our baseline, as it does not release its pruning results, and we are unable to repeat its experiments due to the need for intensive computations. Unless specified otherwise, we focus on the supervised pretraining setting in the experiments. For self-supervised pretraining, we follow the implementation of MoCov2 [45]. The finetuning strategies employed include LP (linear probing), which finetunes the classification head with fixed feature extractor, and FF (full finetuning), which finetunes the entire source model. For FM-based DP method, we utilize K-means clustering to group the ImageNet training data points into  $K = 2000$  clusters for the computation of (1).
In accordance with the terminology convention in model pruning [87, 113-115], we refer the term 'winning subsets' to the obtained source subsets that do not compromise downstream performance. Among these winning subsets, we identify the one with the highest pruning ratio as the 'best winning subset'. We then evaluate the performance of DP methods from the following aspects: the downstream performance of the model pretrained on the pruned source dataset obtained by various DP methods, and the pruning ratio of the best winning subset achieved by DP methods, accompanied by the corresponding time saved in the pretraining phase.
LM/FM improves transfer learning accuracy by identifying 'winning subsets'. We first showcase the significant improvements achieved by our proposed DP methods (LM and FM) compared to baselines. Our methods successfully identify winning subsets of the ImageNet, yielding transfer accuracy on par or even better than scenarios without pruning.
Fig. 4 presents the downstream accuracy of transfer learning vs. different pruning ratios. Here DP is performed using the surrogate model (RN-18) on ImageNet for 8 downstream tasks. The source model (RN-101) is then trained on the pruned ImageNet and the transfer learning accuracy is assessed through LP (linear probing) and FF (full finetuning). We also present the downstream performance without pruning the source dataset (No Prune) as a reference for winning subsets. As we can see, both LM and FM significantly outperform the baselines by a substantial margin. Notably, LM and FM consistently identify winning subsets with significantly larger pruning ratios in all settings. This highlights the effectiveness of our proposed methods in achieving substantial dataset pruning without hurting downstream performance. Furthermore, we observe that the downstream performance of LM and FM initially improves and then declines as the pruning ratio increases. This is not surprising, since the initial increase in performance corresponds to the scenario where harmful source classes are removed, consistent with [41]. When the source dataset continues to shrink, the performance inevitably decreases, as a natural consequence of reducing the size of the source dataset. Moreover, in some datasets of small sizes (e.g., OxfordPets), LP achieves performance comparable to FF. This
Figure 4: Source dataset pruning trajectory given by downstream testing accuracy  $(\%)$  vs. source dataset pruning ratio  $(\%)$  in the supervised pretraining setting. Here the source model RN-101 is trained on each full/pruned source dataset (ImageNet) and finetuned on different downstream tasks through LP and FF. The downstream performance without pruning (No Prune) is marked with the black dashed line. Results are averaged over three independent trials (see exact numbers and variances in Tab. A2).
is attributed to the fact that the performance gap between LP and FF using the large-scale RN-101 model on small-scale datasets tends to diminish.
Table 1: The pruning ratio of the best winning subsets obtained by each DP method for different downstream tasks. The other experiment setup is consistent with Fig. 4. The achieved largest pruning ratio is highlighted in bold in each column. And N/A indicates the scenario where no winning subsets are found.
Dataset	OxfordPeds	SUN397	Food101	DTD	StanfordCars	UCF101	CIFAR10	Flowers102
Finetune method	LP	FF	LP	FF	LP	FF	LP	FF	LP	FF	LP	FF	LP	FF	LP	FF
RANDOM	20%	20%	15%	N/A	N/A	20%	10%	20%	N/A	10%	20%	10%	N/A	15%	40%	35%
MODERATE	15%	20%	15%	20%	N/A	20%	15%	30%	N/A	5%	5%	10%	N/A	20%	40%	35%
GRAND	25%	20%	10%	20%	5%	N/A	15%	20%	N/A	5%	5%	N/A	40%	15%	45%	40%
LM (ours)	80%	80%	65%	70%	65%	70%	55%	50%	65%	75%	45%	40%	40%	40%	55%	60%
FM (ours)	85%	75%	65%	60%	55%	75%	55%	50%	60%	70%	45%	40%	45%	45%	55%	55%
Tab. 1 provides a summary of the pruning ratios achieved by the best winning subsets identified using different DP methods for all 8 downstream datasets. Both LM and FM methods successfully remove more than  $45\%$  of the source classes without downstream performance drop. In contrast, all the baselines experience significant performance degradation when the pruning ratio exceeds  $40\%$ .
Table 2: The downstream performance with different source data pruning ratios in the SSL pretraining setting. A randomly initialized RN-101 is self-supervised pretrained using MoCo v2 on each full/pruned source dataset and finetuned on the downstream task through LP. The best result in each pruning ratio is marked in bold and the performance surpassing the unpruned setting (pruning ratio  $0\%$  ) is highlighted in cyan .
Dataset	OxfordPets	SUN397	Flowers102
Pruning Ratio	0%	50%	60%	70%	80%	0%	50%	60%	70%	80%	0%	50%	60%	70%	80%
RANDOM	69.26	62.32	61.27	59.09	53.75	47.36	45.63	45.08	43.54	39.81	85.17	82.23	82.60	81.03	80.02
MODERATE	63.37	62.45	63.31	57.42	45.73	45.14	44.23	40.82	82.45	81.45	81.69	81.32
GRAND	64.42	63.34	61.14	56.42	45.72	45.58	45.24	41.72	82.85	82.44	82.14	81.73
FM (ours)	69.92	69.99	70.29	70.21	48.46	48.58	47.90	46.00	85.22	85.42	84.37	84.61
Tab. 2 highlights the effectiveness of FM-based DP in the self-supervised pretraining setup for three representative downstream tasks. As we can see, the transfer learning accuracy achieved by using FM consistently outperforms baselines in the self-supervised pretraining paradigm. FM can identify winning subsets for transfer learning even in the challenging regime of large pruning ratios, ranging from  $50\%$  to  $80\%$ . For instance, in the case of SUN397, FM-based winning subsets achieve a pruning ratio up to  $70\%$ , and for Flowers102 the maximum pruning ratio is  $60\%$ . These pruning merits align with the findings for FM in the supervised pretraining setup, as illustrated in Tab. 1.
DP enhances the efficiency of source pretraining. Tab. 3 displays the computation time required to obtain the pretrained source model using LM at different pruning ratios. The reported time consumption includes the entire pipeline, encompassing surrogate model training (RN-18), DP process, and source model training (RN
Table 3: Time consumption of LM/FM in Fig.4 to obtain the pretrained model. The reported time consumption covers surrogate model (RN-18) training, LM/FM dataset pruning, and source model pretraining (RN-101).
Pruning Ratio	0%	20%	40%	60%	80%
Time Consumption (h)	5.4	4.6(15%↓)	3.5(35%↓)	2.4(56%↓)	1.3(76%↓)
101) on the pruned ImageNet dataset. The runtime cost of the conventional transfer learning on the full ImageNet dataset for RN-101 is also listed as a reference. As we can see, DP enjoys high efficiency merit of source training. Taking the 5.4 hours required for source training on the full ImageNet dataset as a reference, LM-enabled  $20\%$  pruned ImageNet achieves a  $15\%$  reduction in training time. Moreover, the efficiency advantage increases to  $76\%$  when the pruning ratio reaches  $80\%$  and these computational benefits do not sacrifice transfer learning accuracy at all.
Next, we compare the efficiency of our methods with the state-of-the-art dataset pruning baselines in Tab. 4, including GRAND, MODERATE, and the brute-force method proposed in [41]. We showcase the efficiency comparison with a pruning ratio of  $60\%$ . As we can see, our method
Table 4: Time consumption comparison with a pruning ratio of  $60\%$  of different dataset pruning methods. Other settings follow Fig. 4.
Method	MODERATE	GRAND	BRUTE-FORCE [41]	ours
Time Consumption (h)	7.6	18.4	>500	2.4
(even using a surrogate model) achieves a substantial computation efficiency improvement over other methods. In particular, [41] involves training thousands of source models. Thus, its computational cost is typically unaffordable in experiments.
DP enables efficient adversarial pretraining. Tab. 5 showcases the improvement in transfer learning accuracy and efficiency achieved by our proposed LP-based DP method when incorporating adversarial training (AT) [116] on either the full or pruned ImageNet dataset. This transfer learning setup is motivated by the findings in [58], showing that enhancing the robustness of the source model against adversarial attacks through AT can improve transfer learning accuracy in both LP and FF-based finetuning scenarios. We then report downstream accuracies
using LP and FF on two specific downstream datasets: SUN397 and DTD, which are intentionally chosen due to the large room for improvement in transfer learning accuracy, as shown in Fig. 4. We also determine the pruning ratios for LM by selecting those that led to the best winning subsets. Our experimental results demonstrate that employing AT on both the unpruned and pruned source datasets can improve transfer learning accuracy. Specifically, we refer to AT on the original unpruned ImageNet dataset as DENSE-AT, and AT on the LM-pruned ImageNet dataset as LM-AT. One notable advantage of integrating LM into AT is the significant improvement in computation efficiency. A key highlight of this approach is that LM-AT achieves a similar computation time as the standard source training on ImageNet (DENSE), while exhibiting almost no accuracy drop compared to DENSE-AT. This observation demonstrates the potential to accelerate AT through DP.
Table 5: Downstream performance of models pretrained on full/pruned source dataset (DENSE/LM) w/wo adversarial pretraining (Adv). For DENSE-ADV and LM-ADV, 3-step adversarial training [116] is used for pretraining. Pruning ratio, downstream test accuracy (Acc.), and time consumptions for obtaining pretrained models are reported.
Method	SUN397	DTD
Pruning Ratio	Acc.(%)	Time (h)	Pruning Ratio	Acc.(%)	Time (h)
LP	FF	LP	FF
DENSE	N/A	51.45	54.21	5.4	N/A	65.91	67.21	5.4
DENSE-ADV	N/A	52.97	55.67	13.7	N/A	67.23	68.92	13.7
LM	70	50.95	54.28	1.9	50	66.25	67.22	2.9
LM-ADV	70	52.07	55.49	4.2	50	67.02	68.54	6.7
Robustness against the surrogate model size. To explore the sensitivity of our proposed method to the surrogate model's size and accuracy, we show the transfer learning performance on the task OxfordPets against different surrogate model sizes in Fig. 5. It is evident that even though the performance of the surrogate model on the source dataset (ImageNet) decreases, the downstream performance of RN-101 pretrained on the LM-based pruned source subsets remains relatively stable. Further, Tab. A6 compares the class indices selected by the corresponding surrogate models. Interestingly, the most relevant source class selections exhibit a high level of agreement across surrogate models of differing sizes.
LM shows robust behavior with different surrogate models, even if they yield different performance on the source dataset due to their different model capacities. For example, RN-32s only achieves a test accuracy of  $40.77\%$  on the source dataset, but it manages to achieve the same maximum pruning ratio of the winning subsets as RN-18. This underscores that LM can accommodate a small surrogate model of  $1\%$  the size of the pretrained model (Rn-32s vs. RN-101). Moreover, it reveals that the surrogate model does not need to attain exceptional performance in the source domain for dataset pruning in trans
(a) ResNets family
(b) VGG family
Figure 5: Source dataset pruning trajectory given the downstream task OxfordPets using different surrogate models. Other settings follow Fig. 4.
Table 6: Downstream performance of different DP methods using ViT-B/16 as the source model and FF as the finetuning method. Other settings are consistent with Fig.4. The best performance at each pruning ratio is marked in bold, and performance exceeding the unpruned setting (0% pruning ratio) is highlighted in cyan .
Dataset	OxfordPets	SUN397	Flowers102
Pruning Ratio	0%	20%	40%	60%	80%	0%	20%	40%	60%	80%	0%	20%	40%	60%	80%
RANDOM	85.75	85.63	83.72	75.94	69.83	43.97	43.74	41.18	36.72	29.41	90.95	90.25	89.38	87.47	84.44
MODERATE	85.39	83.97	82.39	78.32	43.97	41.82	38.89	31.15	90.86	89.77	88.34	86.69
GRAND	85.77	83.88	82.35	79.12	43.32	40.93	38.52	31.00	90.75	89.89	88.54	85.39
LM	86.15	85.72	84.77	80.64	44.24	42.79	39.59	32.35	91.19	89.97	89.07	87.23
DP for ViT. In Tab. 6, we present the consistent transfer learning improvement achieved by LM for Vision Transformer (ViT) pretraining. Specifically, we utilize ViT-B/16 as the source model, RN-18 as the surrogate model for DP, and use FF on the source model for the downstream tasks studied in Tab. 2. As we can see, LM yields a better transfer learning accuracy than baselines at different pruning ratios, consistent with the previous results on RN-101. This observation highlights the model-agnostic advantage of our proposed method in transfer learning. We also observe the pruning ratio of the best winning subset achieved for ViT is smaller compared to ResNet. This discrepancy is possibly due to the higher data requirements of ViT during pretraining [112].
Additional results. We conduct ablation studies on the cluster number for FM in Fig. A2 and the results of selecting data proposed by LM in reverse order in Fig. A3. We show FM can be applied to more advanced SSL framework in Tab. A3. We also investigate the performance of our proposed method in the multi-task setting in Fig. A5, and the scenarios with data biases in Tab. A4. We further examine our method on the few-shot benchmark in Tab. A5. We also present feature distribution analysis in Fig. A6 and image examples for the top classes selected by LM/FM in Fig. A7.
Limitations. Despite the remarkable empirical results above, we admit LM/FM mainly caters to exploring the source data influence in a specific downstream task, and the largest pruning ratio is reduced as more downstream tasks are considered (see Fig. A5), which we consider as a limitation. Meanwhile, we acknowledge that the promising performance achieved by our method still lacks rigorous theoretical analysis, despite the feature distribution and model flatness analysis, which we believe will be a challenging but exciting future work on DP for transfer learning.
In this paper, we first formally define and investigate the problem of DP (dataset pruning) for transfer learning. Recognizing the ineffectiveness of conventional DP methods in the transfer learning setting, we proposed two novel and efficient DP techniques, label mapping and feature mapping. Extensive experiments demonstrate that both methods can effectively prune a large ratio of source dataset and substantially reduce the pretraining computational costs without sacrificing the downstream performance in various transfer learning settings.
The work of Y. Zhang, Y. Zhang, A. Chen, J. Jia, J. Liu, S. Chang, and S. Liu was supported by the Cisco Research Award and partially supported by the NSF Grant IIS-2207052. The work of M. Hong was supported by NSF grants CIF-1910385 and EPCN-2311007.
A Experiment Settings
A.1 Training Settings
General setups for DP (dataset pruning). Regardless of the choice of the downstream tasks and DP methods, each DP process will go through a three-stage paradigm, namely ① source dataset pruning, ② pretraining based on the pruned source dataset, and ③ finetuning on the target dataset using either LP (linear probing) or FF (full finetuning). Regarding baseline methods for source dataset pruning, we strictly follow the baselines' settings provided in the literature. As stated before, we utilize a small surrogate model (ResNet-18) pretrained on the full ImageNet to conduct DP. The proportion of pruned class numbers to the total source class numbers determines pruning ratios for our methods: LM (label mapping) and FM (feature mapping). To ensure a fair comparison with other non-class-wise baselines (RANDOM, GRAND, MODERATE), the same pruning ratio is applied to the total number of training data points.
Pretraining setups. We keep our pretraining procedure consistent for all models and strictly follow the settings released by the existing work [41]. All the models are trained from scratch using Stochastic Gradient Descent (SGD) to minimize the standard cross-entropy loss in a multi-class classification setting. We use a batch size of 1024, momentum of 0.9, and a weight decay of  $5 \times 10^{-4}$ . The training utilizes a cyclic learning rate schedule, which begins with an initial learning rate of 0.5 and peaks at the second epoch. During training, we incorporate data augmentations such as random resized cropping and random horizontal flipping.
Downstream task finetuning settings. Our approach to finetuning the pretrained model on downstream tasks involves using LP and FF. Details of the downstream datasets and the training configurations are presented in Tab. A1, following [98]. For LP, we employ the Adam optimizer, a multi-step decaying scheduler, and an initial learning rate of 0.1 across 50 total training epochs. As for FF, we utilize the Adam optimizer over 200 epochs with a cosine-annealing scheduler, an initial learning rate of 0.01, and a weight decay of  $5 \times 10^{-4}$ . All finetuning experiments employ a batch size of 256 and standard data augmentations, such as random resized cropping and horizontal flipping.
Training settings for SSL (self-supervised learning) and ViT. Our SSL training settings follow the configurations provided by MoCov2. Details of the pretraining and finetuning stages can be accessed at https://github.com/facebookresearch/moco. For the training of ViTs, we rely on the setting released in the original ViT paper [112] (see ViT/B in Table 3).
Table A1: Dataset attributes and training configurations of 8 downstream image classification datasets considered in this work.
Dataset	Train Size	Test Size	Class Number	Batch Size	Rescaled Resolution
Flowers102	4093	2463	102	256	224×224
DTD	2820	1692	47	256	224×224
UCF101	7639	3783	101	256	224×224
Food101	50500	30300	101	256	224×224
OxfordPets	2944	3669	37	256	224×224
StanfordCars	6509	8041	196	256	224×224
SUN397	15888	19850	397	256	224×224
CIFAR10	50000	10000	10	256	160×160
Expanded performance evaluation of in-domain DP methods across all downstream datasets. Fig. A1 expands on the performance comparisons in Fig. 2, providing a more thorough evaluation of various in-domain DP methods on all eight downstream datasets. The trends observed are consistent with those in Fig. 2: Random pruning shows a strong baseline method in DP for transfer learning compared to other state-of-the-art DP methods designed for non-transfer learning. This observation prompts us to explore more effective dataset pruning strategies for transfer learning. MODERATE and
GRAND are also demonstrating strong baselines, motivating us to choose them as the default DP baselines in Section 5.