SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yihua Zhang $^{1,*}$  Yimeng Zhang $^{1,*}$  Aochuan Chen $^{1,*}$  Jinghan Jia $^{1}$  Jiancheng Liu $^{1}$
Gaowen Liu2
Shiyu Chang
<sup>1</sup> Michigan State University, <sup>2</sup> Cisco Research,
<sup>3</sup> University of Minnesota, Twin City, <sup>4</sup> UC Santa Barbara
Equal Contribution
Massive data is often considered essential for deep learning applications, but it also incurs significant computational and infrastructural costs. Therefore, dataset pruning (DP) has emerged as an effective way to improve data efficiency by identifying and removing redundant training samples without sacrificing performance. In this work, we aim to address the problem of DP for transfer learning, i.e., how to prune a source dataset for improved pretraining efficiency and lossless finetuning accuracy on downstream target tasks. To our best knowledge, the problem of DP for transfer learning remains open, as previous studies have primarily addressed DP and transfer learning as separate problems. By contrast, we establish a unified viewpoint to integrate DP with transfer learning and find that existing DP methods are not suitable for the transfer learning paradigm. We then propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings respectively, by revisiting the DP problem through the lens of source-target domain mapping. Furthermore, we demonstrate the effectiveness of our approach on numerous transfer learning tasks. We show that source data classes can be pruned by up to  $40\% \sim 80\%$  without sacrificing downstream performance, resulting in a significant  $2 \sim 5\times$  speed-up during the pretraining stage. Besides, our proposal exhibits broad applicability and can improve other computationally intensive transfer learning techniques, such as adversarial pretraining. Codes are available at https://github.com/OPTML-Group/DP4TL.
The abundance of the training data has long been regarded as the key driver of the contemporary machine learning (ML) algorithms [1-3]. However, the untrimmed, ever-growing training dataset could not only introduce training biases that compromise the model performance [4-6], but also poses an almost insurmountable obstacle to high training efficiency [1, 2]. Therefore, understanding the impact of the training data and selecting the most critical samples has emerged as an important goal, collectively referred to as the problem of dataset pruning (DP).
Although the feasibility and the promise of DP have been unveiled in numerous applications, such as noisy data cleansing [7-9], continue learning [10-13] and active learning [14], greater emphasis is placed on the in-domain training setting, i.e., the pruned training set share the similar distribution as the evaluation set. Examples of methods to condense the training dataset with lossless generalization (on the in-distribution testing dataset) include data influence functions [10, 15-23], model training dynamics [11-14, 24-28], and coreset selection [29-36]. In contrast, our work investigates the
Figure 1: Left: An illustration of the proposed dataset pruning methods (LM and FM) and their performance overview. Large scale source dataset is pruned by LM and FM through a small surrogate model (ResNet-18). Large foundation models can achieve up to  $5 \times$  speed-up on pretraining without no downstream performance drop. Right: Downstream performance overview at different pruning ratios when ResNet-101 [54] is used as the source model (trained on the pruned source dataset) with ImageNet [55] as the source dataset transferring to OxfordPets [56] and StanfordCars [57]. The feature extractor is fixed during finetuning.
problem of DP in the transfer learning paradigm [37-39], which has emerged as a popular way to leverage the knowledge of a foundation model learned on a source dataset (referring to 'pretraining' stage) to further enhance the performance on a cross-domain target dataset (referring to 'finetuning' stage). Recent evidence [40-42] has shown that some source data points could make a harmful influence in the downstream performance. In particular, the previous study [41] showed that removing specific source data classes can improve the transfer learning accuracy of a pretrained model.
Yet, efficiently identifying those harmful source data classes for improved transfer learning is a highly non-trivial task. Firstly, the unique pretrain-finetune paradigm complicates the analysis of the influence of source data on downstream performance, making it an indirect and very challenging process. Consequently, existing gradient and data influence-based in-domain DP methods [10, 19-23] cannot be naively adapted to the transfer setting. Secondly, transfer learning encompasses a wide range of source training methods other than supervised learning, such as self-supervised learning (SSL) [43-52]. Therefore, a generic DP framework is desired given various pretraining scenarios. Thirdly, from an efficiency standpoint, the design of efficient DP algorithms is non-trivial. Even in the non-transfer learning paradigm, a majority of current in-domain DP methods [11, 25, 26] introduce substantial computational overheads. For instance, influence function-based DP methods [10, 15-22] necessitate the calculation of the inverse Hessian matrix of model parameters, which is a highly demanding and computationally intensive process. Moreover, training dynamics-based methods [11-14, 24-28, 53], such as GraNd-Score [26] and Forgetting-Score [11], necessitate model training on the entire dataset multiple times. As a result, the development of an efficient and effective DP framework tailored for transfer learning remains a significant challenge in the field.
The most relevant work to ours is [41], which proposed a brute-force method to evaluate the influence of every source class in the downstream task following a leave-one-out principle. While this method effectively selects the most influential source classes, it is prohibitively slow, as the data selection process demands significantly higher computational resources than performing transfer learning itself. This brings us to the central question addressed in this paper:
(Q) How can we extend DP to transfer learning with minimal computation overhead, broad applicability, and improved target performance?
To address (Q), we formally define the task of DP (dataset pruning) for transfer learning. We start by uncovering the limitations of conventional in-domain DP methods when applied to transfer learning tasks, and establish an intrinsic connection between source dataset pruning for transfer learning and source-target domain mapping. In the supervised pretraining paradigm, we propose an effective and scalable label mapping (LM)-based framework capable of pinpointing the most beneficial source
data labels for a cross-domain target dataset. Furthermore, we extend the idea of LM to feature mapping (FM) for the SSL pretraining protocol, where data labels are unavailable. To achieve a greater practical impact, we demonstrate that our proposed methods (LM and FM) can facilitate effective DP over a small, simple surrogate source model, and further translate the positive impact of the pruned source dataset to other larger, more complex source models. We provide a schematic overview and result highlights in Fig. 1. Our contributions can be summarized as follows:
1 We connect the concept of DP (dataset pruning) to transfer learning for the first time and formalize the problem of DP for transfer learning.
We develop a highly efficient and effective framework for DP in transfer learning, leveraging the source-target domain mapping with customized LM (label mapping) and FM (feature mapping) for both supervised and SSL (self-supervised learning) settings. Our approach is principled and achieves significant speedups compared to the state-of-the-art methods.
We empirically show the effectiveness of our proposals (LP and FP) on 8 downstream tasks. We find that the source dataset (ImageNet) can be pruned up to  $40\% \sim 80\%$  without sacrificing the downstream performance, together with a  $2\times \sim 5\times$  speed-up in the pretraining stage. Our proposal also unlocks a door to prune a dataset using a simple surrogate source model (e.g., ResNet-18) and then reuse the pruned dataset to improve transfer learning on a larger source model (e.g., ResNet-101).
Lastly, we show that our proposed DP framework can benefit other computationally-intensive transfer learning techniques. For example, DP for adversarial pretraining [58] leads to a  $1\% \sim 2\%$  improvement in downstream performance with time consumption similar to standard pretraining.
Dataset pruning. DP (dataset pruning) is an emerging technique to improve the data efficiency of model training by selecting the most representative training samples or removing the less influential ones [10-28]. Thus, the problem of coreset selection [29-36] can also be considered a form of DP. Prior studies on DP range from clustering-based methods [59-61] to the more recent score-based pruning methods [10-28]. In the latter approach, an importance score is assigned to each training data point to quantify its influence on a particular permanence metric during model learning. Specifically, score-based DP methods can be broadly categorized into two main groups: influence function-based approaches [10, 15-23] and training dynamics-based approaches [11-14, 24-28, 53]. The first category measures data influence by examining the effect of data removal on the learning algorithm used during training [15-18] and the model's prediction [10, 19-23]. However, influence function-based approaches typically require high computational costs due to the need for high-order derivatives and complex optimization methods, such as bi-level optimization, as used in [22]. Although an approximate influence score can be obtained efficiently, it may result in a large estimation error [62]. In the second category, training dynamics-based approaches find statistical indicators for pruned data from the training trajectory. Examples of DP metrics include data loss/errror [25-27], prediction confidence [24], model gradient norm [26], forgetting event [11], and compactness [12-14, 28]. However, these methods typically require repeated training to ensure the representativeness of the collected statistics [11, 25, 26]. In addition to improving data efficiency, DP has been applied in a variety of contexts, such as noisy label cleansing [7-9], continue learning [10-13, 63], active learning [14], and reducing annotation cost of a finetuner [64].
Data valuation & attribution. Data valuation [19, 41, 65-70] and attribution [71-73] are research streams related to dataset pruning that aim to quantify the influence of training data points on model's performance. Unlike dataset pruning, these approaches do not focus primarily on training efficiency but instead aim to enhance interpretability [74], adversarial robustness [68, 70], generative model design [75], and data acquisition quality [72]. Representative methods include Shapley value-based methods [65, 66], datamodels [73], sampling-based methods [65, 68], and proxy-based methods [67, 72]. Recently, Kim et al. [35] proposed to select data samples from a large public dataset (Open-Set) for self-supervised learning given a specific target dataset through distribution mismatch. However, it fails to make a general framework in both supervised and self-supervised scenarios, leaving the former under-explored. The most relevant work to ours is [41], which leverages a leave-one-out analysis to quantify the influence of source data on downstream tasks in the context of transfer learning. With this inspiration, we propose to connect DP to transfer learning in this work.
Recent advancements in transfer learning. Transfer learning has been a prominent area over the past decade [37-39]. Significant strides have been made in understanding, analyzing, and improving various aspects of this technique [58, 76-84]. Recent studies [77, 78] have rigorously analyzed the relationship between source model performance and downstream task effectiveness, arguing that a narrow focus on minimizing source training loss may not lead to improved transfer learning results. To improve transfer learning, adversarial training has been shown to benefit the transferability of source pretraining on target downstream tasks [58, 76]. There also exist studies to identify and understand the failure cases in transfer learning [79-84]. Other research [77, 78] has examined model transferability from the perspective of model sharpness and argues that a good pretrained model should be situated in a flat basin in the downstream loss landscape. Last but not the least, transfer learning has progressed in several other directions, such as self-supervised learning [43-52], model weight pruning [85-89], and visual prompting [90-99].
In this section, we introduce some essential preliminaries on transfer learning and DP (dataset pruning), and elucidate the design challenge of DP for transfer learning.
Preliminaries on transfer learning and connection to DP. Let  $g \circ f$  denote a deep neural network that consists of a feature extractor  $f$  and a classification head  $g$ , where  $\circ$  denotes the function composition. Given source and target datasets  $\mathcal{D}_S$  and  $\mathcal{D}_{\mathcal{T}}$ , we study transfer learning in the "pretrainfinetune" paradigm. The primary goal of pretraining is to obtain a high-quality feature extractor  $f_{S} : \mathcal{X} \to \mathcal{Z}$ , which draws a mapping from the input space  $(\mathcal{X})$  to the deep representation space  $(\mathcal{Z})$  in a data-rich source domain  $(\mathcal{D}_S)$ . Popular pertaining recipes include supervised learning (SL) [84, 100] and self-supervised learning (SSL) [43-46] depending on whether the source labels  $(y_S)$  are available or not in  $\mathcal{D}_S$ . In the finetuning stage, the pretrained model is further trained on a specific downstream task under the target dataset  $\mathcal{D}_{\mathcal{T}}$ . Transfer learning expects improved downstream performance over training on  $\mathcal{D}_{\mathcal{T}}$  from scratch. In this work, we consider two finetuning protocols, linear probe (LP) and full-finetune (FF), with  $\mathcal{D}_{\mathcal{T}}$  being a labeled dataset. LP finetunes the linear classification head  $g$  with a fixed feature extractor  $f_{S}$ , acquired from pretraining. In contrast, FF finetunes the entire model  $g \circ f$  from the initialization  $f_{S}$ . FF typically yields a better transfer learning accuracy than LP, but the former takes a higher computation cost.
Our motivation for connecting transfer learning with DP comes from a recent data-based perspective on transfer learning [41]. The study shows that removing certain source data classes from  $\mathcal{D}_S$  can potentially improve the accuracy of a finetuned model on  $\mathcal{D}_{\mathcal{T}}$ . However, the task of evaluating the transfer effects of source data and removing their influence from a pre-trained source model has not been addressed efficiently. The approach developed in [41] involves a leave-one-out analysis to estimate the influence of a source class  $c$  on a target example  $t$ , which is computed as the prediction discrepancy of the finetuned source model at  $t$  when the class  $c$  is either included or excluded from  $\mathcal{D}_S$ . During this process, one must train multiple source models (over 7000 models on ImageNet in [41]) from scratch over different subsets of  $\mathcal{D}_S$  for a given target task. This approach becomes computationally unaffordable when dealing with large source datasets like ImageNet given limited computing resources. To address this challenge, we propose a DP perspective on transfer learning.
Problem of interest: DP for transfer learning. Next, we introduce the background of DP and the problem we focus on. Let  $\mathcal{D} = \{\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_N\}$  denote a dataset consisting of  $n$  samples, where each sample  $z_i$  is represented as a pair  $(\mathbf{x}_i, y_i)$ , with  $\mathbf{x}_i$  denoting the input feature vector and  $y_i$  denoting the corresponding label. DP aims to generate a pruned dataset  $\hat{\mathcal{D}} = \{\hat{\mathbf{z}}_1, \hat{\mathbf{z}}_2, \dots, \hat{\mathbf{z}}_M\} \subset \mathcal{D}$  with  $M < N$ , which can reduce the training cost without a significant decrease in model generalization performance when trained on  $\hat{\mathcal{D}}$ . In the context of [41], instead of individual source data sample  $\{\hat{\mathbf{z}}_i\}$ , the entire source classes are evaluated and selected in terms of transfer influences. Based on the above, we define the problem of our interest below.
(DP for transfer learning) How to prune source data classes to obtain  $\hat{\mathcal{D}}_S$  (a subset of  $\mathcal{D}_S$ ), with lossless or improved transfer learning accuracy of the source model ( $f_S$ ) on a target task  $\mathcal{D}_{\mathcal{T}}$ ?
DP for transfer learning has two key distinctions (1-2) from the vanilla DP setup. First (1), DP must be performed in the source domain  $(\mathcal{D}_S)$ , while its effectiveness is evaluated based on the target
domain  $(\mathcal{D}_{\mathcal{T}})$ . This 'cross-domain' challenge makes the design of efficient and effective DP highly non-trivial. For example, prior work [41] utilizes a computationally-intensive leave-one-out analysis. Classical influence function-based methods [10, 19-23, 101], which trace the eventual model's prediction through the learning algorithm and back to its training data, are also computationally infeasible due to the complex bi-level optimizer and the calculation of high-order derivatives. Second (2), the pre-trained source model  $(f_{S})$  in today's transfer learning regime is typically of a large scale. This motivates us to develop a DP method that can be independent of the source model, while the pruned source dataset  $\hat{\mathcal{D}}_S$  remains effective in the original transfer learning setup. Given this challenge, we will design DP methods for transfer learning using a simple surrogate source model to avoid the computation on the large source model  $f_{S}$  (see Fig. 1 for an illustration).
Conventional DP methods lack effectiveness on transfer learning. Given the challenges (1-2) posed by DP for transfer learning, we further conduct a preliminary study to investigate the effectiveness of existing 8 DP methods, including SP [20], SSP [20], GRAND [26], EL2N [26], MODERATE [3], FORGET [53], INF-MAX [53], and INFSUM [53]. Our results show that these methods are unable to yield significant improvements over random source data prun
ing. Fig. 2 shows the transfer learning performance of the ResNet-101 [54] model trained on different pruned versions of the ImageNet dataset when LP-based finetuning is conducted on the downstream Flowers102 [102] and OxfordPets [56] datasets. As we can see, in transfer learning, random pruning is a solid baseline for various state-of-the-art DP methods, which have demonstrated superior performance to the former in the non-transfer learning regime. Therefore, it is crucial to develop an efficient and effective DP method specifically tailored for transfer learning.
Figure 2: Transfer learning accuracy of existing DP methods on ImageNet at different pruning ratios, where ResNet-101 is the source model, and linear probing (LP) is used for downstream finetuning on the target datasets Flowers102 (Left) and OxfordPets (Right).
4 Label/Feature Mapping-based DP for Transfer Learning
In this section, we first introduce a simple yet powerful DP method called label mapping (LM) by leveraging the class-discriminative capability of a supervised source model. We then extend the concept of LM to feature mapping (FM), suitable for self-supervised pretraining.
LM-based DP for supervised pretraining. Following the notations used in Sec. 3, we represent the model obtained through supervised learning on the source dataset  $(\mathcal{D}_S)$  as  $g_{S} \circ f_{S}$ . This model predicts a source label given an input example  $(\mathbf{x})$ . In DP for transfer learning, the source data classes serve as the variables to be pruned. Thus, we express  $\mathcal{D}_S$  as  $\mathcal{D}_S = \{\mathcal{C}_S^{(1)}, \dots, \mathcal{C}_S^{(N)}\}$  for  $N$  source classes,  $\mathcal{C}_S^{(i)}$  denotes the set of data points belonging to the source class  $i$ . Additionally, the pruner has access to the target dataset  $\mathcal{D}_{\mathcal{T}} = \{\mathbf{t}_1, \dots, \mathbf{t}_n\}$ , which consists of  $n$  target data points. Our objective is to utilize the information provided by  $g_{S} \circ f_{S}$ ,  $\mathcal{D}_S$ , and  $\mathcal{D}_{\mathcal{T}}$  to devise a computationally efficient DP criterion. Importantly, our criterion should not involve model training, distinguishing it from the non-scalable approach in [41]. An important observation by [78] in transfer learning is that transferred knowledge improves transferability. This insight is supported by the loss landscape analysis in transfer learning: The finetuned weights may remain within the flat basin of the pretrained weights for enhanced transfer learning performance.
We next propose to extract the 'transferred knowledge' by leveraging the class-discriminative capability of the source model  $g_{\mathcal{S}} \circ f_{\mathcal{S}}$  on the target data samples in  $\mathcal{D}_{\mathcal{T}}$ . Specifically, we focus on monitoring the responsiveness of the source label predictions made by  $g_{\mathcal{S}} \circ f_{\mathcal{S}}$  when using target samples  $\{\mathbf{t}_i \in \mathcal{D}_{\mathcal{T}}\}_{i=1}^n$  as input data. Here we resize these target samples to ensure their resolution alignment with source data. For the  $i$ th source data class  $\mathcal{C}_{\mathcal{S}}^{(i)}$ , we then define its pruning score below:
$$
\mathbf {s} _ {\mathrm {L M}} \left(\mathcal {C} _ {\mathcal {S}} ^ {(i)}\right) = \sum_ {j = 1} ^ {n} \mathbb {1} \left(g _ {\mathcal {S}} \circ f _ {\mathcal {S}} \left(\mathbf {t} _ {j}\right) = i\right), \text {f o r} i = 1, 2, \dots , N, \tag {LM}
$$
where  $\mathbb{1}(\cdot)$  represents an indicator function that evaluates to 1 when the condition  $\cdot$  is satisfied, and 0 otherwise. We refer to the aforementioned formula as LM (Label Mapping) since the condition  $g_{S}\circ f_{S}(\mathbf{t}_{j}) = i$  establishes a mapping between the predicted labels of the target samples  $\{\mathbf{t}_j\}_{j = 1}^n$  by the source model and the corresponding source labels  $i$ . The larger  $\mathsf{s}_{\mathrm{LM}}$  in (LM) signifies that a specific source class is more frequently utilized to interpret the target data. Consequently, this indicates a more tightly connected relationship between the source class and the transferred knowledge. Therefore, we prune (or select) source data classes with low (or high)  $\mathsf{s}_{\mathrm{LM}}$  values.
Although LM is simple, it offers several advantages. Firstly, it is highly efficient in computation. Given a pre-trained source model  $g_{S} \circ f_{S}$ , the calculation of LM only requires forward passes through the model. In cases where obtaining the pretrained model in the source domain is challenging, our
proposal supports an alternative approach: training a smaller and simpler surrogate model to carry out LM. This surrogate model can effectively replace the complex pretrained model  $g_{\mathcal{S}} \circ f_{\mathcal{S}}$  and facilitate the efficient execution of the pruning process. As we show in Fig. 3a, employing ResNet-18 [54] is sufficient to successfully prune the source dataset (ImageNet [55]). The resulting DP scheme remains effective to improve transfer learning utilizing other larger source models, such as ResNet-101 [54]. In addition, Fig. 3b shows that the computational overhead incurred
(a) Downstream performance.