Improved Fine-Tuning by Better Leveraging Pre-Training Data
Authors: Ziquan Liu
Yi Xu
Yuanhong Xu
Qi Qian
Hao Li
Xiangyang Ji
Antoni B. Chan
Rong Jin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Pre-Training
Fine-Tuning
Learning Theory
Self-Supervision

Abstract: As a dominant paradigm, fine-tuning a pre-trained model on the target data is widely used in many deep learning applications, especially for small data sets. However, recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy once the number of training samples is increased in some vision tasks. In this work, we revisit this phenomenon from the perspective of generalization analysis by using excess risk bound which is popular in learning theory. The result reveals that the excess risk bound may have a weak dependency on the pre-trained model. The observation inspires us to leverage pre-training data for fine-tuning, since this data is also available for fine-tuning. The generalization result of using pre-training data shows that the excess risk bound on a target task can be improved when the appropriate pre-training data is included in fine-tuning. With the theoretical motivation, we propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task. Extensive experimental results for image classification tasks on 8 benchmark data sets verify the effectiveness of the proposed data selection based fine-tuning pipeline. Our code is available at https://github.com/ziquanliu/NeurIPS2022_UOT_fine_tuning.

Improved Fine- Fine-Tuning by Better Leveraging Pre-Training Data
Ziquan Liu Yi Xu 263 Yuanhong Xu², Qi Qian Hao Li Xiangyang Ji', Antoni B. Chan Rong Jin
Department of Computer Science, City University of Hong Kong
2 School of Artificial Intelligence, Dalian University of Technology DAMO Academy, Alibaba Group
Department of Automation, Tsinghua University
ziquanliu2-com, cityu. edu. yxu@dlut edu. cn, {yuanhong xuyh, ri.qian, lihao Athao.1h)ae ah)aalibaba-ine com, xyji xyji@tsinghua edu. cn, abchan@cityu. edu. hk, rongjinemail@@maill com
Abstract
As dominant paradigm, fine-tuning pre-trained model the target data is widely used in many deep learning applications, especially for small data sets. However, recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy once the number of training samples is increased in some vision tasks. In this work, revisit this phenomenon from the perspective of generalization analysis by using excess risk bound which is popular in learning theory. The result reveals that the excess risk bound may have weak dependency on the pre-trained model. The observation inspires us leverage pre-training data for fine-tuming fine-tuning, since this data is also available for fine-tuning. The generalization result of using pre-training data shows that the excess risk bound on a target task can be improved when the appropriate pre-training data is included in fine-tuning. With the theoretical motivation, we propose a novel selection strategy select subset from pre-training data to help improve the generalization image classification tasks on 8 benchmark data sets verify the effectiveness the proposed data selection based fine-tuning pipeline. Our code is available https: / / thub. com http//iiuaanluuuiiiee IPs fine_ tttei_tunnngggg
the target task. Extensive experimental results for
Introduction
After the success on ImageNet [1], deep learning attracts much attention and improves the per- formance of various computer vision tasks significantly, e.g., object detection [2], semantic segmentation[3], However, result of expensive labeling, it is unlikely that we have sufficient labels for every application. Fortunately, given model pre-trained on large-scale data set like ImageNet, an effective model the target data set, which may only have hundreds of examples, can be learned by fine-tuning the pre-trained model. It is because many vision tasks are related [4] and model learned from ImageNet that consists of more than one million examples can contain diverse semantic information and provides better initialization than random initialization.
Despite the prevalence fine-tuning pre-trained models, its theoretical understanding unclear. On the one hand, when sufficient training data is available, some research [5] has shown that training
Work partially done during an internship DAMO Academy, Alibaba Group. indicates corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The proposed pre-training data reusing method, which is moti- vated by our generalization analysis of the effect of pre-training data on fine- tuning. A novel data selection method based on unbalanced optimal transport is proposed to reuse the appropriate pre- training data in fine-tuning, Averaged over classification data sets, the UOT- selection method improves the perfor- mance of vanilla fine- ine-tuning with large margin of 2.93% using a self-super pre-trained model.
Figure 1:
ImageNet
ImageNet Subset
Data Selection
Pre-train
Fine-tune
Model
Target Data
from scratch can achieve the same accuracy as initialization with ImageNet pre-trained models after additional training. As a warm-up of this work, we aim to understand this phenomenon from the theoretical side of generalization analysis [6]. Our analysis reveals that the final prediction precision may have weak dependency on the pre-trained model when the target training set is large enough, On the other hand, is not realistic to have abundant labeled data for every target task. So in many vision applications, training from scratch generally cannot match the performance of fine-tuning pre-trained model. However, our theoretical result tells us that when the pre- training data are too far from the target data, the domain gap will hurt the accuracy of target tasks.
These two observations lead to the following question: can we develop a new strategy of fine-tuning that achieves better generalization performance than the standard framework by reusing pre- pre-training data in fine-tuning to reduce the domain gap? This work will address this question with an affirmative answer. Inspired by the theoretical observation, we propose to leverage the pre-training data, which also available for fine-tuning, for target tasks. Concretely, we propose to reuse pre-training data and optimize its task loss (e.g., cross-entropy loss for classification task) along with the target data when muni-tuning The generalization analysis confirms that the performance on the target data can be improved when an appropriate portion of pre-training data is selected, as illustrated in Figure 1.
Since target data can be from different domains, we study the reusing strategy of pre-training data for different cases. First, when the target data is closely related to the pre-training data, one can randomly sample a number of pre-training data for fine-tuning, which is referred to as random selection. Second, if the label information of pre-training data is available and the classes overlapped with target data are identifiable, one can directly use those data with overlapped classes in fine-tuning. For example, given the data set of CUB [7], which consists of birds images, 59 bird classes [8] in ImageNet can be reused fine-tuning. This scheme referred to as label-based selection. Finally, when the labels between pre- pre-training and target domains cannot match exactly pre- pre-training data has no labels as in self-supervised pre-training the similarity measured by representations extracted from the corresponding pre- pre-trained model will be adopted for selection. The last setting is prevalent in real-world applications and referred to as similarity- based selection.
Given the large scale of pre-training data, the representations from the pre- pre-trained model can capture semantic similarity [9]. Based on this observation, we propose a novel selection algorithm to obtain a subset from pre-training data closest to the target data by solving an unbalanced optimal transport (UOT) problem. Interestingly, the proposed method performs consistently well on other scenarios, e.g., labels are overlapped, which reduces the effort of identifying overlapped pre-training classes. The main contributions of this work are summarized as follows.
From the perspective of generalization analysis, this work explains the phenomenon that training from scratch has a similar final performance as fine-tuning the pre-trained model in some computer vision tasks, when the training data in the target domain is sufficient. We develop the generalization analysis when mre-trainin and target data are used in fine- tuning simultaneously, under some mild assumptions. It demonstrates that the performance on target data will likely be improved when the pre-training data similar to the target data. With the insight of generalization analysis, we propose to select subset of pre-training data with better similarity to the target data to further boost the final performance. novel UOT-based algorithm is developed to handle target data from different scenarios.

---

The performance of the proposed fine-tuning process is evaluated on 8 benchmark data sets for image classification tasks. When a self-supervised pre-trained model used, our method, UOT fine-tuning, surpasses the conventional fine-tuning pipeline by large margin of 2.93% averaged over all tasks, verifying the effectiveness of reusing pre-training data.
2 Related Work
Fine-tuning as a special case of transfer learning [10, 11, 12, 13] aims to improve the performance on the target data by transferring the knowledge from large-scale pre-training data to target domain. For example, supervised pre-trained models on ImageNet have been extensively used in image classification [9], object detection [2, 14] and semantic segmentation [3, 15]. However, the empirical study in [5] shows that the advantage of a supervised pre-trained model over random initialization cannot be observed when the gap between pre-training and target task is large, or the target task has sufficient training data and is trained for sufficient time. Later, [4] demonstrates that self supersupe pre-training improves upon training from scratch in object detection and other vision tasks with strong data augmentation, indicating that self-supervised pre-training learns more general visual representations. Our work considers a general pre-training paradigm including both supervised and self-supervised approaches, and explains why pre-trained models fail to significantly outperform random initialization from the view of generalization theory. Different from existing work that regularizes the fine-tuning optimization explicitly [16, 17], we propose to reuse pre-training data in target training based on the theoretical findings.
There are several existing papers that explore the source data selection [18, 19, 20]. [18] improves fine-tuning by borrowing data from source domain which is similar to the target domain. The difference between [18] and our work are two-fold: 1) our work proposes a novel pre-training and fine-tuning pipeline while [18] is a joint training framework without pre-training: 2) our proposed data selection is global search method using deep features from the pre-training model while [18] uses low-level features and retrieves similar images with local search. Our paper demonstrates the weakness of local search compared to global search in our experiment. [19, 20] proposed similar schemes that pre-train the model on the selected subset from the pre-training data according to a domain similarity measure. but they do not use the selected data along with target data in the fine- tuning but re- re-pre-train on the selected data for every target task, which is a fundamental difference between their works and ours. It is not surprising that such a re-pre-training framework brings benefit to target task but it costs more computational time and resources than our fine-tuning framework. Another type of works uses the relationship between source and target improve fine-tuning [21, 22). [21] exploits the relationship between source and target labels, and [22] trains a policy network to control the gradient mask for backbone's 's blocks. Our paper explicitly adds a selected portion of pre-training data in fine-tuning and the selection strategy can handle self-supervised pre-training since it does not need source labels. [23] proposes to optimize a weight for the target data samples instead of selecting pre- pre-training data for improving fine tine-tuning Such wrget-weighting methods are compatible with ours, which selects source data, and future work will consider their combination.
The similarity- -baed data selection scheme in this work the one based on a variant of optimal transport (OT) optimization. General OT is often used in computer vision to estimate or/and minimize the distance between two probability measures, such as prediction probabilities in classification [24], density maps in crowd counting [25] and the reconstruction loss in generative models [26, 27]. [28] measures the distance between two data sets by using OT and label information. Our paper solves an unbalanced optimal transport (UOT) problem between pre-training and target data to obtain similarity vector for pre-training data to select a portion of data close to the target task.
3
Main Results
3.1 Theoretical Understanding
Preliminary. The target problem of interest that aim to optimize can be formulated
min F(0)= == E(r.,)>>P (f: (0:a0 x; v)]
where 0 is the model parameter be learned; (zc, y) is the input-label pair that follows a unknown
distribution PP; E(r is the expectation that takes over random variable (z, 3) while we use
3

---

E|-] for the sake of simplicity when the randomness is obvious; f(:; 22, v) is a loss function. Suppose we have a set of training data { y1)..... (z } drawn from P are given, where n is the sample size. In practice, we want to solve the following empirical version of problem (1):
min F (0) DERd
Σ f(0:I;:
(2)
Stochastic gradient descent (SGD) [29] is very popular algorithm for solving problem (2) in many computer vision tasks, whose updating is given by 81+1 = 02 of Ii, V..) where t = 0, 1, I.....
? is the learning rate, (of xz y) is the gradient of function f(0; xt y) with respect to variable
0. When the variable to be taken a gradient is obvious, we use ff(;; x. y) for simplicity. We
use the excess risk (ER) as the performance measurement for a solution 0: F(Ø) F(O.), where 0. arg mingeR* (0) is the optimal solution of (1) and 0 is the output of SGD.
In order to describe the pre-traines model, we denote by G(0)) == E(z''r)) [g(0; 3/)] the objective function that the tre-trained model aims to optimize. We also suppose that we have a set of training data {<11 vi..... (a'm }5 drawn from Q Usually, the sample size of pre-training data is larger than that of target data, i.e., > For the sake of analysis, we let be large enough and both the pre-trained model and the target learning task share the same set of parameters. In order to ensure that the model learned by optimizing G(f) will be valuable to the optimization of F(0), we assume that the difference of their gradients is bounded. That is, there exists a constant A > 0 such that [[) (G A, vO E Rd.
Value of Pre-trained Model. To see the value of pre-trained model, we present the excess risk bounds of the pre-trained model Bp and the final model f after fine-tuning 8p for the target task in the following lemma.
Lemma (Informal). We have the following performance guarantees for target task F(0) in expectation. (1) The pre-trained model 8, provides F(6,)- - F(O.) S 0(A2). (2) The final model Oj after fine-tuning Op against a set of n training examples provides F(O,) F(O.) S O log(n
First, the performance gap between pre-trained model 0, and the optimal model O. is bounded by O(A2), where A describes the approximation accuracy when replacing (0) with (G(0). Second, note that A only appears in the logarithmic term, implying that the excess risk bound has a weak dependency on the pre-trained model when n is large. That is to say, when n is lager, the pre-trained model has small effect the final performance, which is consistent with the empirical results found in [5].
Value of Pre-training Data. In reality, a target task often does not have enough data to fine-tune the model for a long time, so the pre-trained models are still better than random initialization in many vision tasks. Nevertheless, Lemma 1 reveals that even is not large, the dependency of generalization on a pre-trained model is potentially weak. This leads to a natural question: is it possible to design a better fine-tuning process that can overcome the limitation of the existing one? To this end, we develop a new approach for fine-tuning that aims to leverage the data used for the pre-trained model during the phase of fine-tuning, In this way, aim solve the following problem during the fine-tuning process:
(3) m Σ a0;a/..0 1/) where is a loss function related to target task and E; := (a/,, 1/,) is drawn from Q. Note that the loss function h can be same as the loss function of the target task. The solution is updated by SGD: 0. = 6₂ - 1/ V ((,,) where f(..) is the stochastic gradient related to (3). The theorem below provides a performance guarantee for fine-tuning via (3).
(F) + (1 - (m)
where a E (0, 1] is a constant, Fa (0) is defined in (2), and Hm (m)(0) = =0
Theorem 1 (Informal). We have the following performance guarantee for target task F(0) in expectation,
F(O,.)- - F(O.) s 50{ alog(n33/0/ + +11-))))
(4)
where 5² max6..s, (E( (.((0:: and Gy. is the final model for the target task.
When 82 is small, by choosing appropriate a E (0, 1}. we may be able to further reduce the error from F(O,.). When 8² is large, that is, when the second term of bound in (4) dominates the total

---

LOT
Top classes
Figure 2: Illustration of UOT selection. (a) UOT selection. Green dots denote target classes and orange dots denote pre- training classes/clusters. Blue and pur- ple arrows show the result of UOT for b; and bj. where the thickness means sim- ilarity. (b) Top- similarity in P1 CUB data set. Most bird classes from Im- ageNet are selected in the top-100 simi- larity vector to CUB.
Target
Nid
Pro Training
16
rum lord
1PL1, 1P11
Samiarky
30"
50
75
100
Rank
error, then it would be worse than the result of standard fine-tuning a pre-trained model in Lemma 1. Therefore, our goal is to select appropriate E,, the training examples from pre-training data, such that Vh(0;; can better approximate F(()). These theoretical observations inspire us to design a selection strategy for pre-training data, that is, to select images similar to those of target data from pre-training data and use these selected images during fine-tuning.
3.2 The Proposed Data Selection Strategy
Theorem shows that the benefit of pre-trained model can be enhanced when pre-training data used during the fine-tuning process. This inspires us to propose data selection strategies and to choose an appropriate portion of pre-training data. In experiments, we follow the standard pre-training practice in computer vision to use a deep neural network pre-trained on ImageNet, and then select data from ImageNet to help fin-ttuning on target classification tasks. We summarize the proposed pre-training data reuse strategies as follows.
Label-based Selection When the label information of pre-training data is available, and the overlapped classes with target data are recognizable, can simply select the overlapped classes and use them during the fine-tuning. For instance, the bird images from ImageNet are all selected when fine-tuning CUB. Unfortunately, this scheme heavily depends on the label match between pre-trane pre-training and target data, which may worsen the performance in some real-world applications without perfectly matched classes.
Random Selection The second data selection scheme is to choose classes with uniform sampling. referred to as random selection. This strategy can improve the performance of target tasks the domain gap 8 between pre-training and target data is small, and keep the weights close to initialization the selected data are sufficiently large. The drawback uniform selection is that the domain gap 8² is not considered in the data reusing process, the performance heavily depends the inherent property of data sets.
Similarity- -based Selection To reduce the domain gap, propose the third data selection scheme, an HOT-based method, to choose data classes from the pre-training set whose distributional distance the target data set is small. The UOT-selection method is able to handle pre-training data with and without labels. When a supervised pre-trained model is used for fine-tuming, there are class labels for pre-training images so the selection unit of UOT class. When a eup-ruuee pre-trained model is used and there are no labels for pre-training images, we index the pre-training data by clustering in the feature space of the corresponding pre-trained backbone. Therefore, for unlabeled pre-training data, the selection unit is cluster.
With the labels or cluster indices, each class/cluster is represented as the mean of deep features from the pre-trained model, e.g., 512-dim features from the penultimate layer of a pre-trained ResNet18 8 model. Since the training set often has balanced classes, all classes or clusters are assigned with unit weights for both pre-training and target set. So we have two density measures for the target set and
pre-training set, i.e. ((a;, w/ K) and [(b;, w5s) 1)] Kg 1= respectively. Denote the features of target and pre-training data as (( and v(0) -E
/n(() and b, = Σ, vi /m(5) where (2( is the number of images in i-th class of target data and m(") is defined similarly for pre-training data. In the general case where Ky * Kg. the two measures have different total masses
5

---

so we propose to compute the unbalanced OT distance between the two by a generalized Sinkhorn iteration [30]. Specifically, the optimization objective is formulated a UOT problem,
min PER K
(P.C) ch(P) + kKLL + T2K 1, w()).
(5)
where Ci. is the distance between a; and by; P is the transportation matrix solved by the generalized Sinkhorn iteration; 71 and T2 determine the constraint on the reconstruction loss of pre-training and target density measures; (T(,,,) and h(.)) are Kullback-Leibhe divergence and entropy function. Note that as a result of unbalanced total masses, we cannot perfectly reconstruct pre-training and target measures at the same time. Using this property, we can create a similarity ranking effect in the P1 vector by using a large value for 72 but small value for 71. P1 is the density measure of pre-training data and 1=1 is the measure for target data. Since we want all classes of the target data to be covered, large 72 is needed; while we need to select a subset of classes.
should be small to relax the constraint. Thus, a large [PI], indicates a high similarity of class-j of pre-training data to the target data. Finally by ranking the elements in P1 and selecting top-K K classes, we obtain the selected classes for target data set. Fig. 2 visualizes the UOT selection and the similarity vector given by UOT on the CUB data set.
3.3 Gradient Computation
We demonstrate how the gradient combination (3) is computed in the experiment of this work. In the case where pre-training data has labels, we add two classification heads on top of the network backbone. One classification head has K K--dim output to predict the target data and the other has K Kg- output to predict the pre-training data. The optimization objective for the labeled case is
Σ f(0; De, :t. + A
m. Σ ((, V..)
(6)
where {zt,, St. } are the target data, f V., are the pre-training pre-training data and m. ñ are batch size. A is the weight for pre-training classification loss, which controls the weight a in (3). Although the classification heads are different for pre-trat pre-training and target data, we assume the optimization variables 0 in ʃ and l are consistent since the output layers only have small amount of parameters compared to the backbone. In the case where pre-training data has no labels, the unlabeled data is used in semi- superviee way,
f(0; Jt Vc.
Σ h(0; p(u,, I2,,))
where the unlabeled pre-training data is processed with weak and strong data augmentation [31] respectively, and the probability prediction of weakly augmented data p(tt, |a's, is taken as the soft pseudo-label for the strongly augmented data H.,- Note that there is only one classification head in (7) and temperature or threshold is not used in the weak-strong training.
4 Experiments
This section presents the empirical analysis of reusing pre- pre-training data in image classification tasks. The experiment uses both supervised and self-supervised pre-trained models to fine-tune a variety of image classification data sets. First, data reusing fine- mee-tuning schemes consistently improves the performance of vanilla fine-tuning, which corroborates theoretical result. Second, the comparison between different data selection strategies demonstrates that the UOT selection is advantageous over random and greedy selection. Third, we simulate the situations where the training data scarce by sub-sampling the given training data and show that as the training data get insufficient, the performance gain of the pre-training data reusing method will increase. Finally, some abl ablation studies on experimental settings are given.
4.1 Experiment Setup
The empirical study is done on both supervised and self-supervised pre-trained models. For the supervised training, we use the official ResNet18 [32] pre-trained on ImageNet. For the self- supervised training, we use the official MoCo-v2 [33] ResNet50 pre-trained with 800 epochs. In similarity-ban -base selection, images are represented in the supervised pre-trained ResNet18 by 512- dim features from the penultimate layer while in MoCo-v2 by 128-dim features from the final

---

(a)
Supervised Pre- Treininin Model
Method
Dogs
Cars
CUB
Pets
SUN
Aircraft
DTD
Caltech
Avg.
Fine-Ture
82.65
85.87
75.49
91.40
58.03
77.62
70.64
90.11
78.98
Baseline
Co-Tuning [21]
80.90
86.48
76.83
89.92
58.73
79.07
69.69
93.08
79.34
Random
83.29
86.52
75.54
91.58
58.18
78.10
70.69
90.64
79.32
Data Selection
Greedy-OT
84.63
86.79
76.92
91.66
58.70
78.43
70.90
90.67
79.84
UOT
84.67
87.03
77.21
91.98
59.06
78.94
71.17
91.11
80.15
(b)
Supersupe Pre- Training Model
Method
Dogs
Cars
CUB
Pets
SUN
Aircraft
DTD
Caltech
Avg.
Baseline
Fine-Tune
78.64
91.05
77.44
90.44
61.12
87.25
75.80
92.82
81.82
Data
Random
79.87
90.85
78.82
91.48
62.42
88.60
77.34
93.26
82.83
Selection
Greedy-OT
79.43
90.89
78.63
91.27
62.27
89.40
76.81
93.36
82.76
w/ Labels
UOT
88,14
90.89
80.98
93.05
64.76
89.28
77.45
93.45
84.75
Data
Random
79.77
90.93
77.96
90.70
62.26
89.17
76.97
92.72
82.56
Selection
Greedy-OT
81.16
90.87
78.63
91.21
63.41
89.29
77.29
93.39
83.16
w/o Labels
UOT
81.47
90.91
78.96
90.39
63.68
89.59
77.07
93.27
83.17
Table 1: Comparison of testing top-1 accuracy (%) different data sets by fine-tuning the supervised and self-supervised pre-trained model. The proposed data selection fine-tuning consistently improves the vanilla fine-tuning, with UOT being the best method. A bold number denotes the top-1 accuracy and an underlined number denotes second best accuracy.
FC layer. The pre-trained model is tested on 8 target image classification data sets, i.c. Stanford dogs (Dogs) [34], Stanford cars (Cars) [35], Caltech-UCSD birds (CUB) [7], Oxford-IIIT Pet (Pets) [36], SUN [37], FGVC-Aireraft (Aircraft) [38], Describable Textures data set (DTD) [39] and Caltech101 (Caltech) [40]. During the fine-tuning process, both the backbone and randomly initialized classification heads are updated using SGD with Nesterov Momentum. The training epochs are fixed to be 100 in our experiment for sufficient training and the learning rate is divided by 10 at 60 and 80 epoch. Other hyperparameters like initial learning rate, weight decay and À are determined by grid search for all selection methods in the comparison (details in the Appendix). When there are no labels in the pre-training data, K-means clustering [41] is used to estimate the cluster assignment with 128-dim features as input and cosine similarity as distance. The cluster number is set as 2000 and we give an ablation study in Table 2 on the cluster number.
We test data selection methods: random selection, greedy selection and UOT selection, and set the number of selected classes to be 100 unless mentioned otherwise. Specifically, we use the OT-based greedy algorithm [19] for comparison. The batch size for fine-tuning data is 256, and pre-training data are reused, the batch size keeps the same as target data which makes total batch size of 512. In random selection, we use the uniform selection over classes or clusters to be consistent with the other data selection methods. In Greedy-OT, we use the same setting as in the original paper where Cij is the la distance In UOT, we set 1.0, Ti 1.0 and 72 = 100.0. The distance cost based on the cosine similarity Cij = cos(a: b,)+1 with c. = 0.01.
For the supervised pre-trained model, we also compare with Co- TonnTnnn [21], a strong transfer learning bascline. The experiment setting follows the original paper and we search the initial learning rate from (1e-4, 3c-4, Ic-3, 3e-3, 1c-2) on a validation set and report the test accuracy trained on the original training or train+val set. Note that Co-Tuning relies on the labels and the classification head in pre- mre-training so it is non-trivial to use Co-Tuning for self-supervised pre-trained model. We only compare with [21] since it is the most recent baseline which surpasses existing baselines on those datasets we evaluate. [22] needs to train another policy network to help the adaptive fine-tuning, which is more complex than ours. [19, 20] are pre-training methods that needs pre-training for every new task. [18] does not consider the pre-training and fine-tuning pipeline and needs low-level features to do data selection.
4.2 Comparison of Data Selection Strategies
Tab. shows the comparison between the standard fine-tuning, Co-Tuning and data reuse methods on 8 image classification data sets, with supervised and self-supervised pre-trained model. In the
7

---

(DT)) CUB
CatyCa
Suh-Sarppiia
Sch-Sarspling Calach
2
dry 1
13
06.2
140
150
250
150 100
130
30
35c
30
Penorragy Triining Dowp
Percertnge of Training Dora
Figure 3: (a) Accuracy of fine-tuning using UOT data selection with different numbers of selected classes using the supervised pre-trained ResNet18. (al) shows the performance on CUB and the blue line is tinettuiing with all birds classes from ImageNet. The UOT selection achieves comparable performance to the label-basee data selection. (a2) shows the increased performance of UOT selection on Caltech as more data reused in UOT, while the performance of random selection is consistently worse than the UOT's. (b) Accuracy and performance gap when sub-sampling training data using the supervised pre-trained R (b1) and (h2) show decreasing trend of performance gain when more training data are added on CUB and Caltech. The advantage of pre-training data reusing is larger when training data sufficient.
K
Dogs
Cars
CUB
Pets
SUN
Aircraft
DTD
Caltech
Avg. 83.04
1000
81.02
90.86
78.48
90.55
63.65
89.16
77.50
93.09
2000
81.47
90.91
78.96
90.39
63 68
89.59
77.07
93.27
83.17
Table Ablation study on cluster number. The test accuracy fine-tuning when different cluster numbers are used in the K-means algorithm demonstrates that K=2000 gives better generalization than K=1000.
labeled data, 100 classes of ImageNet data are reused. the unlabeled data, 200 clusters are used to keep the number of selected images about the same as that in labeled data selection.
The first observation is that, since the pre-training data are large enough to have similar images to target ones, even random selection achieves better performance than the standard fine-tuning in most data sets. Secondly, the benefit of data reuse is amplified by the similarity- predicted by Theorem 1. Thirdly, UOT is better than Co-Tuning in 6 out of data sets and its average accuracy has clear benefit over Co-Tuning, indicating that the effectiveness of Co-Tuning is not robust to task variation. In addition, the proposed data selection is more versatile since it performs well on the cele-super model while Co-Tuning cannot handle the self-supervised model. Finally, the comparison between Greedy-OT and UOT data selections demonstrates the advantage of the global UOT in terms of the similarity measure. Fig. 3.al shows the performance of label-based selection on CUB (blue line), since the birds classes happen to exist in ImageNet. It turns out the accuracy of UOT selection method (77.21% is comparable to the label- basel selection's (76.89 76.89%). In addition, we also test the performance of label- based selection on Dogs (selecting 118 dog classes of ImageNet), the performance of which (85.05%) is again comparable UOT's (84.67%). This comparison demonstrates that the proposed UOT selection is generic yet effective.
data selections, as
The advantage of UOT is more evident in the self-supervised pre-training case than in the supervised pre-training one. The most considerable improvement is achieved in Dogs, Birds and Pets data sets because the animal-related classes are dominant in ImageNet (398 classes of birds, dogs, animals and mammals) and self-supervised training learns good visual features without label information. Once the label information is added to the fine-tuning process by data reuse, the model is taught to recognize those familiar features and achieves giant improvements. Note that the only data on which data reuse does not help is the Cars, indicating that the gap between ImageNet and Cars data is large when measured by the self- suf-ssueer model. When image labels in self-supervised pre- trainiiing are not available, the proposed data selection framework still outperforms the vanilla fine-tuning baseline on most data sets. The benefit of UOT over greedy-OT -OT is large on Dogs and CUB data sets, indicating that UOT is still better at selecting similar images from ImageNet even when label information completely unknown.
8

---

(a) Recall Rate v.S. 6.
(b1) Supervised Moddl
(b2) Supe-Supervised Model
1.00
10
10"
10
Rooal Rote Bate
50-
Accurazy
76.5
10
10"
76.0-
70
01005
75.5
Seperviual Seif Supervised
75.0
50
1.0
6.3
0.1
0.05
0.01
0.00
5
8
9 10
Similarity
Similarit
Figure 4: (a) Sensitivity of recall rate ec. On the supervised pre-trained model, the recall rate is not sensitive to eei on the self-supervised model, when c is small, the performance is stable. (b) The similarity of selected ImageNet split to the downstream CUB data set versus downstream accuracy.
4.3 Simulation of Low-Data Regime and Long-Taie Tail Label Distribution
To study the effect of data reusing in the scarce data scenario, we simulate low-data target tasks in this experiment by sub-sampling CUB and Caltech training data. We select these two data sets because they represent the fine-grained and general classification task, respectively. For each class training data we randomly sample 20%, 40%, 60% and 80% of images get blass-balanced training data. Figure 3b shows that accuracy and performance gap between vanilla and data-reusing mee-tuning when different amounts of training data are available. On both data sets, the performance gap is increased as the training data size decreases, indicating that the UOT-selection data reusing scheme helps more when the target data is insufficient. This experiment demonstrates that the proposed data reusing paradigm is particularly effective when the target task does not have enough data, which could be a typical case real-world applications.
The long-tail mail class distribution is also challenge in real-world applications. We simulate long-tail data sets with CUB, Aircraft and Caltech by sam- pling images from classes with Pareto distribution [42] as in [43], where the number of images in the largest class is 10 times that in the smallest class. The results of fine-tuning a ResNet18 are shown in the Table 3. The proposed method has more significant improvement when the target data is imbalanced compared with the original balanced class data sets. The reason is that the selected pre-training data can be controlled to have balanced label distribution and the gradients of pre-training data have regularization effect, so the model does not easily overfit images in minor classes.
CUB
Aircraft
Caltech
Fine- Fine-Tuning UOT 57.27
52.73
52.74
73.96
53.96
78.20
Table 3: The result of UOT-fine-tuning in data sets with long-tail class distribution.
4.4 Ablation Study
Number of selected pre-training classes. We investigate the effect of selected class numbers on the target classification accuracy. Figure 3a shows the performance of target tasks (CUB and Caltech) when the number of selected classes ranges from 50 to 300 UOT selection. The increased pre-training data added in fine-tuning do not improve the performance of CUB, since there are 59 classes of birds the ImageNet and more reused images enlarge the gap 8², Surprisingly, we observe that only using the bird images (blue line) is not the best strategy on CUB. It is because that there can be a certain number of related classes in ImageNet, which will help the classification of bird images. The result shows that even when labels of pre-training and target data given and overlapped, UOT selection can achieve better performance by including extra relevant classes from pre-training data. On the general classification data set (Caltech), more reused pre-training data help gain the performance improvement because the diverse data set needs a large number of images to have a small domain gap. On both data sets, the UOT selection performs better than the random selection as the number of selected classes changes.
Cluster number in K-means. The cluster assignments are crucial to the performance of similarity- based selection, so we change the cluster number the experiment and show the fine-tuning results of 8 data sets in Table 2. On most data sets, K=2000 achieves better generalization than K=1000,

---

indicating that fine- -ree-gaaiedd clustering helps the following data selection step. However, if we further increase K, the clustering result will be quite noisy and extremely small or large clusters will be found, which makes the fine-tuning result worse than K=2000.
Distance function and 6g- To investigate the influ-
ence of different factors in the UOT selection, we define a recall rate as a metric to make the compar- ison. For a target data set whose classes happen to exist in ImageNet, the ummilarity-based data selec- tion is expected to choose those matched classes, e.g.. selecting all 59 birds classes from ImageNet when fine-tuning on CUB. Thus. the recall rate on CUB is defined as the ratio between the number of bird classes in the top- 100 similar vector or EMD) similarity and 59. With the performance metric, we first compare UOT with Greedy-OT using /2 and cosine distance in Table 4. With a supervised pre-trained model, Greedy-OT is only slightly worse than UOT, while with self-supervised model the weakness of Greedy-OT is amplified. It means that Greedy-OT heavily relies on the label information in supervised training but UOT only needs generic visual features to have good similarity measure. In addition, the cosine distance is better than the 12 distance, especially in the self-supervised model. The importance of cosine distance is due to the cosine similarity loss used in MoCo training [33]. Finally, Fig. 4a shows the recall rate when using different 6.- The supervised model is not sensitive to the choice of 6, but a small < is crucial to the good performance of OT-selection in the sele-supe model. Note that the recall rate of Greedy-OT does not depend on c. so the performance is worse than UOT no matter what 6. is used.
Method
Metrie
GOT-
GOT-
Uor-
UOT-
12
cos
12
cos
Sup.
Roc.
86.44
94.92
94.92
98 31
Acc.
76.92
76.67
77.08
77 21
Rec. 16.95
98.98
16.95
93 22
Self Sup
Acc.
78.63
79.32
78.48
80.98
Table 4: Ablation study on distance. The UOT-
cos selection better than Greedy-OT (GOT) selection in terms of bird classes recall rate (Rec.) and test accuracy of fine-tuning (Acc.).
The impact of the domain gap. We split ImageNet into 10 subsets based on UOT score (higher means higher similarity) to CUB data and reuse the 10 subsets in fine-tuning to see the results, which are presented in Fig. 4.b1-2. With both supervised and self-supervised model, when the UOT similarity score rapidly decreases, the performance of NOT-fine-tuning drops and is lower than the fine-tuning baseline. This experiment indicates that the dissimilar data hurt the performance when added to fine-tuning and highlights the importance of similarity- based data selection.
Subset of Pre- Pre-Training Data. In the case where the pre-training data is larger than the available storage capacity, a subset of the pre-training data could be used. We subsample 50% images from each class of ImageNet and use the 50% ImageNet data in the supervised ResNet18 fine-tuning, On CUB and Caltech, the UOT fine-tuning achieves 77. 77.03% and 91.00% accuracy respectively, which only drop 0.18% and 0.1 11% compared with using the full ImageNet. The experiment indicates that even with half of ImageNet, fine-tuning with UOT data selection still quite effective.
5 Conclusion
This paper provides the generalization analysis of pre-trained models when fine-tuned on target tasks by using excess risk bound, which suggests that the pre-trained model can have little positive influence on learning from target data under certain conditions. It also shows that the performance on the target data can be improved when similar data is selected from the pre-training data for fine-tuning. Inspired by this result, a novel similarity-based selection algorithm is developed, which is evaluated on 8 data sets and shown to be effective. Our future work will further explore the data reusing strategy in other computer vision tasks. Broadly speaking. our research underpins the importance of pre-training data in downstream applications, which is neglected by current research focusing on pre-trained models, and thus has impact in advocating the data-centric Artificial Intelligence (AI) [44]. One limitation of the current work is that sometimes the pre-training data is private (e.g.. JFT-3B [45]) and not accessible. Given the power of pre-training data unveiled by our work, advocate that more pre-training data should be open for the better use of pre-trained models.
Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program, the Funda- mental Research Funds for the Central University of China (DUT No. 82232031) and a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. UityU 1 11215820).
10

---

References
[1] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[2 Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real- sal-time object detection with region proposal networks. Advances in neural information processing systems, 28:91 19991999999 2015.
[3] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[4] Barret Zoph, Golnaz Chiasi. Tsung-Ys Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe Le. Rethinking pre-training and self-training. Advances in Neural Information Processing Systems, 2020.
[5] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918 4927, 2019.
[6] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 222--123,, 2016.
[7] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- uesd birds-200-208 dataset. 2011.
[8] Qi Qian, Juhua Hu, and Hao Li. Hierarchically robust representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7334-7344 2020.
[9] Jeff Donahue, Yangging Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference muchine learning, pages 647-655. PMLR, 2014.
[10] Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22(10):1345 1359, 2009.
[1I] Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. Advances in Neural Information Processing Systems, 32: 1908 1918, 2019.
[12] Xingjian Li, Haoyi Xiong. Hanchao Wang. Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint arXiv: 1wt.:9991.0999... 2019.
[13] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pages 2925-2834. PMLR, 2018.
[14] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2017.
[15] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3440. 2015.
[16] Henry Gouk, Timothy Hospedales, al. Distance- regularisation of deep networks for fine-ture fine-tuning. In International Conference on Learning Representations, 2020.
[17] Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. International Conference on Learning Representations, 2020.
[18] Weifeng Ge and Yizhou Yu. Borrowing treasures from wealthy: Deep transfer learning through selective joint tuning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1086-1095, 2017.
[19] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine- grained categoriza- tion and domain speciffc transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109-4118. 2018

---

[20] Shuvam Chakraborty. Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Sheehan, and Stefano Ermon. Efficient conditional pre-train pre-training for transfer learning. arXiv preprint arXiv:2011.10231 2020.
[21] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning, Advances in Neural Information Processing Systems, 33, 2020.
[22] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4805-4814, 2019.
[23] Yonatan Dukler, Alessandro Achille, Giovanni Paolini, Avinash Ravichandran, Marzia Polito. and Ste- fano Soatto. Diva: Dataset derivative of learning task. In International Conference on Learning Representations, 2022.
[24] Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, and Tomaso Poggio. Learning with a wasserstein loss. arXiv preprint arXiv: / 550695 2015.
[25] Jia Wan, Ziquan Liu, and Antoni B Chan. generalized loss function for crowd counting and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1974- 1983, 2021.
[26] Giorgio Patrini, Rianne van den Berg, Patrick Forre, Marcello Carioni, Samarth Bhargav, Max Welling, Tim Genewein, and Frank Nielsen. Sinkhorn autoencoders. In Uncertainty in Artificial Intelligence, pages 733-743, PMLR, 2020.
[27] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning. pages 214-223, PMLR. 2017.
[28] David Alvarez Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33, 2020.
[29] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statisticy. pages 400-407, 1951.
[30] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-4 607, 2019.
[31] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition Workshops, pages 203-703, 2020.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 770-778, 2016.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[34] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine finengrained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Grained Visual Categorizattin (FGVC), volume 2, 2011.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fne-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[36] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEFE conference on computer vision and pattern recognition, pages 3498-39900 3505, IEEE, 2012.
[37] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485 3492. IEEE, 2010.
[38] Subhransu Maji, Esa Rahtu, Juho Kannala, Mathew Blaschko, and Andrea Vedaldi. Fine- traee-graned visual classification of aircraft. arXiv preprini 100iv:1306.5151, 2013.
[39] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606 3613, 2014.
12

---

[40] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and puttern recognition workshup, pages 178-178. IEEE, 2004.
[41] Stuart Lloyd. Least squares quantization in pem. IEEE transactions information theory. 28(2):12--1377 1982.
[42]
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 46(5):323-351, 2005.
[43] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277, 2019.
[44] Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, Fei-Fei Matei Zaharia, Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, pages 1-9, 2022.
[45] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104- 12113, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [No] We study the general property of fine-tuning in computer vision.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See the supplemental.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplemental.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See the supplemental.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We only report the best result among 3 trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the supplemental.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] See the creators' papers reports.
(c) Did you include any new assets either the supplemental material or as URL.? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] See the creators' papers or reports.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We use public datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "wf7AfZTAaZT", "reviewer": "Reviewer_718u", "summary": "The authors investigate whether a pre-trained model can be enhanced when pre-training data are used during fine-tuning. They develop an approach that performs data selection to choose an appropriate portion of pre-training data to enhance the phase of fine-tuning. Specifically, the authors propose a selection algorithm to obtain a subset from pre-training data closest to the target data by solving an unbalanced optimal transport (UOT) problem, which chooses data classes from the pre-training set whose distributional distance to the target data set is small. \n", "questions": "- What if the training data is a duplicate of the target data? The training data may not further help the target task. For example, CUB has an overlap with ImageNet.\n\n- What if the target label and the source label are semantically similar but not exactly similar. For example, if the target task is to classify dogs vs cats, but the source data have a lot of dog species but not the “dogs” label?\n\n- Fine-tuning is sensitive to hyperparameters [1]. In the paper, the authors performed a large-scale grid search of the optimal hyperparameters for the best performance (5x3x3x3=145 trials per dataset). Is the method sensitive to hyperparameter changes? I am curious if the authors have any observation of whether the hyperparameters are correlated with the similarities of added source data and target data so that the HPO trials can be reduced.\n\n- There are also methods that directly select the data that can benefit the target task (e.g., [2]), which were not discussed or compared.\n\n[1] Li et al, Rethinking the Hyperparameters for Fine-tuning, ICLR 2020\n[2] Dukler et al, DIVA: Dataset Derivative of a Learning Task, ICLR 2022", "limitations": "The limitation is not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is well written. The author provides both theoretical and empirical investigations. The proposed selection method considers to handle pre-training data with and without labels. The paper has an extensive comparison with other strategies.\n\nWeakness: \n\n- The reason why source data can help for target tasks is not well understood. \n- The theoretical study does not provide clear insights on how similar the source datasets affect the fine-tuning performance. \n- It is not clear whether the target task can always benefit from the techniques. The author discussed “the impact of the domain gap”, however, there is no clear criteria whether the method should be used. For example, will a medical dataset benefit from the ImageNet pre-training data?\n- There are also methods that directly select the data that can benefit the target task, which are not discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "qlnIz_SJbGE", "reviewer": "Reviewer_51Vh", "summary": "This work studies the problem of using data of the pretraining task (pretraining data) for the fine-tuning stage.\n\nThey first analyze the excess risk of the target task when the pretrained model and the pretraining data  are used, and show that the proper use of pretraining data can tighten the bound of excess risks. \n\nFor selecting the proper subset of pretraining data, they propose a similarity-based selection method based on the distributional distance between pretraining classes and target data.  \n\nTheir empirical results on several image-classification tasks validates the findings of the previous analysis and the several improvements of their method over the conventional fine-tuning without pretraining data.", "questions": "Please see the questions mentioned above.", "limitations": "I do not see the limitation and negative societal impact sections. Please see my mentioned limitations above. For the societal impact, the authors may consider the impact of the work on utilizing more pretraining data then standard fine-tuning.  ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Problem setting: The use of selected source data to improve target task has been touched before in the transfer learning literatures. This paper studies a seemly new setting: both pretrained model and pretraining data are used in the fine-tuning task. Though it’s new, my question are how important is this setting and under which scenarios is this setting useful? To be specific, I see the overlap of pretrained model and the pretraining data: If the selected pretrained data is more valuable than the pretrained model, why don’t we train the model from pretraining data and the fine-tuning data? Otherwise, as the purpose of the pretrained model is to provide an initially good (and general) representation, if the pretrained model is well-trained, it already well utilized the pretraining data. Also, as training from scratch and pretraining are usually expensive and intensive, does the use of large pretraining data contradict to the purpose of fine-tuning? (In the experiment, the paper use1 up to 100 classes of ImageNet which is approximately 100K samples while the target data (e.g., Stanford Cars) has much less data.\n\n- Analysis: The lemma and theorem are simple but useful to illustrate the idea of this paper. From my understanding, the main contribution of these analyses is to show the usefulness of proper pretraining data selection, rather than study the defined problem in general.\n\n- Similarity-based method:  The method is intuitive.\n    - One concern is how K-mean on high-dimensional data is useful to provide distinguishable clusters (especially with Euclidean distance)? If these clusters are uniformly sampled from the data, the method is likely to be the random selection.\n    - Another question I have is the justification of the real effect of pretraining data. Since the paper simply incorporate the pretraining data via extra regularization term (loss) in the fine-tuning. It’s intuitive that this regularization can at least achieve better performance than the original loss (fine-tuning) with proper regularization parameters. And, the experimental results also show the marginal improvements of the modified loss over the standard fine-tuning and sometimes gets lower performance (table 1b).\n\n- Clarify: the paper is pretty clear to follow and easy to understand.\n\n- Quality:\n    - Writing can be improved:\n        - In paper 2, the statement “From the perspective …” in lines 60—63 is mostly duplicated with lines 64—66\n        - Some statements need to be provided with evidences and more precise contexts. For examples, lines 32—33: “…. training from scratch never matches the performance of fine-tuning …. “\n        - Minors: there are some typos, e.g., ‘datausing’ (Figure 3 caption), ‘images’ (line 172) should be ‘samples’.\n\n- Experiments\n    - Some results seem to be  not representative to me. For example, the Fig.3a shows that 100-class setting provides the best overall performance of method while other settings result in lower performance than the label-based selection. It’ll be better to provide more comparison and robust metrics to the table 1.\n    - While table 1 shows the improvement of using proposed method, the improvement is marginal and unclear whether the gain is from the pretraining data or proper parameter tuning?\n        - As mentioned above, all considered target tasks have smaller number of data compared to the pretraining data. I think it’s better to show with more settings when the pretraining data is much smaller.\n    - Baselines: I suggest the authors to try simple but related baselines: training both pretraining data and target data without pretrained models?\n    - Some results in table 1 are far below the best transfer-learning results on the target datasets. For example, on the Stanford Cars, the latest transfer learning may achieve up to 96% accuracy.\n        - What happen if we use stronger pretrained models?\n    - In figure 3a-2, why don’t we have the label-based line?\n    - In figure 3b, what is the accuracy gap?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hCeMBT1lrN5", "reviewer": "Reviewer_HBDj", "summary": "Pretrained models are widely adopted for various downstream applications and have been demonstrating superior performance. A de-facto technique to utilize such a model is simply fine-tuning the pretrained model to a small downstream task’s dataset. This paper proposes a way to improve this process by selecting a subset of relevant pretraining data and training together with the downstream training data.", "questions": "Q1: Page 6: Is there any reason why the authors chose a small pretrained model? ResNet18 is a bit too small in my opinion. Did they consider larger pretrained models? (e.g. ResNet50 for supervised training or even ResNet101). Is it because larger models may memorize pretraining dataset better and may not benefit from using them?\n\nQ2: Since one of the motivations of using the pretraining data is to prevent “catastrophic forgetting”, I wonder if the authors compared their work to such techniques for example [1,2]. [1] shows it’s much better than simple fine-tuning and contains many baselines on this end. [2] actively prevents catastrophic forgetting even without data.\n\n[2] Hayes, Tyler L., et al. \"Remind your neural network to prevent catastrophic forgetting.\" European Conference on Computer Vision. Springer, Cham, 2020.\n[3] Chen, Sanyuan, et al. \"Recall and learn: Fine-tuning deep pretrained language models with less forgetting.\" arXiv preprint arXiv:2004.12651 (2020).\n", "limitations": "Commonly, we do not have access to the pretraining data because of various reasons (c.f. access, storage space, time etc). This limits the use of this method in such scenarios. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* Employing UOT (unbalanced optimal transport) to select the subset of the pretraining data is novel and was very effective from their empirical evaluation.\n* It is quite surprising that utilizing the pretraining data can improve the fine-tuning performance this much.\n\nWeaknesses\n* Selecting the subset of the pretraining data for the downstream task is not novel (as described in line 92.\n* Authors do not have many baseline methods in their empirical evaluation other than very simple ones. I encourage they compare their work to comparable methods such as [1] or methods that actively prevents catastrophic forgetting [2,3] (3 was not evaluated on the vision but in a similar fashion)\n* (minor) as noted in the limitation, sometimes we do not have the access to the pretraining data. In the contrary [3] does work without the pretraining data.\n\n[1] Chakraborty, Shuvam, et al. \"Efficient conditional pre-training for transfer learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Hayes, Tyler L., et al. \"Remind your neural network to prevent catastrophic forgetting.\" European Conference on Computer Vision. Springer, Cham, 2020.\n[3] Chen, Sanyuan, et al. \"Recall and learn: Fine-tuning deep pretrained language models with less forgetting.\" arXiv preprint arXiv:2004.12651 (2020).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eG17XHYLBjt", "reviewer": "Reviewer_phCA", "summary": "The paper shows how leveraging some pre-training data during fine-tuning can boost a model's pre-train + fine-tuned performance.\nThe authors suggest a few methods to this end, for both the scenario when pre-training data is unlabeled (i.e. self-supervised learning) as well as labeled -- selecting pre-training data by label, randomly, and via similarity with fine-tuning data by way of optimal transport. The latter is the most innovative and performs the best. With several key assumptions (the most important of which is the bound on the distance between the fine-tuning and pre-training loss gradients), the paper provides bounds for the excess risk under normal fine-tuning as well as under their proposed modified fine-tuning loss. They compare against a single recent baseline (Co-Tuning) and study image classification on 8 datasets using ResNets pre-trained on ImageNet and do several ablation studies.", "questions": "Would it be possible to add at least 1-2 NLP experiments as well? Perhaps the same one used in the Co-Tuning paper?", "limitations": "The authors do mention limitations in the Conclusion section, which I felt was adequate. However, more discussion around the assumptions made for the theory would be nice -- how strong are the assumptions? Do they hold in practice? ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Experiments have decent coverage (self-supervised + supervised pre-training ; limited data; 8 datasets) \n* Pretty solid ablation studies\n* Proposed method with UOT selection generally outperforms the other methods.\n* Excess risk bound shown seems semi interesting.\n\nWeaknesses:\n* Novelty is limited -- while I cannot comment extensively of how noteworthy the theoretical analysis is, the suggested approaches for selecting pre-training data seem straightforward.\n* Only co-tuning is compared against (though this method is claimed to be the most competitive recently) and only in the supervised setting. The proposed method with UOT selection loses to co-tuning on 2/8 datasets. Performance is measured as \"the best among 3 trials\" -- running this for more trials and reporting error bars would make the trends more convincing.\n* More experiments would strengthen the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept