Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors investigate whether a pre-trained model can be enhanced when pre-training data are used during fine-tuning. They develop an approach that performs data selection to choose an appropriate portion of pre-training data to enhance the phase of fine-tuning. Specifically, the authors propose a selection algorithm to obtain a subset from pre-training data closest to the target data by solving an unbalanced optimal transport (UOT) problem, which chooses data classes from the pre-training set whose distributional distance to the target data set is small. 

strengths_and_weaknesses: Strength:

The paper is well written. The author provides both theoretical and empirical investigations. The proposed selection method considers to handle pre-training data with and without labels. The paper has an extensive comparison with other strategies.

Weakness: 

- The reason why source data can help for target tasks is not well understood. 
- The theoretical study does not provide clear insights on how similar the source datasets affect the fine-tuning performance. 
- It is not clear whether the target task can always benefit from the techniques. The author discussed “the impact of the domain gap”, however, there is no clear criteria whether the method should be used. For example, will a medical dataset benefit from the ImageNet pre-training data?
- There are also methods that directly select the data that can benefit the target task, which are not discussed.
questions: - What if the training data is a duplicate of the target data? The training data may not further help the target task. For example, CUB has an overlap with ImageNet.

- What if the target label and the source label are semantically similar but not exactly similar. For example, if the target task is to classify dogs vs cats, but the source data have a lot of dog species but not the “dogs” label?

- Fine-tuning is sensitive to hyperparameters [1]. In the paper, the authors performed a large-scale grid search of the optimal hyperparameters for the best performance (5x3x3x3=145 trials per dataset). Is the method sensitive to hyperparameter changes? I am curious if the authors have any observation of whether the hyperparameters are correlated with the similarities of added source data and target data so that the HPO trials can be reduced.

- There are also methods that directly select the data that can benefit the target task (e.g., [2]), which were not discussed or compared.

[1] Li et al, Rethinking the Hyperparameters for Fine-tuning, ICLR 2020
[2] Dukler et al, DIVA: Dataset Derivative of a Learning Task, ICLR 2022
limitations: The limitation is not discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Pretrained models are widely adopted for various downstream applications and have been demonstrating superior performance. A de-facto technique to utilize such a model is simply fine-tuning the pretrained model to a small downstream task’s dataset. This paper proposes a way to improve this process by selecting a subset of relevant pretraining data and training together with the downstream training data.
strengths_and_weaknesses: Strength
* Employing UOT (unbalanced optimal transport) to select the subset of the pretraining data is novel and was very effective from their empirical evaluation.
* It is quite surprising that utilizing the pretraining data can improve the fine-tuning performance this much.

Weaknesses
* Selecting the subset of the pretraining data for the downstream task is not novel (as described in line 92.
* Authors do not have many baseline methods in their empirical evaluation other than very simple ones. I encourage they compare their work to comparable methods such as [1] or methods that actively prevents catastrophic forgetting [2,3] (3 was not evaluated on the vision but in a similar fashion)
* (minor) as noted in the limitation, sometimes we do not have the access to the pretraining data. In the contrary [3] does work without the pretraining data.

[1] Chakraborty, Shuvam, et al. "Efficient conditional pre-training for transfer learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[2] Hayes, Tyler L., et al. "Remind your neural network to prevent catastrophic forgetting." European Conference on Computer Vision. Springer, Cham, 2020.
[3] Chen, Sanyuan, et al. "Recall and learn: Fine-tuning deep pretrained language models with less forgetting." arXiv preprint arXiv:2004.12651 (2020).

questions: Q1: Page 6: Is there any reason why the authors chose a small pretrained model? ResNet18 is a bit too small in my opinion. Did they consider larger pretrained models? (e.g. ResNet50 for supervised training or even ResNet101). Is it because larger models may memorize pretraining dataset better and may not benefit from using them?

Q2: Since one of the motivations of using the pretraining data is to prevent “catastrophic forgetting”, I wonder if the authors compared their work to such techniques for example [1,2]. [1] shows it’s much better than simple fine-tuning and contains many baselines on this end. [2] actively prevents catastrophic forgetting even without data.

[2] Hayes, Tyler L., et al. "Remind your neural network to prevent catastrophic forgetting." European Conference on Computer Vision. Springer, Cham, 2020.
[3] Chen, Sanyuan, et al. "Recall and learn: Fine-tuning deep pretrained language models with less forgetting." arXiv preprint arXiv:2004.12651 (2020).

limitations: Commonly, we do not have access to the pretraining data because of various reasons (c.f. access, storage space, time etc). This limits the use of this method in such scenarios.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper shows how leveraging some pre-training data during fine-tuning can boost a model's pre-train + fine-tuned performance.
The authors suggest a few methods to this end, for both the scenario when pre-training data is unlabeled (i.e. self-supervised learning) as well as labeled -- selecting pre-training data by label, randomly, and via similarity with fine-tuning data by way of optimal transport. The latter is the most innovative and performs the best. With several key assumptions (the most important of which is the bound on the distance between the fine-tuning and pre-training loss gradients), the paper provides bounds for the excess risk under normal fine-tuning as well as under their proposed modified fine-tuning loss. They compare against a single recent baseline (Co-Tuning) and study image classification on 8 datasets using ResNets pre-trained on ImageNet and do several ablation studies.
strengths_and_weaknesses: Strengths:
* Experiments have decent coverage (self-supervised + supervised pre-training ; limited data; 8 datasets) 
* Pretty solid ablation studies
* Proposed method with UOT selection generally outperforms the other methods.
* Excess risk bound shown seems semi interesting.

Weaknesses:
* Novelty is limited -- while I cannot comment extensively of how noteworthy the theoretical analysis is, the suggested approaches for selecting pre-training data seem straightforward.
* Only co-tuning is compared against (though this method is claimed to be the most competitive recently) and only in the supervised setting. The proposed method with UOT selection loses to co-tuning on 2/8 datasets. Performance is measured as "the best among 3 trials" -- running this for more trials and reporting error bars would make the trends more convincing.
* More experiments would strengthen the paper.
questions: Would it be possible to add at least 1-2 NLP experiments as well? Perhaps the same one used in the Co-Tuning paper?
limitations: The authors do mention limitations in the Conclusion section, which I felt was adequate. However, more discussion around the assumptions made for the theory would be nice -- how strong are the assumptions? Do they hold in practice?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work studies the problem of using data of the pretraining task (pretraining data) for the fine-tuning stage.

They first analyze the excess risk of the target task when the pretrained model and the pretraining data  are used, and show that the proper use of pretraining data can tighten the bound of excess risks. 

For selecting the proper subset of pretraining data, they propose a similarity-based selection method based on the distributional distance between pretraining classes and target data.  

Their empirical results on several image-classification tasks validates the findings of the previous analysis and the several improvements of their method over the conventional fine-tuning without pretraining data.
strengths_and_weaknesses: - Problem setting: The use of selected source data to improve target task has been touched before in the transfer learning literatures. This paper studies a seemly new setting: both pretrained model and pretraining data are used in the fine-tuning task. Though it’s new, my question are how important is this setting and under which scenarios is this setting useful? To be specific, I see the overlap of pretrained model and the pretraining data: If the selected pretrained data is more valuable than the pretrained model, why don’t we train the model from pretraining data and the fine-tuning data? Otherwise, as the purpose of the pretrained model is to provide an initially good (and general) representation, if the pretrained model is well-trained, it already well utilized the pretraining data. Also, as training from scratch and pretraining are usually expensive and intensive, does the use of large pretraining data contradict to the purpose of fine-tuning? (In the experiment, the paper use1 up to 100 classes of ImageNet which is approximately 100K samples while the target data (e.g., Stanford Cars) has much less data.

- Analysis: The lemma and theorem are simple but useful to illustrate the idea of this paper. From my understanding, the main contribution of these analyses is to show the usefulness of proper pretraining data selection, rather than study the defined problem in general.

- Similarity-based method:  The method is intuitive.
    - One concern is how K-mean on high-dimensional data is useful to provide distinguishable clusters (especially with Euclidean distance)? If these clusters are uniformly sampled from the data, the method is likely to be the random selection.
    - Another question I have is the justification of the real effect of pretraining data. Since the paper simply incorporate the pretraining data via extra regularization term (loss) in the fine-tuning. It’s intuitive that this regularization can at least achieve better performance than the original loss (fine-tuning) with proper regularization parameters. And, the experimental results also show the marginal improvements of the modified loss over the standard fine-tuning and sometimes gets lower performance (table 1b).

- Clarify: the paper is pretty clear to follow and easy to understand.

- Quality:
    - Writing can be improved:
        - In paper 2, the statement “From the perspective …” in lines 60—63 is mostly duplicated with lines 64—66
        - Some statements need to be provided with evidences and more precise contexts. For examples, lines 32—33: “…. training from scratch never matches the performance of fine-tuning …. “
        - Minors: there are some typos, e.g., ‘datausing’ (Figure 3 caption), ‘images’ (line 172) should be ‘samples’.

- Experiments
    - Some results seem to be  not representative to me. For example, the Fig.3a shows that 100-class setting provides the best overall performance of method while other settings result in lower performance than the label-based selection. It’ll be better to provide more comparison and robust metrics to the table 1.
    - While table 1 shows the improvement of using proposed method, the improvement is marginal and unclear whether the gain is from the pretraining data or proper parameter tuning?
        - As mentioned above, all considered target tasks have smaller number of data compared to the pretraining data. I think it’s better to show with more settings when the pretraining data is much smaller.
    - Baselines: I suggest the authors to try simple but related baselines: training both pretraining data and target data without pretrained models?
    - Some results in table 1 are far below the best transfer-learning results on the target datasets. For example, on the Stanford Cars, the latest transfer learning may achieve up to 96% accuracy.
        - What happen if we use stronger pretrained models?
    - In figure 3a-2, why don’t we have the label-based line?
    - In figure 3b, what is the accuracy gap?
questions: Please see the questions mentioned above.
limitations: I do not see the limitation and negative societal impact sections. Please see my mentioned limitations above. For the societal impact, the authors may consider the impact of the work on utilizing more pretraining data then standard fine-tuning.
review_title: null

Meta-review: metareview: The paper studies reuse of source data (originally used for pre-training) in the fine-tuning phase. Due to the difference between source and target data, use of the entire source data for fine-tuning can degrade generalization for the target task. However, the paper shows that by carefully choosing a subset of the source data, the generalization performance can exceed what fine-tuning on target data alone can achieve. The scheme used for subset selection is based on unbalanced optimal transport and is theoretically justified via a Theorem in the paper. Empirical results on different datasets show that the proposed scheme indeed adds some gain in generalization.
The authors and reviewers were engaged in active discussion. Reviewers raised interesting questions including, when the source data and really benefit learning the target task, choice of neural architectures, relations to catastrophic forgetting, sensitivity to hyperparameters, usefulness of Euclidean distance for clustering in high dimensions, and practicality of the assumption that both pre-trained model, and its data are available at the fine-tuning time.
Authors provided a thorough answer to these questions. Reviewer 51Vh who was the most skeptical raised their score after the rebuttal. While the paper's final score ends up being in borderline, all the scores are on the accept side. I think the contributions of the paper are interesting enough to be published, and i recommend accept. I encourage authors to incorporate the feedback they received from the reviewers in the final version of the paper.
confidence: Certain
Decision: Accept