Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the effect of the pre-training on transfer learning for image classification. Their study covers 7 pre-training datasets, 6 fine-tuning datasets, 2 pre-training methods for both few-shot and full finetuning. They have made several empirical observations. They find that the pre-training dataset is initially important for low-shot transfer and the importance of a good pre-trained model diminishes when fine-tuning on more data.

strength_and_weaknesses: Strength
- The paper empirically studies the effects of pre-training (data source, pre-training methods, loss function) on downstream tasks. 
- The paper provides fine-tuning results with different pre-training methods (e.g., CLIP based ResNet-50 model), which might be new for practitioners.

Weakness
- The questions asked in the paper were already well studied and the answer is not surprising. The major take-away conclusion of “the effect of pre-training diminishes as more training data are available” is a well known fact and was shown in literature on transfer learning and model selection such as [1].
- The downstream tasks are quite limited (only 6) and some of them have quite saturated performance (such as PETS). It is not quite convincing to draw conclusions with these simple datasets.
- As identified by the authors, there is no explanation or insights about why certain pre-trained models work better than others.
- Only a single architecture (ResNet-50) is studied. Not clear whether the findings generalize to other architectures as well.
- [minor] The authors claimed the best performance on the test was picked with variations of learning rates and batch size, however, the appendix said only a single batch size is used.
 
[1] Deshpande et al, A linearized framework and a new benchmark for model selection for fine-tuning, 2021
[2] Bolya et al, Scalable Diverse Model Selection for Accessible Transfer Learning, 2021

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. The result should be easy to reproduce.
summary_of_the_review: This paper provides empirical observations on the effectiveness of pre-training. However, not much new insights are provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Paper 5991 presents an empirical study to address the question of what data and method should be used for pre-training. The scope is limited in the pre-training distribution for image classification. To be specific, the paper investigates the impact of dataset size / pre- training method / whether the dataset is curated on transfer learning. The main take-home message is that pre-training dataset is initially important for low-shot transfer but the differences between distributions are diminished as more data is made available for fine-tuning.
strength_and_weaknesses: # Strength:
A systematic evaluation of the impact of pre-trained data is presented. My educated guess is that this paper requires extensive experiments with massive computation resource. I suggest that authors clearly state or estimate the rough computation resource used for this paper. I suppose curation of those large-scale datasets also contributes largely to the total labor work. In a word, although no novel methods or ideas are presented in this paper, I appreciate the work of verifying common-sense conclusions using large-scale empirical experiments.

Since the authors mention that "our large-scale experiments yield more than 1000 trained networks", all the trained networks can be released if possible, which can be considered as another contribution, just like the paper "Model Zoos: A Dataset of Diverse Populations of Neural Network Models".

# Weaknesses:
The main weaknesses is that conclusions in this paper seem to be known or trivial. It is expected that pre-training dataset is initially important for low-shot transfer but the differences between distributions are diminished as more data is made available for fine-tuning. There are no additional analyses, e.g. on why "Shutterstock and LAION being the best performing pre-training datasets for different downstream tasks, while WIT yields the worst performance in most cases".

In the discussion section, the authors say that "in the future, a sea of pre-trained models will be available for download from the Internet. Therefore, researchers and practitioners will be faced with the question of where to begin." This should be discussed along with the following papers, where they exactly want to address the problem:
LEEP: A New Measure to Evaluate Transferability of Learned Representations, ICML 2020
LogME: Practical Assessment of Pre-trained Models for Transfer Learning, ICML 2021

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read, with good clarity.
The quality is around the borderline.
The novelty is low.
The reproducibility depends on whether they release the code.

summary_of_the_review: This paper presents some expected conclusions verified by large-scale experiments.


--- Post rebuttal review ---
I have read reviews and responses from authors and peer reviewers. It seems we are all concerned with the novelty of those results. There are so many suggestions in the review to be incorporated into the paper, which I think is beyond the scope of a revision. I encourage the authors to dive into several specific findings rather than presenting many take-away messages without going into them deeply.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the effect of pre-training dataset on transfer learning, when pretraining by text caption contrastive matching using CLIP.  Several different pre-training datasets are compared, using six transfer task datasets (three in later experiments), and in both few-shot and full-data scenarios.  The experiments are extensive, with a fairly rich set of results from which conclusions might be drawn.  The paper enumerates many interesting points in these comparisons, and has a few main takeaways, including: (a) choice of pre-training dataset matters most for low-data scenarios, (b) smaller but cleaner pre-training data can out-perform noisier captions-based data for datasets up to around 15x smaller, (c) more pretraining data helps, but also mostly for low-data transfer tasks.

strength_and_weaknesses: The experiments in this paper are extensive, comparing numerous pretraining datasets with different downstream tasks.  This in itself is a useful contribution, particularly if the authors make public the results as a csv file for further analysis, which I would encourage.

However, I think the analysis in this paper and its presentation could be clearer and have more material conclusions.  The largest points (a-c) in my summary above are not particularly informative, as they describe general trends, which at best corroborate known behaviors.  The observation that pre-training choices matter most (quite a lot, in fact) for few-shot settings is perhaps the most interesting one of these.

On the other hand, the paper enumerates many observations describing interesting behaviors in each of the sections --- so some detailed observations are there, but with few conclusions.  For example, at the end of section 4.1, there are 5 enumerated points, which describe many details but few clear conclusions.  Point 2 says "Pre-training a CLIP model on a 2 times larger yet noisy dataset LAION-1m can outperform the IN1K-Flickr-Captions or yield very similar performance".  This merely points to the data and restates in words what the plots show, which is fine if supporting a larger argument, but it is left stranded.  What exact conclusion can be drawn from this?  Does this difference support or elaborate on the larger trends observed in this section?

In addition, most of the figures, while clear, do not always seem the best plots for making the points in the text.  All are organized with number of shots on the x axis, and datasets as point shape/color.  Another view that could work well could be a colored matrix table showing accuracy values as entries, with pretraining datasets on one axis, transfer dataset on the other axis, and transfer training shots as tiled copies of the plot running along the page --- this would make clearer conclusions like 4.1 first sub-section point (2) which says Shutterstock and LAION are best performing, and could also visualize trends in these orderings as transfer shots increase.  There are likely many other figures that could work, and I think more different views could be included to help make more trends apparent.




I also have a few questions on some of the details, enumerated below:


* since SimCLR is better than CLIP pretraining on the three transfer tasks, this immediately brings up the question of whether the rest of the pretraining dataset comparisons might be explored with SimCLR as well.  At the very least, this might be mentioned in the future work section, as the findings are still relevant for CLIP training.

* page 7 sec 4.1 point 4:  should this say "in1k-clip-captions" instead of flickr?

* fig 3, 4, 5:  why not also compare on pets and domainnet?

* sec 4.3:  Why suddenly switch to LP-FT?  Especially if this makes a difference for simclr vs clip, it would be good to compare with FT only as well.  The rest of the paper used this method, so switching methods appears inconsistent.

* sec 4.3 point 2:  Are there any conclusions or hypotheses about why the best-performing dataset is different between clip and simclr?  Perhaps diversity or relevance of images to the downstream tasks, vs of the captions?

* Figure 4:  only SimCLR results are in these figures, so it's difficult to compare between SimCLR and CLIP --- it would be good to have a figure with both included to facilitate this comparison for sec 4.3

* While the pre-training datasets here do differ in the ways described, such as size or noisiness, they have other differences as well --- for example size of the vocab in the captions, diversity of the objects represented in image data.  Do these affect downstream performance as well, and how might those be controlled for or studied?

clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: Overall, this paper presents extensive experiments comparing different pretraining data sources.  This is an interesting set of results, but in my opinion under-utilized in the analysis and presentation.  More exact overall conclusions might be drawn than those like "more data is better" or "cleaner data is better".  The detailed points in each section might be used to help argue for more tangible conclusions, but they aren't clearly stated yet.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the performance of pre-trained models in downstream classification tasks. 
strength_and_weaknesses: Strength:

The experiment is extensive and adds more pre-training models and datasets compared to previous papers.



Weaknesses:

The paper does experiment with different pre-trained models and datasets in the downtream end, which has already been done in previous papers like [1,2,3]. The paper admits that "we extend these results to more pre-training datasets and methods", which is not a substantial contribution to the community according to my criteria. It is better to propose a novel framework to explain the extensive empirical observation. 

[1] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021

[2] Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko. A broad study of pre-training for domain generalization and adaptation. arXiv preprint arXiv:2203.11819, 2022.

[3] Kornblith, Simon, Jonathon Shlens, and Quoc V. Le. "Do better imagenet models transfer better?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

The phenomenon of more downstream data leads to less benefit of pre-training is studies in [4]. Could the author give a discussion on the connection and difference between [4] and this paper?

[4] Ziquan Liu, Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, Xiangyang Ji, Antoni B. Chan, Rong Jin, "Improved Fine-Tuning by Better Leveraging Pre-Training Data" Neural Information Processing Systems (NeurIPS), 2022
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the quality is fine. There is no significant novelty in this paper. 
summary_of_the_review: I would suggest a weak rejection in this phase.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper empirically studies the effects (pre-training data, training losses, low/high-shot finetuning) in transfer learning for the pre-trained models on different downstream computer vision datasets. They found that pre-training is important to low-shot downstream tasks, and the importance diminishes in high-shot downstream tasks. The main concerns of reviews are that the authors do not clearly present their insights, and the novelty is borderline. I suggest the authors spend more time on their manuscript to further separate their contributions from the prior work.   
justification_for_why_not_higher_score: Majority of reviewers have the same conclusion that this paper is below the acceptance threshold. There is one review increased score due to the authors' rebuttal efforts. However, some major parts remained unsolved. 
justification_for_why_not_lower_score: NA
Decision: Reject