Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents an extensive study  on the generalization of ImageNet pre-training. Six real-world datasets spanning from camera traps to satellite imagery are taken into account. The work is motivated by the fact that existing work on visual classifier generalization are often analyzing downstream tasks (e.g. detection) on dataset that are collected with the same strategy as imagenet (web scraping), moreover the datasets on which the transfer learning is performed are often object centric.

The analysis shows that high accuracy on the pre-training dataset not always correlate with high accuracy on real-world tasks for all architectures. Notably some models (E.g. ConvNext) show high correlation.


strength_and_weaknesses: Strenghts

- The study is well formulated and takes into consideration recent neural architectures and a good variety of target datasets
- The use of FID to show difference in dataset distribution is interesting, though the value of FID can be hard to interpret. It could be interesting to just train a linear classifier on each pre-trained representation (before fine-tuning) on the target dataset, this should be a good representative of distribution shift and also more understandable.

Weaknesses
- Unclear if models are downloaded from torchvision with pre-trained weights or re-trained with a fixed set of parameters on Imagenet. This is especially troubling regarding the augmentation strategy that may vary in different pre-trained models (see discussion in ConvNext paper about the pre-training strategy)
- The analysis  could include some more low-level analysis of the architecture behavior. What are the layers that change the most during fine-tuning? How much feature representation shifts from  the pre-trained network and the fine-tuned one?
clarity,_quality,_novelty_and_reproducibility: Paper is clear and well written. See above regarding the pre-training strategy (how are the network pre-trained?) 
The augmentation analysis is conducted on an ablation study, but still it is not clear if pre-trained network are all trained with the same data augmentation strategy.
summary_of_the_review: The paper is interesting and likely useful to the research community.  There are some issues in how experiments are formulated (augmentation) but this can be possibly be clarified in the rebuttal.

Some lower level insights on the learned representation would be interesting to better understand behavior of this models in fine-tuning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work investigates whether models that perform well on the ImageNet dataset also perform as well on real world datasets. By real world datasets they mean datasets that were not scraped off the internet, their investigation includes datasets such as Caltech Camera Traps-20, the Human Protein Atlas Image Classification, EuroSat, APTOS 2019 Blindness, SIIM-ISIC Melanoma, Cassava leave datasets. They extensively fine tune models pre-trained on ImageNet using these datasets and report  that architectures that do well on ImageNet do not necessarily do well on other real world datasets and that the newer models that do well on ImageNet perform worse on these real world datasets than the older ones like VGG Networks.
strength_and_weaknesses: Strengths: 
--The work digs deeper into making the models work for these real world datasets. I really appreciate that the authors focused their efforts on a few datasets and focused their efforts on different model configurations before coming to their conclusions.
--The paper is well written and easy to follow.
--Their findings are some of the things the community has been talking about but had never really been scientifically proven extensively

Weaknesses:
-- I would have wanted to hear more about how your models compare to pre-existing models pretrained on ImageNet and finetuned with the same datasets that you tested in your experiments. Because it is almost like you are disregarding all the work that exists were other researchers have used ImageNet pretrained models(like how you did) on these same real world datasets. Such as: Naushad, R., Kaur, T. and Ghaderpour, E., 2021. Deep transfer learning for land use and land cover classification: A comparative study. Sensors, 21(23), p.8083.
What are your comments on these previous works? Why did you just not take their published recorded accuracies instead of re-doing some of their experiments?
clarity,_quality,_novelty_and_reproducibility: -- In Section 4.1, you mentioned that you removed the background class but you do not mention why. Can you support this decision?
If you are investigating transferability to the real world, surely the background class is very crucial to understanding how well it is performing.
 
-- In section 5.1 , you say "We increase the number of epochs we fine-tune on from 30 to 50 to account for augmentation"- how do you support this statement?
-- In section 6, you say "Asymptotically, it is probably possible to perform well on all web-scraped target datasets simply by collecting a very large amount of data from the Internet and training a very large model on it." Can you back this up by facts?
 
summary_of_the_review: I think the authors did a good job focusing on transferability of these real-world tasks and consolidating the experiments of diverse datasets into one paper. However, I am concerned about the fact that they do not mention other authors who ran some of their experiments on the same real world datasets that they used on some of the same architectures pretrained on ImageNet also. Overall I think this was a well written paper and all of their experiments are clear and sound easy to follow.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates whether models trained on ImageNet with increased accuracy transfer their improved accuracy to multiple datasets. Many models are investigated on a benchmark of six publicly available datasets that try to capture the characteristics practical classification applications. The main finding is that models with higher ImageNet accuracy do not consistently yield performance improvements. 
strength_and_weaknesses: Strengths:
- This paper demonstrates that ImageNet pretrained models do not always lead to better accuracy for certain applications. Thus it highlights that architectural innovations might be overfitting on the ImageNet data highlighting the need for better benchmarks.
- A variety of models exhibiting large imagenet accuracy variance are studied. 
- The datasets under investigation can provide a useful initial benchmark for testing models on a variety of classification applications. 

Weaknesses:
- The choice of the six datasets while practical is not inclusive. For instance, there are a number of additional applications such as robotics (I am thinking indoor classification,), sports, or automotive applications that could also have been added. All of which can be classified as real-world applications that may more closely associate with ImageNet. I believe that the methodology of dataset selection can be made more formal while also including some dataset statistics. 
-  The work fails to also highlight another important of training using pretrained weights which is convergence time. Even though imagenet pretraining alone might not lead to better results, would a model trained from scratch also arrive to the same accuracy?

clarity,_quality,_novelty_and_reproducibility: - Overall, this study does not provide a rapidly new methodology of analysing but some aspects can be seen as novel such as the attempt to predict downstream task accuracy. The authors provide the hyperparameters and experiment details to reproduce the results. 

- In terms of clarity the definition of what is considered a real-world dataset is open to interpretation. Also, it is not unpoppable that images crawled through the web have been uploaded after been taken from “real-world” environments.  This is not a weakness of the particular paper alone but rather a challenge that the community needs to face. 

summary_of_the_review: Overall, the problem tackled in this paper is a challenging one as it attempts to find correlations of real-world tasks with ImageNet trained models. This is difficult to accomplish given that different datasets have different characteristics. But such studies are important, especially to understand the effect that improvements on ImageNet can lead to concrete improvements on real world applications. I fully agree with the final sentence of the paper that it is necessary to have a more diverse set of benchmarks that can improve learning algorithms more comprehensively. Such studies are thus important for the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides experiments suggesting that there is only a very small correlation between performance of a neural classifier on ImageNet and its performance (after fine-tuning) on datasets that are not web-scraped.

strength_and_weaknesses: The numerical experiment seem to be well conducted. Yet, the results of the paper are not very surprising and some claims/conclusions of the authors seem strange or unnecessary. For instance, why this obsession with scrapped VS non-scrapped datasets? Isn't it enough to describe the reason why the 6 chosen datasets have implicitly different underlying distribution?

Why also not mentioning domain adaptation techniques that are beyond fine-tuning?

Some example of affirmation that are doubtful:

"In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from
camera traps or satellites), as opposed to web-scraped benchmarks collected for
comparing models." -> Dataset collected from the web also serve to structuring data in the web. This is a real-world task.

"In particular, we consider classification tasks derived from image data that were specifically collected with the goal of classification in mind. This is in contrast to many standard computer vision datasets – including ImageNet – where the constituent
images were originally collected for a different purpose, posted to the web, and later re-purposed for benchmarking computer vision methods." -> There seems to be a confusion between the intent behind collecting the data and the intent behind the existence of the data. And what was the different purpose in the collection of data for ImageNet if not classification? The notion of intent is very subject to discussions...

"Web-scraped target datasets are by definition within the distribution of data collected from the web, and a sufficiently large model can
learn that distribution. This strategy may not be effective for non-web-scraped datasets, where there
is no guarantee that it is possible to train on data that is close in distribution to the target data, even
if we train on the entire web. Thus, it makes sense to distinguish these two types of datasets." -> While the argument has some logic, I do not find the distinction qualitatively very relevant. There might be a lot to argue. For instance, Camera Traps are a very common object now and many people are already posting it on the web, making the so called web distribution constantly changing.

"If improving ImageNet accuracy alone is insufficient to reach these well-tuned baselines, we can indeed conclude that architecture transfer to this target task is limited." -> Well, this is quite dubious. The improvment the authors perform may lack the power of the training techniques that have been used for these well-tuned baselines. These well-tuned baselines result from specific training approaches that may hypothetically depend on the task at hand, e.g., when employing curriculum learning (which is very much used for real-world applications), and hence not very much fall in the scope of the type of fine-tuning the authors are performing. I would hence be much more careful in their statement.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear, well written and surely reproductible. However, it is not very novel and limited in scope (only picking 6 datasets).

summary_of_the_review: Some affirmations of the paper are dubious, and the novelty is not very important; it is essentially an empirical work. Besides, the conclusion drawn by the paper feels already like common knowledge. For instance, in deep learning for satellite spectral imagery (like the Sentinel-2 example provided by the author), there are already many papers showing how to adapt the convolution specifically for that type of images that have a radically different distribution with respect to more classical images (scrapped or not scrapped...). The authors present some of their statements as if it is something that the community should change. But the truth is that it is already very clear for the practitioners of the various real-world tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper investigates whether models trained on ImageNet with increased accuracy also transfer their improved accuracy to multiple downstream datasets (what they call real-world datasets) via fine tuning. The authors performed extensive experiments testing many models on six publicly available datasets. They also tested some other common practices such as data augmentation in this context. The main finding is that models with higher ImageNet accuracy do not consistently yield performance improvements on the downstream datasets.

Strengths:
1. This paper addresses an interesting and important question: how well does the performance of the model on ImageNet correlate with the performance on the real tasks?
2. The formulation of the study is good, including many recent vision architectures and a variety of target datasets.
3. The paper is well written and easy to follow. 

Weaknesses:
1. The framing (or definition) of the problem as ‘web-scraped vs. real-world (non-scraped)’ is questionable. It is essentially a matter of distribution shift.
2. Related to this, the findings seem somewhat obvious. It is well-known (in particular to practitioners) that distribution shift can cause a significant performance drop and that the performance in the source dataset does not always transfer. 
3. Only the basic fine-tuning is tested and recent progress in transfer learning techniques such as domain adaptation are not considered.
4. The choice of the six datasets is not inclusive enough. The methodology of dataset selection can be made more formal. 
5. With regard to some of the above concerns, there are a little too assertive discussions which could misguide the community. 
justification_for_why_not_higher_score: This paper received split scores of (3,5,8,8), which means two are in favor of acceptance and two are opposed. The reviewers generally acknowledge the motivation and importance of the research, but some concerns are left such as the overall story and argument of the paper, the appropriateness of the experimental design, and the novelty of the results. In particular, one reviewer has raised strong concerns about the framing of the problem, the lack of methodology in terms of transfer learning, and the obviousness of the results, which make sense to the AC as well. It is a valid point that the problem is essentially a distribution shift problem, and it is well known that distribution shifts degrade performance, which is why measures such as domain adaptation are being actively studied. The six data sets selected are also of questionable comprehensiveness to be representative of the real-word tasks.
In the AC’s opinion, papers that, like this one, send a strong high-level message based on empirical results must be highly complete and convincing in terms of logic and supporting results in order not to lead the community in the wrong direction. If the paper cannot convince at least all the reviewers, it would be premature for acceptance. In the discussion between the reviewers and the AC, there was no strong support for the paper, so the AC would like to recommend rejection.
Decision: Reject