Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ### What is the Problem / Question?
There is a ongoing trend in ML right now of exploring larger and larger pre-trained (PT) models. In general, increasing the scale of these PT models has yielded better performance on downstream tasks, though this is not universally the case. This work analyzes the limits of this trend in the computer vision space, investigating how downstream task performance is impacted by scale of the PT model among other factors.

### Why is it impactful?
Pre-training large models is very expensive. This study has the potential to clarify when such large-scale PT is well-motivated, and when it isn't, which could yield significant benefits for future research.

### Why is it hard? Why have previous approaches failed?
Previous approaches have examined this question at a dramatically smaller scale than this study does. For example, Kornblith et al., 2019 examined on the order of 10s of models, as opposed to thousands of models as examined here, and the results found here are correspondingly more detailed than those found in the Kornblith et al. study. _Most critically, the authors here present data that strongly suggests that rather than a linear relationship between downstream and upstream task performance, we should instead expect a saturating, non-linear trend_, which suggests that continually training larger and larger models may not be as worthwhile as was traditionally assumed. The fact that this counters an established result in the literature is an important distinction, because my native assumption would've actually been consistent with the author's finding, so without the literature bias already existing, this finding would in my opinion be less impactful. However, because it is updating a held belief in the published results, it is much more important.

### How do they solve / answer it?
The authors profile a variety of models across different architectural choices, hyperparameter settings, etc. and compare upstream vs. downstream accuracy across various downstream tasks. They further examine a variety of additional angles within this study, including:
  1) Quantifying the extent to which downstream performance is consistent across downstream tasks at a given level of upstream task performance,
  2) How the internal representational space of these models can inform the extent of the saturation of the downstream task.
  3) They examine in which cases upstream and downstream task performance may be at odds with one another
  4) They profile their analyses under various dataset sizes, few-shot settings, and architectures.

main_review: ### Key Strengths
  1. This problem is very important, and these anlyses targeted to assess a key part of that problem.
  2. The scale of these experiments is impressive, and their findings robustly motivated by said scale. It is hard to argue with the observed saturation effect in the context of how extensively you've profiled it. Similarly, the results you've found are I think impactful. You've clearly established that we _cannot_ use upstream accuracy as proxy for _downstream accuracy_, and quantified some other insights as well relating to task consistency. These should be impactful in future research, though some revisions could amplify that effect.
  3. I think that this is a meta-analysis makes the work much stronger--one of my biggest concern at first skim of the paper was precisely that your models wouldn't be trained in a manner consistent with other models published during ordinary research, and thus that your results would be less generalizable. By largely relying on pre-published models and results, you totally alleviate that concern and strengthen your findings.
  4. While I think you could do more here, I really like the representational analysis as well. Showing the consistency between saturation speed and layer preference both establishes some intuitive precedent for your findings and helps hint at some strategies we could take in the future to improve these results as well.

### Key Weaknesses
  1. Your biggest issue here is presentation. This presents itself in two ways. Firstly, you are _constantly_ referring to the appendix. If those appendix plots are so essential to your work, you need to find a way to incorporate them into the main body, or look for a different venue without page limitations (e.g., JMLR). Secondly, the flow of this paper somewhat hinders my ability to get to your key takeaways quickly. I like your introduction -- I think it does a good job of setting up the problem and framing what you're going to do, but I feel like you lose that framing as soon as you branch into other sections. I sort of find myself reading the paper by skipping from figure to figure and trying to back out what take-aways the figures represent from the text, when what you want is to guide the reader through your work with your text directly. To address this, I'd recommend reformulating the work slightly to have a more clear-cut structure, and generously sprinkle forwards and backwards pointers throughout your text and contextulaization sections to keep the reader constantly aware of where they are in your analysis and how that contributes to the big picture. You may also need to relegate some of your findings out of the main text, to secondary publications, or drop them entirely if you really want to crystallize your main take-aways for your readers.
  2. Your work already speaks to the fact that for some tasks you can establish a strong power law fit even without using the full dense dataset you use in your experiments. However, in order for this take-away to be proactively useful in general, it would also be valuable to determine to what extent you can fit a power-law saturation curve to a new model with only a limited computational budget for pre-trained models -- e.g., how few runs and over how few UA data-points can you accurately assess the power-law fit? Additional analyses on these fronts (or clear callouts/explanations to your current findings on these fronts) would be very valuable. 

### Minor Weaknesses
  1. I think you probably spend too much time on the randomized classifier section. I don't think those points are central to your paper, and I think this content could therefore be largely relegated to an appendix. 
  2. I think using accuracy is standard in the field, but I'd be further interested in other metrics like AUROC for these analyses too.
  3. You should re-label your legend entry in Figure 1 for the asymptotic (@ UA = 1) performance value. I didn't find it clear from the legend entry alone and your caption for that figure is already very long.


### Other comments
  1. I think it'd be really impactful if you replicated this analysis on NLP models.

summary_of_the_review: While I think addressing both key weaknesses would greatly strengthen the work, I think this is already an impactful and highly rigorous, technically sophisticated study which meets the bar for this venue, so I vote accept here.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper conducts extensive experiments on ViTs, MLP-Mixers and ResNets with up tp 10B parameters and evaluates transferability on more than 20 downstream tasks. It draws the conclusion that as we improve the performance of the upstream task, the performance of downstream tasks shows a saturating behavior.
main_review: Strengths:

1. The paper is well-written and easy to follow.

2. Extensive experiments and analysis have been conducted, which makes it a strong technical report.

3. Besides the saturating behavior, some insights drawn from the observations are good to be heard in the community, such as increasing the data density of upstream tasks, predicting downstream performance for a given upstream accuracy, and etc.

Weaknesses:

1. This paper is just focusing on the task of image recognition. It will be more beneficial to consider complicated visual tasks and other modalities and to investigate whether the same conclusion still holds. 

2. In terms of predicting downstream performance for a given upstream accuracy, can the authors compare the proposed method with those model selection techniques such as LogME [1]? Which technique is more practical for the real-world transfer tasks to select the best pre-trained model? 

[1] You, Kaichao, et al. "Logme: Practical assessment of pre-trained models for transfer learning." International Conference on Machine Learning. PMLR, 2021.
summary_of_the_review: Overall, the reviewer tends to vote for accept since this work makes huge efforts in terms of experiments and some of the conclusions are beneficial to be heard in the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an empirical study (respectively meta-study) of large-scale supervised pre-training for image recognition tasks. By analysing lots of experiments with varying model sizes, dataset sizes and training durations, the paper reaches the conclusion that simply scaling them up for a generic pre-training task will not lead to proportional gains for downstream tasks that build on the pre-trained model. On the contrary, the paper suggests that with increasing pre-training effort the downstream performance will reach a saturation level below its Bayes error. Moreover, that saturation level appears to vary across different downstream tasks (i.e., image characteristics and class definitions), which is seen as a sign that a generic one-fits-all feature extractor cannot be found by just scaling up.
main_review: Strengths:
- the high-level question that the paper asks makes sense, and becomes more important as models and datasets grow larger and larger, so that fewer organisations have the resources to handle them 
- the sheer size of the study makes it interesting, I would think that many of us are doubting whether some of our findings and beliefs from data- and compute-limited studies hold also for truly huge datasets and models beyond our resources
- it is a clevert idea to analyse model performance in terms of a convex envelope, so as to better approximate the upper performance bound for a given model size and "factor out" architecture and hyper-parameter choices, and at least at first glance it makes a lot of sense

Weaknesses:
- concerns remain whether the "hyper-modern" types of models that are the focus of the paper are representative, I would argue that the majority of computer vision users still employ "old-fashioned" convolutional architectures rather than vision transformers or MLP-mixers, 
especially in data- or compute-constrained scenarios
-the narrow focus on low-shot scenarios and on image-level classification is somehow inconsistent with the "real-world, large-scale" aspirations of the paper; let's face it, at the application level outside the research lab, where transfer learning and dataset constraints really matter, one-label-per-image retrieval-type settings are the exception. The effect of large-scale pre-training  for more structured tasks like
semantic segmentation, detection, etc. would be a lot more relevant
- the paper is overly dense  and overloaded. I am fine with an arbitrarily large and detailed appendix, but the paper itself tries to squeeze too many things into 9 pages. Graphs are so small they are hardly readable; explanatory text is somewhat hard to follow and parse, despite the not overly complicated message; countless references to the appendix make it practically impossible to read the main paper without the appendix; etc.
summary_of_the_review: A brute-force, unashamedly empirical study of large-scale pre-training, albeit in a relatively specialised setting (low-shot, mostly with very recent, not yet widely established network types). The truly large size of the study means it will be interesting and potentially helpful for many people - we probably have to be grateful for such empirical studies, as very few would be able to replicate them, even if they are only "aggregated" and not "trained for the purpose of the study" (although it is perhaps a bad sign for the community that a small oligarchy of organizations now are the only ones who have that capacity). It is the sort of paper that does not strike me as particularly brilliant, but whose message will, and should, raise some eyebrows and spark discussions in the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper gives an important finding that overperformed upstream performance may not transfer well to better downstream performance especially in higher upstream accuracies region. The experiments are both empirically and theoretically evaluated, making the conclusion strong and reliable.
main_review: Strengths:
1). The relationship between (US, DS) accuracy is intuitively convincible especially in the high performance regions (Fig. 1), Appendix Table F.1, F.2 further shows that correlation values change drastically for different choices of (US, DS) tasks. Thanks author[s] for the extensive experiments.  

2). The choice of hyperparameters is important for minimizing the discrepancy between US and DS performances. Appendix C and D show an case study of US weight decay in the projection layer of ViT on the effect of downstream DS accuracy, showing that at the expense of hurting US accuracy could instead lead to the improvement of DS tasks. This is counter-intuitive but technically sounds especially for high performing regions.

3). The proposal for improving performance on the breadth of downstream task by adding upstream data diversity to (data size, model parameters, FLOPs) sounds plausible. It would be excellent if author[s] can give a concrete method on how to do it in future. 


Weakness:
1). If there is any case study that showing US accuracy can accurately predict DS accuracy but (model size, US data size, compute) cannot? Compared to Figure 1, which conclusion cannot not be shown from Figure 4?

2). From the study of head weight decay can we conclude that improving the feature extraction performance of lower layers lead to better generalization performance in ViT (see Figure 6, Appendix F.22, F.23)?

3). In practice it costs long time to get a single US accuracy, if there is any proxy/surrogate way to make a cheap estimation? From Figure 1 I can see the curve is saturated only towards the end, which means an early prediction of US accuracy is not a reliable predictor of DS accuracy. In other words, how can we use power law prediction for model selection in practice (see Appendix Fig. F6, F7, F8, F9)?
summary_of_the_review: This paper gives an interesting study on the prediction power of US accuracy. Extensive results are provided. The conclusions are solid.
review_title: null

Meta-review: This paper provides a very large-scale study on the pretraining of image recognition models. Specifically, three scaling factors (model sizes, dataset sizes, and training time) are extensively investigated. One important phenomenon observed by this paper is that stronger upstream accuracy may not necessarily contribute to stronger performance on downstream tasks---actually sometimes these two types of performance could even be at odds with each other

Overall, all the reviewers enjoy reading this paper and highly appreciate the empirical results presented in this paper. There were only a few concerns raised by the reviewers but most were well addressed during the discussion period. All reviewers reach a consensus on accepting this paper and believe this study is worthy to be heard by the community.
Decision: Accept (Spotlight)