Exploring the Limits of Large Scale Pre-training

1 INTRODUCTION . Recent impressive progress on transfer and few-shot learning ( Brown et al. , 2020 ; Goyal et al. , 2021 ; Kolesnikov et al. , 2019 ; Pham et al. , 2020 ; Dosovitskiy et al. , 2020 ; Dumoulin et al. , 2021 ; Radford et al. , 2021 ) suggests an emerging direction that scaling up models and training them on a huge corpus of data is the main obstacle towards better performance on downstream tasks with less or no data . These developments implicitly encourage two consistent views : 1 ) scaling up the model and data size improves the performance significantly ; 2 ) the performance improvement transfers to downstream tasks in a desirable way . In a more focused empirical study in support of the first view , Kaplan et al . ( 2020 ) show that scaling up the model size , data , and compute appropriately in the language modeling task results in a non-saturating return in performance . Bello et al . ( 2021 ) , Tan & Le ( 2019 ) show that favorable scaling can be achieved in image recognition tasks as well . The second view has also been a subject of recent focused studies . Hernandez et al . ( 2021 ) show that favorable scaling laws similar to that of ( Kaplan et al. , 2020 ; Tay et al. , 2021b ) hold in transfer and few-shot settings in NLP tasks . In perhaps closest prior work to ours , Kornblith et al . ( 2019 ) observe a linear relationship1 between the performances on ImageNet ( Russakovsky et al. , 2015 ) and downstream image recognition tasks . Adopting the above views has major implications moving forward . These views suggest that spending compute and research effort on improving the performance on one massive corpus would pay off because that would enable us to solve many downstream tasks almost for free . It also means while improving our upstream performance , we do not need to be worried about downstream tasks as their improvement is predictable based on a linear trend . While the aforementioned studies provide a compelling story , they suffer from a major shortcoming : due to compute limitations , performance for different choices of hyper-parameter values are not reported . Scaling plots seem more favorable if the hyper-parameter chosen for each scale is fixed or determined by a simple scaling function . 1The linear relationship in ( Kornblith et al. , 2019 ) is achieved after proper logit scaling of accuracy values . We show that with logit or linear scaling , the relationship is not linear . Moreover , often the goal is improving state-of-the-art results , hence naturally most of the efforts in hyper-parameter selection is focused on higher scales , which significantly biases the scaling plots . However , when studying scaling , we are concerned about the best downstream performance of models given all possible values for the hyper-parameters . Additionally , most scaling studies report the behavior within a limited range , and simply extrapolating that scaling without further understanding of the dynamics of scaling can be detrimental as there is no reason , a priori , for the scaling to hold outside of the studied range . In this paper , we systematically investigate the transferability of improvements on a large-scale upstream task to a wide range of downstream tasks in both few-shot and transfer learning scenarios . To address the above shortcomings , part of our work is a meta-study of more than 4800 Vision Transformer ( Dosovitskiy et al. , 2020 ) , MLP-Mixer ( Tolstikhin et al. , 2021 ) and ResNet ( Dosovitskiy et al. , 2020 ) models . The models are pre-trained on either JFT ( Sun et al. , 2017 ) with 303M images and 18K classes or ImageNet21K ( Deng et al. , 2009 ) with 14M images and 21K classes and evaluated on a variety of downstream datasets for few-shot and transfer learning settings . Our 25 downstream tasks cover a wide range of standard datasets that are included in benchmarks like VTAB ( Zhai et al. , 2019 ) , MetaDataset ( Triantafillou et al. , 2019 ) , Wilds ( Koh et al. , 2020 ) and medical imaging . We provide strong empirical evidence that scaling ( and hyper-parameter tuning ) does not lead to a one-model-fits-all solution . There are still many unresolved challenges remaining and at the center is the problem of data diversity for downstream tasks . We provide the first large scale and systematic investigation of this phenomena and discuss the reasons behind it . In Figure 1 , we present downstream ( DS ) vs upstream ( US ) performance plot on variety of models and downstream tasks . We observe that , as we increase US accuracy , for most cases DS accuracy saturates to a value considerably below 100 % . Also , saturating behavior is not an exception but rather the common trend and it is robust to the choice of number of shots and US tasks ( see Figure F.1 ) . We establish that this gap is not due to noise or any other factor that solely depends on DS task ; rather , it depends on the relationship between US , DS tasks . Moreover , given a set of models with similar US accuracy , the best model for different DS tasks varies . Contributions Our main contributions in this paper are as follows : • We establish through extensive study that as we improve the performance of the upstream ( US ) task either by scaling up or hyper-parameter and architectural choices , the performance of downstream ( DS ) tasks shows a saturating behaviour . In our experiments , several DS tasks reach full saturation within the studied range ( Section 2 ) . • We demonstrate that given a set of models with similar US accuracy , the best model for a DS task TDS1 might have much worse performance on another DS task TDS2 compared to the best model for TDS2 ( Figure 5 ) . • Given the scale of experiments , it is crucial for the proposed model to not be impacted by the density of the points in the DS-vs-US plot . We argue and demonstrate that fitting the power law to the convex hull of experiments would circumvent the effect of sampling biases on prediction of downstream accuracy and show the robustness of our model to sample size variations ( Section 2.2 ) . • Having observed the nonlinear relationship between upstream and downstream accuracy , in order to predict downstream performance for a given upstream accuracy , we model their relationship with a power law curve and establish that it captures the behavior well even with small number of samples ( Section 2.2 ) . • We study how scaling up model size , data size , and compute affects DS performance and show that these parameters impact DS performance mainly through the US performance ( Section 3 ) . • We investigate reasons behind the DS performance saturation and show that this behavior can be captured by the usefulness of feature representation in higher layers of the pre-trained model ( Section 4 ) . • We further explore the discrepancy between US and DS performances and show that for some choices of hyper-parameters , they might be at odds with each other . In particular , we showcase how the optimal hyper-parameters for the head used in pre-training ( upstream task ) are different for US and DS . We then uncover the reason behind this discrepancy ( Appendix C , D ) . • Finally , we show our observations are robust to several choices such as size of US data , common scalings of accuracy , number of shots , transfer vs few-shot setting and architecture ( Appendix E ) . Related Work . The closest work to ours is that of Kornblith et al . ( 2019 ) . They investigate the effect of ImageNet ( Russakovsky et al. , 2015 ) pre-training on image classification performance across 12 datasets for few-shot , transfer and random initialization scenarios . They show that performance on ImageNet translates linearly ( in logit scaling ) to performance on DS tasks . However , they do not consider extrapolation of the values . While both works investigate the effect of pre-training via various experiments , there are two main differences in our responses to the question of “ better upstream performance transfer to better downstream performance ? ” . First , we establish that clear “ saturation ” phenomena exists when looking into DS-vs-US performance . In Figure 1 , we see there are various cases when comparing two models , A , B . Where model A has a much higher US accuracy but lower DS accuracy and these are not exceptions to a rule , rather the majority of cases . Essentially , for each DS-vs-US plot two points where one is on the right but lower than the other are instances of such a case . Second , we also establish that , for each DS task you can see best performing models scale with power law as in Equation 1 but for each architecture best performing models are different across DS tasks and this depends on training hyper-parameters , See Figure 5 . In other words , when considering two DS tasks , TDS1 , TDS2 , we have numerous cases where model A has better performance on US and TDS1 but one can not conclude better performance on DS2 . We suspect the difference in conclusion is due to the earlier work being limited in the range of accuracy values they consider . In addition to this difference in conclusions , we investigate the reasons behind this saturation behavior . Moreover , ( in Appendix C ) we consider cases where US and DS performance are at odds with each other , specifically , the scenarios where worse performance on US , leads to performance improvement on DS . Inspired by ( Zhai et al. , 2021 ) who noted that increasing head weight decay during pre-training leads to worse performance on US while improving DS performance ; we investigate head hyper-parameters ( both weight decay and learning rate ) further and show that it can be explained by noting that these manipulations push the information stored in the head down to lower layers . Additional related work are covered in Appendix A . 1.1 EXPERIMENTAL SETUP . The analysis in this paper is based on a study on an exhaustive number of large-scale experiments on image recognition tasks , as well as a set of controlled experiments we conducted to ablate our setup and deepen our understanding of the studied phenomena . We investigate more than 4800 experiments with Vision Transformers , MLP-Mixers and ResNets with different configurations when pre-trained on a large amount of data in a supervised fashion and evaluated on several downstream image recognition tasks through few-shot learning and fine-tuning . For more details , see Appendix G. We emphasize that the large set of experiments we investigate are not trained for the purpose of this paper , rather , we have aggregated models trained by different researchers for different purposes to perform a meta-study on them . This , in fact , positions this meta-study in a unique spot . First , it may not be feasible to run such a number of large-scale trials for the purpose of studying particular phenomena.Second , no implicit or explicit assumption was made in these experiments with respect to the type of analysis we conducted on them afterwards , hence minimizing the systematic biases of the analysis process in the findings . We note that , there might potentially be other biases . For example , researchers usually focus on hyper-parameter tuning to improve SOTA on a specific downstream task ( usually ImageNet ) and this may lead to not do a grid search on high dimensional space of all possible hyper-parameters and possibly affecting the plots . In Section 3 , we investigate this and discuss that in this case the observed trend is similar to performing a grid search . In the main body of the paper , we report the results over eight downstream tasks and provide results for more than 20 downstream tasks in Appendix F. Moreover , the plots corresponding to pre-training on JFT , ImageNet21K are in the main part and Appendix F , respectively .

Summary: ### What is the Problem / Question? There is a ongoing trend in ML right now of exploring larger and larger pre-trained (PT) models. In general, increasing the scale of these PT models has yielded better performance on downstream tasks, though this is not universally the case. This work analyzes the limits of this trend in the computer vision space, investigating how downstream task performance is impacted by scale of the PT model among other factors.  ### Why is it impactful? Pre-training large models is very expensive. This study has the potential to clarify when such large-scale PT is well-motivated, and when it isn't, which could yield significant benefits for future research.  ### Why is it hard? Why have previous approaches failed? Previous approaches have examined this question at a dramatically smaller scale than this study does. For example, Kornblith et al., 2019 examined on the order of 10s of models, as opposed to thousands of models as examined here, and the results found here are correspondingly more detailed than those found in the Kornblith et al. study. _Most critically, the authors here present data that strongly suggests that rather than a linear relationship between downstream and upstream task performance, we should instead expect a saturating, non-linear trend_, which suggests that continually training larger and larger models may not be as worthwhile as was traditionally assumed. The fact that this counters an established result in the literature is an important distinction, because my native assumption would've actually been consistent with the author's finding, so without the literature bias already existing, this finding would in my opinion be less impactful. However, because it is updating a held belief in the published results, it is much more important.  ### How do they solve / answer it? The authors profile a variety of models across different architectural choices, hyperparameter settings, etc. and compare upstream vs. downstream accuracy across various downstream tasks. They further examine a variety of additional angles within this study, including:   1) Quantifying the extent to which downstream performance is consistent across downstream tasks at a given level of upstream task performance,   2) How the internal representational space of these models can inform the extent of the saturation of the downstream task.   3) They examine in which cases upstream and downstream task performance may be at odds with one another   4) They profile their analyses under various dataset sizes, few-shot settings, and architectures.