Exploring the Limits of Large Scale Pre-training

1 INTRODUCTION . Recent impressive progress on transfer and few-shot learning ( Brown et al. , 2020 ; Goyal et al. , 2021 ; Kolesnikov et al. , 2019 ; Pham et al. , 2020 ; Dosovitskiy et al. , 2020 ; Dumoulin et al. , 2021 ; Radford et al. , 2021 ) suggests an emerging direction that scaling up models and training them on a huge corpus of data is the main obstacle towards better performance on downstream tasks with less or no data . These developments implicitly encourage two consistent views : 1 ) scaling up the model and data size improves the performance significantly ; 2 ) the performance improvement transfers to downstream tasks in a desirable way . In a more focused empirical study in support of the first view , Kaplan et al . ( 2020 ) show that scaling up the model size , data , and compute appropriately in the language modeling task results in a non-saturating return in performance . Bello et al . ( 2021 ) , Tan & Le ( 2019 ) show that favorable scaling can be achieved in image recognition tasks as well . The second view has also been a subject of recent focused studies . Hernandez et al . ( 2021 ) show that favorable scaling laws similar to that of ( Kaplan et al. , 2020 ; Tay et al. , 2021b ) hold in transfer and few-shot settings in NLP tasks . In perhaps closest prior work to ours , Kornblith et al . ( 2019 ) observe a linear relationship1 between the performances on ImageNet ( Russakovsky et al. , 2015 ) and downstream image recognition tasks . Adopting the above views has major implications moving forward . These views suggest that spending compute and research effort on improving the performance on one massive corpus would pay off because that would enable us to solve many downstream tasks almost for free . It also means while improving our upstream performance , we do not need to be worried about downstream tasks as their improvement is predictable based on a linear trend . While the aforementioned studies provide a compelling story , they suffer from a major shortcoming : due to compute limitations , performance for different choices of hyper-parameter values are not reported . Scaling plots seem more favorable if the hyper-parameter chosen for each scale is fixed or determined by a simple scaling function . 1The linear relationship in ( Kornblith et al. , 2019 ) is achieved after proper logit scaling of accuracy values . We show that with logit or linear scaling , the relationship is not linear . Moreover , often the goal is improving state-of-the-art results , hence naturally most of the efforts in hyper-parameter selection is focused on higher scales , which significantly biases the scaling plots . However , when studying scaling , we are concerned about the best downstream performance of models given all possible values for the hyper-parameters . Additionally , most scaling studies report the behavior within a limited range , and simply extrapolating that scaling without further understanding of the dynamics of scaling can be detrimental as there is no reason , a priori , for the scaling to hold outside of the studied range . In this paper , we systematically investigate the transferability of improvements on a large-scale upstream task to a wide range of downstream tasks in both few-shot and transfer learning scenarios . To address the above shortcomings , part of our work is a meta-study of more than 4800 Vision Transformer ( Dosovitskiy et al. , 2020 ) , MLP-Mixer ( Tolstikhin et al. , 2021 ) and ResNet ( Dosovitskiy et al. , 2020 ) models . The models are pre-trained on either JFT ( Sun et al. , 2017 ) with 303M images and 18K classes or ImageNet21K ( Deng et al. , 2009 ) with 14M images and 21K classes and evaluated on a variety of downstream datasets for few-shot and transfer learning settings . Our 25 downstream tasks cover a wide range of standard datasets that are included in benchmarks like VTAB ( Zhai et al. , 2019 ) , MetaDataset ( Triantafillou et al. , 2019 ) , Wilds ( Koh et al. , 2020 ) and medical imaging . We provide strong empirical evidence that scaling ( and hyper-parameter tuning ) does not lead to a one-model-fits-all solution . There are still many unresolved challenges remaining and at the center is the problem of data diversity for downstream tasks . We provide the first large scale and systematic investigation of this phenomena and discuss the reasons behind it . In Figure 1 , we present downstream ( DS ) vs upstream ( US ) performance plot on variety of models and downstream tasks . We observe that , as we increase US accuracy , for most cases DS accuracy saturates to a value considerably below 100 % . Also , saturating behavior is not an exception but rather the common trend and it is robust to the choice of number of shots and US tasks ( see Figure F.1 ) . We establish that this gap is not due to noise or any other factor that solely depends on DS task ; rather , it depends on the relationship between US , DS tasks . Moreover , given a set of models with similar US accuracy , the best model for different DS tasks varies . Contributions Our main contributions in this paper are as follows : • We establish through extensive study that as we improve the performance of the upstream ( US ) task either by scaling up or hyper-parameter and architectural choices , the performance of downstream ( DS ) tasks shows a saturating behaviour . In our experiments , several DS tasks reach full saturation within the studied range ( Section 2 ) . • We demonstrate that given a set of models with similar US accuracy , the best model for a DS task TDS1 might have much worse performance on another DS task TDS2 compared to the best model for TDS2 ( Figure 5 ) . • Given the scale of experiments , it is crucial for the proposed model to not be impacted by the density of the points in the DS-vs-US plot . We argue and demonstrate that fitting the power law to the convex hull of experiments would circumvent the effect of sampling biases on prediction of downstream accuracy and show the robustness of our model to sample size variations ( Section 2.2 ) . • Having observed the nonlinear relationship between upstream and downstream accuracy , in order to predict downstream performance for a given upstream accuracy , we model their relationship with a power law curve and establish that it captures the behavior well even with small number of samples ( Section 2.2 ) . • We study how scaling up model size , data size , and compute affects DS performance and show that these parameters impact DS performance mainly through the US performance ( Section 3 ) . • We investigate reasons behind the DS performance saturation and show that this behavior can be captured by the usefulness of feature representation in higher layers of the pre-trained model ( Section 4 ) . • We further explore the discrepancy between US and DS performances and show that for some choices of hyper-parameters , they might be at odds with each other . In particular , we showcase how the optimal hyper-parameters for the head used in pre-training ( upstream task ) are different for US and DS . We then uncover the reason behind this discrepancy ( Appendix C , D ) . • Finally , we show our observations are robust to several choices such as size of US data , common scalings of accuracy , number of shots , transfer vs few-shot setting and architecture ( Appendix E ) . Related Work . The closest work to ours is that of Kornblith et al . ( 2019 ) . They investigate the effect of ImageNet ( Russakovsky et al. , 2015 ) pre-training on image classification performance across 12 datasets for few-shot , transfer and random initialization scenarios . They show that performance on ImageNet translates linearly ( in logit scaling ) to performance on DS tasks . However , they do not consider extrapolation of the values . While both works investigate the effect of pre-training via various experiments , there are two main differences in our responses to the question of “ better upstream performance transfer to better downstream performance ? ” . First , we establish that clear “ saturation ” phenomena exists when looking into DS-vs-US performance . In Figure 1 , we see there are various cases when comparing two models , A , B . Where model A has a much higher US accuracy but lower DS accuracy and these are not exceptions to a rule , rather the majority of cases . Essentially , for each DS-vs-US plot two points where one is on the right but lower than the other are instances of such a case . Second , we also establish that , for each DS task you can see best performing models scale with power law as in Equation 1 but for each architecture best performing models are different across DS tasks and this depends on training hyper-parameters , See Figure 5 . In other words , when considering two DS tasks , TDS1 , TDS2 , we have numerous cases where model A has better performance on US and TDS1 but one can not conclude better performance on DS2 . We suspect the difference in conclusion is due to the earlier work being limited in the range of accuracy values they consider . In addition to this difference in conclusions , we investigate the reasons behind this saturation behavior . Moreover , ( in Appendix C ) we consider cases where US and DS performance are at odds with each other , specifically , the scenarios where worse performance on US , leads to performance improvement on DS . Inspired by ( Zhai et al. , 2021 ) who noted that increasing head weight decay during pre-training leads to worse performance on US while improving DS performance ; we investigate head hyper-parameters ( both weight decay and learning rate ) further and show that it can be explained by noting that these manipulations push the information stored in the head down to lower layers . Additional related work are covered in Appendix A . 1.1 EXPERIMENTAL SETUP . The analysis in this paper is based on a study on an exhaustive number of large-scale experiments on image recognition tasks , as well as a set of controlled experiments we conducted to ablate our setup and deepen our understanding of the studied phenomena . We investigate more than 4800 experiments with Vision Transformers , MLP-Mixers and ResNets with different configurations when pre-trained on a large amount of data in a supervised fashion and evaluated on several downstream image recognition tasks through few-shot learning and fine-tuning . For more details , see Appendix G. We emphasize that the large set of experiments we investigate are not trained for the purpose of this paper , rather , we have aggregated models trained by different researchers for different purposes to perform a meta-study on them . This , in fact , positions this meta-study in a unique spot . First , it may not be feasible to run such a number of large-scale trials for the purpose of studying particular phenomena.Second , no implicit or explicit assumption was made in these experiments with respect to the type of analysis we conducted on them afterwards , hence minimizing the systematic biases of the analysis process in the findings . We note that , there might potentially be other biases . For example , researchers usually focus on hyper-parameter tuning to improve SOTA on a specific downstream task ( usually ImageNet ) and this may lead to not do a grid search on high dimensional space of all possible hyper-parameters and possibly affecting the plots . In Section 3 , we investigate this and discuss that in this case the observed trend is similar to performing a grid search . In the main body of the paper , we report the results over eight downstream tasks and provide results for more than 20 downstream tasks in Appendix F. Moreover , the plots corresponding to pre-training on JFT , ImageNet21K are in the main part and Appendix F , respectively .

Summary: The paper presents an empirical study (respectively meta-study) of large-scale supervised pre-training for image recognition tasks. By analysing lots of experiments with varying model sizes, dataset sizes and training durations, the paper reaches the conclusion that simply scaling them up for a generic pre-training task will not lead to proportional gains for downstream tasks that build on the pre-trained model. On the contrary, the paper suggests that with increasing pre-training effort the downstream performance will reach a saturation level below its Bayes error. Moreover, that saturation level appears to vary across different downstream tasks (i.e., image characteristics and class definitions), which is seen as a sign that a generic one-fits-all feature extractor cannot be found by just scaling up.