Exploring the Limits of Large Scale Pre-training
Authors: Samira Abnar
Mostafa Dehghani
Behnam Neyshabur
Hanie Sedghi
Venue: ICLR 2022 Spotlight (2022)
Keywords: Scaling law
Pre-training
Transfer learning
Large Scale
Vision Transformer
Few Shot
Empirical Investigation

Abstract: Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might  observe that improvements in pre-training would transfer favorably to  most downstream tasks. In this work we systematically study this phenomena and establish that, as we increase the upstream accuracy, performance of downstream tasks \emph{saturates}. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance  that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the observed saturation behavior is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, in order to have a better downstream performance, we need to hurt upstream accuracy.

Published as a conference paper at ICLR 2022
EXPLORING THE LIMITS OF LARGE SCALE TRAINIIG
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi Google Research (samiraabnar dehghani, dehghani, neyshabur, hsedghi]@goooglee
ABSTRACT
Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work we systematically study this phenomena and establish that, as we increase the upstream accuracy, performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet2 IK) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper understand the reasons that give rise to these phenomena, we show that the observed saturation behavior is closely related to the way that representations evolve through the layers of the models. We showcase even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, in order to have better downstream performance, we need hurt upstream accuracy.
INTRODUCTION
Recent impressive progress on transfer and few-shot learning (Brown al. 2020 Goyal et al. 2021
Kolesnikov et al., 2019: Pham et al. 2020 Dosovitskiy et al. 2020 Dumoulin et al. 2021 Rad
ct 2021 suggests an emerging direction that scaling up models and training them on a huge corpus of data is the main obstacle towards better performance on downstream tasks with less no data. These developments implicitly encourage two consistent views: 1) scaling up the model and data size improves the performance signíficantly; 2) the performance improvement transfers to downstream tasks in a desirable way. In more focused empirical study in support of the first view, Kaplan et al. (2020) show that scaling up the model size, data, and compute appropriately in the language modeling task results in non-saturating return in performance. Bello et al. (2021 Tan &
Le (019 show that favorable scaling can be achieved in image recogmition tasks as well. The second view has also been subject of recent focused studies. Hernandez et al. (2021) show that favorable scaling laws similar to that of (Kaplan et al., 2020 TTy et al. 202 hold transfer and few-shot settings in NLP tasks. In perhaps closest prior work to ours, Kornblith et al, (2019) observe linear relationship between the performances on ImageNet (Russakovsky et al. 2015 and downstream image recognition tasks.
Adopting the above views has major implications moving forward. These views suggest that spending massive corpus would pay off
compute and research effort improving the performance because that would enable us to solve many downstream tasks almost for free. It also means while improving our upstream performance, we do not need to be worried about downstream tasks their improvement is predictable based on a linear trend. While the aforementioned studies provide a compelling story, they suffer from a major shortcoming: due to compute limitations, performance for different choices of hyper-parameter values are not reported. Scaling plots seem more favorable
'The linear relationship in Korbllth et al. 2019 is achieved after proper logit scaling of accuracy values. We show that with logit or linear scaling, the relationship is not linear.

---

Published as a conference paper at ICLR 2022
Noldee MLP Miers
1
Comvea Full
126
(atech
L+4
5.3
1:0
Figure 1: The performance of different downstream (DS) tasks of upstream (US) based on more than 1500 different Vision Transformers, 1400 MLP-Mixee and 16 best performing ResNets (Although the number of ResNet samples are small, this does not hurt our investigations. See Appendix G.I for details), with different contigurations. The models are pre-trained on JFT and evaluated in few-shot settings (25 shots). Figure(F] in Appendix shows the same plot but with more than 4800 experiments including two different upstream tasks of JFT, ImageNet21K and 1, 25 shots. We consider the convex hull of the points as well since it captures the performance of a randomized classifier made by choosing these models with different probabilities. As the upstream performance improves, the downstream performance starts to saturate. Even US accuracy reaches 100% accuracy, the DS accuracy will not reach the 100% accuracy and saturates at a lower value. We observe a non-linear relationship between upstream and downstream accuracy and model the relationship with a power law function to predict the DS performance given the US performance. The horizontal linc is the predicted downstream accuracy if upstream accuracy reaches 100%. We investigate DS-vs vs- US plots instead of the usual DS-vs-scale plots to capture the effect of hyper-paane parameter choices and to account for the fact that the scaling impacts DS performance through US performance. Figure Figure[F.2]eppaa depiccs the same plot with log scaling of accuracies done in many related works. Figure F.3 depiccss the same plot when upstream is ImageNet21 K.
if the hyper-parameter chosen for each scale is fixed or determined by simple scaling function. Moreover, often the goal is improving state-of-the-art results, hence naturally most of the efforts in ryper-parameter selection is focused on higher scales, which significantly biases the scaling plots. However, when studying scaling, we are concerned about the best downstream performance of models Additionally, most scaling studies report the
given all possible values for the hyper-parameted behavior within a limited range, and simply extrapolating that scaling without further understanding of the dynamics of scaling can be detrimental as there is no reason, priori, for the scaling to hold outside of the studied range.
In this paper, we systematically investigate the transferability of improvements on large-scale upstream task to a wide range of downstream tasks in both few-shot and transfer learning scenarios. To address the above shortcomings, part of our work is meta- study of more than 4800 Vision Transformer (Dosovitskiy et al. 2020), MLP-Mixer (Tolstikhin et al., (02T)) and ResNet (Dosovitskiy et al. 2020) models. The models are pre-trained on either JFT (Sun al. 2017) with 303M images and 18K classes or ImageNet21 K (Deng et al. (2099) with 14M images and 21K classes and evaluated on a variety of downstream datasets for few-shot and transfer learning settings. Our 25 downstream lasks cover a wide range of standard datasets that are included in benchmarks like VTAB (Zhai et al. 2019) MetaDataset (Triantafillou al. 2019), Wilds (Koh et al. 2020) and medical imaging.
We provide strong empirical evidence that scaling (and hyper-paramettt tuning) does not lead to a mne-model-fit--ll solution. There are still many unresolved challenges remaining and at the center is the problem of data diversity for downstream tasks. We provide the first large scale and systematic investigation of this phenomena and discuss the reasons behind it. In Figure I we present downstream (DS) vs upstream (US) performance plot on variety of models and downstream tasks. We observe that, as we increase US accuracy, for most cases DS accuracy saturates to a value considerably below 100%. Also, saturating behavior is not exception but rather the common trend and it is robust to the choice of number of shots and US tasks (see Figure F.I). We establish that this gap is not due to noise or any other factor that solely depends on DS task; rather, it depends the relationship between US DS tasks. Moreover, given a set of models with similar US accuracy, the best model for different DS tasks varies.
2

---

Published as a conference paper at ICLR 2022
Contributions Our main contributions in this paper are follows:
We establish through extensive study that as we improve the performance of the upstream (US) task either by scaling up or hyper-parametee and architectural choices, the performance of downstream (DS) tasks shows a saturating behaviour. In our experiments, several DS tasks reach full saturation within the studied range (Section 2).
We demonstrate that given a set of models with similar US accuracy, the best model for a DS task TDs, might have much worse performance on another DS task T TDS, compared the best model for TDS, (Figure
Given the scale of experiments, density of the points in the DS-vs-US plot. We argue and demonstrate that fitting the power law to the convex hull of experiments would cireumvent the effect of sampling biases on prediction of downstream accuracy and show the robustness of our model to sample size variations (Section (2.2). Having observed the nonlinear relationship between upstream and downstream accuracy, order to predict downstream performance for given upstream accuracy, we model their relationship with a power law curve and establish that it captures the behavior well even with small number of samples (Section 2.2)
crucial for the proposed model to not be impacted by the
We study how scaling up model size, data size, and compute affects DS performance and show that these parameters impact DS performance mainly through the US performance (Section 33 We investigate reasons behind the DS performance saturation and show that this behavior can be captured by the usefulness of feature representation in higher layers of the pre-trained model (Section 4
We further explore the discrepancy between US and DS performances and show that for some choices of pappe-pprameters, they might be at odds with each other. In particular, we showcase how the optimal
papae-prraeeters for the head used in pre-training (upstream task) are different
for US and DS. We then uncover the reason behind this discrepancy (Appendix D).
Finally, we show our observations are robust to several choices such as size of US data, common scalings of accuracy, number of shots, transfer few-shot setting and architecture (Appendix E
Related Work. The elosest work ours is that of Korblithh al. (2019). They investigate the effect of ImageNet (Russakovsky et al.] (013) pre-training on image classification performance across 12 datasets for few-shot, transfer and random initialization scenarios. They show that performance on ImageNet translates linearly (in logit scaling) to performance on DS tasks. However, they do not consider extrapolation of the values. While both works investigate the effect of pre-training via various experiments, there are two main differences in our responses to the question of "better upstream performance transfer to better downstream performance?". First, we establish that clear "saturation" phenomena exists when looking into DS-vs-US performance. In Figure I we see there are various cases when comparing two models, A, B. Where model A has much higher US accuracy but lower DS accuracy and these not exceptions for each DS-vs-US plot two points where one is on the right but lower than the other are instances of such a case. Second, also establish that, for each DS task you can best performing models scale with power law as in Equation i butt for each architecture best performing models are different across DS tasks (Dehghani et 2021c) and this depends on training byper-parameters, See Figure In other words, when considering two DS tasks, T TDs, we have numerous cases where model has better performance on US and TDS, but one cannot conclude better performance on DS2- We suspect the difference in conclusion is due to the earlier work being limited in the range of accuracy values they consider. In addition to this difference in conclusions, we investigate the reasons behind this saturation behavior. Moreover, (in Appendix [ we consider cases where US and DS performance are at odds with each other, specifically, the scenarios where worse performance on US, leads to performance improvement on DS. Inspired by (Zhai et al. 2021) who noted that increasing head weight decay during pre-training leads to worse performance on US while improving DS performance; we investigate head seper-parameters (both weight decay and learning rate) further and show that it can be explained by noting that these manipulations push the information stored in the head down lower layers. Additional related work are covered in Appendix A
rule, rather the majority of cases. Essentially,
1.1 EXPERIMENTAL SETUP
The analysis in this paper is based on a study on an exhaustive number of large-scale experiments on image recognition tasks, as well as a set of controlled experiments we conducted to ablate our
3

---

Published as a conference paper at ICLR 2022
setup and deepen our understanding of the studied phenomena. We investigate more than 4800 experiments with Vision Transformers, MLP- M-MMMer and ResNets with different configurations when pre-trained on large amount of data in supervised fashion and evaluated on several downstream image recognition tasks through few-shot learning and fine-tuning. For more details, see Appendix
We emphasize that the large set of experiments investigate paper, rather, we have aggregated models trained by different researchers for different purposes to perform meta-study
trained for the purpose of this
them. This, in fact, positions this meta-study in unique spot. First, may not be feasible to run such number of large-scale trials for the purpose of studying particular Saanomena.Seccnne no implicit or explicit assumption was made in these experiments with respect to the type of amalysis we conducted on them afterwards, hence minimizing the systematic biases of the analysis process the findings. We note that, there might potentially be other biases. For example, researchers usually focus on hyper-parameter tuning to improve SOTA task (usually ImageNet) and this may lead not do grid search on high dimensional space of all possible hyper-parameters and possibly affecting the plots. In Section we investigate this and discuss that this case the observed trend is similar to performing a grid search.
specific downstream
In the main body of the paper, we report the results over eight downstream tasks and provide results Moreover, the plots corresponding to pre-training
for more than 20 downstream tasks in Appendix(F F on JFT, ImageNet2 IK are in the main part and Appendix F respectively.
THE DIMINISHING BENEFIT OF SCALING UP IN TRANSFER LEARNING
The prominent goal of transfer learning is to have good performance on downstream tasks. The first question we address is how performance improvement on the upstream task impacts performance on different downstream tasks. We are interested in modeling this effect to facilitate prediction of downstream performance.
2.1 RECAP: RANDOMIZED CLASSIFIERS
Before diving deep into the DS-vs-US performance plots, we recap the concept of randomized classifier since we will be using it extensively throughout this section.
Given two classifiers with US and DS accuracy 01 (al's " a..), a₂ (a a2 D)), one can make a randomized classifier by picking the output of the first classifier with probability Pı and the output of the second classifier with probability the randomized classifier will be p1 @1 + (1 Pi )az That is, the randomized classifier's accuracy is the convex combination of accuracy of the two classifiers. By sweeping the value of Pt. all the points on this convex combination path can be achieved. We extend this notion to the case of more than two classifiers. As the next lemma states, the accuracy of such a randomized classifier would a convex combination of accuracies of its endpoints.
Pı for each input independently. Then the accuracy of
Lemma 2.1. Consider a group of models fj. E [N] that reach accuracy aj (e05s,a55) [N] on some pair of tasks DUS. Construct randomized model Ô as follows: for euch input Ia with probability P5 pick model 0, and output (z(x)). Then the randomized model will demonstrate
accuracy ビ Σ Pjaj.
For proof, see Appendix
Therefore, all the points the convex hull of DS-vs-US accuracy of the trained models achievable and we have the aforementioned method to reach them. This leads to a randomized classifier that shows the accuracy equivalent the convex hull of performances of trained classifiers hand.
Based on the above discussions, in addition to the points corresponding to experiment results, we include the upper hull of the convex hull (representing the highest DS accuracy for every given US accuracy) of the model performances in our analysis. This provides us with a model of DS-vs-US relationship that is robust to density of the points in the plots. We discuss this further in Section 2.2
4

---

Published as a conference paper at ICLR 2022
2.2 SCALING LAWS FOR DOWNSTREAM ACCURACY
Figure I shows DS-vs-US performance for more than 3K experiments where different models are pre-trained JFT and evaluated set of DS tasks in the few-shot setting (25 shots). Figure [F.T in Appendix F depicts similar plot with all the 4800 experiments (pre-trained on JFT or ImageNet2 and for 1, 25 shots). As mentioned in Section L. these models vary in terms of model size and shape, optimization method, compute and other syyper-araameter..
We are interested in predicting how the performance of a DS task will change if US performance improves by investigating the performance of existing models. To do so, we fit curve to the DS-va vs- US performance plot. We emphasize that our analysis differs from earlier works that analyze scaling law (Kaplan et al. 2020; Hernandez et al. 2021 Zhai et al. (021) in that it analyzes DS accuracy vs US accuracy, instead DS accuracy vs dataset size, model size or compute, Since for the most part performance improvement on US is achieved by scaling (dataset size, model size, compute), this approach indirectly captures the impact scaling. We support this argument in Scction
When studying DS-vs-US choosing the right scaling is important. Kornblith et al. (2019) investi- gate DS-vs-US curve for models that are pre-trained on ImageNet and report a linear DS- DS-vs-US performance when plotting the accuracies in the logit scaling. In Figure F2 we depict the same experiments to that of Figure but with logit scaling and we note a nonlinear relationship between DS and US accuracies. Recht al. (2018, 2019) also use logit scaling for investigating relationship between DS and US performance. However, logit scaling shows symmetric behavior around error 0.5, which is not natural for these problems. Therefore, we argue that log scaling which is used in scaling law literature is more appropriate. A linear relationship between US and DS performance in log scaling can be captured follows:
eDs = a(eus))
we note that the behavior not linear. Rather, the performance of DS task saturates at some point and that point varies for different DS tasks.
Looking at Figure 1.
Performance Saturation: We define the saturation point inspired by the observations in Figure and Figure [.1 Then, we mathematically model and investigate saturation value.
Definition 2.2 (Saturation value). Considering downstream vs upstream accuracy, for downstream task Ts saturation value is defined as the value of downstream accuracy as upstream accuracy reaches 1.0
performance saturation also means that there exists a US accuracy value, beyond which the performance improvement on DS is very small and hence it is not worth scaling up data size, compute or model size to improve US accuracy. Since the relationship is not linear, in order to predict DS performance. we need function form to fit the plot. Inspired by recent work on scaling law (Kaplan et al. 2020 Hernandez et al. 2021), we propose the following function form:
Considering Definition 2.2
eDs = =(Euss)* + EIR,
(I)
where eps, evs refer to the error (1- accuracy) of downstream and upstream respectively, k; a are constants and eIR the irreducible error. Irreducible error, <RR captures the value of DS error if US error reaches zero and hence acts similar to a bias term. éIR term captures the nonlinearity trend between US DS accuracies. Meaning that if we plot Equation in log scaling, the dependencies are linear only when eJR is zero.
We sketch the line corresponding to 1- eIR in DS-vs-US accuracy plots of Figure and note that it is not close to 1.0 for many DS tasks and better US performance does not transfer to better DS performance in higher US accuracies. We observe that unlike the common belief, the saturating behavior is not an exception, but typical among DS tasks.
Effect of design choices on power law parameters: we can see Figure F. have different saturating values and this value changes as US task changes. Moreover, eiR changes when we vary the number of shots. To depict this observations more clearly, we plot how different
different DS tasks
2 Mooe precisely, saturation value is the value of DS accuracy when US accuracy reaches its Bayes error. This can be captured by replacing eus with (eys eUS- Bass-Bayesrror in Equation [ loss of generality we do not account for upstream Bayes error in the discussions.
For simplicity and without
5

---

Published as a conference paper at ICLR 2022
3F
2:20
Shd
Sommoee 5iz
Sumple
The effect of sample size on power law curves. The curves are fitted to the convex hull of ex- periments as well as all data points from Figure [.T Prediction error is the difference between power law prediction and observed value of the DS accuracy. Fit- ting error is the difference of power law values from the points that are used in fitting power law parameters. For de moreedetaiiss,se see A Appendix
Figure 3:
Figure 2: Effect number shots and DS/US task on ej of the power law curves. We note that all of them impact eix- To see this effect for all power law parameters see Figures F.4 F5
choices affect the parameters of the power law (Eqquatio in Figures [.4 and It can be seen that the choice of US and DS task affects all parameters, while number of shots mostly impacts and eIR. Specifically, increasing the number of shots results in lower EIR In short, there exists some functions f.), f2)), and fa(.) such that for a specific choice of model and training algorithm, we have
a = (TUs,
k f2 TDS, d), EIR = TDs,
(2)
where df refers to the number of shots in the few-shot setting.
To shed more light into this, we look into correlation of k. a, and eIR with number of shots for different DS, US tasks in Table F.1 For all US and DS choices, and enR correlate negatively with number of shots, while is positively correlated with number of shots. However, correlation values change drastically for different choices of US, DS tasks. In addition, we look into the trend of each of these parameters as we increase the number of shots and present the likelihood of binary correlation in Table F2 Both these tables capture similar phenomena.
Irreducible error is not due to DS Bayes error: We argue that irreducible error, CIR, is not the Bayes error for DS task. Bayes error for a task refers to the error that is intrinsic to the definition of the task. More specifically, the Bayes error captures whether the classification labels are deterministic, i.e., there is non-zero zero probability of a given instance belonging to more than one class. However, as can be seen in Figure F.4, for each DS task, eiR changes significantly by changing the number of shots and choice of US task. Therefore, cIR is not merely due to the Bayes error of DS task, but is also influenced by number of available samples from DS and the difference between US and DS tasks.
Choice of data for fitting the power law: As can be seen in Figure D there is a large variance in DS-vs-US performance across models. When considering the scaling law of the trained models, earlier works fit a scaling curve to all the existing points. We propose another option. To calculate the convex hull of all trained models and fit a scaling curve to the convex hull. The former essentially fits the scaling curve to average model performance. The latter has the advantage of fitting a scaling curve to the best performing models. The location of the points in the DS-vs-US -US plot significantly impact the average model and hence the power law prediction. Whereas, convex hull of points is not affected by the locality of higher density points. A good performing model directly impacts the convex hull with no need to having many such samples. Therefore, we expect the average model to give an incomplete picture of the performance behavior. As we see below, fitting the convex hull more robust to small sample size. Figure [:6 and F.7, depict the power law (Equation ! curves corresponding to these two choices respectively. We plot the predictions from the power law curve on the higher US accuracies as well as the ground truth (prediction target) and observe that power law curve closely predicts the performance of DS. See Figure F.8
F.9
for 1 and 25 shot setting.
Sample size sensitivity analysis: We also investigate the robustness of this fit when we change the number of samples, in terms of error encountered when predicting the plot for higher US accuracies as well as the error in fitting the data. We use the points from higher US accuracies as held out data. Prediction error captures the difference between power law prediction and observed value of the DS accuracy. Fitting error captures the difference between power law values and the points that are used in calculating power law parameters. We plot fitting error and prediction error as the number of samples changes. Figures 33 F.10 summarize these errors when fitting the power law curve to the convex hull of DS-vs-US plot, and all data points. For more details, see Appendix F.
6

---

Published as a conference paper at ICLR 2022
US Acsuncy
15 Accu36)
U/ Accunso
U5 Acouncy
5.00088 cospeeillacci
CIREPT
US AcAAR
US5
U/
US Acsunsy
Figure 4: Controlled scale up experiments of the model size (number of parameters), data size (portion of the pre-trained data), and compute (number of epochs) on different downstream tasks and JFT as upstream. We observe similar trends to Figure [ (1) As increase US accuracy, DS performance saturates. (2) Increasing model size, data size, and compute lead to same curve. (3) The variation from curve due to training
(4) US accuracy has a stronger predictive power for DS accuracy compared to model size, US data size, and compute.
and Figures F.. F.I FF The prediction error is very small across all these choices, which means the proposed model will work well even with smaller number of DS-vs-US samples (trained models). As expected, the fitting error decreases by increasing the number of samples. Note that the prediction error an order of magnitude lower if fit the power law the convex hull vs all samples.
3 EFFECT OF SCALE: A CLOSER LOOK
In this section, we perform a set of controlled experiments where we increase data size, model size, number of epochs. Figure Eegure4depicts depicts how DS-vs-US accuracy changes as we increase US dataset size (from 2% to 100% of JFT), number of model parameters (ViT-Tiny. ViT-Small, Vit-Base, ViT-Large) and number of epochs (7, 14, and 21 epochs). See Figure F.17 for 25 DS tasks. Since we are in the paader-prrmmetrized regime and far from saturating on JFT, the effect of increasing data size is equivalent to increasing training time and the performance on the US improves as we increase training time (Nakkiran et al. 2020). In order to facilitate comparison with earlier experiments, in Figure we overlay the new points (shown in color) to that of Figure !
(shown in grey).
We observe that the controlled experiments in Figure 4 show similar trends to that of Figure [] Figure/F.I] i.e., the DS-vs-US accuracy presents different trends for different DS tasks when scaling up dataset size, model size and number of epochs. For some DS tasks the performance saturates quicker, for instance, colorectal histology (col_ hist) and UC-Merced land use dataset. Furthermore, similar to Figure [ for some of the DS tasks, the benefit of scaling up diminishes gradually, for instance for Cars KKause et al. 2013) or Caltech101 (Li et al.) 2004).
Similar trend:
Grid search equivalence: Effect of model size on improving both US, DS accuracy is more pro- nounced compared to data size and number of epochs. However, we note that if we keep any two of the three parameters fixed and increase the third one, the points reside on the same curve. In Figure the effect of changing data size and number of epochs are on the same curve to that of changing the model size. Therefore, we can trust that even if we did a grid search on all these parameters, Figure would still present the same picture.
On the prediction power of US accuracy: The above observations show that the effect of each of the three parameters (model size, US data size, compute) on DS accuracy is only through US accuracy. That means, conditioned on US accuracy, none these three parameters provide extra information on DS accuracy. To depict this further, we evaluate the effectiveness of using US accuracy predict DS accuracy as follows. Since we have a single value prediction, we consider our prediction based on fitting the power-law of Equation| and compare DS performance. (guure/F.18pl.. the error as well as the power law prediction plot for all DS tasks considered in this paper. In addition, calculate the standard deviation of error (difference between
using average DS accuracy for predicting

---

Published as a conference paper at ICLR 2022
8-
US Acousty
Accuncyy
US5 Acourny
Accuncy
Figure 5: The overlay of convex hull of ImageNet (Red) DS-vs-US plot on DS-vs-US plots of all DS tasks from Figure BBlue) US task is JFT. We observe that best performing ImageNet models perform very similar to best performing models in several DS tasks but not all of them. Moreaver, as the US performance increases, the gap between best performing ImageNet models and best performing DS task models reduces significantly.
Equation Is prediction of DS accuracy and the value of DS accuracy) and report in Table FS We note that standard deviation of the error is much smaller than (which is the standard deviation we would get if we used average as prediction value). This shows that US accuracy has strong prediction power for DS accuracy and conditioned on US accuracy, there is not much left for the rest of parameters (model size, data size, compute) altogether to predict the DS accuracy. This further confirms our choice of parameter to rely on for predicting DS accuracy.
On the role of hyper-parameteryy Moreover, contrary to (Hernandez et al., 2021), these three parameters (data size, model size, number of epochs) are not the only ones that impact the DS accuracy results. When do controlled experiments these three parameters, the points end up in the same curve. The variations observed Figure [ are due to different architecture and choices of training hyper-parametee
and algorithms. The variations caused by the effect of dlyee-paraameter lead to the points not residing on the same curve in Figure [ We observe a distance on the points corresponding controlled experiments from the convex hull (best performing models). For example, for ImageNet, controlled experiments lead to a curve that is close to linear, however, this curve is in the middle of the curve from Figure [ where in addition scaling we change eyperpprrmmetess and further in Appendix
training details. We discuss the effect of
INVESTIGATING DIFFERENT DS- VS-US TRENDS
In this section, we investigate the reason behind the saturation behavior in the DS-vs-US accuracy plots and address why saturation happens much earlier for some DS tasks compared to the others. First, we take a closer look at Figure overlaying convex hulls of different downstream tasks on top of each other. Specifically, of all DS tasks. TigureB3aadd Figure sho this for cases where US task is JFT and ImageNet21 K respectively. We observe that (1) Best performing ImageNet models perform very similar to best performing models in several but not all DS tasks. (2) As the US performance increases, the gap between best performing ImageNet models and best performing DS task models reduces significantly. We also depict Spearman correlation between accuracies on different DS tasks, and between DS tasks and US task in Figure F.20 and F.2 respectively. Therefore, as the next step, we focus on capturing the difference between different DS tasks.
overlay the convex hull of ImageNet DS-vs-US plot DS-vs-US plots
As discussed in Yosinski et al. 2014 Neyshabur et al., 2020). lower layers capture lower level features that are more common across different dataset and tasks, whereas fine fine-grained features reside at top layers in the network. In addition, examples that are learned in higher layers are learned later in training with lower confidence and higher uncertainty (Baldock et al. 2021). Inspired by these observations, we measure the performance of few-shot classifiers when applied on top of representation from different layers of the pre-trate pre-trained model. We look into the depth of the earliest layer that leads to the best performance for a given DS task and check whether this is a proxy of the difference between US and DS, and indicator of how much the DS task will benefit from
8

---

Published as a conference paper at ICLR 2022
US Acteany 1.20
US Accuracy 6.0 c30
US Accirsty 020
US Acteacy 6.0
020
6:30
14
020 1.0
0 0 0
CaP 200lhot
0006
2 2
LEFE
Acourccy
Accuracy
Accu acy
Acourcey
10_/m00000000000 200hoo
cohoch 203001
2
8
Laye
Represemart DIT
Representatons
Represerations
Represemartoom oIT
Figure 6: The effect of choosing representations from different layers on the downstream tasks performance overlay-ed ed with the effect of scaling (model, data, and compute) on downstream performance when upstream task is JFT. The red triangles are performance on downstream task when representation used in the few-shot learning is from different layers of the model. The green cireles overlay the DS versus US performance of different experiments from Figure I on each task. Red triangles use the x-axis on the bottom and the green circles use the x-axis on the top. We note that for those DS tasks that are similar to US, such as ImageNet, the higher the layer the better performance on DS. On the contrary, for those DS tasks that saturate fast, such as UC-Merced and col hist, the optimal layer is not the last one.
present this result. We notice that, for DS tasks that are similar to US, such as ImageNet, the higher the representation layer the better performance on DS. On the contrary, for DS tasks that saturate fast, i.e., do not follow performance improvement on US such as UC-Merced and choosing lower layers as the top layer and skipping the rest of the network leads to same or better performance on the DS.
scaling up the compute or US data size. Figure o Figure F.22 F.23
the optimal layer is not the last one. That is,
Bringing the two discussions together, performance saturation on DS happens when the pre-trained network lacks the fine- tne-ggraied features required to perform well on DS. Therefore, one can get similar performance such DS task when cutting the top layers of the pre-trained model, as seen in Figure o Interestingly, as we see in Figure and Figure F:23 when we overlay the DS-vs-US accuracy curves on DS securacy-vs-layee depth curves, they follow almost exactly the same pattern, which means they are both good proxies for capturing the relation between US and DS tasks.
DISCUSSION AND CONCLUSION
We have investigated the role of scale in few-shot and transfer learning performance in image recognition and have established through extensive study that as we improve the performance of the upstream task either by scaling up or hyper-parameter and architectural choices, the performance of downstream tasks shows a saturating behaviour. In addition, have provided strong empirical evidence that contrary to common narrative, scaling does not lead to a one-model-fitss demonstrated the role of checkpoint that performs well on all possible downstream tasks. We assert that we should refrain from focusing on the performance of only one downstream task, which usually ends up being close breadth
solution. We have and emphasize that one cannot hope to find one pre-trained
to the upstream task. Instead, we should make design choices that improve performance of downstream tasks. Moreover, scaling has both monetary and environmental costs (Patterson et al., 2021 Dehghani et al. 2021a). We argue that, when investing in terms of scaling in terms of data, model parameters and compute, we should think of an additional axis which is data diversity.
The phenomena described in the paper is not limited to the setting reported above. In Appendix we discuss that the observations are robust to several changes in the setting, namely, number of shots, few-shot vs transfer setting, scaling of plots and architecture.
9

---

Published as a conference paper at ICLR 2022
Our paper focuses on the supervised image recognition task. Extending investigation to unsuper- vised pre-training is also of interest. Exploring other modalities, e.g., natural language domain the subject of future work.
REPRODUCIBILITY STATEMENT
All the experiments conducted in this paper are based on Scenic library (Dehghani et al. (0021b). We have shared details on the customization done the controlled experiments Appendix 1.1 All other details, including data preprocessing steps well as training and evaluation hyper-parameter are kept fixed to their default. proof for the theoretical discussion on randomized models (Section 2.1) is provided in Appendix B We have also shared descriptions and references to all the downstream tasks and datasets we used for evaluation in Appendix G.3
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint 210. 2021.
Robert JN Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. arXiv preprint arXiv: (ee::22006099477 2021.
Peter Bartlett Dylan J Foster, and Matus Telgarsky. Spectrally-normaaizze margin bounds for neural networks. arXiv preprint arXiv: 1706:08498, 2017.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv: 1612.030/, 2016.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung-Yr Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint vrXiv:2103.07577 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.. 14165, 2020.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 1855119::865-1883, 2017.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Mostafa Dehghani, Anurag Arna,, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. arXiv preprint arXiv:2110. 1...:2100.12244 2021a.
Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenie: A JAX library for computer vision research and beyond. arXiv preprint arXiv:21 /403, 2021b.
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07000, 2021c
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. image database. In CVPR, 2009.
Imagenet:
large-scale hierarchical
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv preprint ar Xiv:20000.1110 19999, 2020.
Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin. Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches on a unified few-shot classification benchmark. arXiv preprint arXiv:2 024.0263, 2021.
10

---

Published as a conference paper at ICLR 2022
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):123--123, 2013.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang. Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01966 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Danny Hemandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.
arXiv preprint (:Xiv:2102.01290 2021.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. arXiv preprint arXiv:/810.00113, 2018.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 003361 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6908 2014.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao. et al. Wilds: A benchmark of wn-the-wild distribution shifts. arXiv preprint arXiv:2012.07221,222 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint arXiv:1912.11 13770, 6(2):8, 2019.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2661 1671,, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In 4th International IEEE Workshop on 3D) Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. 11-104. IEEE, 2004.
Fei-Fei Li, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. 2021.
arXiv preprint 2rXiv:2111..2000
Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of
influence for transfer learning across diverse appearance domains and task types. arXiv preprint arXiv:2103.13318, 2021.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang. Yair Carmon, and Ludwig Schmidt. Accuracy the line: On the strong correlation between
and in on-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.

---

Published as a conference paper at ICLR 2022
Basil Mustafa, Aaron Loh, Jan Freyberg, Patricia MalWilliams, Megan Wilson, Scott Mayer McKin- ney, Marcin Sieniek, Jim Winkens, Yuan Liu, Peggy Bui, et al. Supervised transfer learning at scale for medical imaging. arXiv preprint w7Xh:22011005913, 2021.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap: Good online learners are good offline generalizers. arXiv preprint arXiv:2010.08127, 2020.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring general- ization in deep learning. arXiv preprini arXiv: /00. 077067 2017.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008.1 /687, 2020.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07056 2018.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang. Lluis-Mique Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv: 104. 10350, 2021.
Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoe Le. Meta pseudo labels. arXiv preprint arXiv:2003. 10580, 2020.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedrie Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009.1322 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 10..2103.0000 103.00020, 2021.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint ar 19Xix:9922007200 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:/806.004: 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Michael S Ryoo, AJ Piergiovanni, Anurag Arna,, Mostafa Dehghani, and Anelia Angelova. Token- learner: What can learned tokens do for images and videos? arXiv preprint arXiv:2106.11297, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114 14. PMLR, 2019.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint +=Xiv:2007.00644 2020.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Onnidirectional representations from transformers. arXiv preprint arXiv: 03005 2021a.
12

---

Published as a conference paper at ICLR 2022
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler, Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint 0rXh:22109.10680 2021b.
Eu Wern Teh and Graham W Taylor. Metric learning for patch classification in digital pathology. In International Conference on Medical Imaging with Deep Learning-Extendel Extended Abstract Track, 2019.
lyy Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint www.:::205.0000:00 2021.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint 1903v:9003.03000 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint arXiv: 1411. 2014.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1910::1::00044440
2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint (tt.:22006045660 2021.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoe V Le. Rethinking pre-training and self-training. arXiv preprint 068882200.00688,, 2020.
13

---

Published as a conference paper at ICLR 2022
Appendix
A ADDITIONAL RELATED WORK
Large scale transfer learning by pre-training on JFT (Kolesnikov et al.; 2019 Dosovitskiy al.
2020 Ryoo et al. 202 Mustafa et al.) 2021 Tay et al. 2021a Puigcerver et al. 2020: Ngiam et al.
2018 or ImageNet2IK (Dosovitskiy et al. 2020 Kolesnikov al. | Mustafa et al.) 2021] |02T]Amab Amab et al. 2021 Likhosherstov et al. 2021; ?: Puigcerver et al. 2020, Zhai et al. 20119) has been done extensively. Mensink et al. 2021 considers a two-step transter chain, where the model is pre-trained on ImageNet, fine-tuned on the source task and then transferred to the target task. Then they look into the effect of different
on this transfer chain. They conclude that the effect of transfer learning vanishes the target domain size increases. This is very different from the setting we consider, that is when the size target domain is very small (few-shot setting).
Raghu et al. (2019) investigate the performance of models pre-trained on ImageNet when they used to transfer to medical images. They conclude that the family of smaller lightweight convolutional networks performs comparably to standard ImageNet models, despite having significantly worse accuracy on ImageNet. Hence, ImageNet performance is not predictive of medical performance. Neyshabur al. (2020) also studies transfer learning from models trained on ImageNet. They note that improved accuracy from pre-training can be achieved in less steps of fine-tuning than what is done in practice.
In the context of transfer learning, generally, model selection is major step. Given a set of pre-trained models, and downstream task, we need techniques or methods help us predict the performance of the pre-trained models on the downstream task. There have been several efforts in this direction, including NCH[3], LEAPS[4], and most recently LogME[2]. The main focus of the aforementioned methods is to estimate the transfer performance of the models efficiently without actually having to fine-tune them on the given downstream task. In this paper, on the other hand, our aim is not to predict the transfer performance of specific model. We are taking the big picture into account, and focusing on understanding the limits of scaling up, i.e., the dynamics of the scaling behaviour of DS performance with respect to US accuracy. The framework can in general be used to answer the question of "How beneficial it would be for the downstream task if we improve the performance of the models the upstream task by scaling up data, compute, model-size?". We show that having enough data points with a wide enough range of accuracies on the upstream task, one can fit pawer-law curve that can be used to predict the best possible accuracy on the DS task with respect to any given us accuracy. Ideally, this would mean that we can predict the performance of models that do not yet exist. (We are predicting the best performance one could achieve on DS task by pretraining on given US task). Here we use few-shot accuracy as an indicator of the performance the models on the downstream tasks. Methods such as LogME can be beneficial for such analysis, e.g., instead of tracking the few-shot performance of the models, one could look into other transferability metrics.
propose and employ in this paper,
B PROOF OF LEMMA 2.1
Proof. Since pj are probability values we have pj 2 0 for all Σ₁ Pj definition of accuracy and simple counting, follows. Accuracy captures total number of correct predictions total number of predictions. Let refer accuracy of 0, i.e., à (aUS DSDs) let
The proof follows the
14

---

Published as a conference paper at ICLR 2022
nus, pps refer total number of predictions for upstream and downstream respectively. That is pz1(0)(z)) - ys)
N
UUS
(1)
1
yi)
2
TUS
RUS
TUS
Pa
((,) = y.)
(4)
pjaj
TAUS
j=1
where (1), (4) are due to the definition of accuracy, (2) is achieved by the construction of the randomized classifier and (3) is due to commutative property of addition. Similarly,
1
aos
1 nos N
a
DS
1 = v1)
Σ
(x;) = v:)
TDDS
1DS
5=1
RAS ((,)
= wi)
Σ pja
ps
Pj
nps
Putting these two together gives us
N
N
N
à = (aUS (SD)
Σ
p,ayys
US
Σ
praps
Σ pja'.
j=1
j=1
Note that, this is the definition of convex hull of a..j j E [N].
C DISCREPANCIES BETWEEN US AND DS PERFORMANCES: A CASE STUDY
In the last section, we observed that there exist cases where increase the US accuracy does not translate to performance improvement in DS. In section investigated the hyper parameters that are relevant to scaling, i.c., number of training epochs, number of training examples, and model size.
In this section, we build on the observations in Section 3 on the role of training hyper- parameters in the DS-vs-US performance plot. Here, inspired by (Zhai et al. 2021) we focus on hyper-pan saper-parameters related to the head (projection layer). Zhai et al. (2021) observed the impact of decoupling head weight decay on the performance of DS and US tasks. Specifically they noted that higher head weight decay during pre-training leads to worse performance on US, while improving DS performance. In this section, we take a closer look at the effect of the head (the projection layer).
We present cases where there are discrepancies between US and DS performances when we change head hyper- parameters. We investigate the phenomena observed in (Zhai et al. [021]] further and provide explanations on why this happens. Moreover, we show that one can observe similar phenomena by decoupling and decreasing learning rate of the head during pre-training. In addition, for both head WD, LR, we conclude that the optimal value for each DS task depends on the DS task.
The experiments in this section, are aligned with discussion on Section 3 on the effect of hyper- parameters and show that when we consider a point on the DS-vs-US accuracy plot, changing the hyper-parametery may lead to moving in different directions toward the convex hull. It does not necessarily lead to a vertical improvement where you keep US accuracy fixed and increase DS accuracy. There can be even cases where improving DS accuracy comes the expense of hurting US accuracy.
C. EFFECT OF HEAD WEIGHT DECAY
Figure shows the performance for DS when increase US head weight decay. In this experiment, the weight decay for the rest of the network kept at 0.01. Figure F.24 in Appendix (.4 depicts the same plot for 25 downstream tasks. We observe that
15

---

Published as a conference paper at ICLR 2022
For US, increasing the head weight decay up threshold (optimum head WD) improves the performance on US and increasing it beyond that threshold leads to over-regularization and worse performance.
The optimum value for head WD is different for US and different DS tasks. That is, there are cases where increasing WD on US head, results deteriorating performance on US but improves performance for some DS tasks. Therefore, head weight decay an important hyper-parameter and should be optimized for DS.
The optimal head weight decay for different DS tasks can be very different, i.c., If we take different DS tasks into account when tuning the value for this hyper-parameter we will end up with different optimal values. This is illustrated in Figure D.3 are cases where increasing or decreasing US head WD results in improved performance for a DS task and degraded performance for another DS task. Therefore, one cannot simply save a checkpoint of a model pre-trained on an upstream task and use for all downstream tasks.
F.25 In other words, there
The optimal weight decay for DS is usually higher than the optimal one for US, as also shown in (Zhai et al. 2021).
The impact of increasing weight decay on the head more prominent when the number of shots lower. For example, we observe that the effect more prominent on I-shot performance on all DS datasets than on 20-shot performance.
This phenomenon is robust to the number of training steps in US, i.e., increasing the number of training epoch does not change the trend. See Figure in Appendix[F2]
C.2 EFFECT OF HEAD LEARNING RATE
Next, we look into the effect of decoupling head learning rate, i.e., changing the learning rate of the head relative to the learning rate of the rest of the network. In this experiment, the learning rate for the rest of the network kept at 0.008. We notice similar patterns when decreasing head learning rate to that of increasing head weight decay. Figure F:27 shows the discrepancy between DS (Imagenet and Caltech) and US (JFT) when we change head learning rate. Considering the trend for all DS tasks, we note that impact of head learning rate DS is different from impact US. When decrease head learning rate, for a number of DS tasks the performance remains the same or improves when US accuracy degrades.
We also look into optimal head learning rate for different DS tasks in Figure F.28 in Appendix F.4 and observe that depends on the DS task and for an optimal performance, one needs to tune head learning rate for each DS task.
D INVESTIGATING THE EFFECT OF HEAD HYPER-PARAMETER
First, we investigate the Ln-morm of layers a proxy of the amount of information stored in them, as we change the head WD. In this experiment, the WD for the rest of the network is kept at 0.01. We observe that as we increase the WD on the upstream task, the norm of the weights in the higher layers increases while it does not change much in lower layers. Figure D.2 shows the sum of the norm of all layers before the head as we increase head weight decay [] We observed similar pattern in distance to initialization as we increase head WD, we do not see a change in lower layers, but the distance to initialization increases for higher layers we increase head WD.
It has been widely discussed that a network's margin of error (also called prediction margin) can predict its generalization performance well (Neyshabur et al.) 2017 Bartlett al. 2017 Jang et al.) 2018). We refer to the margin for single data point as the difference between the score of the correct label and the maximum score of other labels. We report average margin value over train data. The classical notion of margin refers to the scores at the head. More recently, Diang et al. (2018) proposed the notion of margin at different layers which normalizes the score difference by the norm of gradient differences at that layer. Due the correlation of margin to how well the model separates the data at each layer, in order to investigate this phenomena we look into how head margin and
FFiuure
P:29 Appendix R4 shows this trend for each layer separately.
16

---

Published as a conference paper at ICLR 2022
caltech
cars
difar100
hist
34-
09 Accuro
DB Acstracy
Accuracyy
Exemony
Head Weight Decay
Head Weight Dooay
Head Woight Decay
Hoad Weight Decey
imagenet
pats
UC_ mmmmmedded
Acournay
4.3
D9 Aourracy
D6 Acouraa
1003 10020OO
Aceuncny
Sonuntoy
Weight Decny
Hoad Wiight Dooay
Weight
Weight Decay
Figure D.I: The effect of increasing head weight decay in performance of upstream (JFT) versus performance of downstream (all shots). Note that not only the optimum value of head WD for upstream and downstream different, but also the optimum value changes for different downstream tasks.
Traneformer Blocks
Heac
Head Margin of Pholog Layer Mergnn
12-
Lest homm
Hoad Muggin
10
20
Cie-
1:00
Best Head WD
Head WD
Head WD
Layer norm and layer margin for US as a function of head weight decay. As we increase the head weight decay, the sum of norms of all the layers up the head, as well the pre-logit layer margin increases, while the head's norm and margin decrease. Since these two metrics are correlated with amount of information stored in layer, we conclude that increasing head weight decay pushes the information stored in the head the layers below, similar to the effect of decreasing the head learning rate.
Figure D.2:
0.09
Dataset
Figure D.3: Optimal head weight decay for each DS task for different number of shots. The optimum value different for each DS task.
pre-logit (penultimate) layer margin changes as we increase head WD. We observe that as we increase the weight decay, the pre-logit layer margin increases, while the head layer margin decreases; See Figure D.2
For US although the head margin decreases with increasing head weight decay, which is also reflected in the performance drop on US (see Figure D.I. the margin for pre-logit improves. This shows that the information is being pushed down from head to pre-logit layer.
Since these two metrics are correlated with amount of information stored in a layer, the above two investigations suggest that we increase head weight decay the information pushed down to layers below the head. Moreover, these are still top layers in the network and the effect does not propagate nor affect early layers the network.
Next, we look into the margin on DS datasets. We note that the margin (calculated on training data) trend completely reflects the accuracy trend on DS test data. Although this is expected in classical machine learning, it is still intriguing that observe this pattern for large-scale deep learning model where the margin has occasionally failed to capture generalization performance. We note that for datasets that saturate more slowly such as ImageNet, the margin increases as we increase head WD, and for datasets that saturate fast, such as Caltech 101 and Cars, the margin does not change. See Figure F:30 in Appendix F.4 for DS margin plots.
17

---

Published as a conference paper at ICLR 2022
CH98
V/AE Sho0
1740
VIA
1.6 As3recy
0.10
1.0
205
6.30
0.5 6:0
Figure E.1: Performance of models presented Figure 4 transfer learning setup on VTAB Zhai et al. 2019 benchmark. VTAB defines total of tasks, grouped into three categories: (i) Natural. which contains natural images captured using standard cameras, specialized, which contains images of the world that captured through specialist equipment, and Structured, which contains tasks that designed to assess comprehension of the structure of scene, mostly generated syntactically using simulated environments, In the transfer learning setup, fine-tune all parameters the pre-trained model well a randomly initialized head using all examples in the training set of the downstream lask. Results for individual VTAB tasks are shown in Figure phenomena similar to few-shot setting. (I) Performance saturation happens in the transfer learning setting as well. (2) The effect of increasing model size, pre-training data size, compute lead to the same curve. (3) Strong power of US accuracy compared to model size, pre- pre-training data size, compute in predicting DS accuracy.
We observe a
We observe that as we decrease the head learning rate, the norm of the head decreases while the sum of the norm of other layers increases. A similar pattern is captured in US margin and norm plots when decreasing head learning rate as to increasing head weight decay (Figure F.3 in Appendix (F.4). We note that the effects of these two interventions (increasing head WD, decreasing head LR) similar. When we increase the head weight decay, as discussed above, we are pushing the information compressed in the network down lower layers. On the other hand, when we decrease the head learning rate, we encourage lower layers to be more active and learn more. Both lead to the same impact.
Next, we look into the optimal WD as a function of the rank correlation between the performance of US and DS in Figure D.4 We calculate the rank correlations as follows. Given the list of model checkpoints, we make two rank lists based on US and DS performance and then calculate the correlation between the two lists. It can be observed in Figure D.4 that for DS task optimal WD is high when we have a high correlation between performance on US and DS. The reason being when the correlation is high, one would want to move all the information that resides in the head the lower layers and not lose any information. Since the head is removed for few-shot transfer, storing more information in the rest of the network leads to better performance in the DS. But when US and DS are different and hence uncorrelated, we do not need a high WD as there is not much information in the head that will help in the DS performance and one can even remove the head and some of the top layers as seen in the analysis of Figure 6
bind
caitcch
cans
olla
Box eru
0.81
Imagenel
resisc45
oun
meroed
Con/DS
Figure D.4: Optimal weight decay as func-
tion of rank correlation between the perfor- mance of US and DS for different. DS tasks.
E ON THE GENERALIZATION OF OBSERVED PHENOMENA
The phenomena we described in the paper is not limited to the setting reported above. this section, discuss that the observations robust to several changes in the setting.
Number of shots:
The DS-vs-US performance saturation phenomena and effect of head hyper-
parameters (WD. LR) are robust to the number of shots the downstream task. This can be seen in
Figures D.3 F25
and F.26
Transfer vs. few-shot: In addition to robustness to number of shots in few-shot setting, the reported phenomena is consistent across both few-shot and fine-tuning setting (aka transfer learning). Note that this not direct implication of the previous assertion. In few-shot setting we keep the network weights fixed and only replace the head and train it for the downstream task. In fine-tuning
18

---

Published as a conference paper at ICLR 2022
setting however, weights from the rest of the network are also updated using the training data for the downstream task. Figure E. presents the results of the effect of scaling in the fine-tuning setup on VTAB Benchmark. Note that VTAB considers low-sample regime (1000-examples), which reflects performance under a reasonable labelling budget. Results in Figure E. correspond to the same controlled experiments that we performed in from Figure 4 for few-shot setting. Results on VTAB as well as VTAB subsets, i.c., natural, specialized, and structured shows similar general trends to few-shot setup. See Appendix H for additional observations and more detailed results in the fine-tuning setup.
Scaling of plots: Many of the works that consider transfer performance accuracy or how model
accuracy changes by scaling up (Kornblith et al. 2019 Kaplan et al.) 2020; Hernandez et al. 202I). scale the accuracy by passing it through a logit transformation (logit(p) = =log(
log(p) - 1 - p), i.e., instead of plotting accuracy, they plot logit of accuracy. Logit function (which is the inverse of sigmoid function) has the drawback being sensitive to low values. Meaning that if we plot range of values that includes values close to zero, the logit plot is mainly influenced by values between 0 and 0.15 and the bigger values are collapsed mostly on top of each other. To mitigate this sensitivity, one can instead plot the the second term log(1 p). We considered both these scaling options as well as not scaling the accuracies and observed that both phenomena presented
in the paper are robust to choice of scaling. For corresponding plots to logit and loog1 p) see Figure F22 in Appendix F.1
Architecture: In this work, we investigated family of architectures and a number of different architectural changes in different models from the Vision Transformers, MLP-Mixer and ResNets. has been widely inspected that in large data regime the role of inductive biases and architecture- specific parameters diminishes. Moreover, there is evidence indicating that choice of architecture does not impact the power law governing the DS performance (Kaplan et al. 2020 Taori et al. 2020) Miller et al. 2021). This is also observed in (Komblith et al., 2019) that effect of architecture is only observed through US performance. Therefore, we expect that our results generalizes to other large scale architectures such as ResNet- 151 and EfficientNet (Tan & Le 2019) (that is made of CNN
blocks).
F ADDITIONAL FIGURES
F.1 ADDITIONAL FIGURES FOR SECTION
depicts similar plot to Figure 1 with all the 4800 experiments (pre-trained on JFT or ImageNe121K), for 1 and 25 shots.
Figure F.T
Figure F2
presents scaled version of Figure given the scaling of downstream accuracies, discussed
in Section
F.1.1 DETAILS THE EXPERIMENTAL SETUP FOR FITTING EQUATION T
Figures and F.7 illustrate the fitted curves to the the convex hull and all data points in the US-vs-DS accuracy plots respectively. We use the points from the lower US accuracies (0.0, 0.45) fitting data and higher US accuracies (0.45-0.50) as held out data to fit equation For the convex hull fit, we first compute the convex hull of the given data points, and find the fit to the convex hull. In Figure F.8 and [99 we compare the fitted curves when we fit equation | to all data points or the convex hull of all data points for shot and 25 shot.
To measure the sensitivity of the predictive power of the fitted equation to the number of samples, we conduct the experiment with different number of data points sampled randomly (uniform distribution across all data points), and for each sample size we repeat the experiment 10 times (where we take a new sample for each trial). We use the points from higher US accuracies as held out data. Prediction error captures the difference between power law prediction and observed value of the DS accuracy. Fitting error captures the difference of power law values from the points that are used in calculating power law parameters, We plot fitting error and prediction error as the number of samples changes. Figures F.12 F.13 F. F.144 F.155 and F.166 depict the mean prediction error and mean fitting error for each sample size as well their standard deviation across the 10 trial.
19

---

Published as a conference paper at ICLR 2022
Figure F.l: Performance of different downstream tasks upstream based experiments (2974 Vision Transformers, 1593 MLP-Mixers, 249 ResNets). The experiments are grouped based on the upstream dataset (JFT or ImageNet21K] dotted line shows the convex hull of points on the DS-vs-US plot. The fact that saturation happens at different values for the two upstream tasks suggests that saturation does not solely depend on DS task, rather, it is about the relationship between US, DS tasks.
than 4800 different
number shots few-shot evaluation (1 and 25). The
2
Figure F.2: Performance of upstream vs downstream (8 different tasks) based on more than 3K different VIT models with different configurations, tre-trained JFT and evaluated on few-shot (25 shots), where downstream accuracies are scaled using- log(1 p).
20

---

Published as a conference paper at ICLR 2022
Wiinn Translomers
02 Acouracy
US Acqurayyyy
Figure F3:: The performance of downstream (8 different tasks) of upstream based more than 1.4k different Vision Transformers, MLP mixers and 233 ResNets, with different configurations. The models are pre-trained trained on ImageNe:21K and evaluated in few-shot settings (25 shots). As the upstream performance improves, the downstream performance saturates. Even if US accuracy reaches 100% accuracy, the DS accuracy may not lower value. We observe non-linear relationship between upstream
reach the 100% accuracy and saturates and downstream accuracy and model the relationship with power law function predict the downstream performance given the upstream performance. The plot also show horizontal line which predicted downstream accuracy if upstream accuracy reaches 100%.
1.1
0.58
12
cars
03-
cltr100
Hat
0.
0.40
Iragenet
peta
0.8
0.30
MC, muummrroed
0.5
0.25
0.-
0.10
Shot
Shot
Figure F.4: Effect number shots and DS task on the value parameters of power law curves, upstream lask is JFT. We note that DS task affect all parameters, while number of shots mostly impacts & and eiR-
1.0
077
0.-
cars
0.8
45
clar100 100
h8t
4.
0.7
0.6
3.0
0.4
pets
m.mmmerdd
0.5
2.5
0.3
0.3
02-
1.0
0.5
Shot
Shot
Figure F.5: Efffect number shots and task on the value parameters of the power law curves, upstream task is ImageNet 21k, We note that DS task affect all parameters, while number of shots mostly impacts k and
21

---

Published as a conference paper at ICLR 2022
A60000 Dur
6.30
D3 ACQURSY
DS ACQuESc
1.0
DG Accuriiy
DIS AccuUTTTT
05
03 Accurssy
6.2 Accurary
03 Accurasy
Accurasy
Figure F.6: Power law curves that are fitted the points convex hull corresponding to experiment results from Figure [.T]] We plot the predictions from the power law curve on the higher US accuracies to the ground truth (prediction target) and observe that power law curve closely predicts the performance of DS..
DS Azsunay
D6 AcoutS0
22 2000110
D8 Acounio
08 Acturaty
DS Accuracy
055
DS
Power law that fitted all point corresponding experiment results from Figure F.1 We plot the predictions from power law curve the higher US accuracics to the ground truth (prediction target) and observe that power law curve closely predicts the performance of DS.
Figure F.7:
22

---

Published as a conference paper at ICLR 2022
00 A00 2017
DS Acpunecy
DS Acoureey
2.50
6.00
DS Accuraccy
DS6 Recumscs
05
Figure F.8: Comparing fitted curves when convex hull (Figure F.6 (Figure when number shots is 1.
when all samples
Doua rysted
DS Aoturary
DSC
DS ACOUS22y
05
41
0F
41
62
38
Aocurazy
Recuracy
Rocuraay
Acouracy
Figure F.9: Comparing fitted curves when we use convex hull (Figure (Figure when number shots is 25.
when we use samples
23

---

Published as a conference paper at ICLR 2022
Image Mngeleeliiaveeee...
artovergg
0427
1009
Play
Cove fal reigg piy
5401
6009
1000
18
Figure F.10: The effect of sample size power law The experiments as well as data points from Figure We use the points from higher accuracies as held out data. Prediction error captures the difference between power law prediction and observed value of the DS accuracy. Fitting error captures the difference of power law values from the points that are used in calculating the number of samples changes. For more power law parameters. We plot fitting error and prediction details see Appendix F.1.I
fitted the convex hull of
24

---

Published as a conference paper at ICLR 2022
Sample
Somple
Sunde
Gampr 50c
Scnce 53m
Samph
Sample
1121118211
Figure F.11: Effect of sample size when fitting the power law to convex hull of the samples the average fitting error.
25

---

Published as a conference paper at ICLR 2022
Samoe
13111
Momp Morn
Landa
Samdi
Figure F, 1212: Effect of sample size when fitting the power law to convex hull of the samples on the average prediction error.
1wgge i-dhot
wwsgetaecc/Wveagg.... 255kel
aProweuge
5.59
R
8
Sample Stxc
Stmpte
Sample
Sample
Figure 1:13: Efffect of sample size when fitting the power law to convex hull the samples on the average fitting error.
26

---

Published as a conference paper at ICLR 2022
lNtweerae 25-dhet
artlewengg 1-thet
TTeeeppp 25-Shal
Predap Ert
00000
0:009
0000
Sample
Sample
Sample
Figure F.14: Effect of sample size when fitting the power law to convex hull the samples on the average prediction error.
190 www.eeee IiCeerggg 25-5hl
3.003
2,30
2,003
88
8
Sample
Somple
Scmple
Sample
Figure F.15: Effect sample size when fitting the power law to samples on the average fitting error.
01/10
02040
COLP
PredSoon Engy
Prodioon Engor
02000
3.007M
Figure F.16: Effect of sample size when fitting the power law to all samples the average prediction error.
27

---

Published as a conference paper at ICLR 2022
DS
US
Parameter
Correlation with Number of Shots
caltech
ImageNer21K
K
-0.777892
caltech
ImageNer21K
a
-0.582066
caltech
ImageNet21K
EIR
-0,845368
caltech
JFT
K
-0.620526
caltech
JFT
0.259305
caltech
JFT
CIR
-0.762856
cars
ImageNet2IK ImageNet21K
K
0.720391
cars
a
0.960490
cars
ImageNet21K
CRR
-0.737273
cars
JFT
K
-0.976599
cars
JFT
-0.034033
cars
JFT
FIR
-0.809016
cifar100
ImageNer21K
K
-0.918914
cifar100
ImageNet21K
0.683485
c0f
ImageNet21K
EIR
-0.587304
c0far100
JFT
K
-0.934455
c0
JFT
a
0.707966
cifar100
JFT
GIR
-0.754030
col col_hist hist col hist
ImageNet21K
K
-0.756297
ImageNet2IK
0.947 101
col hist ImageNet2IK
GIR
-0. 104776
col hist
JFT
K
-0.534724
col hist col hist dtd ImageNet21K dtd ImageNet21K ImageNet21K
JFT
a
0.466138
JFT
GTR
-0.848960
K
-0.892400
a
0.810935
dtd
CTR
-0.532797
dtd JFT
K
0.392218
dtd JFT
a
-0.751290
dtd JFT
GIR
-0.806674
imagenet ImageNer21K
K
-0.923350
imagenet
ImageNet21K
a
0.464193
imagenet
ImageNe12IK
ETR
-0.590325
imagenet
JFT
K
0.618935
imagenet
JFT
a
-0.866692
imagenet
JFT
ETR
-0.847294
pets
ImageNet21K ImageNet21K
K
-0.895292
pets
a
0.707 198
pets
Ie12
6RR
0.936508
pets
JFT
K
0.398171
pets
JFT
a
0.937076
pets
JFT
EIR
-0.003738
uc_ me_
I K
K
-0.986538
uc ue_merced
ImageNet21K IeageNer21K
a
0.942120
uc_merced
ETR
-0.724245
uc uc_mereed
JFT
K
-0.821492
uc_mereed
JFT
0.
uc_merced
JFT
6IR
0.019906
Table F.1: Correlation of each parameter with number of shots
28

---

Published as a conference paper at ICLR 2022
Parameter
Correlation for US=JFT
Correlation for US=
ImageNet 21k
-0.65
-0.81
0.60
0.75
EIR
-0,88
-0.79
Table F.2: The Likelihood that each of the parameters of the scaling law increases/decreases the number shots increases, average over all DS tasks.
29

---

Published as a conference paper at ICLR 2022
F.2 ADDITIONAL FIGURES FOR SECTION 3
Figure F.17 shows the effect of scaling model, data, and compute on all downstream tasks. This is more complete version of Figure 4 in the main paper that includes all 25 different downstream tasks.
200he
Geadan
01
05
Upstream Acturacy
Upatnom Accumsry
Upseream Anouacy
Upsrrsm Aczuncy
Upimam Accurccy
1.3
18
071400
Upstram Accuracy
Upstram Accurccy
Upstroa
Upstraa Accurccy
Upsiramm Accerocy
Upsleam Acourayy
Uptheom Accurssy
Updtremm Acquracy
Upebsam Accuray
Upsttem Accuracy
101
Upptream Acturac
Upannam Accundty
Upohoum Accency
Upartam Accenccy
0
Upsream
Upstrem Aoourady
Accusacy
Upstram
Figure F.17: Effect of controlled scale up with respect the model size (number of parameters), data size (portion of the pretrained data), compute (epochs) on 25 different downstream tasks in few-shot setup (20-shots).
30

---

Published as a conference paper at ICLR 2022
2'v1000
Dredited
Imaganel
cbent
den Jedance
100 Saaeee
D5
p6t
Sowert
DC Accunsy
09 Accu82
US Noursey
D5
STabroo denddo
D5 Hamay
DS 2couaay
05
U55 Acuussy
US Accursoy
Accurscy
Accutaty
Figure F.18: Fitting the scaling law to points plotted Figure F.17 and depicting the value of error incurred in predicting DS accuracy.
31

---

Published as a conference paper at ICLR 2022
DS
(Y
birds
0.154270
caltech
0.102052
camelyon
0.402138
cars
0.197948
cifar10
0.235078
cifar100
0.242331
clevr_count
0.093481
clevr_distance
0.093481
col hist
0.155221
dmlab
0. 0.126028
dsprites_location
0.059326
dsprites__oreentatinn
0.059326
dtd
0.088551
curosat
0.258027
flowers
0.141492
imagenet
0.188222
kitti
0.438465
pets
0.141251
resisc45
0.188155
retinopathy
0.446441
smallnorb azimuth
0.049473
smallnorb -ettttorreeeevttoonn
0.049473
sun397
0.085017
svhn
0.082023
uc_ oc_merced
0.158118
Table F.3: Root squared error predicted DS accuracy when fitting the points in Figure F22 with Equation
32

---

Published as a conference paper at ICLR 2022
F.3 ADDITIONAL FIGURES FOR SECTION 4
US AUAEROY
AcouRtO
US Acourity
Acounsy
1.5
1.50
Figure F.19: The overlay the convex hull of ImageNet DS-vs-US plot DS-vs- US plots of all DS tasks from Figure [ US task ImageNet21 K. We observe that Best performing ImageNet models perform very similarly to best performing models several DS tasks but not all DS tasks. Moreover, as the US performance increases, the gap between best performing ImageNet models and best performing DS task models reduces significantly.
Figure F. F.20 depicts Spearman correlation between accuracies different downstream tasks. Fig- ure F22 shows Spearman correlation between accuracies on different downstream tasks and the upstream task.
-0.5
0,0
0.5
1.0
Figure F.20: Spearman correlation between accuracies on different downstream tasks.
Figure F222 illustrates the quality of representations from different layers all downstream tasks.
This is more complete version of Figure 6 in Section 4 that includes all 25 different downstream tasks.
33

---

Published as a conference paper at ICLR 2022
02
3.4
0.6
0.8
1.0
1-Shot
Shot
magaee
ImageRet21K
1-Seet
25-Shot
CABOY
Figure F.21: Spearman correlation between accuracies on different downstream tasks and the upstream task, based on more than 3K different ViT models with different configurations.
34

---

Published as a conference paper at ICLR 2022
CO OP
Rapsserakra
Rap aaraaraarrr
Acazncy
Acoutry
20 010
010
6n
130
LX
US Acourcy
US ACCUCO
US Accunsyy
US Accassy
05:000
US
US
120 010
630
5 1
Figure F.22: Investigating the effect of choosing representations from different layers on the downstream tasks performance overlay-ed with with the effect of scaling (model, data, and compute) on downstream performance when upstream task is JFT. The red triangles in the plots are performance on downstream task when representation used the shot learning is from different layers the model, The green circles in the plots overlay the US versus DS performance of different experiments from Figure on each task. Here we sketch the plots for 25 downstream tasks.
35

---

Published as a conference paper at ICLR 2022
US Accuracy
US Acouracy
US Acouracy 0.28
US Accuracy
6/2
0,29
5.00
9.0
0.50
130
0.00
39
1.0
1.0
clant 02 2002100
De00 25shel
cong-25ahul
imageeatt
C
8
%
Rogrosentations
Representatioss
Popresontations
Representations
US Accuracy
US Accuracy
US 030
US Accurasy 630
200
2.21
1.90 0.0
2:01
1330
0:31
022
Dre 01
03 (160 250025
caltcch 200tter
015 200000
08
Ueer
Represembaaions
Pepresenbations
Pepresentations
Represemtations
Figure F.23: Investigating the effect of choosing representations from different layers the downstream tasks performance overlay-ed with US-vs-DS performance when upstream task ImageNet21K. The red triangles plots are performance on downstream task when representation used different layers of the model. The green circles in the plots overlay the DS versus US performance of difffrrent experiments from Figure F.3 on each task. Red triangles use the x-axis the bottom and the green circles use the x-axis on the top. We note that for those DS tasks that are similar to US, such ImageNet, the higher the representation layer the better performance UC-Mereed and col_hist, the optimal layer is not the last one and the model can cut at lower layers leading to better performance.
few-shot learning is from
DS. On the contrary, For those DS tasks that saturate fast, such as
36

---

Published as a conference paper at ICLR 2022
F.4 ADDITIONAL FIGURES FOR APPENDIX
Figure illustrates the effect of increasing head weight decay on all downstream tasks. Figure D. in the main paper that includes all downstream tasks.
Shot
BrightDooyy
Weight
Waight
Dhuigh
Besd WeightSoayy
Meight
Figure F.24: The effect of increasing head weight decay in the performance of upstream verses downstream (all shots, all datasets).
37

---

Published as a conference paper at ICLR 2022
Figure F.25 show the best head weight decay for all downstream tasks. This figure is more complete
version of Figure D.3
100
shot
O
Best Head WD
10
5-
20
0.1
0.001
birds
caltech
camelyon
did
resisce5
uc moromerroo
olevr count
euroeat
sur397
swhn
ool Het
metmopal
smalinoto elevation
cars
imaginet
ofar 00
diaar
Dataset
Figure F.25: Optimum head weight decay for different downstream tasks and for different number of shots.
Figure F.26 illustrates the effect of changing head weight decay on all downstream tasks, when we train longer (for 14 epochs instead of that is reported in Figure D.I}. The changes is consistent across different epochs as well as different number of shots.
reight
0
Rpscha
atarro
plart
02
08
Acouracy
Accuracy
Accurnyy
Acouracy
calledh
om
0.1
030
0.38
Q29
US Acouracy
US Accuracy
UB Accuracy
UB Accuracy
Figure F.26: The effect of changing head weight decay when trained for 7 or 14 epochs for different number of shots.
Figure E33 illustrates the effect of increasing head weight decay on pre-logit layer margin for all downstream tasks.
38

---

Published as a conference paper at ICLR 2022
0.0001
0.0004
0.0008
caltech
imagenet
0.332
0.524
+Acuuracy
Accuracy
822
8.00
05
0.500
0.616
0.614
0.504
Accuracy
Accuracy
Figure F.27: The effect of increasing head learning rate in performance of upstream (JFT) versus performance of downstream (ImageNetlk and Caltech101)
shot
0.0008
Best Head LR
0.0004
10
20
0.0001
0.00008
0.00004
0.00002
devt. couunuutt
devt distance
smallroos camelyon
otarto
olaa 00
col his
Rowers
pets
dmtaa
UP mecoo
bind
Imagenet
Dataset
Figure F.28: Optimum head learning rate for different downstream tasks and for different number of shots.
M
M
0
Figure F.29: 1.2 Norm of different layers of the ViT model for different values of head weight decay.
39

---

Published as a conference paper at ICLR 2022
birds
caltcch
camelyon
GesT_murt
decance
Col_
ded
flowers
imagenst
ktti
retinopethy
amallorb. stevvtion
cuhnn
U._ merced
Figure F.30: The effect of increasing head weight decay the pre-logit layer margin for downstream (all shots,
all datasets). In this plot term for downstream few-shot classifiers is set 4096.
40

---

Published as a conference paper at ICLR 2022
Figure F.31: 1.2 Norm of different layers of the ViT model for different values of head learning rate.
41

---

Published as a conference paper at ICLR 2022
G EXPERIMENT SETUP
G.1 CONFIGURATIONS OF THE MODELS USED IN THE LARGE SCALE META-STUDY
We investigate more than 4800 experiments with Vision Transformers, MLP-mixers and ResNets with different configurations (2974 Vision Transformers, 1593 MLP-Mixers 249 ResNets.), when pre-trained on large amount of data supervised fashion and evaluated on several downstream image recognition tasks through few-shot learning and fine tuning. We note that while there are much fewer ResNet experiments in the meta data we have collected, these are the best performing ResNet models as researchers know how to tune myper-parameee performance. However, for Vision Transformers and MLP-Mixers, best practices for parameeeree tuning is yet to be figured out as these newer architectures. In addition, data suggests that the convex hull would not be affected significantly by having more ResNet models. Therefore, due to computational and environmental costs, we refrain from training many ResNets. These experiments vary in terms of the upstream dataset (either JFT-300M (Sun et al. (017)) with 303M images and 18k classes or ImageNer2IK (Deng et al. 2009) with 14M images and 21K classes), model size and shape (different sypeerparameter of the architecture), optimization (e.g. different learning rate values and learning rate schedules, different weight decays, different optimizers), compute (e.g. number of epochs) and other knobs that researchers changed during development of models for various purposes.
for these class of models to achieve best
In the experiments we run ourselves (Section 3), we mainly use ViT-B/32, which is the base model with patch size 32 x 32. We also have tiny (9.4687e+6 parameters), small (2.9536e+7 parameters), base (1.0152e+8 parameters) and large (3. 242624+ parameters) models for the controlled scaling experiments. We pre-train our models on JFT for epochs and evaluate more than 20 tasks. For the downstream evaluation, we mainly focus on few-shot learning setup (1, 5, 10, and 20 shots) as well as fine-tuning for some of the ablations. This is motivated by the fact that the effect of transfer learning vanishes as the number of downstream data points increases (Kornblith et al., 2019 Zoph setting where transfer learning shines the
et al. 2020 Mensink et al. 2021). Hence, we focus most. In both aggregated and controlled experiments, in the few-shot setup, a linear classifier is trained on top of the representations from the frozen pre-trained model, given only fixed number of training examples per class. In the fine-tuning setup, we follow VTAB standard (Zhai et al., (2019) and use 1000 training samples from the downstream task and update all the parameters of the model besides the downstream head. The details on upstream and downstream task benchmarks and training setup appear in the Appendix G
G.2 TRAINING DETAILS
For the controlled experiments, train all models using Adam (Kingma & Ba 2014) with $1 0.9, 32 = 0.999. In all experiments, the batch size is set to 4096. The deffault weight decay used in the experiments is 0.1, unless the changed value is mentioned in the description of the experiment. For the learning rate, we se the value to 8e - 4 (unless for large models that we use 4e 4) and use a linear decay, with a warmup of 1000 steps.
G.3 DATASETS
Table G.4 summarizes the datasets used in our experiments.
42

---

Published as a conference paper at ICLR 2022
Table G.4: Summary of datasets used in our experiments, part
Datasct ImageNet
Description
Reference
1.28M labelled natural images.
Deng et al. 2009
http:
Caltech101
The task consists in classifying pictures of objects (101 classes plus background clutter class), including animals, airplanes, chairs, or scis- sors. The image size varies, but it typically ranges from 200-300 pixels per edge.
//ww.vvssin www.sinnnctttcceeuu.
Image_ Doapp_Dattasets/Catttchh01/1/
https: wttps//www.ss.trootoooooo toronto.ed/
CIFAR-10
The lask consists in classifying natural images (I0 classes, with 6000 training images each). Some examples include apples, bottles, di- nosaurs, and bicycles. The image size is 32x32.
-kr "kriz/cifar .ttll
https://ww.ccr
httt://wwww....oooooooo/ toronto.eddu
CIFAR-100
The task consists classifying natural images (100 classes, with 500 training images cach). Some examples include apples, bottles, di- nosaurs, and bicycles. The image size 32x32.
-kriz/cifar.
DTD
The task consists classifying images of textural patterns (47 classes. with 120 training images each). Some of the textures are banded, bub- bly. meshed, lined, or porous. The image size ranges between 300x300 and 640x640 pixels.
Cimpoi et 2014
https://www. robots.aae o.. uk/
Pets
classifying pictures of and dog breeds (37 classes with around 200 images cach), including Persian Chihuahua dog, English Setter dog, Bengal Images dimensions typically 200 pixels larger.
The task
-vgg/data/pets/
Sun397
The Sun397 task a scenery benchmark with 397 classes and, at least, images per class. Classes have hicrarchy structure, and include cathedral, staircase, shelter, river, archipclago. The images (colour) 200x200 pixels larger.
https://vision.prineeett edu/
projects/2010/SNN/
https: rwbots..obooo.ooo uk/
Flowers102
The task consists in classifying images of flowers present in the UK (102 classes, with between 40 and 248 training images per class). Aza- lea, Californian Poppy, Sunflower, or Petunia are some examples. Each image dimension has at least 500 pixels.
wvgg/data/Flowrr/1002
SVIN
This task consists in classifying images of Google's street-view house numbers (10 classes, with more than 1000 training images cach). The image size is 32x32 pixels.
http://ufldl. http://ufldl.sttafordd .dw/
housenumbers/
CLEVR/count
CLEVR is visual question and answer dataset designed to evaluate algurithmic visual reasoning. We use just the images from this dataset, and create a synthetic task by setting the label equal to the number of objects in the images.
(Johnson et al. 2017)
CiEVRddistance
Another synthetic task we create from CLEVR consists of predicting the
(Johnson et al.
2017
depth of the closest object the image from the camera. The depths
bucketed into size bins.
hittps//wwwe kaggle. com/c/
Retinopathy
The Diabetic Retinopathy dataset consists of image-lahel pairs with high-resolution retina images, and labels that indicate the presence of Diabetic Retinopahy (DR) in 0-4 scale (No DR, Mild, Moderate, Se- vere, or Proliferative DR).
disbetic-retiinpatyy-ee det.ereeinnppttyyyyeeeetiii data
http://www.vise www.vviicisieeeeeeeeee htt.visipediaCLL-200....
birds
image dataset with photos of 200 bird species (mostly North American).
43

---

Published as a conference paper at ICLR 2022
Table G.5: Summary of datasets used in experiments, part II
Dataset
Description
Reference Tch & Taylor 2019
Patch Camclyon
The Patch Camelyon datasct contains 327,680 images of histopatho- logic seans of lymph node sections. The classification task consists predicting the presence of metastatic tissue given image (i.e., two classes). All images are 96x96 pixels.
Resise45
The Remote Sensing Image Scene Classification (RESISC) datasel is scene classification task from remote sensing images. There are classes, containing 700 images each, including tennis court, ship, island, lake, parking lot, sparse residential, stadium. The image size is RGB 256x256 pixels.
Cheng el al. 2017
EuroSAT
The task consists classifying Sentinel-2 satellite images into 10 differ- ent types of land use (Residential, Industrial, River, Highway, etc). The spatial resolution corresponds to 10 meters per pixel, and the image size is 64x64 pixels.
(Helber et al. 2019
https://github. wwt.:/cccc///mmmeeeee deepmind/ ssprites-dataser
dSprites/lcaation
The dSprites dataset was originally designed to assess disentanglement properties of unsupervised learning algorithms. In particular, each im- age is 2D shape where six factors are controlled: color. shape. scale, rotation, and (x.y) center coordinates. Images have 64x64 black-and- white pixels. This task consists in predicting the (horizontal) coordi- nate of the object. The locations are bucketed into 16 bins
https://github.. http:ichuu.ccoddeeeeeeeeeeeee cow/deepmindllll.ceeeeeecceocmmm
dSprites/orientation
We create another task from dSprites consisting in predicting the orien- tation of each object, bucketed into 16 bins.
Aisprites-o cetsssdatttttttttpeeeee
//github. ceetthub.com/deeppaee con/deepmind//
dspr ites-tes-aae datasett/
SmallNORBBazimuth
The Small NORB datasct contains images of 3D-toys from 50 classes, including animals, human figures, airplanes, trucks, and cars. The age size 640x480 pixels. this case, we define labels depending the azimuth (angle of horizontal deviation), in intervals of 20 degrees (18 classes).
LoCCun ct al. 20044
SmallNORRB/llevtiinn
Another synthetic task we create from Small NORB consists in prodict- ing the elevation in the image. There are 9 classes, corresponding to 9 different elevations ranging from 30 to 70 degrees, in intervals of 5 degrees
LeCCnn et al.
2004
DMLab
The DMLab (DeepMind Lab) is a set of control environments focused on 3D navigation and puzzle-solving tasks. The Dmlab dataset contains frames observed by the agent acting in the DeepMind Lab environment, which are annotated by the distance between the agent and various ob- jects present the environment. The goal is to evaluate the ability of visual model to reason about distances from the visual input in 3D environments. The Dmlab dataset consists of 360x480 color images in 6 classes. The classes are close, far, very far x positive reward, negative reward respectively.
(Bcattic et al. 2016
KITTI
The KITTI task consists in predicting the (binned) depth to the vehicle (car, van, or truck) in the image. There are 4 bins / classes.
(Geiger et al. 2013)
https: //www. tensorflos.oo org//
ColHist
Classification of textures in colorcctal cancor histology. Each example a 150 x 150 x 3 RGB image of one of 8 classes.
datasets/catalog/coooo
histology
https://usdahsi...
https///udahsi.uueereedd ucmerced.
UC Merced
21 class land use image dataset
edudatasets/ hattttasseets/aadds.s.ete html
http:
cars
The Cars datasct contains 16.185 images of 196 classes of cars. The data split into 8,144 training images and 8,041 testing images, where each class been split roughly in 50-50 split. Classes are typically the level of Make, Model, Year, e.g. 2012 Tesla Model S 2012 BMW M3 coupe.
/tan. faa..staffrrd.ddu/1tt edd///jjrrkuse// cars/car_dataser
44

---

Published as a conference paper at ICLR 2022
H TRANSFER TO VTAB
In this Section, we provide additional experiments for the transfer learning scenario and use VTAB as downstream task. Figure H.32 shows the effect of controlled experiments, scaling up model size, data size and compute for transfer learning setting on VTAB dataset. Note that these experiments are based on the standard VTAB setup (Zhai al. 2019) that uses only 1000 examples for each dataset to reflect performance of transfer learning under a reasonable labelling budget in downstream tasks. We use the same objective function for both upstream and downstream (Sigmoid cross entropy) and update all of the pre-trained parameters during fine-tuning. Table H.6 presents presennn results of models that are pre-trained with differed head weight decays in the transfer setup on the VTAB test set. this setup, we use SGD momentum with batch size 512 for fine-tuning all the parameters of the model using the training set the downstream task.
Accurccy
Asunsc
Acouracy
0.00000 (3080 pa
US
US
DRngh
Redgg
Crungl
2000 02
125 Accundy
U51 Apuuncc
125 Ac/.
00
Goy
[
Figure H.32: Effect of controlled scale up with respect to the model size (number of parameters), data size (portion of the pre-trained data), and compute (epochs) tasks in VTAB-IK benchmark (1000 training example per task) the transfer setup.
45

---

Published as a conference paper at ICLR 2022
Table H.6: Results of VYT-B/32 on fine-tuning (transfer) setup on VITAB-IK benchmark, when pre-trinained with different head weight decays. Note that the selected head WD for these experiments are set to and 5.0, which are rather extreme values, to highlight the effect on different datasets.
Dataset
HWD=0.0
HWD=5.0
caltech101
0.89
0.91
cifar100
0.51
0.79
clevr-count
0.72
0.42
clevr-distance
0.65
0.49
dlabeeti-retinopathy-deettti
0.74
0.72
dmlab
0.42
0.36
dsprites-location
0.68
0.56
dsprites-orientant
0.58
0.58
dtd
0.66
0.72
eurosat
0.94
0.95
kitti
0.76
0.70
oxford-flowers 102
0.98
0.99
oxford-iitt-et
0.93
0.94
patch-camelyon
0.78
0.77
resise45
0.82
0.83
smallnorb-azimuth
0.27
0.22
smallnorb-elevation
0.47
0.36
sun397
0.42
0.65
swhn-cropped
0.72
0.60
VTAB-Natural
0.69
0.78
VTAB-Speecialzzed
0.82
0.82
VTAB-Structured
0.57
0.46
VTAB-ALL
0.69
0.68
46


Reviews:
[{"review_id": "xkBbg9UEyO", "reviewer": "Reviewer_pqjL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Key Strengths\n  1. This problem is very important, and these anlyses targeted to assess a key part of that problem.\n  2. The scale of these experiments is impressive, and their findings robustly motivated by said scale. It is hard to argue with the observed saturation effect in the context of how extensively you've profiled it. Similarly, the results you've found are I think impactful. You've clearly established that we _cannot_ use upstream accuracy as proxy for _downstream accuracy_, and quantified some other insights as well relating to task consistency. These should be impactful in future research, though some revisions could amplify that effect.\n  3. I think that this is a meta-analysis makes the work much stronger--one of my biggest concern at first skim of the paper was precisely that your models wouldn't be trained in a manner consistent with other models published during ordinary research, and thus that your results would be less generalizable. By largely relying on pre-published models and results, you totally alleviate that concern and strengthen your findings.\n  4. While I think you could do more here, I really like the representational analysis as well. Showing the consistency between saturation speed and layer preference both establishes some intuitive precedent for your findings and helps hint at some strategies we could take in the future to improve these results as well.\n\n### Key Weaknesses\n  1. Your biggest issue here is presentation. This presents itself in two ways. Firstly, you are _constantly_ referring to the appendix. If those appendix plots are so essential to your work, you need to find a way to incorporate them into the main body, or look for a different venue without page limitations (e.g., JMLR). Secondly, the flow of this paper somewhat hinders my ability to get to your key takeaways quickly. I like your introduction -- I think it does a good job of setting up the problem and framing what you're going to do, but I feel like you lose that framing as soon as you branch into other sections. I sort of find myself reading the paper by skipping from figure to figure and trying to back out what take-aways the figures represent from the text, when what you want is to guide the reader through your work with your text directly. To address this, I'd recommend reformulating the work slightly to have a more clear-cut structure, and generously sprinkle forwards and backwards pointers throughout your text and contextulaization sections to keep the reader constantly aware of where they are in your analysis and how that contributes to the big picture. You may also need to relegate some of your findings out of the main text, to secondary publications, or drop them entirely if you really want to crystallize your main take-aways for your readers.\n  2. Your work already speaks to the fact that for some tasks you can establish a strong power law fit even without using the full dense dataset you use in your experiments. However, in order for this take-away to be proactively useful in general, it would also be valuable to determine to what extent you can fit a power-law saturation curve to a new model with only a limited computational budget for pre-trained models -- e.g., how few runs and over how few UA data-points can you accurately assess the power-law fit? Additional analyses on these fronts (or clear callouts/explanations to your current findings on these fronts) would be very valuable. \n\n### Minor Weaknesses\n  1. I think you probably spend too much time on the randomized classifier section. I don't think those points are central to your paper, and I think this content could therefore be largely relegated to an appendix. \n  2. I think using accuracy is standard in the field, but I'd be further interested in other metrics like AUROC for these analyses too.\n  3. You should re-label your legend entry in Figure 1 for the asymptotic (@ UA = 1) performance value. I didn't find it clear from the legend entry alone and your caption for that figure is already very long.\n\n\n### Other comments\n  1. I think it'd be really impactful if you replicated this analysis on NLP models.\n", "summary_of_the_paper": "### What is the Problem / Question?\nThere is a ongoing trend in ML right now of exploring larger and larger pre-trained (PT) models. In general, increasing the scale of these PT models has yielded better performance on downstream tasks, though this is not universally the case. This work analyzes the limits of this trend in the computer vision space, investigating how downstream task performance is impacted by scale of the PT model among other factors.\n\n### Why is it impactful?\nPre-training large models is very expensive. This study has the potential to clarify when such large-scale PT is well-motivated, and when it isn't, which could yield significant benefits for future research.\n\n### Why is it hard? Why have previous approaches failed?\nPrevious approaches have examined this question at a dramatically smaller scale than this study does. For example, Kornblith et al., 2019 examined on the order of 10s of models, as opposed to thousands of models as examined here, and the results found here are correspondingly more detailed than those found in the Kornblith et al. study. _Most critically, the authors here present data that strongly suggests that rather than a linear relationship between downstream and upstream task performance, we should instead expect a saturating, non-linear trend_, which suggests that continually training larger and larger models may not be as worthwhile as was traditionally assumed. The fact that this counters an established result in the literature is an important distinction, because my native assumption would've actually been consistent with the author's finding, so without the literature bias already existing, this finding would in my opinion be less impactful. However, because it is updating a held belief in the published results, it is much more important.\n\n### How do they solve / answer it?\nThe authors profile a variety of models across different architectural choices, hyperparameter settings, etc. and compare upstream vs. downstream accuracy across various downstream tasks. They further examine a variety of additional angles within this study, including:\n  1) Quantifying the extent to which downstream performance is consistent across downstream tasks at a given level of upstream task performance,\n  2) How the internal representational space of these models can inform the extent of the saturation of the downstream task.\n  3) They examine in which cases upstream and downstream task performance may be at odds with one another\n  4) They profile their analyses under various dataset sizes, few-shot settings, and architectures.\n", "summary_of_the_review": "While I think addressing both key weaknesses would greatly strengthen the work, I think this is already an impactful and highly rigorous, technically sophisticated study which meets the bar for this venue, so I vote accept here.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "wLl1Txg2l4D", "reviewer": "Reviewer_2EkD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the high-level question that the paper asks makes sense, and becomes more important as models and datasets grow larger and larger, so that fewer organisations have the resources to handle them \n- the sheer size of the study makes it interesting, I would think that many of us are doubting whether some of our findings and beliefs from data- and compute-limited studies hold also for truly huge datasets and models beyond our resources\n- it is a clevert idea to analyse model performance in terms of a convex envelope, so as to better approximate the upper performance bound for a given model size and \"factor out\" architecture and hyper-parameter choices, and at least at first glance it makes a lot of sense\n\nWeaknesses:\n- concerns remain whether the \"hyper-modern\" types of models that are the focus of the paper are representative, I would argue that the majority of computer vision users still employ \"old-fashioned\" convolutional architectures rather than vision transformers or MLP-mixers, \nespecially in data- or compute-constrained scenarios\n-the narrow focus on low-shot scenarios and on image-level classification is somehow inconsistent with the \"real-world, large-scale\" aspirations of the paper; let's face it, at the application level outside the research lab, where transfer learning and dataset constraints really matter, one-label-per-image retrieval-type settings are the exception. The effect of large-scale pre-training  for more structured tasks like\nsemantic segmentation, detection, etc. would be a lot more relevant\n- the paper is overly dense  and overloaded. I am fine with an arbitrarily large and detailed appendix, but the paper itself tries to squeeze too many things into 9 pages. Graphs are so small they are hardly readable; explanatory text is somewhat hard to follow and parse, despite the not overly complicated message; countless references to the appendix make it practically impossible to read the main paper without the appendix; etc.", "summary_of_the_paper": "The paper presents an empirical study (respectively meta-study) of large-scale supervised pre-training for image recognition tasks. By analysing lots of experiments with varying model sizes, dataset sizes and training durations, the paper reaches the conclusion that simply scaling them up for a generic pre-training task will not lead to proportional gains for downstream tasks that build on the pre-trained model. On the contrary, the paper suggests that with increasing pre-training effort the downstream performance will reach a saturation level below its Bayes error. Moreover, that saturation level appears to vary across different downstream tasks (i.e., image characteristics and class definitions), which is seen as a sign that a generic one-fits-all feature extractor cannot be found by just scaling up.", "summary_of_the_review": "A brute-force, unashamedly empirical study of large-scale pre-training, albeit in a relatively specialised setting (low-shot, mostly with very recent, not yet widely established network types). The truly large size of the study means it will be interesting and potentially helpful for many people - we probably have to be grateful for such empirical studies, as very few would be able to replicate them, even if they are only \"aggregated\" and not \"trained for the purpose of the study\" (although it is perhaps a bad sign for the community that a small oligarchy of organizations now are the only ones who have that capacity). It is the sort of paper that does not strike me as particularly brilliant, but whose message will, and should, raise some eyebrows and spark discussions in the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cMD1ip2HX8f", "reviewer": "Reviewer_SKn8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well-written and easy to follow.\n\n2. Extensive experiments and analysis have been conducted, which makes it a strong technical report.\n\n3. Besides the saturating behavior, some insights drawn from the observations are good to be heard in the community, such as increasing the data density of upstream tasks, predicting downstream performance for a given upstream accuracy, and etc.\n\nWeaknesses:\n\n1. This paper is just focusing on the task of image recognition. It will be more beneficial to consider complicated visual tasks and other modalities and to investigate whether the same conclusion still holds. \n\n2. In terms of predicting downstream performance for a given upstream accuracy, can the authors compare the proposed method with those model selection techniques such as LogME [1]? Which technique is more practical for the real-world transfer tasks to select the best pre-trained model? \n\n[1] You, Kaichao, et al. \"Logme: Practical assessment of pre-trained models for transfer learning.\" International Conference on Machine Learning. PMLR, 2021.", "summary_of_the_paper": "This paper conducts extensive experiments on ViTs, MLP-Mixers and ResNets with up tp 10B parameters and evaluates transferability on more than 20 downstream tasks. It draws the conclusion that as we improve the performance of the upstream task, the performance of downstream tasks shows a saturating behavior.", "summary_of_the_review": "Overall, the reviewer tends to vote for accept since this work makes huge efforts in terms of experiments and some of the conclusions are beneficial to be heard in the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PSzFan1eKp", "reviewer": "Reviewer_3WYy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1). The relationship between (US, DS) accuracy is intuitively convincible especially in the high performance regions (Fig. 1), Appendix Table F.1, F.2 further shows that correlation values change drastically for different choices of (US, DS) tasks. Thanks author[s] for the extensive experiments.  \n\n2). The choice of hyperparameters is important for minimizing the discrepancy between US and DS performances. Appendix C and D show an case study of US weight decay in the projection layer of ViT on the effect of downstream DS accuracy, showing that at the expense of hurting US accuracy could instead lead to the improvement of DS tasks. This is counter-intuitive but technically sounds especially for high performing regions.\n\n3). The proposal for improving performance on the breadth of downstream task by adding upstream data diversity to (data size, model parameters, FLOPs) sounds plausible. It would be excellent if author[s] can give a concrete method on how to do it in future. \n\n\nWeakness:\n1). If there is any case study that showing US accuracy can accurately predict DS accuracy but (model size, US data size, compute) cannot? Compared to Figure 1, which conclusion cannot not be shown from Figure 4?\n\n2). From the study of head weight decay can we conclude that improving the feature extraction performance of lower layers lead to better generalization performance in ViT (see Figure 6, Appendix F.22, F.23)?\n\n3). In practice it costs long time to get a single US accuracy, if there is any proxy/surrogate way to make a cheap estimation? From Figure 1 I can see the curve is saturated only towards the end, which means an early prediction of US accuracy is not a reliable predictor of DS accuracy. In other words, how can we use power law prediction for model selection in practice (see Appendix Fig. F6, F7, F8, F9)?", "summary_of_the_paper": "This paper gives an important finding that overperformed upstream performance may not transfer well to better downstream performance especially in higher upstream accuracies region. The experiments are both empirically and theoretically evaluated, making the conclusion strong and reliable.", "summary_of_the_review": "This paper gives an interesting study on the prediction power of US accuracy. Extensive results are provided. The conclusions are solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a very large-scale study on the pretraining of image recognition models. Specifically, three scaling factors (model sizes, dataset sizes, and training time) are extensively investigated. One important phenomenon observed by this paper is that stronger upstream accuracy may not necessarily contribute to stronger performance on downstream tasks---actually sometimes these two types of performance could even be at odds with each other

Overall, all the reviewers enjoy reading this paper and highly appreciate the empirical results presented in this paper. There were only a few concerns raised by the reviewers but most were well addressed during the discussion period. All reviewers reach a consensus on accepting this paper and believe this study is worthy to be heard by the community.
Decision: Accept (Spotlight)