Revisiting Neural Scaling Laws in Language and Vision
Authors: Ibrahim Alabdulmohsin
Behnam Neyshabur
Xiaohua Zhai
Venue: NeurIPS 2022 Accept (2022)
Keywords: Scaling Laws
Computer Vision
Neural Machine Translation
Language Modeling
Big Bench

Abstract: The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and  language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain. 

Second, it has been used Revisiting Neural Scaling Laws in Language and Vision
Ibrahim Alabdulmohsin Google Research, Brain Team Zürich, Switzerland ibomohsin@googleecomm
Behnam Neyshabur
Google Research, Blueshift Team Mountain View, United States
neyshabur@google..cm com
Xiaohua Zhai
Google Research, Brain Team Zürich, Switzerland xzhai cahaiegoooge.coo com
Abstract
The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for more rigorous methodology based on the extrapolation loss, instead of reporting the best- fitting (interpolating) parameters. We then present recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.
1 Introduction
and the natural language processing (NLP) [ [1] domain. Recent work has found that scaling up the data size (34), the model size [26 (35) the training schedule [5] 39) or all of them together [8 40) often lead to improved performance. More importantly, scaling up the data size and the model size together can better utilize the compute resources. Scaling laws have been properly studied in several works, e.g.
Scale has led to innovative research both the vision domain 110 14, 26
35
40
I8 20 23), and has been found that the performance f fz) (e.g. excess loss) often follows power law f(z) ~ fx for some 3 > 0 and < 0 as one varies a dimension of interest such as the data or the model size.
18
While theoretical arguments alone seldom predict scaling law parameters in modern neural archi- tectures 2
has been observed that the benefit of scale could be predicted empirically 23 28 31] The general approach to acquire a learning curve, i.e. a collection of samples (xr, (x)). where a dimension of interest such the training data size while f(x) is a measure of performance, such as the validation loss. After that, parameters are estimated, e.g. by computing the best-fitting values of and in the model f(x) fxx.. Given the estimated scaling law parameters, one can then extrapolate by predicting f(x) for large values of x.
21
32
B. 4
18
20
22
Such learning curve extrapolation has found many applications, of which four seem to be more prominent. First, it offers a tool for understanding deep neural networks; e.g. how the architecture and data distribution impact scaling behaviors 18
30 32
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

10 10-shot enes
1.0
M1
1.0
M2
1.0
M3
1.0
MM (Ours)
Used
0.5
0.5
0.5
10'
10a
19f
10
10" Upstream Examples
10'
10"
10
Upstrcam Examptes
Upstream Examples
Exampits
Methed
Average
1.2
Image Casaliratton
Exnguage Modeling
BiG-Bench
Figure 1:
We introduce an estimator M4 (see Section 3 of scaling parameters that extrapolates more accurately from learning curves and compare it against previous methods denoted Mi, MM2, and
M3 (see Section 2 TOP: The y-axis is ImageNet 10-shot error rate while the rr-axis is the number
of examples in JFT-300M [4 seen during pre-training. The architecture is BiT/101x3 (26) (see
Section for further details). Values in amber are not seen when fitting the scaling law. BOTTOM:
Comparison across four domains. We report the fraction of time (y-axis, higher is better) in which method achieves the best extrapolation error in the given domain's 's tasks (see Section D Because
several methods may perform equally well in one task, average rankings do always sum one.
Third, can reduce the environmental footprint of experimentation by terminating experiments early and accelerating parammte search [33 20) 22 Forth, it has been applied in neural architecture search (NAS) [15] 20, 25). In addition, learning curve prediction offers different methodology for comparing performance; e.g. instead of comparing accuracy single dataset, one can also examine the full (hypothetical) scaling curves.
for sample size planning, particularly in data-scarce domains such medicine 4
17 28
However, in order to achieve such benefits in practice, it is imperative that scaling laws extrapolate accurately instead of merely interpolating the learning curve. To our knowledge, a validation of this sort based extrapolation is often lacking in the literature and previous works have generally reported the best-fitting (interpolating) parameters. We demonstrate why this can misleading in Section[4] 4 where we illustrate how the scaling exponent that extrapolates best can be quite different from the exponent that best fits the given (finite) learning curve. addition, we propose an estimator for scaling laws denoted M4. which extrapolates more accurately than previous methods as shown in Figure I
We validate the proposed estimator in several domains, including image classification, neural machine translation, language modeling, and other related tasks.
Our contributions are:
We argue in Section for more rigorous methodology to validate scaling law parameters based on extrapolation, instead of only reporting the best-fitting (interpolating) parameters.
2. We propose recipe in Section estimate scaling laws reliably from learning curves. The new estimator is verified across several domains: image classification, neural machine trans- lation (NMT), language modeling, and tasks from BIG-Bench evaluation benchmark (6].
3. We use the proposed recipe to study the impact of the neural architecture's 's type and size on scaling exponents.
4. We release benchmark dataset consisting of 90 tasks to accelerate research in scaling laws.
2
Related work
Power law scaling in deep neural architectures has been verified in a wide range of domains, including
image classification 2 20 32 40. language modeling (0 23 32.. NMT 3 20) and speech
recognition (20). To explain this theoretically, at least for data scaling, several works have argued

---

for a power law behavior under various contexts. For instance, in the universal learning setting 7 under the realizable case with a 0-1 misclassification loss, power law scaling emerges with exponent = 1 if the hypothesis space has an infinite I Littlestone tree but not an infinite VC-LLitleestone tree 77. Another argument for the exponent = 1 can made in the non-realizable setting the chosen loss is sufficiently smooth and the model size is limited by deriving the variance of the empirical solution around its population limit [2]. A more relevant setting for deep neural networks is to assume that the model size is effectively infinite and the loss is Lipschitz continuous (e.g. continuous Ioss in bounded domains). Under the latter assumptions, it has been argued that the scaling exponent would satisfy O(1/d) where d is the intrinsic dimension of the data manifold 22 21 (32, This is consistent
with the fact that scaling exponents often satisfy c < and that the exponent seems to be independent of the neural network architecture size as long as the architecture is sufficiently large [20 23 32]
Writing x for the dimension of interest (e.g- data size) and & for the error/loss of the model as a
function of x; three function classes have been used in the literature to model the performance &n as function of while capturing its expected power law behavior:
M1 The simplest model assumes a power law throughout the domain of Ex 3z" This
has been used, for example, to estimate the required sample size in healthcare (9] neural machine translation (NMT) [19], and language models [23]. among others 220
M2: : To capture saturating performance for large (i.e. when the Bayes optimal risk is bounded away from zero), parameter &oe added: Ez - E .zz.. This is, perhaps. the most
commonly used model in the literature; see for instance [
13
17
19
20
22
28
30 31.
M3: : A different parameterization has been recently used in NMT 3 Exr 3(x +T) where 3 >> 0,7 2 0 and e < O. Variants of this approach were used previously in studying, for example, scaling laws in vision transformers [40,, accelerating hyper-parameter optimization
13 and (more generally) in learning curve prediction 255
In this work, we introduce a fourth estimator M1 and verify experimentally that it outperforms the above methods in terms of its extrapolation capability in several domains, as summarized in Figure We describe M4 and discuss its rationale in Section 3
3 The Scaling Law Estimator Ma
Motivation. The function class M2, in which it is assumed that Ex = Eso + 3rx, captures (by definition) what it means for the excess risk to follow power law. Hence, question naturally arises: do we need any other function classes to estimate the scaling law parameters Eoo, 3 and c?
To see why using M2 can occasionally fail, consider the following simple classification problem whose optimal Bayes risk is known. Suppose that the instances x E $d-1 are generated uniformly at random from the unit sphere gd-1 = (x E Rd ||æ||| = 1}. In addition, let the (ground-truth) labeling function be given by:
v(x) = {- sign ((w", x)), with probability 1 - 5 sign(("", x)), otherwise,
for some fixed w* Sd-1_ If a classifier is trained using, for example, logistic regression, the misclassification error rate e. of the learning algorithm function of the data size would typically undergo three stages as illustrated in Figure 2 20). First, we have saturating performance for small sample sizes shown on the left, in which the trained model does not perform much better than random guessing. Second, we have a transitional stage in which the performance of the model improves quickly but it does not constitute power law yet. Third, we have the final power law regime in which the excess risk ε - Eoo fits power law curve.
Let Do {(x,, {x)} be the learning curve and write D, ((z, Ex): x 2 T} C Do for the restriction of the learning curve to 2 т. To extrapolate from learning curve, we train each of the four models M1. M2, M3. and M4 on the learning curve after applying a cutoff r to mitigate the effect of small data samples. Then, we plot the excess risk e e predicted by each model, where 8 is the (ground-truth) Bayes risk. Since ex is known exactly, an accurate model that extrapolates well would produce a linear curve in each plot. As shown in Figure 2. M2 is accurate only when the data resides entirely in the power law regime (rightmost figure), whereas M4 works well in all cases.
3

---

Cutoff 0
Cutoff 109
Cutoff 000
15"
Excoss
10-1
10-
H5, cm.8.12
H3, Co 0.17
K3. C- 0.12
M4, cm-8.57
M6. cm- 49
R6,
Cm-8 .49
18
10-2
15.
10
Data Size
Data Size
Data Size
Figure 2: The excess risk Ex E% is plotted against the training data size for logistic regression where instances x E Rd are drawn uniformly at random from the surface of the unit sphere and the label is binary y E [-1,+1} with noise rate (see Section 3). In this experiment, d 100 and 5 = 0.2. In each figure, only the data sizes that exceed the indicated cutoff value are used to estimate the scaling law parameters, M2 is accurate only when the data resides entirely in the power law regime (rightmost figure), whereas M4 works well in all cases.
Derivation. The function class M4 arises from several natural requirements. First, we would like function class be sigmoid-like like so that it fails only gracefully when the data deviates from the expected power law behavior; e.g. to avoid failures like that Mi and M2 in Figure(left) left). Second, would like function class F reduce power law functions ((x - Eoo Bx as x oo. More precisely, we require that:
log(f(x)= - log
vf FF 3c 8,<08>> 0: lim
To reiterate, this is because power law behavior has been empirically verified in wide range of domains (see Section 2). Third, we would like our function class to be expressive enough to contain all of the functions in MMg i.c. M2 C MM, that using M4 becomes equivalent to using M2 when the observed learning curve resides entirely in the power law regime.
If we take the first requirement above on the shape of the function, general approach to achieve this is to write the performance as a convex combination of the form:
E, = (())))))())))())-))) + +1+1 (1+)) E.o;
2
for some function 7(x) that satisfies lim,- ((x) and lim (xx) = co. Here, 6 is the predicted limiting performance when - while is the performance at the random-guessing level. To meet the second requirement, we set ((x) = 3x for some learnable parameters 3 > 0 and
< O. Rearranging the terms yields (Ez 6xx) (co exr) = ((x) = fxr.. Finally, we introduce learnable parameter a > 0 to meet our final requirement (see the ablation in Appendix A..I:
Ex Exc (co - E2)
Bx
(M₂)
With a = 0, our function class M4 reduces to M2 as required. By differentiating both sides of the equation above and noting that < O, we deduce that e, remains monotone decreasing function of x for all a 2 0 as expected. In addition, by rearranging terms and using both the binomial theorem and the Lagrange series inversion theorem, we have the following asymptotic expansion for the excess loss En Exo as r + DC:
Ex Eoc (Eo (Bx²) a(Eu 20 -a-
(3)
When = 0, recover the power law estimator M2 Setting 0 allows M4 to handle measurements that deviate from the power law behavior; i.e. when the learning curve does not fall into the asymptotic power law regime. The difference is 2(ax22) (suppressing other constants).
The parameters to fitted here are a 2 0.,3% >> 0, e < and ex > 0. The parameter én corresponds to the value of the loss at the random-guessing level and can be either fixed optimized. We fix so in our evaluation be equal to the loss the random- -uuesing level, although we observe similar results when it is optimized.

---

Loss Function. In this work, scaling law parameters in all the four function classes are estimated by minimizing the square-log loss, similar to the approach used in (28] This serves two purposes. First, it penalizes the relative loss and, hence, treats errors at all scales equally. Second, it allows us to compute subset of the parameters in closed-for form using least squares. Specifically, in M4. for example, we minimize:
((0,3, = =Ex (log(Es EXO) - log(Eo Ex) -log log =
(4)
In M2 we optimize the same loss above while fixing a = 0. In M1, we fix both a and foo to zero.
In M3 we optimize the following loss:
C33 7) = Ex [loge> - log - clog (x + +
(5)
In all function classes, we use block coordinate descent, where we compute a, 3 and c in closed form using least squares, and estimate the remaining parameters (if any) using gradient descent, with learning rate of 10-7. We repeat this until convergence.
4 Validating Scaling Laws using the Extrapolation Error
common approach in the literature for estimating scaling law parameters assume parametric model, e.g. MM,, and reporting its best-fitting parameters to an empirical learning curve (see for example the prior works discussed in Section Afterwards, patterns are reported about the behavior of the scaling law parameters; e.g. how the exponent varies with the architecture size. We argue, next, for a more rigorous methodology based on the extrapolation loss, instead only reporting the best-fitting (interpolating) parameters. Specifically, choices of scaling law parameters that achieve small interpolation error do not necessarily achieve a small extrapolation error so they may not, in fact, be valid estimates of scaling law parameters. Scaling law parameters should be validated by measuring how well they extrapolate.
To see why validation of this sort matters, consider the following example. If we pretrain a vision transformer ViT/B/16 [4] on subsets of JFT-300M (a proprietary dataset with 300M examples and 18k classes (34) using the Adam optimizer 24 and evaluate the 10-shot error rate on ImageNet- ILSRCV2012 []] we obtain the learning curve shown Fleft green). Evidently, power law emerges; i.e. ImageNet 10-shot error rate (shown in green) follows a linear curve on log-log plot. Hence, one might estimate, for example the scaling exponent using least squares.
However, consider now the family of curves shown in Figure left), all corresponding to M2 but with scaling exponents that vary from about e = -0.24 33 e.o). Note that all five curves overlap with each other significantly. Choosing the best fitting parameters on the learning curve would favor a scaling exponent of e -0.24 as shown in Figure Bright). However, if we validate the parameters by evaluating how well they extrapolate (i.e. how well they predict performance when the number of seen examples x > 10"), different picture emerges. We observe that a more accurate estimate of the scaling exponent is c = -0.40. This is shown in Figure left) and in Figure rright) by measuring the extrapolation loss. Here, validation measured using the root mean square error (RMSE) to the log-loss:
-0.4 (while fitting the parameters
RMSE / (xog log Ex)²
in which x uniform over the set [10", 2 x 10%) where éx is the predicted loss while & is the actual. We apply the logarithm that penalize the relative error and, hence, assign equal importance to all error scales (both large and small
In summary, scaling law parameters that give the best fit on the learning curve (i.e. lowest interpolation loss) do not generally extrapolate best. When using extrapolation loss instead, different scaling law parameters emerge. In this work, we use extrapolation to evaluate the quality of scaling law estimators.
witt a base learning rate of 5e-4, batch-size 4.096, and dropout rate of 0.1
'E.g. if we have single measurement where e.. 0.99 and the estimator predicts é. = 1.0, the RMSE in 6 is approximately equal 1 1%. Similarly, it is approximately equal to 1% when e= 0.099 while é O.
5

---

7x10
c=-0.24
c=-0.01
Criterion Interpolation
7x18-1
Nevertheless, we observe that prior to reaching peak performance, training examples behave as if they were fresh samples. This observation generalizes the bootstrapping phenomenon observed in Shot Error
cm-8.48
10-Shot Error
0m
6x10
10a
Extrapolation
New
RMSE
5x10
5x10-
4x10-3
4x18-1
10a
10²
10a
10'
8.0 -..2
-B.4
-8.6
-0,8
-1.8
Examples Seen
Examples Seen
Figure 3: In this experiment, ViT/B/16 14] is pretrained on JFT-300M 34], where the evaluation metric is 10-shot ImageNet emror rate. LEFT & MIDDLE: the learning curve is plotted. Different scaling exponents using the function class M2 can the learning curve almost equally well. Values in green correspond to the data used train the scaling law estimator (up to 500M seen examples) while values in yellow are used to evaluate the extrapolation loss. RIGHT: Best fitting parameters do not necessarily coincide with the scaling parameters that achieve small extrapolation loss.
Table 1: A list of the six architectures used in the evaluation study in the image classification domain.
Residual Nets
Vision Transformers
MLP Mixers
Model
# Parameters
Model
# Parameters
Model
Parameters
BIT/50/1
61M 494M
VIT/S/16 VIT/B/16
32M 110M
MiX/B/16 MiX/L/16
73M
BIT/101/3
226M
Experiments
We provide an empirical evaluation of the four scaling law estimators in several domains, including image classification (72 tasks), neural machine translation (5 tasks), language modeling (5 tasks), and other language-related evaluations (10 tasks). The dataset for neural machine translation is available at [3]. The code and dataset for the remaining tasks used in this evaluation are made publicly available facilitate further research in this domair In all experiments, we divide the learning curve into two splits: (1) one split used for training the scaling law estimators, and (2) one split used for evaluating extrapolation. Setting
Fmazr/2 22, where maar is the maximum value of x in the data, the first split is the domain E [0, r] while the second split is the domain E (r, 2r]. We measure extrapolation error using RMSE in (6) All experiments are executed on Tensor Processing Units (TPUs).
5.1 Image Classification
Architectures and Tasks. We use three families of architectures: (1) big transfer residual neural networks (BiT) (26). (2) vision transformers (ViT) [14), and (3) MLP mixers (MiX) [37]. For each family, we have two models of different sizes as shown in Table T in order assess the impact of the size of the architecture on the scaling parameters. We pretrain on JFT-300M (34). Since pre-training
task performance is not representative of the downstream performance 40 we evaluate the few-shot
accuracy downstream on four datasets: (1) ImageNet [T]] 1 (2) Birds 200 (38] (3) CIFAR 100 (27), and (4) Caltech (16). For each dataset, report 5/10/25-shot accuracy. It results in 72 tasks for the combinations of architecture, dataset, and metric. Following [4] 26), we removed duplicate
pre-training examples between upstream JFT-300M dataset and all the downstream train and test sets.
Bootstrapped Examples. In the few-shot image classification setting under the transfer learning setup, overfitting can occur if the upstream dataset is small, where training beyond a particular number of steps would reduce the downstream validation accuracy This is demonstrated in Figure where we pretrain on subsets of JFT-300M (upstream) and evaluate ImageNet 10-shot error (downstream).
*Code and benchmark dataset will be made available at: gog.leereaarra///e/////////r/err
wwt. wtt.//oogle---eeeerrr.ooooo lavs

---

(IFT), Bir,5/1, few shet 103
0.6
(JFT, /VI/S/1.. "lew shut 107
0.7
FIFI,, MXX8/15; (few shet 103
Accursyy
0.5
Acursoy 22220020000 000-0-2---0 24
10:1
12M
0.44
105M
12M
0.4
100M
12M
2004
002
203N
200M
3031
011
303M
100M
0.0
0.0
10t
Exemples Sewn
Examples Seen
Exemples Seen
(JPT, 011/102/,, Tee shot 10)
CJFT, VVit/azee 'ew shot 121
CJIT, "M00//16) TTey shot, 10")
0.6
303M
10
30²
3019
10a Examples Soon
16
Exampies Scon
Examotes Soon
Figure 4: In this experiment, we pretrain each vision architecture on subsets of JFT-300M 34 and report the 10-shot ImageNet accuracy (T]]. When the architecture is pretrained on subset of size 12M, we observe overfitting, where the performance hegins to drop if the model is pretrained for a large number of steps. Nevertheless, prior reaching peak performance, training examples behave as if they are fresh samples, which is consistent with the earlier observations reported in 29
MI
M2
M3
M4
1.0
VIT8/166
0.5
0.5
0.5
10a
10a
109
107
10
109
1.0
1.0
1.0
M0X011
0.5
0.5
0.5
0.5
10'
100
10f
10a
100
10a
10¹
10
10a
10'
10a
1.P
Examples Seen
Examples Seen
Examples Seen
Examples Seen
Figure 5: ImageNet 10-shot accuracy (y-axis) vs. the number of bootstrapped examples seen during upstream training in ViT/B/16 and MiX/I/16 (see Figure [ for BiT/101x3 and Appendix AA.3 for all remaining figures). The curves in each column correspond to the scaling law learned using the corresponding function class. The values marked in amber are reserved for evaluating how well the scaling law parameters extrapolate. Generally, M4 extrapolates better than previous methods.
291. where it showed that training examples behave as fresh samples prior convergence, which would be equivalent to our observation if no overfitting occurs. Throughout the sequel. we refer to the examples seen during training prior to peak performance "bootstrapped examples" and use their number as the independent variable x when evaluating the scaling law estimators in this section.
Figure Saare55jiluuuee iillustrate how well each of the the four scaling law estimators MM. M3. and M4 can extrapolate from a given learning curve. The complete set of figures is provided Appendix [A.3] We observe that M4 extrapolates better than other methods and produces learning curves that approximate the empirical results more faithfully. As shown in Figure [ M4 outperforms the other methods in more than 70% of the tasks in this domain.
Impact of the Architecture. F downstream task is nt-hot accuracy on ImageNet. We observe that within each family of models, larger models have more favorable scaling exponents. In addition, M4 yields estimates of the scaling exponents that larger in absolute magnitude than in other methods. Figure 7 shows that such differences in scaling exponents show up indeed in the slopes of the learning curves as expected.
the scaling exponent c in each architecture when the
5.2 Neural Machine Translation (NMT)
Next, we evaluate the scaling law estimators on NMT. We use the setup studied in B. in which models are trained with the pek-token cross-entropy loss using Adafactor optimizer 33 with a batch-size of 500K tokens and a dropout rate of 0.1 3 We use the encoder-decoder transformer models 61.61,, 28L.6l and 61.28I, where 281.6L. means that the architecture has 28 encoders and 6 decoders. We also use the two architectures: decoder-only with language modeling loss (D/LM) and the transformer-encodee with LSTM decoder (TE/LSTM). These correspond to the architectures used
7

---

Model
Model VIT
Model
0.00
0.00
0.00
-0.25
-0.50
-0.50
-0.50
-0.75
101/3
-0.75
5/16
8/16
-0.75
1/16
-1.00
-1.00
-1.00
10
15
20
25
10
15
20
25
10
15
20
25
Model
Model VIT
Model MiX
0.25
0.25
0 -0.50
-0.50
-0.50
-0.75
-0.75
-0.75
101/3
B/16
1/16
-1.00
-1.00
-1.00
10
15 20
25
10
15 20
25
10
15 20
25
Figure 6: The scaling exponent is plotted for each architecture when pretraining on subsets of JFT-300M and evaluating performance using n-shot accuracy on ImageNet, where is the æ-axis. Top row is for M2 (nearly identical in this case to Ma) while the bottom row is for M4. We observe that M4 suggests more favorable scaling behavior (larger estimates of the scaling exponents) than in previous methods. Also, larger architectures within the same family have more favorable exponents.
Excess Risk
BIT/50/1
6x
VIT/S/1
MIX/B/16
BIT/101/3
VIT/B/16
MiX/L/16
10-1-1
10-1
3
10
2
10a
10a
10a
10
10a
10a
Examples Seen
Examples Seen
Examples Seen
Figure The excess risk 10-shot ImageNet classification when pretraining on JFT-300M is plotted against the number of examples seen upstream. The slope of each curve is its scaling exponents
in Figure 1 in 33. In all cases, performance is measured using log- perpplexxxxty hold-out dataset. To evaluate the accuracy of the scaling law estimator, we fit its parameters on the given learning curve (for up to 256M sentence pairs) and use it to predict the log-perplexity when the architecture trained on 512M sentence pairs. Because the learning curves contain few points, we only evaluate on the 5 512M sentence pairs. Table displays Tableyd the RMSE of each estimator. Clearly, M4 performs better than the other methods as summarized in Figure 1 which is consistent with the earlier results in image classification.
2158e107
1.00
92
M3
1.00
N4
0.93
0.93
6,90
3.00
0.50
v1. 11.34108
3.0
1.0
0.0
0.9
32a
1214
1051
107
20t
100
20"
109
1211
101
125
P2422420
1.0
3.0
1.0
0.9
0.9
10PP
14.53e1 28
0.8
0.9
0.6
0.9
1214
10'
131
130
P1076:00
1.0
1.0
1.0
1.0
5.0
0.9
0.
12
1090
2011
10a
1244
2014
10a
20M
20th
191
Number of Tokens
Figure 8: We evaluate scaling law estimators on language modeling tasks (see Section various model sizes indicated the left side of each row. Table provides the RMSE scores.
5.3 with

---

Table 2: Extrapolation RMSE of the scaling law estimators on the five NMT tasks (top) and the five language modeling tasks. See Sections 5.2 and for further details.
Model
M1
M2
M3
M4
NMT
Enc, Layers
10"1
x 10
Enc. Layers
10
6 Enc, 28 Dec Layers
2.3 10
5.3 10
1.6 X 10
3.0
Decoder-only /LM
2.5 x
10
3.9 x 10
8.9 x 10
1.0 x
10
Transforer Enc /LSTM Dec
1.9 10
1.3 x 10
6.2 X 10
1.2
10
Language Modeling
1.68e.07
1.5 10
6.0 10-4
10
10 44
1.34e-08
0.6 10-2
1.7 10-3
6.6 3.0 10
1.4
10
2. 2..22-08
0.5 10
1.9 10
5.2 0.9 10
1.8
0.5
4. 08
1.7 10-2
7.4 10
6.6 3.8 10
7.5 -
5.
1.07 1.07e=09
1.7 0.4 10
1.7 0.3
4.5 0.4
1.3
0.2
10
-0.36
-0.02
-0,350
-0.03
-03375
-0.04
-0.400
-0.40
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e9
1e9
1e9
1e9
Figure Scaling exponent e plotted against the language model size. Unlike image classification with transfer learning (see Figure 6 |c] seems to decrease using M4 the model size increases.
Table 3: summary of BIG-Bench evaluation results using the extrapolation RMSE in 66 See Section 5.4 for further details. Both M3 and M4 perform best in this domain.
M2
M4
linguietic mappiiigg 1-shot Linguistic_mappinge 2-shot
1.6 10
1.6 10
1.6 10
1.7 10
1.7 - 10
1.2 10
1.7 10
9.2 1.4
qi.kiaaata:: I-shot
4.2 10-3
4.4 10
4.2 10-3
4.4
ge.wikidata: 2-shot
14 10-3
4. 10
14 10
1.7
unit_convarston:
8.1
1.5
0.
uoit_conversion: 2-shot
1.1 x0
11 10
7.5 10
1.2
vutt_data_urangl.ee 1-shot
1.1 1.110.. 10
1.1
1.1 10
1.3 1 0.3 10
muls_dera_.vanglling 2-shot
1.6 0.4
1.4044
16404x
6.2
late_understanding: I-shot
3.2 3.2 10
3.2 10
4.7 10-3
1 10" 1
date dit__understandinn 2-shot
29
18 10
18
5.3 Language Modeling
Next, we evaluate the scaling law estimators in language modeling, where the goal to predict the next token. We use the LaMDA architecture used in (36), which is decoder-only transformer language model. Five model sizes are used, ranging from 10 to 10 model parameters. In each model, we rescale validation loss to the unit interval [0, 1]. Figure g and Table summarize the results. We observe that M4 and Ma perform best, with M, tending perform better. As stated earlier, M4 becomes equivalent to M2 when the learning curve resides entirely in the power law regime, hence the similar performance. Figure ddisplays the scaling exponents e predicted by each estimator as function of the architecture size. We observe that Mi and M3 produce estimates of that are small absolute magnitude. However, in Ma and M4 the scaling exponent close to -1 1133 and decreases (in absolute magnitude) for larger models.

---

5.4 Scalable Tasks from the BIG-Bench Evaluation Benchmark
Finally, we evaluate the scaling law estimators on language tasks from the BIG-Bench collab- orative benchmark [6]. Here, we pretrain 262M-parameter decoder-only transformer (middle architecture in Figure X on language modeling and evaluate its I-shot and 2-shot capabilities in five language-related tasks. We choose the five tasks that exhibit the highest learnability from the benchmark (i.c. improvement in performance when pretrained on language modeling, see [6] for details). The five tasks are: Linguistic_mapei mapphings, qa_ va_wikidata, unit_ conversion, la__ddattawaagllll
and date_ aaate__nddsttadding We use the benchmark's preferred metrics in all cases, which is either "multiple choice grade" or "exact string match" depending the task. Table Bland Figure summarize the results. In this evaluation, both M3 and M4 perform best and equally well. In addition, we observe that Mi and M2 perform equally well and consistently worse than the other methods. One possible reason is that the learning curves are quite noisy (see Appendix A.2)
6 Discussion
The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer training schedules. Several works observe that the benefit of scale can be predicted empirically by extrapolating from learning curves and this has found important applications, such as in sample size planning and neural architecture search. However, to achieve such benefits in practice, it is imperative that scaling laws extrapolate accurately. We demonstrate that scaling parameters that yield the best fit to the learning curve do not generally extrapolate best, thereby challenging their use valid estimate of scaling law parameters. Hence, we argue for a more rigorous validation of scaling law parameters based on the extrapolation loss. In addition, accurately than in previous works, which we verify in several state-of-the wide range of domains. To facilitate research in this domain, we also release benchmark dataset comprising of 90 evaluation tasks. We believe that the proposed scaling law estimator can be utilized, for example, to accelerate neural architecture search (NAS), which we plan to study in future work.
present recipe for estimating scaling law parameters that extrapolates more architecture across a
Acknowledgements
The authors would like acknowledge and thank Behrooz Ghorbani for his feedback on earlier drafts of this manuscript as well as Ambrose Slone and Lucas Beyer for their help with some of the experiments. We also would like thank Daniel Keysers and Olivier Bousquet for the useful discussions.
References
[1] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint arXiv:21 10.02095, 2021.
[2] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.061 2021.
[3] Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in NMT: The effect of noise and architecture. arXiv preprint arXiv:2202.01994 2022.
[4] Claudia Beleites, Ute Neugebauer, Thomas Bocklitz, Christoph Krafft, and Jürgen Popp. Sample size planning for classification models. Analytica chimica acta, 760:25-33, 2013.
[5] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher patient and consistent, arXiv preprint arXiv:2106.05237, 2021.
[6] BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabil- ities of language models. In preparation, 2021.
10

---

[7] Olivier Bousquet, Steve Hanneke, Shay Moran, Ramon Van Handel, and Amir Yehudayoff. A theory of universal learning. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 532-541, 2021.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 33:1877-1901, 2020.
[9] Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. How much data is needed to train a medical image deep learning system achieve necessary high accuracy? arXiv preprint arXiv: 1.06488, 2015.
[10] Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS, 19933965-397,,, 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[12] Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18X0v:/810.0000 04805, 2018.
[13] Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperpa- rameter optimization of deep neural networks by extrapolation of learning curves. In IJCAI, 2015.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16
words: Transformers for image recognition at scale. ICLR, 2020.
[15] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. JMLR, 20(1):1997-201 2019.
[16] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004.
[17] Rosa L Figueroa, Qing Zen--Treitler, Sasikiran Kandula, and Long H Ngo. Predicting sample size required for classification performance. BMC medical informatics and decision making. 12(1):1-10, 2012.
[18] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740, 2021.
[19] Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In EMNLP, pages 5915-5922, 2021.
[20] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arX1v:1712.00409, 2017.
[21] Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021.
[22] Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 455, Melbourne, Australia, 2018. Association for Computational Linguistics.
[23] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arxiv preprint arXiv:2001.0836, 2020.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1xXiv.141..6980, 2014.

---

[25] Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In ICLR, 2017.
[26] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In ECCV, pages 491-507, 2020.
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[28]
Sayan Mukherjee, Pablo Tamayo, Simon Rogers, Ryan Rifkin, Anna Engle, Colin Campbell, Todd R Golub, and Jill P Mesirov. Estimating dataset size requirements for classifying dna microarray data. Journal of computational biology, 10-)):19-142,, 2003.
[29] Prectum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. ICLR, 2020.
[30] Jonathan Rosenfeld. Scaling laws for deep learning. arXiv preprint urXiv:2108. 076866 2021.
[31] Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization enror across scales. arXiv preprint vrXiv:190912666 2019.
[32] Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. JMLR, 1339):1-34, 2022.
[33] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML, 2018.
[34] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference
on computer vision, pages 843-852, 2017.
[35] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In /CML, pages 6105-6114, 14,, 2019.
[36] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 1082399 2022.
[37] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MLP-mixer: An all-MLP architecture for vision. NeurIPS, 34, 2021.
[38] Welinder, Branson, T. Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
[39] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint 10.0046, 2021.
[40] X Zhai, A Kolesnikov, N Houlsby, and L. Beyer. Scaling vision transformers. arXiv preprint 0465:2 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [N/A] We are aware of many appli- cations using learning curve extrapolation, and plan to apply our new recipe to new applications (c.g. NAS) to land the impact.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We provide a study of scaling laws and an improved estimator of scaling law parameters. We do not anticipate any negative socictal impacts of this work.
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] We plan to release the code and a benchmark dataset to facilitate research in this direction. Some of the datasets used in our experiments are proprietary and cannot released, such as JFT-300M. We also include experiments on publicly available datasets, such as Big-Benn Big-Bench for reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We describe the data, training procedure and architectures in details. See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Please see
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments are executed in Tensor Processing Units (TPUs). See Section 5
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes] We cite all the datasets and architectures we use.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [Yes] We plan to release code and a benchmark dataset.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "j0SNfh649W8", "reviewer": "Reviewer_am89", "summary": "The authors propose an extrapolation-based fitting for better estimating the scaling properties of deep learning models (e.g., the larger model lowest Val/Test loss). The author compares interpolation methods, e.g.,  power-law fitting, with the proposed extrapolation method on different tasks image classification, NMT, language modeling, tasks from BIG-Bench, and architecture type and sizes. The results show that the proposed extrapolation method is more accurate in predicting the model's performance. ", "questions": "- The loss shown in the appendix is quite important to highlight the difference between different methods. Why not include it in the main paper?", "limitations": "No this is missing in the paper.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The experiments to support the claims are extensive and explore multiple modalities and architectures. \n- The paper is well written, well-motivated, and easy to follow. \n- Achieving better prediction in terms of model scaling (e.g., size, data size) is important in many scenarios, once for all neural architecture search. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dnBZRAc1giz", "reviewer": "Reviewer_frKW", "summary": "This paper argues that predicting the benefit of scale should be conducted based on learning the learning curve 'extrapolation' rather than 'interpolation'. In addition, based on this, this work proposes an estimator model for the scaling laws, which extrapolates more accurately than previous methods, and in various experiments, the proposed estimator indeed shows better performance.", "questions": "- Isn’t it natural to derive the $\\mathcal{M}_4$ based on the empirical observations rather than the synthetic experiment as in Section 3?\n- I’m not sure why the first experiments were conducted in a few-shot setting instead of the pre-training task itself.\n- In figure 3, does the x-axis, examples seen, means the number of data examples that are fed during training instead of the size of the dataset? Then, is it the realistic experimental setting where the interpolation region is set around $3\\cdot10^8$ examples seen, which is the same as the number of data samples of JFT-300M?\n- The observation that the $\\mathcal{M}_4$ tends to yield estimates of the scaling exponents that are larger in absolute magnitude than in other methods means what?\n- I’m confused that, for example in Figure 3, shouldn’t the difference in $c$ be reflected as different linear functions with different slopes rather than different curves in log-log plots?", "limitations": "This paper does not address the limitations and considerations about the negative societal impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- I believe the task dealt with in this paper will be able to give a significant impact in the era where scaled-up DNN models show surprising performance.\n- To support the claim of this work, extensive experiments with various models, datasets, and experimental environments are conducted.\n\n### Weaknesses\n\n- There are several concerns about the experiments and I listed the questions below.\n- I guess the derivation of $\\mathcal{M}_4$ arises from the motivation part, and the motivation part says an accurate model that extrapolates well would produce a linear curve, however, I was not able to get why it is true.\n- It was a little confusing when I first read the paper, so I recommend providing the range of $\\beta, c$ in the Introduction to help readers imagine the shape of the graphs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bohDsjp17O1", "reviewer": "Reviewer_7qUq", "summary": "The paper contributes a solid work with a new recipe to estimate scaling laws reliably from learning curves, which outperforms previous work on a variety of tasks. The paper also shows that in order to validate scaling law parameters, we should base on extrapolation of data points instead of interpolation of data points. The paper also contributes a benchmark dataset consists of 90 tasks, which I believe it is an important contribution to the community.\n\n", "questions": "(see the strengths and weakness)", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The paper is well-written for most parts.\n- The new recipe is ingenious to me.\n- The experiment section is solid. It covers a lot of different tasks with clear results supporting the benefits of using M4, the proposed method.\n- Overall the paper contains materials that are very nice contribution to the field.\n\nCons:\n- I don't think it is clear in the paper while the new recipe works that well compared to M2. Is it because M4 contains M2 as a special case or is it because M4 has a sigmoid like shape, which avoids failures of M2 as in Figure 2 left? I think this needs to be elaborate more.\n\n- It is also not clear what exactly it means to be \"extrapolation\". In the introduction, the authors said that: ``Given the estimated scaling law parameters, one can then extrapolate by predicting the performance f(x) for large values of x\". However, in the experiment section, it was explained that: \"we divide the learning curve into two splits: (1) one split used for training the scaling law estimators, and one split used for evaluating extrapolation performance. We set the cutoff between the two splits to be equal to xmax/2, where xmax is the maximum value of x\". With that, I am not sure the second split, which used for evaluating extrapolation performance reflects what was stated in the statement (i.e. large values of x).\n\nNote: Equation 3 is the one that I could not follow. Perhaps a more explanation on this would help readers a lot.\n\nTypo:\n\nof which four seem to be more prominent -> four seem???\n\nThis is consistent with the fact that scaling exponents often satisfy c -> satisfies?\n\n-------\nUpdate: I wanted to thanks authors for answering my questions. I still have an issue with extrapolation and hopefully the authors will put some more efforts on explaining that in the paper. However, my overall impression is that the recipe is ingenious to me and it seems to work. So I just raised my score from 6 to 7. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept