Broken Neural Scaling Laws
Authors: Ethan Caballero
Kshitij Gupta
Irina Rish
David Krueger
Venue: ICLR 2023 poster (2023)
Keywords: Scaling Laws
Scaling
Scale
Big Learning
Deep Learning
Artificial Neural Networks

Abstract: We present a smoothly broken power law functional form (referred to by us as a broken neural scaling law (BNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as the amount of compute used for training, number of model parameters, training dataset size, or upstream performance varies) for various architectures and for each of various tasks within a large and diverse set of upstream and downstream tasks, in zero-shot, prompted, and fine-tuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, robotics, out-of-distribution (OOD) generalization, continual learning, uncertainty estimation / calibration, out-of-distribution detection, adversarial robustness, molecules, computer programming/coding, math word problems, arithmetic, unsupervised/self-supervised learning, and reinforcement learning (single agent and multi-agent). When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models and extrapolates scaling behavior that other functional forms are incapable of expressing such as the non-monotonic transitions present in the scaling behavior of phenomena such as double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. See arXiv for longer version of this paper. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws

Published as a conference paper at ICLR 2023
BROKEN NEURAL SCALING LAWS
Ethan Caballero
Kshitij Gupta
Mila, McGill University
Mila, University of Montreal
ethan. victor. waw.aaerremmeeeillll........ com ethan. www.aaaalalllllaaauuuuueee quebec
Irina Rish
David Krueger University of Cambridge
Mila, University of Montreal
ABSTRACT
We present a smoothly broken power law functional form (referred to by us as a broken neural scaling law (BNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies the amount of compute used for training. number of model parameters, training dataset size, or upstream performance varies) for various ar- chitectures and for each of various tasks within large and diverse of up- stream and downstream tasks, in zero-shot, prompted, and fine-tuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, Al alignment, robotics, out- of of-distribution (OOI) generalization, continual learning, uncertainty estimation calibration, out-of-distributie detection, adversarial robustness, molecules, com- puter programming/codingg math word problems, arithmetic, unsupervised/sell- supervised learning, and reinforcement learning (single agent and multi-agent). When compared to other functional forms for neural scaling behavior, this func- tional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models and ex- trapolates scaling behavior that other functional forms are incapable of express- ing such as the non-monotonic transitions present in the scaling hehavior of phe- nomena such double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this func- tional form to glean insights about the limit of the predictability of scaling be- havior. See arrXiv for longer version of this paper. Code is available at httpss //github. wwt.///oeeettttteeeeeeeee
sca liaallingllllll laws
INTRODUCTION
The amount of compute used for training, number of model parameters, and training dataset size of the most capable artificial neural networks keeps increasing and will probably keep rapidly increas- ing for the foresceable future. However, no organization currently has direct access to these larger resources of the future; and it has been empirically verified many times that methods which perform best at smaller scales often are no longer the best performing methods at larger scales (e.g., one of such examples can be seen in Figure 2 (right) of Tolstikhin al. (202 To work on, identify, and steer the methods that are most probable to stand the test-of-time as these larger resources come on- line, one needs way to predict how all relevant performance evaluation metrics of artificial neural networks vary in all relevant settings as scale increases.
Neural scaling laws (Cortes et al. 1994 Hestness et al. 2017: Rosenfeld et al. 2019 Kaplan et al., 2020 Zhhii al., 2021 Abnar et al., 2021 Alabdulmohsin et al.) 2022 Brown et al. 2020) aim to predict the behavior of large-scale models from smaller, cheaper experiments, allowing to focus on the best-scaling architectures, algorithms, datasets, and so on. The upstream/in-diste -iditiiiiiiittiittiionnn test loss typically (but not always!) falls off as power law with increasing data, model size and compute. However, the wwtpssrrammoouooo-dsriitittettii
performance, and other evaluation metrics of interest (even -iitaamminnisttiiuttionn evaluation metrics) are often less predictable, sometimes exhibiting inflection points (on a linear-linear plot) and non-monotonic behaviors. Discovering universal scal- ing laws that accurately model a wide range of potentially unexpected behaviors is clearly important not only for identifying that which scales best, but also for AI safety, as predicting the emergence of novel capabilities at scale could prove crucial to responsibly developing and deploying increasingly advanced AI systems. The functional forms of scaling laws evaluated in previous work are not up to this challenge.

---

Published as a conference paper at ICLR 2023
One salient defect is that they can only represent monotonic functions. They thus fail model the striking phenomena of double-descent (Nakkiran et al., 2021). where increased scale temporarily decreases test performance before ultimately leading to further improvements. Many also lack the expressive power to model inflection points (on linear-linear plot), which can be observed empir- ically for many downstream tasks, and even some upstream tasks, such task, or the modular arithmetic task introduced by Power et al. (2022) in their work on "grokking".
N-digit arithmetic
To overcome the above limitations, we present broken neural scaling laws (BNSL) functional form that generalizes power laws (linear in log-log plot) to "smoothly broken" power laws, í.e. smoothly connected piecewise (approximately) linear function in a log-log plot. An extensive empirical evaluation demonstrates that BNSL accurately model and extrapolate the scaling behaviors for various architectures and for each of various tasks within large and diverse set of upstream and downstream tasks, in zero-shot, prompted, and fine-tuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning. contrastive learning, AI alignment, robotics, out-of-distribution generalization, continual learning, uncertainty estimation calibration, out-of-distribution detection, adversarial robustness, molecules, computer learning, and
programming/coding math word problems, arithmetic, reinforcement learning (single agent and multi-agent). When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that considerably more accurate on this set. It captures well the non-monotonic transitions present in the scaling behavior of phenomena such as double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic.
THE FUNCTIONAL FORM OF BROKEN NEURAL SCALING LAWS
Decomposition of BNSL into Power Law Segments
103
Performance Evaluation Metric
102
101
BNSL
Break 1
Break 2
100
Break
Segment
10-1
Segment 2
Segment 3
10-2
Segment 4
Limit
10-3
103
107
10
10
10
10
Quantity Being Scaled
Figure 1: A Broken Neural Scaling Law (BNSL) (dark black solid line) (with 3 breaks where purple dotted lines intersect with dark black solid line) that contains 4 individual power law segments (where the dashed lines that yellow, blue, red, and green overlay the dark black solid line). The 1st and 2nd break are very smooth; the 3rd break is very sharp. See Section 2 for more details.
The general functional form of broken neural scaling law (BNSL) is given follows:
7
1/34
-e-eft
y = a
bx
co
II
de
where represents the performance evaluation metric (e.g. prediction error, cross entropy, cali- bration error, AUROC, BLEU score percentage, reward, Elo rating, or FID score) (downstream or upstream) and x represents a quantity that is being scaled (e.g. number of model parameters, amount of compute used for training, training dataset size, or upstream performance). The remaining parameters b, Co, C1...Cas dj dn. fi...fn unknown constants that must be estimated by fitting the above functional form to the (ær, 3) data points. (In our experiments, SciPy curve-fitting library /Virtanen al. 2020) was used.)
2

---

Published as a conference paper at ICLR 2023
The constants in equation I are interpreted as follows. Constant nt represents the number of (smooth) "breaks" (i.e. transitions) between n + consecutive approximately linear (on a log-log plot) seg- ments, for a total of + approximately linear segments (on a log-log plot); when n = 0, equation 1 becomes y = a + bx- Constant a represents the limit as to how far the value of (perfor- mance evaluation metric) can be reduced (or maximized) even if (the quantity being scaled) goes to infinity. Constant b represents the offset of functional form on a log-log plot (analogous to the intercept in y mz + on a linear-linear plot). Constant co represents the slope of the first ap- proximately linear region on a log-log plot. Constant y; represents the difference in slope of the (i)th approximately linear region and (i + 1)th approximately linear region on a log-log plot. Constant d, represents where on the x-axis the break between the (i)th and the (i+ 1 1tth approximately linear region (on log-log plot) occurs. Constant Ji represents the sharpness of break between the (i)th and the (i + 1 1)th approximately linear region on log-log plot; smaller (nonnegative) values of fa yield a sharper break and intervals (before and after the (i)th break) that are more linear on a log-log plot; larger values of fi yield a smoother break and intervals (before and after the (i)th break) that are less linear on a log-log plot.
are
For mathematical analysis and explanation of why Equation is smoothly piece-wise (approxi- mately) linear function on a log-log plot, see Appendix A.I For mathematical decomposition of Equation D intt the power law segments it is composed of (e.g. in Figure D see Appendix A..2
Note that, while intuition for using such smoothly connected approximately piece- linear (in log-log plot) function was that, with enough segments, it could fit well any smooth univariate scaling function, it remained unclear whether BNSL would also extrapolate well; yet we demonstrate below, it extrapolates quite accurately. Additionally, we find that the number of breaks needed to accurately model an entire scaling behavior is often quite small.
RELATED WORK
To the best of our knowledge, Cortes et al. (1994) was the first paper model the scaling of multi- layer neural network's 's performance as a power law (also known as scaling law) (plus a constant) of
the form y = ax" + c in which x refers to training dataset size and y refers to test error; we refer to that functional form as M2. Hestness et al. many orders of magnitude. Rosenfeld et al. (019) demonstrated that the same functional form, M2, applies when x refers to model size (number of parameters). Kaplan et al. (2020) brought "neural" scaling laws to the mainstream and demonstrated that the same functional form, M2, applies when refers to the amount of compute used for training. Abnar et al. (202I) proposed to use the same functional form, M2, to relate downstream performance to upstream performance. Zhai et al. (202I) introduced the functional form y = a(x d) + c, (referred to by us as M3) where d represents the scale at which the performance starts to improve beyond the random guess loss (a constant) and transitions to power law scaling regime. [Alabdulmohsin et al.] (2022) proposed functional form (y - -xe))/ < )) = bx. (referred to by us as M4) where 6 is irreducible entropy of the data distribution and eo is random guess performance, for relating scale to performance and released scaling laws benchmark dataset that we use in our experiments.
(017)) showed that the functional form, M2, holds over
Hernandez al. (2021) described smoothly broken power law functional form (consisting of constants after reducing redundant variables) in equation 6. of their paper, when relating scale and downstream performance. While this functional form can be summed with an additional constant representing unimprovable performance to obtain a functional form whose expressivity is equivalent to our BNSL with a single break, it important to note that (i) Hernandez et al.) (2021) describes this form only in the specific context, when exploring how fine-tuning combined with transfer learning scales as a function of the model size- thus, their functional form contains a break only with respect to number of model parameters but not with respect to other input quantities which we do explore such as dataset size, amount of compute, and upstream performance; (ii) Hernandez et al.) (2021) mentioned this equation in passing and as result did not try to fit or verify this functional form on any data; (iii) they arrived at it simply via combining the scaling law for transfer (that was the focus of their work) with scaling law for pretraining data; (iv) they did not identify it as a smoothly broken power law, or note any qualitative advantages of this functional form; (v) they did not discuss the family of functional forms with multiple breaks
Finally, we would like to mention that smoothly broken power law functional forms, equivalent to equation [ are commonly used in the astrophysics literature (e.g. dam (017)) they happen to model well a variety of physical phenomena. This inspired us to investigate their applicability to a wide range of deep neural scaling phenomena well.
3

---

Published as a conference paper at ICLR 2023
4 THEORETICAL LIMITATIONS OF PREVIOUSLY PROPOSED SCALING LAWS
Our of BNSLs is inspired by the observation that scaling is not always well predicted by simple power law; nor are many of the modifications which have been applied in previous works sufficient capture the qualitative properties of empirical scaling curves. Here show mathematically two qualitative defects of these functional forms:
1. They are strictly monotonic (first-order derivative does not change its sign) and thus unable fit double descent phenomena.
2. They cannot express inflection points (second-order derivative does not change its sign), which are frequently observed empirically. An exception this is M4, proposed by Alabb dulmohsin et al. 2022).
Note that these functional forms can exhibit inflection points on the log-log axes which are com- monly used for plotting scaling data (as was observed several prior works). However, for inflec- tion points linear-linear plot, the extra expressiveness of broken neural scaling laws appears to be necessary (and sufficient). Figure [ and Figure 4 provide examples of BNSLs producing non- monotonic behavior and inflection points, respectively, establishing the capacity of this functional form to model these phenomena that occur real scaling behavior.
name
f(x)
f'(x)
f (x:
M1
ax
abx"-1
ab(b 1)r²~2
M2
ax
+ in (1++)))(4+11)))
abah-1
ab(b 1)x²
M3
a(x 1 + d)
ab
abæeb (b-( 2) + dz)( (b 2dx)
Table 1: Previously proposed functional forms MI, M2, M3 and their (first and second order) derivatives. See Equation 2 for M4.
M1, M2, M3 functional forms cannot model non-monotonic behavior or inflection points: First, recall that expressions of the form m² can only take the value 0 if m 0. We now examine the expressions for the first and second derivatives of M1, M2, M3, provided in Table [ and observe that they are all continuous and do not have roots over the relevant ranges of their variables. i... x > 0 in general and < 0 in the case of M3 (we require æ > O because model size, dataset size, and compute are always non-negative). This implies that, for any valid settings of the parameters a, b, c, d, F, these functional forms are monotonic (as the first derivative never changes sign), and that they lack inflection points (since an inflection point must have fx)x)) 0).
M4 functional form cannot model non-monotonic behavior. The case of M4 is a bit different, since the relationship between y and æ in this case is expressed an inverse function, i.e.
1/c
x = gy) = b(co - y)m
-- 600
However, non-monotonicity of y as an inverse function y = y (x) ruled out, since that would imply two different values of x: g(y) can be obtained for the single value of this is impossible, since f(y) maps each y deterministically to a single value of æ. As a result, M4 cannot express non-monotonic functions.
M4 functional form can model inflection points. It is easy to see that if = -g-((x))) had an inflection point, then g(y) would have it well. This is because inflection point is defined as a point x where (x) changes from concave to convex, which implies that g changes from convex the point(s) at which this change occurs. Using Wolfram Alpha[aam and matplotlib (Hunter 2007). we observe that
concave, since the inverse of convex function concave; the root(s) of q²(v)
M4 is able to express inflection points, e.g. (a, b. co. 6. v) = 1, 10-2,,3 2/4, 1/4, 1/ 1/v3,5, 5/8). or (a, b, C, 60. 6oo, x. v) -,,
2/3, 1/3, 55/6 23/2)222,1/////// 1/v3).
https:/wwww.ooffraaalpp..ccmm
4

---

Published as a conference paper at ICLR 2023
5 EMPIRICAL RESULTS: FITS AND EXTRAPOLATIONS OF FUNCTIONAL FORMS
We now show the fits and extrapolations of various functional forms. In all plots here and onward and in the appendix, black points are points used for fitting a functional frrm, green points are the held-out points used for evaluating extrapolation of functional form fit to the black points, and a red line is the BNSL that has been fit to black points. 100% of the plots in this paper here and onward and in the appendix contain green point(s) for evaluating extrapolation. Please refer Appendix Section A.6 for further experimental details fitting BNSL.
Except when stated otherwise, each plot contains single break of BNSL fit to black points that are smaller (along the x-axis) than the green points.
All the extrapolation evaluations reported in the tables are reported in terms of root mean squared log error (RMSLE) + root standard log error. See Appendix A.3 for definition of RMSLE and Appendix
A.4 for definition of root standard log error.
Domain
M1 1
M2 1
M3 1
M4 1
BNSL
Downstream Image Classification Language
2.78%
4.17%
9.72%
13,89%
69.44 %
10%
5%
10%
0%
75%
Percentage of tasks by domain where each functional form is the best for extrapolation of scaling behavior. Numbers for M1, M2, M3, and M4 were obtained via correspondence with authors of Alabdulmoohsin et al. (2022). See Sections 5.1 and 5.2 for more details.
Table 2:
5.1 VISION
Using the scaling laws benchmark of Alabbuulmossin al. (2022). we evaluate how well various functional forms extrapolate performance on vision tasks as training dataset size increases. In this large-scale vision subset of the benchmark, the tasks that are evaluated are error rate on each of various few-shot downstream image classification (IC) tasks; the downstream tasks are: Birds 200 (Welinder et al. 2010), Caltech101 (Fei-Fei et al. [2004), CIFAR- 100 (Krizhevsky et al. (2009), and ImageNet (Deng et al. (200)) The following architectures of various sizes pretrained on subsets of JFT-300M (Sun et al. 2017): big-transfer residual neural networks (BiT) (Kolesnikov et al.,
2020 MLP mixers (MiX) (Tolstikhin et al. 2021), and vision transformers (VTT) (Dosovitskiy 2020). As can be seen in Tables and 3 yields extrapolations with the lowest RMSLE (Root Mean Squared Logarithmic Error) for 69.44% of tasks of any of the functional forms, while
et al.
the next best functional form performs the best on only 13.89% of the tasks.
To view plots of BNSL on each of these tasks, see figures 24 25 26 30 in Appendix A.26 To view plots of M1, M2, M3, M4 on each of these tasks, see Appendix A.4 of Alabdulmohsin et (0222).
we additionally show that BNSL yields accurate extrapolations of performance on large-scale downstream vision tasks when amount of compute used for (pre-)training is on the x- axis and compute is scaled in the manner that is Pareto optimal with respect the performance evaluation metric on the y-axis (downstream accuracy in this case).
In Section A.8
In Section A.9 we additionally show that BNSL yields accurate extrapolations of the scaling behav- ior of diffusion generative models of images when amount of compute used for (pre-) prre-itrannin is on the x-axis and compute is scaled the manner that is Pareto optimal with respect to the performance evaluation metric on the y-axis (NLL and FID score in this case).
In Section A0 we additionally show that BNSL yields accurate extrapolations of the scaling be- havior of generative models of video.
In Section A.20
we show that BNSL yields accurate extrapolations of robotics scaling behavior
fout-of-distributiin generalization and in-distribution generalization).
In Section A.19
BNSL accurately extrapolates the scaling behavior of continual learning.
In Section A.15
BNSL accurately extrapolates the scaling behavior of adversarial robustness.
In Section A.24
we show that BNSL accurately extrapolates the scaling behavior of the downstream
performance of multimodal contrastive learning (i.e. gon-eeenrteeive unsupervised learning).
In Section A. we additionally show that BNSL, yields accurate extrapolations of the scaling be- havior when data is pruned Pareto optimally (such that each point along the x-axis uses the subset of the dataset that yields the best performance (y-axis value) for that dataset size (x-axis value)).
In Section A.12 we additionally show that BNSL yields accurate extrapolations when upstream performance on the x-axis and downstream performance is the y-axis.
In Section A.7 we additionally show that BNSL accurately extrapolates to scales that are an order of magnitude larger than the maximum (along the x-axis) of the points used for fitting.
5

---

Published as a conference paper at ICLR 2023
Task
Model
MI
M2 4
M3
M4 4
BNSL
Birds 200
1h-
BIT/101/3
9.13e.2 3
9 2.8e
9.13e 2.8c
2,95e:2 1.3e-3
1.76e-2 = 1.1e-
Birds 200 10-shor
B/T/50/1
6.886-2 7.56-4
6.886-2 7.50-4
5,240-2 = 6.20-4
2.66e-2 5.30-4
1.196-2 3.5e-4
Birds 200 10-shor
MiX/B/16
9.15c-2 1.16-3
9.15e-2 1.16.3
3.95e-2 7.0c4
4.62e-2 8.20-4
3.04c-2 6,9e4
Birds 200
10-shor
MiX/L/I6
5,510-2 1
5.51e-2 1,4033
5,51e-2 + 1 1
5,15c-2 = 1.7c-3
Birds 200 10-shot
VIT/B/16
6.776-2 1.1e33
6.77e-2 1.le3
3.52e-2 8.1e.4
1.51e-2 6.2e-4
1.69c-2 7.0e-4
Birds 10-shol
VIT/S/16
3.95 1.2e-3
3.95e-2
3.74c:2 1.1e:3
1.8ec2 7.9e-4
1.09c2 6.1e-
Birds 200 25-shoc
BIT/101/3
9,41e-2 3,20-3
9,21e-2=3.26-3
9,41e-2 : 3.20-3
6,38c-2 : 2,0c-3
1.55e-2 1,3e3
Birds 200 25-shot
B1T/50/1
1.10e-1 1.0e33
8.209
1,97e=2 5.6e-4
1.33e-2 4.4e44
Birds 200 25-shor
MiX/B/16
1.40--1 1.96-
1.00-11
6,93e-2 1.2c-3
2.11c=2 6.9e-4
1.64c-2 6.6e-
Birds 200 shot
MiX/L/16
1.12e-1 3.0e.
1.10
1.12e. + 2.0e-3
5,44e-2 3.8e-3
2.08c=2 1.1e33
Birds 200 25-shol
VIT/B/16
9.02e-2 1.6e-3
9.02e-22
3,75e 1.0c-3
1,51e-2 5.7e-4
1.62c-2 6.le
Birds 200 25 shot
VIT/S/16
5.06e 3
5.06e.2
4.96e 1.4e-3
4.02e 1.2e
1.03e-2 6.6e-4
Birds 5-shot
B/T/101/3
8.176-2
8.176:222.06-3
8.17e-2 : 2.0o-3
3,38c-2 +: 1.3e-3
8.2e-4
Birds 200 5-shot
BIT/50/1
5,440-2 5.66-4
5.46--2
5,440-2 +
2.59c-2 5,4o44
1.34c-2 3.76--
Birds 200 5 shot
MiX/B/16
8.27e-2 3.0e-3
8.27e-2 1.0e
5.49e 7.8c
2
1.39g-2 4.1e-
Rirds 5-shot
MIX/L/16
5.68e-2 1.6-3
5.68e-2 1.4e-3
5.68e-2 1.4e-3
3.20e-2+9.7e-4
1.85c-2 6.4c-4
Birds 200 S-shol
VIT/B/16
3.40e-228.9e
2.
3.40ee 8.9c
6.77
1
Birds 200 5-shot
VIT/S/16
2.990-27..66-4
2.75e-2
2.75c-2 7.99c-
1,20e-2
7,39e-3 4.55-4
CIFAR-100 10-shor
BIT/101/3
8.57c-2 3.8c-3
8.576-2 3.80-3
8.25e-2 3.70-3
4.77c-2 3.0e-3
2.58e-2 2.3e-3
CIFAR-100 10-shor
BIT/S00
7.44c-2 1.5c-3
5.24
2.08c-2 = 7.20-4
1.24c-2 5.8e-4
1.83c-2 8.30-4
CIFAR-100 10.shox
MiX/R/16
8.77c-2 1.9e:3
1.9e33 8.77e-2
2.71e-2 1.2e-3
2.37e-2 9.9e-4
2.44c-2 = 9.5e-4
CIFAR-100 100 10-shoo
MiX/L/16
1.05e-1 3.1e-3
1.05e- 1 3.1e
4.85e 2.6e-
4,97e 1.6e-3
4.75e-2 2.6e-3
CIFAR-100 10-shor
VIT/B/16
8.98o-2
8.98:-2
8,98c-2 2.00-3
4,98c-2 1.70-3
3.71e-2 1.4e-3
CIFAR-100 10-shox
VIT/S/16
6.84e-2 1.1e:3
6.
3.35e-2 2 8.6c-4
2.54e-2 7.5eo4
2.57e=2 7.5e-4
CIFAR-100 25-shor
BIT/101/3
8.776-2
8.776-2
4.440-2 2 3.5o-3
3.40c-2 : 2.7c-3
2.88e-2 = 3.0e-3
CIFAR-100 25-shot
BIT/500
7.31e-2+
2.35e-2 1,5ee3
3.65e 1.8c-3
2.35e:2 1.5e-3
1.89 1.1e33
CIFAR- 100 25-shol
MiX/B/16
1.0%:-1
1.75e:221..66:
2.10e-2 :: 9.4e-4
2.24c:2 9.9e-4
2.67e-2 1.1e:3
CIFAR-100 25-shor
MIX/L/16
9.79e-2
2.22-
3.670-2 1.7c-3
2.98e-2 1.4e-3
3.45o-2 1.60-3
CIFAR-100 25-shot
VIT/B/16
1.07c-1 1.963
1.07e-1
6,54e-2 : 1.6 1.6e-3
4.80e-2 1.4e-3
3.02e-2 4.5e-3
CIFAR-100 25-shor
VIT/S/IG
8.03c-2 1,20-3
7.4
3.13e-2 8:40-4
2.27c-2 7.1-4
2.14c-2 6.9c
CIFAR-100 S-shha 5-shot
BIT/101/3
5.94e-2 3,2e:3
5.94e-2+3.22 3.92e-2+3.22..
3.M0e-2
3.78
CIFAR-100 S-shol
BTT/5I
4.87e-2 1.3e-3
4.87e-2
1.69e 1 8.8e-4
1.87e:22
1.77
CIFAR-100 5-shot
MiX/B/16
7.07e 1.2e-3
7.07e:2
2.78c 8.4e
1.76e
1.70k=2 6.3e-
CIFAR-100 5-shot
MiX/L/16
7.06e-2 1.6e-3
7.06e-2
4.17e-2 1.4e-3
1.32e=2=1..20-
1.00-
CIFAR-100 5-shot
VIT/B/16
6.276-2 1.6e-3
6.276-2=1.6665
6,27c-2 # 1.6o-3
4.30c-2 1.3e-3
2.82e-2 1.0e-3
CIFAR-100 5-shot
VIT/S/16
6.93e-2 1.2e:3
2.84e=2
3.88e 8.0e
3.16e:2 7.5e
3.50e-2
Caltech101 10-shot
BIT/101/3
3.07c-1
3.07
1.51e-1 + 1.3e-2
1,00e-1 1.1c-2
4.75e-2
Caltech101 sboshot
BIT/50/1
3.29e 1.6e
2
1.13e 6.0e
2
1.77e-2
Cahtech01 10-sbot
MiX/B/16
1.35e-1 1.4e22
1.35e-1
1.35e- = 1.4e-2
1.92e-1 1.6e-2
2,04e-1 9.770-
Caltech101 10-shot
MiX/L/16
1,25c-1 1.36-2
1.256-1
1.25e-1 1.3e-2
1.30e-1 1.2o-2
2.13c-1 1.5002
Caltech101 10-sbot
VIT/B/16
7.760-2 4.36-3
4.330-
3.11e-2=3.0k 3.0e-
3.75c-2
=.020-2
Cahtech101 10-shot
VIT/S/16
1.95e-1 6.0e-3
3.99e-2
2.40e-2 2.0ee3
3.41e-2+
2.40=-2
Caltech101 sbot
B/T/101/3
1.15e-1
3.
1.15e:1 6.5e
8.06- 6..5:
Caheeh101 25-abot
B/T/5001
3.60e-1 1.9c-2
5.550-
1,43e-1 7.66-3
3.76--233.6-3
1.66-
Caltech111 25-sbot
MiX/B/16
8.28e-2 1.2e-2
1.22-
8.28e-2 1.2e-2
1.65e-1
1.93e-1
Cahech101 25-abot
MIX/L/16
9.66c-2 1.06-2
3.06e-221.00-2
9,660-2 1.0c-2
9.66e-2 1.0e-2
1
Caltech101 25-shot
VIT/B/16
1.03e-1 = 3.6e-3
2,55e
4.46e-2 3.6e-3
3.33e-2 2.5e-3
3.95e-2=5.4e-3
Caltech101 sbot
VIT/S/16
1.77e. = 3.
3.11e-2
2.80ee2 1.8e-3
3.79e-2
2.
Caltech101 S.shot
BIT/101/3
2.12e 1.2e
2.12e-1 1.2e
2.12e 1.2e-2
9
1.87e-2 = 4.3e-3
Caltech101 5-shot
2.34c-1
4.13e:222.1e-3
1.6le2 1.3e-3
4.69e-2
4.10c-2 =
Caltech101 5-shot
MiX/B/16
2.43c-1 1.20-2
2.430-1 1.77e-
2.35c-1 = 1.10-2
7,28c-2
1.92e-2 1.9c-3
Caltech101 5-shot
MIX/L/16
1.38c-1 = 3.76-3
1.38e- 9.7e-3
1.97e-1
1.63e-1 1.1e-2
Caltech101 5-shot
VIT/B/16
1.10c-1 = 6.36-3
6.02e-2 4.76-3 6.36--16636-3
6.81c-22 4.8c-3
3.87c-2 3.4e-3
Caltech101 shol
VIT/S/16
1.90e 1 3.7e.3
2.
5.04e 2.9e-3
3.82e 2.6e
2.78c-2 1.8e33
ImageNet 10-shot
B/1/101/3
1.27e-1 2.0e33
1.27e-1122.0e-
7.36c-2 1.1e-3
3.06e-2
6.65e-3 3.8e-4
ImageNet 10-shot
BIT/50/1
9,54p-2 7.20-4
9,54c-2
5.75c-3 2.0c-4
1.86e-2 2.8c4
3.84c-3 1.5c-4
ImageNet 10-shot
MIX/B/16
9.34c-2 7.9c44
9.34e-2
3,37c-2 2.90-
3.00-
4.22e-3 1.5e-4
ImageNet 10-shot
MiX/L/16
9.83c-2 1.36-3
9.83e-2 1.3e-3
9.83e-2 1.3e-3
4.01e-3 1.9c-4
4.33e-3 = 1.8c-4
ImageNett 10 shot
VIT/B/16
4.62e.2
2.
4.62e 7.1ee
1.44e 3.0e
5.70e-3 = 2.0e4
ImageNet 10-shot
VIT/S/16
5.76
5.746-225.66-4
1.66e-2 2.5e-4
7.18o:3
3.71e-3 1.4c-4
ImageNet shool
B:T/101/3
1.42e+1
1.42e-1
6.67e-2 9.1e-4
8.3
4.76e.3 4
ImageNet 25-shot
BIT/50/1
1.176-1
4.06e-3
1.84c-2
4.670-3 = 1.6e-4
ImageNet 25-shot
MIX/R/16
9.59e-2+9.3.4
9.59e-2#9.364
5.39c-2 4.9e.4
2.04c-2 3.1e-4
4.17c-3 1.7c-4
ImageNet 25 shot
MiX/L/16
1.03e-1 3
1.03e:1
1.03e 1.3e
6.33e:3 2.2e-4
7.60e 2.6e-4
ImageNet shothot
VIT/B/16
8.
5.17e:2+8.8e.
5.17e:2
1,52e-2 3.8e
4.96c-3 + 2.0c-4
ImageNet 25-shol
VIT/S/16
5.52e-2
3.1ee-2
9.65e a 2.3e-4
6.11e-3 2.4e-4
ImageNet 5-shot
B/T/101/3
9.24c-2 1.4-3
9,24c-2
9.24c-2
1.40-3
2,09c-2 7.9c-
8.05c-3 5.0c-4
ImageNet 5-shot
BIT/50/1
8.95e-2 6,7e-4
6.97e-2+6.7c.4
1.53e-2 2.2e-4
1.Ile-2
7.94e-3 2,1o44
ImagoNet 5-shot
MIX/B/16
9.09c-2 = 7.20-4
9.09e-2 7.2004
3.01e-2 + 2.8c4
1.95e-2 + 2.7c-4
6.49e-3 = 2.20-4
ImageNet s-shat
MiX/L/16
7.99e-2 9.7e
7.99e-2
7.99c 9.7ee
9,92e.3 4.5e
5.68c-3 2.4e-4
ImageNet S-shot
VIT/B/16
4.11e-2
4.1Ie-2 6.3e-4
4.11e-2 a 6.3e-4
1.55e-2 a 2.8e-4
1.29e-2 2.7e-4
ImagoNet 5-abot
VIT/S/16
4.200-2
4.20c-2 2 4.1o-4
2.400-2 2.6c-4
8.02c-3 + 1.9044
4.720-3 1.6e-4
Table 3: Extrapolation Results on scaling behavior of Downstream Vision Tasks. See Section 5.1
for more details. Numbers for M1, M2, M3, and M4 obtained via correspondence with authors of Alabdulmohsin et al. (2022).
6

---

Published as a conference paper at ICLR 2023
5.2 LANGUAGE
Using the scaling laws benchmark of Alabdulmohsii et al. (2022), we evaluate how well various functional forms extrapolate performance on language tasks as the (pre-)training dataset size in- creases. In this large-scale language subset the benchmark, the tasks that are evaluated are error rates on each of the various downstream tasks from the BIG- EIG-Bench (BB) (Srivastava et al., 2022) benchmark and upstream test cross-entropy of various models trained to do language modeling (LM) and neural machine translation (NMT). All LM and BB tasks use decoder-only language model. As can be seen in Tables 2 Squared Logarithmic Error) for 75% of tasks of any of the functional forms, while the next best functional form performs the best on only 10% of the tasks.
and 4 BNSL yields extrapolations with the lowest RMSLE (Root Mean
To view all plots of the BNSL on each of these tasks, see Figures 27 28 29 in Appendix A.26 To
view plots of M1, M2, M3, and M4 these tasks, Figure of Alabdulmohsin et al. (2022)
In Section A.13 additionally show that BNSL yields accurate extrapolations of performance on large-scale downstream language tasks when number of model parameters is on the x-axis. In Section A.14 we show that BNSL accurately models and extrapolates the scaling behavior of sparse models (i.e. sparse, pruned models and sparsely gated mixture-of-exper models). In Section A.25 we additionally show that BNSL yields accurate extrapolations of performance on large-scale downstream audio (speech recognition) tasks.
In Section [A.6]] we show BNSL accurately models and extrapolates the scaling behavior with fine-
tuning dataset size on the x-axis and the scaling behavior of computer programming / coding.
In Section A.23
BNSL. accurately extrapolates the scaling behavior of math word problems.
In Section A.22
BNSL accurately extrapolates the scaling behavior of tasks involving molecules.
In Section A.2I
BNSL accurately extrapolates the scaling behavior of OOD detection.
In Section A.17
we additionally show BNSL accurately models and extrapolates the scaling behav-
ior of uncertainty estimation/ / calibration.
Domain
Tusk
Model
M1
M2
M3
M4 2
BNSL 1
BB
date understanding, I-shot
2.620+8 Param
9.
9.60- 3.19
10-4 4.676-3
3.19-22
3.40c-3 # 7.9c-5
BB
date understanding. 2-shot
2.62e+88 Param
2.86e 6.2e 5..5-2
6.22e.
4.83e-34 le
2
4.38e-3 4.0e-4
linguistic mappings. 1-shol
2.62e+8 Param
5.4e-2
1.66e-225.5e-4
1.33e-2 6.50
2.22-4
BB
linguistic mappings. 2-shot
2.62c+8 Param
1.700-26.5cc-4
6.500-2
5.06
BB
mult data wrangling. I-shot
2.62++8 Param
1.07-22
1.07c-22
1.07e-2 1.0c-3
3.666-317.3e--
6.39e-322446er
molt data wrangling, 2-shot
2.620+8 Param
1.57e-2 1.5c-3
1.576-2+ 1.5e-3
1.57e-2
1.5e-3
5.79t-3+7.0e-
2.67-
BB
wikidatn, I-shot
2.62++8 Param
4.27e.3 8.9e-4
4.32e.3+8.2e-4
8.27e.34
4.32e.348.2e
7.33e:
BB
wikidata, 2-shot
2.62++8 Param
7.00--
4.66e-3
4.39e:32
6.02e-3
8.05e-3
BB
unit conversion, 1-shot
2.62++8 Param
8.30e
$.30e-344.4e-4
2.77e
4.79e.343.4e
2.55e:2
conversion, 2-sbot
2.62++8 Param
1.77--224.4c-4 6.70 1.66e:2
1.77--224.e-44
5.500=335.55-44
1.55e=315.10-4
7.02e-3
LM
upstream test cross-entropy
1.07c+9 Param
1.6
5.500-3+5.9e-5
3.292-3
9.71c-4
3.2e-5
LM
upsiream test cross entropy
4.53et8 Param
7.4.e.449.8e.5
6.58e
7.
5.86e-4
7.7e-5
IM
upstream test cross-entropy 2622+88 Param
1.55e-2 7.2e-4
9.20e-4+9.7e.5
3.97e-3+
1.3e44
9.70--44
5.
LM upsiream lest entropy 1.34+88 Param
4.88e
1.46e
6.66e.4
66.8--3
5.55e
LM upstream test cotrs-c
168c+7 Param
6.376-3
3.03e-4
1.56--3
3.03c-4
1.34e-4=1.8o-5
AMTupstream test cross-cntropy
Enc, Dec
1.71e-1
5.64e-2=
3.376-2#
1.81c-2+
L669-2
NMT upstrcaam test cross-cntropy 6 Enc,
2:
5
1.65c-2
4.44o-2
1.56c2
upsuteeeam test cross-
6 Fnc. 6 Dec
2.62e +0
3.84e-2=
8.926-2+
2.05e-2+
1.37e33 +
NMT upsireemm lest cross entropy Decy LM
2.52e 10
1.03e-220
3.28e-2+0
8.43e-3
7.33e-3 a
NMT upstrean test cross-cutropy
Transformer-
1.900-110
1.260-220
6.320-2
1.26--210
8.30c-3 1
Ence LSTM-
Doc
Table 4: Extrapolation Results on scaling behavior of Language Tasks. See Section 5.2 for more
details. Numbers for MI, M2, M3, and M4 were obtained via correspondence with authors of Alabdulmohsin et al. (2022). BB stands for BIG-Bench (Srivastava et al. 2022). NMT stands for Neural Machine Translation. LM stands for Language Modeling.
5.3 REINFORCEMENT LEARNING
We show that BNSL accurately models and extrapolates the scaling behaviors of various multi-agent and single-agent reinforcement learning algorithms trained in various environments. In the top left plot and top right plot and bottom left plot of Figure 2 BNSL accurately models and extrapolates the scaling behavior of the AlphaZero algorithm trained to play the game Connect Four from Figure 4 and Figure 5 and Figure 3 respectively of Neumann & Gros (2022); the x-axes respectively are compute (FLOPs) used for training, training dataset size (states), and number model parameters. In Figure bottom left and bottom right respectively, BNSL accurately models and extrapolates the scaling behavior of the Phasic Policy Gradient (PPG) algorithm (Cobbe et al.. 2021bb trained to play the Procgen (Cobbe et al. 2020) game called StarPilot and the scaling behavior of the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) trained to play the Proegen (Cobbe et al. 2020) game called Heist.
7

---

Published as a conference paper at ICLR 2023
In Section A.18 we find BNSL accurately extrapolates the scaling behavior of pretrained language model finetuned (i.e. aligned) via Reinforcement Learning from Human Feedback (RLHF) to be helpful from Figure of Bai et al. (2022)
Figure 2: Extrapolation of BNSL on Reinforcement Learning Scaling Experimental Data. Experi- mental data of the top left plot and top middle plot and top right plot is from Figure 4 and Figure 5 and Figure 3 respectively of Neumann & Gros (2022). Experimental Data of the bottom left plot from Figure left of Hilton et al.] (2023). Experimental Data of the bottom right plot is from Figure 2 of Cobbe et al. (2020). Top left and bottom left plot is the compute-optimal Pareto frontier. See Section 1.33mm5533r3rm for more details.
5.4 NON-MONOTONIC SCALING
We show that BNSL accurately models and extrapolates non- non-monotonic scaling behaviors that are exhibited by Transformers (Vaswani et al. (017)) in double descent (Nakkiran et al.] 2021) in Fig- ure 3 Various other functional forms are mathematically incapable of expressing non-monotonic behaviors (as shown in Section
Figure 3: Extrapolation of BNSL on Double Descent. Both plots are of transformers trained do neural machine translation via minimizing cross-entropy. Experimental data of left figure is obtained from Figure 8 top of Nakkiran et al. (202T); "Model Width" on the x-axis refers to embedding dimension dmodee of the transformer; note that model width is linearly proportional to number of model parameters, so number of model parameters on the x-axis would yield same results. Experimental data of the right figure is obtained from Figure 1l Nakkiran al. (202T). The plot on the left contains two breaks of a BNSL fit to the black points. See Section 5.4 for more details.
5.5 INFLECTION POINTS
We show that BNSL is capable of modeling and extrapolating the scaling behavior of tasks that have inflection point on linear-linear plot such as the task of arithmetic (4-digit addition). Here we model and extrapolate the scaling behavior of a transformer model (Vaswani et al. (2017) with respect to the training dataset size the 4-digit addition task. Various other functional forms are mathematically incapable of expressing inflection points on linear-linear plot (as shown in Section 4 and as a result, are mathematically incapable expressing and modeling inflection points (on linear-linear plot) that are present in the scaling behavior of 4-digit addition. In Figure 4 left, we show that BNSL expresses and accurately models the inflection point present in the scaling behavior of d-digi addition and result accurately extrapolates the scaling behavior of 4 digit addition. For further details about the hyperparameters please refer to the Appendix Section A.5
8

---

Published as a conference paper at ICLR 2023
of Da 2N05 of Dittdaitto
Figure 4: Extrapolation of BNSL on 4 Digit Addition. Note these plots are linear-linear. Each point in left plot is mean of greater than 1000 seeds at that dataset size. In left plot, each point is gathered from model trained to do task of 4 digit addition. In right plot, each point is gathered from a noiseless simulation of the BNSL of the task of 4 digit addition. See Sections 5.5 A.5 more details.
for
6 THE LIMIT OF THE PREDICTABILITY OF SCALING BEHAVIOR
We use BNSL to glean insights about the limit of the predictability of scaling behavior. Recent pa- pers (Ganguli et al. 2022 Wei et al. 2022a) have advertised many tasks as having "unpredictable" "emergent" "phase transition/change" scaling behavior, the most famous of which is the task of arithmetic. In the previous section and in Figure Ieeft, we successfully predicted (i.e. extrapolated) the scaling behavior of 4-digit addition (arithmetic). However, we are only able to accurately ex- trapolate the scaling behavior if given some points from training runs with training dataset size of at least 720, and the break in which the scaling behavior of 4-digit addition transitions from one power law to another steeper power-law happens at around training dataset size of 415.
Ideally, one would like to be able to extrapolate the entire scaling behavior by fitting only points from before the break. In Figure 4 right, we use a noiseless simulation of the BNSL of 4-digit addition to show what would happen if one had infinitely many training runs / seeds average out all the noisy deviation between runs such that one could recover (i.e. learn via a curve-fitting library such SciPy (Virtanen et al. 2020 the learned constants of the BNSL as well as possible. When using this noiseless simulation. we find that we are only able to accurately extrapolate the scaling behavior given some points from training runs with a training dataset size of at least 415, which is very close to the break.
This has few implications:
1) When the scaling behavior exhibits greater than 0 breaks that are sufficiently sharp, there is a limit as to how small the maximum (along the x-axis) of the points used for fitting can be if one wants to perfectly extrapolate the scaling behavior, even if one has infinitely many seeds / training runs.
2) If an additional break of sufficient sharpness happens at scale that is sufficiently larger than the maximum (along the x-axis) of the points used for fitting, there does not (currently) exist a way to extrapolate the scaling behavior after that additional break.
3) If break of sufficient sharpness happens at scale sufficiently smaller than the maximum (along the x-axis) of the points used for fitting, points smaller (along the x-axis) than that break are often useless for improving extrapolation.
CONCLUSIONS
We have presented a smoothly broken power law functional form that accurately models and ex- trapolates the scaling behaviors of artificial neural networks for various architectures and for each of various tasks from a very large and diverse set of upstream and downstream tasks. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, Al alignment, robotics, out-of-distribution generalization, continual learning, uncertainty estimation calibration, out-of-distribution detection, adversarial robustness, molecules. computer programming/coding, math word problems, arithmetic, unsupervised/self- supervised learning, and reinforcement learning (single agent and multi-agent). When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Additionally, this functional form accurately models and extrapolates scaling behavior that other functional forms are incapable of ex- pressing such as the non-monotonic transitions present in the scaling behavior of phenomena such as double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we used this functional form to glean insights about the limit of the predictability of scaling behavior. See arv for longer version.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
We place relatively high probability on the claim that variants of smoothly broken power laws per- haps are the "true" functional form of the scaling behavior of all(?) things that involve artificial neural networks. Due the fact that BNSL is a variant of smoothly broken power laws, an ethical concern one might have about our work is that revealing BNSL might differentially (Hendrycks & Mazeika 2022) improve I capabilities progress relative to A()) safetylalignment progress. A counter-argument is that BNSI will also allow the A safetylalignment field to extrapolate the scaling behaviors of its methods for aligning A()) systems and as result will also accelerate align- ment/safety progress. Existing scaling laws besides BNSL struggle especially to model downstream performance, e.g. on safeet-relevant evaluations (especially evaluations (such as interpretability and controllability) that might exhibit non-monotonic scaling behavior in the larger scale systems of the future); we believe our work could differentially help in forecasting emergence of novel capabilities (such reasoning (Wei al. (0226)) or behaviors (such deception dishonesty (Evans al. 2021
Lin et al., 2021)), and thus help avoid unpleasant surprises.
A potential limitation of the current approach is the need to collect enough samples of the system's performance (i.e. the (x.y) points required for estimating the scaling laws parameters). A small number of samples sometimes may not be sufficient to accurately fit and extrapolate the BNSL functional form, and obtaining large number of such samples can sometimes be costly. This has the ethical implication that entities with more compute to gather more points maybe will have considerably more accurate extrapolations of scaling behavior than entities with less compute. As result, entities with less compute (e.g. academia) maybe will have less foresight than entities with more compute (e.g. Big Tech), which could maybe exacerbate the gap between entities with more compute (e.g. Big Tech) and entities with less compute (c.g. academia).
ACKNOWLEDGMENTS
We are thankful for useful feedback and assistance from Kartik Ahuja, Ibrahim Alabdulmohsin, Ankesh Anand, Jacob Buckman, Guillaume Dumas, Leo Gao, Andy Jones, Behnam Neyshabur, Gabriel Prato, Stephen Roller, Michael Trazzi, Tony Wu and others.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Direct detection of a break in the teraelectronvolt cosmic-ray spectrum of electrons and positrons. Nature, 652(7683))63666, nov 2017. doi: 10.10388/aturce44455. URL Inttps:// /rg/ 10. 1038 103822Fntture244777
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. 2021.
Ibrahim Mansour Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scal- ing laws in language and vision. In NeurlPS 2022, 2022. URL hoKlhttps:://xxl..oro/////
2209.0680 06640
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint Stdi:2204.05822. 2022.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Ko- rnbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. Nature Communications, 13 (I), may 2022. doi: 101003884144---00222999-9--
URL https://dci.org/10..11
2Fs41467-022-29335-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Pagannii, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference Machine Learning, pp. 4057- 4086. PMLR, 2022.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp. 2048- 2056. PMLR, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021a.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In Interna- tional Conference on Machine Learning, pp. 2020-2027. PMLR, 2021b.
Corinna Cortes, Lawrence Jackel, Sara A Solla, Vladimir Vapnik, and John Denker. Learning curves: Asymptotic values and rate of convergence. In Advances in Neural Information Process- ing Systems, pp. 327-334. 1994.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint vrXi::2010.11929 2020.
Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. Truthful ai: Developing and governing ai that does not lic. arXiv preprint 10.067/ 2021.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEFE, 2004.
11

---

Published as a conference paper at ICLR 2023
Elias Frantar and Dan Alistarh. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 2023.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In 2022 ACM Conference Fairness, Accountability, and Transparency, pp- 1747-1764, 2022.
Dan Hendrycks and Mantas Mazeika. arXiv:2206.05882, 2022.
X-risk analysis for ai research. arXiv preprint
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling. arXiv e-prints, art. 1011:20100.1401,, October 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling Laws for Transfer. arXiv e-prints, art. arXiv:2102.01293, February 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep Learning Scaling is Predictable, Empirically. arXiv e-prints, art. 17X.::111..004000 December 2017.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint 2xxiv22301.13442 2023.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):
90-95, 2007. doi: 10. 109/MCSE.2007.55
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint 2xix:2207.05222 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv e-prints, art. arXiv:2001.0836/ January 2020.
Andrej Karpathy. mingpt. httppp////g.cca...ccc///aaatattp
2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- on computer vision, pp. 491-507, Springer, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07990 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL htthttps::// htt.tpp:://peeeeviieee wet.ee.weeeeeeeeeeeeeeeeee.eeeee wtw.:/orrrmmmddrr..ae..
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, (12) 2021.
Oren Neumann and Claudius Gros. Scaling laws for multi-agent reinforcement learning model. arXiv preprint arXiv:2210000849 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:22
2022.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez. and Ethan Dyer. Effect of scale catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A construuctive of the generalization error across scales. CoRR, abs/ (tss/1909.1ne 12673, 2019. URL http:pt/p///xiii org/abs/ 909. 12673
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707:00044444 2017.
Mohit Shridhar, Lucas Manuelli, and Dicter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Meyood neural scaling laws: beating power law scaling via data pruning, 2022. URL httpss////aa abs/ 2as//2006.14488
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Aonsoo, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 1rXiv:2206.04615. 2022.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:24261- 24272, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau,, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: funda- mental algorithms for scientific computing in python. Nature methods, 17(3):261-272 2020.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint urXiv:2206.07683 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1190. 2022b.
13

---

Published as a conference paper at ICLR 2023
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference Learning Representations, 2020. /RUlhttss:/openrreviee wwwpeeww......eeeeeeeeeeeeeeeeoe forum?
idd-HyJhCEFDS)
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Re- arranging the visual world for robotic manipulation. In Conference Robot Learning, pp. 726- 747. PMLR, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. CoRR, abs/2106.044. 04560, 2021. hRLhttps: / /arxivvvv.v.. org/ abs http:/bb//22..0.. 2206. 04560
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 ANALYSIS AND EXPLANATION OF WHY BNSL IS SMOOTHLY CONNECTED PIECEWISE (APPROXIMATELY) LINEAR FUNCTION ON LOG-LOG PLOT
Analysing Equation reveals why BNSL is smoothly connected piecewise (approximately) linear function log-log plot. Considering y as function of = log(æ), applying logarithms both sides and setting = 0 yields:
n
[epp(=) 1
1/5
log(y) log(b) co= Σ
cafi log
de
We can now see the terms in the sum resemble the well-known softplus function: softplus(z) log(1 + exp(z)). which smoothly interpolates between the constant 0 function and the identity. By plotting one such term for different values of Ci d,, St. easy to confirm that they influence the shape of the curve as described in Section 2
A.2 DECOMPOSITION OF BROKEN NEURAL SCALING LAW INTO POWER LAW SEGMENTS THAT COMPOSED OF
We now show way to decompose BNSL (Equation 1 with 3 breaks into the power law segments that it is composed of. This decomposition is what we used to produce segments 1-4 overlaid in Figure [ and is usable when values of in Equation T are not too large. This decomposition pattern is straight-forward to extend to n breaks.
segment, b (x) (ro)
segment2 b (d1) (co) (r /d1) (c)+to)))))))) (cs)
segmenls b (di) -(co) (co)
(d2 d1 (crte) (xr /d2) - (cxtor
segment4 b (di)~ (00) (d2 /2/ (os+o)) (d)/d2) (02+or+0)) ( +00t
A.3 DEFINITION OF ROOT MEAN SQUARED LOG ERROR
Root_Menn_Sppe SirioonSqureddaaa Log_Errre RMSLE
(log(u)) - 1o)())))))/n
A.4 DEFNITION OF ROOT STANDARD LOG ERROR
eтror
(log(vi) - log(ū:)))
N
Herror
error
N
1=1
N
Ceroor
(error; - Merror)
Terror
Root Standard Error _
Perror
Verror
Vlen(p)
A.5 EXPERIMENTAL DETAILS OF SECTION
5.5
We perform an extensive set of experiments to model and extrapolate the scaling behavior for the 4-digit arithmetic addition task with respect to the training dataset size. Our code is based on the minGPT implementation (Krrpthyy] 2020). We set the batch size equal to the training dataset size. We do not dropout GPU and each run took less than 2 hours. For our experiments we train the transformer model using the following set of hyperparameters:
learning rate decay here. Each experiment was
single V100
15

---

Published as a conference paper at ICLR 2023
Dmodel
128
DMLP
512
Number of heads
Number of transformer blocks (i.e. layers)
Learning rate
0.0001
Weight Decay
0.1
Dropout Probability
0.0
Dataset sizes
144-1008
Vocab Size
10
Table 5: Hyperparameters for 4-digit addition task
A.6 EXPERIMENTAL DETAILS OF FITTING BNSL
We fit BNSL as follows: We first use seipy.optimize. brute to do a grid search the values of the constants (a, b, Co, Cl...Cn, di...n. fI...nn) of BNSL that best minimize the mean squared log error (MSLE) between the real data and the output of BNSL. We then use the values obtained from grid search as the initialization of the non- non-linear least squares algorithm of scipy.optimize.cum www.i/eeeccreefii.tee We then use the non-lineat least squares algorithm of scipy.opting cotpyyoptiiizzcuu curve to minimize the mean squared log error (MSLE) between the real data and the output of BNSL.
The version of MSLE we use for such optimization is the following numerically stable variant:
Numerically Stable MSLE Σ (log(yy + +1 Log(ū: + 1))2)/u
16

---

Published as a conference paper at ICLR 2023
A.7 EXTRAPOLATION TO SCALES THAT ARE AN ORDER OF MAGNITUDE LARGER THAN THE MAXIMUM (ALONG THE X-AXIS) OF THE POINTS USED FOR FITTING
ImageNet 5-Shot
100
Test Error Rate
6 60-10-1
40-1
3 10
107
10
109
Training Dataset Size
ImageNet 25-Shot
10
Test Error Rate
6x 10-1
10-1
3x-1-1
107
10a
10a
Training Dataset Size
Figure 5: Extrapolation Results of BNSL to Scales that are Order of Magnitude larger than the maximum (along the x-axis) of the points used for fitting. Experimental data of scaling behavior obtained from scaling laws benchmark of Alabdulmohssi al. (2022). The upstream task is super- vised pretraining MLP mixers (MiX) (Tolstikhin et al. (021) on subsets (i.c. the x-axis of plot) of JFT-300M (Sun et al. 2017h. The Downstream Task is n-shot ImageNet classification (i.c. the
y-axis plot). See Section A.7 for more details.
In Figure we show that BNSL accurately extrapolates to scales that are order of magnitude larger than the maximum (along the x-axis) of the points used for fitting. The upstream task is supervised pretraining of MI.P mixers (MiX) (Tolstikhin et al. 202T) on subsets (i.e. the x-axis of plot) of JFT-300M (Sun et al. 2017). The downstream task is n-ohot ImageNet classification (i.c. the y-axis of plot). The experimental data of this scaling behavior obtained from Alabdulmohsin et al.. 2022).
17

---

Published as a conference paper at ICLR 2023
A.8
EXTRAPOLATION RESULTS FOR DOWNSTREAM VISION TASKS WHEN TRAINING RUNS ARE SCALED TO BE COMPUTE-OPTIMAL.
Task
Model
M3 L
BNSL 1 9.79e-3 = 4.70e-3 9.37e-3 2.60c-3
ImageNet I0-Shot
VIT
1.91e-2
6,48e-3
ImageNet Finetune
VIT
1.14e-2 2.42e-3
Table 6: Extrapolation Results for Downstream Vision Tasks when training runs are scaled using the compute-optimal scaling (i.e. Pareto frontier) with respect to downstream performance. Experi- mental data obtained from Figure 2 of et (02TT. See Section A.8 for more details.
Tosk: imagefiet Fiieetue Model: VTT
Tasc: ImageNet 10-Shot; Model: VIT
00
6x
4x
Tost Enorr Enorrate
Teat Enoor Eate
3x.10
2x
2
10"
10+P
2014
1021
2224
105
1070
10/3
FLOPs
FLOPS
Figure 6: Extrapolation Results of BNSL for Downstream Vision Tasks when training runs are scaled to be compute-opting Experimental data obtained from Figure 2 of Zhai et al. (202T). See Section for more details.
In Figure 6 via fitting BNSL, we additionally obtain accurate extrapolations of scaling behavior of large-scale downstream vision tasks when compute (FLOPs) used for (pre-)training on the x- axis and compute is scaled in the manner that is Pareto optimal with respect to the performance evaluation metric (downstream accuracy in this case). The experimental scaling data was obtained from Figure 2 of Zhai et al. (202T), and as result in Table 6 we compare extrapolation of BNSL to the extrapolation M3 (which was proposed in Zhai et al. (20II)); we find that BNSL that yields extrapolations of scaling behavior that more accurate these tasks.
18

---

Published as a conference paper at ICLR 2023
A.9
EXTRAPOLATION RESULTS FOR DIFFUSION GENERATIVE MODELS OF IMAGES
Mathod: Diffusion Model | Evellation Dataset: Imagellet 64x64
Mathod: Model;
VLL (LL:pitaamm
3,00 107
2
2.xx 10
28101
3.55
Comouto FLLOPs Tooiing
Compate (HLOrss)
training
Extrapolation Results of BNSL for scaling behavior of Diffusion Generative Models of Images. Frechet Inception Distance (FID) score is on the y-axis in the left plot. Negative log- likelihood (NLL) is the y-axis in the right plot. For both plots, compute used for training is on the x-axis and Imagenet 64x64 is the evaluation dataset. Experimental data of scaling behavior obtained from Figure 10 of Nichol & Dhariwal (202T). See Section A.9 for more details.
Figure 7:
In Figure 7 we show that BNSL accurately extrapolates the scaling behavior of Diffusion Generative Models of Images from Figure 10 of Nichol & Dhariwal (2021) when Negative Log-likelihood (NLL) or Frechet Inception Distance (FID) score is on the y-axis and compute used for training is on the x-axis; compute is scaled in the manner that is Pareto optimal with respect to the performance evaluation metric on the y-axis.
10 EXTRAPOLATION RESULTS FOR GENERATIVE MODELS OF VIDEO
Video Scoling
16x 16
15x 300
14X10 10
13x
10
101
1914
10.5
20
Compute FLLP uned for Training
Figure 8: Extrapolation Results of BNSL for scaling behavior of Generative Models of Video. Upstream Test Cross-Entroppy is on the y-axis. Videos scraped from the web are the evaluation dataset. During training, compute (used for training autoregressive transformer) on the x-axis is scaled in the manner that is Pareto optimal with respect to the performance evaluation metric on the y-axis. Experimental data of scaling behavior obtained from top right plot of Figure Heeighga et al. (2020). See Section A.10 for more details.
In Figure 8 we show that BNSL accurately extrapolates the scaling behavior of generative models of video. Each frame is downsampled to a pixel resolution of 64x64; each frame then tokenized via a pretrained 16x16 VQVAE (Van Den Oord et al.) (017) to obtain 256 tokens per frame. 16 consecutive frames are then input to an autoregressive transformer as length 4096 (16x16x16) sequence. The dataset is 100 hours of videos scraped from the web. See section 2 of Henighan et al. (2000) for more details.
19

---

Published as a conference paper at ICLR 2023
A.11
EXTRAPOLATION RESULTS WHEN DATA IS PRUNED PARETO OPTIMALLY
Model: Resnet50; ; Dataset: ImoqaNet
6X10
The Sate
1
1
2x10
10
101
107
Training Dahhaen Sine
Figure 9: Extrapolation Results of BNSL for scaling behavior when data is pruned Pareto optimally (such that each point along the x-axis uses the subset of the dataset that yields the best performance (y-axis value) for that dataset size (x-axis value)). Experimental data of scaling behavior obtained from Figure 3D of Sorscher et al. (2022). See Section A.1T for more details.
In Figure 9 we show that BNSL accurately extrapolates the scaling behavior when data is pruned Pareto optimally (such that each point along the x-axis uses the subset of the dataset that yields the best performance (y-axis value) for that dataset size (x-axis value)) from Figure 3D of Sorrcher al. (0022).
A. EXTRAPOLATION RESULTS WHEN UPSTREAM PERFORMANCE IS ON THE X-AXIS
Tosk ImageNett 20-Shot
5x10
Doeratmam Frer Sate
4x100%
40-0 Upstream Accuracy Rote
Figure 10: Extrapolation Results of BNSL for scaling behavior when Upstream Performance is on the x-axis and Downstream Performance is on the y-axis. Experimental data of scaling behavior obtained from Figure 5 of Abnar al. (2021) The upstream task is supervised pretraining of ViT (Dosovitskiy et al.) 2020) on subsets of JFT-300M (Sun et al. (017). The Downstream Task is 20- shot ImageNet classification. See Section A.12 for more details.
In Figure 10
we show that BNSL accurately extrapolates the scaling behavior when upstream per- formance is on the x-axis and downstream performance is on the y-axis. The upstream task is super- vised pretraining of VIT (Dosovitskiy et al. (020)) on subsets of JFT-300M (Sun et al., 2017). The downstream task is 20-shot ImageNet classification. The experimental data of this scaling behavior obtained from Figure 5 Abnar et al. (202))
20

---

Published as a conference paper at ICLR 2023
A.13 EXTRAPOLATION RESULTS FOR DOWNSTREAM LANGUAGE TASKS WHEN NUMBER OF MODEL PARAMETERS IS ON THE X-IISS.
Taicks En (Multi BLEJ) Shots: Rex
Tesk: 4 Digit Addition; Shots. Zera Shot
189
x10
Text Eieer Eate hate
150 BLE me Scor
2.8 10
8k10
1
96% 10
102
101t
101
10
1011
Mudkl
Task: En-eRo 16 [Multi- BLEU): Shots: Zero Shot
Jasc: Symbol Insertion; ; Shots: Zero-Shot
100 mins SLEU Scoe harcottoge
99x10-1
Text Emor
86x
92x10-1
8K100
103
20
10
101E
1001
Number of Model Parameters
Number of Modzl Forameters
Tosk: TriviOAA Shots: See--Sht
Task. LAMBACA (Errrr) Shots. Zero Shot
63 10
Ton Enorr Rato
40
That
30
2x30-1
122
12"
10
101
Numter of Model Paremeters
Number of Model Paremeters
Extrapolation Results of BNSL for Downstream Language Tasks when Number of Model Parameters is on the x-axis. Experimental data obtained from Table H.1 of the GPT-3 arXiv paper (Brown et 2020). See Section A.13 for more details.
Figure 11:
We find in general for each of every modality that the variance between seeds is higher when number of model parameters is on x-axis (as opposed to e.g. training dataset size on the x-axis). Table H.1 of the GPT-3 arXiv paper (Brown et al. (020) release includes results for numbers of model param- eters. In Figure we include examples of when 8 numbers of model parameters (7 for fitting, and largest held-out to evaluate extrapolation) are sufficient for obtaining accurate downstream extrap- olation from BNSL due variance between seeds being low enough. For many other downstream tasks with number of model parameters on the x-axis, the variance between seeds is much higher such that number considerably larger than 7 points along the curve is needed to obtain an accurate extrapolation.
21

---

Published as a conference paper at ICLR 2023
A.14 EXTRAPOLATION RESULTS FOR SPARSE MODELS
Tink- Unstmom Mudell: SeSf Mixhure of Experts
Unnimom: ; Medill: SirkhemmAASF Mixtur Experts
222x 19
2.
Tet Chstrriee
Tet Crose-
2.12x10
21X 10"
21x 10
2
2.00 10
2.0x 10
10.4
10t
Numbbr of Nodel Parameters
Number of Experts
Task: WidTor22 (owrstream)) Model OPT Pruned have Sparsity
2x
Figure 12: Extrapolation Results of BNSL for Sparse Models. Experimental data of top 2 figures are obtained from Figure 22 of Clark et al. (2022). Experimental data of bottom figure obtained from Figure I right of |Frnntar & Alistarh (2023). The y-axis is Test Cross-Entropy. The x-axis the number of model parameters that the model contains. See Section A.14 for more details.
In Figure 12 find BNSL accurately extrapolates the scaling behavior of various sparse models (i.e. sparse, pruned models and sparsely gated mixture-of-cxperr models).
A.15 EXTRAPOLATION RESULTS FOR ADVERSARIAL ROBUSTNESS
Adverserial Robustness
51
4.9
4.
47x
4.
1C300
3X10 4 10/4
Extrapolation Results of BNSL for Adversarial Robustness. Test Error Rate is on the y-axis. FLOPs of the forward pass of a model of that size is on the x-axis. Experimental data of y-axis is obtained from Table 7 of Xie & Yuille (2020); experimental data of x-axis is obtained from Figure Xie & Yuille 2020p. See Section A.15 for more details.
Figure 13:
In Figure 13 we find BNSL accurately extrapolates the scaling behavior of adversarial robustness. The adversarial test set is constructed via projected gradient descent (PGD) attacker (Madry et al.) 2018) of 20 iterations. During training, adversarial examples for training are constructed by PGD attacker of 30 iterations.
22

---

Published as a conference paper at ICLR 2023
A.16 EXTRAPOLATION RESULTS WITH FINETUNING DATASET SIZE ON THE X-AXIS (AND ALSO FOR COMPUTER PROGRAMMING CODING)
Tesk: linetuue on Code after pretraining on Text
The Cross Entrony
10
9x10
10a
10'
301
10'
1020
Finetniig Datscet Size
Figure 14: Extrapolation Results of BNSL with Finetuming Dataset Size on the X-axis. Experimen- tal data is obtained from Figure 1 of Hernandez et al.] (202T) The figure is of transformer a model that is pretrained on large amount of mostly English text from the internet and then finetuned on a large amount of python code. The y-axis is Test Cross Cross-Entropy on the distribution of python code. The x-axis is the size (measured in number of characters) of the Finetuning (not pretraining) Dataset. See Section A. for more details.
In Figure [4] we find BNSL accurately models and extrapolates the scaling behavior with finetuning dataset size on the x-axis (i.e. model that pretrained on a large amount of mostly english text from the internet and then finetuned on a large amount of python code).
A.17 EXTRAPOLATION RESULTS FOR UNCERTAINTY ESTIMATION/ / CALIBRATION
Uncertainty Estimation on BIG Bench
0x10
3x10
2X10"
112
100
Number Matel Parometers
Extrapolation Results of BNSL for Uncertainty Estimation / Calibration. Expected Calibration Error is on the y-axis. Number of Model Parameters is on the x-axis. Experimental data obtained from "Lettered Choices (5-shot)" evaluation protocol plot from Figure 4 right of Kadavath ct al. 2022). See Section A. 17 for more details.
Figure 15:
In Figure [5] we find BNSL accurately extrapolates the scaling behavior of downstream uncertainty estimation calibration on BIG-Bench (Srivastava et al. (022).
23

---

Published as a conference paper at ICLR 2023
A.18 EXTRAPOLATION RESULTS FOR AI ALIGNMENT VIA RLHF
Method: RLHF [Reinforcement Leaming from Human Feadback)
Task: Heistuiness (Nl Alignmenti
So
10t
6x
107
10th
Number Modzl rorameoors
Figure 16: Extrapolation Results of BNSL for Downstream AI Alignment when Number of Model Parameters is on the x-axis. Experimental data obtained from the Static HH RLHF results from Figure of Bai et al. (2022). See Section A. for more details.
In Figure 16
find BNSL accurately extrapolates the scaling behavior of a pretrained language model finetuned (i.e. aligned) via Reinforcement Learning from Human Feedback (RLHF) to be helpful from Figure 1 Ba al. 2022). The y-axis is Elo score based on crowdworker preferences. The x-axis is the number of model parameters that the language model contains.
19 EXTRAPOLATION RESULTS FOR CONTINUAL LEARNING (L.E. CATASTROPHIC FORGETTING)
Leaning Sraling
26x
Pelrrrrhats hate
2.4
2x
18
Number Porometers
Figure 17: Extrapolation Results of BNSL for Continual Learning (i.e. Catastrophic Forgetting). Experimental data obtained from the Domainnet/Clipart section of the bottom right Figure 2 of (Ramasesh et al. 2022). X-axis is number of model parameters in the ResNet model. In this setup, model is trained (in sequence, not simultancously) on task and then task B. Y-axis is mean of the test error rate on task A and task B. See Section A9 for more details.
In Figure 17 we find that BNSL accurately extrapolates the scaling behavior of continual learning (i.e. catastrophic forgetting).
24

---

Published as a conference paper at ICLR 2023
A.20 EXTRAPOLATION RESULTS FOR ROBOTICS GENERALIZATION AND DISTIIIRIUUIIO GENERALIZATION)
Toxks axw.mbbiingg boswg unteersscooos
Taak: lowenechhlant requasenncooos
34320
127
2x321
407
Warder of Thans Dermaminesces
Warder of Traiiing Demooitretons
Task: towe Is of-innnnoi 50a unw.on.ccooooo corors
6x
Figure 18:
Extrapolation Results of BNSL for Robotic control (and
Generaliza-
tion). Experimental data obtained from the transporter (Zeng et al. (202T) model results from Table of Shridhar et al. (202T). X-axis number of training demonstrations. Y-axis task success score (mean percentage) obtained via 100 evaluations. See Section A.20 for more details.
we find BNSL accurately extrapolates the scaling behavior of a transporter (Zeng et al., 2 model trained via imitation learning to do robotic control tasks. Plots with "unseen-colors" in the plot title evaluate on a test set that contains colors that are unseen (i.e. out-of-distribution)) with respect the training set. Plots with "seen-colors" in the plot title evaluate test set that contains colors that are seen (i.e. in-distribution) with respect to the training set.
In Figure [8]
A.21 EXTRAPOLATION RESULTS FOR OUT-OF- DISTRIBUITTIIIIO DETECTION
Out foution (COD) Detection
13-10" 10
CL0000
8330-1
Nember of Medil Paramotara
Extrapolation Results of BNSL for Out-of- Distribuuuti Detection. Number of model parameters is on the x-axis. Y-axis AUROC. Experimental data obtained from the Outlier Expo- sure results from Figure 23 of Bai et al. (022) when exposed to 30 outlier examples. See Section for more details.
Figure 19:
In Figure 19 find BNSL accurately extrapolates the scaling behavior of Out-of-Distribution Detection performance.
25

---

Published as a conference paper at ICLR 2023
A.22 EXTRAPOLATION RESULTS FOR MOLECULES
Molacules
This Force Mean Abrolai (ecu]]
10
1x.10
SK310
3P
Dataent (tramen)
Figure 20: Extrapolation Results of BNSL for Molecules. Experimental data obtained from
"NequlP L=3" results for the aspirin molecule in MD-17 of Figure 8 of the arXiv version of Batzner ct al. 2022 Y-axis is the test force mean absolute error [eV/A]. X-axis is the training dataset size (frames). See Section A.22 for more details.
In Figure 20
find BNSL accurately extrapolates the scaling behavior of Neural Equivariant Interatomic Potentials (NequIP) graph neural networks (Batzner et al. 2022) trained via minimizing the weighted sum of energy and a force loss terms in order to predict the forces of molecules.
A.23 EXTRAPOLATION RESULTS FOR MATH WORD PROBLEMS
Math Word Proslems
2xx3C
This Rale Ch))
121
10
Frnetningg Datacet Size
Figure 21: Extrapolation Results of BNSL for Math Word Problems. Experimental data obtained from the 12 billion parameter model results in Figure 2 left of Cobbe et al. (2021a). Y-axis is the test solve rate. X-axis is the finetuning dataset size. See Section A.23 for more details.
In Figure 21 we find BNSL accurately extrapolates the scaling behavior of large language models finetuned to solve math word problems.
26

---

Published as a conference paper at ICLR 2023
A.24 EXTRAPOLATION RESULTS FOR DOWNSTREAM PERFORMANCE OF MULTIMODAL CONTRASTIVE LEARNING (L.E. NON-GENERATIVE UNSUPERVISED LEARNING)
Method: CLIP Taak: Imagenet, Zero-Shot
Methvel IIIP: Twak Invegenn Firehiaed inwar Pro
107
23
21
Port Bhor Aate
2.
2x10-1
Test
1.9
30-3
18
17
10
32P
307
12'
GPLOPstmage
GELOPStrrage
Figure 22: Extrapolation Results of BNSL for Downstream Performance of Multimodal Contrastive Learning (i.e. Non-Generative Unsupervised Learning). Experimental data of scaling behavior obtained from Table 10 and Table 11 in arXiv version of Radford et al. (202T). The upstream task is "Contrastive Image Language Pretraining" k.. CLIP) of ResNets training dataset consisting of hundreds of millions of image-text pairs. The x-axis is GFLOPs/image (GigaFLOPs/image) of the forward-pass of model. The Downstream Task is ImageNet classification (i.e. the y-axis of plot). The y-axis of left plot is Zero-Sun Shot Downstream. The y-axis of right plot is performance of model with finetuned linear probe on it. See Section A.24 for more details.
In Figure 23 we show that BNSL accurately extrapolates the scaling behavior of the Downstream Performance of Multimodal Contrastive Learning (i.e. Non-Generative Unsupervised Learning).
A.25 EXTRAPOLATION RESULTS FOR DOWNSTREAM PERFORMANCE ON AUDIO TASKS
Method: Whisper: Task Multinguul Speoch Becogition (Flours): Zuro Shot
8x
Wavd Frw Rate 10:47 Langaages
7x
3
4x30
30
10
Number of Mocel Paramabers
Figure 23: Extrapolation Results of BNSL for Downstream Audio Tasks when Number of Model Parameters on the x-axis. Experimental data obtained from the second plot of Figure Whisper paper (Radford et al. 2022). The downstream task in the plot is downstream zero shot multilingual speech recognition performance on the FLEURS dataset of "Whisper" speech recognition model pretrained on a dataset of 681,070 hours of audio. See Section A.25 for more details.
In Figure 23
show that BNSL accurately extrapolates the scaling behavior of the Downstream
Performance on Audio Tasks.
27

---

Published as a conference paper at ICLR 2023
A.26 PLOTS OF BNSL EXTRAPOLATIONS ON SCALING LAWS BENCHMARK OF ALABDULMOHSIN ET AL. (2022)
144.0
2012
2. 20 Nold 2033
1:17
Eup
33-17"
201220 Nai 970219
Tax 20 Pan 0811
400
Figure 24: Extrapolation Results of BNSL on Downstream Birds 200. X-axis is pretraining dataset size. See Section 5.1 for more details.
28

---

Published as a conference paper at ICLR 2023
Teas prus polles Jomn11
31133
Figure 25: Extrapolation Results of BNSL on Downstream CIFAR-100. X-axis is pretraining dataset size. See Section for more details.
29

---

Published as a conference paper at ICLR 2023
2120
4017
209
To Casenbee Mi-11
COMMES
2000
1447
2117
2007
Ted. Coleebci J03321
21.0
14:37
Figure 26: Extrapolation Results of BNSL on Downstream Caltech101. X-axis is pretraining dataset size. See Section [.I for more details. From eyeballing, we think the subset Caltech100 01 with unsatisfactory extrapolations has unsatisfactory extrapolations due to the maximum (along the x-axis) of the black point used for fitting being near or before a break; this accentuated by not having enough points for fitting for the SciPy fitter to be able to determine whether the break is an actual break or just noisy deviation. See Section for more details on this explanation.
30

---

Published as a conference paper at ICLR 2023
utw.creet 20 chace Siae frseate map
Figure 27: Extrapolation Results of BNSL on Downstream BIG-Bench (BB). X-axis is pretraining dataset size. See Section (52 for more details. From eyeballing, we think the subset of with unsatisfactory extrapolations has unsatisfactory extrapolations due to the maximum (along the x-axis) of the black point used for fitting being near before break; this is accentuated by not having enough points for fitting for the SciPy fitter to be able to determine whether the break is an actual break or just noisy deviation. See Section 6 for more details on this explanation.
Figure 28: Extrapolation Results of BNSL on Neural Machine Translation (NMT). See Section 5.2 for more details.
Figure 29: Extrapolation Results of BNSL on Language Modeling (LM). See Section 5.2 for more details.
31

---

Published as a conference paper at ICLR 2023
Cort
31.0
4014
MIGOIN 21.50 Not 317/31
Snoute 15500 Mal (9.83
3120
11-22
Figure 30: Extrapolation Results of BNSL on Downstream ImageNet. X-axis is pretraining dataset size. See Section 5.1 for more details.
32


Reviews:
[{"review_id": "HOtXho9HwB", "reviewer": "Reviewer_hC3X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose using an functional form of smoothly broken power law namely broken neural scaling laws (BNSL) for extrapolating and generalizing to model scaling behaviors across diverse set of upstream and downstream tasks such zero-shot, prompted, and fine-tuned settings of unsupervised language, vision, reinforcement learning, and arithmetic. The results shows that using BNSL is significantly more accurate in extrapolating the scaling laws based on dataset size and model complexity. These benefits are driven from two important characteristics of BNSL to model non-monotonic transitions in scaling behavior (e.g in double descent) or and inflection points (e.g. in delayed and sharp transitions in arithmetic). ", "summary_of_the_review": "The paper has clear and solid and intuitive idea and observations, however, limited presentation to support the theory behind the scaling law. The presentation is not rigorous in terms of theoretical foundation and too much dependent on the intuitive explanation of the formula's property. The experimental results can be improved and acceptable by including longer extrapolation horizon to show the true capability of this functional form for more sophisticated extrapolation (not just more sophisticated modeling) and this functional form can be compared to non-functional scaling laws (the reviewer were not able to find precedent to non-functional forms of scaling laws in recent literature). To the reviewer, proposing one functional form for all the task settings and networks is not convincing. There might be the case that different scaling formulas can be applied to different neural networks and deeper analysis of the scaling formula in combination with the network properties gives better view of the scaling behavior.    ", "strength_and_weaknesses": "The strengths of this paper include but not limited to clarity of the idea presented in presenting a formula to model scaling behavior neural networks in learning tasks is very clear and somehow intuitive idea. Second, this paper is extremely concise and the basic mathematical idea behind the paper sounds solid and intuitive. Another strength is significant enhancement in the performance of the scaling law in neural networks, at least compared to other M1 to M4 methods. \n\nSome of the weaknesses are presented in the A.3 section. Although the results show stronger performance compared to other proposed formulations, the figures show significant deviations in performance. This is not necessarily a weakness however it is worthy of addressing in the paper. In addition, the performance for cases with more monotonic extrapolation sounds reasonable in some simpler cases however in more complex nonlinear cases, the duration of predictions sound too short to give any notion of the performance. My concern is the number of test points has been cherry picked based on the strength of this approach. In addition, the practicality of modeling scaling behavior is way more valuable when we can predict a certain behavioral patterns sooner than later. In other words, modeling with fewer datapoints and extrapolating on more points than is presented. Finally, I am curious how does the scaling formula is compared to other methods aside from direct formulations such as predictive models and machine learning extrapolations. The reason for my notion is my observation in the A.3 that this formulation misses some basic locally linear extrapolations in some cases. For example, in Fig.6 the results sounds like a simple linear regression in log-log plot models the scaling behavior better and the BNSL actually is underestimating the scaling behavior. This also intensifies my previous concern that longer extrapolation horizon might show even higher and more staggering deviations between results and the BNSL prediction. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Tk73efaSB", "reviewer": "Reviewer_TK8J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a piecewise defined \"broken power law functional\" that can accurately fit the performance of neural networks when quantities like the amount of compute used for training, number of model parameters, or training dataset size varies across several orders of magnitudes. The accuracy of the functional is verified on many different tasks. Compared with existing power laws, the proposed functional can fit non-monotonic behaviors like the double-descent phenomenon", "summary_of_the_review": "The paper proposes a new functional that can more accurately fit the scaling behavior of neural networks. The concern is that the accuracy of the functional comes from the flexibility provided by undetermined parameters, rather than characterizing the true pattern behind the scaling behaviors. ", "strength_and_weaknesses": "Strength: \n\nThe broken power law functional can fit the performance of neural networks over the change of its size more accurately. It can work on more networks, problems, and metric of performances.\n\nWeaknesses:\n\nCompared with existing power laws, the proposed functional has more free parameters. Hence, it is not surprising that it can fit the scaling behavior more accurately. To show the power of this proposed functional, the authors need to show that it can give more accurate predictions to the performance of neural networks than the power laws, or the prediction works in a larger range of network size (or other factors). As shown in the numerical results in the appendix, it seems in many cases the functional does not provide good prediction to the green points even though those points only span less than one order of magnitude in the horizontal axis. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "28rv9z1swOH", "reviewer": "Reviewer_72f9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new functional form for neural scaling laws that corrects for two weaknesses in previous functions: (1) that they could only model strict monotonic behaviour, and (2) they could not express inflection points. The function is intuitive and well-designed and outperforms previous scaling law functions on learning curve fitting, sometimes by orders of magnitude. These are evaluated on many vision and language tasks to demonstrate better accuracy than other methods.\n", "summary_of_the_review": "Overall the paper presents a simple, intuitive function to train and demonstrates its value extensively on both vision and language tasks. Although the paper identifies limitations, it would be nice to also evaluate the degree of the limitations, especially since they are not too hard to test.", "strength_and_weaknesses": "Strengths:\n- The paper has many extensive experiments and the function is intuitive and can be easily implemented\n\nWeaknesses:\n- The authors remark on a limitation which I wish that they numerically investigated. Just how many points do we need to fit this function? Simple neural scaling laws can be fit with even less than 10 points, whereas since this function has lots of parameters, I suspect it can be challenging. It would be nice to run some ablations on this, especially since it is an easy test.\n\n- How do we determine $n$ when forming a BNSL? This seems to be a parameter that must be specified a priori and not learned. Is there intuition in terms of how we can go about selecting it? Is there a consequence for selecting n too large or too small?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster