Title: 4+3 Phases of Compute-Optimal Neural Scaling Laws
Venue: NeurIPS 2024
Accepted: yes
Keywords: scaling laws

compute-optimal curves

high-dimensional probability/statistics

random matrix theory

stochastic optimization
Abstract: We consider the solvable neural scaling model with three parameters: data complexity, target complexity, and model-parameter-count. We use this neural scaling model to derive new predictions about the compute-limited, infinite-data scaling law regime.  To train the neural scaling model, we run one-pass stochastic gradient descent on a mean-squared loss.  We derive a representation of the loss curves which holds over all iteration counts and improves in accuracy as the model parameter count grows.  We then analyze the compute-optimal model-parameter-count, and identify 4 phases (+3 subphases) in the data-complexity/target-complexity phase-plane.  The phase boundaries are determined by the relative importance of model capacity, optimizer noise, and embedding of the features. We furthermore derive, with mathematical proof and extensive numerical evidence, the scaling-law exponents in all of these phases, in particular computing the optimal model-parameter-count as a function of floating point operation budget. We include a colab notebook https://tinyurl.com/2saj6bkj, nanoChinchilla, that reproduces some key results of the paper.

4+3 Phases of Compute-Optimal Neural Scaling Laws

Abstract

We consider the solvable neural scaling model with three parameters: data complexity, target complexity, and model-parameter-count. We use this neural scaling model to derive new predictions about the compute-limited, infinite-data scaling law regime. To train the neural scaling model, we run one-pass stochastic gradient descent on a mean-squared loss. We derive a representation of the loss curves which holds over all iteration counts and improves in accuracy as the model parameter count grows. We then analyze the compute-optimal model-parameter-count, and identify 4 phases (+3 subphases) in the data-complexity/target-complexity phase-plane. The phase boundaries are determined by the relative importance of model capacity, optimizer noise, and embedding of the features. We furthermore derive, with mathematical proof and extensive numerical evidence, the scalinglaw exponents in all of these phases, in particular computing the optimal modelparameter-count as a function of floating point operation budget. We include a colab notebook nanoChinchilla 3 that reproduces some key results of the paper.

1 Introduction

The advent of large language models (LLMs) has changed our perceptions of the landscape of optimization and is resulting in the emergence of new interesting questions related to scaling. Prior to LLMs and other large models, we often viewed the large-scale optimization problems as being limited by the amount of data. In training language models, in contrast, data can be effectively infinite. Thus, compute budgets can be the limitation. This leads to the following natural question: given an architecture, given a fixed compute budget, and having unlimited data, how should one select the model size to minimize loss?

To formally address this question, let us consider the general learning problem,

the number of parameters d is large, and the data vector x is drawn from an unknown distribution. We solve (1) using stochastic algorithms, such as stochastic gradient descent (SGD) with batch size B , under various parameter sizes d , that produce a sequence of iterates { θ r } . A standard formula used in practice to measure compute is the "6ND" formula [26], that is,

Compute (flops 4 f ) = ( iterations of alg. ( r ) × batch size ( B )) × parameters ( d ) . (2)

Therefore, we can plot the loss curve P ( θ r ; d ) = P ( r ; d ) = P ( f / ( d · B ); d ) as a function of flops (see Fig. 1). The question now is: given a fixed number of flops f and given batch size B , how should we choose the parameters d so that we get the best loss, i.e. find d glyph[star] solving the constrained problem

Main contributions. In this work, we analyze a three parameter simple model, which we call powerlaw random features (PLRF) [30, 5]. The three parameters in the PLRF are the data complexity ( α ), target complexity ( β ) and model-parameter count d . Using this model, we derive a deterministic equivalent for the expected loss, as a function of α , β , and d , that captures the training dynamics of one-pass SGD. This can be used to derive numerical predictions for the scaling laws. We also extract exact expressions for the compute-optimal scaling laws and the optimal parameter d glyph[star] ( f ) ∈ argmin d P ( f d · B ; d ) for large 5 d , and give some estimates on the order of d necessary for these scaling laws to take hold.

We also observe for a large portion of the ( α, β ) -phase plane, the optimal parameter is d glyph[star] ( f ) = f 1 / 2 , suggesting a regime of universal scaling behavior (see Fig. 4a and Table 2). This verifies theoretically the Chinchilla scaling [24].

The PLRF is not only analyzable, but also exhibits a rich behavior of compute-optimal curves/loss curves, which are qualitatively and quantitatively different depending on the strengths of the data ( α ) vs. target ( β ) complexity. Particularly, we show that there are 4 distinct (+3 sub phases) compute-optimal curve/loss curve behaviors.

Model constrained compute-optimal curves. In two of the phases (Phase Ia,b,c and Phase II), it is the underlying model that dictates the curves. The algorithm has little/no impact. This appears in two forms. The first behavior are compute-optimal curves controlled by the capacity of the model (Phase Ia,b,c). Here once the algorithm reaches the limiting risk value

 

 

 

 

⌈}√√/(∑{˜⌈}√√}

∐⌈√[∐(//(̂˜√∐(/

〈((/

〈((/

〈((/

〈((/

〈((/

〈((/

̂}⌉√√√˜(}√√]⌉∐⌈

˜]glyph[arrowtp]˜̂(̂}⌉√√√˜

˜⌈}√√

possible (capacity), it is better to increase the model-parameter d . Another type of loss dynamics is due to poor model feature embedding (Phase II). Here the features are embedded in a way which is difficult to train. After an initial large decrease in the loss value, this feature embedding distortion frustrates the algorithm and training slows, but it continues to solve. However, solving to capacity wastes compute, in that it is compute-favored to increase the model parameter count d .

Algorithm constrained compute-optimal curves. For some choices of ( α, β ) (Phase III and IV), it is the noise produced by the SGD algorithm that ultimately controls the tradeoff. Here the algorithm matters. Indeed, another algorithm could change the compute-optimal curves for these phases.

Related work. The key source of inspiration for this work are [24, 26], which identified compute optimality as a fundamental concept in scaling large language models and made a substantial empirical exploration of it. The problem setup was formulated by [30], where additionally data-limited scalings were considered, but compute optimality was not (nor indeed any algorithmic considerations); see also [8] where gradient flow is considered in the same setting.

compute

optimal

F

pp

F

0

flops

loss curve

compute

optimal

Phase II

F

pp

F

ac

F

0

flops

loss curve

compute

optimal

Phase III

K

pp

F

ac

F

0

flops

loss curve

Ia

II

III

IVa

IVb

Ib

Ic

no

power law

high-dim.

line

1

-

1

√

2

0

.

25

α

β

0.5

0.5

pentuple

point

(a) Phase Diagram

compute

optimal

Phase IVb

F

pp

K

pp

F

0

flops

loss curve

compute

optimal

Phase IVa

F

pp

K

pp

F

0

flops

loss curve

There is a substantial body of work considering scaling laws of losses (trained to minimum-loss) of dataset size vs parameter count, in a variety of settings (linear, random features, deep networks). See especially: [5, 39, 41], wherein a 'hidden-manifold' model is considered for the data. We note that as we consider one-pass SGD, some dataset/parameter-count scaling laws are implicit from the results here; however, the training method (one-pass SGD) is, in some regimes, suboptimal given unlimited compute.

For additional related work on random features models (and sample complexity), random matrix theory in machine learning, and other deterministic equivalents for SGD, see Section A. We note that while this paper is fundamentally about computation, but the novel mathematical contributions could also be recast in terms of generalization bounds of one-pass SGD, some of which are new. For a detailed comparison of the convergence rates and sample complexity, see Table 4.

1.1 Problem Setup: SGD on Power-law Random Features

In this work, we analyze the three parameter power-law random features (PLRF) model, that is,

We embed the data vector x ∈ R v in R d through the matrix W ∈ R v × d and construct noiseless targets 6 by dotting a fixed b ∈ R v with the sample x . The use of the matrix W allows the model to have variable capacity ( d ) independent of the data set size. The samples x ∈ R v and labels b ∈ R v have power law dependence, whereas the matrix W has entries distributed as N (0 , 1 /d ) .

(a) Compute-optimal Front

(b) IsoFLOP

100

Scaling Law Exponent

-2.6

Theory Prediction: 0.643

10-1

Measurement: 0.648

-2.8

104

Approach 2: f(x) = 0.17×0.508

Training Loss

10-2

(ssol)

3.0

#Params

Approach 1: f(x) = 0.13x0.525

3.2

10-3

SGD: (α, β) = (0.7, 0.7)

-3.4

10~4

10-5

Volterra

IsoFLOP Slice: 2.4e8

Scaling Law Prediction

Compute-optimal Front

f(x) =50.81x-0.648

104

106

Flops (6ND)

10#

1010

3.6

3.8

-4.0

2.0

2.4e8

1.5e7

3.8e7

9.5e7

6.0e8

2.5

3.0

Log10 (N)

3.5

4.0

4.5

103

107

108

Flops (6ND)

109

1010

Assumption 1 (Data and labels, α and β ) . The samples x ∈ R v are distributed according to ( x j ) ∼ j -α z j for all 1 ≤ j ≤ v and { z j } v j =1 ∼ N (0 , 1) . The labels are scalars constructed by dotting the sample x with a signal b ∈ R v whose entries ( b j ) = j -β .

Without the random matrix W , the α, β are related to what is known in the literature as source and capacity conditions [10, 11, 20, 36]. For a detailed comparison of the parameters and related work, see Section A and Table 3.

The dimensions we consider throughout are always such that v ≥ Cd for C > 1 . Throughout both v and d need to be large, but for some choices of α and β , the v will need to be comparable to d .

Definition 1.1 (Admissible v and d ) . We assume that v ≥ Cd with C > 1 and v, d →∞ . Above the high-dimensional line, which is when 2 α > 1 , we suppose v/d → r ∈ (1 , ∞ ) ∪ {∞} . 7 On the other hand, below the high-dimensional line (2 α < 1) we limit v to be v/d → r ∈ (1 , ∞ ) . 8

One can rewrite the expression in (4) using the convenient form:

Algorithmic set-up. To solve the minimization problem in (5), we use one-pass SGD with minibatches of size B (independent of d ) 9 and constant learning rate γ > 0 : letting θ 0 = 0 , we iterate

The learning rate and batch size will need to satisfy a condition to ensure convergence (Prop. 2.1).

Main goal. Under this setup, our main goal is to characterize the compute-optimal frontier. Precisely, we want to find the parameter count exponent ξ and scaling law exponent η , such that,

Notation. We use P ( θ r ) = P ( r ) when we want to emphasize the iteration counter r . We say A ( r, v, d ) ∼ A ( r, v, d ) for functions A ( r, v, d ) , A ( r, v, d ) > 0 if for every ε > 0 and for all admissible v and d , there exists an r 0 , d 0 such that for all d > d 0 and r ≥ r 0

We write glyph[equivasymptotic] if the upper and lower bounds hold with some constants c, C in place of 1 ∓ ε respectively and glyph[lessorsimilar] , glyph[greaterorsimilar] if only one inequality holds.

2 Learning Dynamics of SGD

Compute-optimal curves (3) for the random features model (4) rely on accurate predictions for the learning trajectory of SGD. Similar to the works of [33, 35], we show that the expected loss under SGD satisfies a convolution-type Volterra equation (for background on Volterra equations, see Section C.3)

The forcing function F ( r ) and kernel function K ( r ) are explicit functions of the matrix ˆ K = D 1 / 2 WW T D 1 / 2 , where D = Diag ( j -2 α , 1 ≤ j ≤ v ) , and Γ ⊂ C a contour enclosing the spectrum of ˆ K ∈ [0 , 1] ,

Intuitively, the forcing function is gradient descent on the random features model and the kernel function is the excess risk due to 1 unit of SGD noise.

Deterministic equivalent. The forcing function F ( r ) and kernel function K ( r ) are random functions depending on the random matrix ˆ K . Indeed, it is the resolvent of ˆ K , ( ˆ K -z ) -1 , which plays a significant role in F and K . We remove this randomness from the expression by using a deterministic equivalent - a technique from random matrix theory.

Formally, we define the deterministic equivalent for the resolvent of ˆ K , denoted by R ( z ) , implicitly via a fixed point equation

This deterministic equivalent R ( z ) is viewed, roughly, as E W [( ˆ K -z ) -1 ] ≈ R ( z ) ; though it is not formally the expectation over W . By replacing the resolvent of ˆ K with R ( z ) , there exists a

03

(a) Scaling Law Exponent

(b) Empirical vs Theory (

α

= 0

.

7

)

1.0

1.0

Params Exponent (Theory)

Params Exponent (Chinchilla approach 1)

0.7

Params Exponent (Chinchilla approach 2)

0.8

0.8

la

0.6

Exponents

SL Measurement

0.5

Scaling Law Exponents

0.6

0.6

0.4

0.4

0.4

Wa

0.2

la

川I

0.2

0.2

Univsersal Scaling

Ic

0.1

0.0

s xp

0.0

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

β

β

deterministic function P ( r ) which solves a convolution Volterra equation, matching (7):

The solution to the Volterra equation with deterministic equivalent (10) numerically exactly matches the training dynamics of SGD, see Fig. 3. A discussion of the deterministic equivalent for ( ˆ K -z ) -1 can be found in Sec. E. All our mathematical analysis will be for the deterministic equivalents, going forward. 10 The derivation of the Volterra equation for the expected loss can be found in Sec. B.

An immediate consequence of (10) is that for convolution Volterra equations bounded solutions occur if and only if the forcing function is bounded and the kernel norm ‖ K ‖ def = ∑ ∞ s =0 K ( s ) < 1 . This directly translates into a sufficient condition on the batch size and learning rate of SGD.

Proposition 2.1 (Sufficient conditions on learning rate and batch) . Suppose learning rate γ and batch B satisfy ‖ K ‖ < 1 and γ ( B +1) < 2 . Then P ( r ) is bounded.

Remark 2.1. Below the line 2 α = 1 , the kernel norm diverges with v for fixed constant γ , and so we must take γ → 0 to ensure bounded solutions. Thus, provided γ ∼ v 2 α -1 , then

Thus, the kernel norm, ‖ K ‖ , is always constant order for all α .

The batch B and γ can depend on d . For simplicity, we only consider B order 1 in this work. For a proof of the necessary and sufficient conditions on γ and B , see Prop. C.2, and see Cor. G.1 for the asymptotic on ‖ K ‖ .

The Volterra equation in (10) can be analyzed to give a more explicit formula for P (see Section C.3.2 for proof).

Observed Exponent

10 4

10 8

10 12

10 16

10 20

10 24

0.30

0.35

0.40

0.45

0.50

Asymptotic Prediction

α

=

0.3

α

=

0.35

α

=

0.4

α

=

0.45

D

P

/(

F

0

+

F

pp

+

F

ac

+

K

pp

/

γ

)

200

400

800

1600

3200

1

10

100

1000

10 4

10 5

10 6

1.0

1.5

2.0

2.5

3.0

3.5

Step

Compute

-

optimal slope

SL exp

Data exp

1000

10 5

10 7

10 9

0.45

0.50

0.55

0.60

0.65

0.70

Num.

params

(

d

)

(a)

(b)

(c)

Compute

-

optimal slope

SL exp

Data exp

1000

10 5

10 7

10 9

0.45

0.50

0.55

0.60

0.65

0.70

Num.

params

(

d

)

Compute

-

optimal

slope

SL exp

Data exp

1000

10 5

10 7

10 9

0.45

0.50

0.55

0.60

0.65

0.70

Num.

params

(

d

)

Theorem 2.1 (Approximation solution for P ) . Suppose γ and B are at most half the convergence threshold and 2 α +2 β > 1 , α > 1 4 . 11 There exists an M > 0 large and a constant C = C ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M ,

The convolution K ∗ F further simplifies

for some constants ˜ c = ˜ c ( α, β, M ) and ˜ C = ˜ C ( α, β, M ) > 0 independent of d .

Remark 2.2. If we were to run gradient descent instead of SGD (i.e., γ small), then we would only have the forcing term, that is, P ( r ) = F ( r ) . The measurable effect of SGD comes from the second term that contains the kernel function. For this reason, we refer to SGD noise as 1 γB · K ( r ) .

In light of (13) and (14), we have trapped the training loss between the sum of F and K , so it suffices now to understand the forcing and kernel functions.

2.1 Forcing function and kernel function

We decompose the forcing function (11), F , and the kernel function, (12), K , into

Each term is explicit and has an asymptotic equivalence (when 1 glyph[lessorsimilar] γBr glyph[lessorsimilar] d 2 α ) given by

The two error terms are such that for large d with 1 glyph[lessorsimilar] γBr glyph[lessorsimilar] d 2 α ,

for some constant C > 0 . For γBr glyph[greaterorsimilar] d 2 α , the forcing function F ( r ) glyph[equivasymptotic] F 0 ( r ) , the limiting risk value. The terms arise from different parts of the spectrum of the deterministic equivalent for ˆ K (see Fig. 6).

Point mass at 0 : F 0 (0) = F 0 ( r ) is the limiting value of P ( r ) glyph[equivasymptotic] d -2 α +max { 0 , 1 -2 β } as r →∞ . It occurs because the loss is irreducible ( d < v ) , that is a component of the target is not in the image of the RF model (or equivalently that ˆ K has a kernel).

Aligned features: The function F pp ( r ) represents gradient descent on the components of features which are aligned to the underlying population features. Indeed, if we ran gradient descent on the population loss without a random features map (or a diagonal W ), this would be the loss curve.

Distorted features: The function F ac ( r ) is the result of feature distortion, where the matrix W leads to an embedding where a small component of the leading features is distributed across many different eigenmodes. These are still solvable, and given enough compute these will eventually be used, but they are much slower to solve.

Aligned kernel: K pp ( r ) is the excess risk due to 1 unit of SGD noise, which is then solved according to population gradient descent.

Out of brevity, we relegate the exact definitions of F 0 , F pp , F ac , and K pp and all proofs of the asymptotics in Table 1 and analyses of the functions to Section F, G, and H.

3 The 4 Phases

We now put together a coherent picture of the effect of different choices of α (data complexity) and β (target complexity) and their impact on the compute-optimal frontier. By Theorem 2.1, we estimate

Explicitly, we show that the functional form, or scaling law for the PLRF model is

Fig. 5a. shows empirically that this equivalence of P ( r ) is quite good. The first two terms F pp ( r ) (i.e., r -σ 1 ) and F 0 ( r ) (i.e., d -τ 1 ) are often called in the literature as time and model bottlenecks respectively. The functional form using only these two components, i.e., P ( r, d ) glyph[equivasymptotic] r -σ 1 + d -τ 1 were used to find compute-optimal exponents in [24, 9] and in concurrent work [28]. Because the functional form considered in [9, 28] are missing the two other terms (cross-term F ac and SGD noise K pp ) , the compute-optimal curves in [9, 28] agree only in Phase Ia with our results. Importantly, we show that the cross-term, i.e., F ac ( r ) , and SGD noise, K pp ( r ) , can indeed affect the compute-optimal exponents. (The cross-term also appeared in concurrent work on ridge regression [18].)

The 4 distinct phases (see Fig. 2a) decompose the ( α, β ) -plane based on the shape of the loss curve P ( r ) , that is, which of the distinct components of the forcing function (i.e., F 0 , F pp , F ac , ) and/or kernel function (i.e., K pp ) dominate the loss curve at a given iteration r . See Table 2 for loss description in each phase. Cartoon pictures of the different features of the loss curves are shown in Fig. 2. For each phase, we derive a compute-optimal curve in Section 3.1.

The high-dimensional line, which occurs where 2 α = 1 , distinguishes the phases where the v -dimension can be big and independent of d (Phase Ia, II, III, 2 α > 1 ) and the phases where d and v must be related to each other (Phase Ib, Ic, IVa, IVb, 2 α < 1 ). When 2 α +2 β < 1 , the loss does not exhibit any power-law decay as the limit level stops going to 0 as d →∞ (purely as a consequence of having selected the regime v > d ). Moreover, there exists an interesting critical point α = β = 1 2 where all the parts of the forcing function and kernel mix and interact with each other. The behavior of the loss at the pentuple point (see Fig 2a) we leave for future research. Across each of the phase boundaries the compute-optimal curves are continuous, but not necessarily differentiable; in contrast, d glyph[star] is discontinuous across some phase boundaries.

3.1 Compute-optimal Curves

To simplify the computations for compute-optimal curves, we introduce the following curve

The function ˜ P ( r, d ) achieves the same power law behavior as the original compute-optimal curve P ( r, d ) (i.e., the slope of the compute-optimal curve is correct) and deviates from the true curve by an absolute constant (independent of d and f ). Note that some of the terms in the max function

(19) should be taken to be 0 when not defined for the different phases. Therefore, we derive the compute-optimal curves by solving the problem

See Table 2 for the exact expressions for d glyph[star] ( f ) and the compute-optimal curve ˜ P glyph[star] ( f ) for each phase. A more detailed description with proofs can be found in Section C.4 and Section D.

Now to derive d glyph[star] and ˜ P glyph[star] , we recall that the functions F 0 , F pp , F ac , K pp take the form c × d -σ i × ( f d · B ) -τ i (16). Therefore, ˜ P glyph[star] ( f / ( d glyph[star] · B ) , d glyph[star] ) must occur at corner point where two functions meet. These tradeoffs between the two functions for which the compute-optimal point occurs are shown in Fig. 2 and Table 2.

Details for each phase. We describe the qualitative and quantitative properties of compute-optimal curves for each phase. These are broken down into model constrained (Phase I, II) vs. algorithm constrained (Phase III, IV), i.e., whether the PLRF model or SGD is the constraining feature.

Phase Ia, Ib, Ic. Capacity constrained. Phase Ia ( 2 α > 1 , 2 β < 1) , Ib ( 2 α < 1 , 2 β < 1 , 2( α + β ) > 1 ), Ic are characterized by having the simplest loss description, P ( r ) glyph[equivasymptotic] F pp ( r ) + F 0 ( r ) . Here the SGD noise is irrelevant and one would have the same loss (and thus compute-optimal curve) as gradient descent on the population loss. Compute optimality is characterized by training the model completely (to its limit loss) and choosing the model parameter count large enough so that at the end of training, the smallest loss is attained. The main distinctions between Phase Ia, Ib, Ic are the model capacities (i.e., F 0 ( r, d ) = d -2 α +1 -2 β in Ia, Ib, and F 0 ( r, d ) = d -2 α in Ic) and the dependence of dimension in the learning rate due to Ib,Ic being below the high-dimensional line. Consequently, while the qualitative features of the loss curve are the same for Ia, Ib, and Ic, the actual values of the compute-optimal curve vary across the different regions. Notably, in Phase Ib, the compute-optimal parameter is d glyph[star] = f 1 / 2 and it is independent of α and β .

Phase II. Distortion constrained. Phase II ( 2 α > 1 , 2 β > 1 , β < α ) has a loss curve where the F ac is important, that is, P ( r ) glyph[equivasymptotic] F pp ( r ) + F ac ( r ) + F 0 ( r ) . The F ac term becomes the dominant term after running for some intermediate amount of time d c ; in fact it is compute-optimal to stop at this point, and then select the number of model parameters so to minimize the loss with this early stopping criterion. It transpires that across all phases, it never pays to solve through the F ac part of the loss curve - it is always better to just increase the number of model parameters.

Phase III. SGD frustrated, distortion constrained. In this phase (2 α > 1 , 2 β > 1 , β > α ) , SGD noise is important. The loss curve is P ( r ) glyph[equivasymptotic] F ac ( r ) + F 0 ( r ) + 1 γB K pp ( r ) . Notably, in this phase, the compute-optimal parameter is d glyph[star] ( f ) = f 1 / 2 , which is independent of α and β . PLRF that fall within this phase have the same scaling law regardless of data complexity and target complexity. Moreover, the tradeoff occurs, like in Phase II, once the optimizer reaches the F ac -dominated part of the loss curve. Unlike in Phase II, the optimization is slowed by SGD noise ( K pp ) leading up to that point. We note that there is a dimension-independent burn-in period required for SGD noise to dominate, and for small numerical simulations, one may actually observe an ( F pp , F ac ) tradeoff.

Phase IV. SGD frustrated, capacity constrained. Like Phase III, SGD noise is important. The SGD algorithm in Phase IV will be distinguished from gradient descent. As one approaches the high-dimensional line ( 2 α = 1 ) in Phase III, the F ac ( r ) disappears. It becomes too small relative to F pp and K pp . Moreover at the high-dimensional line, F pp becomes important again. Thus, the loss curve in Phase IV (a and b) look like P ( r, d ) glyph[equivasymptotic] F pp ( r, d ) + F 0 ( r, d ) + 1 γB K pp ( r, d ) . The distinction between Phase IVa ( 1 -1 √ 2 < α < 0 . 5 , 2 β > 1 ) and Phase IVb ( 1 4 < α < 1 -1 √ 2 , 2 β > 1 ) is where the compute-optimal tradeoff occurs. It changes from K pp = F 0 (Phase IVa) to F pp = K pp (Phase IVb). In particular it can be (Phase IVb) the SGD noise is so large that increasing the model parameter count is compute-optimal. We note that in this phase d must be taken very large (in particular larger than we could numerically attain) to get quantitative agreement between the exponents and theory.

Other observations. In Phase III, Ib, and IVa, the optimal parameter d glyph[star] = f 1 / 2 (see dashed lines in Fig. 4a). These phases, taken together, encompass a large section of the ( α, β ) -phase plane. This suggests that there is a potential universal scaling law. Moreover using 1 A100-GPU-day of compute, one reaches scales of d where the observed exponents in the scaling laws - SGD, the theoretically-derived Volterra equation eq. (10), and the equivalence of P ( r ) eq. (17) - are still changing (see Fig. 5b and c). This serves as a potential warning for empirically derived scaling laws. Additionally, although we have identified the lower-left of the phase diagram ( α + β < 1 / 2 ) as "no power-law", this designation relies on the assumption v > d , which could be relaxed to interesting effect in more realistic (e.g. non-linear) models.

Compute-optimal learning rate and batch. Previously, we have used B = 1 and the maximal learning rate allowed. One can also consider finding the compute-optimal curves with respect to batch size and learning rate, i.e., find d glyph[star] , γ glyph[star] , B glyph[star] such that

In Section I, we show that P ( f dB , γ, d ) is monotonic in B and therefore B = 1 is optimal. Similarly for γ , in Phases I, II, III, the loss P ( f dB , γ, d ) is monotonic and thus the maximally stable learning rate is optimal. For Phase IV, this is not true. There does exist an optimal γ glyph[star] (with B = 1 ),

where the tradeoff occurs between 1 γ K pp ( r ) = F 0 ( r ) and Phase IVa and IVb collapse to a single phase. This is proven in Proposition I.1.

Conclusion. Weanalyze a simple three parameter model, PLRF, and derive deterministic expressions for the training dynamics (see Volterra equation (10)). We then extract compute-optimal scaling laws for large d . We identify 4 phases (+3 subphases) in the ( α, β ) -phase plane, corresponding to different compute-optimal curve/loss behaviors. These phase boundaries are determined by the relative importance of model capacity (Phase I, IV), poor embedding of the features (Phase II, III), and the noise produced by the SGD algorithm (Phase III, IV). The latter suggesting that another stochastic algorithm might change the compute-optimal curve; we leave this interesting direction to future research. We also show evidence of a universal scaling law which we also leave for future research to explore.

Acknowledgments and Disclosure of Funding

C. Paquette is a Canadian Institute for Advanced Research (CIFAR) AI chair, Quebec AI Institute (MILA) and a Sloan Research Fellow in Computer Science (2024). C. Paquette was supported by a Discovery Grant from the Natural Science and Engineering Research Council (NSERC) of Canada, NSERC CREATE grant Interdisciplinary Math and Artificial Intelligence Program (INTERMATH-AI), Google research grant, and Fonds de recherche du Québec - Nature et technologies (FRQNT) New University Researcher's Start-Up Program. Research by E. Paquette was supported by a Discovery Grant from the Natural Science and Engineering Council (NSERC). Additional revenues related to this work: C. Paquette has 20% part-time employment at Google DeepMind.

References

Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From highdimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks. In The Thirty Sixth Annual Conference on Learning Theory (COLT) , pages 1199-1227. PMLR, 2023.

Søren Asmussen, Soren Asmussen, and Sren Asmussen. Applied probability and queues , volume 2. Springer, 2003.

Krishna B Athreya, Peter E Ney, and PE Ney. Branching processes . Courier Corporation, 2004.

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science , 6(1):26-50, 2024.

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proc. Natl. Acad. Sci. USA , 121(27):Paper No. e2311878121, 8, 2024.

Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices , volume 20. Springer, 2010.

Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla Scaling: A replication attempt. arXiv preprint arXiv:2404.10102 , 2024.

Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A Dynamical Model of Neural Scaling Laws. In Proceedings of the 41st International Conference on Machine Learning (ICML) , volume 235 of Proceedings of Machine Learning Research , pages 4345-4382. PMLR, 2024.

Blake Bordelon and Cengiz Pehlevan. Learning Curves for SGD on Structured Features. In International Conference on Learning Representations (ICLR) , 2022.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7:331-368, 2007.

Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random features. Advances in Neural Information Processing Systems , 31, 2018.

Chen Cheng and Andrea Montanari. Dimension free ridge regression. arXiv preprint arXiv:2210.08571 , 2022.

Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models. Inf. Inference , 13(4):Paper No. iaae028, 107, 2024.

Romain Couillet and Zhenyu Liao. Random Matrix Methods for Machine Learning . Cambridge University Press, 2022.

D. Cruz-Uribe and C. J. Neugebauer. An Elementary Proof of Error Estimates for the Trapezoidal Rule. Math. Mag. , 76(4):303-306, 2003.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime. Advances in Neural Information Processing Systems , 34, 2021.

Stéphane d'Ascoli, Marylou Gabrié, Levent Sagun, and Giulio Biroli. On the interplay between data structure and loss function in classification problems. Advances in Neural Information Processing Systems , 34:8506-8517, 2021.

Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. Dimension-free deterministic equivalents for random feature regression. arXiv preprint arXiv:2405.15699 , 2024.

Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal Distributed Online Prediction Using Mini-Batches. Journal of Machine Learning Research , 13(1), 2012.

Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large step-sizes. The Annals of Statistics , 44(4):1363 - 1399, 2016.

Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM Journal on Mathematics of Data Science , 6(2):400-427, 2024.

Gustaf Gripenberg. On the resolvents of nonconvolution Volterra kernels. Funkcial. Ekvac. , 23(1):83-95, 1980.

Walid Hachem, Philippe Loubaton, and Jamal Najim. Deterministic equivalents for certain functionals of large random matrices. Ann. Appl. Probab. , 17(3):875-930, 2007.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems , volume 35, 2022.

Hong Hu, Yue M. Lu, and Theodor Misiakiewicz. Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime. arXiv preprint arXiv:2403.08160 , 2024.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alex Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Kiwon Lee, Andrew Cheng, Elliot Paquette, and Courtney Paquette. Trajectory of mini-batch momentum: batch size saturation and convergence in high dimensions. Advances in Neural Information Processing Systems , 35:36944-36957, 2022.

Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling Laws in Linear Regression: Compute, Parameters, and Data. arXiv preprint arXiv:2406.08466 , 2024.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34:18137-18151, 2021.

Alexander Maloney, Daniel A. Roberts, and James Sully. A Solvable Model of Neural Scaling Laws. arXiv preprint arXiv:2210.16859 , 2024.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75(4):667-766, 2022.

Gabriel Mel and Jeffrey Pennington. Anisotropic random feature regression in high dimensions. In International Conference on Learning Representations , 2021.

Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality. In Proceedings of Thirty Fourth Conference on Learning Theory (COLT) , volume 134, pages 3548-3626, 2021.

Courtney Paquette and Elliot Paquette. Dynamics of stochastic momentum methods on largescale, quadratic models. Advances in Neural Information Processing Systems , 34:9229-9240, 2021.

Courtney Paquette, Elliot Paquette, Ben Adlam, and Jeffrey Pennington. Homogenization of SGD in high-dimensions: exact dynamics and generalization properties. arXiv preprint arXiv:2205.07069 , 2022.

Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems , 31, 2018.

Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in Neural Information Processing Systems , 30, 2017.

Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International conference on machine learning , pages 71-79. PMLR, 2013.

Utkarsha Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802 , 2020.

Jack W Silverstein and Zhi Dong Bai. On the empirical distribution of eigenvalues of a class of large dimensional random matrices. Journal of Multivariate analysis , 54(2):175-192, 1995.

James B. Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory. In International Conference on Learning Representations (ICLR) , 2024.

Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Last iterate convergence of SGD for Least-Squares in the Interpolation regime. In Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 21581-21591, 2021.

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Benign overfitting of constant-stepsize SGD for linear regression. Journal of Machine Learning Research , 24(326):1-58, 2023.

4+3 Phases of Compute-Optimal Neural Scaling Laws

Supplementary material

Broader Impact Statement. The work presented in this paper is foundational research and it is not tied to any particular application. The set-up is on a simple well-studied random features model with synthetic data and solved using a commonly deployed algorithm - stochastic gradient descent. We present (theoretical) compute-optimal curves for this model. The results are theoretical and we do not anticipate any direct ethical and societal issues. We believe the results will be used by machine learning practitioners and we encourage them to use it to build a more just, prosperous world.

Outline of the paper. The remainder of the article is structured as follows: in Section A, we provide additional related work. In Section B, we derive the convolution-type Volterra equation for the expected risk under SGD, (7). In Section C.1, we analyze the Volterra equation under the deterministic equivalent. A discussion on the convergence threshold for P ( r ) including a necessary and sufficient condition for bounded solutions of (10) (Proposition C.2) and a proof of Proposition 2.1 are provided in Section C.2. Some background on Volterra equations and their solutions are provided in Section C.3.1 followed by the proof of Theorem 2.1 in Section C.3.2. We finish this section with a detailed description and proofs for the risk curves in all phases, Section C.4. Section D is devoted to deriving and proving the compute-optimal curves in Table 2. We follow this by Section E which analyzes the deterministic equivalent for the resolvent of ˆ K . Here we examine the spectrum of ˆ K from a random matrix point of view. In particular, in this section, we prove estimates on the fixed point equation, m , see eq. (9). We then give explicit descriptions of the components of the forcing function, F 0 , F pp , F ac , as contour integrals and show that the error terms error F are small, see Section F. We do the same with the kernel function K and kernel norm in Section G. In Section H, we derive the asymptotic formulas for the components of the forcing and kernel functions (see Table 1) used in the compute-optimal curve derivations. Finally, we end with some additional numerical experiments (and their experimental setups) as well as detailed descriptions of the different approaches to estimating the exponents in the scaling law and optimal model-parameter, Section J.

Contents

A Additional Related Work.

Random features and random matrices. This paper uses random matrix theory to analyze a random features problem, which in statistical language would be the generalization error of the one-pass SGD estimator. Random matrix theory has played an increasingly large role in machine learning (see for [14] for a modern introduction).

The input we need for our random matrix analysis is for sample covariance matrices with power-law population covariance (i.e. linear random features). The analysis of sample covariance matrices precedes their usage in machine learning (see e.g. [6]), but to our knowledge, a detailed study of all parts of the spectrum of sample covariance matrices with power-law population covariances has not appeared before. The narrower study of ridge regression has been extensively investigated (see for e.g. [4, 12]), and the concurrent work [18] provides a complete analysis of the ridge regression problem when 2 α > 1 . However, while (roughly speaking) ridge regression requires analysis of resolvent ( A -z Id ) -1 statistics for negative spectral parameter z (which might be very close to 0 ), the analysis in this work requires resolvent statistics for essentially all z .

There is a larger theory of nonlinear random features regression, mostly in the case of isotropic random features. Including nonlinearities in this model is a natural future direction; for isotropic random features with proportional dimension asymptotics this has been explored in works such as [31] and for some classes of anisotropic random features in [32, 17, 29] (we mention that lots of the complexity of the analysis of power-law random features arises from the analysis of the self-consistent equations - indeed the self-consistent equations we use date to [40], but the analysis of these equations may still be novel). This strongly motivates non-proportional scalings (which would be inevitable in power-law random features with nonlinearities); in the isotropic case, the state of the art is [25].

Random features regression, 'source/capacity' conditions, and SGD. A large body of kernel regression and random features literature is formulated for 'source/capacity' conditions, which are power-law type assumptions that contain the problem setup here, when 2 α > 1 (the low-dimensional regime). For convenience, we record the parameters

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

Here we have taken r as the limit of those r 's for which the source/capacity conditions hold (see Table 3). We note that in this language r is often interpreted as 'hardness' (lower is harder), and that r ∈ (0 , 0 . 5) , r ∈ (0 . 5 , 1 . 0) and r ∈ (1 . 0 , ∞ ) correspond to 3 regimes of difficulty which have appeared previously (see the citations below); they are also precisely the 3 phases Ia, II, and III.

The authors of [37] establish generalization bounds for random feature regression with power-law structures in 2 α > 1 case. These bounds were sharpened in [16] and extended in [18] (see also the earlier [10] which shows kernel ridge regression is 'minimax optimal' under various 'source-capacity conditions'); we give a comparison to these bounds in Table 4, but we note that the problem setup we have is not captured by 'minimax optimality' (in particular minimax optimality is worst-case behavior over a problem class, and our problem setup is not worst-case for the traditional source/capacity conditions)

We note that this paper is fundamentally about computation, but the novel mathematical contributions could also be recast in terms of generalization bounds of one-pass SGD, some of which are new. The work of [11] compares SGD to kernel ridge regression, showing that one-pass SGD can attain the same bounds as kernel ridge regression and hence is another minimax optimal method (again under 'source-capacity' conditions). See also [20] which considers similar statements for SGD with iterate averaging and [36] for similar statements for multipass SGD; see also [38, 19] which also prove the single-batch versions of these. These bounds attain the minimax-optimal rate, which are worse than the rates attained in this paper (see Table 4 for a comparison).

Dynamical deterministic equivalents, Volterra equations and ODEs. Using the deterministic equivalents for random matrix resolvents [23], we in turn derive deterministic equivalents for the risk curves of SGD.

The method of analysis of the risk curves in this paper is by formulation of a convolutional Volterra equation [33]. This can be equivalently formulated as a system of coupled difference equations for weights of the SGD residual in the observed data covariance, which generalizes beyond the

least-squares context [13]; in isotropic instances, this simplifies to a finite-dimensional family of ODES [1]. This can also be generalized to momentum SGD methods [34] and large batch SGD methods [27]. Convolution-Volterra equations are convenient tools, as they are well-studied parts of renewal theory [2] and branching process theory [3].

Another method of analysis is dynamical mean field theory. The closest existing work to this one in scientific motivations is [8], which uses this technique. This formally can be considered as a type of Gaussian process approximation, but for a finite family of observables ('order parameters'). In instances of one-pass SGD (including in anisotropic cases), this is rigorously shown to hold in [21]. The analysis of the resulting self-consistent equations is nontrivial, and [8] does some of this analysis under simplifying assumptions on the structure of the solutions of these equations.

Besides these works, there is a large theory around generalization error of SGD. The work of [42] gives a direct analysis of risks of SGD under 'source/capacity' type assumptions which formally capture the F pp parts of the Phase Ia/II loss curves. The risk bounds of [43] give non-asymptotic estimates which again reproduce tight estimates for the F pp parts of the loss (note that to apply these bounds to this case, substantial random matrix theory needs to be worked out first); see also concurrent work [28] where some of this is done.

B Derivation of Volterra equation

We begin by deriving a Volterra equation for the population loss P ( θ ) , (5). Fix a quadratic q : R d → R , i.e., a function q ( x ) = x T Ax + e T x + c for fixed matrix A ∈ R d × d , vector e ∈ R d and constant c ∈ R . Let us consider the filtration F r = σ ( W,θ 0 , . . . , θ r ) which conditions on W and the past iterates. Then we have from Taylor theorem,

We need to plug the expression for SGD in (6) into the above. The first thing we observe is that we need moments of Gaussians via Wick's formula: for fixed vectors v i ∈ R v , i = 1 , 2 , 3 , 4 , and x ∼ N (0 , D )

Here we recall that the ( v × v ) -matrix D def = Diag ( j -2 α : 1 ≤ j ≤ v ) . Using these moment calculations, we can compute explicitly each of the terms in (22).

Gradient term. First, we consider the gradient term in (22). A simple computation yields

Quadratic term. We now turn to the quadratic term in (22). Supposing x and ˆ x are independent samples,

Continuing, we have

Let ∇ 2 q = ∑ v j =1 v j ⊗ ˜ v j . Now we note the following

This, after taking expectations, is in the form for us to apply the moment computations in (23). Using these moments, we get the following expression:

Now we simplify the 2nd term in the summand

Moreover, as x and ˆ x are independent, we see that

As a result, we deduce by combining (27), (28), (29), and (30) with (26) gives the following representation for the expected quadratic term

Volterra equation. Using the simplified gradient and quadratic terms, we can now state the expected change in any quadratic q : R d → R evaluated at an iterate of SGD (6):

We can write R v = Im ( W ) ⊕ W ⊥ . Thus, there exists ˇ b ∈ R d and ˙ b ∈ R v such that one can write b = W ˇ b + ˙ b , that is, something in the image of W and something in the co-ker of W , i.e.,

One-step update formula. Using this observation, we have a formula for the expectation of the quadratic q : R d → R ,

We observe that all the terms on the right hand side of the above (34) involve the matrix W T DW ∈ R d × d . Consequently, let ( λ j , w j ) for j = 1 , . . . , d be the eigenvalue-eigenvector of W T DW with ‖ w j ‖ = 1 . Now define

We will write our Volterra equation in terms of ρ j 's. Note we can express the loss P ( θ r ) = ‖ D 1 / 2 ( Wθ r -b ) ‖ 2 by

We can now plug ρ 2 j into (34). For this, we need to compute ∇ ρ 2 j and ∇ 2 ρ 2 j :

Then we have that

Using an integrating factor, we can implicitly solve this expression

and thus, we have a discrete Volterra equation

Let us define ˇ K def = W T DW . Using the expression in (36),

Let us define the kernel

Discrete volterra equation for the loss for ˇ K = W T DW . Let r be the number of iterates of SGD. Then

We can also write (40) in terms of ˆ K def = D 1 / 2 WW T D 1 / 2 . To see this, set D 1 / 2 W = V √ Ω U T where ˇ K = U Ω U T and ˆ K = V Ω V T . Then we see that

Discrete volterra equation for the loss with ˆ K = D 1 / 2 WW T D 1 / 2 . Let r be the number of iterates of SGD. Then

C Analysis of Volterra Equation

From now on, we consider the setting where the initialization of SGD is θ 0 = 0 . Let us introduce the forcing function:

and recall the kernel function K ( s ) :

While these representations are easy to see from the derivation of the V olterra equation, a more useful representation of the forcing function and the kernel function is through contour integrals over the spectrum of ˆ K . With this in mind, let Γ be a contour containing [0 , 1] . Note that by the assumptions

on ˆ K , the largest eigenvalue is normalized to be 1 ; hence Γ contains the spectrum of ˆ K . Then the forcing function takes the form

and the kernel function

Then one can write the Volterra equation (41) as the forcing function plus a convolution with the kernel and the expected loss, i.e. ,

C.1 Deterministic equivalent of the loss under SGD

The forcing functions F ( r ) and kernel function K ( r ) are random functions as they depend on the random matrix W . Moreover the expressions via contour integration show that both of these functions can be described in terms of the random matrix ˆ K = D 1 / 2 WW T D 1 / 2 . Indeed it is the resolvent of ˆ K ,

which plays a significant role in F and K and thus in the expected loss E [ P ( θ r ) | W ] . To analyze the power law behavior of the expected loss, it would be helpful to remove the randomness in ˆ K , i.e., W . We do this by finding a deterministic equivalent for the resolvent of ˆ K , R ( ˆ K,z ) = ( ˆ K -z ) -1 , using techniques from random matrix theory. Intuitively, we want to take the expectation over the random matrix W ; though not formally true.

Formally, we define the deterministic equivalent for the resolvent R ( ˆ K,z ) , denoted by R ( z ) implicitly via a fixed point equation

As mentioned earlier, this deterministic equivalent, R ( z ) can be viewed, roughly as,

though it is not formally the expectation over W .

Using this deterministic expression for the resolvent of ˆ K , we defined deterministic expressions for the forcing function via the contour representation of F ( r ) in (43)

and the kernel function in (44)

Using the deterministic expressions for the forcing function F and kernel function K , we define the deterministic function P ( r ) : N → R as the solution to the (discrete) convolution-type Volterra equation:

We note the similarity with the Volterra equation for SGD. We conjecture that the two processes are close: for { θ r } the sequence of iterates generated by SGD with θ 0 = 0 and any ε > 0 ,

for all admissible v , d with probability going to 1 as d →∞ .

We leave this for future research; we suspect it is true based upon existing of deterministic equivalence theory for random matrices and numerical evidence.

C.2 Convergence threshold

A natural question is: for what choices of batch B and learning rate γ does P converge? To answer this, we introduce an additional quantity, the kernel norm defined as

Proposition C.1 (Kernel norm) . The kernel norm is satisfies

If 2 α > 1 , then v be taken to equal ∞ , that is,

In the case that 2 α < 1 , we have

In all cases, we choose γ so that the kernel norm is asymptotic to a strictly positive constant.

A well-known result about convolution-type Volterra such as (49) is that the solution of convolutiontype Volterra equation is bounded if and only the forcing function F ( r ) is bounded and the kernel norm ‖ K ‖ < 1 . This naturally leads to conditions for our specific forcing function and kernel function.

Remark C.1 (Convergence threshold conditions.) . The forcing function F is bounded and the kernel norm ‖ K ‖ < 1 for (47) and (48) , respectively, if and only if

The first term ensures that the forcing function of the Volterra equation in (49) goes to 0 (i.e., bounded) and the second condition is the same kernel norm bound. Moreover, we can think of condition (i). as the same condition needed for gradient descent to converge while the kernel norm is the effect of noise from SGD.

We also note that in light of Proposition C.1 the kernel norm does not involve the batch size B . Therefore the condition ‖ K ‖ < 1 only places a condition on the learning rate (see below).

We now state necessary/sufficient conditions on the batch size and learning rate (51) (Proof of Prop. 2.1).

Proposition C.2 (Necessary/Sufficient conditions on learning rate and batch size) . The learning rate, γ > 0 and batch size, B > 0 , satisfy

if and only if the solution P ( r ) to the convolution-type Volterra equation (10) is bounded.

Proof. From (51), we need that | 1 -2 γBλ j + γ 2 B ( B +1) λ 2 j | < 1 , for all λ j ∈ [0 , 1] . For this, we consider two cases.

First, suppose that 1 -2 γBx + γ 2 B ( B +1) x 2 < 1 for all x ∈ [0 , 1] . We have that

The roots are precisely x = 0 and x = 2 γ ( B +1) . If 2 / ( γ ( B +1)) > 1 , then the inequality in (53) always holds. Therefore, we need that γ ( B +1) < 2 .

The roots of the left-hand side are complex and thus the inequality always holds.

Remark C.2. Below the high-dimensional line, 2 α < 1 , the kernel norm diverges with v for fixed constant γ , and so we must take γ → 0 to ensure bounded solutions. Furthermore, with γ → 0 (at any rate depending on v ) we have the asymptotic equivalence

For a proof of the asymptotic for ‖ K ‖ , see Corollary G.1.

Remark C.3. Similar results hold for the expected SGD loss (via the Volterra equation (45) ) by replacing ‖ K ‖ with ‖ K ‖ .

C.3 Simplification of the Volterra Equation

While convolution-type Volterra equation such as (49) are quite nice and well studied in the literature (e.g., [22]), we need an approximation of the solution to it to have better understanding of computeoptimal curves. In this section, we show that we can bound (above and below) P ( r ) by a constant multiple of the forcing function F and kernel function K .

C.3.1 Background on Volterra equations

To do this, we need some background on general convolution-type Volterra equations of the form:

where f ( t ) is a non-negative forcing function and K ( t ) is a monotonically decreasing non-negative kernel function.

Under mild assumptions such as ‖ K ‖ = ∑ ∞ t =0 K ( t ) < 1 and the forcing function f is bounded, then there exists a unique (bounded) solution P ( t ) to (55) and the solution is given by repeatedly convolving the forcing function with K (see, e.g., [22, Theorem 3.5]),

This representation of the solution to (55) enables us to get good bounds on P ( t ) . First, we state and prove a lemma attributed to Kesten's Lemma [3, Lemma IV.4.7].

Lemma C.1 (Kesten's Lemma) . Suppose the kernel function K is positive and monotonically decreasing and ‖ K ‖ < ∞ . Moreover suppose for some ε > 0 , there exists a T ( ε ) > 0 such that

Then for all n ≥ 0 ,

Proof. Define

Then a 1 = 1 , and we are trying to prove

By definition of the convolution, we have that

By the hypothesis (56),

For t < T , we have

( K

where the last equality follows by the equality ‖ K ∗ n ‖ = ‖ K ‖ n , [22, Theorem 2.2(i)]. In conclusion, by monotonicity, we have that

Hence we have that

Developing the recursion,

The result is proven.

Remark C.4. If the assumption (56) holds only for ˆ T > t > T , then the statement of Lemma C.1 still holds with

We now give a non-asymptotic bound for the general convolution-type Volterra equation.

Lemma C.2 (Non-asymptotic Volterra bound) . Let K and f be non-negative functions. Suppose K is monotonically decreasing and for some ε > 0 , there exists a T ( ε ) > 0 such that

Moreover, suppose the convergence threshold condition 2(1 + ε ) ‖ K ‖ < 1 holds. Then

Proof. We consider the upper and lower bound separately.

Lower bound: Since K and f is are non-negative, then ∑ ∞ j =1 ( K ∗ j ∗ f )( t ) ≥ ( K ∗ 1 ∗ f )( t ) ≥ ( K ∗ f )( t ) . Recall the solution to the convolution-type Volterra equation takes the form,

It immediately follows from ∑ ∞ j =1 ( K ∗ j ∗ f )( t ) ≥ ( K ∗ f )( t ) the lower bound.

Upper bound: The solution to a Volterra equation (in L 1 ) is

By Lemma C.1 and the hypothesis, there exists a T > 0 and ε > 0 such that

and (2 ‖ K ‖ (1 + ε )) j -1 < 1 . Hence, we have that

The result is shown.

C.3.2 Proof of Theorem 2.1

We are now ready to show one of the main tools used to analyze the loss function, Theorem 2.1. The result relies on approximations for the kernel and forcing functions found in Section F and Section G. We restate the theorem statement to remind the reader of the result.

Theorem C.1 (Approximation solution for P ) . Suppose γ and B is at most half the convergence threshold and α > 1 4 . There exists an M > 0 large and a constant C = C ( α, β, M ) , independent of d , so that for all admissible v and d , for all M < γBr ,

The convolution further simplifies. For any glyph[epsilon1] > 0 , there exists an M > 0 and a constant C = C ( α, β, M ) independent of d so that for all M < γBr ,

Proof of Theorem C.1 / Theorem 2.1. Note for all γBr > 1 /Md 2 α , we have that c F 0 ≤ F ( r ) , K ( r ) ≤ C F 0 ( r ) for some C, c > 0 . This is where the limiting level starts to dominate. We begin by showing (58). Fix ε > 0 . From Proposition G.2, we have that there exists an M > 0 sufficiently large so that the hypothesis for Kesten's Lemma, i.e.,

Therefore, we get (58) by Lemma C.2.

To prove (59), we begin by

where we used monotonicity of F and K .

Using Proposition H.2 and Proposition H.4, for large d 2 α /M ≥ γBr ≥ M , we have that F ( r 2 ) glyph[equivasymptotic] F ( r ) since F is power law for large r (see also Corollary F.1). The same holds for K , using Proposition H.5 and Proposition G.2, K ( r 2 ) glyph[equivasymptotic] K ( r ) for d 2 α /M ≥ γBr ≥ M for some M > 0 .

For small γBr ≤ M , we have that F ( r/ 2) ≤ C and K ( r/ 2) ≤ C for some C > 0 . Since F and K are monotonic, we can choose a constant so that F ( r/ 2) glyph[lessorsimilar] F ( r ) and K ( r/ 2) glyph[lessorsimilar] K ( r ) for γBr ≤ M .

Now using Proposition C.1 and Proposition H.6, we have that

For the lower bound, we have that

where we used monotonicity of K and F .

We note that F ( s ) glyph[equivasymptotic] C for γBs ≤ M for all M > 0 . Therefore,

On the other hand, by Proposition C.1, for any glyph[epsilon1] > 0 , there is an M so that for any γBr ≥ M ,

This proves the lower bound.

C.4 Details of risk curves for the phases

We can now put together a coherent picture of the effect of different choices of α and β and their impact on the Pareto frontier. We will have 4 distinct phases where the expected loss will exhibit a power law decay and 1 region ( α + β ≤ 0 . 5) for which the expected loss has no power law decay (see Figure 2a). We will describe each of the 4 power law phases below marked by their boundaries.

First, we recall the forcing function F and kernel function K introduced in Section 2.1.

Forcing function. For the forcing function,

The function F 0 ( r ) is the component of the forcing function corresponding to the point mass at 0 , F pp ( r ) is the component of the forcing function corresponding to the pure point part of the spectrum, and lastly, the most complicated part of the spectrum, the forcing function corresponding to the distorted features. In particular, we will show in Section F the exact definitions of F 0 , F pp , F ac and | error F | are small, and, in Section H, we derive asymptotic-like behaviors for these functions. See Table 5 for definitions and asymptotics.

Kernel function. Similarly, the kernel function K is

Note here that the kernel function has a multiplication by the eigenvalue of ˆ K and so the point mass at 0 will not contribute. In Section G, we will give an explicit definition of K pp and show error terms are small and, in Proposition H.5, we give the asymptotic-like behavior of K pp .

Now we describe in detail the different risk curves that arise for the different phases.

C.4.1 Above the high-dimensional line (Phases Ia, II, III)

This setting is commonly known as the trace class. It is characterized by four components:

learning rate γ can be picked independent of dimension;

loss curve does not self average, that is, the loss curve does not concentrate around a deterministic function;

v ≥ d , but v has no upper bound and so we can take v →∞ ;

batch, B , is constrained to be small (see Proposition C.2).

When 2 α > 1 , or the trace class phase, the loss will exhibit 3 different phases. We described these phases in detail below.

Phase Ia: (2 β < 1 , 2 α > 1) . In this phase, it notable for three characteristics:

absolutely continuous part of the forcing function does not participate;

level at which SGD saturates is affected by β ;

SGD noise does not participate.

In this case, the loss curve is just a constant multiple of gradient flow. Hence, we have that

Proposition C.3 (Phase Ia: 2 β < 1 , 2 α > 1 ) . Suppose 2 β < 1 and 2 α > 1 . Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

Proof. By Theorem C.1, we know that it suffices to look at the forcing function F and kernel function K . Moreover, in this regime, we have that γ and B are constant (see Proposition C.2).

The rest of the argument relies on the bounds found in Proposition H.2, ( F pp ) , Proposition H.4 ( F ac ), Proposition H.3, ( F 0 ) , and Proposition H.5 ( K pp ).

For the forcing function, F ac ( r ) = 0 as 2 β < 1 (Proposition H.4). Therefore the forcing function is composed of F pp ( r ) and F 0 ( r ) .

First, we have that ( γBr ) -2+1 / (2 α ) < ( γBr ) -(1+ β/α )+1 / (2 α ) as β < α in this phase. Thus, using Proposition H.2 and Proposition H.5, for γBr > M , where M is some constant, we have that 1 γB K pp ( r ) ≤ C × F pp ( r ) for some C > 0 . Hence the result is shown.

As a consequence of the argument above, we know that

Phase II: (2 β > 1 , 2 α > 1 , β < α ) For this phase, we see that

limit level is unaffected by β ;

absolutely continuous spectrum takes over for r ∈ ( Md α , d 2 α /M ) for some M ;

SGD noise does not participate.

Therefore, in this case, we have that

Proposition C.4 (Phase II: 2 β > 1 , 2 α > 1 , β < α ) . Suppose 2 β > 1 , 2 α > 1 , and β < α . Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

Proof. By Theorem C.1, we know that it suffices to look at the forcing function F and kernel function K . Moreover, in this regime, we have that γ and B are constant (see Proposition C.2).

The rest of the argument relies on the bounds found in Proposition H.2, ( F pp ) , Proposition H.4 ( F ac ), Proposition H.3, ( F 0 ) , and Proposition H.5 ( K pp ).

γBr ≤ M 0 , for some M 0 : First, we have that 1 γB K pp ( r ) ≤ C 0 × F pp ( r ) (Proposition H.5) and F ac ( r ) ≤ C 0 × F pp ( r ) (Proposition H.4) for some constant C 0 > 0 . The constant M 0 is where the asymptotic of F pp starts to apply.

M 0 ≤ γBr ≤ M 1 , for some M 0 and for all M 1 > M 0 : We see that ( γBr ) -2+1 / (2 α ) < ( γBr ) -(1+ β/α )+1 / (2 α ) as β < α in this phase. Thus, using Proposition H.2 and Proposition H.5, we have that 1 γB K pp ( r ) ≤ C 1 × F pp ( r ) for some C 1 > 0 . A quick computation shows that F ac ( r ) ≤ F pp ( r ) .

M 1 ≤ γBr ≤ M 2 d 2 α , for any M 1 and some M 2 : The M 2 is the smallest of the two endpoints for the asymptotics of F pp and F ac . As in the previous regime, we have that 1 γB K pp ( r ) glyph[lessorsimilar] F pp ( r ) . In this region, F ac ( r ) glyph[equivasymptotic] d -1 ( γBr ) -1+1 / (2 α ) and F pp ( r ) glyph[equivasymptotic] ( γBr ) -(1+ β/α )+1 / (2 α ) . We see at ( γBr ) = d 2 α that ( γBr ) -β/α ≤ ( d 2 α ) -β/α = d -2 β ≤ d -1 as 2 β > 1 . Therefore, at γBr = d 2 α , F pp ( r ) glyph[lessorsimilar] F ac ( r ) and we started, i.e., when r = M 1 with F ac ( r ) glyph[lessorsimilar] F pp ( r ) . Therefore, we must change in this regime to being F ac dominate.

As a consequence of the argument above, we know that

Phase III: SGD noise appears, (2 β > 1 , 2 α > 1 , β > α ) In this case, we see that SGD changes the dynamics over gradient flow. In particular,

limit level is unaffected by β ;

absolutely continuous forcing function takes over for iterations r ∈ ( Md,d 2 α /M ) for some M ;

SGD noise regulates convergence.

Thus, we have that

Proposition C.5 (Phase III: 2 β > 1 , 2 α > 1 , β > α ) . Suppose 2 β > 1 , 2 α > 1 , and β > α . Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

Proof. By Theorem C.1, we know that it suffices to look at the forcing function F and kernel function K . Moreover, in this regime, we have that γ and B are constant (see Proposition C.2).

The rest of the argument relies on the bounds found in Proposition H.2, ( F pp ) , Proposition H.4 ( F ac ), Proposition H.3, ( F 0 ) , and Proposition H.5 ( K pp ).

γBr ≤ M 0 , for some M 0 : First, we have that F pp ( r ) ≤ C 0 × 1 γB K pp ( r ) (Proposition H.5) and F ac ( r ) ≤ C 0 × 1 γB K pp ( r ) (Proposition H.4) for some constant C 0 > 0 . The constant M 0 is where the asymptotic of K pp starts to apply.

M 0 ≤ γBr ≤ M 1 , for some M 0 and for all M 1 > M 0 : We see that ( γBr ) -2+1 / (2 α ) > ( γBr ) -(1+ β/α )+1 / (2 α ) as β > α in this phase. Thus, using Proposition H.2 and Proposition H.5, we have that F pp ≤ C 1 × 1 γB K pp ( r ) for some C 1 > 0 . A quick computation shows that F ac ( r ) ≤ K pp ( r ) .

M 1 ≤ γBr ≤ M 2 d 2 α , for any M 1 and some M 2 : The M 2 is the smallest of the two endpoints for the asymptotics of K pp and F ac . As in the previous regime, we have that 1 γB K pp ( r ) glyph[lessorsimilar] F pp ( r ) . In this region, F ac ( r ) glyph[equivasymptotic] d -1 r -1+1 / (2 α ) and K pp ( r ) glyph[equivasymptotic] r -2+1 / (2 α ) . We see at γBr = d 2 α that ( γBr ) -1 ≤ ( d 2 α ) -1 = d -2 α ≤ d -1 as 2 α > 1 . Therefore, at ( γBr ) glyph[equivasymptotic] d 2 α , K pp ( r ) glyph[lessorsimilar] F ac ( r ) and we started, i.e., when r = M 1 with F ac ( r ) glyph[lessorsimilar] K pp ( r ) . Therefore, we must change in this regime to

being F ac dominate.

As a consequence of the argument above, we know that

C.4.2 Below the high-dimensional line (Phases IVa, IVb, Ib, Ic)

One of the main differences between the previous regime and this regime is that V can not be taken to ∞ independent of d . As a result, we call this below the high-dimensional line and it is precisely bounded by whether 2 α is summable or not.

The four main characteristics of this regime are:

learning rate γ scales like v -1+2 α ;

SGD loss, i.e., E [ P ( θ r )] self-concentrates;

v can not be too large, i.e., d and v are proportional;

batch can be large (i.e., γB ≤ 1) since the learning rate is small ( γ ∼ v -1+2 α ).

In Phases IV, Ib, and Ic, because j -2 α is not summable, the summation of the j depends on the dimension v . Thus, the kernel norm is

where the learning rate γ is chosen so that ‖ K ‖ is constant, i.e., γ = ˜ γ ‖ K ‖ where ˜ γ > 0 is a constant.

Phase IV, (2 β > 1 , 1 4 < α < 1 2 ) . In this phase, we have the following

limiting value of the loss that SGD converges to is unaffected by β ;

pure point forcing function plays a role;

absolutely continuous part of the spectrum does not contribute to the forcing function;

SGD noise affect the loss curves.

In this phase, the loss curve is

The following gives the precise statement.

Proposition C.6 (Phase IV: 2 β > 1 , 1 4 < α < 1 2 ) . Suppose 2 β > 1 and 1 4 < α < 1 2 . Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

Proof. By Theorem C.1, we know that it suffices to look at the forcing function F and kernel function K . Moreover, in this regime, we have that γ decreases like d 2 α -1 (see Proposition C.2).

The rest of the argument relies on the bounds found in Proposition H.2, ( F pp ) , Proposition H.4 ( F ac ), Proposition H.3, ( F 0 ) , and Proposition H.5 ( K pp ).

We first note there is no F ac ( r ) glyph[lessorsimilar] F 0 and therefore it is too small to contribute.

γBr ≤ M 0 , for some M 0 : First, we have that 1 γB K pp ( r ) ≤ C 0 × F pp ( r ) for some constant C 0 > 0 . The constant M 0 is where the asymptotic of F pp starts to apply.

M 0 ≤ γBr ≤ M 1 , for some M 0 and for all M 1 > M 0 : We see that γ ( γBr ) -2+1 / (2 α ) < ( γBr ) -(1+ β/α )+1 / (2 α ) since γ glyph[equivasymptotic] d 2 α -1 and 2 α < 1 in this phase. Thus, using Proposition H.2 and Proposition H.5, we have that 1 γB K pp ( r ) ≤ C 1 × F pp for some C 1 > 0 .

M 1 ≤ γBr ≤ M 2 d 2 α , for any M 1 and some M 2 : The M 2 is the smallest of the two endpoints for the asymptotics of K pp and F pp . In this region, F pp ( r ) glyph[equivasymptotic] ( γBr ) -(1+ β/α )+1 / (2 α ) and γ × 1 γB K pp ( r ) glyph[equivasymptotic] γ × ( γBr ) -2+1 / (2 α ) glyph[equivasymptotic] d 2 α -1 × ( γBr ) -2+1 / (2 α ) . We see at r = d 2 α that d 2 α -1 ( γBr ) -1 = d -1 ≥ d -2 β = ( d 2 α ) -β/α . Thus F pp ( r ) glyph[lessorsimilar] 1 γB K pp ( r ) and we started, i.e., when r = M 1 with K pp ( r ) glyph[lessorsimilar] F pp ( r ) . Therefore, we must change in this regime to being K pp dominate.

M 2 d 2 α ≤ γBr for all M 2 : In this case, all terms are bounded above by F 0 ( r ) .

As a consequence of the argument above, we know that

Phase Ib, (2 β < 1 , 0 . 25 < α < 0 . 5 , 2( α + β ) > 1) . Phase Ia, Ib, and Ic are quite similar as the dynamics of SGD only depend on the forcing function pure point and limiting value. In this phase, the learning rate γ is dimension dependent, unlike Phase Ia, and the following hold

limiting value of the loss that SGD converges to is d -2 α +1 -2 β ;

absolutely continuous part of the spectrum does not contribute to the forcing function;

SGD noise not does affect the loss curves.

In this phase, the loss curve is

Although we did not prove the statement for α < 0 . 25 as we do not have estimates for the kernel function, we believe that statement still holds. We believe that the kernel function stops becoming power law when α < 0 . 25 , but the forcing function is still power law.

The following gives the precise statement.

Proposition C.7 (Phase Ib: 2 β < 1 , 1 4 < α < 1 2 , 2( α + β ) > 1 ) . Suppose 2 β < 1 , 2( α + β ) > 1 , and 1 4 < α < 1 2 . Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

Proof. By Theorem C.1, we know that it suffices to look at the forcing function F and kernel function K . Moreover, in this regime, we have that γ decreases like d 2 α -1 (see Proposition C.2).

The rest of the argument relies on the bounds found in Proposition H.2, ( F pp ) , Proposition H.4 ( F ac ), Proposition H.3, ( F 0 ) , and Proposition H.5 ( K pp ).

We first note there is no F ac ( r ) .

γBr ≤ M 0 , for some M 0 : First, we have that 1 γB K pp ( r ) ≤ C 0 × F pp ( r ) for some constant C 0 > 0 . The constant M 0 is where the asymptotic of F pp starts to apply.

M 0 ≤ γBr ≤ M 1 d 2 α , for any M 0 and some M 1 : The M 1 is the smallest of the two endpoints for the asymptotics of K pp and F pp . In this region, F pp ( r ) glyph[equivasymptotic] ( γBr ) -(1+ β/α )+1 / (2 α ) and γ × 1 γB K pp ( r ) glyph[equivasymptotic] γ × ( γBr ) -2+1 / (2 α ) glyph[equivasymptotic] d 2 α -1 × ( γBr ) -2+1 / (2 α ) . We see at r = d 2 α that d 2 α -1 ( γBr ) -1 = d -1 ≤ d -2 β = ( d 2 α ) -β/α . Thus 1 γB K pp ( r ) glyph[lessorsimilar] F pp ( r ) and we started, i.e., when

r = M 1 with K pp ( r ) glyph[lessorsimilar] F pp ( r ) . Therefore, F pp must dominate.

We expect Prop. C.7 to hold with the same conclusions for 2 β < 1 , 2 α < 1 , and 2( α + β ) > 1 . As a consequence of the argument above, we know that

Phase Ic, (2 β > 1 , 0 < α < 1 4 . Lastly, we consider Phase Ic, which is similar to Phases Ia and Ib. The following holds in this phase.

limiting value of the loss that SGD converges to is d -2 α +1 -2 β ;

absolutely continuous part of the spectrum does not contribute to the forcing function;

SGD noise not does affect the loss curves.

In this phase, the loss curve is

Under the assumption that Theorem C.1 holds for α > 1 / 4 , we get the following.

Proposition C.8 (Phase Ic: 2 β > 1 , 0 < α < 1 4 ) . Suppose 2 β > 1 and 0 < α < 1 4 and Theorem C.1 holds. Suppose the learning rate γ and batch B > 0 satisfy at most half the convergence threshold in Proposition C.2. Then there exists an M > 0 large and constants C = C ( α, β, M ) and c = c ( α, β, M ) , independent of d , so that for all admissible v and d , for all γBr > M

We can not prove this statement as we do not have sharp bounds on the kernel function in this region. We believe that the kernel function stops becoming power law, but the forcing function is still power law. Thus, it should become even more forcing function dominate.

We believe the loss curve follows similar behavior to Phase Ia and Phase Ib, that is,

D Compute-optimal curves

Throughout this section, consider the deterministic equivalent loss function P ( r ) = P ( r, d ) . Moreover as batch size B is order 1, it only effects the compute-optimal curves by a constant. Therefore, we can set B = 1 . For each iteration r , the SGD costs d flops, or equivalently r/d = flops, f . The goal is to find the optimal compute line as a function of the number of flops f :

If d glyph[star] ( f ) def = arg min d P ( f d , d ) , the optimal compute line is precisely P ( f d glyph[star] ( f ) , d glyph[star] ( f ) ) .

To do this, we simplify the loss curve P ( f d , d ) . While it is possible to minimize this as a function of d , an alternative function considered is the following

which achieves the right power law behavior as the true compute-optimal curve and deviates from this true curve by an absolutely constant (independent of d, f ) (see Theorem C.1). Note here some of the terms should be taken as 0 when not defined for the different phases.

Using this alternative loss function, ˜ P ( r, d ) , the compute-optimal line must occur at one of the corner points, i.e., where any pair of functions equal each other. The following lemma gives a useful characterization of these points.

LemmaD.1. Suppose C 0 , C 1 > 0 are constants and γ 0 , γ 1 , p 0 , p 1 > 0 exponents such that a function ˆ P ( r, d ) equals

Then replacing r ↦→ f d the minimizer in d satisfies

and the optimal value is

Proof. The proof is a straightforward computation. The minimizer of ˆ P ( f , d ) in d must occur where the two terms in the maximum are equal, i.e.,

Solving for this d gives d glyph[star] . Plugging in the value of d glyph[star] into ˆ P ( f , d ) gives the optimal value.

Remark D.1. The possible minimal values of (73) , i.e., where pairs of functions in the max are equal, can be reduced further. For instance, if F ac ( r, d ) exist for the phase, then for some 0 < r 0 < r 1 < r 2

Thus, there are only a maximum of three points to check in order to find the optimal compute curve.

Remark D.2. In view of Lemma D.1, to find the optimal compute curves, we first find the potential curves (i.e., all the possible combinations of two functions in the loss curve are equal while still lying on the loss curve). Then the curve which has the smallest exponent on the flops, f , is the optimal compute curve.

D.1 Compute-optimal curves: Above the high-dimensional line (Phases Ia, II, III).

To ease notation, we introduce several constants that will be used only in this Section D.1:

where the asymptotics only hold in specific regions of the space of γBr . For additional details on the derivation of these asymptotics and the constraints on γBr where asymptotics hold, see Section H.

Remark D.3. The constants in the asymptotics are dimension independent and only depend on α, β .

The compute-optimal curves are summarized in Table 6.

D.1.1 Phase Ia

In this case, the approximate loss curve is given by

With this, we give a description of the optimal compute curve.

Proposition D.1 (Phase Ia: Compute-optimal Curve) . Suppose we are in Phase Ia, that is, 2 β < 1 and 2 α > 1 . The compute-optimal curve using ˜ P ( f d , d ) in (74) occurs when F pp ( f d , d ) = F 0 ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

and the compute-optimal curve is

Proof. We apply Lemma D.1 with

D.1.2 Phase II

In this case, the approximate loss curve has three terms (Proposition C.4 with (63))

Proposition D.2 (Phase II: Compute-optimal Curve) . Suppose we are in Phase II, that is, 2 β > 1 , 2 α > 1 , and β < α . The compute-optimal curve using ˜ P ( f d , d ) in (75) occurs when F pp ( f d , d ) = F ac ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

Proof. Using the Remark D.2 after Lemma D.1 and Proposition C.4 with (63), we only need to check two intersections: F pp = F ac and F ac = F 0 . The curve which has the smallest (i.e., largest negative) exponent (i.e, steepest curve on a log-log plot) is the compute-optimal curve.

Case 1: Consider F pp ( f d , d ) = F ac ( f d , d ) . We apply Lemma D.1 with

Case 2: Consider F ac ( f d , d ) = F 0 ( f d , d ) . As before, we apply Lemma D.1 with

and the compute-optimal curve is to get that the minimum is

and the optimal value is

to get that the minimum is and the optimal value is

One can check that

Therefore, Case 1 is the optimal overall.

D.1.3 Phase III

In this case, the approximate loss curve has three terms (Proposition C.5 with (66))

Proposition D.3 (Phase III: Compute-optimal Curve) . Suppose we are in Phase III, that is, 2 β > 1 , 2 α > 1 , and β > α . The compute-optimal curve using ˜ P ( f d , d ) in (76) occurs when 1 γB K pp ( f d , d ) = F ac ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

Proof. Using the Remark D.2 after Lemma D.1 and Proposition C.5 with (66), we only need to check two curves: 1 γB K pp = F ac and F ac = F 0 . The curve which has the smallest (i.e., largest negative) exponent (i.e, steepest curve on a log-log plot) is the compute-optimal curve.

Case 1: Consider F ac ( f d , d ) = F 0 ( f d , d ) . We did this for Phase II in the proof of Proposition D.2. Thus, we have

Case 2: Consider 1 γB K pp ( f d , d ) = F ac ( f d , d ) . We apply Lemma D.1 with

and the compute-optimal curve is to get that the minimum is

and the optimal value is to get that the minimum is

and the optimal value is

One can check that

Therefore, Case 2 is the optimal overall.

D.2 Compute-optimal curves: Below the high-dimensional line (Phase IV, Ib, Ic), 2 α < 1

This section main distinction with above the high-dimensional line section is the dependency of the learning rate on v . In deed, we have that v/d → c ∈ (0 , ∞ ) and the learning rate is chosen so that the kernel norm is constant, i.e.,

where ˜ γ is the positive constant so that γ = ˜ γ × v 2 α -1 . Consequently, we also need to keep track of the learning rate in the various terms.

In this section, we assume that v/d → r ∈ (1 , ∞ ) as v, d →∞ .

We state for completeness the d and r dependency on the forcing and kernel function, including the learning rate γ . We note that these asymptotics only hold for a set of γBr values which depend on the spectral properties of K (see the propositions listed next to the terms for details).

D.2.1 Phase IV (a) and (b)

In these cases, the approximation loss curve is given by (Proposition C.6 with (68))

As one crosses the 2 α = 1 , line the F ac disappears and F pp emerges. Consequently, there leaves two possible corners where the compute-optimal value could occur at. When α goes below α = 1 / 4 , the K pp decreases.

The difference between IVa and IVb is simply where the compute-optimal occurs. In IVa, the tradeoff occurs between K pp and F 0 , whereas in IVb, the tradeoff occurs at F pp and K pp .

We give the compute-optimal curve for Phase IVa.

Proposition D.4 (Phase IVa: Compute-optimal Curve) . Suppose we are in Phase IVa, that is, 2 β > 1 and √ 2 -1 √ 2 < α < 1 2 . The compute-optimal curve using ˜ P ( f d , d ) in (77) occurs when d 1 -2 α K pp ( f d , d ) = F 0 ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

Proof. Using the Remark D.2 after Lemma D.1 and Proposition C.6 with (68), we only need to check two curves: F pp = d 1 -2 α × K pp and d 1 -2 α × K pp = F 0 . The curve which has the smallest (i.e., largest negative) exponent (i.e, steepest curve on a log-log plot) is the compute-optimal curve.

Case 1: Consider F pp ( f d , d ) = d 1 -2 α K pp ( f d , d ) . We apply Lemma D.1 with

to get that the minimum is and the optimal value is

Case 2: Consider d 1 -2 α K pp ( f d , d ) = F 0 ( f d , d ) . We apply Lemma D.1 with

to get that the minimum is and the optimal value is

and the compute-optimal curve is and the compute-optimal curve is

Proof. The computations are exactly the same as in Proposition D.4. For the α 's and β 's in this region, we see that Case 1 has the smaller exponent on f than Case 2.

In this region of α 's and β 's, Case 2 has a smaller exponent on f than in Case 1.

As for Phase IVb, we have the following.

Proposition D.5 (Phase IVb: Compute-optimal Curve) . Suppose we are in Phase IVb, that is, 2 β > 1 and 1 4 < α < √ 2 -1 √ 2 . The compute-optimal curve using ˜ P ( f d , d ) in (77) occurs when c k d 1 -2 α K pp ( f d , d ) = c f F pp ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

D.2.2 Phase Ib

In this case, the approximate loss curve is given by (Proposition C.7 with (70))

Note this is true for α > 1 / 4 , but we expect this to hold without this extra assumption.

With this, we give a description of the optimal compute curve.

Proposition D.6 (Phase Ib: Compute-optimal Curve) . Suppose we are in Phase Ib, that is, 2 β < 1 , 1 4 < α < 1 2 , and α + β > 1 2 . The compute-optimal curve using ˜ P ( f d , d ) in (78) occurs when F pp ( f d , d ) = F 0 ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

and the compute-optimal curve is

Proof. We apply Lemma D.1 with

We expect the conclusions of Prop. D.6 to hold for the ( α, β ) pairs where 2 β < 1 , 2 α < 1 , and 2( α + β ) > 1 .

D.2.3 Phase Ic

In this case, the approximate loss curve is given by (Proposition C.8 with (72))

Note again that this is speculative as we do not have the bounds for the kernel. However we do believe that this is correct. With this, we give a description of the compute-optimal curve.

Proposition D.7 (Phase Ic: Compute-optimal Curve) . Suppose we are in Phase Ic, that is, 2 β > 1 and 0 < α < 1 4 and suppose (79) is true. The compute-optimal curve using ˜ P ( f d , d ) in (77) occurs when F pp ( f d , d ) = F 0 ( f d , d ) . Precisely, the optimal d glyph[star] which minimizes ˜ P ( f d , d ) is

and the compute-optimal curve is

Proof. We apply Lemma D.1 with

∐⌈√[∐(//(̂˜√∐(//(〈((//(∧((/

√√˜̂√√∐(glyph[arrowvertexdbl]˜]˜[√˜̂(]{(〈∧{∖}([∐√{̂˜√∐}

 

 

 

 

 

 

/

/

/

/

/

/

/

̂˜{√]√glyph[arrowbt]

˜⌉√]√]̂∐⌈(̂˜{√]√glyph[arrowbt]

˜]glyph[arrowtp]˜̂(√}]{√

E Spectrum of ˆ K : random matrix theory

In this section, we analyze the spectrum of ˆ K = D 1 / 2 WW T D 1 / 2 . For this, we use standard tools from random matrix theory to derive a fixed point equation for the Stieljes transform of ˆ K . Indeed, by knowing the Stieljes transform of ˆ K , one can recover the spectral properties.

In particular, we will need the spectra of ˆ K decomposes into 3 parts:

Point mass at z = 0 : There will be a point mass at z = 0 of mass v -d for trivial reasons since v glyph[greatermuch] d .

Pure point outliers: There will be a set of outliers, the pure point spectra, which are at constant order and nearly equal to j -2 α for j = 1 , 2 , . . . ,

Absolutely continuous part: The spectral bulk, the absolutely continuous part, which form a density on a shrinking window.

In fact, we will not need to give a complete picture about all the spectra.

In this section, we state the deterministic equivalent for the random matrix ( ˆ K -z ) -1 and give some properties of its 'self-consistent spectra.' The starting point for this is the self-consistent equation

The identification ←→ can be made rigorous by showing that

for deterministic sequences of test matices { A } with bounded nuclear norm and generally with very high probability in d . We note that we would need a more precise quantification of errors to be useful for establishing the scaling law for the actual random matrices. In Figure 6, we solve the theoretical spectrum by solving the fixed point equation for m ( z ) using a Newton method on a grid of complex z .

The function m can also be related to the trace of ( ̂ K -z ) -1 . From the definition of m , we can derive the explicit representation theorem.

Lemma E.1. For any v, d and z ∈ C with Im( m ( z )) > 0

Proof. Multiplying through by z we should evaluate

Adding and subtracting j -2 α m ( z )

Using the definition of m ,

Then we have

For each fixed z ∈ H , this is a strict self map into the lower half-plane of m . Self-maps are strict contractions in the hyperbolic metric (from the Schwarz-Pick lemma), and thus there is a unique solution of m ( z ) = G ( m ( z ); z ) .

We now introduce F according to the formula

so that F ( m ; z ) = m/G ( m ; z ) . Introduce the stability operator

By the Schwarz-Pick lemma (in the half-plane version), we have

and hence in some sufficiently small neighborhood of G ( m ( z )) = m ( z ) , we therefore have

Hence also, in a sufficiently small neighborhood of m ( z )

glyph[negationslash]

Substituting this in completes the proof.

While an explicit solution of m is unavailable, we can derive many properties of m , starting with:

Proposition E.1. Suppose X is a real random variable compactly supported in [0 , ∞ ) . For every z ∈ H = { z ∈ C : Im z > 0 } , there is a unique solution of (80) satisfying Im m ( z ) < 0 . Moreover, this solution is an analytic function, and it can be solved by iterating the fixed point map

initialized with m ≡ 1 . Furthermore, if we consider the equation for z ∈ H

then this is solved uniquely by m ( z ) and moreover it is stable in that ∂ m F = 0 in a neighborhood of the solution.

Proof. Let G be the mapping glyph[negationslash]

While this proposition does not state what happens on the real line, in regions of the line where m has a finite, real-valued limit, it agrees with its reflection in the lower half-plane. Hence in any open subset of R where lim η → 0 m ( x + iη ) exists and is real, m will be analytic.

From Proposition E.1, we can derive some explicit estimates on m , which will be sufficient for deriving the estimates on the forcing and kernel functions. We summarize these properties in the following:

Proposition E.2. Let X be any random variable with support in (0 , 1] . Then the following hold:

Near 0 : Suppose that a < 0 is a real-valued solution of

Then m is analytic in a neighborhood of z = 0 and m ( z ) = az + O ( z 2 ) . If furthermore if for some interval [0 , z 0 ] the equation

is solvable for a < 0 , then m is analytic in a complex neighborhood of the whole interval [0 , z 0 ] .

Far away : Let glyph[rho1] 0 ( z ) be the distance of z to [0 , 1] . Suppose that z is such that 16 v d E X ≤ glyph[rho1] 0 ( z ) 2 . Then for some absolute constant C > 0 ,

Moreover suppose that ∣ ∣ ∣ v d E ( X X -z ) ∣ ∣ ∣ ≤ glyph[epsilon1] < 1 4 , and suppose that

Then

We need a lemma on stability of solutions.

Lemma E.2. Suppose F is an analytic map of -H → C , Suppose there is a c, δ > 0 and m 0 ∈ -H so that for all m ∈ -H with | m -m 0 | ≤ δ and δ < | Im m 0 | ,

If m 0 is an approximate solution of F ( m ) = 1 in that it satisfies

then there is an solution m ∈ -H with F ( m ) = 1 so | m -m 0 | ≤ δ .

Proof. We introduce an ODE (which is the continuous limit of the damped Newton's method)

Then we note that along this ODE

Hence we have (1 -F ( m t )) = (1 -F ( m 0 )) e -t for however long the ODE exists. In particular, if we have that on some open set U of admissible m containing m 0 that

then provided the ODE does not exit U ,

Hence as there is a neighborhood of m 0 of size δ on which ∂ m F ( m ) | ≥ c then as | (1 -F ( m 0 , z )) | < cδ we have

Proof of Proposition. Part 1, Near 0 : For the component near 0 , we consider a change of variables and look at q ( z ) = m ( z ) /z which is therefore the unique solution of the fixed point equation:

Then by hypothesis, we have a solution at z = 0 given by F ( a, 0) = 1 . We wish to continue this solution to a neighborhood of ( a, 0) , and so it would suffice to know that the differential equation

has a solution in a neighborhood of the point. Solving for the derivative of q ,

We note that if we solve the equation along the real line, then q stays real-valued. Furthermore, for all q with q < 0 we have

By analyticity, we can extend this solution into a neighborhood in the upper half plane and on an interval of the real line where this solvable. Hence m is analytic in this neighborhood and has boundary values given by m ( z ) /z → a .

Part 2, Far away : For the other parts, we use that m is the solution of

Hence we have

Define a distance

Provided that | m -1 | ≤ δ and v d ( E X ) | z | ≤ glyph[rho1] ( z ) 2 / 2 (which is trivially satisfied if v d ( E X ) /glyph[rho1] ( z ) ≤ 1 2 )

For any z with glyph[rho1] ( z ) > 0 , we can find a unit speed geodesic σ ( t ) connecting z to ∞ so that glyph[rho1] ( σ ( t )) = glyph[rho1] ( z ) + t (this will just be a straight line). Then along this geodesic, v d ( E X ) | σ ( t ) | ≤ glyph[rho1] ( σ ( t )) 2 / 2 , since v d ( E X ) | σ ( t ) | ≤ v d ( E X )( | z | + t ) ≤ v d ( E X )( | z | + t | z | /glyph[rho1] ( z )) ≤ ( glyph[rho1] ( z )) 2 (1+ t/glyph[rho1] ( z )) / 2 ≤ ( glyph[rho1] ( z )+ t ) 2 / 2 .

Hence along the geodesic, and provided that | m -1 | ≤ δ , we conclude

Integrating along this line segment from infinity, we conclude that provided the right hand side is less than δ ,

For the second conclusion, we use Lemma E.2 with this F (holding z fixed). We let m 0 = 1 -v d E ( X ( X -z ) ) , and we consider an 2 glyph[epsilon1] neighborhood of 1 , U By assumption, on U

satisfies | ∂ m F | ≥ 3 4 . We also have that

Hence applying Cauchy-Schwarz

The conclusion now follows from Lemma E.2.

E.2 The region near 0 and the spectral bulk

We now bound the contribution of the region near 0 .

Proposition E.3. The function m ( z ) is analytic in a neighborhood of z = 0 of radius c ( α ) d -2 α for some c ( α ) > 0 . Furthermore, m is negative on (0 , cd -2 α ) , vanishes at 0 , and has | m ′ (0) + κ ( v/d ) d 2 α | ≤ Cd 2 α -1 for all d sufficiently large where

Moreover, we introduce f ( z ; V/d ) where f : H →-H which solves

This extends analytically to the interval [0 , c ) . Then f and m are close in that for any compact subset K ⊂ ( C \ ([ c, ∞ ] ∪ [ -∞ , 0]) ) , we have

We furthermore have that in the case 2 β < 1

and in the case 2 β > 1

Proof. For the first part, we look to apply Proposition E.2 part 1. The equation we need to solve is

for a < 0 . We change variables by setting a = -d 2 α κ and z = d -2 α z , in terms of which

By monotonicity, for κ positive,

and moreover the lower bound is only less than the upper bound by at most 1 d uniformly in κ > 0 . In the case that 2 α < 1 , we can bound for κ ∈ [0 , 1]

Hence there there is an interval [0 , c 0 ] (bounded solely in terms of ( v/d ) ) on which (81) is solvable and is uniformly bounded away from 0 over all d . Hence, the solution of κ of ∫ v/d 0 κ d x κ + x 2 α = 1 satisfies

Following the same bounds on ∫ V/d 0 d x κ + x 2 α shown above, we conclude that the true solution κ of (81) with z = 0 satisfies | κ -κ | = O (1 /d ) . This concludes the proof when 2 α < 1 . In the case that 2 α > 1 ,

On the other hand for κ ∈ [0 , 1]

and hence once more there is an interval [0 , c 0 ] independent of V/d on which this is solvable and moreover the conclusions now follow in the same way as in the case that 2 α < 1 .

Convergence to f. The existence and uniqueness of f follows from Proposition E.1, where we define

(making appropriate choices of v/d and X ).

We further have, from the previous part, that f takes negative values on an the interval (0 , c ) . In what follows we fix a compact set K ⊂ ( C \ ([ c, ∞ ] ∪ [ -∞ , 0]) ) . Further, we claim the stability operator

is nonvanishing on K in a neighborhood of f . Off of the real line, this follows from Proposition E.1. On the real line, it follows from monotonicity of F for f < 0 .

Hence it follows that on K , there is a constant C ( K ) and a δ 0 > 0 so that if z ∈ K and m satisfies

then

Define

Then with β = 0

We will define a = v/d , and we will define ∆ as the separation

Then by bounding the errors in a trapezoid rule approximation

where we have used a bound on the x derivative of the integrand

(which relies on z being bounded away from 0 and on z being bounded in modulus). Hence if m ( zd -2 α ) is the solution of then we have

provided m is bounded.

To see that m remains bounded, we let ∂ m F be the stability operator of the equation

Then once more

Approximating the sum for ∂ m S

Differentiating the fixed point equation, we have the differential equation for m

As the same equation holds for f and is non-degenerate in a neighborhood of f (as the stability operator does not vanish in a neighborhood of the solution), we conclude that

uniformly on compact sets for all d sufficiently large

Sum formula. Hence having approximated f , we can turn to estimating S ( m ; z, β ) . When 2 β < 1 , we may repeat the Riemann sum approximation argument. Specifically, we have

where to bound the errors, we now must estimate

Setting x = wd , we arrive at

We may subsequently replace in this expression m by f .

In the case that 2 β > 1 , we subtract from S the divergence c β /m and then express

Bounding the difference leads to

Remark E.1. There is an exactly solvable case where even more can be said. Note that when 2 α > 1 and v/d = ∞ , the equation for f becomes

Changing variables (which requires a contour deformation which restricts the branches considered) by letting x 2 α z = -fy 2 α , so that x = ( -f/z ) 1 / (2 α ) y . Then

Hence with c α = ∫ ∞ 0 d x 1+ x 2 α , we have that f is the solution of

If for example α = 1 , then with g = ( -f ) 1 / 2 we have g satisfies the quadratic equation

with ± chosen so that Im g ≥ 0 and Re g > 0 . We note that c 1 = π 2 and conclude that

with the branch chosen to ensure Im f < 0 when Im z > 0 .

E.3 The mesoscopic region

We will need the following technical estimate on sums over lattice points.

Lemma E.3. Suppose that z and w are complex numbers and -z/w glyph[negationslash]∈ Z

Moreover, if we suppose | Im( z/w ) | ≥ | Re( z/w ) | then there is an aboslute constant C > 0 so that for any N ∈ N

Proof. Note that we may remove a factor 1 w from all statements and instead look at the case (with y = -z/w )

or solving

Then by a residue computation (applied to the function π cot( πz ) 1 z -y )

where we have used that cot is odd.

Now by pairing terms, we have

Making a common fraction, we have

Now Re( z/w ) 2 < 0 , and hence the claim follows.

Proposition E.4. Let α, β ≥ 0 with neither equal to 1 2 . We further assume 2 α + β = 1 2 . For u, η, a, b ≥ 0 consider with m = a -ib

for real A,B . We suppose that the a, b, u, η and glyph[epsilon1] satisfy

Then there is an glyph[epsilon1] 0 > 0 , c > 0 , and C > 0 so that if glyph[epsilon1] ∈ (0 , glyph[epsilon1] 0 ) such that

where c β = ∑ ∞ j =1 j -2 β (if β > 1 2 ) or c β = 0 otherwise and where o α + β is the indicator function of α + β < 1 2 . Furthermore, let A = A ( u + iη ) be the same sum with m → 1 . Then and moreover

Proof. We look to estimate the expression

glyph[negationslash]

on the regime considered, where m = a -ib . The dominant contribution of the sum will either come from j -2 α a ≈ u or possibly, when 2 β > 1 , from small j . So the analysis will be done by separately considering windows around the transition window j -2 α a ≈ u , and another analysis for large/small j . We use the notation for I ⊂ { 1 , 2 , . . . , v } that A I and B I are the restrictions of this sum to the range of j ∈ I.

The transition window. We begin by setting j 0 to be the integer which minimizes | j -2 α 0 a -u | . We can estimate this difference, noting that

We can estimate j -2 α by Taylor approximation, giving

Now we divide j according to whether

or if not, for a large M = M ( glyph[epsilon1] ) glyph[equivasymptotic] 1 /glyph[epsilon1] 2 . Let I the largest possible symmetric interval of j around j 0 that satisfies the above display.

On this interval, we would like to justify that the Taylor approximation holds. For this, we shall require that √ M ( η + j -2 α 0 b ) j 2 α 0 ≤ glyph[epsilon1] . Note under this condition

Thus the largest difference of | j -j 0 | on I is bounded above, up to constants, by √ M ( η + ub ) u -1 -1 / (2 α ) . Hence the Taylor approximation is justified in that on I

with the implied constants bounded in terms of | 1 -a | and c . It follows that for terms outside of I , we have ( j -2 α 0 a -j -2 α a ) 2 > c ′ M ( η + j -2 α 0 b ) 2 for some absolute c ′ (provided c ( α ) was picked sufficiently small).

The contribution of I terms now follows the same path as was done in the first case:

The error terms ξ 1 are bounded by

We then do a second replacement, freezing the j -2 β in the numerator, and so we need to estimate

which we do simply by

Thus with M glyph[equivasymptotic] 1 /glyph[epsilon1] 2 and using the second assumption of the lemma, we get

where we have expressed

The sum we can now evaluate using Lemma E.3. Note this makes z nearly -i ( b + η/u ) and w nearly -u 1 / (2 α ) (2 α ) , and hence z/w is almost purely imaginary. Thus the error estimate in the Lemma applies and we have (using | ( η + bu ) u -1 -1 / (2 α ) | glyph[greaterorsimilar] log(1 /glyph[epsilon1] ) and η < glyph[epsilon1]u )

The small j regime, imaginary part. Recall the terms of small j , which is to say those with j less than those in I , are denoted S. For these terms, we have j -2 α a -u ≥ c √ M ( η + ub ) . For the real and imaginary parts of the sum we have

We shall focus on the imaginary part first. We introduce an approximation for this sum, coming from approximating the denominator by j -4 α a 2 . Thus we introduce

Let c β be as in the statement of the Proposition. Then

We turn to estimating the difference of B S -iB ′ S . Using that ( j -2 α a ) 2 -( -u + j -2 α a ) 2 ≤ 2 u ( j -2 α a ) , we can estimate

To estimate these differences, we break these sums into scales. We let S k to be those j for which

Then we can estimate the number of terms in each of these k by

For small k , i.e. those for which ( η + ub )2 k ≤ u , we can estimate | S k | ≤ C ( α )( η + ub )2 k u -1 -1 / (2 α ) . Call the small k terms S ′ and the remainder S ′′ . Then for larger S ′′ terms,

For the difference of the imaginary parts on small k , we may bound j -2 β as a multiple of j -2 β 0 and so we arrive at

Then for the difference of the imaginary parts on large k

This we further estimate

In the event that the exponents are non-negative, which can only occur when 2 β > 1 , we may lose a factor which is boundable by the largest k term (which is constant order) or by a logarithm in the case the exponent is 0 . If either exponent is negative, the expression is dominated by its smallest k term, for which ( η + ub )2 k glyph[equivasymptotic] u . In all we have

The large j regime, imaginary part. We break the sum into two parts L ′ and L ′′ , those with j < 1 . 1 j 0 and those with j ≥ 1 . 1 j 0 . For the terms in L ′ we again break into scales, much like in the small j regime. We let L k to be those j for which

Then we can estimate the number of terms in each of these k by

This sum is always dominated by the smallest k , and so we have

As for larger j , we first remove a potentially divergent term, and so define

In the case that α + β < 1 / 2 , we have that (comparing to an integral and using monotonicity)

Otherwise,

As for comparing the this divergence with the sum, we have

Then if 2 α + β > 1 / 2 , this leaves

or in the case 2 α + β < 1 / 2

The real part. For the real part, we shall prove a comparison with

which we note is a special case of A with a -ib = 1 . The arguments are now very similar in all regimes to the imaginary parts, and so we just give a summary of the arguments.

The main difference is for j ≈ j 0 . Note that using the previous bounds on the transition window, we may discard an interval of | j -j 0 | ≤ √ Mηj 1+2 α 0 from A and incur an error of only glyph[epsilon1]u β/α -1 / (2 α ) . On a larger interval, J , given by those j with

by pairing j 0 + r with j 0 -r , we can bound

Then for the imaginary part

Moreover, the difference we can bound by

For small j , where we redefine S as those j smaller than those in J , we further divide to hose j with | j -j 0 | ≤ j 0 / 2 and those S ′ which are further from j 0 .

For the large j terms, we redefine L as those j larger than those in J . Again dividing to those with | j -j 0 | ≤ j 0 / 2 and otherwise, we arrive at

This, as in the large terms for the imaginary part, leads to

Finally, we observe that A satisfies an estimate of the form

which arise from the transitionary region, the small j region and the large j region.

glyph[negationslash]

Proposition E.5. Assume α = 1 4 and α = 1 2 . With z = u + iη ( u ) , with glyph[negationslash]

there is a c > 0 and an glyph[epsilon1] 0 so that for all glyph[epsilon1] ∈ (0 , glyph[epsilon1] 0 ) there is a c glyph[epsilon1] > 0 so for all u ∈ [ d -2 α /c glyph[epsilon1] , c glyph[epsilon1] ] (with A as in Proposition E.4)

Proof. We claim that m is approximately equal to

Hence we arrive at where m is the solution of

with Im m< 0 . Hence the result boils down to checking:

in a neighborhood of m 0 , using Lemma E.2.

and secondly that

For showing that | F ( m 0 , z ) -1 | we first observe that on the contour selected, if α < 1 2 and glyph[epsilon1] 0 , c glyph[epsilon1] is chosen sufficiently small

Moreover the claimed estimates on 1 -F now follow directly from Proposition E.4.

For the stability, we have that

Taking modulus, we have

Now we break the estimation of the sum into regions of j , as in the proof of Proposition E.4. We let j 0 be the integer which minimizes ( j -2 α 0 a -u ) 2 . We define S, L, I, J to be the sets where

and the rest in J , we let X A be the restriction of the sum X to the set of indices A . For the terms in S ,

with the final sum holding for all δ > 0 sufficiently small. For the terms in L ,

where o α is the indicator of 2 α < 1 . For the terms in I we have

where we have used that the number of terms in this regions is on order of j 2 α +1 0 ( ub + η ) . Now taking η a sufficiently large multiple of uβ , we conclude that the terms in X I ≤ 1 8 . For the terms in J

here the range of r is such that at its smallest value j -2 α -1 0 ( r ) glyph[equivasymptotic] ( ub + η ) , and so we arrive at

Hence picking δ sufficiently small that X S , X L are both less than 1 8 , and subsequently increasing the lower bound on η/ ( ub ) sufficiently far, we conclude that all four components can be made less than 1 8 and hence that

E.4 The large z region

Proposition E.6. For any compact set U ⊂ C of distance at least δ > 0 from [0 , 1] and any α = 1 / 2 there is a C ( α ) such that

glyph[negationslash]

and such that

Furthermore, on the same set

Proof. We apply Proposition E.2 part 2. We have

and the result follows directly from Proposition E.2. We turn to evaluating the sum

Then taking the partial derivative in m ,

which is uniformly bounded on U and on the set m so | m -1 | < δ/ 2 . It follows that on U

For the second part, we start by observing that we can estimate

We further have

Hence, combining all these errors we conclude the claim.

F Approximation of the forcing function

We now apply the technical estimate to find good approximations for the function F . Recall

We decompose the forcing function into a sum of three functions

which will be introduced in the course of the approximation.

2

glyph[epsilon1]u

u

1

-

1

/

(2

α

)

d

·

1

log(1

/glyph[epsilon1]

)

u

1+1

/

(2

α

)

d

·

1

log(1

/glyph[epsilon1]

)

Γ

caps

= Γ

L

+Γ

R

Γ

C

= Γ

1

+Γ

2

Γ = Γ

C

+Γ

caps

glyph[epsilon1]d

-

2

α

Γ

0

Γ

1

Γ

2

glyph[epsilon1]

Γ

R

Γ

L

spectrum of

ˆ

K

Re(

z

)

Im(

z

)

d

-

2

α

M

˙

d

-

2

α

1

1

M

d

-

α

0

The contour we select will come in three parts. The contour Γ 0 is an arbitrarily small contour enclosing 0 . The contour Γ will be in three parts which is symmetric under the reflection z ↦→-z . The main part will be Γ C parameterized by z = u + iη ( u ) with η ( u ) as in Proposition E.5 for u ∈ [ u 0 , u 1 ] where u 0 = u 0 ( d ) = Cd -2 α for some large C > 0 and u 1 is a small positive constant. This is connected by two curves, one which is a smooth curve Γ L which is on scale d -2 α and which is reflection symmetric, connects u 0 + iη ( u 0 ) to its conjugates and crosses the imaginary axis on [0 , cd -2 α ] (with c as in Proposition E.3). The other Γ R connects u 1 + iη ( u 1 ) to its conjugate by a smooth curve which avoids an glyph[epsilon1] neighborhood of [0 , 1] .

For Γ 0 , using Proposition E.3, we have

This can be evaluated explicitly in terms of a residue at 0 .

Proposition F.1. The function F 0 ( r ) is constant and

Proof. From Proposition E.3, we can apply the residue formula. Evaluating the residue and bounding the sum produces the statement.

The contours Γ R and Γ L both contribute error terms. Define the sum of the two as

Proposition F.2. There are positive functions f ( r ) and g ( r ) satisfying f ( r ) ≤ C exp( -cγBrd -2 α ) and g ( r ) ≤ C exp( -cγBr ) so that

Furthermore, for any M > 1 we can choose u 0 = Td -2 α and u 1 = 1 /T with T sufficiently large that F caps ( r ) satisfies for γBr ≤ M and γBr ≥ Md 2 α and some other C > 0

Hence this will appear as essentially constant on the loss curves. When combined with F 0 we have that F 0 ( r ) + F caps ( r ) is bounded above and below by constants times d -2 α +(2 β -1) + -1 .

Proof. Both the contributions from Γ L and Γ R give exponentially decaying errors, albeit at much different scales and lead to the f and g terms respectively. For the Γ R terms, we simply bound, using Proposition E.6,

On the Γ R contour, having picked the contour sufficiently close to [0 , 1] (independent of v, d ), we have for some δ > 0

Then we have

Hence this decays exponentially.

By construction of the Γ R contour, we can close the contour with an additional (nearly vertical) segment Γ V with real part u and height glyph[epsilon1]u . Moreover this can be chosen to evenly divide two poles { j -2 α } , by adding small horizontal segments. Then we can estimate on Γ V (essentially by Proposition E.4, with an extension for very small imaginary part when we split two poles)

Then integrating over Γ V , we get

Having enclosed the poles, we can apply the residue formula, and we have

for some j 0 with j 0 glyph[equivasymptotic] u -1 / (2 α ) 1 . Hence both contributions of Γ V and Γ R decay like g ( r ) for an appropriate choice of δ, C .

For Γ L , we use similar arguments. We use Proposition E.3 to replace the summation by a d -independent quantity, which also requires rescaling the contour by d -2 α . Then we have

Hence we are left with a dominant contribution of

In the case that 2 β > 1 we instead are left with

As the spectral support of f has a left edge, these decay exponentially. In either case, we can then deform the contour to run twice along the real axis and then vertically to the ends of the Γ L contour. The component along the vertical portion can be estimated by

(and using the boundedness of f, 1 /f ). This can be made to decay faster than the contribution from f .

Finally, the dominant contributions arise from the contour Γ C . We define:

where c β is as in Proposition E.5. Then this gives us the principal contribution to the limit:

Proposition F.3. Set for r ≥ 0

Then F C ( r ) is real-valued and satisfies for some constant C independent of u 1 , u 0 , α, β

Moreover, there is an M = M ( u 0 , u 1 ) > 0 and a positive bounded function C ( r ) so that if γBr ∈ [ M,d 2 α /M ] then

Furthermore, for any glyph[epsilon1] > 0 there is a M ( glyph[epsilon1],u 0 , u 1 ) large enough that C ( r ) ≤ 1 + glyph[epsilon1] for γBr ∈ [ M,d 2 α /M ] .

Proof. These follow in a similar way to the earlier Propositions, and so we do not enter the details. Instead, we give a brief overview, using the estimates given in Proposition E.5 and Proposition E.4.

Along F C , we can approximate m uniformly by

where c is real-valued and bounded and M = M ( glyph[epsilon1] ) . Hence using Proposition E.4,

for real valued A and c β = ∑ ∞ j =1 j -2 β if β > 1 / 2 or 0 otherwise as in Proposition E.4. Integrating each of these imaginary terms over Γ C produces F pp and F ac respectively. The real part is negligible, as the contour is close to the real axis (in particular the imaginary part of the contour is smaller than the real part by a factor of glyph[epsilon1] ).

Combining all of these propositions, we have the following conclusion glyph[negationslash]

Corollary F.1. For any α, β with α, β = 1 / 2 and α + β > 1 2 there is a function C ( r ) bounded above for all r so that

Moreover, for any glyph[epsilon1] > 0 there is a M ( glyph[epsilon1] ) large enough that C ( r ) ≤ 1 + glyph[epsilon1] for γBr ∈ [ M,d 2 α /M ] and for γBr > Md 2 α .

Proof. This follows directly from Proposition F.3, F.2 and F.1, and needs that the F curve is monotone to fill the gaps on which the approximations are made. ( There are potentially two windows on which the various approximations do not overlap: when γBr is a large constant and when it is on order d 2 α )

G Estimation of Kernel function

We can now give the approximation of the kernel function, which is represented by

with the same contours that were used for the forcing function.

Using Lemma E.1, we therefore can represent the kernel function as

By the residue theorem, the contribution from Γ 0 disappears, as does the V z term. Hence we are left with the representations

When α > 1 4 , the dominant contribution comes once more from the contour Γ C for which we get

It now follows swiftly from the estimates on m :

Proposition G.1. Suppose α > 1 4 . There is a positive function C ( r ) so that

and C ( r ) is bounded independent of d by a function of M for all rγB < d 2 α M . Moreover for any glyph[epsilon1] > 0 there is an M sufficiently large so that for rγB ∈ [ M,d 2 α /M ] , C ( r ) < 1 + glyph[epsilon1] .

Proof. By reflection symmetry, the real part of contour integral for K ( r ) vanishes. Also, for a given contour Γ A , we will define

and we will estimate each piece of Γ = Γ R +Γ C +Γ L separately.

We begin with the contributions from Γ R . Using Proposition E.6, we have that on Γ R

Following the same steps as in the proof of Proposition F.2, we can extend the Γ R contour by a straight line Γ V to enclose some residues, which leads to

with j 0 glyph[equivasymptotic] u -1 / (2 α ) 1 . Moreover, the contribution of the Γ V contour can be estimated (for some δ > 0 which can be made small by increasing u 1 ) by

Meanwhile making a Riemann sum approximation (and changing variables by j -2 α = u )

for 2 γBr < 1 u 1 . Hence by taking u 1 sufficiently small, we conclude that for 2 γBr < 1 u 1 ,

and also that | K R ( r ) | glyph[lessorsimilar] e -2 γBru 1 (1 -δ ) for larger (2 γBr ) .

The contributions from Γ C give, in a similar way for 2 γBr > 1 u 1 and 2 γBr < 1 u 1

Moreover for any glyph[epsilon1] > 0 there is an M > 0 sufficiently large that when (2 γBr ) is in [ M,d 2 α /M ] ,

by first choosing glyph[epsilon1] > 0 , then choosing the contour as in Proposition E.5 sufficiently far, and then possibly shrinking [ u 0 , u 1 ] . For larger r , it further satisfies an estimate that for some δ > 0 , which can be made smaller by increasing u 0 , K L ( r ) glyph[lessorsimilar] e -2 γBru 0 (1 -δ ) .

Finally, the contributions from Γ L , we have after changing variables

This can be compared to the same expression with m ( zd 2 α ) → f ( z ) and replacing (1 -2 γBzd -2 α + γ 2 B ( B +1) z 2 d -4 α ) r → exp( -2 γBr (Re z ) d -2 α ) . This gives for 2 γBr glyph[lessorsimilar] d 2 α

where f ( u ) = -1 π lim glyph[epsilon1] → 0 f ( u + iglyph[epsilon1] ) is the spectral density corresponding to f . Hence it follows that for 2 γBr glyph[lessorsimilar] d 2 α ,

Remark G.1. In contrast, when α < 1 4 , the dominant contribution to K is from K L , so that

and moreover the density f ( u ) glyph[lessorsimilar] u -1 / (2 α ) so that the integral is convergent (which is implicit in Proposition E.5)

We conclude with noting that for ther norm of K we can directly evaluate it using a contour integral. Summing the contour integral expression

We additionally can more generally evaluate a partial norm

Combining this with Proposition E.6, this leads directly to tight estimates for the kernel norm.

Corollary G.1. When 2 α > 1 and γ ( B +1) < 2 ,

When 2 α < 1 (and recalling that we take γ on the order d 2 α -1 in this case),

Furthermore, for any glyph[epsilon1] > 0 there is an M > 0 so that if γBr > M then

Proof. For the first case with 2 α > 1 , Proposition E.6 gives on Γ R

Using this, and completing the Γ R contour via a vertical line, we get a residue contribution which matches the claim, up to some number of terms j 0 (which can be made as large as desired). Proposition E.5 and E.3 can be used to control the parts of the contour near 0 and in the middle.

For the second case 2 α < 1 , since γ is small, we may deform the contour to be at a fixed distance from [0 , 1] , and then once more we can use Proposition E.6 which gives in 1 step that

For the final statement, under the conditions given on r

uniformly over the contours, and the estimate follows directly.

From here, we can derive the 'sub-exponential' property of K .

Proposition G.2. Suppose α > 1 4 . For any glyph[epsilon1] > 0 , there is an M sufficiently large so that for γBr ∈ [ M,d 2 α /M ]

Proof. We note that in the range of r given, we can conclude that for any δ, glyph[epsilon1] > 0 , by increasing M

furthermore that for s > r/ 2

and that finally for s > r/ 2

where the final estimate follows by estimating

once γBr > M , and moreover their ratio tends to 1 as M →∞ .

With these estimates in place, we can break the estimate up as

The final sum is bounded by

Meanwhile, for the first sum, rδ

2

∑

s

=0

where we have used that K pp ( r (1 -δ )) ≤ 1+ glyph[epsilon1] 1 -glyph[epsilon1] (1 + O ( δ )) K pp ( r ) .

H Asymptotics of forcing function and kernel function

With this, we now analyze the asymptotics of each of these terms individually. These asymptotics often rely on a result about how close a Riemann sum is to its integral. We state below the main result of this nature that we used:

Proposition H.1 (Trapezoidal Rule, [15]) . If f is continuous, then for each integer n > 0 , the integral of f on [ a, b ] is approximated by

where x i = a + i ( b -a ) /n , 0 ≤ i ≤ n . Define the error in the trapezoid rule

If f has an integrable first derivative as an improper integral, thens

K

(

s

)

K

(

r

-

s

)

2(1 +

O

(

glyph[epsilon1]

))(1 +

O

(

δ

))

‖

K

‖

K

pp

(

r

)

,

H.1 Pure point forcing term, F pp ( r )

In this section, we prove an asymptotic for the pure point forcing term, see (85),

Proposition H.2 (Pure point forcing term) . Suppose 2 α +2 β > 1 . For any glyph[epsilon1] > 0 , there is an M > 0 so that for γBr ≥ M ,

where

Furthermore, for any ˜ M > 0 , there exists some constants C, ˜ C,c > 0 independent of d so that

Proof. First, a simple computation shows that

Let ρ = -(1 + β/α ) + 1 / (2 α ) . A simple computation, using the change of variables w = 2 γBru , yields

Then we have that

Since ∫ ∞ 0 w (2 β -1) / (2 α ) exp( -w ) d w , there exists a M large so that

and if r > ˜ Md 2 α ,

Thus the first result is shown.

If γBr < ˜ M , then

Moreover, we have that exp( -2 γBru ) ≥ exp( -2 ˜ M ) . Therefore, we get that

Now suppose γBr > ˜ Md 2 α . By the previous part, we know that F pp ( r ) ≤ (1 + glyph[epsilon1] ) g ( r ) . Moreover, we see that g is decreasing. As a result, we see that up to constants

for some constants C, ˜ C > 0 . Hence the result is shown.

H.2 Model misspecification, point mass at 0 , F 0 ( r )

Recall, from Proposition F.1, the forcing function point mass at 0 , satisfies

In this section, we provide an asymptotic for F 0 ( r ) (see Proposition H.3) which represents the limiting value the loss obtains as r → ∞ . Unlike the pure point process above, this asymptotic depends on whether 2 β > 1 .

We begin by showing that the κ defined implicitly in (87) is uniquely determined and dimensionless. Lemma H.1. Suppose v and d are admissible such that the ratio v d > 1 . Then the equation

has a unique solution κ such that 0 < κ < ∞ .

Proof. Let w def = κ and F ( w ) def = ∫ v/d 0 1 w + u 2 α d u and set G ( w ) = wF ( w ) . To solve the fixed point equation, we want to find G ( w ) = 1 . First, it is clear that lim w → 0 G ( w ) = 0 . Second, we see that as lim w →∞ G ( w ) = v d > 1 . As G ( w ) is continuous, it follows that there exists a solution κ to G ( κ ) = 1 .

To show that κ is unique, amounts to showing that G ( w ) is strictly increasing for w ≥ 0 . First, we see that

We note that w ↦→ u 2 α w + u 2 α is strictly decreasing in w . So w ↦→ 1 -u 2 α w + u 2 α is strictly increasing in w . Hence G ( v ) is strictly increasing and there is a unique solution to G ( κ ) = 1 .

Now we give an asymptotic for F 0 .

Proposition H.3 (Asymptotic for F 0 ) . Suppose v and d are admissible such that the ratio v/d > 1 and suppose 2 α +2 β > 1 . Let 0 < κ ( v/d ) < ∞ be the unique solution to

Then as d →∞

Proof. We consider 2 cases. Let κ = κ ( v/d ) .

Case 1: Suppose 2 β > 1 : Let ˜ C def = ∑ v j =1 j -2 β , which is finite as 2 β > 1 . Consider the following

To handle the large j values, we see that there exists a j 0 large so that

where we used that j -2 α κd 2 α +1 > 1 . For the small j , we use that d can be large. Hence,

For sufficiently large d , we can make the right-hand-side small. Therefore, E 1 ( r ) is small for sufficiently large d and hence, the result holds.

Case 2: Suppose 2 β < 1 : To show this case, we define the following errors

It is clear, for sufficiently large d , E 22 ( r ) is small.

For the first error term, we use a Riemann sum approximation, that is,

Letting a = 1 /d , b = v/d , n = v -1 , x j = 1 /d + j/d , and f ( x ) = x -2( α + β ) x -2 α κ +1 , we can approximate the summation with an integral. Using Prop. H.1,

One can check that ∫ v/d 1 /d | f ′ ( x ) | d x ∫ v/d 0 u -2 β d u κ + u 2 α < C where C is independent of d . For sufficiently large d ,

Hence, Case 2 is shown.

H.3 Absolutely continuous forcing function, F ac ( r )

We now turn to the absolutely continuous forcing function, defined as

where c β = ∑ v j =1 j -2 β if 2 β > 1 and 0 otherwise. From this, we derive a simple asymptotic formula.

Proposition H.4. There exists a constant C ( α, β ) > 0 such that

Suppose now 2 α > 1 and 2 β > 1 . For any glyph[epsilon1] > 0 , there is an M > 0 so that for γBr ∈ [ M,d 2 α /M ] ,

Furthermore, for any ˜ M > 0 , these exists some constants C, c > 0 independent of d so that

Proof. We proceed by cases. The case 2 β < 1 is immediate as c β is only non-zero for 2 β > 1 .

Case: 2 β > 1 and 2 α < 1 . In this case, we just bound directly bound F ac ( r ) . Dropping the exponential, we get

We know that F 0 ( r ) glyph[equivasymptotic] d -2 α +max { 0 , 1 -2 β } and thus the result is shown.

Next we show (89).

Case: 2 β > 1 and 2 α > 1 . First, we make the following observation. The integral is

Define C = c β 2 α . Let us consider the following

First, we see

Case: E 1 . Suppose γBr ≤ 1 /Md 2 α . Here we can just use directly the u and disregard the exponential:

for some constant ˜ c . Now we have that

By choosing M large, this can be made small.

Case: E 2 . Suppose γBr ≥ M . Let us consider

It follows that

Therefore, by choosing M large, we have that this can be small. This proves (89).

To finish the proof, let us first suppose that γBr ≤ ˜ M . Then we have that

When γBr ≥ ˜ Md 2 α , we have that

H.4 Kernel function asymptotic.

We recall the term K pp defined as

We now give an asymptotic for such a function.

Proposition H.5 ( K pp asymptotic) . Suppose α > 1 / 4 . For any glyph[epsilon1] > 0 , there is an M > 0 so that for γBr ≥ M ,

where

Moreover, for any ˜ M > 0 , there exists constants c, C, ˆ C > 0 , such that when 2 α > 1 ,

Furthermore, for any ˜ M > 0 , there exist a constant ˜ C > 0 , such that

Proof. The first part of the argument, (90), follows immediately from the proof of F pp , Prop. H.2.

If 2 α > 1 , then we always have γ 2 B is constant order. Therefore, using the same argument as F pp (see Prop. H.2), we get that there exists constant c, C > 0 such that c ≤ K pp ( r ) ≤ C for γBr ≤ ˜ M .

If 2 α < 1 , then γ glyph[equivasymptotic] d 2 α -1 . Therefore, for γBr ≤ ˜ M , we have that

The later inequality follows using the same bounding argument as F pp ( r ) .

As γ 2 B ≤ C , then the same argument in F pp ( r ) shows for γBr ≥ ˜ Md 2 α that K pp ( r ) ≤ ˜ C F 0 ( r ) .

We now turn to the last quantity that appears in the Volterra equation.

Proposition H.6 (Forcing function norm) . Provided 2 β > 1 , we have that

and when 2 α < 1 , for some constant M > 0 .

Next, suppose 2 β < 1 . Then there exists an ˜ M,M > 0 such that

and it follows for all γBr ≤ Md 2 α ,

Proof. Suppose 2 β > 1 . First note that in this region ∑ Md 2 α / ( γB ) s =0 F 0 ( r ) glyph[lessorsimilar] 1 γB for any fixed M > 0 , Proposition H.3.

Next, let us consider the pure point part of F , i.e., F pp . Choose M and ˜ M so that F pp is behaving like the asymptotic in Proposition H.2, i.e., for γBr ≥ ˜ M ,

and for γBr ≤ ˜ M , F pp ( r ) ≤ C for some constant C > 0 independent of d . It follows then that

To handle the rest of the sum, we see that

Here we use that the Riemann sum approximation with a = ˜ M d 2 α , b = M + ˜ M d 2 α , n = Md 2 α and f ( x ) = x -(1+ β/α )+1 / (2 α ) .

Using a similar argument for F ac ( r ) (Proposition H.4)when 2 α > 1 (otherwise we do not need to worry about F ac ), we have that

When r ≤ ˜ M/ ( γB ) , we have that F ac ( r ) glyph[lessorsimilar] d -1 . Hence, ∑ ˜ M/ ( γB ) r =0 F ac ( r ) glyph[lessorsimilar] 1 γB . The first result, (91), then follows from Corollary F.1.

Consider 2 β < 1 and 2 α < 1 . We do not need to worry about F ac in this region because it does not exist in this region. Choose M and ˜ M so that both K pp and F pp are in their asymptotic regions and, using Proposition G.1, K ( r ) glyph[equivasymptotic] K pp ( r ) . Using a similar argument as above, we estimate the summation of F pp as an integral. For any r ∈ [ ˜ M/ ( γB ) , Md 2 α / ( γB )]

Using the asymptotic for K pp (Proposition H.5),

We will show that this is less than F pp ( r ) . Using the asymptotic for F pp ( r ) (Proposition H.2), let us suppose

In this region, the learning rate is γ glyph[equivasymptotic] d 2 α -1 . Thus, we see that

This is true and so we have that

For F 0 , with r ∈ [ ˜ M/ ( γB ) , Md 2 α / ( γB )]

Therefore, we get that

We will show that this is less than F pp . For this, we see

Hence, we have that

Since there is no F ac in this region, we immediately get from Corollary F.1

For s ∈ [0 , ˜ M/ ( γB )] , we have that F ( s ) glyph[lessorsimilar] C . Thus we immediately get that

for all r . This proves the result for 2 β < 1 and 2 α < 1 .

Consider 2 β < 1 and 2 α > 1 . As in the previous case, we do not need to consider F ac as it does not exist here. The proof will be similar to the previous case. Choose M and ˜ M so that both K pp and F pp are in their asymptotic regions and, using Proposition G.1, K ( r ) glyph[equivasymptotic] K pp ( r ) .

First, by the same argument as in 2 β < 1 and 2 α > 1 , we immediately have for s ∈ [0 , ˜ M/ ( γB )] , we have that F ( s ) glyph[lessorsimilar] C ,

for all r .

As before, we have for any r ∈ [ ˜ M/ ( γB ) , Md 2 α / ( γB )]

Note here that γ is constant. We will show that this is less than F pp ( r ) . Using the asymptotic for F pp ( r ) (Proposition H.2), let us suppose

Hence, we have that

For F 0 , with r ∈ [ ˜ M/ ( γB ) , Md 2 α / ( γB )]

Therefore, we get that

We will show that this is less than F pp . For this, we see

Now we see that ( γBr ) β/α glyph[lessorsimilar] d 2 β glyph[lessorsimilar] d 2 β +2 α -1 . Hence, we have that

The result is thus shown in this case.

I Optimizing over batch and learning rate

The previous sections use batch size B = 1 and the maximal learning rate allowed. In this section, we consider optimizing compute-optimal curves with respect to batch size and learning rate, i.e., find d glyph[star] , γ glyph[star] , B glyph[star] such that

I.1 Optimal batch size

We see that batch essentially scales out of the problem and therefore the batch has no effect on the compute-optimal curves. To see this, we observe from Table 5 that

It immediately follows that the batch size has no effect on the compute-optimal curves. Therefore any batch size (e.g., B = 1 ) that satisfies the necessary and sufficient condition for convergence (Prop. C.2), that is, γ ( B +1) < 2 , will yield the same compute-optimal curves.

This is not necessarily true for the learning rate as we will see in the next section.

I.2 Optimal learning rate

Without loss of generality, we let the batch size B = 1 . From the expressions in (95), we see that F pp and F ac are monotonically decreasing in learning rate γ . Moreover in Phase III, 1 γB K pp , is also monotonically decreasing in the learning rate. Therefore, in Phases I, II, and III, the optimal learning rate choice is to choose γ maximally. In the cases of Phase Ia, II, and III, this would mean γ constant (see Prop C.2) and in Phase Ib, Ic, γ ∼ d 2 α -1 . It remains to understand the effect of the learning rate in Phase IV.

From Proposition C.6, we know that the loss is given by

By taking derivatives, we see that

We need to check that γ glyph[star] is feasible, i.e., γ glyph[star] < 1 (which it is) and γ glyph[star] < d 2 α -1 . For the later, a simple check shows that

when α > 1 / 4 and 2 β > 1 , i.e., precisely Phase IV. The compute-optimal curve in Phase IV with optimal stepsize is the following.

Proposition I.1 (Phase IV, optimal γ , compute-optimal curve) . Suppose 1 / 4 < α < 1 / 2 and 2 β > 1 . Then

The trade off occurring where 1 γB K pp = F 0 .

We note that there is only one Phase IV (and no sub-phases).

J Experimental Results

To measure the exponents of the scaling law and parameter count, we follow approach 12 1 and 2 from [24]. We explain the method below using ( α, β ) = (0 . 5 , 0 . 7) as an example. The theoretical prediction of the scaling law and parameter count exponents for this example are η = 0 . 5 and ξ = 0 . 5 , resp. (see Table 2). We then repeat this procedure for total of 32 pairs of ( α, β ) in the phase diagram; see Fig. 15 and Fig. 16. The theoretical predictions of these two exponents are shown in the heatmaps Fig. 8.

First, we run SGD for parameter counts

The SGD learning curves for ( α, β ) = (0 . 5 , 0 . 7) with parameters d ∈ [800 , 1600 , 3200 , 6400 , 12800] are shown in Fig. 9a.

(a) Parameter Count Exponents (

ξ

)

(b) Scaling Law Exponents ( η )

1.0

0.6

8°0

la

0.5

Param Count Exponents

0.4

0.6

-0.3

0.4

b

IVa

0.2

IVb

0.2

0.1

Univsersal Scaling

Ic

dαf0.5

0.0

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

β

1.0

0.7

0.8

la

0.6

0.5

Scaling LawExponents

0.6

0.4

0.4

Hia

0.3

0.2

0.2

Univsersal Scaling

Ic

0.1

dxf0.5

0.0

0.4

-0.2

0.0

0.2

0.4

0.6

8'0

1.0

0.0

β

J.1 Measuring the Scaling Law Exponent

We follow Sec 3.1 in [24]. First, we choose an IsoFLOP window [ f min , f max ] and construct f j 's using a geometric spacing between f 1 = f min and f n = f max. For each IsoFLOP slice, f j , (e.g., f j = 2 e 7 is the vertical line in Fig. 9a), we find the minimum loss across all d . We denote this minimum value by P glyph[star] ( f j ) and the associated optimal parameter by d glyph[star] ( f j ) . As an example, in Fig. 9a, P glyph[star] ( f j ) = 1 . 6 e -3 and the associated optimal parameter d glyph[star] ( f j ) = 6400 .

We obtain the compute-optional frontier (highlighted in red in Fig. 9b) by plotting

We then fit a power-law curve P glyph[star] ( f ) = a × f -ˆ η to predict the relationship between the compute f and optimal loss P glyph[star] . This is shown as the dashed line in Fig. 9b. For ( α, β ) = (0 . 5 , 0 . 7) , this gives

whereas our theoretical result predicts and the optimal parameter count

J.2 Measuring Parameter Count Exponent: Approach 0

One benefit of our theoretical framework is that the solution of the Volterra equation (eq. 10) is deterministic. As such, precise numerical evaluation can determine the instantaneous slope of the compute-optimal curves using a new approach that is not necessarily feasible when dealing with noisy

(a) IsoFLOP

ScalingLawExponent:0.515(Measurement)vs 0.500(Theory)

100

10-1

Training Loss

10-2

10-3

SGD:(α,β)=(0.5,0.7)

Compute-OptimalFrontier

10-4

Prediction f(x)= 22.61x-0.51

104

106

108

1010

Flops ('6ND)

100

d=800

d=12800

d=1600

IsoFlop: 2e7

d=3200

IsoLoss: 1.6e-3

10-1

d=6400

★

Compute-Optimal

Training

10-2

10-3

10-4

103

104

105

106

107

108

109

1010

Flops (6ND)

(b) Compute-Optimal Frontier

(a) IsoFLOP Window [1e6, 1e8]

(b) IsoFLOP Window [2e6, 0.5e8]

Optimal Param Count (Approach 1)

Approach 1:f(x) = 0.25x0.551

#Params

104

103

107

108

109

1010

Flops (6ND)

105

Optimal Param Count (Approach 1)

Approach 1:f(x)=0.15x0.58

#Params

104

103

107

108

109

1010

Flops (6ND)

SGD curves. Specifically, we search for the unique tangent line that intersects the loss-versus-flops curves for two adjacent values of d , i.e. we numerically solve the following system for f 1 and f 2 :

where P 1 is the loss curve for d = d 1 and P 2 is the loss curve for d = d 2 . When d 1 and d 2 are close, we obtain an accurate estimate of the parameter count exponent by measuring the discrete logarithmic derivative, (log( d 2 ) -log( d 1 )) / (log( f ∗ 2 ) -log( f ∗ 1 )) .

J.3 Measuring Parameter Count Exponent: Approach 1

To predict the optimal parameter count exponent, we fit the function d glyph[star] = a × f b , a, b constants, to the measurements in (97) (see e.g., Fig. 10a). For the example ( α, β ) = (0 . 5 , 0 . 7) (Fig. 10a), this approach gives

Note that the fit of the exponent is very sensitive to the choice of IsoFLOP window. When we change the window from [1 e 6 , 1 e 8] to [2 e 6 , 0 . 5 e 8] , the parameter count exponent changes from 0 . 51 to 0 . 58 , as shown in Fig.10b. The theoretical prediction of this exponent is 0 . 5 .

J.4 Measuring Parameter Count Exponent: Approach 2

For each IsoFLOP slice, f j , we obtain a set of training loss values depending on d , { P ( f j , d i ) } 1 ≤ i ≤ m . In our running example, d 1 = 200 and d m = 12800 (see Fig. 9a). We then fit a parabola (quadratic

(a) IsoFLOP Quadratic fit

(b) Approach 2

Phase Ill:(α,β)=(0.50,0.7).Exponent(Theory):0.500

OptimalParamCount

Approach2:f(x)=0.17x0.565

#Params

104

103

107

108

109

1010

Flops('6ND')

-1.00

-1.25

-1.50

(SS07

-1.75

(Train

-2.00

0

5

1

6

-2.25

6.0e6

-2.50

1.2e7

-2.75

2.2e7

4.3e7

-3.00

2.50

2.75

3.00

3.25

3.50

3.75

4.00

4.25

4.50

Log10 (d)

(a) IsoFLOP Window

[1

e

6

,

5

e

8]

(b) IsoFLOP Window

[1

e

6

,

5

e

8]

(c) IsoFLOP Window

[1

e

6

,

5

e

8]

(d) IsoFLOP Window

[1

e

6

,

1

e

8]

(e) IsoFLOP Window

[1

e

6

,

1

e

8]

(f) IsoFLOP Window

[1

e

6

,

1

e

8]

1.0

Params Exponent (Theory)

α=0.7Slice

params exponent approach 2

1.0

α=0.27 Slice

params exponent approach 2

1.0

params exponents

β=0.7 Slice

params exponent approach2

0.8

SL Exponent (Theory)

params exponent approach 1)

sl exponent:empirical

0.8

SL exponents

params exponent approach 1

sl exponent:empiricall

8'0

params exponent:empirical

SL exponents

sl exponent:empirical

Exponents

0.6

0.6

0.6

0.4

0.2

la

IIII

0.2

Ib

IVB

0.2

Ic

bIVa

IIII

I

0.0

0.0

0.2

0.4

=

0.6

0.8

1.0

1.2

0.0

0.4

0.6

0.8

1.0

1.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

β

betas

alpha

1.0

0.8

SL Exponent (Theory)

params exponent approach 1)

Params Exponent (Theory)

α=0.7Slice

params exponent approach 2

sl exponent:empirical

1.0

80

SL exponents

params exponeits

params exponent approach 1

α=0.27 Slice

params exponent approach 2

sl exponent:empirical

1.0

0.8

SL exponents

params exponent:empirical

params exponents

β=0.7Slice

params exponent approach2

sl exponent:empirical

Exponents

0.6

exponents

0.6

0.6

0.4

0.4

0.2

la

IIII

0.2

Ib

IVb

0.2

Ic

IVa

IIII

0.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

0.0

0.4

0.6

0.8

1.0

1.2

0.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

β

betas

eudie

function) to { (log P ( f j , d i ) , log d i ) } 1 ≤ i ≤ m , i.e., we find ( a, b, c ) such that

This is shown in Fig. 11a. After solving for ( a, b, c ) , we find the d glyph[star] ( f j ) that minimizes a 2 log 2 d + b log d + c . Repeating this procedure for all f j 's gives a set of pairs { ( f j , d glyph[star] j ) } 1 ≤ j ≤ n . In the final step, we power-law fit this set. For the example ( α, β ) = (0 . 5 , 0 . 7) (see Fig. 11b), this gives

J.5 Exponents comparison: Theory vs Measurement

We compare the empirical measurements of the exponents against their theoretical predictions in Fig. 12. We chose three slices across the phase diagram

1. α = 0 . 7 Slice (Fig. 12a), in which ( α, β ) goes from Phase Ia, II and III.

2. α = 0 . 27 Slice (Fig. 12b), in which ( α, β ) goes from Phase Ib to Phase IVb.

3. β = 0 . 7 Slice (Fig. 12c)' in which ( α, β ) goes from Phase Ic, IVb, IVa, III and to II.

For the scaling law exponents, the empirical measurement agrees with the theoretical prediction quite well. For the parameter count, the agreement is good but not as good as that of the scaling law exponents. Noticeably, there is disagreement between Approach 1 and Approach 2. Such disagreement is not surprising, as empirical measurements are sensitive to the choice of the IsoFLOP windows and we use the same IsoFLOP window [1 e 6 , 5 e 8] for all ( α, β ) . This is clearly suboptimal. We briefly discuss this in the next subsection.

J.6 Instantaneous slope

In this section, we demonstrate that there can be strong finite-size d effects in the measurements of the scaling law and parameter count exponents. We measure the instantaneous slope as a function of parameter count d for the Volterra equation (10). See Fig. 13. To do so, we generate the Volterra solutions for a geometrically spaced sequence of d 's with ratio 1 . 05 . We then apply Approach 1 with a very dense IsoFLOP window (100 IsoFLOP Slices between [2 e 4 , 2 e 7] ). We then slide a smaller IsoFLOP window (20 IsoFLOP slices) from left to right to generate a sequence of scaling law (parameter count) exponent, as shown in the middle (right) plot in Fig. 13. These exponents varying slowly when the window slides from a small flops regime to a large flops regime. For example, the scaling law exponent η changes from η = 0 . 440 to η = 0 . 413 from left to right, while the parameter count exponent changes from ξ = 0 . 450 → 0 . 575 . Using the global window (100 IsoFLOP) to measure these exponents, we have η = 0 . 42 and ξ = 0 . 52 which are very close to their average over all small windows: η = 0 . 418 and ξ = 0 . 526 .

100

0.440

measurement

0.575

measurement

0.435

mean

Param count exponent

0.550

mean

Loss

10-1

exponent

0.430

0.525

Training

0.425

0.500

Small d fit

10-2

0.420

0.475

Large d fit

0.415

f(x)=14.379 x^-0.412

0.450

103

105

107

109

4.5

5.0

5.5

6.0

4.5

5.0

5.5

6.0

Log10(Flops)

Log10(Flops)

J.7 Negative β .

In Fig. 14, we show that our theoretical results work well for β < 0 .

(α,β)=(0.7,-0.2)

(α,β)=(0.7,-0.15)

(α,β)=(0.7,-0.1)

D=200

D=3200

3×100

D=200

D=3200

D=200

D=3200

D=400

D=6400

D=400

D=6400

D=400

D=6400

4 ×100

D=800

D=12800

D=800

D=12800

D=800

D=12800

3×100

D=1600

Theory

2×100

D=1600

Theory

100

D=1600

Theory

2 ×100

100

6×10-1

106

108

1010

106

108

1010

106

108

1010

(α,β)=(0.7,-0.05)

(α,β)=(0.7,0.0)

(α,β)=(0.7,0.2)

100

D=200

D=3200

D=200

D=3200

D=200

D=3200

D=400

D=6400

100

D=400

D=6400

D=400

D=6400

100

D=800

D=12800

D=800

D=12800

D=800

D=12800

D=1600

==

Theory

D=1600

==

Theory

10-1

D=1600

Theory

10-1

10-2

10-1

10-3

106

108

1010

106

108

1010

106

108

1010

J.8 Additional plots for different phases

We summarize the measurement of the scaling law exponents and optimal parameter count exponents in Fig. 15 and Fig. 16, resp.

Expnent 0.169 (Measurement) vs 0.222 (Theory)

Expnent 0.234 (Measurement) vs 0.281 (Theory)

Expnent 0.243 (Measurement) vs 0.285 (Theory)

Expnent 0.008 (Measurement) vs 0.020 (Theory)

100

100

100

4 ×100

Training Loss

3×100

10-1

10-1

SGD: (α,β)= (0.2, 0.7)

SGD:(α,β)= (0.25, 0.7)

SGD: (α,β)= (0.26,0.7)

SGD:(α,β)= (0.27,0.25)

Volterra

Volterra

Volterra

2×100

-===-

Volterra

10-1

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Prediction f(x)= 4.18x-0.17

Predictionf(x)=6.35x-0.23

Predictionf(x)= 6.43x-0.24

Predictionf(x)=1.91x-0.01

104

106

108

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

Expnent 0.057(Measurement) vs 0.070 (Theory)

Expnent 0.137(Measurement) vs 0.170(Theory)

Expnent 0.193(Measurement) vs 0.270 (Theory)

Expnent 0.230 (Measurement) vs 0.281(Theory)

3×100

100

10°

100

2 ×100

SGD:(α,β)= (0.27,0.3)

SGD:(α,β)= (0.27, 0.4)

SGD:(α,β)= (0.27,0.5)

10-1

SGD:(α,β)=(0.27,0.6)

100

Volterra

Volterra

10-1

Volterra

Volterra

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Predictionf(x)= 2.77x-0.06

Predition f(x)= 4.41x-0.14

Predictionf(x)= 5.74x-0.19

Predictionf(x)= 6.47x-0.23

104

106

108

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

Expnent 0.259 (Measurement) vs 0.288 (Theory)

Expnent 0.277(Measurement) vs 0.292(Theory)

Expnent 0.289 (Measurement) vs 0.295 (Theory)

Expnent 0.308 (Measurement) vs 0.298(Theory)

100

10-1

10-1

10-1

SGD: (α,β)=(0.27, 0.7)

SGD:(α,β)= (0.27,0.8)

SGD:(α,β)= (0.27,0.9)

SGD:(α,β)=(0.27,1.0)

Volterra

Volterra

Volterra

Volterra

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

10-2

Compute-Optimal Frontier

Predictionf(x)= 7.34x-0.26

10-2

Prediction f(x)=7.76x-0.28

10-2

Predictinf(x)=7.89x-0.29

Predictionf(x)= 9.35x-0.31

104

106

10

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

Expnent 0.308(Measurement) vs 0.300 (Theory)

10°

Expnent 0.273(Measurement) vs 0.290 (Theory)

10°

Expnent 0.293(Measurement) vs 0.300 (Theory)

10°

Expnent 0.400 (Measurement) vs 0.400 (Theory)

10-1

Training Loss

10-1

10-1

10-1

SGD: (α,β) = (0.27, 1.1)

SGD:(α,β)= (0.28,0.7)

SGD: (α,β)= (0.3,0.7)

10-2

SGD:(α,β)=(0.4,0.7)

Volterra

Volterra

Volterra

Volterra

10-2

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Compute-Optimal Frontier

Predictionf(x)=8.38x-0.31

10-2

Predictio f(x)= 8.1x-0.27

10-2

Predictionf(x)= 8.56x-0.29

10-3

Predictionf(x)= 12.33x-0.4

104

106

108

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

100

Expnent 0.516 (Measurement) vs 0.500 (Theory)

100

Expnent 0.586 (Measurement) vs 0.583（Theory)

10°

Expnent 0.415 (Measurement) vs 0.417 (Theory)

10°

Expnent 0.476 (Measurement) vs 0.500 (Theory)

10-1

10-1

10-1

10-1

Training Loss

10-2

10-2

10-2

10-2

10-3

10-3

SGD: (α,β) = (0.5, 0.7)

Volterra

Compute-Optimal Frontier

10-4

SGD:(α,β)= (0.6, 0.7)

Volterra

Compute-Optimal Frontier

10-3

SGD:(α,β)= (0.7, 0.3)

Volterra

Compute-Optimal Frontier

10-3

SGD:(α,β)= (0.7,0.4)

Volterra

Compute-Optimal Frontier

10-4

Predictionf(x)= 23.02x-0.52

Prediction f(x)= 24.23x-0.59

Predition f(x)= 7.77x-0.42

10-4

Predictionf(x)= 9.33x-0.48

10-5

104

106

10

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

100

Expnent 0.535 (Measurement) vs 0.583(Theory)

10°

Expnent 0.586 (Measurement) vs 0.615(Theory)

100

Expnent 0.631 (Measurement) vs 0.643 (Theory)

10°

Expnent 0.664 (Measurement) vs 0.643 (Theory)

10-1

10-1

10-1

10-1

Training Loss

10-2

10-2

10-2

10-2

10-3

SGD:(α,β)= (0.7,0.5)

Volterra

10-3

10-4

SGD:(α,β)= (0.7,0.6)

Volterra

10-3

10-4

SGD:(α,β)= (0.7,0.7)

Volterra

10-3

10-4

SGD:(α,β)=(0.7,0.8)

Volterra

10-4

Predictionf(x)=12.2x-0.54

Compute-Optimal Frontier

10-5

Predictionf(x)=15.92x-0.59

Compute-Optimal Frontier

10-5

Predictionf(x)=20.83x-0.63

Compute-Optimal Frontier

10-5

Predictionf(x)=24.66x-0.66

Compute-Optimal Frontier

104

106

108

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

100

Expnent 0.667 (Measurement) vs 0.643 (Theory)

100

Expnent 0.675 (Measurement) vs 0.643 (Theory)

100

Expnent 0.691 (Measurement) vs 0.643 (Theory)

10°

Expnent 0.682 (Measurement) vs 0.643(Theory)

10-1

10-1

10-1

10-2

10-2

10-2

10-3

10-3

10-4

SGD:(α,β)= (0.7,0.9)

10-4

SGD: (α,β)=(0.7,1.0)

10-4

10-4

SGD:(α,β)=(0.7,1.2)

10-s

10-6

Preditionfx)=19.2x-0.67

Compute-Optimal Frontier

Volterra

10-6

Predictionf(x)= 17.93x-0.67

Volterra

Compute-Optimal Frontier

10-5

10-6

Predictionf(x)=19.75x-0.69

Compute-Optimal Frontier

Volterra

10-5

10-6

Predictionf(x)=15.33x-0.68

Volterra

Compute-Optimal Frontier

104

106

108

1010

104

106

108

1010

104

106

108

1010

104

106

108

1010

Expnent 0.656 (Measurement) vs 0.667(Theory)

Expnent 0.691 (Measurement) vs 0.688(Theory)

Expnent 0.706 (Measurement) vs 0.706 (Theory)

Expnent 0.688 (Measurement) vs 0.722 (Theory)

100

10°

10°

1-01

10-2

10-2

10-2

10-3

Training Loss

10-4

10-4

10-5

10-4

10-7

10-6

10-6

SGD:(α,β)= (0.8,0.7)

SGD:(α,β)=(0.9,0.7)

SGD:(α,β)=(1.0,0.7)

SGD:(α,β)=（1.1,0.7)

10-6

Compute-OptimalFrontier

Volterra

10-9

Volterra

Compute-Optimal Frontier

10-8

Compute-OptimalFrontier

Volterra

10-8

Volterra

Compute-Optimal Frontier

Prediction f(x)=14.98x-0.66

10-11

Prediction f(x)=14.82x-0.69

01-0L

Predictionf(x)=11.49x-0.71

10-10

Predictionf(x)=5.17x-0.69

104

106

108

1010

104

HEHR

106

108

1010

104

106

108

1010

H

104

106

108

1010

Flops (6ND)

Flops (6ND)

Flops (6ND)

Flops (6ND)

Phaselc:(αβ）=(0.200.7）.ExpnentTheory):.556

Phase lc: (α,β) = (0.25, 0.7). Exponent (Theory):0.563

Phase IVb: (α, β) = (0.26, 0.7).Exponent (Theory): 0.567

Phase Ib: (α, β) = (0.27, 0.2). Exponent (Theory): 0.500

104

Param Count

104

104

104

103

103

Optimal Param Count

Approach2:fx)=0.09x.51

103

★

Aproach2:f(x)=0.08x0.571

Optimal Param Count

103

★

Aproach2:f(x)=0.07x0.579

Optimal Param Count

Optimal Param Count

Approach 2:f(x)=0.02x0.566

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

。

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Approach1:f(x)=0.16x0.53

Approach1:f(x)=0.45x0.489

Approach1:f(x)=0.53x0.483

102

Approach1:f×x)= 0.06x0.524

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

PhaseIb:(αβ）=(0.270.3）.Expent Thery):00

PhaselIb:(αβ）=(0.27,0.4)）.Exponent Thery):00

PhaseIb:(αβ）=0.27,0.5）.ExponentThery):500

Phase IVb: (α,β) = (0.27, 0.6). Exponent (Theory):0.545

★

Optimal Param Count

Param Count

104

Approach2:fx)=0.07x0.539

Aproach1:f(x)=0.05x0.561

Optimal Param Count (Approach 1)

104

104

104

103

103

Approach2:f(x)=0.09x.553

Optimal Param Count

103

Aproach2:f(x)=0.09x0.561

Optimal Param Count

103

★

Approach2:f(x)=0.07x0.582

Optimal Param Count

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Aproach1:f{×x)=0.14x0.532

Aproach1:f{x)=0.31x0.502

Approach 1:f×x)=0.57x0.477

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

Phase IVb: (α,β) = (0.27, 0.7). Exponent (Theory):0.572

Phase IVb: (α,β)= (0.27, 0.8).Exponent (Theory): 0.590

Phase IVb: (α,β) = (0.27, 0.9). Exponent (Theory):0.603

Phase IVb: (α,β) = (0.27, 1.0). Exponent (Theory): 0.613

104

104

104

★

Optimal Param Count

★

Optimal Param Count

★

Optimal Param Count

Optimal Param Count

103

Approach2:f(x)=0.07x0.585

103

Aproach2:f(x)=0.06x0.598

103

Aproach2:f(x)=0.06x0.597

103

Approach2:f(x)=0.04x0.62

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Approach1:f(x)=0.56x0.481

Approach1:f(x)=0.86x.463

Approach1:f(x)=0.89x0.462

Aproach1:f(x)=1.14x0.453

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

Phase IVb: (α,β) = (0.27, 1.1). Exponent (Theory):0.621

Phase IVb: (α, β) = (0.28, 0.7). Exponent (Theory): 0.577

Phase IVa: (α,β) = (0.30, 0.7). Exponent (Theory):0.500

Phase IVa:(α,β)=(0.40, 0.7).Exonent(Theory):0.500

105

★

Optimal Param Count

Approach2:f(x)=0.07x.609

Optimal Param Count (Approach 1)

Approach1:f(x)=3.64x0.397

Param Count

104

104

104

104

103

Approach2:f(x)=0.06x0.598

Optimal Param Count

Optimal Param Count (Approach 1)

103

大

★

Aproach2:fx)=0.05x0.603

Optimal Param Count

Optimal Param Count (Approach 1)

103

Approach2:f(x)=0.06x0.597

Optimal Param Count (Approach 1)

Optimal Param Count

103

Approach 1: f(x)=1.51x.439

Approach1:f(x)=0.61x0.478

Approach 1:f(x)=1.11x0.451

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

Phase Il:(α,β) = (0.50, 0.7). Exponent Theory):0.500

Phase Il:(α,β) = (0.60, 0.7).Exponent Theory):0.500

Phase la:(αβ)=(0.70,0.3）.Exponent (Theory):0.417

Phase la: (α, β) = (0.70, 0.4). Exponent (Theory): 0.417

10

★

Optimal ParamCount

Approach2:fx)=0.11x0.593

★

Optimal ParamCount

Approach2:f(x)=0.2x0.548

104

104

Optimal Param Count (Approach 1)

Optimal Param Count(Approach1)

Approach1:f(x)=3.86x0.395

Approach 1:f(x)=1.37x0.44

Count

104

104

Param

103

Approach2:f(x)=0.58x0.43

Optimal Param Count

103

Approach2:f(x)=0.53x0.445

Optimal Param Count

103

103

。

Approach1:f（x)=0.3x0.462

Optimal Param Count (Approach 1)

Aproach1:f(x)=1.32x0.396

Optimal Param Count (Approach 1)

107

108

109

1010

107

108

109

1010

107

108

109

0101

107

108

109

1010

Phasela:(αβ）=(0.70,0.5）.ExponentTheory):0.417

PhaselI: (α,βB) = (0.70, 0.6). Exonent (Theory): 0.462

Phasell(αβB）=0.700.8).Expoet Theory):000

★

Optimal Param Count

★

Optimal ParamCount

★

Optimal ParamCount

★

Optimal Param Count

Approach2:f（x)=0.49x0.459

Approach2:f(x)=0.33x0.492

Approac2:f(x)=0.26x0.518

Approach2:f（x)=0.17x0.551

104

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Param Count

Approach1:f(x)=0.67x0.443

104

Approach1:f(x)=0.51x0.467

104

Approach1:f(x)=0.96x0.48

104

Approach 1:f(x)=1.56x.43

103

103

103

103

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

★

Phasel:(αβ）(0.700.9).Exonent Theory):500

Optimal Param Count

105

Optimal Param Count

105

★

Phase Il:(α,β) = (0.70, 1.1).Exponent Theory):0.500

Optimal Param Count

105

★

Phase Il:(α, β) = (0.70 1.2). Exponent (Theory):0.500

Optimal Param Count

Approach2:f（x)=0.19x0.55

Approach2:f(x)=0.14x0.58

Approach2:f(x)=0.13x0.587

Approach2:f(x)=0.12x0.597

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

OptimalParamCount(Approach1）

Optimal Param Count (Approach 1)

Aproach1:f(x)=2.44x0.12

Approach 1:f(x)=3.03x0.406

Approach1:f(×x)=6.94x0.367

Approach1:f(x)=6.94x0.367

Param

Count

104

104

104

104

103

103

103

103

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

PhaselI:(α,β) = (0.80, 0.7). Exponent (Theory): 0.467

PhaselI: (α,βB) = (0.90, 0.7). Exponent (Theory): 0.437

Phasell:(α,βB) = (1.00, 0.7).Exponent (Theory): 0.412

PhaselIl (α,β)= (1.10, 0.7). Exponent (Theory): 0.389

★

Optimal Param Count

Approach2:f(x)=0.41x0.473

104

104

Optimal Param Count

104

OptimalParam Count(Approac1)

Approach1:f(x)=0.5x0.463

Approach1:f(x)=0.53x0.402

Param Count

103

103

103

Optimal Param Count

★

Optimal Param Count

103

Approach2:f(x)=0.52x0.44

Approach2:fx)=0.66x0.409

Optimal Param Count (Approach 1)

Optimal Param Count (Approach 1)

Approach1:f(x)=0.68x0.429

Approach1:f(（x)=0.31x0.449

107

108

109

1010

107

108

109

1010

107

108

109

1010

107

108

109

1010

Flops (6ND)

Flops (6ND)

Flops (6ND)

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the statements stated in the abstract and in the introduction are proven explicitly in the remainder of the paper - with most proven in the Appendix. We clearly state that we are studying a simple model with the exact description described in Section 1.1.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We clearly state in Section 1.1 that we are working on a simple quadratic power law random features model with assumptions on the data and targets. The algorithmic set-up follows that section. We indicate that we are only considering fixed stepsize SGD. All the results are proven in the Appendix. We also make note that we only prove our main result for α > 1 / 4 .

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems and results are proven in the Appendix. We make sure to clearly state any assumptions and limitations. For example, we do not prove the setting for α < 1 4 , but we clearly state this (see e.g., remark after Theorem 2.1).

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In each of the figures, we provide an explicit description of how the image was generated including the numerical set-up. In this case, the experiments are simple and they do not require significant coding or datasets. This is mainly a theory paper and the data was synthetically generated. The model has already been used before in other papers and the data comes from Gaussians. This is all clearly stated in our set-up, Section 1.1. We intend to release the code for the numerical computation of the Volterra equation and how it matches the loss curves under SGD.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments that require significant code. The model we analyze is a simple random features model with power law applied to synthetic data. As such, the code can be readily reproduced by following the set-up seen in the caption of the figures. We intend to release the code for the numerical computation of the Volterra equation and how it matches the loss curves under SGD.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental design, including the power law exponents, fixed stepsizes, choices of d and v , and the numerical simulations for solving Volterra equations are all written in the captions of the figures. We also intend to release the code for numerically computing the Volterra equation.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide (and explain) in the figures/captions the error bars across multiple runs of SGD. We record how we generate the empirical compute-optimal exponents using statistical tools that were first deployed in other papers such as [24]. We are careful to explain when and why the theory deviates from the numerical simulations. Often this is due to finite d and v effects and the slow behavior of the theory to the asymptotics.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As this paper is about compute-optimal curves, we provide details on the exact number of flops required to perform the experiments. Since the model we study is a simple least squares model, the compute resources are also well known in by the community.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics. We have made our utmost attempt to adhere to the guidelines provided by NeurIPS. We do not use any human subjects nor any datasets. We did our best to cite all the relevent related work. Given that our work is in the foundational research, it is difficult to mitigate all the risks as the downstream effects of theory are long, but we have done our best. The model is completely synthetic using the standard SGD algorithm; thus we don't, to the best of our knowledge, anticipate any risks.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work presented is purely foundational research and is not directly tied to any particular application. We study a simple random features model with power law data and target and we solve the model using a common algorithm SGD. Given the theoretical nature of this work, we do not anticipate any direct ethical and societal issues. See our Broader Impact Statement in the appendix.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks. The work is purely theoretical and uses only synthetic data generated from a normal distribution. The models employed are standard statistical model (e.g., least squares) which are textbook learning problems.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We acknowledge via citations that the model we study was introduced before by others (e.g., Maloney, Roberts, and Sully paper). We are not using any datasets or other assets beyond numpy and thus do not need to name any license or cite any dataset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not intend to release any new assets. The work is a theoretical analysis of SGD on a random features model.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. The work is purely theoretical on a simple model.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The work does not involve crowdsourcing nor research with human subjects. The data used in this work is generated synthetically.

Peer reviews:
comments: SUMMARY:
The authors consider a simple scaling model and derive scaling laws for one pass SGD in an asymptotic limit. From the scaling laws they identify several phases and subphases where certain components of the loss dominate and the compute-optimal model parameter count is affected. The loss components are related to the model capacity, SGD noise, and feature embedding. Detailed calculations are performed to completely characterize the behaviors in different regimes and are accompanied by simulations to give supporting evidence. Interestingly large regions of the phase space result in a "universal scaling" independent of the problem parameters.

STRENGTHS:
Understanding scaling behavior of learning models is of great theoretical and practical importance. This work provides a very thorough and comprehensive analysis of the simplest, non-trivial model. Despite the simplicity of the model a rich variety of behavior is observed which requires effort to analyze and catalogue. The observations of universality and effects of finite size are interesting and potentially relevant to more realistic settings.

WEAKNESSES:
1. It is unclear how novel the mathematical techniques are and how general purpose they are, some discussion would be helpful.
2. It is unclear how related the simple model is to observations about more realistic models so some more commentary could be useful.

QUESTIONS:
1. Is there potential for a more general explanation for universality since the same scaling appears to hold in the Chinchilla scaling law? 
2. Can any of the techniques be extended to analyze more complex models?
3. Are there any lessons for practical LLM scaling?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the Power Law Random Feature (PLRF) that depends on three parameters: data complexity, target complexity, and model parameter count. They derive a deterministic closed expression for the dynamics of SGD using a clever mapping to Volterra equations. They are able to determine the compute-optimal frontier for different learning scenarios. The theoretical claims are supported by extensive numerical simulations.\n\nSTRENGTHS:\nThe main strength of this paper stands clearly in being both theoretically and numerically exhaustive. The numerical illustrations and details backing up the theoretical claims are explored in detail both in the main and appendices. Moreover, the questions addressed are of great interest to the theoretical machine learning community.\n\nWEAKNESSES:\nThe strong weakness of this submission is the presentation. It is challenging for a non-expert reader to navigate the related literature with only thirtheen references and eleven lines devoted to the related works.\n\nQUESTIONS:\nAs mentioned above, a large part of my concerns resides in the presentation of the results and the framing of the present submission in terms of the related literature. See below for some examples of needed citations/explanations. \n\n\n- Deterministic equivalent at page 2, what is it and how is it used in machine learning theory? Cite reference for this, e.g., [1].\n- What is a \"neural scaling model\". This appears in the first line of the abstract. \n- When introducing PLRF there is no mention to what classical RF looks like [2]. Many works could be cited that use also Random Matrix Theory tools to connect at the previously described deterministic equivalent, e.g. [3] among many others. \n-  [4,5] are two papers that drew phase diagrams for the training dynamics of SGD. Although they do not seem related to the present submission, they deal with the optimal tuning of SGD from a theoretical point of view. \n\nThe setting presented in the manuscript has many limitations-- which is completely acceptable for a primarily theoretical work. However, the technical challenges that the authors would face if they were to lift them are never discussed. For example, square loss, lack of non-linearity in the PLRF, deterministic teacher vector $b$, need for $v>d$, etc. All these assumptions are reasonable, but must be compared to related works on the subject and why it is difficult to lift them. \n\n### Minor points\n- Explain in deeper detail footnote number 6. Why is this the case? \n- The mapping to Volterra equations is nice and I believe it would deserve more space in the main body. \n- Different subfigures might have different aspect ratios (see e.g. Figures 2 and 3). \n### References \n\n- [1] Random Matrix Methods for Machine Learning. Coulliet & Liao 2022, Cambridge University Press. \n- [2] Random Features for Large-Scale Kernel Machines. Rahimi & Recht. NeurIPS 2007. \n- [3] The generalization error of random features regression: Precise asymptotics and double descent curve. Mei & Montanari. Communications on Pure and Applied Mathematics 2021. \n- [4] Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs. Arnaboldi et al. ICML 2024.\n- [5] Dissecting the effects of SGD noise in distinct regimes of deep learning. Sclocchi et al. ICML 2023.\n\nLIMITATIONS:\nThe suggestions for improving the description of the theoretical limitations are given above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a linear random feature model trained on power law data under online SGD. By using a Volterra approach and leveraging deterministic equivalence, they characterize the loss in the high dimensional limit. From this, they extract scaling laws for this model, that determine compute optimality. The show that depending on the spectral decay of the covariates and the task vector, SGD noise as well as finite width effects can effect the scaling of the loss curves with time. A consequence of this is that the optimal allocation of compute can change. By systematically studying finite width and time effects, the authors characterize the scaling laws one should expect in this model.\n\nSTRENGTHS:\nThe paper investigates an important problem. The reviewer has checked much of the math in the (extensive) appendix. The derivations are technically sound. The conclusions of this paper certainly add meaningfully to the present machine learning literature and our understanding of the role of the optimizer on neural scaling laws. The experiments are beautifully done and add meaningfully to the presentation of the paper. I commend the authors on the care put into the experiments and encourage them to make the code public.\n\nWEAKNESSES:\nThere are two primary weakness:\n\n1. The paucity of citations in this work is stunning\n\nMany authors have studied random feature models before and concurrently with the Maloney paper. The set of random feature model papers that deserve a citation are many, and I leave it to the authors to do a thorough review. Certainly Mei and Montanari https://arxiv.org/abs/1908.05355, as well as Bach's paper that also derives the same scalings as Maloney: https://epubs.siam.org/doi/abs/10.1137/23M1558781\nBefore all of this, Hastie et al in their original paper studied random projections on structured data as well:\nhttps://arxiv.org/abs/1903.08560\nAlthough not explicitly calculated, the results of Mel and Pennington give as a special case the generalization error formula in Maloney et al  and was published much earlier. They just didn't calculate the scaling law exponents. \nhttps://openreview.net/forum?id=JfaWawZ8BmX\nSimilar models were studied and exactly solved by Loureiro and colleagues in several papers, for example\nhttps://arxiv.org/abs/2102.08127\n\nMoreover, although I have not read all of the Bordelon paper due to lack of familiarity with DMFT, I can see that they explicitly treat the case of SGD noise. Given the substantial overlap in model and problem studied I think it is worth clarifying sharply what this paper puts forward beyond the initial studies of that one.\n\nI think this work is important. Putting it in the context of works that came before is also important. The authors are hurting the reception of their paper in the broader community by citing so sparsely. \n\n2. Although the readability of this paper in the main text is reasonable, the accessibility of the appendix is quite poor. \n\nThe first section deriving the Volterra equation is written quite clearly and accessibly. From there forward, it becomes increasingly impossible to read. \n\nBy hopping around the theorems and propositions one can eventually recover your bounds on the forcing function and the kernel function. I strongly recommend restructuring the order of your theorems and propositions. I also strongly encourage an \"overview of proofs\" to start the appendix off so that a reader can navigate the dense math more carefully. \n\nThe contour presented in Figure 7 comes out of nowhere. Even after reading most of this paper several times over I still have no idea how the authors arrived at it. I've managed to reproduce most of the results and I don't need this type of analytic contour argument at all. \n\n\nIf these two concerns can be addressed, I will be happy to raise my score. Specifically, I would like the authors to list all of the relevant papers that they will cite in the revised draft, and I would strongly encourage them to follow through.\n\nQUESTIONS:\n1. It is not obvious without going deep into the appendix why these terms are called $K_{pp}$, $F_{ac}$, etc. It would be much better if you explained this sooner rather than later. \n\n2. If I am not mistaken, the limit that isn't high-dimensional, $2 \\alpha < 1$, would never be encountered for a kernel trained on natural data.  Although there do indeed seem to be interesting phenomena below that line, I am wondering why the authors have decided to so carefully study this? Is is purely out of a desire to characterize the entire $\\alpha, \\beta$ plane? \n\n3. Further, $\\beta > 1$ corresponds to tasks that are ``in the Hilbert space'' of the original $v$ dimensional space. Again, this is never the case for natural data, where the spectral decay is much much slower. If anything this paper seems to tell us that SGD has very little effect on the scaling laws on realistic data.  Please let me know if I am incorrect in this characterization. Otherwise, given the discussion of scaling laws, there will likely be some practitioners that read this paper who are confused about how to interpret $\\alpha$ and $\\beta$. I think the authors would do well to state clearly the relationship between the regions in the $\\alpha$ and $\\beta$ plane and the values one would expect in real datasets.\n\n4. Why are you using the term `SGD frustrated'? In the statistical physics literature this is taken to mean something else entirely. \n\n5. As a smaller comment, the notation for the exponents is quite different from other works. In linear regression its standard to report things in terms of the \"source and capacity\" exponents, where $\\alpha$ is the decay exponent of the spectrum. Of course the authors don't need to change their notation but a table comparing notations with other works would be very useful and go a long way.\n\nLIMITATIONS:\nGiven that this is a toy model of neural scaling laws, I do not expect major societal implications. However, the principled understanding of such scaling properties may well have important impacts on the future of the field.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the generalization error dynamics of one-pass SGD in a sketched linear regression setting, where the data and target signal are distributed according to certain power laws, and SGD optimizes a linear model on top of a Gaussian random projection. Using random matrix theoretical tools, the authors precisely computed the asymptotic generalization error of the SGD iterates in high dimensions; this reveals various scaling laws and allows the authors to characterize the compute (flop) optimal frontier.\n\nSTRENGTHS:\nThis is an emergency review, so I shall simply state that this is probably the most interesting submission in my batch and should be accepted.\n\nWEAKNESSES:\nMy main concern is that the authors do not adequately discuss the overlap with prior results.\n\n1. Generalization error of the random features model under various source and capacity conditions has been extensively studied. For instance, (Rudi and Rosasco 2016) derived scaling laws of random features regression with respect to the model width and number of training data, and this result is later extended to SGD in (Carratino et al. 2018), taking into account the scaling of iteration number. The authors should explain the similarity and differences in the findings to highlight the advantage of a precise analysis.  \n* (Rudi and Rosasco 2016) *Generalization Properties of Learning with Random Features.*  \n* (Carratino et al. 2018) *Learning with SGD and Random Features.*  \n\n2. If we do not take optimization into account, then neural scaling laws in the form of $(\\text{data}^{-\\beta} + \\text{param}^{-\\gamma})$ have been established in many prior works on nonparametric regression (Schmidt-Hieber 2017) (Suzuki 2018) -- these are rigorous versions of the hand-wavy arguments in (Bahri et al. 2021) which the authors cited. \nIf we interpret the number of training data to be the same as the iteration number, can we obtain similar plots on the compute-optimal front as reported in Figure 2(a)(b)?  \n* (Schmidt-Hieber 2017) *Nonparametric regression using deep neural networks with ReLU activation function.*  \n* (Suzuki 2018) *Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality.*  \n\nSome additional questions:\n\n1. In light of empirical findings that training small models for a longer time can be beneficial, can the authors comment on the possibility of extending this analysis to the multiple-pass setting? In the kernel regression literature, it is known that multi-pass SGD has statistical superiority (Pillaud-Vivien et al. 2018).  \n* (Pillaud-Vivien et al. 2018) *Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes.*\n\n2. Does the sample size scaling match the minimax optimal rate in (Caponnetto and de Vito 2007) in certain regimes?  \n* (Caponnetto and de Vito 2007) *Optimal rates for regularized least-squares algorithm*. \n\n3. What are the technical challenges to show the asymptotic equivalence between the SGD dynamics and the deterministic equivalent description that the authors analyze?\n\n4. Can the authors comment on the restriction to proportional $v,d$? Why is there a sharp transition at $2\\alpha=1$ that decides the scaling of $d$?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied a solvable neural scaling law model (the power-law random features, PLRF) that involve three parameters (data complexity: $\\alpha$, target complexity: $\\beta$ and the number of parameters: $d$). The PLRF model here is trained by applying the stochastic gradient descent (SGD) algorithm to the mean-squared loss. For different choices of the parameters $\\alpha$ and $\\beta$, the optimal number of parameters $d$ is solved by minimizing the expected loss. A corresponding phase diagram is drawn with respect to $(\\alpha,\\beta)$. An extensive set of numerical experiments is also conducted to justify the main theoretical results.\n\nSTRENGTHS:\n1. This is a technically solid paper with rigorous mathematical proof. For each of the transition boundaries in the phase diagram, an intuitive explanation is provided to help the readers understand the key idea behind. \n2. An extensive set of numerical experiments are included to justify the theoretical results.\n\nWEAKNESSES:\nOne possible drawback of the current version of this paper is that the list of references is incomplete. It seems to the reviewer that the discussion on related work is incomplete and ignores many recent and concurrent work on the theoretical aspects of neural scaling law. See for instance [1,2].\n\nQUESTIONS:\nOverall, the reviewer enjoyed reading both the proof and the experiments presented in the paper a lot. One possible broad question that the reviewer is interested in is how the theoretical results presented here can be linked to previous work on the learning of timescales in two-layer neural networks [3] (In general, the key question is also to address which features get learned at which stage).\n\nLIMITATIONS:\nIt seems that the authors didn't perform experiments on large-scale language models (LLMs). One possible way to make the work more impactful is to do some empirical investigation on LLMs to see if certain parts of the theoretical results still hold when the model complexity drastically increases. \n\nReferences:\n\n[1] Jain, A., Montanari, A., & Sasoglu, E. (2024). Scaling laws for learning with real and surrogate data. arXiv preprint arXiv:2402.04376.\n\n[2] Lin, L., Wu, J., Kakade, S. M., Bartlett, P. L., & Lee, J. D. (2024). Scaling Laws in Linear Regression: Compute, Parameters, and Data. arXiv preprint arXiv:2406.08466.\n\n[3] Berthier, R., Montanari, A., & Zhou, K. (2023). Learning time-scales in two-layers neural networks. arXiv preprint arXiv:2303.00055.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}