Title: Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf’s Law
Venue: NeurIPS 2025
Accepted: yes
Keywords: scaling law

optimization

gradient descent

sign descent

bigram model

high dimension

power-law

zipf's law
Abstract: Recent works have highlighted the optimization difficulties encountered by gradient descent in training the first and last layer of transformer-based language models, which are overcome by optimizers such as Adam. The problem appears linked to the heavy-tailed distribution of words in text data, where the frequency of the $k$th most frequent word $\pi_k$ is proportional to $1/k$, following Zipf's law. To better understand the impact of the data distribution on training performance, we study a linear bigram model for next-token prediction when the tokens follow a power-law $\pi_k \propto 1/k^\alpha$ parameterized by the exponent $\alpha$. We derive optimization scaling laws for deterministic gradient descent and sign descent as a proxy for Adam as a function of the power $\alpha \geq 0$. This setting differs from existing theoretical investigations in scaling laws which assume that the eigenvalues of the data decay as a power with power $\alpha > 1$. This assumption effectively makes the problem "finite dimensional" as most of the loss comes from a few of the largest eigencomponents. In comparison, we show that the problem is more difficult when the data have heavier tails. The case $\alpha = 1$ as found in text is ``worst-case'' for gradient descent, in that the number of iterations required to reach a small relative error scales almost linearly with dimension. While the performance of sign descent also depends on the dimension, for Zipf-distributed data the number of iterations scales only with the square-root of the dimension, leading to a large improvement over gradient descent for large vocabularies.

Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law

Abstract

Recent works have highlighted optimization difficulties faced by gradient descent in training the first and last layers of transformer-based language models, which are overcome by optimizers such as Adam. These works suggest that the difficulty is linked to the heavy-tailed distribution of words in text data, where the frequency of the k th most frequent word π k is proportional to 1 /k , following Zipf's law. To better understand the impact of the data distribution on training performance, we study a linear bigram model for next-token prediction when the tokens follow a power law π k ∝ 1 /k α parameterized by the exponent α > 0 . We derive optimization scaling laws for deterministic gradient descent and sign descent as a proxy for Adam as a function of the exponent α . Existing theoretical investigations in scaling laws assume that the eigenvalues of the data decay as a power law with exponent α > 1 . This assumption effectively makes the problem 'finite dimensional' as most of the loss comes from a few of the largest eigencomponents. In comparison, we show that the problem is more difficult when the data have heavier tails. The case α = 1 as found in language is 'worst-case' for gradient descent, in that the number of iterations required to reach a small relative error scales almost linearly with dimension. While the performance of sign descent also depends on the dimension, for Zipf-distributed data the number of iterations scales only with the square-root of the dimension, leading to a large improvement for large vocabularies.

1 Introduction

Recent works have shown that one of the primary benefits of Adam (Kingma and Ba, 2015) in training transformed-based language models (Vaswani et al., 2017) lies in how it handles the first and last layers (Zhang et al., 2025; Zhao et al., 2025). For language models, the input and output dimensions correspond to distinct words in the vocabulary, where the k th most frequent word has frequency π k ∝ 1 /k following Zipf's law (Piantadosi, 2014). Kunstner et al. (2024) provide evidence that this heavy-tailed distribution leads to optimization difficulties for gradient descent that Adam is able to overcome. They argue that Zipf's law is 'worst-case' in that it combines a large imbalance in frequencies, while decaying slowly enough that most samples come from the tail.

Our objective is to formalize this empirical observation, and to describe the impact of the heavytailedness of the data distribution on the convergence of gradient descent (GD) and sign descent (SD) as a proxy for Adam (Tieleman and Hinton, 2012; Bernstein et al., 2018; Balles et al., 2020; Chen et al., 2023). We consider a linear bigram model for next-token prediction trained with the square loss, where the token frequencies π k follow a power law π k ∝ 1 /k α with exponent α > 0 . While this problem could be solved directly rather than with iterative methods, it is a good starting point for the theoretical investigation of optimization dynamics. Despite its apparent simplicity, this model already reproduces the observation that GD performs poorly on Zipf-distributed data (see Fig. 1). The behavior of gradient and sign descent are also not well described by current results, see Section 1.2.

Our approach is inspired by the line of work on theoretical scaling laws, also known as asymptotic convergence as the dimensionality grows (e.g., Caponnetto and De Vito, 2007; Advani et al., 2020;

Francis Bach

0

250

750

1000

Steps

0

.

0

0

.

5

1

.

0

Relative error

π

k

∝

1

k

1

/

2

0

250

750

1000

Steps

π

k

∝

1

k

0

250

750

1000

Steps

π

k

∝

1

k

2

d

= 10

6

10

5

10

4

10

3

10

2

Berthier et al., 2020; Bahri et al., 2021; Cui et al., 2021; Maloney et al., 2022; Paquette et al., 2024). Instead of analyzing the generalization error of online gradient descent as the dimension of the model and sample size grow, we study the convergence rate of GD as the dimension grows. Spectral assumptions on the eigenvalues of the Hessian following a power-law are common in the literature, which in our case correspond to assumptions on word frequencies. But these works focus on power-laws that are not 'too' heavy-tailed, 1 /k α with α > 1 , which lead to sublinear rates independent of dimension. In contrast, we focus on the case α ≤ 1 where it becomes impossible to make progress unless the number of iterations grows with the dimension of the problem. Our contributions are as follows.

We propose a simplified model of the word frequencies that leads to tractable dynamics for GD and SD, that captures the difference in their training performance experimentally (Fig. 2).

We derive scaling laws for GD and SD in this model as a function of α > 0 , covering power-laws that decrease as slow or slower than Zipf's law ( α ≤ 1 ). This setting is often ignored in existing analyses and leads to qualitatively different results, with scaling laws that are not power-laws and require the number of iterations t to grow with d .

For GD on Zipf-distributed data ( α = 1 ) the number of iterations required to reach a small relative error scales almost linearly with dimension, t ∼ d . This setting is 'worst-case' in that the case α < 1 results in a better scaling of t ∼ d α , while α > 1 does not require t to scale with d .

In comparison, SD under Zipf-distributed data only requires t ∼ √ d , provably confirming its benefits over GD on a language task, but not in all settings as SD exhibits worse scaling if α > 1 .

1.1 Overview of the results

We consider a simplified language modeling tasks, given a vocabulary of d words, we train a linear bigram model with square loss to predict the next word y ∈ [ d ] given the current word x ∈ [ d ] , represented as one-hot vectors x , y ∈ { 0 , 1 } d . The dynamics of the problem depends on the word frequencies π k and conditional frequencies π k | j , which we assume follow a power law 1 /k α , formalized in Section 2. The scaling we analyze is how the loss changes as the dimension d and number of iterations t increases, depending on α . For GD on α > 1 , we recover the result that the loss after t steps, L d ( t ) , follows the power law L d ( t ) ∼ c 1 t -p + c 2 as d →∞ for some power p and constants c 1 , c 2 . Equivalently, we could write the relative optimality gap as

where ∼ denotes asymptotic equivalence, r d ( t ) ∼ t -p means lim d →∞ r d ( t ) /t -p = 1 . This rate is independent of d , but specific to GD with α > 1 . Our results show that in other settings, the number of iteration t needs to scale with d to achieve an ε -relative optimality gap, r d ( t ) ∼ ε . The following is a simplification of our main results, made formal in Theorems 3.1 and 4.5.

Informal theorem. To reach an ε -relative optimality gap, the number of iterations t should scale as

where t ≍ f (equivalently, t = Θ( f ) ) hides constants and factors that depend on ϵ but not on d .

10

0

10

2

10

4

Optimization horizon

T

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Relative error

Real data

SD

asym.

asym.

10

3

10

4

Vocab. size

d

10

0

10

1

10

2

10

3

10

4

Time to

ε

Gradient descent

10

3

10

4

Vocab. size

d

10

0

10

1

10

2

10

3

10

4

Time to

/epsilon1

Sign descent

ε

=

3

4

1

2

1

4

1

16

For language and Zipf-distributed data ( α = 1 ), our scaling predicts that the number of iterations required to reach ε relative error scales almost linearly with d for GD if ε is small while it scales as d 1 / 2 for SD. For common vocabulary sizes of d = 10 4 tokens, this leads to a 100 -times speedup. Zipf-distributed data is also 'worst-case' for GD, as other values of α lead to better scaling in d .

We recover the power-law scaling for GD when α > 1 , but obtain different functional forms for other settings (Theorems 3.1 and 4.5). For α = 1 , the relative optimality gap behaves as

We confirm these predictions experimentally using real data on OpenWebText, shown in Fig. 2.

1.2 Related work

Convergence of Adam and sign descent. The benefit of Adam has been argued to stem from its similarity to sign descent, in that the updates are uniform across coordinates (Bernstein et al., 2018; Balles et al., 2020; Chen et al., 2023). This 'scale-freeness' can reduce the dependence on the condition number (Zhuang et al., 2022), but this does not imply SD outperforms GD as known convergence rates for sign-like methods instead depend on the dimension d (e.g., Safaryan and Richtárik, 2021; Das et al., 2024; Liu et al., 2025). In the bigram problem with Zipf-distributed data, the dimension grows faster than the condition number, leading to worse guarantees for SD. We compare our asymptotic analysis to existing rates in Appendix B.

SDEapproximations of sign methods. Scaling laws have been derived for online sign-like algorithms through stochastic differential equations (Ma et al., 2021; Malladi et al., 2022; Xiao et al., 2024; Compagnoni et al., 2025). The focus of these works is on the scaling of the step-size with batch size and the asymptotic stationary distribution of the algorithm which controls the generalization error. As noise is not necessary to reproduce the performance gap between GD and Adam (Kunstner et al., 2023), we instead focus on the impact of heavy-tailed data on the deterministic dynamics.

Scaling laws and asymptotic results. Empirical scaling laws have been developed to extrapolate the performance of deep networks at scale and how to balance compute across model and data sizes (Rosenfeld et al., 2020; Kaplan et al., 2020; Hoffmann et al., 2022). Many works have contributed to the theoretical understanding of this scaling behavior through high dimensional analyses and random matrix theory (Advani et al., 2020; Bahri et al., 2021; Maloney et al., 2022; Bordelon et al., 2024a; Lin et al., 2024; Paquette et al., 2024), classical source/capacity conditions from learning theory (Caponnetto and De Vito, 2007; Berthier et al., 2020; Cui et al., 2021), also used in an optimization context (Velikanov and Yarotsky, 2024). However, those works study problems where the spectrum decays fast, and does not cover case α ≤ 1 . This regime, covering Zipf's law, might be more relevant when considering scaling the vocabulary size, as in the work of Gowda and May (2020) and Tao et al. (2024). While they hypothesize that larger vocabularies might lead to worse performance due to overfitting, as larger vocabularies implies fewer examples per word in addition to more compute per step, we show that larger vocabulary size might also need more steps to get the training error down. Closest to our work is perhaps the blog post of Bulatov (2023), which argues that

the loss under GD should approximately behave as -log( t/d ) on a problem matching our setting with α = 1 . Our work provides a formal justification for this scaling.

2 Problem setup

In this section, we present the problem setting, a linear bigram model with square loss, the modeling assumptions used to make the problem tractable and the approach we use to derive our results.

Problem 2.1 (Linear bigram model) . Let x i , y i ∈ { 0 , 1 } d for i = 1 , . . . , n be one-hot encodings from d classes (or tokens), with their concatenation X , Y ∈ { 0 , 1 } n × d , fit with a linear model,

We define π k and π k | j as the frequencies and conditional frequency statistics of the data,

The analysis of GD on quadratics typically uses an eigenvalue decomposition. Consider a d -dimensional quadratic f ( x ) = 1 2 ( x -x ∗ ) ⊤ A ( x -x ∗ ) with minimizer x ∗ where the eigenvalues/vectors pairs of A are ( λ i , v i ) ∈ R × R d for i = 1 , . . . , d . The dynamics of GD with step-size η , x t +1 = x t -η A ( x t -x ∗ ) , decompose in terms of the distance along eigenvectors, δ i ( t ) = ⟨ v i , x t -x ∗ ⟩ , as

For the d 2 -dimensional bigram model (2.1), the dynamics depend on the frequencies π k and π k | j .

Proposition 2.2. The dynamics of gradient descent on Problem 2.1 initialized at W =0 are described by the eigenvalues and distances to solution λ ij , δ ij (0) for i, j = 1 , . . . , d , using L d ( t ) = L d ( W t ) ,

where L d ∗ = min L d , as the eigenvalues and distances to solution are λ ij = π i and δ ij (0) 2 = π 2 j | i .

Proof sketch. The Hessian of L d is diagonal, with diagonal blocks X ⊤ X /n = Diag([ π 1 , . . . , π d ]) repeated d times. The eigenvectors are the standard basis with eigenvalues λ ij = π i . The solution is at W ∗ = ( X ⊤ X ) -1 X ⊤ Y , where w ∗ ij = π j | i as [ X ⊤ Y /n ] ij = π j | i π i , giving δ ij (0) 2 = π 2 j | i .

2.1 Modeling assumptions

Getting an interpretable form of the rate in Eq. (1) requires assumptions on the values of λ i and δ i . Assuming µ ≤ λ i ≤ L leads to the typical smooth (strongly-)convex rates (e.g., Nesterov, 2018),

While valid, these worst-case bounds are too coarse to capture the richness of the behavior of GD and becomes vacuous if µ → 0 or ∑ d i =1 δ i ( w 0 ) 2 →∞ as d →∞ . To obtain fine-grained results, we assume that the frequencies π k and conditional frequencies π k | j follow power laws.

Assumption 2.3 (Heavy-tailed data) . We assume that the frequencies and conditional frequencies follow a frequency-rank power law with exponent α > 0 . That is, assuming the frequencies are sorted ( π k ≥ π k +1 ) and defining the sorting permutations ρ j such that π ρ j ( k ) | j ≥ π ρ j ( k +1) | j , where by π k ∝ 1 /k α we mean that the frequencies are normalized, π k = 1 /zk α for z = ∑ d k =1 1 /k α .

This assumption may appear strong, as it would be satisfied for example if the words were sampled i.i.d. with frequencies π 1 , . . . , π d as π k | j = π k . But it does not require that all conditional distributions

1

10

10

3

10

4

Rank of word

k

10

-

5

10

-

3

10

-

1

Frequency

Word frequencies

π

k

data

∝

1

/k

1

10

10

3

10

4

Rank of next word

k

Conditional frequencies

π

ρ

j

(

k

)

|

j

median

∝

1

/k

5-95%

10-90%

25-75%

be the same. The distribution of the next word after j can depend on j . This assumption merely asks that, once sorted, the next-word frequencies follow a power law with the same exponent. Some distributions might deviate from this trend if a token can only logically be followed by specific tokens, or if the word being conditioned on is rare and our dataset is relatively small. 1 While we do not expect the assumption to be exactly satisfied in practice, it appears to be a reasonable high-level approximation of real-world data, as shown in Fig. 3 in comparison to the empirical distributions on OpenWebText, and leads to accurate predictions as shown in Fig. 2.

Relation to other spectral conditions. Even though Problem 2.1 is d 2 -dimensional, the dynamics of GD are equivalent to those run on a d -dimensional problem as Proposition 2.2 can be rewritten as L d ( t ) -L d ∗ = 1 2 ∑ d i =1 π i (1 -ηπ i ) 2 t ∆ 2 i for ∆ 2 i = ∑ d j =1 δ ij (0) 2 . Many works have considered decay conditions on the eigenvalues and distances to the solution, π k ∝ k -a and ∆ 2 k ∝ k -b , similar to the source/capacity conditions (Caponnetto and De Vito, 2007). However, their focus is typically on a fast decay, a + b > 1 , which leads dimension-independent power-laws (see e.g., Paquette et al., 2024, and references therein). While Assumption 2.3 is a special case corresponding to ( a, b ) = ( α, 0) , we study the case α ≤ 1 to understand the behavior of optimizers on heavy-tailed data.

2.2 Strategy for the analysis

Our goal is to derive scaling laws for the loss of Problem 2.1 in d dimensions after t steps, L d ( t ) , as d →∞ . Such scaling laws can be interpreted as approximating the convergence rate for large d , or serve as a guide on how to scale the hyperparameters of the optimizer as we increase the vocabulary size. Formally, we compute the asymptotic limit of the rate r ( t ) at which the relative loss decreases,

Works on scaling laws typically model the absolute value of the loss. This approach degenerates when the loss at initialization vanishes or diverges as d →∞ which happens when α ≤ 1 . Considering the relative decrease circumvents the issue, as also noted by Bulatov (2023) and Tao et al. (2024).

Another potential degeneracy is the scaling of time. If the problem becomes more difficult as d grows, it might be impossible to make progress in finite time. To take a concrete example, suppose that L d ∗ = 0 and L d ( t ) = r d ( t ) L d (0) with r d ( t ) = (1 -1 /d ) t . If we take the limit as d →∞ for a fixed t , we obtain lim d →∞ (1 -1 /d ) t = 1 . The rate no longer depends on t , and we cannot make progress unless t grows with d . If we instead introduce a rescaled time variable τ and scale t d ( τ ) = τd , we recover a linear rate in the rescaled time τ as (1 -1 /d ) τd d ∼ e -τ . A similar issue arises in random matrix theory, where the dimensions of the matrix are taken to grow jointly with a fixed ratio to avoid degenerate solutions (Potters and Bouchaud, 2020). It can be verified that t d ( τ ) = τd is the 'right' scaling, as the limit r d ( t d ( τ )) degenerates otherwise. Using f ( x ) ≪ g ( x ) for lim x →∞ f ( x ) /g ( x ) = 0 , we have r d ( t d ( τ )) d ∼ 1 if t d ( τ ) ≪ d and r d ( t d ( τ )) d ∼ 0 if t d ( τ ) ≫ d ; we either make no progress or solve the problem instantly. Our results are derived by taking the finite dimensional rate r d ( t ) with a scaling t d such that the asymptotic rate r ( τ ) is well-defined in terms of the rescaled time τ ,

10

-

2

10

τ

=

T/d

α

10

-

4

10

-

3

10

-

2

10

-

1

10

0

Relative error

α

= 0

.

5

1

-

α

α

e

-

τ

1+

τ

10

-

2

10

τ

:

t

=

d

τ

α

= 1

1

-

τ

1

10

2

10

6

10

8

τ

=

t

α

= 2

d

= 10

2

d

= 10

4

d

= 10

6

d

= 10

8

C

τ

1

-

1

α

3 Scaling laws for gradient descent

We study the relative error of GD with the step-size η = 1 /π 1 , the inverse of the largest eigenvalue,

Before diving into the main result, we provide some intuition on those dynamics. The performance of GD depends on the speed of convergence for each word, (1 -π k /π 1 ) , and the proportion of the error coming to that word, π k . The parameter α controls both. If we increase α , the frequencies π k ∝ 1 /k α decay faster. Low-frequency words converge more slowly, (1 -π k /π 1 ) = (1 -1 /k α ) , but contribute less to the error, π k = 1 /zk α where the normalization term is z = ∑ d k =1 1 /k α .

For α > 1 , the error is dominated by high-frequency words, which converge quickly. The error attributed to the first K words, ∑ K k =1 π k , is a constant approximation of the total. Increasing the vocabulary size d does not make the problem much harder as low-frequency words contribute little. For α < 1 , the error associated with the first K words vanishes if K is fixed and d grows, indicating that most of the error comes from low-frequency words. However, their convergence speed improves as α decreases, with the extreme case of uniform frequencies at α = 0 , making the problem easier. The case α = 1 of Zipfian data exhibits the worst of both settings. The decay is slow enough that the contribution of low-frequency words is significant, but fast enough that their convergence is slow.

The following theorem formalizes these intuitions.

Theorem 3.1 (Scaling for gradient descent) . On the bigram problem (Prob. 2.1) with distributions following a power law with exponent α > 0 (Assumption 2.3), gradient descent with a step-size 1 /π 1 , with time scaling t d ( τ ) has the following asymptotic convergence rate (Eq. (2) ).

where Γ is the Gamma function, E is the generalized exponential integral, B is the Beta function, and ζ is the zeta function (DLMF, §5.2, §5.12 §8.19 §25.2), and C = Γ ( 1 -1 α ) /αζ ( α ) .

Proof. We sketch the proof for α = 1 and leave the remaining cases to Appendix C. Under Eq. (1) and Assumption 2.3 the dynamics of the normalized loss r d ( t ) (Eq. (3)) reduce to

where H d,α = ∑ d k =1 k -α . To simplify the analysis, we use the integral form of the sum as we can use Laplace's method to estimate its behavior for large d , see Appendix C for a formal justification;

0

5

10

15

0

1.0

|

δ

i

(

t

)

|

Individual directions

δ

1

δ

2

δ

3

δ

4

Assumption

0

5

10

15

0

1.0

Relative error

Total error (

d

= 4

)

Exact

Assumption

0

500

1000

1500

10

-

5

1

Relative error

Budget

T

vs. step-size

η

η

= 2

-

6

η

= 2

-

7

η

= 2

-

8

η

= 2

-

9

η

= 2

-

10

t

t

t

after the change of variable k = d z or z = log( k ) / log( d ) . As the normalizer H d, 1 d ∼ log( d ) , we only need to consider the limit of the integral. Taking d →∞ with t fixed, the integral converges to 1 and we make no progress, regardless of t . To make progress, t needs to scale as 2 t = d τ for τ ∈ [0 , 1] ,

For a fixed τ and as d →∞ , the integrand converges to 0 if z < τ and 1 if z > τ . As it is bounded by a constant, we can exchange limits and integrals by the dominated convergence theorem to obtain

The results highlight different regimes depending on α . The number of iterations needs to scale with dimension if the data decays as slow as or slower than Zipf's law ( α ≤ 1 ) whereas it is not necessary for lighter-tailed data ( α > 1 ). We show in Fig. 4 that the dynamics on data satisfying Assumption 2.3 converge to the asymptotic rates of Theorem 3.1 and are accurate even for common vocabulary sizes.

4 Scaling laws for sign descent

The dynamics of SD differ qualitatively from those of GD as they take a uniform update in all directions, regardless of the magnitude of the derivatives. As we will see, this makes it better suited for the linear bigram model with Zipf-distributed data, but comes with additional challenges. For SD, we need to address two issues. First, the sign descent update is not linear; we need an alternative to the closed form solution of GD in Eq. (1). Second, SD does not converge with a fixed step-size; we need to scale step-size as a function of the iteration budget and dimension.

If run with a constant step-size, the update of sign descent with a step-size of η is

As the Hessian of Problem 2.1 is diagonal, the update applies independently to each parameter. Letting δ ij ( t ) be the distance along the ( i, j ) th parameter at step t ,

The difficulty in the analysis comes from the fact that | δ ij ( t ) | does not converge to 0 . Instead, | δ ij ( t ) | will oscillate between some c ∈ (0 , η ) and c -η , unless t = | δ ij ( t ) | /η is an integer and the distance to the solution reaches exactly 0 . Keeping track of these oscillations is cumbersome, as each of the d 2 parameters will oscillate between different constants. To simplify the analysis, we assume that the distances decrease while | δ ij (0) | ≥ tη then go to η/ 2 to model the oscillatory regime, essentially 'averaging' the oscillations, as illustrated in Fig. 5.

Assumption 4.1. We assume that sign descent with step-size η follows the dynamics

1

10

4

τ

=

T

10

-

8

10

-

5

10

-

2

φ/d

α

= 0

.

25

1

1+

α

1

-

α

4

τ

2

10

-

1

10

τ

: 4

T

2

=

d

τ

0

.

0

0

.

5

1

.

0

x

:

φ

=

d

x

α

= 0

.

5

1

-

τ

10

-

2

10

2

τ

: 4

T

2

=

τ

2

d

10

1

10

3

φ

α

= 1

d

= 10

2

d

= 10

4

d

= 10

6

d

= 10

8

1

1+

τ

2

Those dynamics do not capture the fact that a direction might reach exactly 0, after which sign descent would not oscillate, but this can only happen for a few directions if | δ ij (0) | ∝ 1 /j α , and their impact is small with large d . With this assumption, we have the following dynamics.

Proposition 4.2. If the conditional distribution follows a power law with exponent α as in Assumption 2.3, the dynamics of sign descent with step-size η in Assumption 4.1 lead to the loss

and k ∗ is the number of directions in the decreasing regime, k ∗ = max k k : π k > tη

Proof. By Proposition 2.2, λ ij = π i does not depend on j . By Assumption 2.3, there is a permutation ρ i such that δ iρ i ( j ) (0) = π j . As a result, the dynamics of δ iρ i ( j ) ( t ) do not depend on i . Writing δ j ( t ) as a shortcut for δ i,ρ i ( j ) ( t ) for any i and using that ∑ d i =1 π i = 1 ,

We then split the sum depending on whether | δ k ( t ) | is decreasing or oscillating.

The dynamics of SD in Proposition 4.2, differ qualitatively from those of GD in Eq. (1). The progress in each direction is not scaled by π i , because the update is uniform across directions. This is what will enable SD to make faster progress on low-frequency words. However, we now have another challenge in that we need to choose the step-size η to trade-off between the oscillations of magnitude ( η/ 2) 2 on low-frequency words and still making progress on high-frequency words that are not yet in the oscillatory regime. This is easy when α is small and the frequencies are close to uniform, giving a small spread for the initial distances δ k (0) , but becomes more difficult as α increases. From this, we expect SD to perform better than GD for small α , and worse for large α , but we need to understand how to set the step-size to understand where the transition happens.

4.1 Scaling of the step-size

As SD with a fixed step-size eventually enters an oscillatory regime, the loss we converge to as t grows depends on η . To describe the performance achievable after tuning η for a given budget T , we need to estimate how η scales with T and d . This effect is illustrated in Fig. 5 (right). We use capital T to emphasize that we are modeling the loss at the end of a training run of T steps with a fixed step-size which depends on T . Getting the exact form of η ∗ = arg min η L d ( T, η ) is out of reach, but we establish bounds on the optimal step-size.

Proposition 4.3. The step-size η ∗ that L d ( T, η ) in Proposition 4.2 given T and d , satisfies

Proof. If η ≤ δ d (0) /T , all directions are still in the decreasing regime of Assumption 4.1 at time T . As long as Tη < δ d (0) , increasing the step-size leads to more progress. Similarly, if Tη ≥ δ 1 (0) , all directions are in the oscillatory regime, and reducing the step-size reduces the oscillations.

1

10

4

τ

=

T

10

-

4

10

-

3

10

-

2

10

-

1

10

0

Relative error

α

= 0

.

25

c

2

α

2

(2

τ

)

2

-

4

α

10

-

1

10

τ

: 4

T

2

=

d

τ

α

= 0

.

5

1

-

τ

10

-

2

10

2

τ

: 4

T

2

=

τ

2

d

α

= 1

d

= 10

2

d

= 10

4

d

= 10

6

d

= 10

8

1

1+

ζ

(2

α

)

τ

2

As our initial distances follow a power law, δ k (0) = π k = 1 zk α where z = ∑ d k =1 k -α , Proposition 4.3 suggests an alternative parameterization of the step-size as

where ϕ controls how many directions are still decreasing. We now define the following scaling of ϕ .

Definition 4.4. We define the following scalings as a function of the dimension d and rescaled time τ

While those scalings need not be optimal, they match the empirical behavior of the best stepsize computed by grid-search, as shown in Fig. 6. For α > 1 / 2 , the step-size is only accurate for τ 2 ≥ 1 / (2 α -1) or τ ≥ 1 for α = 1 . We justify those estimates in Appendix D.

4.2 Asymptotic behavior

Using the scalings for T and ϕ in Definition 4.4, we define the asymptotic rate of sign descent as

Theorem 4.5 (Scaling for sign descent) . Given scalings for T and ϕ in Definition 4.4, the asymptotic convergence rate of sign descent (Eq. (4) ) is, with c 1 = 1 -1 2 α , c 2 = α 1 -α ,

We leave the proofs in Appendix D. The results also show different forms of scaling depending on α , with a threshold at α = 1 / 2 instead of 1 . The scaling in dimension is flipped compared to GD. SD needs t to scale with d when α is large, which is the regime where GD can make progress with finite t . For the case of Zipf-distributed data ( α = 1 ), SD only needs a scaling in d 1 / 2 compared to the d 1 -ε scaling of GD, showing that it achieves better performance for ε < 1 / 2 . We show in Fig. 7 that the asymptotic rates of Theorem 3.1 are accurate even for finite d .

10

0

10

2

10

4

Time

t

10

-

2

10

-

1

10

0

Relative error

Gradient Descent

d

= 2

6

2

8

2

10

2

12

0

1

2

Rescaled time

τ

=

log(

t

)

log(

d

)

10

-

2

10

-

1

10

0

GD, Rescaled time

exp(

-

2

τ

2

)

1

-

τ

10

0

10

1

10

2

10

3

Time

t

10

-

6

10

-

4

10

-

2

10

0

Sign Descent

t

-

2

5 Conclusion

We have presented scaling laws for gradient descent (GD) and sign descent (SD) on the linear bigram model as a function of the power law exponent α of the word frequencies. Rather than hide the dimension dependence in problem specific constants, we consider the scaling of running time and dimension as the problem grows in size to get precise estimates of the scaling. Our results highlight the benefit of SD and the need to address ill-conditioning to improve the performance of GD.

Our results show that the power-law scaling is specific to the regime α > 1 . This regime may accurately describe cases where the training dynamics converge to a well-defined limit, such as when increasing width or depth (Yang et al., 2021; Bordelon et al., 2024b; Noci et al., 2024), it misses a large dimension dependence as we scale the vocabulary size. The scaling we obtain for α ≤ 1 have a different functional form and highlight the dependency on dimension. For GD on Zipf-distributed data, the scaling of d 1 -ε shows a non-trivial interplay between the desired error ε and the dimension. Our results suggest that increasing the vocabulary size might require a larger training budget, not only because each iteration is more costly due to the larger embedding matrices, but also because more iterations are needed to reach the same error. Algorithms that target this dimension dependence, for example by estimating word frequencies (Li et al., 2022), would be an interesting next step.

Our approach however has limitations. We do not cover the online case, for which the analysis should be extendable using existing tools. The addition of momentum for sign descent would be more complex but particularly interesting to dampen oscillations, and getting finite-dimensional results by tracking a correction term for finite d would be enlightening, as the convergence to the asymptotic regime can sometimes be slow, especially in the case α = 1 . A more difficult extension would be to consider models leading to non-linear dynamics, such as bilinear models (Mikolov et al., 2013) or the cross-entropy loss. But it is not clear how to obtain closed-form solutions or sufficiently accurate approximations even for GD in the deterministic setting. We can however probe the behavior of GD and SD experimentally, and present preliminary results with the cross-entropy loss.

Empirical behavior with cross-entropy loss. We experiment with a variant of the linear bigram model trained with cross-entropy loss on synthetic data satisfying Assumption 2.3 for α = 1 . We show in Fig. 8 the result of training models with increasing vocabulary, with the step-size set by grid-search for both GD and SD to minimize the loss at the given horizon. The results suggest that the gap in scaling between GD and SD is even larger than with the quadratic loss; GD appears to require t ∼ d , as in the quadratic case, while the performance of SD appears independent of d .

Acknowledgments and Disclosure of Funding

We thank Si Yi (Cathy) Meng, Aaron Mishkin, and Victor Sanches Portella for helpful discussions and providing comments on the manuscript, and for the feedback from anonymous reviewers. This work has received support from the French government, managed by the National Research Agency, under the France 2030 program with the reference 'PR[AI]RIE-PSAI' (ANR-23-IACL-0008). Frederik Kunstner is supported by a Marie Skłodowska-Curie Fellowship from the European Union's Horizon Europe Research and Innovation program under Grant Agreement No. 101210427.

References

Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky (2020). 'High-dimensional dynamics of generalization error in neural networks'. In: Neural Networks 132, pp. 428-446.

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma (2021). 'Explaining Neural Scaling Laws'. arXiv/2102.06701.

Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux (2020). 'The Geometry of Sign Gradient Descent'. arXiv/2002.08056.

Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar (2018). 'SIGNSGD: Compressed Optimisation for Non-Convex Problems'. In: International Conference on Machine Learning (ICML) .

Raphaël Berthier, Francis R. Bach, and Pierre Gaillard (2020). 'Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Blake Bordelon, Alexander B. Atanasov, and Cengiz Pehlevan (2024a). 'A Dynamical Model of Neural Scaling Laws'. In: International Conference on Machine Learning (ICML) .

Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan (2024b). 'Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit'. In: International Conference on Learning Representations (ICLR) .

Stephen Boyd and Lieven Vandenberghe (2004). Convex Optimization . Cambridge University Press.

Yaroslav Bulatov (2023). Gradient descent under harmonic eigenvalue decay . Blog post. https:// machine-learning-etc.ghost.io/gradient-descent-under-harmonic-eigenvaluedecay-average-case-analysis/ .

Andrea Caponnetto and Ernesto De Vito (2007). 'Optimal Rates for the Regularized Least-Squares Algorithm'. In: Foundations of Computational Mathematics 7.3, pp. 331-368.

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le (2023). 'Symbolic Discovery of Optimization Algorithms'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurélien Lucchi (2025). 'Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise'. In: International Conference on Learning Representations (ICLR) .

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová (2021). 'Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Rudrajit Das, Naman Agarwal, Sujay Sanghavi, and Inderjit S. Dhillon (2024). 'Towards Quantifying the Preconditioning Effect of Adam'. arXiv/2402.07114.

DLMF (2025). NIST Digital Library of Mathematical Functions . https://dlmf.nist.gov/ , Release 1.2.4 of 2025-03-15. F. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.

John C. Duchi, Elad Hazan, and Yoram Singer (2011). 'Adaptive Subgradient Methods for Online Learning and Stochastic Optimization'. In: Journal of Machine Learning Research (JMLR) 12, pp. 2121-2159.

Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex (2019). OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus .

Thamme Gowda and Jonathan May (2020). 'Finding the Optimal Vocabulary Size for Neural Machine Translation'. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre (2022). 'An

empirical analysis of compute-optimal large language model training (Training Compute-Optimal Large Language Models)'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei (2020). 'Scaling Laws for Neural Language Models'. Tech. report. arXiv/2001.08361.

Diederik P. Kingma and Jimmy Ba (2015). 'Adam: A Method for Stochastic Optimization'. In: International Conference on Learning Representations (ICLR) .

Taku Kudo and John Richardson (2018). 'SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing'. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) .

Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt (2023). 'Noise is not the main factor behind the gap between SGD and Adam on transformers, but sign descent might be'. In: International Conference on Learning Representations (ICLR) .

Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti (2024). 'HeavyTailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Yan Li, Dhruv Choudhary, Xiaohan Wei, Baichuan Yuan, Bhargav Bhushanam, Tuo Zhao, and Guanghui Lan (2022). 'Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits'. In: International Conference on Learning Representations (ICLR) .

Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter L. Bartlett, and Jason D. Lee (2024). 'Scaling Laws in Linear Regression: Compute, Parameters, and Data'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han (2020). 'Understanding the Difficulty of Training Transformers'. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) .

Yuxing Liu, Rui Pan, and Tong Zhang (2025). 'AdaGrad under Anisotropic Smoothness'. In: International Conference on Learning Representations (ICLR) .

Chao Ma, Lei Wu, and Weinan E (2021). 'A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms'. In: Mathematical and Scientific Machine Learning (MSML) .

Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora (2022). 'On the SDEs and Scaling Rules for Adaptive Gradient Algorithms'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Alexander Maloney, Daniel A. Roberts, and James Sully (2022). 'A Solvable Model of Neural Scaling Laws'. arXiv/2210.16859.

Tomás Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean (2013). 'Efficient Estimation of Word Representations in Vector Space'. In: International Conference on Learning Representations (ICLR) .

Yurii E. Nesterov (2018). Lectures on Convex Optimization . Vol. 87. Springer.

Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto (2024). 'Super Consistency of Neural Network Landscapes and Learning Rate Transfer'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington (2024). '4+3 Phases of Compute-Optimal Neural Scaling Laws'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Steven T. Piantadosi (2014). 'Zipf's word frequency law in natural language: A critical review and future directions'. In: Psychonomic Bulletin & Review 21, pp. 1112-1130.

Marc Potters and Jean-Philippe Bouchaud (2020). A First Course in Random Matrix Theory: for Physicists, Engineers and Data Scientists . Cambridge University Press.

Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit (2020). 'A Constructive Prediction of the Generalization Error Across Scales'. In: International Conference on Learning Representations (ICLR) .

Mher Safaryan and Peter Richtárik (2021). 'Stochastic Sign Descent Methods: New Algorithms and Better Theory'. In: International Conference on Machine Learning (ICML) .

Rico Sennrich, Barry Haddow, and Alexandra Birch (2016). 'Neural Machine Translation of Rare Words with Subword Units'. In: Annual Meeting of the Association for Computational Linguistics (ACL) .

Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong (2024). 'Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Tijmen Tieleman and Geoffrey E. Hinton (2012). RMSPROP: Divide the gradient by a running average of its recent magnitude . Lecture notes http://www.cs.toronto.edu/~tijmen/ csc321/slides/lecture_slides_lec6.pdf .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2017). 'Attention is All you Need'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Maksim Velikanov and Dmitry Yarotsky (2024). 'Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions'. In: Journal of Machine Learning Research (JMLR) 25, 81:1-81:78.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors (2020). 'SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python'. In: Nature Methods 17, pp. 261-272.

Ke Liang Xiao, Noah Marshall, Atish Agarwala, and Elliot Paquette (2024). 'Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects'. arXiv/2411.12135.

Ge Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao (2021). '(Tensor Program V) Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer'. In: Advances in Neural Information Processing Systems (NeurIPS) .

Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P. Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun (2025). 'Adam-mini: Use Fewer Learning Rates To Gain More'. In: International Conference on Learning Representations (ICLR) .

Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham M. Kakade (2025). 'Deconstructing What Makes a Good Optimizer for Autoregressive Language Models'. In: International Conference on Learning Representations (ICLR) .

Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona (2022). 'Understanding AdamW through Proximal Methods and Scale-Freeness'. In: Transactions on Machine Learning Research (TMLR) .

Supplementary Material

The supplementary material is organized as follows.

Appendix A gives experimental details and information on how to reproduce the figures.

Appendix B compares our results to standard convergence rates in the literature.

Appendix C gives the main results for gradient descent Theorem 3.1.

Appendix D gives the main results for sign descent Theorem 4.5.

A Experimental details

This section goes over the technical details of the experiments needed to reproduce the figures.

A.1 Computational complexity

We use d to denote the size of the vocabulary, but the number of parameters W is d 2 as we have to learn the conditional probability table π k | j . As the number of iterations t has to scale with dimension, the problem scales in d 3 , which becomes prohibitive fast. To circumvent this issue, we use the fact that the training dynamics of gradient descent and sign descent on data following Assumption 4.1 can be simulated in O ( d ) . The error after t iterations can then be computed in closed-form if initialized at 0 , making it possible to compute the loss after t steps without computing the intermediate steps.

Proposition A.1 (Reduction of the dynamics for gradient descent) . Under Assumption 2.3, the dynamics of gradient descent with step-size 1 /π 1 can be computed in O ( d ) as

Proof. We use the dynamics using the eigendecomposition notation presented in Section 2,

Using Assumption 2.3 gives that λ ij is independent of j and δ ij is independent of i as

Plugging in those together and using that the step-size is η = π 1 = 1 /z gives

Proposition A.2 (Reduction of the dynamics for sign descent) . Under Assumption 2.3, the simplified dynamics of sign descent (Assumption 4.1) with step-size η ( T, ϕ ) = 1 /zTϕ α following the reparameterization of Proposition 4.3 where z = ∑ d k =1 k -α can be computed in O ( d ) as

Proof. Using the same derivation as above for Proposition A.1 but using the update dynamics assumed in Assumption 4.1. Note that those dynamics imply that δ ij ( T ) is independent of i . Writing ∆ j =

δ ij ( T ) for any i and using that ∑ d i =1 π i = 1 , we have

Expanding ∆ j ( T ) using Assumption 4.1 gives the result.

For the real data experiments in Fig. 2, computing the dynamics cannot be reduced to O ( d ) . We still use the fact that the dynamics can be computed in closed-form to avoid running t steps of gradient/sign descent. For sign descent, we do not use the simpler model of Assumption 2.3 but the full dynamics by computing the point reached after t steps, including oscillations.

Proposition A.3. Under the dynamics of sign descent with step-size η ,

the distance after t steps is given by

A.2 Additional details about the figures

Fig. 1 shows the dynamics of gradient descent on Problem 2.1 on data satisfying Assumption 2.3.

Fig. 2 shows the dynamics on real data on the OpenWebText dataset (Gokaslan et al., 2019). Using the SentencePiece (Kudo and Richardson, 2018) implementation of BPE Sennrich et al., 2016, we train tokenizers with vocabulary sizes of 1 000 , 3 612 , 10 000 and 31 622 tokens on the first 2 000 000 entries of the dataset with a maximum sentence length of 16 768 . We compute the frequencies and conditional frequency tables for each vocabulary size using the entire dataset. We use the closed form formulas for the loss after t steps using O ( d 2 ) computation detailed in the previous section to avoid having to run gradient and sign descent on those large models.

Gradient descent uses the empirically-derived step-size of 1 /π 1 . For sign descent, for a given time horizon T , we optimize over the step-size numerically. Because the loss after T steps as a function of the step-size is unimodal, we use the default bounded bracketing method in scipy (Virtanen et al., 2020, minimize_scalar ) starting with the interval [ η min /d, dη max ] where η min , η max are the bounds derived in Proposition 4.3. The optimal step-size can vary drastically if it is computed on even or odd iterations as the loss oscillates. To avoid this issue, we only show even iterations.

Fig. 3 shows the frequencies computed as for Fig. 2 for the largest vocabulary size, d = 31622 .

The rightmost plot of Fig. 5 shows the simplified dynamics of sign descent.

Fig. 4, Fig. 6 and Fig. 7 show the convergence of the loss in d dimension computed using the equations in Appendix A.1. For sign descent, the best step-size is obtained by grid search. We know the optimal step-size satisfies ϕ ∈ [1 , d ] (Proposition 4.3), so let ϕ = d x where x comes from a logarithmically spaced grid-search on x from -10 to 0 , taking every 1 / 32 th powers;

Fig. 8 shows the dynamics of GD and SD on the linear bigram problem trained with the cross-entropy loss. For both GD and SD, the step-size is selected by grid-search with a similar 1 / 32 th power logarithmic grid as above, to minize the loss after t steps. As for the plots of SD, Fig. 8 does not not show a single run but the envelope of the performance achievable with a constant step-size for T steps. We have not found a way to simplify the computational complexity of the experiments using the cross-entropy loss. Each run requires running GD or SD for t steps on the full d × d matrix. As t needs to scale with d , computing a run of GD or SD takes O ( d 3 ) time, which limits the vocabulary sizes we can consider.

B Comparison with worst-case rates

In this section, we compare our rates against results obtained using classical analyses to highlight the benefit of the asymptotic analysis in capturing the dependence on dimension. Our goal is not to imply those bounds are poor; each of the work cited below studied a specific problem and the assumptions were selected to highlight the impact of the condition number, non-convexity, variance, or other issue. However, due to their worst-case generality, existing results do not capture the dimension dependence on the problem of the linear bigram problem (Problem 2.1) with Zipfdistributed frequencies (Assumption 2.3) and predict worse behavior than actually observed.

In this section, we focus on Zipf-distributed data ( α = 1 ) as it is the most relevant to text data. To simplify notation, we assume that the conditional frequencies directly follow a power-law π k | i ∝ 1 /k , instead of assuming that there exists a reordering ρ i such that π ρ i ( k ) | i ∝ 1 /k as in Assumption 2.3. This reordering does not affect the dynamics of the loss and can be ignored without loss of generality.

B.1 Standard smooth, (strongly-)convex rates.

Classical results in smooth, convex optimization are derived under the assumption that the objective function L d is L -smooth and µ -strongly convex with µ ≥ 0 . We write the function rates in matrix form for the loss L d defined in Problem 2.1, but this could equivalently be transformed to a vector form using and ∥ x -x ∗ ∥ 2 2 = ∥ W -W ∗ ∥ 2 if x = vec( W ) and x ∗ = vec( W ∗ ) where vec stacks the columns of W as a single vector. For a twice-differentiable function, this is equivalent to assuming that the eigenvalues of the Hessian are bounded by µ ≤ λ ij ≤ L for all i, j ∈ [ d ] at every possible input. We compare against simple forms available in this setting (Nesterov (2018, Cor. 2.1.2), Boyd and Vandenberghe (2004, Eq. 9.18)). While it is possible to slightly improve the constants in these bounds, these constants do not meaningfully affect the asymptotic behavior as d grows.

To better compare these rates with our results, we normalize them by L d (0) -L ∗ d ,

Proposition B.1 (Values of the constants) . On Problem 2.1 with frequencies following a power-law with α = 1 (Assumption 2.3) initialized at W 0 = 0 , the smooth convex sublinear rate r sub d ( t ) and the smooth strongly-convex linear rate r lin d ( t ) are asymptotically equivalent to

Proof. The proof follow from substituting the constants with the values

where z = ∑ d k =1 1 /k d ∼ log( d ) . The eigenvalues are λ ij = π i = 1 / zi after normalization, giving L = 1 / z and µ = 1 / zd . Using that δ ij (0) = 1 /zj gives the loss and distance at initialization,

Both rates struggle to predict the progress in 'early' iterations, when t is much smaller than d . The sublinear rate requires a scaling t ∝ d / log( d ) while the linear rate predicts t ∝ d . Neither captures the progress that can be made by running t = d 1 / 2 iterations, which reaches an error of ε = 1 / 2 . Instead, both rates predict no progress. We visualize the given rates in Fig. 9 after rescaling the number of

0

1

τ

=

log(

t

)

log(

d

)

0

1

Relative error

Standard rates

1

-

τ

0

1

τ

=

log(

t

)

log(

d

)

10

-

3

1

Relative error

Standard rates (log scale)

Linear

d

=10

2

d

=10

4

d

=10

8

d

=10

16

Sublinear

d

=10

2

d

=10

4

d

=10

8

d

=10

16

steps to our normalized time τ = log( t ) / log( d ) . The linear and sublinear rates are not converging to r ( τ ) = 1 -τ . Instead, they exhibit a sharper and sharper transition between not predicting any progress for τ < 1 ( r ( τ ) ≈ 1 or r ( τ ) > 1 ) and that the problem is solved if τ > 1 .

B.2 Rates for sign descent

Analyses on sign-like methods in the literature typically target more complex algorithms such as RMSProp (Tieleman and Hinton, 2012) or AdaGrad (Duchi et al., 2011) for Das et al. (2024) and Liu et al. (2025), or consider more general problems including non-convex functions for Bernstein et al. (2018) and Safaryan and Richtárik (2021). We are not aware of existing analyses that specifically target sign descent on diagonal quadratic problems such as Problem 2.1. This makes a direct comparison difficult. It might be that the rates described in those papers for the chosen problem setting or algorithm are tight. However, our message is that the resulting rates are too pessimistic even for a problem as simple as Problem 2.1 and suggest runtimes for sign descent that are off by a factor depending on the dimension.

The main difficulty in studying sign descent and sign-like methods more generally is the strong dependence on the coordinate system used. For Problem 2.1 the dynamics perfectly separate along coordinates which makes it possible to derive a closed form for the dynamics. Other works typically rely on assumptions on the Hessian that quantify how close to diagonal it is. For example, bound the Hessian with a diagonal matrix L , H ⪯ L in Loewner ordering, and obtain rates that depend on the trace of L (e.g., Bernstein et al., 2018; Liu et al., 2025). For Problem 2.1, the Hessian is diagonal and made of d diagonal copies of X ⊤ X /n = Diag([ π 1 , ..., π d ]) , thus Tr( L ) = Tr( ∇ 2 L d ( W )) = d .

Anisotropic smoothness and AdaGrad. Using this assumption, Liu et al. (2020, Theorem 4.1) show the following convergence rate for AdaGrad. To simplify their results and show the rate in its best light, we assume there is no noise in the gradient ( ∥ σ ∥ 1 = 0 in their notation), that AdaGrad is run with the parameter ϵ = 0 , that the algorithm is run with projections onto the constrained set W = { W : ∥ W ∥ ∞ ≤ π 1 } and that we initialize at W = 0 .

Normalizing the loss and simplifying the constants using the same approach as in Proposition B.1 gives the following asymptotic upper bound

Although we might expect Adagrad to outperform sign descent as it uses decreasing step-sizes to avoid the oscillations, this rate estimate that the number of iterations should scale with d log( d ) instead of the scaling of d 1 / 2 we find for sign descent.

Preconditioning effect of Adam. Das et al. (2024) study RMSProp, or Adam without momentum ( β 1 = 0 ) but with momentum on the moving average of the squared gradient. They use high-probability arguments to handle the dynamics of the preconditioner and random initialization. Their rate shows that Adam can perform better on diagonal quadratics if the condition number scales worse than linearly with the dimensionality, by replacing the condition number κ with κ Adam = min { d W +1 , κ } where d W is the dimensionality of W . Assuming that their bound holds with probability 1 with W 0 = 0 and ignoring logarithmic factors in d and ϵ , their rate for diagonal quadratics is (Das et al., 2024, Thm. 2)

Unfortunately, on Problem 2.1 the dimensionality is d W = d 2 while the condition number scales as κ = d with Zipfian eigenvalues ( α = 1 ) so the proposed approach does not improve over gradient descent. Normalizing the loss and using the same approach as in Proposition B.1 gives

This scaling predicts the same performance for Adam and gradient descent (up to log factors depending on d and ϵ that we ignored) whereas our analysis shows a scaling of d 1 / 2 for sign descent.

Non-convex results. Results in the non-convex setting (Bernstein et al., 2018; Balles et al., 2020; Safaryan and Richtárik, 2021; Liu et al., 2025) give convergence results to stationarity instead of convergence in optimality gap, measured using the 1-norm of the gradient instead of the Euclidean norm. Because ∥ v ∥ 2 1 ≤ ∥ v ∥ 2 2 d for a d -dimensional vector v , the time required to get the 1 -norm small might be much worse than the time required to find a stationary point in Euclidean norm or to minimize the function value. To illustrate this point, we show that it is possible to have arbitrarily small relative error on Problem 2.1 and arbitrarily large gradients in 1-norm in high dimension.

Proposition B.2. On Problem 2.1 with Zipf-distributed data (Assumption 2.3 with α = 1 ), SD with simplified dynamics (Assumption 4.1) with t d ( τ ) = τd 1 / 2 / 2 and ϕ d ( τ ) = (1 + 1 /τ 2 ) -1 satisfies

Proof. Computations similar to Proposition 4.2 show that the 1-norm of the gradient is

where k ∗ is the number of directions that are still in the decreasing regime after T steps with stepsize η . As ∥ vec( ∇L d ( W 0 )) ∥ 1 = ∑ d k =1 π k = 1 , this expression is also the normalized 1-norm of the gradient. Using the parameterization η = 1 /ztϕ , where z = ∑ d k =1 1 /k , we get the update

Using the same scaling as in Definition 4.4, ϕ d ( τ ) = (1 + 1 /τ 2 ) -1 and 2 t d ( τ ) = τd 1 / 2 , we get

While getting a small error only requires scaling t with d 1 / 2 , getting the magnitude of the gradient in 1-norm smaller than a constant independent of d requires scaling t with d/ log( d ) .

C Proofs for gradient descent

This section gives the proof of Theorem 3.1 for the scaling of gradient descent.

C.1 Standard results

We start with standard results that are used in the subsequent proofs. The following classical relationships between sums and integrals of monotone functions will be used to bound the approximation error induced by analyzing the asymptotics of the integral instead of the sum.

Lemma C.1 (Sum-Integral) . For a function f that is monotone on [ a, b ] ,

To apply these sum-integral relationships to the dynamics of gradient descent in Theorem 3.1, we need to describe when they are increasing or decreasing.

Lemma C.2 (Unimodal sequence) . The sequence s ( k ) = k -α (1 -k -α ) t is non-negative on k ≥ 1 and unimodal. It monotonically increases until k ∗ = (1 + t ) 1 /α , then monotonically decreases.

Proof. As s ( k ) is non-negative, we can instead look at its logarithm,

The denominator is positive on k ≥ 1 , and the numerator is positive for small k until the derivative changes sign at αt -α ( k α -1) = 0 , or k ∗ = (1 + t ) 1 /α .

At the partial sum H d,α = ∑ d k =1 k -α , appears in the proof of gradient and sign descent, we give its asymptotic behavior independently.

Lemma C.3 (Normalizer Asymptotics) . As d grows, the partial sum H d,α = ∑ d k =1 k -α behaves as

where ζ is the zeta function, defined as the limit of H d,α , ζ ( α ) = ∑ ∞ k =1 k -α < ∞ for α > 1 .

Proof. For α > 1 , the sum converges to ∑ ∞ k =1 k -α = ζ ( α ) . For α ≤ 1 , the sum diverges as d grows. As the sequence k -α is decreasing in k , we can use the sum-integral bounds (C.1) to get

If α < 1 , the integrals evaluate to

and both terms are asymptotically equivalent to d 1 -α / (1 -α ) as d →∞ . If α = 1 , this gives

Both terms are asymptotically equivalent to log( d ) .

The main purpose of the sum-integral bounds (C.1) and the Unimodal Lemma (C.2) is to bound on the error incurred by approximating the sum with the integral form of the loss.

Lemma C.4 (Approximating error) . The approximation error between the following sum and integral,

can be bounded by the following error term,

Proof. By the Unimodal Lemma (C.2), the sequence s ( k ) is increasing until k ∗ = (1 + 2 emt ) 1 /α then decreasing, which lets us use the sum-integral bounds (C.1).

For large t . Suppose that t is sufficiently large such that k ∗ ≥ d and 1 + 2 t ≥ d α , meaning that the sequence s ( k ) is increasing on [1 , d ] . Then,

Using that s (1) = 0 gives | I d ( t ) -S d ( t ) | ≤ s ( d ) when t is large.

For small t . If t is small and k ∗ < d the sequence flips from increasing to decreasing on [1 , d ] . We still use the same idea, but bound the increasing and the decreasing subsequences separately.

Upper bound. As the sequences s ( k ) in increasing on [1 , k ∗ ] and decreasing on [ k ∗ , d ] ,

Summing both bounds and adding the remaining terms s ( ⌊ k ∗ ⌋ ) , s ( ⌊ k ∗ ⌋ +1) ,

where the last inequality uses the following simplifications,

Lower bound. Now using the lower bound,

Summing both bounds, we can complete the integral by adding and subtracting ∫ ⌊ k ∗ ⌋ +1 ⌊ k ∗ ⌋ s ( k ) d k and adding the remaining terms s (1) and s ( d ) to obtain

where the last inequality uses that s (1) = 0 , s ( k ) ≤ s ( k ∗ ) .

Combining the results for the small t regime gives

The final bound in Eq. (5) expands s ( x ) = x -α (1 -x -α ) 2 t and replaces k ∗ by (1 + 2 t ) 1 α .

C.2 Scaling laws for gradient descent

We are now ready to move to the proof of Theorem 3.1, for which we recall the theorem statement.

Proof sketch. We first give a sketch of the proof, which will be formalized in the next lemmas. Based on the reduced dynamics for gradient descent in Proposition A.1, we know that

where H d,α = ∑ d k =1 k -α . Let S d and I d be the sum and integral variants of the denominator,

First, we establish in Lemma C.5 that the integral form converges to the rate r ( τ ) in Theorem 3.1,

Next, we show in Lemma C.6 that the error incurred by approximating the sum S d by the integral I d is negligible, in the sense that | I d ( t ) -S d ( t ) | ≤ δ d ( t ) and

This gives the results that

with the values of r ( τ ) given in Theorem 3.1.

Lemma C.5 (Asymptotics of the integrals) . Let I d ( t ) be the integral form given in Eq. (7) and t d ( τ ) be the scaling given in Theorem 3.1. The following limits hold.

Proof. For α > 1 . We use the change of variable z = k -α to get

As d →∞ , the integral converges to definition of the Beta function

As lim d → α H d,α = ζ ( α ) < ∞ (Lemma C.3),

As it is not easy to intuit the rate from the Beta function, we give an additional asymptotic equivalence for large t . Using Stirling's formula, the Beta function behaves as

For α < 1 we use the change of variable z = 2 tk -α to get

To have a well-defined integral, we need to introduce the scaling 2 t d ( τ ) = τd α ,

The factor of d 1 -α will cancel out with the normalizer as H d,α = Θ( d 1 -α ) (Lemma C.3). The remaining integral should simplify for large d , as (1 -z/τd α ) τd α ≈ e -z , and converge to where E p is the generalized exponential integral. To swap the limit and integral, we can verify that the dominated convergence theorem applies. The integral can be written as

The integrand a ( z, d ) converges pointwise to f ( z ) = z -1 α e -z and is dominated by f which is integrable as ∫ ∞ τ f ( z ) = τ 1 -1 α E 1 α ( τ ) . Combined with the fact that H d,α d ∼ d 1 -α / (1 -α ) , we get

To simplify for large τ and obtain E 1 / α ( τ ) τ ∼ e -τ / τ , we use the fact that the generalized exponential integral E p ( z ) in decreasing in p , meaning that E ⌊ 1 / α ⌋ ( τ ) > E 1 / α ( τ ) > E ⌈ 1 / α ⌉ ( τ ) , and that for

integer values of p we have e -τ / τ + n ≤ E n ( τ ) ≤ e -τ / τ + n -1 (DLMF, §8.19(ix)). Both bounds are asymptotically equivalent to e -τ / ( τ +1) .

For α = 1 we use the change of variable k = d z or z = log d ( k ) to get

The normalizer scales as H d,α d ∼ log( d ) (Lemma C.3) so only the integral remains. To make meaningful progress, we introduce the scaling 2 t d ( τ ) = d τ for τ ∈ [0 , 1] ,

As d → ∞ , the integrand converges to 0 if z ∈ (0 , s ) and to 1 if z ∈ ( s, 1) , and is dominated by f ( x ) = 1 so by the DCT we can swap the limit and integral to get

Lemma C.6 (Approximation error is negligible) . Let δ d ( t ) be the upper bound on the approximation error derived in the Approximation Error Lemma (C.4). We have that

Proof. Recall that the bound approximation error δ in Approximation Error Lemma (C.4) is

For α > 1 , t does not scale with d so we are in the small t regime, 1 + 2 t ≤ d α . In this regime,

The error δ d ( t ) does not vanish with d , but it goes down as O (1 /t ) . As the integral I d ( t ) is of order Θ(1 /t 1 -1 α ) , the relative error is of order O (1 /t 1 α ) , and vanishes for large t .

For α < 1 , we scale t with d as 2 t = τd α . Whether t is small or large depends on τ . If τ < 1 , we are in the small t regime as 1 + τd α ≤ d α and

If τ ≥ 1 we are in the large t regime and

In both cases lim d →∞ δ d ( τd α ) → 0 and the relative error also vanishes.

For α = 1 we scale t with d as 2 t = d τ for τ ∈ [0 , 1] . Taking d → ∞ puts us in the small t regime, 1 + 2 t = 1 + d τ ≤ d . In this regime, which also vanishes with d .

D Proofs for sign descent

This section gives the derivation for the scaling of time and the step-size for sign descent given in Definition 4.4 and the resulting asymptotic convergence rates of Theorem 4.5. Each result start from the relative loss defined as follows.

Definition D.1 (Normalized loss for sign descent) . Let L d ( t, η ) be the loss after with step-size η as defined in Proposition 4.2, and η ( T, ϕ ) = 1 / H d,α Tϕ α be the reparameterization of the step-size derived from Proposition 4.3. The relative loss after T steps of the simplified sign descent dynamics on Problem 2.1 with power-law frequencies as in Assumption 2.3 is

where H n,p = ∑ n k =1 k -p .

Proof. Starting from Proposition 4.2 and using the fact that, if ϕ ∈ [1 , d ] , the number of components in the decreasing phase of the simplified sign descent dynamics is ⌊ ϕ ⌋ , we expand the square and replacing the sums by H n,p ,

Our rates are given for a choice of scaling of the step-size ϕ d ( τ ) and time T d ( τ ) , as

D.1 Scaling of sign descent for α = 1 / 2

Proposition D.2. For the relative loss defined in Definition D.1, if α = 1 / 2 , the scalings

are obtained by setting ϕ d ( τ ) = d x ∗ ( τ ) where x ∗ ( τ ) is the solution to

These choices result in the scaling r ( τ ) = 1 -τ .

Proof. We start from the normalized loss given ϕ ,

Taking 4 T 2 = d τ and ϕ = d 1 -τ , most terms vanish as d →∞ as H n, 1 2 ∼ 2 √ n , H n, 1 ∼ log( n ) , and

The first term is the only one remaining, and gives the scaling

The optimum is at x ∗ ( τ ) = 1 -τ and gives r ( τ ) = lim d →∞ r d ( T ( d, τ ) , d 1 -τ ) = 1 -τ .

D.2 Scaling of sign descent for α < 1 / 2

Proposition D.3. For the relative loss defined in Definition D.1, if α < 1 / 2 , the scalings

where c 1 = 1 -1 2 α and c 2 = α α -1 , are obtained by setting ϕ d ( τ ) = dx ∗ ( τ ) where

These choices result in the scaling

Proof. Substituting ϕ = dx , taking the limit as d → ∞ , and using that H d,p ∼ d 1 -p 1 -p for p < 1 , define f τ ( x ) as the limit of r d ( τ, dx ) as d grows,

We will show that our choice of step-size corresponds to taking r ( τ ) = min 0 <x ≤ 1 f τ ( x ) . Gathering terms, f τ ( x ) is proportional to the following polynomial

which has a unique stationary point at

If x stat ( τ ) ̸∈ (0 , 1] , we know the function f is decreasing on [0 , 1] as lim x → 0 f τ ( x ) = ∞ , f τ (1) is finite, and there is no stationary point in (0 , 1] ), r ( τ ) = f τ (1) . If the stationary point is in (0 , 1] , it is the minimum as f τ must be decreasing from 0 to x stat ( τ ) . This gives

which can be simplified for large τ as f τ ( x ∗ ( τ )) τ ∼ ( α 1 -α ) 2 α 1 (2 τ ) 2 -4 α .

D.3 Scaling of sign descent for α > 1 / 2

For α > 1 / 2 , the expression for the loss does not simply as d →∞ . The conditional frequencies decay fast, meaning that most of the loss comes from the few high-frequency words. As a result, we cannot define the scaling of the step-size as the minimization problem for the optimal scaling in the limit d → ∞ . Instead, we use the fact that the (normalized) loss can not converge to 0 unless all components enter the oscillatory regime, at which point we can compute an optimal step-size.

Proposition D.4. For the relative loss defined in Definition D.1, if α > 1 / 2 and 4 T 2 ≥ d -1 2 α -1 , the optimal-step size is given by

This gives the following scaling for τ 2 > 1 / (2 α -1)

Proof. If ϕ ≥ 2 , the normalized loss is lower-bounded by the error on the first two components,

This is lower-bounded by a constant C > 0 independently of T , and implies that we cannot make progress by running longer unless ϕ < 2 . If only the first component is oscillating, the optimal ϕ is

To be consistent with only having two components oscillating, this requires ϕ ∗ ( d, T ) ≤ 2 , giving the constraint that this only holds when (1 + d -1 4 T 2 ) 1 /α ≤ 2 or 4 T 2 ≥ d -1 2 α -1 . Taking the scaling 4 T d ( τ ) 2 = τ 2 d gives the limit

and the asymptotic loss

where H d, 2 α d ∼ ζ (2 α ) , the Riemann zeta function.

Proposition D.4 and Theorem 4.5 only gives guarantees for the regime τ 2 > 1 / (2 α -1) . The extension of the scalings to the regime τ 2 ≤ 1 / (2 α -1) was decided arbitrarily to fit empirical data. To fit the empirical the empirical data when both τ and α are small ( α ≤ 1 ), the asymptotic scaling presented in Theorem 4.5 uses the following step-size scaling

and the following approximation for the loss,

Both expressions are asymptotically equivalent as d → ∞ and τ → ∞ , but the above proposals (given in Definition 4.4) fit the observed best step-size and loss scalings better.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claim of the abstract are outlined in Section 1.1, which gives an overview of the results of Theorems 3.1 and 4.5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The problem setting and main assumption are given in Section 2. The sketch of the proof for the main theorem is given in the main paper after Theorem 3.1 for one case, and the other cases are describe in Appendices C and D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental details are described in Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available in the supplementary material. The data used is freely available and the experimental details are given in Appendix A.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details are given in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The experiments in the paper are deterministic.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments in the paper are lightweight and did not require the use of high performance compute resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms witht he NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The focus of the paper is on a mathematical description of the convergence rate of optimization algorithms. While this theory improves our understanding of optimization algorithms and could lead to developments that make it easier to develop machine learning models, we have not identified a societal impact relevant specifically to this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets used are described in Appendix A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve human subjects and no IRB approval is required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core contribution of this paper does not rely on LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates scaling laws for the convergence of Gradient Descent (GD) and Sign Descent (SD) optimizers for a simplified linear bigram model trained on data that follows power law distributions 1/k^α (in both token frequency, conditional token frequency, and consequently eigenvalues of the Hessian), where k is the frequency rank of each token.  This simplified model is chosen because it allows closed-form solutions for both GD and SD, while still reproducing differences between GD and SD seen in practice.  The final deliverable is a set of empirically-validated scaling theorems that show how many iterations are required to reach a relative loss target ϵ as a function of (1) the dimensionality and (2) the exponent of the power law α. GD is shown to perform particularly poorly on Zipfian α=1 data (for small ϵ), scaling almost linearly in the dimensionality, whereas SD scales closer to the square root of the dimensionality.  These findings formalize some prior empirical observations regarding the impact of heavy-tailedness of the data (as Zipfian reflects real language data).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality:\n- Technical soundness: The theoretical derivations are rigorous and are supported by experimental results on both synthetic (controlled α) and real data (OpenWebText).\n- Supported claims: The results on GD and SD scaling are clearly demonstrated experimentally (Figure 2). Their simplified theoretical model aligns well with empirical data.\n\nClarity:\n- Structure: Generally well-organized; contributions clearly stated.\n- Figures: Good illustrations effectively show theoretical predictions against empirical outcomes.\n\nSignificance:\n- Key insight: Shows explicitly why and when Sign Descent (proxy for Adam-like methods) outperforms Gradient Descent for power-law-distributed data.\n- Practical Impact: The analysis provides insights into optimizer performance with increasing vocabulary size, which is practically relevant for training language models.  However, because of the many assumptions and restrictions needed for the study, I do not foresee an immediate impact on practical LLM training arising directly from this work.\n\nOriginality:\n- Novelty of results: The paper uniquely addresses a gap by studying heavier-tailed data (α ≤ 1), which previous literature largely ignores.\n\nWeaknesses:\n\nScope limitation:\n- As noted above, the analysis is restricted to the simple linear bigram model with squared loss.  A number of other assumptions are made. While informative, the impact may not directly generalize to complex transformer layers or cross-entropy objectives common in practical NLP applications.\n- The assumption that conditional distributions also follow the same power-law as the marginal frequencies (Assumption 2.3) seems idealized. The authors investigate and acknowledge deviations (Figure 3), but the implications for practical scenarios are less clear. It's also unclear if α < 1 cases are practically motivated, as most natural language settings specifically reflect α ≈ 1.  This is not really investigated or discussed.\n\nClarity:\n\nThe paper would benefit greatly (and potentially have larger impact) by adding more general topic sentences and definitions at the beginning of sections. These can serve to anchor and provide scaffolding for readers who are not experts in this area (but may generally be interested in scaling laws, optimization, etc.).  That is, this paper may serve as a good entry point for such readers and therefore I would recommend greater clarity and also perhaps an extended related work section in an appendix to clarify the progress in understanding that has been made in this area over the last several years.\n\nAs an example of something requiring clarification, I note that the term “scaling law” is mentioned in the title, abstract, intro, Figure 1, related work, and if we jump ahead to the conclusion.  But none of this actually defines what an optimization scaling law actually is?  Readers familiar with scaling laws in other contexts will expect a clear and precise definition: here we consider a law that looks at how X changes in Y and Z (or whatever).\n- The related work mentions scaling laws for parameters like N and D (in C, for compute-efficient training) and η with B (for well-tuned training), but on a first pass, I wasn't able to extract similar scaling laws for optimization.\n- These sections of the paper definitely hint at what form the scaling law might take, but I think the paper’s clarity would have really benefited if the introduction had made the specific scaling law clear, which seems to happen at the start of Section 2.2: “Our goal is to derive scaling laws for the loss of Problem 2.1 in d dimensions after t steps, Ld(t), as d → ∞.”\n- So, in the intro, you could say something like, “specifically, we look at how the [squared-error] loss of a [linear] model, of dimensionality d, scales [after t steps] for tokens generated by a bigram LM with power law exponent α.” (where the bracketed parts depends on how specific you want to be; the key is to mention L scaling in d). You could also work this into the caption of Figure 1 in some form.\n\nAlso, it would have helped on a first read of the paper if we had more clearly defined what is meant by eigenvalues of the “data” (i.e., eigenvalues of the Hessian), and how the power law in these eigenvalues specifically relate to the power law in the unigram (and bigram) frequencies.  While you use α for both in the intro, the full connection wasn’t clear to me until after I read Section 2.  I definitely assumed that we were talking about the same α, but you could have helped by saying something like, “in our case, the eigenvalues follow the same power law as the bigram frequencies, thus we naturally consider cases where α < 1 by virtue of ...” or whatever.\n\nOther clarity issues:\n- 1.2 “Given matrices X and Y ∈ {0, 1}^{n×d} containing the one-hot encodings of n pairs of tokens”.  This was actually hard to parse at first (reading it as X and Y *each* contain pairs of tokens?).  Can we slow down a little bit here, say that x is a 1-hot-encoding of a token, and y is a 1-hot-encoding of the NEXT token, and we have a dataset of n such pairs (next-token prediction), which we put into matrix form as X and Y.  That would have helped a lot.\n- 2.1: Can we be a bit more explicit rather than defining things as we go?  Like, “let x* be the solution (loss minimizer?) of a convex quadratic loss in reduced form, f(x).  Let … be the eigenvectors/eigenvalues of A.  The loss …”\n- Figure 3, right: this plot was a bit confusing on first read.  The following words help: “once sorted, the next-word frequencies also follow a power law with the same exponent,” but the xlabel says “Rank of word k”, not “Rank of word k among followers of word j,” which I think is really meant.\n\nVery minor for the final paper:\n- Note it does say to “delete this instruction block” in the NeurIPS checklist\n- “Our approach is inspired by the line of work on theoretical scaling laws, or asymptotic convergence as the dimensionality grows” – could you revise to clarify the semantics of “or” here, i.e., does this mean “also known as” or are you talking about two distinct areas of work?\n\nQUESTIONS:\nCould you make the connection to LLM studies a bit more clear?  I.e., do modern methods of creating LLM vocabularies avoid the degenerate case where a specific token is always followed by a specific other token (e.g., \"et\" and \"cetera\" being two separate tokens), thus perhaps helping justify your power law assumptions?  Recent work has also shown that increasing vocab size with scale can improve loss, which effectively means that changing the vocab size can change the slope of the scaling law.  I wonder how this relates to your work.\n\nMotivating α < 1: Can you provide stronger justification or evidence from practical datasets (beyond Figure 3) that data following α < 1 is worth studying, and thus your results may be practically meaningful?  Is this something that arises outside of language?  In other words, why do we care about very, very heavy tails?\n\nHyperparameter implications: You mention your scaling laws can guide hyperparameter selection as vocabulary size increases - can you elaborate precisely on how practitioners might use these scaling laws to adjust specific hyperparameters (e.g., step size or optimizer choice)?\n\nGeneralizability: How do you anticipate your results might generalize beyond linear bigram models (e.g., to transformers or non-squared-loss settings)? Which of your assumptions are most critical and least likely to hold in more complex models?  Do you have further thoughts on the first/last layer issues seen previously?  Do your findings imply things about parameters in non first/last Transformer layers?  Does the theory in this work make any testable predictions for LLMs?\n\nConditional distribution deviations: Can you quantify or bound how deviations from your conditional power-law assumption might impact your theoretical predictions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider a linear bigram model, where the distribution of the current token and the conditional \ndistribution of the next token both follow a power law. They study the loss curve of gradient descent and sign descent \nin this setting, and derive loss-iteration scaling laws that take the dimensional dependency into consideration and\nare more accurate than the usual worst-case rate from the generic convex optimization analysis in the early phase of \nthe training. The scaling laws also show that when the data follow a power law distribution, sign descent has a \nbetter dimensional dependency than vanilla gradient descent.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n* This is a neat paper in the sense that the task itself (bigram model) is simple, but the analysis is clean \n  and complete. The overall presentation is also clear and easy to follow. \n* The introduction of the rescaled time variable is new in the theoretical analysis of scaling laws (as far as I know).\n  The authors let the time $t$ be a function of the dimension $d$, so that the loss $L_d(t)$ does not degenerate when\n  $d \\to \\infty$. It is surprising that such a simple trick can lead to a wide variety of non-trivial results. \n* The theoretical analysis is quite complete, as the authors derive the scaling laws for both gradient descent and sign\n  descent in fo all different regimes of the power law exponent $\\alpha$. Experiments/simulation are also provided to \n  verify their theoretical claims. \n\n**Weakness**\n* The inputs are one-hot encoding vectors of the tokens, which are automatically axis-aligned. This greatly simplifies \n  the analysis of sign descent. It is not clear if the sign descent analysis can be extended to other orthogonal (rotated version of the one-hot embeddings) or \n  near orthogonal (random Gaussian vectors) embeddings. Though one may argue that algorithms such as Muon will remove the effect of a rotation, and it \n  behaves similar to axis-aligned sign descent, this is not obvious and does not apply to the near orthogonal case. \n\n**Typos and suggestions**\n* Line 144: \"$\\pi_{\\rho_j(k+1)|j} \\ge \\pi_{\\rho_j(k)|j}$.\" I believe the inequality should be reversed.\n* Line 225: \"$\\eta_* = \\min_\\eta...$\" Should be $\\mathrm{argmin}_\\eta$\n* Line 954: \"At she partial sum ...\"\n* I found Proposition 2.2 confusing until I reached the proof Theorem 3.1. It might be good if the authors could comment \n  near Proposition 2.2 that this proposition is intended to be used together with Eq (1) to get a clean formula for the \n  loss.\n\nQUESTIONS:\n* See the weakness section. For the sign descent analysis, what will happen if we apply a rotation to the input embedding?\n  What if the embeddings are only near orthogonal (for example, random Gaussian vectors)? Do the analysis/result/intuition \n  carry over to those cases? \n* Could you provide some intuitive explanations on why sign descent has a better rate than gradient descent in this task?\n* What will happen if the condtional/unconditional distributions have different exponents? I guess we will not be able \n  to get clean formulae in this case. However, I wonder given a scaling law formula, whether it is always possible to find exponents \n  $\\alpha_k$ such that gradient descent or sign descent realizes this scaling law, or this model can realize only a \n  certain class of scaling laws? I am asking this purely out of curiosity and do not expect a complete answer.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the optimization scaling law for GD and sign GD on a linear bigram model with square loss. They posit a power law distribution for the frequency of each word $\\pi_k \\sim k^{-\\alpha}$. They provide scalings for the MSE loss as a function of $t$ for both GD and sign GD. One key takeaway for Zipfian data $\\alpha = 1$ is that GD requires $d^{1-\\epsilon}$ steps while sign GD requires $\\sqrt d$ steps to converge where $d$ is the vocabulary size. The authors verify their approximations and scaling results with numerical simulations. Their work establishes the importance of optimizer in improving efficiency of training on large vocabularies with Zipfian structure.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper studies an important problem of analyzing neural scaling laws for sign-GD. \n2. They provide a novel analysis of sign-GD on a bigram model and show that for Zipfian data, sign-GD outscores SGD significantly as a function of the token dimension ($\\sqrt d$ instead of $d$ steps).  \n3. They also include several experiments of their bigram model showing the validity of their analytical results.\n\nWeaknesses\n1. Connections to known scaling law results for LLMs could be improved or clarified (see questions below).\n\nQUESTIONS:\nQuestions\n\n1. In the related works section, the authors claim that previous scaling law studies required $\\alpha \\geq 1$ in order to have well defined infinite width and depth limits. I think both the Paquette paper and the Bordelon scaling law paper could consider either $\\alpha > 1$ or $\\alpha \\leq 1$ since their formulas depended on a general set of limiting eigenvalues. Could the authors explain what they meant in this section?\n2. The observed scaling laws often do have the form of $\\text{Loss} \\sim t^{- \\alpha}$. For GD this would require having $\\alpha > 1$ to get this functional form and it could allow for a variety of observed scaling law relationships. From Theorem 4.5 a power law scaling for time $t$ is achieved for any $\\alpha > \\frac{1}{2}$ where it goes as $t^{-2}$. Empirically the LLM scaling laws have an exponent for time of $\\approx 0.75$ (Kaplan). Do the authors think that this is compatible with their model?\n3. Have the authors experimented with LLM training using varying tokenizers? The theory suggests that time to train should depend on $\\sqrt{d}$. Validating this (or perhaps pointing to existing experiments on this) would be very impressive and strengthen the paper significantly. \n4. Do the authors have a sense of what finite model size scaling laws look like in this model? If they added an untrainable projection layer $\\hat{Y} = X A W$ where $A$ is $d \\times k$ (could be random and frozen as in Bordelon or Paquette papers) and $W$ is now $k \\times d$ can they describe the dependence of the final error on $k$ if $k < d$? Does it recover reasonable scaling laws (ie consistent with LLM papers?)\n\nNits\n\n1. \"would\" on line 34 should probably be \"could\"\n2. Should $\\lambda_{ij}$ under line 129 be $\\lambda_i$?\n\nLIMITATIONS:\nThe authors address limitations of the present study.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates why gradient descent struggles to train language models with large vocabularies, focusing on how word frequency distributions affect optimization. The authors study a linear bigram model where word frequencies follow a power law (like Zipf's law in real text) and derive scaling laws showing how training time grows with vocabulary size.\n\nTheir main contributions are: (1) They prove that for Zipf-distributed text data (where word frequency is proportional to 1/rank), gradient descent requires nearly linear scaling with vocabulary size, explaining why GD performs poorly on language model embedding layers. In contrast, sign descent (a proxy for Adam) only needs iterations proportional to the square root of vocabulary size, providing up to 100x speedups for typical vocabularies. (2) They identify different scaling regimes based on how heavy-tailed the word distribution is: for very heavy-tailed data (including Zipf's law), the number of iterations must grow with dimension, while for lighter-tailed distributions, convergence is dimension-independent. (3) They develop a simplified model that captures the essential difficulty - frequent words dominate the loss but rare words in the long tail prevent fast convergence. Their theoretical predictions closely match experiments on real text data, providing a formal explanation for why Adam-like optimizers are crucial for training the first and last layers of transformers where vocabulary size matters most.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Strong theoretical analysis: rigorous asymptotic scaling laws for both gradient descent and sign descent on a well-motivated problem. Generally non-trivial analysis.\n- Practical relevance: addresses a real phenomenon in deep learning - why Adam outperforms SGD on transformer embedding layers.\n- Some of the simplifying assumptions are reasonably well motivated (eg Figure 3)\n- Convincing empirical validation of the theoretical results\n- Well organized and reasonable clarity\n- Clear articulation of weaknesses\n\nWeaknesses\n- Some of the assumptions are fairly unrealistic. Eg, the model is linear, uses square loss, assumes deterministic full-batch updates, and collapses Adam to SD via an additional oscillation-averaging assumption (Assumption 4.1) whose fidelity is neither proved nor ablated.\n- Empirical validation is still in a toy domain, and does not test how well the theory generalizes outside of the fairly restrictive modeling assumptions.\n- The paper is fairly dense and would benefit from additional intuition.\n\n[This is outside my area of expertise so I am unable to judge novelty and relation to prior work.]\n\nQUESTIONS:\n- How well does sign descent capture the behavior of Adam in practice? An additional experiment which shows the convergence of Adam (or otherwise relaxing assumption 4.1) would strengthen the results.\n- Similarly, how much do the other modeling assumptions change empirics? E.g. using mini-batches, alternative loss function, etc.?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of why vanilla gradient descent (GD) slows down while Adam-like \"sign descent\" (SD) maintains performance when training embedding layers of language models with Zipfian token frequencies, using a tractable linear bigram next-token predictor with squared loss where both unigram and conditional distributions follow power laws π ∝ 1/k^α. The authors derive closed-form optimization dynamics for both GD and SD, establishing scaling laws that show the number of iterations t needed to reach fixed relative training error ε scales as d^α for GD (nearly linear in vocabulary size d for the realistic Zipf case α=1) but only as √d for SD when α≥½, yielding a substantial √d versus d^{1-ε} speedup advantage for SD over GD in the Zipfian regime. Through validation on synthetic power-law data and real OpenWebText bigram statistics, the analysis formally demonstrates that Zipfian spectra represent a \"worst-case\" scenario for gradient descent and provides theoretical justification for the empirical observation that Adam outperforms SGD on large-vocabulary language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Rigorous asymptotic derivations, with proofs sketched in the main text and deferred to appendices.\n- Paper is well organised: assumptions ➔ theorems ➔ figures ➔ discussion\n- Extends classical spectral‑decay analyses, which assumed α>1, to the heavy‑tailed case and to sign‑methods.\n\nWeaknesses \n- Core proofs rely on strong simplifications: squared loss, linear model, deterministic GD/SD;, momentum and cross‑entropy are ignored.\n\n- The key SD analysis rests on Assumption 4.1 (replace oscillations by a constant η/2), whose correctness is argued only heuristically.\n\n - Because the model is linear, direct impact on full transformer training remains speculative.\n\nQUESTIONS:\n- Training language models uses softmax cross‑entropy, not squared error. Do the scaling exponents change when the loss is replaced by cross‑entropy on the same linear bigram model? A short derivation or experiment would clarify applicability.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}