Seesaw: Accelerating Training by Balancing Batch Size and Learning Rate Scheduling
Authors: Alexandru Meterez
Depen Morwani
Jingfeng Wu
Costin-Andrei Oncescu
Cengiz Pehlevan
Sham M. Kakade
Venue: ICLR 2026 Poster (2026)
Keywords: optimization
batch size
cbs
scheduler
llm
pretraining
Primary area: optimization
TL;DR: We introduce a new learning rate scheduler and batch size ramp up scheme that reduces serial run time in LLM pretraining.

Abstract: Increasing the batch size during training --- a “batch ramp'' --- is a promising strategy to accelerate large language model pretraining. While for SGD, doubling the batch size can be equivalent to halving the learning rate, the optimal strategy for adaptive optimizers like Adam is less clear. As a result, any batch-ramp scheduling, if used at all, is typically tuned heuristically. This work develops a principled framework for batch-size scheduling and introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/\sqrt{2}$ and doubles the batch size, preserving loss dynamics while reducing serial steps. Theoretically, we provide, to our knowledge, the first finite-sample proof of equivalence between learning-rate decay and batch-size ramp-up for SGD on noisy linear regression, and we extend this equivalence to normalized SGD, a tractable proxy for Adam, under a variance-dominated regime observed in practice. Empirically, on 150M/300M/600M-parameter models trained at Chinchilla scale using a constant (critical) batch size, Seesaw matches cosine decay at equal FLOPs while reducing wall-clock time by $\approx 36\%$, approaching the theoretical limit implied by our analysis.

Published as a conference paper at ICLR 2026
SEESAW: ACCELERATING TRAINING BY BALANCING LEARNING RATE AND BATCH SIZE SCHEDULING
Alexandru Meterez"
Depen Morwani"
Harvard University ameterezég, harvard.edu
Harvard University
dmorwant i @g. haw.ard.
Jingfeng Wu
Costin-Andrei Oncescu
University of California, Berkeley uuujf9berkeley.co edu
Harvard University concescuêg harvard. edu
Cengiz. Pehlevan
Sham Kakade
Harvard University
Harvard University
opehlevan@seas. harvard.
sham@ harvard.
ABSTRACT
Increasing the batch size during training "batch ramp" -amp"-inp a promising strat- egy accelerate large language model pretraining. While for SGD, doubling the batch size can be equivalent to halving the learning rate, the optimal strategy for adaptive optimizers like Adam less clear. As result, any batch-ramp schedul- ing, if used at all, is typically tuned heuristically.
This work develops principled framework for batch-size scheduling and intro- duces Seesaw: whenever standard scheduler would halve the learning rate, See- saw instead multiplies by 1/v22 and doubles the batch size, preserving loss dynamics while reducing serial steps. Theoretically, we provide, to our knowl- edge, the first snite-sample proof of equivalence between learning-rate decay and batch-size ramp-up for SGD noisy linear regression, and extend this equivalence to normalized SGD, a tractable proxy for Adam, under a variance- dominated regime observed in practice. Empirically, 150M/300M/600M- parameter models trained at Chinchilla scale using constant (critical) batch size, Seesaw matches cosine decay equal FLOPs while reducing wall-clock time by = 36%, approaching the theoretical limit implied by our analysis.
INTRODUCTION
recent years, large language models (LLMs) have demonstrated remarkable progress across di- verse tasks, including outperforming humans in competitive benchmarks and international compe- titions (Huang & Yang. 2025; Petrov et al., 2025; El-Kishky et al., 2025). A central driver this progress has been the steady increase in pre-training compute, measured in floating point operations (FLOPs) (Kaplan et al., 2020; Hoffmann et al., 2022). However, hardware improvements have not kept pace with the rapid escalation of training requirements, resulting in wall-clock times extending to several months for state-of-the-art models (Erdil & Schneider-loseph, 2024).
widely studied strategy reduce wall clock time is increasing the batch size (You et al., 2017: Goyal et al., 2017). Empirical studies show that larger batches can proportionally reduce the number of optimization steps required for convergence (Zhang et al., 2024; McCandlish et al., 2018; Shallue et al., 2019). However, beyond a maximum batch size termed as critical batch size (CBS), further scaling reduces sample efficiency and limits gains in training speed.
While most prior work assumes fixed batch size, recent large-scale LLM training runs employ batch size schedules that gradually increase batch size over the course of training (Dubey et al.,
"qqual contribution

---

Published as a conference paper at ICLR 2026
2024: Touvron et al., 2023; Adler al., 2024; OLMo al., 2024; Team, 2025). This practice has been observed to further reduce training times without compromising model performance. However, to the best of our knowledge, the "batch ramp" schedules are not theoretically grounded and instead tuned heuristically. The lack of theoretical justification leaves open whether these heuristics are close optimal, motivating the central question of our study: what is the optimal batch size schedule for minimizing serial runtime while not sacrificing performance?
1.1 THEORETICAL CONTRIBUTIONS
We theoretically prove, to the best our knowledge, the first asymptttioi equivalence result be- tween learning rate decay and batch size ramp up in SGD in linear regression with additive noise. We introduce an informal version our main theorem here, as well the corollary leading up to Seesaw, and we formalize the statements in Section 5.
Theorem (Informal version of Theorem 1). Consider mini-batch SGD D total samples. Consider a base process where we run with a stepwise batch ramp up schedule which doubles the batch size at certain points while keeping the learning rate fixed. Consider alternative process where at the same points we instead halve the learning rate, while keeping the batch size fixed and adjust the number of steps such that the total processed samples remains D. Then, the excess risk of the base process is within constant factor of that of the alternative process.
Corollary (Informal version of Corollary 1). Under mild assumptions, we extend the equivalence to normalized SGD with different schedulers. Consider base process where we run with a stepwise butch ramp up schedule which doubles the butch size at cerluin points while decaying the learning rate by V2. Consider the same alternative process as before. Then, the excess risk of the base process is within constant factor of that of the alternative process.
1.2 EMPIRICAL CONTRIBUTIONS
Based on the theoretical analysis, we introduce Seesaw, learning rate and batch size scheduler that reduces the serial runtime of LLM pre-training runs by approximately 36% via increasing the batch size during training specific points. We provide empirical results in Figure I and show that at (or below) the critical batch size, our method achieves a significant serial runtime acceleration across several model and data scales, while maintaining the same performance training with cosine decay.
RELATED WORK
Role of batch size in scaling, Understanding batch size ramp up schemes during training has been a topic of interest in recent years due its crucial role in decreasing wall clock runtime. Various methods of increasing the batch size have been used in common LLMs such as LLaMA (Dubey al., 2024; Touvron et al., 2023), Nemotron (Adler et al., 2024), OLMo (OLMo al., 2024; Groeneveld et al., 2024), Apertus (Team, 2025). The reason behind ramping up the batch size is take advantage of the parallel computation of samples and thus reducing the total number of sequential steps. However, since increasing the batch size reduces the total number of gradient steps taken by the model during training, there is maximal batch size which can achieved without becoming data inefficient, called the critical batch size (CBS) (Erdil & Schneider-Joseppe 2024; Jain et al., 2018; Zhang et al., 2024; Shallue et al., 2019). Recent work also looks at the effect of batch size SGD optimization in LLMs (Srećković et al., 2025; Marek al., 2025), following previously established theoretical results in noisy quadratic models (Zhang et al., 2019).
SGD for linear regression. Recently, Zhang et al. (2024) have analyzed the CBS using weight averaging in linear regression and established scaling laws as a function of data and model size. The bias-variance analysis used by Zhang et al. (2024) has longstanding history in the literature (Jain et al., 2017) and has been used to study batch ramp-up up schemes in SGD (Jain et al., 2018). These rates have been recently made tight by (Zou et al., 2021; Wu et al., 2022a;b) for general spectra of the data covariance. Recently, (Meterez et al., 2025) have used simplified mathematical framework for rederiving the same bounds by rotating the dynamics in the eigenbasis of the data. A similar
2

---

Published as a conference paper at ICLR 2026
Batch size:
12.
256
512
1024
Sched dee::
Cosine
a.. Seasan
150M
300M
600M
10
Valibatton Loos
Vaa Validation Losa
100
10P
105
10
10.
10
10f4
Tikcess
Tokees
3.6
3.6
Vilcatton Love
3.6
3.4
3.4
3.4
3.0
2.8
15000 Steps
25000
10000
20000 30000
Figure 1: Seesaw comparison with cosine decay in 150M (left), 300M (middle) and 600M (right) models trained Chinchilla scale. Seesaw matches the loss dynamics of cosine annealing in FLOPs (top row), but achieves a significant speed up in terms of serial runtime (bottom row). Runs are swept over learning rates and plotted the best learning rate for cosine annealing terms of validation loss. at each batch size. The validation losses at the end of training are provided in Table 1. Note the axes: the top plots are on a logarithmic scale while the bottom are on a linear scale. For more experimental details, see Section 4.
diagonalizing idea has also been previously used in literature by Bordelon & Pehlevan (2021); Wu et al. (2023b;a).
Stochastic Differential Equations (SDEs). Another point of view for studying the interaction between batch size and learning rate in optimization is through SDEs (Li et al., 2021; Xie et al., 2020; Compagnoni et al., 2024; Jastrzębski et al., 2017). Malladi et al. (2022) study how to scale the learning rate function of the batch size in adaptive algorithms, extending previous work that introduced the square root scaling rule (Granziol et al., 2022; You et al., 2019).
Empirical Work. Scaling laws for the CBS and the optimal batch size have also been recently observed by (Bergsma al., 2025). In line with our conclusions regarding SGD, the linear scaling rule for SGD has been observed by (Smith et al., 2017), showing that in SGD, linearly increasing the batch size is equivalent to decreasing the learning rate. McCandlish et al. (2018) propose metric based the Hessian and the noise that correlates with the CBS over training. While their proposed metric is based on having access to the Hessian, which is prohibitive for current large-scale runs, they find that the noise scale increases during training run, which aligns with our theoretical predictions. Lastly, perhaps the most similar to our work is Merrill et al. (2025), who propose a batch size warmup scheme based starting from checkpoint with various multiples of the current batch size, and pick the largest k* where the loss is e-close to the original loss. Based on this methodology, they propose the scaling rule B = 2B, and M = v2n,. In contrast, we propose simple drop-in replacement for existing cosine schedulers, motivated rigorously by (normalized) SGD on quadratics. Moreover, we argue that the scheduler proposed by (Merrill et al., 2025) will lead to instabilities and divergence after fixed number of steps, based on our theoretical analysis in Lemma 4.
SEESAW: ALGORITHMIC DETAILS
We begin by providing an intuitive derivation of Seesaw, and the practical implementation of our algorithm. To build intuition, consider different SGD processes. In process take 2 steps learning rate n/2 and batch size B, and the other we take step at learning rate 7 and batch size 2B. Intuitively, both processes should look the same up first order: the deterministic part of the
3

---

Published as a conference paper at ICLR 2026
update stays the same, and the noise averages out. Consider general smooth loss function [(x) and let go = V((xx). Then, through a simple Taylor expansion up to first order in 77, we have the loss of the (n, 2B) process and the loss of the 2 half step process (n/2, B) respectively:
((xi) = ((x)) - 18. (go + 5) +
o² Ia Cov(E') = 2B
((x2) = [(xi)
" B
2 28 (2go + Eo + (1) + 0(1)²
Cov(Es) = B Id.
o²
Note that the processes are equivalent up to first order both in the deterministic part and in the noise terms up to O(j)),, an argument which has been previously shown by Malladi et al. (2022). We formalize this SGD intuition Theorem and extend to normalized SGD as an analytical proxy to Adam.
3.1 EXTENSION TO NORMALIZED SGD
From the previous subsection, intuitively, for SGD, cutting the learning rate by factor of a should be equivalent to increasing the batch size by factor of To design practical training algorithm based on the SGD analysis and arrive at Seesaw, we begin with the Adam update rule and simplify until we obtain normalized SGD (NSGD), which a commonly used tractable analytical proxy for Adam (Jelassi et al., 2022; Zhao et al., 2024; Xie et al., 2024). Suppose we are optimizing over parameters and denote the gradients at each time step g Then, for learning rate 7) and ignoring the bias correction, the parameter update is:
m, = 81m + (1 (1- 31)g(
V, = 32 +
(2)
m.
0. = 0, Vv +
(3)
where m, is the momentum term, V: is the second moment term, 31, 82 are their respective expo- nential decay rates, and ensures stability. For NSGD, approximate the per-coordinate updates of Adam will full parameter updates, set B1 = B2 = 0 and replace the denominator with the true expected value of the squared gradient norms over the population:
g
0, = 02
g+
Equation 4 describes the NSGD update rule, which is a crucial component of de- signing Seesaw. While the full analysis is deferred to Appendix B, the expected gra- dient norms can be decomposed as:
Algorithm Scesaw
Inputs: (initial learning rate), Bo (initial batch size), a > (step decay factor), (steps at which input scheduler cuts 17 by a)
I/ For a given input scheduler, S₁ are the steps where n ////"
mean + variance B g: 2
(5)
n < nu. B < Bu, scheduler +
where the variance scales down with the batch size. To design Seesaw, assume that the variance dominates the expected gradient squared norms (Assumption 3), and we motivate why this assumption is reasonable in Appendix B. This step re- duces (up to constant factors) the NSGD
for E S do
1 - 1/
B B+B.a
scheduler. apppedd B)
end return scheduler
update rule to SGD with a rescaled learning rate, allowing us extend risk equivalence NSGD (Corollary 1) in Section 5. For NSGD, informally, Corollary 1 shows that any learning rate cut by factor of and batch size increase by a factor of are equivalent as long as a 3 held constant. We further empirically compare Seesaw with other possible schedulers in Figure 4.
3.2 ACHIEVABLE SPEEDUPS
While our theory is established for step decay schedulers, in practice we approximate cosine decay with step decay by considering a decay of a, and passing the times (as measured in tokens) where
4

---

limit will have a total of 2T x steps. This yields maximum theoretical serial runtime reduction of Published as a conference paper at ICLR 2026
the cosine would cut the learning rate by input to Seesaw. Then, at these points, instead cut the learning rate by va and increase the batch size by 3. where the schedulers are equivalent in terms of loss long as we keep the product a fixed. However, cannot arbitrarily increase the batch size at time I and expect the risk to match the underlying process. Lemma 4 quantifies this and the main takeaway is stated below:
The most aggressive ramp up scheme argument see Lemma 4)
Remark 1.
use is given by VB. (for a formal
In Section 4.1 I we empirically verify this constraint and show that a = /B is the most aggressive scheme we can choose without divergence, which is the reason for presenting Algorithm in this setting.
At the most aggressive limit, can compute the theoretical speedup would hope to achieve where the standard scheduler is the cosine decay.
Lemma 1 (Maximum Theoretical Speedup under Cosine Decay), Consider a baseline training process of T total steps using a constant batch size and cosine learning rate schedule nttt) Yo cos( zf. ). An equivalent process run with batch ramping schedule like Seesaw, in the continuous 21
(1 (1-2) 2) 36.3%.
Lemma 1 provides an intuitive upper bound on the acceleration from Seesaw. The speedup is sig- nificant but less than 50% because most of the training progress under a cosine schedule occurs early, when the learning rate is high and the batch size must consequently be relatively small. While Seesaw aggressively increases parallelism in the later stages of training, the initial, more sequential phase remains the primary bottleneck on total runtime.
EMPIRICAL FINDINGS
In this section, we present the experimental details and methodology for evaluating Seesaw. We denote by D the dataset size, N the number of parameters.
B=128
B=256
B=512
B=1024
150M (cosine)
3.0282
3.0353
3.0696
3.1214
1 550M (Seesaw)
3.0208
3.0346
3.0687
3.1318
300M (cosine)
2.8531
2,8591
2.8696
2.9369
300M (Seesaw)
2.8452
2.8561
2.8700
2.9490
600M (cosine)
2.6904
2.6988
2.7128
600M (Seesaw)
2.6883
2.6944
2.7132
Table 1: Final validation losses picked at the best learning rate (for the cosine annealing scheduler) for each batch size, for a 1.1. Note that the dynamics match robustly across the 2 schedulers when trained at CBS.
Model and Dataset. We pretrain models of size 150M, 300M and 600M (non- (non-embedding) param- eters at Chinchilla scaling i.e. D = 20N (Hoffmann et al., 2022). We use the OLMo (Groeneveld al., 2024) codebase to train all of our models. For each experiment, we do learning rate warmup for 10% of the total amount of tokens, followed by learning rate decay following cosine scheduling or Seesaw. We report the architectural details of each model as tuple (depth, # heads, width), and thus we have for 150M (12, 16, 1024), 300M (24, 16, 1024) and for 600M (24, 22, 1408). Un- less mentioned otherwise, each model is trained using AdamW, with weight decay
0.0 (no 10 8 For each run we sweep over learning rates 1 E (0.001, 0.003, 0.01, 0.03) and initial batch sizes B E (128, 256, 512, 1024), at sequence length
weight decay), 3 0.9, 32 = 0.95,
In the continuous-time limit, consider aggressive (non-divergent) batch size ramp that maintains the relationship a = VB. Consequently, the total number of sequential steps given by the integral of the normalized learning schedule: Jo" nit) dt for cos( zt )dt
2T.
5

---

Published as a conference paper at ICLR 2026
a
2,0=1
,
2
V22
v2,8=2
a=2414 --2-112 -=1,0=4
B=256
B=512
3.125
Validation Loss
3.15
3.100
Validation
3.075
3.10
3.050
3.05
2.00
2.25
2.50
2.75
3.00
3.25
2.00
2.25
2.50
2.75
3.00
3.25
Tokens
1e9
Tokens
1e9
Figure 2: 150M models trained at batch size 256 (left) and 512 (right) with a and values following the line of equivalence ayB = 2 described in Table Note that the target to match is the blue trace, and our theory (Lemma 4) predicts that the red and purple traces should not match the baseline (blue trace) due to instabilities.
L = 1024. Similar to the OLMo training codebase, we enable z-loss during training, but provide ablations over it in Appendix D showing that it does not affect the model performance at our scales. All our models are pretrained on the C4 dataset (Raffel et al., 2020), tokenized with the T5 tokenizer.
Experimental Design. We compare Seesaw with cosine annealing by training models at the crit- ical batch size (CBS) 8* approximated based on (Zhang al., 2024), namely 8 s 256 L (150M), 3 zs 512 x L (300M) and 3 se 1024 x L (600M) tokens. The main results comparing Seesaw and cosine annealing at equal FLOPs are provided in Figure 1. The precise final losses obtained by the 2 schedulers are provided in Table 1.
4.1 CAN WE DO BETTER?
Recall that based on Corollary and Lemma 4, we have a family of equivalent schedules in NSGD, given by a fixed product av/3, under the constraint that a 2 v3. Ideally, we would like make as large as possible, since this would lead to larger batch sizes, and thus assuming enough devices are available, the lowest serial runtime. Crucially, the constraint prevents us from using a too agressive batch size scheduler. In this section. we empirically verify our theoretical prediction by testing schedulers positioned various points on the (ax, (3) axis. Namely, we train 150M models at fixed
2
2//
2/22
23/1
3
1
23/2
2
2/2
2
Table 2: a, 3 values used to test the extreme values of the equivalence.
batch size and Chinchilla scale, and we approximate cosine decay with a step decay scheduler that halves the learning rate at the token counts where the cosine schedule's 's learning rate would halve. This gives us the baseline a = and B = 1, with the product / = Based on the theoretical constraint and the equivalence line, the most aggressive scheduler we could use a = /2 and compare with
== 2. To validate our hypothesis, geometric intervals. Table 2 gives an overview of the experimental design, and Figure 2 shows that indeed the most aggressive schedules tend underperform.
and / 4, and points in between at
4.2 WHEN DOES ASSUMPTION FAIL?
Up to this point, crucial assumption for the development of our theory and the design of Seesaw has been Assumption 3. Recall that Assumption 3 states that the expected gradient norms namely, the denominator of the NSGD update step. dominated by the additive noise. Intuitively, since the noise variance decreases with the batch size as O(1/B) one can see that past certain batch the additive noise will become small, and thus Assumption will fail. In Figure 3,
that at sufficiently

---

Published as a conference paper at ICLR 2026
large batch sizes, indeed Seesaw starts to perform worse compared to the underlying cosine schedule. The first hypothesis could be that it is still possible to match the underlying schedule, but
Scheduler:
Cosine
Fixed LR
Seesaw
B B=1024
B=2048
4.0
4.0
Validation Loss
3.8
Validation Loss
3.8
3.6
3.6
3.4
3.4
3.2
3.2
B 4=9096
B=8192
6.0
6.0
5.5
Validation Loss
5.0
5.0
4.5
4.5
4.0
4.0
3.5
3.5
10a Tokens
10a Tokens
Figure 3: 150M models trained past CBS (roughly 256), batch sizes 1024, 2048, 4096 and 8192, for 3 schedulers: cosine decay (blue), constant learning rate with increasing batch size based on Seesaw (orange) and Seesaw (green). Note that none of the proposed schedules is able to match the cosine curve, with the discrepancy increasing as the batch size grows more.
with a learning rate equivalence as given by mean dominating in the denominator. As mean does scale with batch size, therefore, using the equivalence schedule required by SGD could be promising candidate. We explore this option in Figure 3, and it turns out that this schedule performs even worse than the Seesaw schedule. We hypothesise that beyond certain batch size, it is not possible to match the performance of learning rate decay by any equivalent batch size ramp up for Adam or normalized SGD, which we motivate using the following toy example.
For simplicity, we look at NGD in ID, for the quadratic loss L(x) = } hz², where æ, E R and 0. Training with NGD, have the loss gradients with respect the parameters and the update rule:
V = h
T++1 Tr sign
A 1y/a sign(rt)
x² is the distance of the current iterate from the minimizer. Note that if x, > O, then A, 17h and z; O. then A, -n,, implying that the model does not reach the minimizer and instead converges to a stable cycle of (()) around the minimizer. In order to escape this stable cycle and reach the minimizer, Therefore, if we slightly relax the setup and think of large training batch as being close to NGD regime, we can see that further increasing the batch size does not change the dynamics. Therefore, past certain batch size, it is fundamentally impossible to formulate batch size ramp up scheme with fixed learning rate that achieves the same loss as a learning rate scheduler fixed batch size.
where hign(r;) _t_ and A,
|.,|
thus necessary to decay the learning rate.
5 THEORETICAL ANALYSIS
In this section introduce the main theoretical contributions of work. Namely, under mild assumptions, we establish a formal equivalence between learning rate decay and batch size ramp up in SGD and normalized SGD.

---

Published as a conference paper at ICLR 2026
Setup and Notation. We use the notation S g to mean that there exists some constant such that f(x) I cg(z) for any æ. We also use the notation f ~ g f(x) S g(x) I f(x) for all x. We denote the samples (x, y) where x R and y R. with the distribution and risk:
1 [W (ww, x) y)²
x (O,
y|x ~ N NNWWw, x²)
R(w) =
where the expectation is over the (x, y), w is the minimizer, and o² is the variance of the additive noise. We also use ([W, 17) denote the risk at time for process trained with 7). but we drop the parameter when it is clear from context. We consider step decay schedules for the learning rate, where, the learning rate in the 4 phase denoted by 7is and Pj denotes the total number of data samples used in the ttth phase. Similarly, for batch ramp schedules, B₁ denotes the batch size in the h phase. For discussion, we will use the bias varianar decomposition terminology of risk (Jain et al., 2018; 2017: Zou et al., 2021; Wu et al., 2022a;b; Meterez et al., 2025). Informally, bias corresponds to the risk of the averaged iterates, while variance corresponds to the noise in the iterates, and (ww)) bias, variance,. We will denote the stochastic gradient at time t by gr and let E g/l represent its expected squared norm under the population distribution.
5.1 MAIN RESULTS
In this section, we first introduce the main assumptions and discuss their implications, followed by the main theoretical results. Our first assumption states that the risk is almost non expansive, in the sense that at any point during training after starting the scheduling. the risk is close the starting risk.
Assumption (Bounded risk.). Suppose an SGD process and a given scheduling scheme, and let la be the time where the scheduler starts. Then, we assume that there exists constant c 1 such that R(w,)= S co² for all > to.
In general, we expect every "well tuned" scheduler to start cutting when R(wt,) S x2, as we want to minimize the bias component of the risk before cutting down the learning rate to reduce noise in the iterates. Moreover, for well-behaved schedule, as we expect the risk to decrease over time, this condition should hold throughout the process.
Our second assumption characterizes the gradient norms in the normalized SGD update rule.
Assumption 2 (NSGD oracle access). For normalized SGD, we assume access an oracle that provides, at every step, the exact value of the expected squared gradient norms E| [l]]g:l]
In general, we don't have access to the ground truth gradient norms and rely on an exponential moving average controlled by the 32 hyperparameter in Adam, in order to estimate the gradi- ent norms. Assumption 2 simplifies the analysis by giving us access to the true expected gradient squared norms. Our final assumption states that the expected gradient squared norms of the NSGD update rule are dominated by the additive noise term.
Assumption (Variance dominated.). Assume that Elg 12 B.
Under Assumption 3, the NSGD process effectively reduces to SGD with a rescaled learning rate, up to constant factors. Based on the previously established assumptions, we can now state the equivalence result. We use the notation R(,, B.) denote the risk time t of an SGD process trained with the learning rate scheduler 1) and batch size scheduler B,, where we omit the time subscript to denote constant learning rate or batch size respectively.
Theorem (SGD Equivalence). Fix 0.01 T(H 2 1) > 0, B O, and parameters a1,02 B1, ,2 with 3131 = 02 Define the two phase-indexed schedules
(x), (na BBt)
(TiC B,) == (naz",BB)) 0,
and run two SGD procedures in phases 1, 0, so that, in phase k. each procedure processes the same number of samples (possibly depending on k) under its respective schedule. Let R(m, Bx) and Ratfs BB) denote the (population) risk of the two procedures at the end of phase k. If Assump- tion I holds (for both procedures) with constant c, then
R(1.01 7k. BB) Sa R(nKs Ba) So R(1ho B³),
8

---

Published as a conference paper at ICLR 2026
B3a) denotes the risk of the second procedure when its entire learning- rate schedule where R(x- n/6 multiplied by a uniform factor À > 0, and A Se B means S C(c) B for a numerical constant C(c) depending only с (and absolute constants).
We defer the full proof to Appendix A.1. Now, we extend this result to Normalized SGD. Under Assumption 3, NSGD reduces to SGD with a rescaled learning rate 1/ /B
/T(HH) (Equation equa-
tion II). Consequently, we can extend Theorem I to the normalized SGD case. We formalize this in the following corollary:
Corollary (Normalized SGD Equivalence). Fix 0.01
TH( 2 1 B and parameters 01, 02 >
1, P1: 82 > with (11 /B1 022 Define the two phase- innse- schedules
(mh, Bx) = (πaT² k B st), (Ther BB) = (π B2).
= 0,1,,,
1.... so that, in phase each procedure processes the same number of samples (possibly depending k) under its respective schedule. Let Rim,, B&) and R(the B¹)) denote the (population) risk of the two procedures at the end of phase k. Assumption I and 3 holds (for both procedures) with constant e, then
and run two normalized SGD procedures in phases
R(1.01 Th BB)) So R(TK, B&) Se R(1hs B¹),
where R(x 16 B¹²) denotes the risk of the second procedure when its entire learning-rate schedule is multiplied by a uniform factor X > O, and A Se B means A S C(c) B for a numerical constant Ccc) depending only on e (and absolute constants).
DISCUSSION AND CONCLUSIONS
In this work we have introduced Seesaw, drop-in batch size and learning rate scheduler, theoret- ically motivated by optimization quadratics using normalized SGD. We rigorously show that for stepwise schedulers there exists equivalence between learning rate decay and batch size ramp-up, and empirically compare our scheduler with cosine annealing using a stepwise approximation the cosine. Crucially, also show that there exists maximally aggressive batch size ramp up scheme without leading to instabilities and divergence during training. In the current implementation. See- saw is able decrease the serial runtime of training run by Rs 36%, bringing significant speedups to current pretraining pipelines. To conclude, we believe that our scheduler is principled way of decreasing the runtime of any LLM pretraining run an optimizer agnostic way.
ACKNOWLEDGEMENTS
AM would like to thank Jacob Zavatone-Vett and Alex Damian for helpful discussions. The authors would also like to thank Max Shad and Bala Desinghu for their help with the cluster. AM, DM acknowledge the support of Kempner Institute Graduate Research Fellowship. CP is supported by an NSF CAREER Award (IIS- -23-7233,,) DARPA grants DIAL-FP-038 and AIQ-HRO01 12520041, the Simons Collaboration on the Physics of Learning and Neural Computation, and the William F. Milton Fund from Harvard University. AM, SK and DM acknowledge that this work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation establish the Kempner Institute for the Study of Natural and Artificial Intelligence. SK and DM acknowledge support from the Office of Naval Research under award N0001422-1-225 1-2377 and the National Science Foundation Grant under award #IIS 2229881. 1. DM is also supported by Simons Investigator Fellowship, NSF grant DMS-2134157, DARPA grant W91 NNF211002 and DOE grant DE-SCC022199.
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11 1104,, 2024.
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Power lines: Scaling laws for weight decay and batch size in Ilm pre- mre-training. arXiv preprint arXiv:2505. 13738, 2025.

---

Published as a conference paper at ICLR 2026
Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd structured features. arXiv preprint arXiv:2106.02713. 2021.
Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov, Frank Norbert Proske, Antonio Orvieto, and Aurelien Lucchi. Adaptive methods through the lens of sdes: Theoretical insights on the role of noise. arXiv preprini 1rXiv:241 1.144....15.... 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025.
Ege Erdil and David Schneider-Joseph. Data movement limits to frontier model training. arXiv preprint arXiv:2411.01 2024.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:/706.0220 706.02677. 2017.
Diego Granziol, Stefan Zohren, and Stephen Roberts. Learning rates as a function of batch size: A random matrix theory approach to neural network training. Journal of Machine Learning Research, 23(1 (1173):1-65, 2022.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402. 00838, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025. arXiv preprint arXiv:2507.15855 2025.
Prateek Jain, Sham M Kakade, Rahul Kidambi, Praneeth Netrapalli, Venkata Krishna Pillutla, and Aaron Sidford. A markov chain theory approach to characterizing the minimax optimality of stochastic gradient descent (for least squares). arXiv preprint 11ti:::11100943000 2017.
Prateek Jain, Sham M Kakade, Rahul Kidambi, Pranceth Netrapalli, and Aaron Sidford. Paralleliz- ing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of machine learning research, 18(223):1-42 2018.
Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623. 2017.
Samy Jelassi, David Dobre, Arthur Mensch, Yuanzhi Li, and Gauthier Gidel. Dissecting adaptive methods in gans. arXiv preprint arXiv:2210.0431 10.000044, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems, 34:12712- 12725, 2021.
Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the sdes and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35: 7697-7711, 2022.
10

---

Published as a conference paper at ICLR 2026
Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Goldblum. Small batch size training for language models: When vanilla sgd works, and why gradient accu- mulation is wasteful. arXiv preprint arXiv:2507.07101, 2025.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 2018.
William Merrill, Shane Arora, Dirk Groeneveld, and Hannaneh Hajishirzi. Critical batch size re- visited: A simple empirical approach to large-batch language model training. arXiv preprint arXiv:2505.23977 2025.
Alexandru Meterez, Depen Morwani, Costin-Andrei Oncescu, Jingfeng Wu, Cengiz Pehlevan, and Sham Kakade. simplified analysis of sgd for linear regression with weight averaging. arXiv preprint arXiv:2506. 1.t::2500.55555. 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprini arXiv:2501.00656, 2024.
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva. Kristian Minchev, Mislav Balumović, Nikola Jovanović, and Martin Vechev. Proof or bluff? evaluating Ilms on 2025 usa math olympiad. arXiv preprint urXiv:2503.21934 2025.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(1 12):1 19911149911 2019.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00450 2017.
Teodora Srećković, Jonas Geiping, and Antonio Orvieto. Is your batch size the problem? revisiting the ada gap language modeling. arXiv preprint 2rXiv:2506.12545 2025.
Apertus Team. Apertus: Democratizing Open and Compliant LLMSs for Global Language Environ- 2025.
ments. https: https/////ggggggggggeeeee co/ ht///wssssss ato//vssaaail/peeruus---00
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Last iterate risk bounds of sgd with decaying stepsize for overparameterized linear regression. In International Conference on Machine Learning, pp. 24280-24314, PMLR, 2022a.
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, and Sham Kakade. The power and limitation of pretraining-finetuinng for linear regression under covariate shift. Advances Neural Information Processing Systems, 15:33041-33033 2022b.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? arXiv preprint ar 3rXv:23310.88311, 2023a.
Jingfeng WI, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Sham M Kakade. Fimile-sample analysis of learning high-dimensional single relu neuron. International Confer- on Machine Learning, pp. 37919-37951. PMLR, 2023b.
Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits geoometr of loss land- scape via coordinate- wise adaptivity. arXiv preprint arXiv:2410.08198, 2024.

---

Published as a conference paper at ICLR 2026
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamies: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03495, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 130.00388, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1904vv0900000922. 2019.
Guodong Zhang. Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger В Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. Advances in neural information processing systems, 32, 2019.
Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Fos- ter, and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21676, 2024.
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstruct- ing what makes good optimizer for language models. arXiv preprint arXiv:2407.07972, 2024.
Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Benign overfitting of constant-stepsize sgd for linear regression. In Conference on Learning Theory, pp. 4633-4635 PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
A PROOFS FOR SECTION 5
A.1 PRELIMINARIES
We take as convention for eigenvalues ordering Amax X, A2 2..>>> 0. For two matrices A and B we use the notation A 3 B to denote that B - A is positive semi-definite (PSD). We denote (u, v) for the inner product between u and V. Moreover, with slight abuse of notation, we use the notation < as elementwise comparison, namely u S v if u; S V; for all i and A < B if Asj S By for all i.j. To simplify the analysis, we will follow the approach of Meterez et al. (2025) and work in the eigenbasis of the data covariance H. Denote the eigendecomposition of H = QAQ For the sake of completeness, we restate the main derivation for the bias and variance iterates in the case of constant learning rate and constant batch size, starting from the SGD update rule:
B
B
W+41 - w'
7
X:X 1=1
(wi - (w)
"
X,C
B
B
1=1
1 E.41 = - - HH, +1² (1+B) HE,H- + THE H +
D H
B
B
= MM+ = M. M, AA,, +1² (1+
1 B
AM, + TMAMM +
T
n
o (6)
B
B
where in the last equation M, QΣ,Q is the iterate covariance matrix rotated in the eigenbasis of H. Since we can write the excess risk as:
R(w,) - R(w*) 2
TT(A.M)) = 2 (A m.)
where m, (M,, it suffices push diag operator through equation equation 6. Finally, get:
I- 2nA n7
1 1+
A
n AA
17022 X A'mo + n00² t-1 m,
B A'X
Σ
m
B
B
B
2²
A are the bias and variance iterates respectively, Before we begin proving the main statements, we introduce several helpful lemmas that we will use. Lemma 2. For n S 0.01 O,H) and a 2 1, we have the elementwise inequality:
where m, := A mo and m, := B
a 1
77 -- 2 )
II-
-1 a x
7
217
Proof. We have:
)
I
[ 17
-1
I
1
A - 2 71
A ))
2h
a2
A (2
=²aa )
at
2n
-1
Note that trivially we also have the other direction by noticing that 2- w x gives us the conclusion.
S
Multiplying by
Lemma 3. For 17 S 0.01/Tr(H) and a1, 02 B1, B2 > 1 such that 01 = ag and a1 a22 we have:
2
2
2
I
1.01
A
1
I
7
1
I
a
ai
0g
13

---

Published as a conference paper at ICLR 2026
Proof. RHS bound. Since both sides diagonal matrices, it suffices to prove the scalar inequality for every x = 77}
252
23
x
x 2
1
Taking logarithms and defining
2/3 log(1 - x//a/) of log((1 - x/a g(mi)
f(x) 2 log(1 x//a/) a log(1 x/a5)
g(az)²
where g(v) y log(1 z/v). For 0 < 1 and y > 1. g(v) is monotonically increasing, for an I a2 we have g(a1) I g(az) and hence g(an)/ g(2z) 2 (since g(az) < 0). Thus f(x) 2 1, which proves the RHS inequality.
LHS bound. Similarly, we use the scalar inequality and the bounds
2 In(1
x)
2
x x²
2
Since In(.) is monotone, we apply it to both sides:
1.01
1.01
1.01
8 In
1
3
0
02
20
1
1
g In
aj
2
ai
x
a
2k
It suffices to prove that:
1.01
1.012
B
x
8/
x2
0
2a²
a
Using
P. and
Lz
82
we obtain:
02
01
022
ayaa
1
1
a
(1.01) + (1.01)
2a
x
aj
a1
2k
x
2
0,
0.01
x
5
LOT
41
2a
Using 0n S 02, we get
< ai
0.01
x
1012
which holds automatically under 17 S (.0)/TT(H)) and x n7, This concludes the proof.
A.2 PROOFS OF MAIN STATEMENTS
Proof Theorem I. Consider 2 processes: process will have a learning rate step decay factor of Q1 and a batch size ramp up factor of {1 and process 2 will have 02 and 32 respectively. Define the transition matrices:
v2
A
A
+
(A + AAT)
Ba
2k
gk
1
122
C
A
+
(A
0
Bo
2k
Denote process 1 as my) and process 2 r& (1) where they depend on the base learning rate note that we skip the indexing on 7) when it is clear from context. In order to keep both the per stage data count, m, does B Ps steps per stage, and r does 8 Ps steps per stage. We begin by establishing the upper bound first. Note that we assume that a = a2 and without loss of generality due to symmetry, that B1 2 82 (and consequently on 0:2).
14

---

Published as a conference paper at ICLR 2026
Upper bound. Before begin, introduce the idea behind the proof. We define M₁ = B Pk. and N₂ = 3 Pk. The derivation proceeds by unrolling the recurrence first over a single step, then over 322 steps, and finally over Ps stages.
M AAMNNa-1
Bat 34
1P02
I aj
A
mN -1 + (1+ + 2c)
A, Bafk 3
which follows from Assumption 1.
MNi1
77
A
2
M 3/
+ + +1+ 2c)
Ba 3K
1700
1j
83
Σ
1
11
17
A
2i
A
==0
7)
2,8
1²202 Ba 2k
17
A
205
17
27
]
m Ni.k 84 + +(1+ 2c)
A
A.
$1
a1
k
k
Applying Lemma 2, we have:
23
na²
205
MN.
a
"I
A
m M - 83 + (1 2c)
Bat 31
I-
I-
af
1]
A
1
29
1²002
"I
A
m NIAA 85 + + 2c)
A.
B
By Lemma 3, we can replace the term with one involving (a2, Bi):
[1
17
23
17002
B2
MN I
a22
A
N
+ 2c)
B
a1
X.
Following, we can unroll over Pa:
2Mc
1²02
P
20%4
MN
I
MN +
20)
Σ
I
A
X.
B
a
p1
0
1=0
Finally, recursively unrolling across lo yields:
k
2M,7
mN <
II
1 A
m
1/02
2M,
P
2
+ +111 + 2c)
I
A
Σ
I
?
A
A.
B
01102
02
a
+1
1=1
15

---

Published as a conference paper at ICLR 2026
For the lower bound, we follow similar strategy, by bounding the term AA 2 0:
2
0/222
77
MMisk
2
I
Bog
2k
32
2
7 0²
1
11
17
I
M 87
+
A
A
02
Buz
B
02
k=0
2.0f
+ 1222
27 A
-1
1
7 A
1 2
I
T
Bait
I
I
I
I
02
02
2-0
2.0
1
no²²
1
I
E
TM -
88
+
2 Bab 32
1
Lemma 2
Q2
a2
11 A
2-04
1 17²0²
B1
I
02
TM 89
+
4 B
02
X
I
02
2-Mx
TM
4 B
11/002
03332
B1
k P
Σ
2
1]
k
A
2044
A
k
2-M,
1
I
A
ro
02
1
11700
2.M
P
237
I
A
I
A
A
4
B
0102
a
02
La=r+1
i=0
Note that the bias terms equal Tai m and the variance terms are mN 2 4(1 + Mc.. Dotting the terms into X gives us the upper bound from Theorem 1.
Lower bound. We now turn our attention towards proving the lower bound in Theorem Note that the bias terms have an exponentially decaying dominating term. In order to obtain an inequality in the reverse direction for these terms, we compare m(y) with r(1.0ln). We begin with lower bounding m:
2
12002 d
1
mN.a
(1)
2
I
A
m +
Bac* ot
2
1 022
1 k
M
M 11:k-99 24 +
4 B
a1a2
2Nr
17700²
P -1
28/1
T
I
MN
-
A
x
4 B
0102
a
1=0
2N,
7
I
mg
a
1
0
2N,
P
1) A
2
+
I
I
4
B
0102
-=++1
i=0
Now we need to establish an upper bound for r(1.01n). We follow a similar analysis as we did for the upper bound subsection:
16

---

Published as a conference paper at ICLR 2026
Mis (1.01n)
1.01n A
2
- + 1.013 1 2c)
22
Ba 5/
1.01n
23
11002
I
A
=f 2201. (1 + 2c)
X
a
B
0102
2/3
1/02
I
" A
TM of +2 1.01 (1 + 2c)
A
Lemma
B
0102
2NN
q²0²
k P P--1 1
28
1
A
- +2. (1 + 2c)
Σ 1=0
1
7
A
A
B
(100
2N.
I ai
A
ro
10222
2
k
k
2N,
P. 1
2,55
+ 1.01 (1+ 2c)
II
71
A
Σ
rJ
A
X
0102
a
1
Comparing the bias and variance terms gives us the conclusion.
17

---

Published as a conference paper at ICLR 2026
B
NORMALIZED SGD ANALYSIS
Under the setup introduced Section 5, we have the update rule for normalized SGD is:
Wi+1 = W.
EE
g+
where gr 1 Σ.= Bi (6)
for i indexing the sample and batch size B.
For MSE and y (w*) + e, the loss is:
B
[(wi) = 2B
Σ
(w (1
(0)
i=1
B
(w: -w')==" w=) - c)3
2B
==1
If we look at the risk at time we have:
1
R(w))
B[(w; (w w*) x(() (w; -w") c²]
(7)
2B
1
B
B[(W: xx(1) (w, - -w")] + a²
(8)
2B
2
iml
B[(we wT) xx" (we - -w")] + o²
(9)
2
2
Tr(HE,) + a²
(10)
2
So the risk is equal to:
2 Tr(HE,) + a² R(w.) - R(w") TT(H)
1
R(w:)
2
2
Analyzing the gradients Taking the gradient for sample:
V w. L(+) (w, x(() xx x(1) (x(0) w*) - (x())
gi
So we have:
1 B
Σ (0) (x(0) (w, - w*) 1
Σ xx
g:
Moving forwards, need calculate the term in the denominator. Skipping the time index when convenient in order to simplify the notation, and writing ò; == w, w', we compute E||g|| by conditioning on w, and then applying the tower rule:
E Elgl|
E
|gl
w.]]
B Σ
E[|g||
2
wo]
LE 16 (1)
(xxx
(x To wr]
B
j=1
1
Σ
8 (6) ( w.].
B
18

---

Published as a conference paper at ICLR 2026
First term Let H := E[xx'] | For i * independence gives
E x (9) (x x (x())) =H² H².
For = j. assume
(N H), so that the Gaussian fourth-moment identity yields
E|xx xx ] Tr(H)) 2H.
Therefore,
E o x (xe (x) xx (x(()) Toc wi] = =6, H ( - (1p)) + 6.y (T + 2H²) 6r.
)o.
Summing over i yields
B
B
E
x 1
(x(0)
xx(0 (x(0)) w.]=o= ( (Tr(H)H + 2H²) + -11 B H²) H²) 6z
B
= =6,
(
( Tr(H)H + (1+ 1 T3 H2 )o
B
Second term
Using Bie [03 o2drz and ()( Tx(()) = Tr(H) we get
B
EE ( (1) (x) xx())
wi] = B Σ
B
[2 x( 0² (T(H))
B
B
So altogether, conditional on Wt
[[1gg] 2 wi] = o7 Tr(H)H (1 (+11)) H oz + d² Tr(H)
B
B
B
Taking the outside expectation and letting E, =- E[ we obtain
E Elgg 12 Tr( ( MHHHH + (1+ B H E Tr(H)
B
(H))))((H)))) (1+ TT( + +o² Tr(H)
B
Since E, ((02) (Lemma 8) (Jain al., 2018), then for large enough t. we have that the gradient norms are dominated by the additive variance, which is captured in Assumption 3. For the remainder of this paper we will assume is large enough for this assumption to hold, and with an abuse of notation we will write (as opposed to R):
E E
TT)
B
Under Assumption 3, we have the following update rule:
B
W++1 = Wt - 0 (r(H)
Vw, L
(11)
Note that this is simply SGD with a learning rate 7j = 1)
/B
B. HOw AGGRESIVE CAN THE SCHEDULER BE?
In this section we provide a short lemma explaining what is the most aggressive scheduler we could possibly used, based on hard contraints on a, B.
Lemma (Divergence conditions.). Suppose we are in the same setting as Corollary 1. For a fixed initial learning rate 1,, the training dynamics diverge asymptotically if a vB as the training time goes to infinity, for and 3 constants independent of time.
/B² Note that vB > o, then at every cut we are effectively increasing the learning rate. Thus, there must exist 0 such that 7j > 7maax where max is the maximum convergent learning rate for SGD (Wu et al., 2022b; Jain et al., 2018), leading to divergence.
Proof. To see this, we focus on the scaling of îją: = 17
19

---

Published as a conference paper at ICLR 2026
C
COMPARISON TO OTHER SCHEDULERS
We compare our scheme with other schedulers this section.
Batch Size: 128
Batch Size: 256
3.20
3.20
5,0. 2
3.15
Validation Loss
3.15
n/2,
Validation
n/v
3.10
3.10
3.05
3.05
3.00
3.00
109
x 10 Tokens
3 10"
10a
2 10 Tokens
x 10
Figure 4: 150M models trained with different schedules, at CBS (right) and just below (left). Blue trace keeps learning rate fixed and doubles batch size, orange trace keeps learning rate fixed and quadruples batch size, green trace halves learning rate at fixed batch size, and red trace is Scesaw. Note that the naive scheduling (blue) severely underperforms the baseline (green) and Seesaw (red).
20

---

Published as a conference paper at ICLR 2026
D AUXILIARY LOSSES
In this section we ablate over the effect of z-loss on the training dynamics (OLMo et al., 2024). We observe no difference in the training stability our models at 150M scale in Figure 5;
Learring rate: 0.003
Leamiing rate: 0.03
Learning rate: 0.03
5.50
3.2
5.00
4.75
4.7
6.30
6.00
2.73
3.75
2.50
3.53
350
345
5.23
5.5
3.50
1.00
3.00
Tooess
14kms
Figure 5: 150M models trained with cosine decay in Chinchilla scale, across 3 learning rates and batch sizes. Note that the final validation losses are equal whether Z-Loss is enabled or not.
However, while the final validation loss does not change as an effect of z-loss our scale, we have observed certain instabilities in the z-loss towards the end of training when using Seesaw in Figure 6. We speculate that the way we are scaling the learning rate and batch size might not the proper way to do it for 7-loss, and we leave this study for future work.
0.0025
0.0020
7-1065
0.0015
0.0010
0.0005
0.2
0.4
0.8
1.0
Takens
1c10
Figure 6: 600M models trained with Seesaw decay in Chinchilla scale, with Z-Loss.
21

---

Published as a conference paper at ICLR 2026
E WEIGHT DECAY
In this section we provide experiments on 150M models trained with AdamW, sweeping weight decay À E {0.000001, 0.00001, 0.0001, 0.001, 0.01, 0.1, and learning rate "J {0.001, 0.003, 0.01, 0.03), and the rest of the parameters are as explained in Section 4. For ev- ery figure we pick the best (n. pair on cosine annealing, and we use the values for Seesaw. Across all batch sizes (128, 256, 512), the optimal (1) A) pair from the sweep turned out to be (n. A) (0.003, 0.0001). Figure shows the results:
Size:
Batch
Batch 512
Schooakr
3.25
Sceeaw
1:23
3.20
Oriane Douay
3:23
2.25
2.55
3.15
2.30
2.35
107
3.20
2
102
20
1'7 3.00
30
20300
Figure 7: 150M experiments with weight decay across different batch sizes (128, 256, 512) for cosine annealing and Seesaw, for learning rate and weight decay values (n, X) (0.003, 0.0001). Note that the losses overlap during training. We provide the final validation losses in Table 3.
Table 3 shows the final validation losses:
B=128
B=256
B=512
I 15MM (cosine)
3,0125
3.0220
3.0559
1 50M (Seesaw)
3.0027
3.0210
3.0588
Table 3: Final validation losses picked at the best learning rate (for the cosine annealing scheduler) for each batch size, for a = 1.1 and weight decay 0.003. Note that the dynamics match robustly.
22

---

Published as a conference paper at ICLR 2026
F OVERTRAINED RUNS
In this section we provide experiments for 150M models in the overtrained regime. We train for 4x Chinchilla (so approximately 13.2B tokens), while sweeping over learning rates and batch sizes in the same range as Section 4. We show in Figure 8, and the final losses of these runs in Table 4, where the plots are done at the optimal learning rate for cosine.
Batch
532
1024
Schedider
Cosine
Seessw
3.6
Waldbttin MOS
3.4
3.2
3.0
2.8
Figure 8: Seesaw comparison with cosine decay in 150M models trained at 4x Chinchilla scale. For more experimental details, Section 4. Note that the schedulers agree in the final losses, with the actual values shown in Table 4.
B=256
B=512
B=1024
I 15M (cosine)
2.8762
2.8814
2.8990)
150M (Seesaw)
2.8724
2.8820
2.9016
Table 4: Final validation losses for 150M models trained at 4x Chinchilla.
23

---

Published as a conference paper at ICLR 2026
G ADDITIONAL FIGURES
8m256
3.100
5.075
3.04
3.04
3.02
3.02
3.075
2.8
2.6
2.8
3.0
3.2
Tikens
103
Thions
102
Thi
10
Figure 9: 150M models trained batch size 128, 256, 512 with and values following the line of equivalence a / = 2 described in Table 2.
10
Validation Less
Volidation
0.50
0.25
0.50
0.75
1.50
1.25
4.0
Tokens
169
4.0
Tolons
1c9
4.0
Tokees
1610
3.8
Valdwwem Loss
Velidaekmm LeNS
3.4
3.4
32
3.2
3.2
3.0
3.0
5000
10000
15000 Stops
20000
25000
10000
20000
30000
40000
50000
10000
20000
30000
40000
50000
Staps
Stops
Figure 10: Seesaw comparison with cosine decay in 150M (left), 300M (middle) and 600M (right) models trained at Chinchilla scale. The validation losses at the end of training are provided in Table For more experimental details, see Section
0.22
Caiira
0.0
3.30
3.05
0.8
0.5
Scretslo
Sentaw
0.52
Sanma
Tahoro
Traro
Takro
Figure 11: Downstream evals comparison with cosine decay in 150M models between Seesaw and Cosine decay trained CBS (256) for 1 x Chinchilla, the optimal learning rate for cosine. Note that the methods have similar performance. The shades represent standard deviations over 5 seeds, taken due to the noisy nature of the evals.
24


Reviews:
[{"review_id": "j0dyTCpolX", "reviewer": "Reviewer_r7Qn", "summary": "The authors present a theoretically grounded approach to batch size scheduling, specifically showing that decay schedules can be replaced by a combination of decay schedules and batch size schedules (e.g. instead of decaying by a factor of 2 you can scale LR by 2 and decay by a factor of sqrt(2)). This allows you to either exploit more hardware or (more realistically) exploit the hardware you have more effectively.\n\nThey conduct small experiments up to 600M \"Chinchilla\" that demonstrate that the theory works out, showing both that their schedule is equivalent and also that a more aggressive schedule (in terms of higher effective LR) won't work, or works less well.", "questions": "Clarification about Lemma 3's relationship to Merrill et al 2025 would be helpful. I feel like I'm missing something.\n\nsome experiments with actual weight decay would be helpful since that is how LLMs are usually trained. \n\nI'm also curious about behavior at sub-CBS (e.g. 150M with B=128).\n\nIs it possible to exploit the empirical observation from Merrill et al 2025 that their measured CBS seems to increase as training progresses? Or is your contention that their estimate of CBS is inherently flawed?\n\nrelatedly, it would be nice to see this holding for the overtrained ( say 4x Chinchilla) regime too. I wouldn't expect to see all scales to a multiple of chinchilla. Given that many real models are overtrained, it would be nice to see. (This may be asking too much and that's ok.)", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall I think this is a nice paper: It is generally very well written. It's a nice example of trying to deploy actual theoretical insights with practical advice, and I think it's a fairly creative way of going about it. I think one could have gotten to the same conclusions from older SDE theory about Adam LR's relationship to batch size, but it's good to have this new approach too.\n\na nice extension of the theory from Meterez, et al 2025 (and others) coupled with some small but very convincing experiments demonstrating the idea. \n\nThe experiments show that their theory holds up remarkably well.\n\nThis isn't a world-shattering paper but, modulo my concerns, it is a solid contribution and one i'd be keen to test myself.", "weaknesses": "It is easy to say that it would be nice to see bigger experiments, but that doesn't feel necessary here.\n\nOne small thing is that the authors appear to have moved their theoretical results to the end of the paper late in the drafting process: assumption 3 is referenced in 3.1 and 4.2 but not defined until section 5. This is confusing but easily remedied.\n\nI think it's a little much to claim a 36% \"wall clock\" speedup since it would need ~proportionally more compute for those phases (or considerably smaller wall clock gains from improved MFU)... I get the point that it's possible to use more compute in those circumstances when you might otherwise be constrained by CBS (though there are other ways to do that too through model parallelism)\n\nI am quite confused by how Lemma 3 is a refutation of the approach proposed by Merrill et al... They're trying to hold effective LR constant. They increase B by \\beta = 2, which would reduce the effective LR by sqrt(2) (consistent with your analysis of NSGD and other prior results with SDE etc.) But they then increase the LR by a factor sqrt(2), and so alpha=sqrt(2), and so the effective LR is held constant? Right?\n\nThe absence of weight decay in the experiments makes sense given its weird interaction with LR, but it's not realistic. In particular, decoupled weight decay means LR impacts steady state of the weight norms, and so you would end up with different results... Does the theoretical analysis transfer to this more realistic setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OUxbsQZQiS", "reviewer": "Reviewer_fg72", "summary": "The proposed SEESAW scheduling method replaces standard learning rate halving by concurrently multiplying the learning rate by \\sqrt{\\frac{1}{2}}  and doubling the batch size. This reduces serial steps while preserving the loss trajectory. Experiments on 150M-600M models with C4 and Chinchilla scaling show SEESAW matches cosine scheduling's final loss while achieving near-theoretical-maximum speedups.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper establishes a non-asymptotic equivalence between learning rate decay and batch size growth under SGD, extending it to NSGD via an equivalence family where the product \\alpha \\sqrt{\\beta} is conserved. This links theoretical insight to practice, forming an actionable framework for designing training protocols.\n\n2. The proposed algorithm features a remarkably simple structure and achieves true zero intrusion, meaning it can be seamlessly integrated into existing training pipelines without requiring any modifications to the model architecture, optimizer, or other components. Its plug-and-play nature makes it highly accessible and easy to adopt in practice.\n\n3. The theoretical upper bound derived in the paper aligns closely with experimental results.", "weaknesses": "1. The theoretical derivation relies heavily on Assumption 3; however, Figure 3 shows that as the batch size increases, Seesaw diverges from cosine scheduling, indicating a fundamental limitation in the method's applicability when this assumption breaks down.\n\n2. The paper lacks comprehensive experimental validation across a broader range of conditions. It evaluates only three medium-scale models on a single dataset and omits comparisons across diverse downstream tasks, additional optimizers, or extensive hyperparameter settings. More extensive experiments would be necessary to convincingly demonstrate the generalization and robustness of the proposed method.\n\n3. The notion of acceleration is measured in terms of reduced serial steps rather than actual wall-clock time. While the paper reports the ratio of achieved serial step reduction to the theoretical limit, it does not provide end-to-end wall-clock evaluations under realistic distributed training scenarios, including cross-node throughput, communication overhead, and memory constraints. Furthermore, extreme members of the proposed equivalence family exhibit instability in practice, raising concerns about their usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2l5EqUOqE", "reviewer": "Reviewer_nmrR", "summary": "This paper proposes Seesaw, a principled batch size scheduling algorithm designed to reduce wall-clock training time for LLM pre-training. The key insight is to replace LR decay with slower decay and increasing batch size. Specifically, when a standard schedule would decrease LR by factor $\\alpha$, Seesaw instead decays LR by $\\sqrt{\\alpha}$ while increasing batch size by $\\alpha$.\n\nThe authors motivate this rule through theoretical analysis of SGD on noisy linear regression, extending to Normalized SGD (NSGD) as a proxy for Adam. Under a \"variance-dominated\" regime, analysis on NSGD results in the Seesaw scheduler. Experiments on 150M-600M parameter models at Chinchilla scale demonstrate ~36% wall-clock time reduction while matching baseline validation loss.", "questions": "1. Could the authors elaborate on why NSGD was chosen as a proxy for Adam, given that recent literature increasingly favors SignSGD as a more faithful conceptual model? Would a theoretical analysis based on SignSGD yield the same $\\sqrt{\\alpha}$​ scaling rule, or would different dynamics emerge?\n\n2. The theory analyzes the specific case of a factor 2.0 decay. Could the authors provide a proof for the general case of decay by factor $\\alpha$?\n\n3. The practical implementation discretizes the continuous cosine decay (e.g., using $\\alpha=1.1$ in Table 1). How sensitive is the method's performance to this discretization factor? Would finer-grained approximations better track the cosine curve, and are there practical trade-offs to consider in choosing $\\alpha$?\n\n4. Could the authors revise Figure 1 to use consistent linear scaling for tokens to enable fairer visual comparison?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Strong empirical results.** The paper's main contribution is its compelling empirical validation: Seesaw achieves approximately 36% reduction in wall-clock training time while matching the baseline's final validation loss. For practitioners with access to parallel compute resources (for processing larger batch sizes), this translates directly to significant cost savings. Another advantage is its simplicity, as it can be implemented as a straightforward drop-in replacement for standard learning rate schedules.\n2. **Principled derivation of a simple heuristic.**  The authors bridge theory and practice by grounding their scheduler in a principled theoretical framework rather than relying on pure empirical tuning. This approach elevates batch size scheduling from ad-hoc experimentation to a more systematic methodology.\n3. **Rigorous validation of theoretical predictions and failure modes.** The evaluation of where its theory succeeds and where it breaks down strengthens the paper. Figure 2 validates the stability condition derived in Lemma 3, confirming that overly aggressive schedules ($\\alpha < \\sqrt{\\beta}$​) lead to performance degradation as predicted. Figure 3 demonstrates the failure of the \"variance-dominated\" assumption at very large batch sizes, where Seesaw no longer tracks the baseline. This transparency about limitations adds credibility and helps practitioners understand the method's applicability boundaries.", "weaknesses": "1. **The NSGD proxy inadequately represents Adam.** The choice of Normalized SGD appears driven by mathematical tractability rather than fidelity to Adam's behavior. NSGD's global L2 normalization differs fundamentally from Adam's coordinate-wise adaptivity. A strong consensus in recent literature suggests that SignSGD, which respects the sign-based nature of Adam's updates, is a much better conceptual proxy. Since Seesaw's square-root scaling rule is specifically tailored to the analysis on NSGD, it's unclear whether similar insights hold for SignSGD or Adam. The paper should explicitly acknowledge this limitation and discuss why the NSGD-derived heuristic succeeds despite this mismatch.\n\n2. **Oversimplified theoretical foundation.** The noisy linear regression setting vastly oversimplifies LLM training's non-convex, high-dimensional landscapes. More critically, the \"variance-dominated\" assumption (Assumption 3) appears overly strong and may only hold near the end of training. The paper does not sufficiently justify why analyzing this simplified setting provides a good proxy for LLM training dynamics, even though the empirical results suggest the heuristic works in practice. \n\n3. **Performance degradation at large batch sizes.** While overall results are strong, Table 1 shows Seesaw slightly underperforming the cosine baseline at large batch sizes (e.g., $B=1024$ for all 150M, 300M, 600M models). This gap along with Figure 3 suggest diminishing benefits as batch size grows beyond the variance-dominated regime. This ceiling on applicability could limit the method's utility for larger-scale training runs in practice.\n\n4. **Proof presentation and limited technical novelty.** The proofs in Appendix A suffer from unclear notation. Multiple symbols (e.g. $Q$, $\\lambda$, $\\Lambda$) are used without proper definition, making the derivations difficult to follow. Presumably these relate to an eigendecomposition $H=Q\\Lambda Q^T$ with $\\lambda = \\text{diag}(\\Lambda)$, but this is never explicitly stated. Additionally, the theory specifically analyzes a factor of 2.0 step decay but does not provide a proof for general drops by factor $\\alpha$, limiting its generality. Beyond these presentation issues, the technical contribution itself is limited—the proof is straightforward and follows the same approach as Meterez et al. (2025).\n\n5. **Inconsistent visualization in Figure 1.** The figure uses log scale for tokens but linear scale for steps, which is inconsistent and potentially misleading. The log scale can obscure small but meaningful differences in sample efficiency, while the linear scale visually exaggerates the speedup. Using consistent linear scaling for both plots would provide a more transparent and fair comparison of the method's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHyRGbBuED", "reviewer": "Reviewer_gioo", "summary": "This paper introduces a technique to trade off a decrease in LR for an increase in batch size for LLM training.  The recipe is an application of the square-root scaling rule previously proposed for Adam training: whenever you double the batchsize, you should increase the LR by \\sqrt(2).  Here, the idea is that when we would normally drop the LR by 2 according to its decay schedule, we immediately apply the square-root scaling rule, with the net effect that the LR gets decreased by only \\sqrt(2), and the batch size is doubled.  Experiments on 150M, 300M, and 600M models trained to a compute-efficient 20 tokens-per-parameter show this approach can equal the loss of the baseline recipe, but using fewer steps (which you can calculate in advance).", "questions": "- Suppose I apply the Merrill et al approach at exactly those points where I drop the LR by 2x.  That is, I double the batch size, and then scale the LR back up by √2 (after dropping it by 2).  So the net change to the LR is to change by √2/2 = 1/√2.  Doesn’t this result in the exact prescription from this paper: whenever you would drop the LR by 2x, you instead double the batch size and decrease the LR by 1/√2?  If so, is the key difference from the Merrill approach just the timing of when you do this adjustment?  How does their proposed timing differ from yours?\n\n- For Figure 3, what does “Seesaw” mean in terms of α and β?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "It makes sense to revisit ideas like \"Don't decay the LR, increase the batch size\" in the context of modern LLMs.  Companies are competing to train the next generation of models as quickly as possible; if we can train faster, without spending additional FLOPs, it's of definite benefit.  \n\nTesting a simple recipe, as in the paper, is definitely the next step; if it works, people can immediately apply it.  Moreover, the ability to deterministically calculate the number of steps in advance is useful for resource planning.  I.e., the schedule is not based on, e.g., online measurements of the gradient noise scale as in https://arxiv.org/abs/2411.00999, but rather are motivated in advance from the LR schedule.\n\nI would absolutely encourage the authors to keep working in this direction!", "weaknesses": "In its current form, the paper is only half-baked, in terms of writing, experimental rigor, theory, awareness of prior work, etc.  The paper gave me the feeling that the ICLR reviewers were doing the work of proofreading the paper, rather than mentors or co-authors, which was unsettling.\n\n- The idea that Adam is basically normalized SGD, and that this necessitates dividing the learning rate by the square root of the batch size adjustment, was previously articulated in, e.g., in \"Hilton - Batch size-invariance for policy optimization - 2110.00641v3\", where (citing Hardin 2017), they similarly note \"Adam divides the gradient by a running estimate of the root mean square gradient\".  Hilton et al. divide this estimate into the square root of {gradient-mean-squared plus gradient-variance}, and show how the batch size reduces the variance, and essentially leads to the square root LR adjustment.  Since this paper wasn't discussed or cited, I'm not sure what the connections are, or the extent to which the theory as presented goes beyond this.\n\n- Regarding soundness, I was concerned why we tested the extreme values of the equivalence (Figure 2) at 2x the CBS?  Especially since they're much closer at 256, it makes me suspect that they're even closer at 128, yet this wasn't presented.\n\n- Also regarding soundness: I just feel there isn't the depth and breadth of experiments that we would normally find in an ICLR paper.  Beyond this, I feel like there's not enough depth and breadth to convince me to use this approach in my own training.\n\n- Limitations were not discussed, e.g., not acknowledging that state-of-the-art models are trained with non-zero weight decay, to higher tokens-per-parameter, etc.  That this was only on one dataset with one tokenizer with one optimizer, etc.\n\n- The claim in the abstract that we \"*approach the theoretical limit implied by our analysis*,\" is misleading, as approaching this limit is not, like, a sign the model is training well or whatever, but just that the continuous-time approximation that you used returns a number that is close to the actual number of steps, right?  I mean, you could run a simple dry-run simulation to exactly determine how many steps the model will take given your algorithm, it's not like \"abstract-level-ITALICIZED-claim\" significant that your continuous-limit version provides a close answer, right?  The significant thing is that the losses match, but, like I said, that's got nothing to do your continuous-time version.  Unless I'm really misunderstanding something, which is possible, because...\n\nThe paper is poorly written and organized, and not from like a non-native-speaker grammar perspective, but from a thinking-and-planning-out-the-paper-clearly perspective.  Some points on these lines:\n\n- It's confusing as heck to use α and β to represent the adjustments to the LR and batch size --- used in the product that should remain constant --- and to use them as actual values that get substituted into this equation, essentially α := √α, β := (√α)^2. You know what I mean?  Like, setting β = α is needed in order to satisfy α = √β.  It's almost non-sensical.\n\n- \"where the schedulers are equivalent in terms of loss as long as we keep the product α√β fixed\" – what schedulers???  The one where B doesn’t change and the one where it does?\n  - Let’s use α and β to be the adjustments to the LR and batch size, respectively.\n  - Case 1: α = C and β = 1.  I.e., the points where the LR schedule would drop by C and we don’t change the batch size.  α√β = C * √1 = C.  (e.g., C=2 would give us the abstract of the paper)\n  - Case 2 (proposed): α = √C, and β = something.  To maintain the invariant, β = C, only this way will α√β = √C√C = C as before.\n  - As far as I can tell, that’s our only option for β in order to keep the product fixed.  But then we say, of all the ways we could adjust the batch size, i.e., of all the β values, the most aggressive we can use must satisfy α = √β, i.e., β = α^2.  So if α = √C, β = C.  But isn’t this our only option to have an equivalent scheduler?\n  - I REALLY wanted to understand what you’re saying here, but I just could not.  Maybe on a second reading of the paper... but I'm just a reviewer, make my life easier please!\n\n- Not enough context prior to the experiments\n  - E.g., there's a section on “Assumption 3”, but this assumption is only mentioned parenthetically before this point\n\n- I don’t really understand why all the content that was collected into Section 3 is in there:\n  - Intuition\n  - Theoretical results, with lots of pointers to other parts of the paper.\n  - The formal algorithm\n  - Like, you say, “the NSGD update rule, which is a crucial component of designing Seesaw”, but it seems to me that using a simple square-root scaling rule would have been sufficient here.\n\n- When you present your main findings (Table 1), maybe provide some interpretation?  Like, what is the take-home message here?\n\n- Too much required for understanding is in the appendices.\n\nI could go on.\n\nNitpicks:\n- For the Taylor expansions, it would have been helpful to define x_0, x_1 and x_2, and to define the noise terms precisely.  Should there be some expectations here?  Can you cite the specific section of Malladi that “shows this argument”?\n- Might be worth pointing out somewhere that actually changing the batch size in a fine-grained way is challenging for modern large-scale LLM GPU deployments… although perhaps not on other hardware…\n- “Understanding batch size ramp up schemes during training has been a topic of interest in recent years”.  Really?  I mean, people have used it, but has \"*understanding* ramp-up\" been a topic of interest?\n- Wrong use of \\citet vs \\citep, e.g., “Recently, (Meterez et al., 2025) have used” should instead by \\citet\n- Typo: “We further empirically comapre* Seesaw”\n- Table 1: “Note that the dynamics match” --- do you mean the final losses?  Might be good to highlight (e.g., color/italicize/shade/bolden) the CBS cells somehow.\n- Says \"trained using AdamW\", but then weight decay = 0.0, so isn’t this just vanilla Adam?\n- For the CBS, do you get the numbers directly from Zhang, or do you use their power law estimates based on tokens, or something else?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)