MT-DAO: Multi-Timescale Distributed Adaptive Optimizers with Local Updates
Authors: Alex Iacob
Andrej Jovanovic
Mher Safaryan
Meghdad Kurmanji
Lorenzo Sani
Samuel Horváth
William F. Shen
Xinchi Qiu
Nicholas D. Lane
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed Training
Foundation Models
Large Language Models
Optimizers
Communication Efficiency
Federated Learning
Distributed Systems
Optimization Theory
Scaling
Primary area: foundation or frontier models, including LLMs
TL;DR: MT-DAO, a multi-timescale optimizer, closes the performance gap from infrequent communication in distributed training. It cuts wall-clock time by 6-27% and allows a 720M model to reach its target 35% faster with 5-24% fewer steps than standard DDP.

Abstract: Training large models with distributed data parallelism (DDP) requires frequent communication of gradients across workers, which can saturate bandwidth. Infrequent communication strategies (e.g., Local SGD) reduce this overhead but, when applied to adaptive optimizers, often suffer a performance gap relative to fully synchronous DDP.
We trace this gap to a time-scale mismatch: the optimizer's fast-moving momentum, tuned for frequent updates, decays too quickly to smooth gradients over long intervals, leading to noise-dominated optimization. To address this, we propose MT-DAO, a family of optimizers that employs multiple slow- and fast-moving first momenta or the gradient to track update dynamics across different time scales, for which we provide the first convergence guarantees. 
Empirically, for language-model pre-training, this eliminates the performance gap with DDP, outperforming infrequent-communication baselines in perplexity and reducing iso-token wall-clock time by 6-27% on Ethernet interconnects. At the 720M scale, MT-DAO reaches a target perplexity in 24% fewer steps and 35% less time than the single-momentum DDP baseline.  MT-DAO enables effective cross-datacenter training and training over wide geographic areas.

Published as a conference paper at ICLR 2026
MT-DAO: MULTI-TIMESCALE DISTRIBUTED ADAPTIVE OPTIMIZERS WITH LOCAL UPDATES
Alex lacob f.1,2
Andrej Jovanović*
Mher Safaryan
Meghdad Kurmanji'
Lorenzo S.
Samuel Horváth*
William E. Shen!
Xinchi Qiu!
Nicholas D. Lane
ABSTRACT
Training large models with distributed data parallelism (DDP) requires frequent communication of gradients across workers, which can saturate bandwidth. In- frequent communication strategies (e.g,, Local SGD) reduce this overhead but, when applied to adaptive optimizers, often suffer performance gap relative fully synchronous DDP. We trace this gap to time-scale mismatch: the optimizer's fast-moving momentum, tuned for frequent updates, decays too quickly to smooth gradients over long intervals, leading to noise-dominated optimization. To address this, we propose WTTDDAO, family of optimizers that employs multiple slow- and fast-moving first momenta the gradient to track update dynamics across different time scales, for which we provide the first convergence guarantees. Empirically, for language-modd model pre- pre-training, this eliminates the performance gap with DDP, outperforming
baselines in perplexity and reducing iso-token wall-clock time relative to DDP by 6-27% Ethernet interconnects. At the 720M scale, DAO reaches a target perplexity in 24% fewer steps and 35% less time than the single-momentum DDP baseline. MAODAO enables effective cross-datacenter training and training over wide geographic areas.
INTRODUCTION
The scalability of training infrastructure is impeded by the communication required for Distributed Data Parallelism (DDP). Infrequent parameter-averaging strategies like Local SGD (Stich, 2019) reduce this overhead, yet extensions to adaptive optimizers (Cheng & Glasgow, 2025; Charles et al., 2025) show a performance gap relative DDP (Sani al., 2025; Charles et al., 2025). Charles et al. (2025) finds that infrequent averaging, even with Nesterov momentum at round boundaries (Reddi al., 2021), underperforms DDP for models up 2.4B parameters and worker counts exceeding 2.
We hypothesize this gap stems from a timescale mismatch. Optimizers use fast-moving momenta (low 31 ss 0.9) that smooth high- feghfrreuency noise under DDP but decay too rapidly between infrequent synchronizations. This decay prevents a stable shared trajectory, leading to our central question:
Can distributed adaptive optimizer with B's suited for infrequent communication close the performance gap with DDP while providing convergence guarantees?
We propose MT-DAO, which brings multi- momimnum optimizers (Lucas et al., 2019; Pagliardini et al., 2025) to the distributed, crrrquuentcommmuiicttoo regime. MI-DAO resolves the mismatch by using slow-moving momenta (e.g.. | 0.999) to preserve trajectory information across synchronizations while remaining responsive via a fast momentum. In its simplest quasi- hyauriblypebooicc form (Ma & Yarats, 2019), MT-DAC uses the current gradient as the fast momentum, adds no memory or communication overhead, and requires only one additional hyperparameter. Crucially, unlike methods that use
faa 1 302 cam..amm.a.mm ac.uk; 'Equal contributions; University of Cambridge; Flower Labs; *Institute of Science and Technology Austria; "Mohamed hin Zayed University of Artificial Intelligence

---

Published as a conference paper at ICLR 2026
nomentum-based outer optimizer (Reddi al., 2021; Douillard et al., 2023) at synchronization boundaries, MT-DAO needs no extra memory buffers or multiple outer hyperparameters.
Empirically, slow momentum acts as regularizer, improving update alignment by increasing cosine similarity between worker pscudo-gradient (Reddi et al., 2021). This stability lets MT MT-DAO improve perplexity over low.communication baselines. Furthermore, MT-DAO matches exceeds its DDP analogue at larger scales, closing the perplexity gap for models up to 720M parameters. We validate these findings across six inner optimizers (ADOPT, Adam, Muon, AdEMAMix x, AggMo, and QHM) at scales up to 1.3B, and show that the benefits extend to streaming communication regimes (Appendix H.4), varying numbers of workers and batch sizes (Appendix H.3), and multiple momentum counts (Appendix H.2).
Contributions
1. A Provably Convergent Multi- Timescale Framework. We introduce MT-DAO, the first frame- work to integrate multi momentum strategies into distributed settings, with convergence guarantees for heterogeneous momentum timescales and synchronization frequencies.
Closing the Performance Gap Efficiently, MT-DAO matches synchronous DDP, outperforming baselines in perplexity, reducing wall-clock time by 6-27%; 720M reaches targel perplexity in 24% fewer steps and 35% less time than a DDP baseline. We demonstrate these gains across six inner optimizers and scales to 1.3B, including downstream task cvaluation.
3. Noise Suppression and Information Retention. MT MT-DAO's slow momentum preserves mutual information across rounds and reduces inter-worker momentum variance.
4. Resilience to Infrequent Communication. MT-DAO lowers the rate of change of parameters and momenta, improving tolerance low communication frequencies.
5. Aligument of Worker Trajectories. MT-DAO increases cosine similarity local worker update trajectories, which reduces worker drift and aligns the overall model update.
2 MULTI-TIMESCALE DISTRIBUTED ADAPTIVE OPTIMIZERS (MT-DAC
= 0.9099
302 09:99
10-11
00
-00
120
Clomuniiciiio
(a) Distance to optimum vs. steps.
(b) Contour plot of trajectorics.
Figure 1: To highlight the stability benefit of MT-DAO, we illustrate its performance on toy non- convex problem. Crucially, under a high momentum decay Local Adam (Cheng & Glasgow, 2025) become unstable and fail to converge, whereas MT-DAO maintains its rapid and stable convergence. We optimize the non-convex Rosenbrock function f(21, #2) = (1 )2 + 100(x2 zf)² with M 256 workers and IID Gaussian noise (a 2).
0.9999, prior stateful methods like
We characterize the conflict between optimizer momentum timescales and communication intervals. Let M workers perform K local updates per round minimize f(x) 1 Σ fm (æ), where fm (x) = -e--p,,. (x; Performance degradation in this regime stems from mismatch between the optimizer's memory and the communication period.
2.1 TIMESCALE MISMATCH
The first momentum in adaptive optimizers is an Exponential Moving Average (EMA): u, -Bu4-1 0.5
(Pagliardini al., 2025). 9)),, with effective memory quantified by half-life (8.) In
2

---

Published as a conference paper at ICLR 2026
A typical 81 = 0.9 yields To.5 6.6 steps. suitable for frequent communication, A conflict arises when K > To.5, common in communication efficceent training (K E [32, 512).. Unrolling over K local steps gives U++K = =3 u + (1 Ex=o 9**K- The influence of the global state ue decays as 3* (= 0.03 for 3100.9, K=32), leaving workers reliant on noisy local gradients. For example, if noise independent across workers, the variance of the final local momentum Var(u++k) 1+3 (1 32k (see Appendix F), with of being the gradient variance of worker m. As B the factor 1-f suppresses variance. For 3 + O, variance approaches o exposing local updates to coise-induced instability. Importantly, while increasing the batch size B reduces on linearly, 3 controls variance exponentially through the drift term 11-32k). This means that in infrequent-commuiicati
T+
regimes, precise control of / is far more important than DDP settings, where the exponential time-decay is absent (see Appendix F.1 for the full derivation and Appendix H.3 for empirical validation).
An alternative interpretation of this memory decay is offered by information theory, which quantifies the preserved signal between the initial global momentum U, and the final local momentum U++K via their mutual information, I(U++K: Ue). By modeling the local updates UL+K = 3º + L, where L is the accumulated local gradient noise, a closed-forn expression can be derived when assuming Gaussian distributions for the states and noise with covariances Ey, and Σn respectively. The mutual information is T(U++ U2) = log det(T + 328 Ey, E (see Appendix F). As 3K + O, mutual information vanishes, implying statistical independence with respect to the momentum at the start of the interval. As 3 - 1.0, the initial signal is preserved.
linear process
2.2 THE CHALLENGE OF HIGH-B OPTIMIZERS
Although both arguments above encourage the use of large 3 values as a solution to this timescale mismatch problem, previous work has shown that high-momentum optimizers are often unfeasible in practice (Lucas et al., 2019). Without modification, they are insufficiently responsive to changes in the loss landscape and are prone to oscillations (see the top of Fig. 2), yielding subpar performance.
0.001
Freq Syne
Sync
Clbaat Pacaueetet (x)
Local SGD with Momentum, 0.9
Local SGD with Momentum, f 0.990
10-
-10-
300
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Clinit Pacammeee (A)
MT-DAO-SGDM (4) 0.95), 8
MT-DAO-SGDM (w) 0.95),
0.999
0-
300
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Communiication Rounds
Communicaion Rounds
Figure 2: Comparison of Local SGDM with standard momentum (top) and MT MT-DAO-SGDM (N = momentum, W1 = 0.95) (bottom) for the function f(; A) = xx with R for various parameters controlling the rate of change À and and sync frequencies (frequent: solid, infrequent/slow: dashed). For 3 0.9, Local SGD with momentum can quickly find the sole global optimum, which does not hold for non-convex functions, however, MT-DAO-SGDM converges despite the slow momentum dampening it. While both optimizers are stable at / = 0.9, at high momentum (3 = 0.999) Local SGD with standard momentum becomes unstable for high À while MT -ED---GGDM remains stable.
This instability motivates using multi multi-momentum methods (Lucas et al., 2019; Ma Yarats, 2019; Pagliardini et al., 2025). Such methods compose the optimizer update linear combination of slow and fast-moving first momenta, or the gradient in the case of Quasi-byperbolic methods (Ma & Yarats, 2019). This avoids common pitfalls of high-momentum methods by responding changes in the loss
3

---

Published as a conference paper at ICLR 2026
landscape via the fast momentum//raadent. Recent works (Pagliardini et al., 2025; Semenov et al., 2025) have shown that such optimizers can provide SOTA results, outperforming popular optimizers such Adam (Loshchilov & Hutter, 2019), Muon (Jordan al., 2024), and Dion (Ahn et al., 2025).
2.3
THE MT-DAO METHOD AND ALGORITHM
Algorithm IMT-DAO-Adane local bias correction omitted to save space.
Require: Model tensors, hyper-parameters
1:
zo E R [u__) E (IR"),, 8-1 R initial params, N momenta, second momentum
2:
(B1.))=1, 83 [0 1) decay rates for each momentum
3:
(ws])== [0, 1] convex combination coefficients for first momenta, > wy 1.0
P [v= - clipping radius, learning-rafe schedule
5: T.M E N4 - total optimization steps and number of workers
6:
K2,1K, LN K. K E (N+))+² communication periods for parameters and states
7: OuterOpt R + R" update params using an outer optimizer, averaging by default
Ensure: 3T; fu4-1 D VT-1
8: for each worker m: initialize 20 (u- h
9: for t- O,.... T,
do
10:
for all workers m = O.....M, M- in parallel do
11:
8t" <- (xx?"" p) for = N do
clipped stochastic gradient update momenta
12:
13:
u a. - Srzu/_ (1 31,)
b t if (1 mod K2 = 0) then E. [u]-m] else d.
syne every K2 sleps
15:
2nth - 8201 + (1 {g"
16:
8t t if (1 mod Ko 0) then Em 6m else o
every Ke steps
17:
Al"
(1 Ee= wy)a Ejrlwyy
form combined update direction
Yor 4e
18:
I141 Zt m
19:
(t mod K. 0) then (Bm else
syne a every Ka steps
Algorithm 1 formalizes MT-DAO for Adam (variants: ADOPT in Algorithm 2, SGDM in Algorithm 3). It uses N first-order momenta {u²} and one second-order momentum v, combined via weights {wy) (purple), with the current gradient receiving weight - - wnmpllmmmmniiii the QHM structure. OuterOpt represents any aggregator (e.g., FedAvg (McMahan al., 2017), Nesterov (Huo et al., 2020), FedOPT (Reddi et al., 2021)) 1)); we default to averaging per prior analyses (Cheng & Glasgow, 2025; lacob et al., 2025). MT MT-DAO-Adam reduces communication costs by K, j= K, + K. ) over DDP.
This generalized framework recovers previous distributed adaptive optimizers (Stich, 2019; Douillard et al., 2023; Cheng & Glasgow, 2025; Iacob et al., 2025). also introduces the first-ever formula- tions for provably convergent distributed variants of multiimmoeentm optimizers (Lucas et al., 2019; Ma & Yarats, 2019; Pagliardini et al., 2025). Figure (bottom) shows an example of MT-DAO- SGDM converging for both high and low 31 with quasi-hyperbolic formulation while the Local SGD with momentum averaging method fails for high 31. To highlight the stability of MT-DAO- Adam, Figure 1 illustrates its convergence
common toy non-convex problem (Pagliardini et al., 2025) under high momentum (3 = 0.9999), setting where prior provably convergent methods like Local Adam (Cheng & Glasgow, 2025) become unstable and do not reach the optimum. Beyond Adam and ADOPT, we empirically validate that MT-DAO is effective with Muon (Appendix H.7), AdEMAMix, x, and AggMo (Appendix H.2), and that it composes with streaming communication (Ap- pendix H.4). We recommend the robust default =0.999 W1 =0.98 for the quasi-hyperbolic variant, which transfers from 16M to 1.3B without re-tuning (Section 5.5).
CONVERGENCE GUARANTEES FOR -DA
This section provides a theoretical convergence analysis for the proposed MT -DAO approach using the SGDM optimizer. The analysis, detailed in Appendix D, relies on the following standard assumptions.
4

---

Published as a conference paper at ICLR 2026
Assumption 1 (Lower bound and smoothness). The overall loss function f: R - B is lower bounded by some f E R and all local loss functions fm are L-smooth:
& L|| - yll. for any v. y E Rdd
Assumption 2 (Unbiased noise with bounded stochastic variance). The stochastic gradient g" of local loss function fm computed by machine m is unbiased and the noise has bounded variance:
Big fm),,, Billg" Vm fm S o², for any x E R.
Assumption (Bounded heterogeneity).
For any x E IR,,, the heterogeneity is bounded by
1 M Σ
[m S G2 (()
These are standard assumptions in smooth non-convex optimization (Yu al., 2019; Karimireddy et al., 2020b; Wang et al., 2021; Yuan et al., 2022), covering homogeneous data as a special case (Ga = 0, B = 1). For analytical tractability, we model periodic synchronization every K steps probabilistic event. Model parameters are averaged with probability
1/Kg, the j-th momentum averaged with probability Pj 1/K3. The gradient treated momentum with 8 0.
Theorem 1. Let Assumptions 1, 2 and hold. Then, choosing the step size = min x where
def No
1/(AL max(Bw, 6. max(1, B 1))) with constants دزي 1-85" and vp def 4(1-p=l N
Bw def EØ=1
(1- (1-8,11-P) W
Pa
f=1
T- D3)F3
the average iterates Ze- = Em [x:""] of STTMT--AAAE DAD-SGDM converge with the following rate:
1J (f(zo)- - 2M
Lo²
+ 0 1+8
The derived bound in (2) achieves the optimal O(1/VT) asymptotic rate for smooth non-convex stochastic optimization (Arjevani et al., 2023). Distributed factors, such client drift and data heterogeneity, are contained within the siep constraint and the higher-order O D(//T) term, thus not affecting the asymptotic rate. The step size r is constrained by Ba and tb The dependence d = O shows that model synchronization frequency Ps is critical. The impact of momentum synchronization is nuanced: reducing a momentum's sync frequency increases its contribution to J.. but this is modulated by its decay rate 3y. This implies "slower" momenta (larger B3) more robust to infrequent synchronization. This analysis reveals a trade-off: large 33 values constrain the step size via 3. but reduce the communication penalty in v/. Furthermore, synchronizing only the model always (i.e., Ps = 1, Pj = 0) is algorithmically equivalent to synchronizing only the momenta always (i.e., Pz 0, Pj 1). In the boundary case where only model parameters are
synced (p.- = 1,P; = 0), & = 0 and the rate recovers that of mini-batch SGD (Liu et al., 2020).
4 EXPERIMENTAL DESIGN
Building on our analysis, our experimental design answers the following research questions:
RQ1
Does MT-DAC reduce momentum noise and preserve mutual information, predicted?
RQ2 Does MT-DAO better preserve task performance when decreasing communication frequency?
RQ3 How does MT MT-DAO perform against DDP and prior communication efficienn optimizers?
RQ4
How does slow momentum affect local optimization trajectories between synchronizations?
RQS How does MT-DAO impact downstream task performance vs baselines?
4.1
SETUP
Models and Data. We use peri-norm (Kim et al., 2025) GPT-style transformer models 16M, 125M, 360M, and 720M parameters (Table 2). The 16M model is used for hyperparameter sweeps and qualitative investigations, while the 125M and 720M models are used for scaling experiments and baseline comparisons. We additionally train 1.3B parameter model 8OB tokens using Adam
5

---

Published as a conference paper at ICLR 2026
to demonstrate uptimizer-agnosse scalability (Section 5.5). All models are trained with a sequence length 2048 on the SMolIM2 mixture (Allal et al., 2025). We evaluate all models using validation perplexity held-out 10% portion of the training mixture, and the 1.3B model additionally downstream zero-shot and few-shot benchmarks. For further details, please see Appendix B.
Optimizers and Tuning Methodology. We use the ADOPT optimizer, a variant of Adam whose convergence rate is independent of the second momentumm decay rate 3g and preserves perfor- mance (Taniguchi et al., 2024); we fix 32 = 0.9999 to isolate the first momentum dynamics (governed by B1 defaulting to 0.9 and w). For Adam we use (31, 82) (0.9, 0.999) by default as recommended by Semenov et al. (2025). We use the CompleteP parameterization for one-shot transfer of the learning rate (LR) from small to large models (Dey et al., 2025). For each combination of convex coefficients (w's) and momentum decays (9's), we tune the learning rate on the 16M model and transfer the optimal hyperparameters to larger models directly without ever re-tuning (w's) and (8's). To establish strong DDP baselines, we tune w. 91 parameters in the DDP setting and reuse them for MT-DAO. For complete details see Appendix B.2. We always quasi-hyperbolic MT -TT-DO DAO (N 1) which does not require additional memory and reduces comms costs by (R_ Ri + R. DDP.
over
Baselines. We compare MT-DAO against: the base optimizer (ADOPT/Adam) with DDP and DDP analogues to MT-DAO such as Quasi-hyperbolic Momentum (QHM) (Ma & Yarats, 2019). For communication-effiieent baselines we use the provably convergent and stateful Local 1 Adam (Cheng & Glasgow, 2025) approach. We also compare against using Nesterov momentum the outer optimizer (Charles et al., 2025). We evaluate ML performance for communication-eficceet methods under the same, fixed synchronization frequency. Unless otherwise stated we use K K K₁ K, = 32 steps, based on prior work finding a practical balance of performance efficiency (Charles al., 2025). We split the dataset in an IID fashion across 4 workers using H100 per worker.
We analyze flattened models/momenta 9; I' using several metrics. The relative change over K steps measured || s || 2222- To quantify the dispersion among M worker vectors, we compute the cross-worker variance, defined as M Em= |||m - 8 $122- The statistical dependency between two random vectors different timesteps captured by their mutual information, x(U++K; U.). Finally, we measure alignment between vectors using cosine similarity.
Other Metrics
5 EVALUATION
This section empirically validates MT-DAO, showing its slow momentum preserves information and aligns workers (Sections 5.1 and 5.4), which improves stability under infrequent communication (Section 5.2) and allows it to close the performance gap with DDP at scale (Section 5.3). We further demonstrate that these gains translate downstream task performance the 1.3B scale using Adam (Section 5.5). In total, our evaluation spans six inner optimizers, model scales from 16M to 1.3B, and 235 additional experiments detailed in the appendix.
5.1 DAO REDUCES MOMENTUM NOISE AND PRESERVES MUTUAL INFORMATION (RQ1)
We now empirically validate the motivation of MT-DAO. Our results in Fig. 3 demonstrate that the slow-momentum of MT-DAO both preserves information about the global optimization direction across communication rounds and suppresses the variance induced by local updates.
Slow Momentum Is Preserved: The slow momentum in MT DAO preserves its direction across communication rounds. This directional memory also reduces the influence of local gradient noise, leading to Jower momentum variance across workers and more stable optimization path.
5.2 MT-DAO IS RESILIENT TO INFREQUENT COMMUNICATION (RQ2)
We now investigate if MT-DAS provides greater resilience against infrequent synchronization, as predicted by our analysis in Section showing that reducing the communication frequency of momenta with higher 3 has diminished impact the step size.

---

Published as a conference paper at ICLR 2026
Murual Information
0.00
0.2
Varusee
0.02
0.01
0.00
50
Consumnication
Commonication
Figure 3: A comparison of MT -DO (8) = 0.999) versus Local ADOPT baseline (3, 0.95) with a communication frequency K = 32. For each communication round, we plot metries computed between the momentum at the start (t) and end (t + K) of the round. MT-DAO's slow momentum preserves mutual information, I [U;;; U++K), across rounds while the baseline's ' momentum decays losing the global optimization direction (Ieft). Furthermore, MT-DAO reduces inter-worker momen- tum variance, Var ((+K))) indicating greater stability against local noise (right).
0,201006
2:734-33
(Parames
0.0015
[-9.007)
0,00114
20111
0014
( (Momentum
0.95
1:1.04 23-
(223346)
16:246:333
0,0000
0.95
(46 (0000)
5,00801
[-4000
0.210
(46.00)
0:00:
(48 (041)
0,0991
0012
0.99
9:8kk-0 2b
2:30--h
Mean Rela Relostee Change
0.99
01800071))
0,2140
(1.0.99)
0.30044
0.010
Mean Relative Change L2
0.999 0.998
E:00-05
14-00)
4:40-C5
(.:.3600)
2.13x-45
[1.00.-21]
2::0-c0
0U00006
1.0 0.999 0.995
(40700072)
2,00845
1-0 00101
0.21
1:22 00300030
B,00027
0.00177
(031)
0.008
(::11x-00)
[=1.0023)
(126-4))
000000
(48000075)
1:22 0089)
(0271
0.006
1.0
5:t0wdb
D-20EES
0:0051
(=1.1648)
121.50 36333
(32:3445)
[2:.16233]
0.004
O
000002
0.95
0.99
0.999
0.9999
0.95
0.59
0,959
0.9359
Figure 4: Mean relative L2 change and standard deviation across communication rounds of (left) model parameters and (right) the first momentum state, as a function of momentum decay (31) and weight (wi). In both cases, MT-DAO shows a significantly reduced relative rate of change with high (3,wr) w1) (minimum in gold), which reduces worker drift and thus makes parameter averaging more effective, Each point on the grid corresponds a configuration evaluated with its own independently tuned learning rate. Local ADOPT corresponds to (3) = 0.95, W1 = 1.0).
Table 1: Demonstration of how parameter synchronization period (K,) affects final perplexity for two MT-DAC configurations with momentum periods K₁ = K, for our 16M models. Values show the percentage increase in validation perplexity over the K2 16 baseline. Higher 3 leads to less performance degradation as K increases.
32
64
128
256
512
1024
01 0.99
(37.72)
+1.7%
+3.0%
4-3.9%
+-.11%
4-.66%
46.2%
91 0.995
(3766))
+1.0%
+1.6%
+33.2%
+2.8%
+3.4%
+3.7%
Table I shows that MT-DAO configuration with higher 81 suffers less degradation as K₂ increases. This resilience is explained by the reduced rate of change in the model parameters, quantified in Figure 4. High (31, w1) values reduce the local model change Em [| xella], bounding inter- worker variance and mitigating drift. Workers thus compute gradients on models closer to the global mean, improving convergence robustness (Li al., 2020).
Slow Momentum Anchor: Long-term momentum (high 81 and w) reduces the of change of parameters and optimizer states. This stability ensures worker models diverge less prior to synchro- nization, which reduces the performance impact of infrequent synchronization.
7

---

Published as a conference paper at ICLR 2026
135
(24
MT-2NO (LU6)
ADOPT (25.30)
Lood ADOPT (0L22)
EncI ADOPT
Hastorow ADO (2048)
ADOFT (1441)
Valiaation Perplesty
Nattoo 400PT
(0)
123
AJCI T-OPP (2.42)
NNDDIO AOOEPT (7.43
ADOPT. DDP
DN AROFT (3:40)
CHADOPT T-OOP-OOPPPP
Valelatenn
124
3.8
13
18
02
25
(Houm)
(Houm)
(Hour))
MT-DAD (24 66:
MT-240 29:2
ADOPT
Local ADOPT (DLI1)
12.
Loul ADOPT
(15
Nattaos ADOPT (UA25)
Porpeaity
ADOIPT (55
Valdaton Popteelty
ADOPT.0OP (10.25)
A3OPP (2 0)
CH-ADOTT-OO BBP (2441)
QH-DOPT DDP
1:2
QHAADOR DUP (7 58)
22
Valdeson
72
(Blllon))
Takene (1100oo))
Tokene (13l0on))
(a) 16M
(b) 125M
(c) 720M
Figure 5: Validation perplexity versus wall-clock time and training tokens for MT VD- (8) = 0.999, W/ 0.98) and baselines models of increasing size. Horizontal lines denote the two DDP baselines (ADOPT-DDP and SHADOPT-DDPP For each non-DDP method, a colored marker on the x-axis marks the point at which its curve attains lower/equal perplexity to DDP variant.
5.3 MT-DAO OUTPERFORMS PRIOR LOW-COMMS OPTIMIZERS AND MATCHES DDP (RQ3)
We evaluate DAO on 16M, 125M, and 720M parameter language models against other baselines. We report validation perplexity as function of both training tokens and wall-clock time. Timings are measured on 4 cloud H100s connected via 50-100 Gbit/s Ethernet, including constant implementation overheads, and accounting for communication-copptaae are specific to this hardware; Appendix provides bandwidth model that compares communication- efficient methods to DDP across wider range of interconnects. When reporting time-to-target perplexity, we give improvements in both wall-clock time and training tokens.
overlap in DDP. These measurements
Across all scales, MT -DA consistently improves over ADOPT-DDE and Local ADOPT in both tokens and time, closing the gap to synchronous training and reducing end-to-end wall clock by 6- 2.27%. At 720M, relative to single-momentum DDP, MT-DAO reaches the same perplexity in 24% fewer tokens and 35% less time. Relative to CHADOPT-DDE 125M, and at 720M reaches the QHADCPT-DDP target perplexity about 8% faster in wall-clock and R 5% fewer tokens. The additional improvements in time are due to MT-DAO communicating 10x less than DDP. The outer Nesterov baseline performs better than Local ADOPT in our setting yet remains below MT-DAO and DDP; matching the findings of Charles et al. (2025, Table 4). Mech- anistically, Nesterov coalesces per-round gradients via an outer momentum, whereas MT-DAO implements a finer- ther-grained inner multi- multi-timescale modification. We note that MT (N = 1) does not require the additional momentum buffer Nesterov and has only one additional hyperparameter to tune instead of two.
MT-DAO trails at 16M, matches at
Improved Performance and Efficiency at Scale: MT-DAO improves performance w.n. all base- lines across model scales, closing the performance gap to DDP. These gains are not specific ADOPT: we confirm similar trends with Adan up to 1.3B (Section 5.5), Muon (Appendix H.7). AdEMAMi x, and AggMo (Appendix H.2). We further show robustness across batch sizes and worker counts (Appendix H.3), in the streaming regime (Appendix H.4), and under varying numbers of momenta (Appendix H.2).
8

---

Published as a conference paper at ICLR 2026
MT-DAO
Local ADOPT Nesterov
ADOPT w/
Local PG to Global Mom
Local PG to Local Mora
Local PG tu Global PG
Global PG to Global Mom
100
3.0
1.0
Cleite Sintleity
0.8
0.5
0.5
0.5
0.6-
0.0
20
(1
20
50
Commonication Roumds
Communccation Rounds
Communication Rounds
Commimictiion
Rounds
Figure 6: A comparison of update vector alignments for MT-DAO (31 = 0.999, = 0.98) versus Local ADOPT (31 0.95), and Local ADOPT (81 0.95) with Nesterov. Cosine similarity measured between: (1) the local pseudo-gradient and global momentum, (2) the local pseudo- gradient and the local momentum, (3) the local and global pseudo-gradients, (4) the local and global momentum. Pscudo-gradient and momentum have been abbreviated as PG and Mom.
5.4 MT-DAO ALIGNS WORKER UPDATE TRAJECTORIES RQ4)
Having established the performance benefits of MT-DAO, we now investigate the underlying mecha- nism. We hypothesize that the slow momentum reduces worker drift by keeping the optimization trajectories of individual workers aligned with the global optimization direction. To validate this, we measure the cosine similarity between key optimization vectors. We define the per-round local update as the "pseudo-gradient" (A -t+K - xx"") and the global pscudo- grandeng as the average of local ones (Reddi al., 2021). To provide a comprehensive comparison, for each method: for MT-DAO and Local ADOPT, is the average of worker momenta at the end of round, while for the Nesterov variant, it is the state of the outer Nesterov momentum.
define the "global momentum"
The results in Fig. 6 show that MT-DAO achieves near-perfect alignment (cosine similarity >> 0.95) across all four metrics. This indicates that: (1) each worker's update is consistent with its momentum history (Local PG to Local Mom), (2) workers are in strong agreement with each other (Local PG to Global PG), and (3, both local and global updates are aligned with the long-term global trajectory (Local/Global PG to Global Mom). This demonstrates that the slow momentum acts as regularizer, ensuring all workers maintain stable and shared optimization path.
In contrast, the Nesterov outer optimizer presents mixed results. As an EMA of global pseudo- gradients. is better aligned the global -rrudigg than the Local ADOPT momentum and improves the alignment between the local and global souudoggaddeetsssoo compared to standard Tocal ADOPT. However, never reaches the degree of alignment of MT-DAO in any metric.
Slow Momentum as Regularizer: MT MT-DAO's slow momentum acts as a regularizer for each worker, ensuring that local updates remain aligned with their history and the global trajectory.
5.5 MT-DAO IMPROVES DOWNSTREAM TASK PERFORMANCE (RQ5)
To validate that perplexity gains translate to practical utility, we evaluate downstream zero-shot and few-shot accuracy at the 1.3B parameter scale using Adam (Fig. 7). MT-DAO demonstrates consistent superiority over Ada aggregate performance, excelling on the most challenging reasoning tasks LARC-CHALLENGE HELLLASAAG). Compared to QHAdamW-DDP. MT-DAO remains compet- itive early in training and progressively closes the gap as training continues, mirroring the perplexity trends. Using 4 machines with 8 H100 GPUs each connected via 100 Gbit inter-node links, MT-DAO achieves target accuracies earlier in time than its DDP counterparts (Fig. 12).
6 RELATED WORK
Standard Distributed Data Parallelism's (DDP) per-step synchronization creates communication bot- tleneck (Sergeev & Balso, 2018). This is mitigated by two orthogonal strategies: payload compression and infrequent synchronization. Compression shrinks transmissions via quantization (Alistarh et al.,

---

Published as a conference paper at ICLR 2026
ARC-Challage (B-chot)
ARC-Eeay (0-dhon)
Store (1)
Soore (%6
Store
5 8
ANG DETNEE
3ES DMG
Anw Desa DE
Anlen to
to
d0
70
do
50
Tokena (Bills
Tokera (Ball.
Tokena (Billi
HellaSwing (D-ahat)
(ssbo)
PIQA (a-shoe)
Score (56
Scorn (56)
86
2044 29 A
30
20u8
-L38000 2012
Quieie 2000 (1.66.3
20
40
50
20
40
60
50
Tokens (Billion))
Tobens (BAllloos))
Tokens (Billios)
Figure 7: Downstream task accuracy versus training tokens at 1.3B scale on ARRC-CHALLENE. ARC-EASY, HELLASWAG, MMLU, and PIQA. MT-DAO exceeds AdamM-DDP the aggregate and on reasoning-heavy tasks; closes the gap to QHAdamW-DDP with training. Tome-normalized results: Figure 12.
2017), sparsification (Lin al., 201 8b), mixes thereof (Wang al., 2023b), low-rank updates (Robert et al., 2025), or communicating select momentum components (Peng et al., 2024). Our work advances infrequent synchronization (Stich, 2019; McMahan et al., 2017) which allows local updates between communications and is complementary to compression.
Adapting stateful optimizers like Adam infrequent synchronization not straightforward, Local Adam (Cheng & Glasgow, 2025) provided the first convergence proofs at the cost of synchronizing all optimizer states. Douillard al. (2023); Charles et al. (2025) showed that a Nesterov-based outer optimizer improves performance. Recently Iacob et al. (2025) improved the communication efficiency of Local Adam by decoupling parameter and momentum syne frequencies. However, these methods use single-timescale optimizers with small 31 values ill-suited to low communication frequencies due to momentum decay. While naively increasing momentum often harms task performance, recent optimizers that track gradients across multiple timescales have shown significant benefits. QHM (Ma & Yarats, 2019) decouples momentum decay from gradient weight, while AggMo (Lucas et al., 2019) averages multiple velocity vectors for stability. Building on this, AdEMAMix x (Pagliardini et al., 2025) mixes fast and slow momenta to accelerate convergence, demonstrating that slow momentum acts as memory, reducing forgetting in L.Mss. More recently, Muon (Jordan al., 2024) uses Newton-Schut preconditioning for update orthogonalization. We show that MT DAO is compatible with all of these optimizers. providing empirical validation with Adam, ADCPT, Muon, ACEMAMix. x, and AggMo (Appendix H). For further related work see Appendix G.
CONCLUSION
A persistent challenge in distributed training has been the performance gap between fully-synchronous and communication-effiieee optimizers. We identify the rapid decay of momentum as a key cause: standard momenta are temporally mismatched with the long intervals of infrequent communication. MT-DAO resolves this with a multi-timescale optimizer whose slow momentum persists across communication rounds. Our theory shows that higher 3 reduces sensitivity to synchronization frequency, and our experiments demonstrate that this approach closes the performance gap with DDP by maintaining stable, shared optimization trajectory. These findings establish momentum timescale management as a critical factor for performant distributed training, providing a practical path for communication-constaanned cross-datacenter, and wide-area settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
All costs for the computational resources used for this work were funded by Flower Labs, and the research conducted by a team of researchers from Flower Labs, The University of Cambridge, The Institute of Science and Technology of Austria, The University of Warwick, and Mohamed bin Zayed University of Artificial Intelligence. Support for university-based researchers came from a variety of sources, but in particular, the following funding organizations are acknowledged, the European Research Council (REDIAL), the Royal Academy of Engineering (DANTE), the Ministry of Education of Romania through the Credit and Scholarship Agency, and the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 101034413.
REFERENCES
Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv: 2504. 05295, 2025.
Dan Alistarh, Demjan Grubie, Jerry Li, Ryota Tomioka, and Milan Vojnovie. QSGD: communication- efficient SGD via gradient quantization and encoding. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. In Conference Neural Information Processing Systems (NeurIPS), 2018.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafitti, Hynek Kydlícek, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model. arXiv preprint arXiv:2502.02737, 2025.
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming. 199( 2):165-214 2023.
Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd with majority vote communication efficient and fault tolerant. arXiv preprint arXiv:1810.05291, 2018.
Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication effficient language model training scales reliably and robustly: Scaling laws for diloco. arXiv preprint arXiv:2503.09799 2025.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hicu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms. In NeurlPS, 2023a.
Yineng Chen, Zuchao Li, Lefei Zhang, Bo Du, and Hai Zhao. Bidirectional looking with A novel double exponential moving average to adaptive and non-adaptive momentum optimizers. ICML, Proceedings of Machine Learning Research, 2023b.
Ziheng Cheng and Margalit Glasgow. Convergence of distributed adaptive optimization with local updates. In International Conference on Learning Representations (ICLR), 2025.
Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Bill Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don't 't be lazy: Completep enables compute-efficient deep transformers. arXiv preprint arXiv: 2505.01618 2025.
Arthur Douillard, Qixuang Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.088105 2023.
11

---

Published as a conference paper at ICLR 2026
Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Gar- rett, Gabriel Teston, Dave Lacey, Ross Mcllroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Mare' Aurelio Ranzato, and Paul Barham. Streaming diloco with overlapping communication: Towards a distributed free lunch. arXiv preprint 25Xiv:2501.185/2 2025.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07949; 2020.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. Conference on Neural Information Processing Systems (NeurIPS), 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals. and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint (rXiv:1909006355, 2019.
Zhouyuan Huo, Qian Yang, Bin Gu, Lawrence Carin, and Heng Huang. Faster on- on-device training using new federated momentum algorithm. arXiv preprint arXiv: 2002.02090, 2020.
Alex Iacob, Lorenzo Sani, Mher Safaryan, Paris Giampouras, Samuel Horváth, Andrej Jovanovic, Meghdad Kurmanji, Preslav Aleksandrov, William F. Shen, Xinchi Qiu, and Nicholas D. Lane. Des-loc: Desynced low communication adaptive optimizers for training foundation models. arXiv preprint arXiv: 2505.22549 2025.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: / //kellerjordan github. ic wwtp//psss/////o /mmon/
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He. Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song. Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. arXiv preprint arXiv: arXiv: 1912,04977, 2021.
Satyen Kale, Arthur Douillard, and Yanislav Donchev. Eager updates for overlapped communication and computation in diloco. arXiv preprint 250. 2025.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.0336 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference Machine Learning (ICMI., 2020b.
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontack Oh, Beomjun Kim, Tachwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-In: Revisiting layer normalization the transformer architecture. arXiv preprint arXiv: 2502.02732, 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
12

---

Published as a conference paper at ICLR 2026
Jaerin Lee. Bong Gyun Kang, Kihoon Kim, and Kyoung Mu Lee. Grokfast: Accelerated grokking by amplifying slow gradients. arXiv preprint arXiv: 2405.20233, 2024.
Royson Lee, Minyoung Kim, Da Li, Xinchi Qiu, Timothy Hospedales, Ferenc Huszár, and Nicholas
Lane. Fed12p: Federated learning personalize. Advances in Neural Information Processing Systems, 36: 1481 8-14836, 2023.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In Proceedings of Machine Learning and Systems (MLSys), 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning (ICML), 2021.
Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. arXiv preprint arXiv: 0888.07211 2018a. 8a.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local SGD. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations (ICLR), 2018b.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training In International Conference Learning Representations (ICLR), 2024.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang. Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon scalable for LLM training. CoRR, (i///5022.66822 2025.
Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. arXiv preprint arXiv:2007.07995 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
James Lucas, Shengyang Sun, Richard S. Zemel, and Roger B. Grosse. Aggregated momentum: Stability through passive damping. In International Conference on Learning Representations (ICLR), 2019.
Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and adam for deep learning. In International Conference on Learning Representations (ICLR), 2019.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 1812.06162, 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-eefiiieee learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
Matteo Pagliardini, Pierre Ablin, and David Grangier. The adEMAMix optimizer: Better, faster, older. In International Conference on Learning Representations (ICLR), 2025.
Bowen Peng, Jeffrey Quesnelle, and Diederik Kingma. Demo: Decoupled momentum optimization. arXiv preprint arXiv:2411.19877 2024.
Xinchi Qiu, Javier Fernandez-Marquee Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208. 02507, 2022.
13

---

Published as a conference paper at ICLR 2026
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný. Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference Learning Representations (ICLR), 2021.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A cottmuunication-ffffiee federated learning method with periodic averaging and quanti- zation. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Thomas Robert, Mher Safaryan, Konut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. International Conference on Learning Representations (ICLR), 2025.
Lorenzo Sani, Alex Iacob, Royson Lee Zeyu Cao, Bill Marino, Yan Gao, Wanru Zhao, Dongqi Cai, Zexi Li, Xinchi Qiu, and Nicholas D. Lane. Photon: Federated Ilm pre-training. In Fighth Conference on Machine Learning and Systems, 2025.
Felix Sattler, Simon Wiedemann, Klaus-Rober Müller, and Wojciech Samek. Robust and communication-effiien federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
Bryan Van Scoy, Randy A. Freeman, and Kevin M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control. Syst. Lett. 201 8.
Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv: 2509.01440, 2025.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprini arXiv: /000:0099999 2018.
Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations (ICLR), 2019.
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing. 2024.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. International Conference on Machine Learning (ICML), 2013.
Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong, Go Nagahara, Tomoshi liyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. ADOPT: modified adam can converge with any B2 with the optimal rate. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization, arXiv preprint 1Xiv:2000.074881 2020.
Janny Wang, Zachary Charles. Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Pranceth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi. Sashank Reddi, Peter Richtarik. Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi. Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang. Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A field guide to federated optimization. arXiv preprint arXiv:2 107.0691 2021.
Jue Wang. Yucheng Lu, Binhang Yuan, Beidi Chen, Perey Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference Machine Learning (ICMI.), 2023a.
14

---

Published as a conference paper at ICLR 2026
Jue Wang. Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang. Christopher De Sa, Christopher Ré, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In /CML, PMLR, 2023b.
volume 202 of Proceedings of Machine Learning Research, pp.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. arXiv preprini 10xx::/905.08/7,, 2019
Kun Yuan, Xinmeng Huang, Yiming Chen, Xiaohan Zhang, Yingya Zhang, and Pan Pan. Revisiting optimal convergence rate for smooth and non-convex stochastic decentralized optimization. arXiv preprint arXiv:2210.07800 2022.
Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How does critical batch size scale in pre-training? In International Conference Learning Representations (ICLR), 2025.
15

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Limitations
B Experimental Details
17
B.1 Architecture Details and Parametrization
17
B.2 Optimizer Hyperparameter Sweeping Procedure
18
B.3 Optimizer Tuning Results
18
Deterministic Optimizer-specffie Variants MT-DAO
MT -TAO-ADDDPT
XT-DAO-SGDM
D Convergence Analysis of MT-DAO-SGDM
20
D.1 Key Lemmas..
25
E Wall-Clock Time Modeling
29
E.1 Wall-Clock Time Model
29 
Experimental
.e..............................
E,3
Modeling Results
30
F Derivations of Mutual Information and Variance
31
F.1 Variance of Local Momentum
31
E2 Mutual Information
32
Extended Related Work
33
H Additional Results
34
H.l
Downstream Evaluation
36
H.2
Ablations the Number of Momenta N
36
H.3
Ablations on the Number of Workers and Batch Size
38
H.4
MT-DAA Comparison Against St reami ngDi loCoo
39
H.5 Ablation on the clipping threshold
42
Ablation with K
43
H.7 Efficacy with Muon
44
LIM Usage Declaration
49
16

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
Limitations. While we have substantially expanded our empirical evaluation to models up to 1.3B parameters, multiple optimizers (ADOPT, Adam, Muon). and several ablation dimensions (N, batch size, workers, clipping, streaming), our validation remains limited to language-model pre-training. Extending MT DAO to other modalities (e.g.. vision, multimodal) and to non-IID data distributions beyond the homogeneous setting is left for future work. Additionally, while convergence analysis covers -T-DAOGSGDM, extending theoretical guarantees to adaptive variants and to the streaming setting remains open direction. Finally, experiments focus the memory- and communication-efffieenn N=1 variant, which we justify empirically (Appendix H.2); whether higher N provides benefits in regimes with very long communication intervals (K > 512) warrants further investigation.
B EXPERIMENTAL DETAILS
Here we provide additional experimental details complementing those in Section 4.1, including: a) model architecture details and the model parameterization (Appendix B.1), b) our hyperparameter sweep procedure to select optimizer-specffic settings (Appendix B.2), and c) the results of our tuning sweeps for MT DAO.
B.1 ARCHITECTURE DETAILS AND PARAMETRIZATION
Table 2: Model architecture and training hyperparameters. Architectural parameters include the number transformer blocks (#Blocks). attention heads (#Heads), embedding dimension (dmodet), vocabulary size (VV),, and feedforward expansion ratio (Exp. Ratio). Key training parameters are the global batch size (Baa)) and the total number of training steps (T). All models use RoPE positional embeddings (Su et al., 2024), the S LU activation function, norm-based gradient clipping with
bound of p. and are initialized with a typical (Semenov et al., 2025; Dey et al., 2025) o 0.02. For Adam the p values recommended by Semenov et al. (2025). Sequence length is standard for
models at these scales.
Ausodal
[V]
Exp--Ratio
Radupt
Padaa
Seq
[Rg]
16M
256
50K
4
4
10930
5:100
0.02
1.0
1.0
2048
64
46908, 12228, 41960
125M
12
768
b0K
12
4
10000
S110
0.02
1.0
0.5
2048
256
4608, 12288, 41960
300M
24
1024
50K
16
4
10030
5110
0.02
0.25
2048
256
12288, 40960
720M
50K
2048
12258,
1.3B
21
2048
50K
4 10000
5110
0.02
1.0
0.1
2048
1024
12288, (0860
Table 2 summarizes the architectural details of our models, which follow established practices for large language models at their respective scales. To improve training stability and final performance, adopt two key modifications. First, following the recommendations of Kim al. (2025), use Peri-LayerNNorm transformer structure instead of pre-norm. Second, we use the Complets eteP (Dey al., 2025) parametrization with parameters from small model to its larger-scale counterparts in a one-shot manner. This property allows us perform comprehensive hyperparameter sweeps on our smallest model size and reserve computationally expensive scaling experiments for direct comparisons against baselines.
1.0, which enables the effective transfer of optimizer hyper-
We set batch sizes and training durations following recent best practices (Zhang al., 2025). For the smallest model size, the initial batch size is determined using the noise-scale estimator for the critical batch size (McCandlish et al., 2018) and then doubled until the efficiency deviates from linear trend by 20%. For our 125M and 720M models we follow the batch size recommendations from Semenov et al. (2025). Training durations are set as multiples of the compute-optimal token budget (Hoffmann et al., 2022): :): for the 16M model, we tune using x 2 this budget and run baseline comparisons at ss 16x; ; for the 125M model, we use se 8x; and for the 720M model, we use z 2.83 We chose the 720M model size good balance between scale and computational efficiency following Semenov et al. (2025), with the 360M being chosen as an efficient middle- grouled between 720M and 125M
For the 1.3B use standard batch size of 2M tokens.
17

---

Published as a conference paper at ICLR 2026
All models trained using the harmup-stable-dee (RSD) learning rate schedule (Hägele et al., 2024), with warmup and decay periods selected based on established recommendations (Zhang et al., 2025; Hägele et al., 2024; Allal et al., 2025; Semenov et al., 2025).
For all longer training runs and baseline comparisons, we use the industry-standee warmup of TWARM 2048 steps. We use cooldown period equal to the warmup period in all cases, using 1-sqrt cooldown (Hägele et al., 2024).
We use slightly different tuning configurations for Adam. For the 16M model tuning runs for ADOPT, which last 4608 steps, the warmup period is set to TWARM 512 steps. For Adam we use 12288 steps the tuning period with warmup period set TWARM 2048.
B.2 OPTIMIZER HYPERPARAMETER SWEEPING PROCEDURE
Our tuning procedure is designed to ensure that both our method and the baselines are evaluated under their optimal DDP configurations, providing fair comparison. Given that previous work has shown that the learning rate (LR) tends to transfer effectively between DDP and distributed settings (lacob et al., 2025), we first tune all parameters to achieve the best possible performance under DDP and then transfer these settings to MT-DAO. Unlike methods such AdEMAMix that schedulers for optimizer parameters, we employ a simple switch from a base optimizer (e.g.. ADOPT) T) to its multi-timescale variant at the end of the warmup period. This necessitates two-phase LR tuning process to ensure identical starting conditions for both optimizers:
Phase Base Optimizer Tuning. We first tune the learning rate for the base optimizer over the entire training run achieve the lowest final perplexity. This ensures the baseline itself is as strong as possible.
2. Phase 2:
Tuning, Using the model state from the end of the base optimizer's s warmup, we then tune the learning rate for the post-switch phase of MT-DAO and of its DDP analogue. With WSD scheduler, this corresponds to tning the LR for the constant "stable" portion of training and for the cooldown.
While more complex scheduling manipulations might yield further gains for DAO, this two-phase approach provides the cleanest methodology for comparison. For every combination of momentum decay rates (f's) and convex coefficients (w's) used by MT-DAO, we independently perform this tuning procedure. For ADOPT the LR sweeps both phases search over values between 2-10 and 2-6 using powers of two, with the search grid refined by manually adding 2 8.5 2 7.5 6.5) around the optimal value.
steps (e.g..
B.3 OPTIMIZER TUNING RESULTS
First, tuning of ADOPT for DDP revealed optimal Ir 17" 2- while tuning for Adam revealed an optimal learning rate of 17" = 10-3 We now present the results of our tuning for the post-warmup Ir for MT DAODDO-DD with N first momenta in Fig. 8.
clear trend emerges from our results: methods with higher momentum decay rates (3s) or higher weights (ws) ascribed the slow-moving momenta can tolerate significantly higher learning rates than standard momentum methods. This finding is in strong agreement with the previous findings of Lucas al. (2019), who similarly found that AggMo can effectively utilize learning rates that are orders of magnitude higher than those suitable for classical momentum.
Takeaway: Multi- tinesciieccl optimizers that emphasize slow-moving momenta (via high 3 or w values) are not only more stable but can also leverage much higher learning rates, enabling faster convergence than their manglettinescle counterparts.
18

---

Published as a conference paper at ICLR 2026
Figure 8: Visualizing the learning rate sweeps for different MT-DAO configurations. Each subplot shows the final perplexity for given convex coefficient (w) and momentum decay (31), where 32 = 0.9999 was kept constant. The sweep demonstrates that the optimal learning rate and final performance are highly dependent on the choice of these internal hyperparameters, with S1 (0.995, 0.999] and w E [0.9, 0.99) performing best for these short tuning experiments. The vertical line each subplot marks the persorerronnng for that confiuurttinnSwiichh seale referes to the multiple of the base learning rate that we select, the chosen learning rate can be computed via multiplication with BBSES.
C DETERMINISTIC OPTIMIZER-SPECIAE VARIANTS OF MT-DAO
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 -NT-DAO-ADOPT
Require: Model tensors, hyper-parameters
1: #0 E R", { E (IR")", 8-1 E R" initial params, N first momenta, second momentum ($1.))-1,88 82 [0, 1] - decay rates for each momentum state
2:
3:
[wy]]=1 [0, convex combination coefficients for first momenta, Σ S 1.0
4:
clipping and lcarning-rate schedules 5: T.M E N4 total optimization steps and number of workers
(ce)
6: K2, K2,(K, 1= K. E (N+)*+2 communication periods for parameters and states
7: OuterOpt R + R update params using outer optimizer, averaging by default
Ensure: Tr. fur-1)/-1 VT-1
8: for each worker m: initialize 20', [u-T), _n1
9:
do
10: for all workers ....., M in parallel do
11:
of" <- VF(="";
stochastic gradient
12:
"f" <- -2 + (1 (9
update second momentum with gradient sync every steps
13:
+- if (t mod K2 0) then Fm (01 else et"
14:
normalize gradient (ADOPT step)
15:
9tth < c ct)
clip the nonnalized gradient
16:
for to N do
update first clipped, normalized gradient sync u² every K2 steps form combined update direction
17:
us 81
-
18:
i/. + if (t mod K, = 0) then Em [u else ui.
19:
Al" + [1 - Σド= w, )²)" - Σ== uy
20:
2f f 3; A,
apply combined update K2 steps
21:
T141 + if ((t 1) mod K₂ 0) then (utcropt(E.m(1))) else rf+1
Algorithm 3 MT-DAO-SGDM
Require: Model tensors, hyper-parameters
1: zo E Rd, [___] - E (R initial params, first momenta
[81 E [0, 1) decay rates for each momentum state
(wi]i=1 [0, 1] - convex combination coefficients first momenta, Σ 45 1.0
PE R+. [v:]T= - clipping radius, learning-rate schedule
T. E N+ - total optimization steps and number workers
6: Kr, Ke,KK
(N+)*+ communication periods for parameters and states
7: OuterOpt Rd Rd update params using an outer optimizer, averaging by default
Ensure: 2r, 1
8: for each worker m: initialize x 3...
9: for tort=00 O.....,, T' 1 do
10: for all workers m = D....,M. M in parallel do 9i" + ('." p.)).
11:
clipped stochastic gradient update first momenta
12:
for j = 1 N do
13:
un - u _/___ (1 (1- BB
14:
पे < if (t mod
K3 = 0) then E. (u else un
sync u² every K2 steps form combined update direction (unnormaiize))
15:
Al" + [1 - Σ w/ ))" Σ uyu.
x f- Z; A
17:
Æt④1 + ((t + 1) mod K2 = 0) then Outeroptt(Emmp else 3:4
syne a every K2 steps
D CONVERGENCE ANALYSIS OF MT OT-DAO-SGDM
In order to facilitate the technical presentation, we model synchronization frequencies by assigning probabilities to each averaging event. For example, the parameters x:" synchronized with the
20

---

Published as a conference paper at ICLR 2026
Algorithm 4 MT MMT-DA-SGDM, probabilistic variant
Require: Model tensors, hyper-paramettrss
1: 20 E R" {u' 1) E (R=")* initial parameters, N first momenta
2: (8,))= E [0, 1) decay rates for each momentum state
3:
(wyh-1 [0, 1 convex combination of non-negative coofficients for first momenta, Σド= =
4:
(nol=o - Icarning-rate schedule
5:
T. E N4 total optimization steps and number of workers
6:
K2 (p, K,
bー [0, [0,1] communication periods/probabilitis for parameters and states
Ensure: 2r: 1
for worker m: initialize 20 (w2r))
8: fo O....,,
do for all workers m = O,..., M in parallel do
9:
10:
Si" + Vn (
stochastic gradient update first momenta
11:
for j = 1 to N do
Bm [ f. (1- 3,)9%)
with probability Py with probability 1 Pb
ut mm
syac 1x
3 + (1 3,)9("
13:
A
Lj=1 Wi wyrr?.m
form combined update direction
Bm m "/=""" with probability px
#++1
symc IT:
A.
with probability 1 Pr.
probability K. which is statistically equivalent to performing the averaging in every 1 = K₂ iteration. Similarly, momentum u.'" synchronization happens with probability differ from Pa- Note that QHM structure included since we can choose 31 = 0 and get w. = ll" Auxiliary notation. Let EE and E, be the averaging operators with weights J. across M workers and wy across momenta.
P. R. which can
def
Em (u?
Bru + (1 B5)gt, where 9: = E, (m."))
j.m [+1
def
Em [xtim qu?
with probability Pa
with probability 1 Pr
2 def
Em Em [m/44]]]] Id -
2Ht E, [x/111111 (line 14).
For the sake of notation, we also let unth = A Eyluf"") upcoming derivations.
Em Bm["), 24 = Em [x7]]] in the
Step (virtual iterates). Letting x2_ zgj Fg. define the global virtual iterations follows
def
3 1 t-1
and 2t def [2]]]
for 2 0.
B
B3
21

---

Published as a conference paper at ICLR 2026
The key property of this virtual iterates are going to exploit in the next steps is that they follow averaged gradients, namely for any t 2 we have
2++1
1
Bj
(
1
B5
E
Bg 2+1
1 1- Bj
2/
3j
x
1- Bg
2
3g
E
1 8y (12+1 -xt)
(s)
st
Bg
Bj (-me--1n
E
83 (-mpu) 1 33
71 (u) 8y -B
E2
EByl-1gd
-119t.
3g
Step 2 (smoothness over virtual iterates). Then we apply smoothness of the global loss function f over these global virtual iterates.
f(z++1) S f(zx) + (V)(=)) 2041 - 22) 22 "1 ||:0+1 20|1
f(z)) + V f(re), 2111 21) + f - vf(xx)) 2041
21) +
2 2 - 2011?
111
In the next step, we separately bound each term appearing in the above bound. Step 3a (one step progress). Bounding term I.
B(V ( 2++1 =z)
nn (1)))
Σ gt
BB
1
Σ
V
(x
M
M
M
1 M
-
7 E
V fm("
M 2
E
vf(xt)
V 7f (I+T)
2
2
M
M
m=
M
1 M
V (x
+ M E
(m - v. (m
2
2
M
2
M
m=1
m=1
M
M
< 2 2
-"EIV
"B
V
)
E (m - V (m
M
2.M
m=1
< -"
2E
M Σ
2
nLL
M
Bli x7
M
2M
m=
m=1
Lemma 3
Step 3b (one step progress). Bounding term II.
M Σ
E (2²) Vf(x)), Z441 - 2x)
-nB Vs(z)) -((z)-v((a))
V mm (x
M
m=
M
nP
V)(2))) vf(x1)11
7
V fm (x
2
2p
M
mpL
2
20 2
+ 17 E
M
V fm (x
2
2p
M
Lemma 2
m==1
22

---

Published as a conference paper at ICLR 2026
Step 3c (one step progress). Bounding term III.
L²L
M
B|12++1
ze12
2
M
172L.
Σ 96
M
V (a
+
17 E
1
V
fm (2.
2
M
2
M
mm
m
L
M
178
M
19 Th
fm fm )1²
E
V fm (z
2M
M
m=1
m=11
1
r E
M
<
0² +
V m
(x
2M
2
M
m2
Step 3abe (one step progress). Combining previous bounds.
Ef(z+11 - Es(x) <
E (V 2111
21) +B (V(2))
V[(x)),
#++1 2x) +E
7 - #el1
111
nL2
-
fm (x
Tt
M
2M
Lauma 3
Mpl
17 E 2p
E 21
xzll
(z
M
Lemma
1
02
E
fm
(I
2K
2
M
7 - Vf
17
1
1
nL
E
1
fm
my
2
P
M
7lhn
nL
M
17
+
E
E
xz
11
+
EE
20
xt
0
2
2M
2M
Lemma
m=1
Lemma 3
23

---

Published as a conference paper at ICLR 2026
Step 4 (final). Now we average over the iterates and apply the bounds derived in Lemmas 2 and 3.
((f(2r) f(z0]]
1
Σ
-
T
T
7 (7(1
E
(x)))
1
nL
E
V
(x,")
2T
2
P
T
M
1=0
t=0
npl.
T-1
11
T-1
M
+ 1 0² 2M
Bl - -eel 2
xt 112
2
TM
Lemma
Lemma 2
<
2T
17
T-1
EE [
2 1
2
P
-
nT
E
M
M
mm (x,")
n
2M
0²
t=0
m=1
1pL2
1P
T-1 Σ
M
0² + +17
E
V (x
2
M
M
m=1
11?
+
12nr² 1)01
V )12
+ 4n7 2002 + +G2)
2
T
(1
LB 1)6)
2
T
t=
1
1
1
1n - B2LL
1
ΣE
M
fm (x(T)
2 p
M
m=1
L
11 P2
0²
0² 2n (0² + 3G2).
2M
2M
Next, we choose p = and step size 77 such that
1
12 (B2
1)0 -
to bound the first term
1
nI + 277² 10,2, S
to bound the second term
120 2
from Lemma 3
Notice that
def (AT.max (Bw, 6V 16 max B 1)
110
satisfies all three bounds. Then, with any 17 S we get
E(S(=T) [(20)
T-1
5
1)
E [xxz)]
T
4T
1²² L
17 p2pL2
+
0²
+
0² + 21 22 + +3G2).
2M
2M
Noticing that zo and frr < f(zT), we have
T
[( < A(f(2o) - -()
20L
L
1/T +
x + M
0² + L3 + +36²).
M
24

---

Published as a conference paper at ICLR 2026
Furthermore, choosing 77 min(no, ) we get the following rate:
B| B (x:))1
4(f(2o) 17) + 2L
AL2
8L2 +36
max
1, nooT
+
VT
MVT
MT
T
4(f(r)) - f*)
+ 2L
4(f(x0) - -f¹) +
2022
SL2 3G9
+
+
VT
MVT
70T
MT
T
VT (I(OT)) - + Lo²
4
+
P +
+ 0
2M
T
D.1 KEY LEMMAS
Lemma 2. For all T 2 1, we have
T-1 1
T-1
M
Σ t=
|| - x0l1
-
To² + 17 ~
Σ
E
Σ my
V
(x?")
(3)
M
M
1=0
where
wrBr
Bw
Σ
33
Proof. Since u-1 0, unrolling the update rule of momentum, for any t 2 0 we get
ut -3,u/=1+ (1- = (1 3j) 0=0
Σ gg gg.
Using this and the definition of the average iterates, we have
n/83
- = 1 33
(x] x/-1) = 1 1- 33
uf
n
3/- 97
-=0
24 Tt Eylzd
-n135 Σ
87-
9T 11
Σ
By [
gr
0=0
7=0
N
8(1 gr:
where 3(7) = R, [ = Σ w
Let us make another notation for the sum of weights in the above sum and bound it as follows:
def
w
7=0
r= j=
Σ
N
W
g
+1
Σ
N
Wj
Bj -gt42
N
wrBO
del
Ba-
1- Bj
1 33
j=1
T=0
1=1
1=1
Using convexity of squared norm, we have
+11)
, (1-++1)
2|
2:11 = n
97
<
2
8
|| 11?
< n Σ
plt ( 1)
|
r=0
T=0
==0
25

---

Published as a conference paper at ICLR 2026
Summing over the iterates yields
T
E 2
xell?
BB
34 E
1=0
t=0 ==0
T
Bw
3 B 1- = Br
((0)
E 9- 11
Bj 33 TT-++
Br
Wj
1|1
Bg
T-1
< 17 3
B||g+11 1?
1/ 8
97
V (x
+ +1 2 8
E
mm (2 m
M
M
1/8
E
11
V (2x) )1
+ +
E
fm (I77
M
M
T=0
M
P To
+ 7
E
(2 m
M
M
m=1
Lemma 3. 2f L2 < 1, then
T-1 M
Σ Σ
R rt
2m11/ S
12² -1)
Σ
+ +30
MT
where
4(1 - pp))
N
(1- (1-3))(1 - Pp))
Wj 1- (1 p,)
p2
j=1
Proof. Let us expand the term E|| x/iill3 using "ii 's probabilistic update rule:
E 0|
0+ (1 - p.)) El[x; 774 (a:)7² myu )|2
(1 - -Pa) Ellare xx 17(4; w, )1
(1 - p1 + s)Ellz, x1'11² + 772 p ) + 1/s)Ellu, ur'l
s 12(1- - (1) + 1/2) Σ (11pp)) + +))) - ==
T=1
26

---

Published as a conference paper at ICLR 2026
where 0 will be chosen later. Next expand the term B||bu] f. ussng uf uf"""s probabilistic update rule:
M
E/ 1/2 = pp.0 (1 -p))
Σ
(Byu11 p. + (1 - --))9==)) - (Byuh-7 1 + +1 B,)
M
m=1
(1 p,) 8, - - (1 3 - 92-1) 1
I
(1 p,)
- - + (1 - p;)(1 3,) 96 ||
(1 - p,)(1 3,) Σ (1--p)) -,),,) Blgg+ - 9 97
ut 112
W
u ne 11
W/ (1 - (1- 33,)
-Bllg- 9²'1
==0
N
Σ j=1
wj(1 - -p;((1 - -3,)((1 -1;1;,,
Bll,t - 11'
N
Σ wy(1 P1) -,)0
Ellgr 97 07"112
Denote Cr. = (1 P1) + 8), d' = (1 - -P=)(1 + 1/4) and = (1 - Pr;88. Combining the previous
two bounds, get
M
Bllae xt 1|?
M
m=1
M
1f
4x
1-T
M
E
" 1
(4)
mm
M
11
w7(1 p,)(1 B,095
Ellg, - 1/
M
m=
v=
N
1
d 4
w3(1 p;)(1 1 (8))
1-T-1 --
9j
E Elgg. g. 9-'1
M m1=1
1=1
7=1 ==
N
1-1
M
1/ 44 Σ
w1(1 - p1) (11--1)
1-T 9j
Ellg 0mm1"
M
1=1
1= +=0==
m=1
N
1-1
M
1/ 12 Σ
w - (1 Σ
L
E [g
g gm1
4j
M m=1
j=1
9 v
124 Σ
w)(11 31) - Pp))
M Σ E
Eg
9. 11
Qr 4j
1=1
27

---

Published as a conference paper at ICLR 2026
Next, we bound the gradient term above.
M
K
1
gell
E
9
96
M
M
M
M
M
2
IE
96
V fm
1
Σ
(sf vf((x())
K
M
2
1
V
mm (x7")
M
M
(Lemma
2
Σ
Vm (x )| 2E
C Thhh v.n
M
M
1
E E| x7 1721|
6G
M
m=1
12L
M
20 +
E 24 - + - - 6G
M
m=1
Averaging over the iterates and plugging this bound to the previous one, we get
T T-1
M
22 - 11
MT
t=0
m=11
T M
E E| 11?
MT
t=1 m=1
n d
L-T qj
1 M
Σ
w (1) (3-11)p)) P,)
E Bll 9 1
T
9r 9j
M
1=1
d
T-
T
41 1-T oj
M
- - 3,)(1 -P)) Σ
Bllgr - 9 11
T
4e 4j
M
1=7+1
m=1
wy p.)
T-1
qa(1
4 T--)
g,(1
1 M
E
El|g
9, 112
4s -9i
1 1-4a
1 Qj
M
m=1
<
n2
wy(1 - -1) PÚ)
9r
M
E
g-
9 112
T
9x
Qu
- 9j
M
n'4
W/ (1-1111)) - (1:B))(111 PO)
M
Bligr 11
T
9x) (1 43)
M
m=1
Now, let optimize the factor
d/
(1 (1-pa) 1/a)
4x
(1 - (1+
by choosing optimal value for s introduced earlier. By the first order optimality condition, we find that the optimal value is 8"
-í-P. Hence, the minimal value of the factor is
1
La
Pz
1 Qx
(1 VI )2)
(1 - P1) II- ?
(1 - (1) + VI 1Ipp))
4(1 Pw)
<
(1
/1 Pa 22 + V1 )2
2
p2
28

---

Published as a conference paper at ICLR 2026
Letting
4(1
Pw)
N
(1
81)(1 - -P))
4(1 Pv.
Σ
N
(1- (1- - pj)
wij
wij
P
1 (1 py)
1=1
1=1
and continuing the chain of bounds, we get
22"1
MT
v/
E 99: 96" 11?
T
1
S
npv.
-1)_===))) + 20 + 6G2 E - x
M
m=l
T-1
M
<
L2
E1 - xt x211
TM
1=0 m=1
+ (B² - 1)
E 2 + +3G
T
Assuming 12y < 1/2 and reordering the first term in the bound, we arrive
T-1
Σ
B It - 1111 S BB² -1)+ + 002 20000((+330"0 +3G
MT
Lemma 4, Under smoothness and bounded heterogeneity assumptions 1 and 3, we have
1 .
Σ
Σ vi((
6L
M
for any y² ,.... nh E R and y Em [..
Proof. The proof follows from Lemma 5 of (lacob al., 2025) as the result does not depend on the optimizer.
E WALL-CLOCK TIME MODELING
To assess the practical benefits of our proposal, we analyze its impact on total wall-clock time by modeling two distinct synchronization strategies: simple unified frequency approach and desyn- chronized approach based on optimizer state half-lives. We adopt the model from DES-LOC (lacob al., 2025) for estimating total training time.
E WALL-CLLOCK TIME MODEL.
The total wall-clock time is modeled as the sum of computational and communication time: tooaa = fcompute foomns The computation time, fcompute function of model and dataset size, while the communication time, fcommss depends on the number and size synchronization events.
For training process of T total steps, the communication time for AllReduce operation (Sergeev Balso, 2018) depends on the payload size, number of workers M, bandwidth B, and latency The total time for different methods and strategies is:
29

---

Published as a conference paper at ICLR 2026
Unified Frequency Methods: Parameters and all optimizer states synchronized together every steps. The total payload 3d (for parameters, first and second momenta). This applies to Local Adam and baseline version of our method, -T-DAO (Unified).
2(3d) B
Eootal, Unified
Ecompute
+ K
1
+1
(5)
M
Parameters (Ka), first momentum (Ku), and second momentum (K₂) are synchronized at different frequencies. This applies to DES-LOC and our proposed method, MT-DAO (Half-Life).
Half-Life Based Methods:
2d B
total Half-Life
Ecompute +
1
+1
(6)
K₂
K
K
M
Limitation: This model does not account for any potential overlap between computation and communication.
E.2 EXPERIMENTAL CONFIGURATION
We compare the two synchronization strategies. The Unified Frequency strategy serves as baseline, where all states are synchronized together every K₂ 32 steps. This includes Local Adam, Local ADOPT, and a variant of our method, MT-DAC (Unified), which uses high B3 value but is forced to syne at the same frequent rate as its parameters. These methods have equivalent communication costs and will overlap for an iso-token budget, however, the results Section 5 show that MT DAO achieves the same perplexity as Local ADOPT in many fewer optimization steps, outperforming on time-to-pent mone-to-perplxxtt metrics.
The Half-Life Based strategy aims to improve efficiency by synchronizing states less frequently if they change slowly. The synchronization frequency is set based on the state's 's half-life, (3) = l0(0.5)/ (n))). This includes DES-LOC and MT-DAO (Half-Life). The quasi-hyperbollic (QH) configuration of MT DDO allows it to use an extremely high 31 = 0.999, leading to a very long half-life and thus much lower communication frequency for its first momentum. We use 32 = 0.999 for ADAM variants and 32 0.9999 for ADOPT variants.
Table 3 details the configurations for both strategies.
Table 3: Hyperparameter configurations and synchronization frequencies (K) for modeled methods, grouped by synchronization strategy. For the Half-Life strategy, momentum frequencies are set the closest power of two to their half-life.
Strategy
Method
Values
3 Values
& Value
Syne Freq. K.,
Syne Freq. K,
Unified Frequency (All states sync every K2
32 steps)
Unified
Local Adam
N/A
(0.95)
0.99
32
32
Unified
Local ADOPT
N/A
[0.95]
0.9999
32
32
Unified
MI-DAO-Adam (Unified)
(0.95)
(0.999)
0.999
32
32
Unified
MI-DAO-ADOPT (Unified)
(0.95)
(0.999)
0,9999
32
32
Half-Life Based Frequency (States syne different rates from K2
32)
Half-Life
DES-LOCCADDM
N/A
(0.95)
0.99
32
69
Half-Life
DES-LOC-ADOPT
N/A
[0.95)
0.9999
32
6931
Half-Life
MT DAO-Adam (Half-Life)
{0.95)
(0.999)
0.999
693
693
Half-Life
MI-DAO-ADOPT (Half-Life)
(0.95)
(0.999)
0,9999
693
6931
E.3 MODELING RESULTS
The following figures present the estimated wall-clock time and communication costs when training IB model on 4 H100 machines with batch size of 2M tokens and sequence length of 2048. The results demonstrate that MT-DAC significantly reduces communication cost with both strategies, with the half-life one being generally more effective.
30

---

Published as a conference paper at ICLR 2026
ADAM Variants WalkClork Tine
ADOPT Vartants Wall--oork Time
124
D00
201
124
724
377.00 (EESE
20
182
16
Basbeitl
PS Raddaiall
(a) Adam Variants
(b) ADOPT Variants
Figure 9: Estimated total wall-clock time as a function of interconnect bandwidth. For both (a) Adam and (b) ADOPT, methods using the Half-Life strategy outperform those using Unified frequency.
ADAM Variato Communicaton Cort
ADOPT Variants Commmisiion Cost
12a0
NETOMO (Calett
LET (iiile)
Charsaaoke thee K0)
D&S LAC (200L:0
Chassaaoo tten 60)
DCSLOC (LUSLR
107
20F
10h
167
32
Basbeial (Chbni)))
(a) Adam Variants
(b) ACOPT Variants
Figure 10: Estimated total communication time. The plots clearly distinguish the two strategies. The Unified frequency methods (Local Adam and MT-DAC (Unified)) have identical high costs. The Half-Life methods are more efficient, with MT-DAO (Half-Iiffe) being the most efficient due to its ability to leverage high-3 momentum that requires infrequent updates.
Takeaway: MT-DAO can significantly reduce communication costs across bandwidths.
F DERIVATIONS OF MUTUAL INFORMATION AND VARIANCE
This section provides the detailed derivations for the expressions referenced in the main text.
F.1 VARIANCE OF LOCAL MOMENTUM
The variance of the final local momentum, Var(u++ ), is derived under the assumption that the stochastic gradients g: are independent and identically distributed random variables with variance o
The unrolled momentum update over K local steps is given by:
3k u4 (1 3) Σ 3 9
u4+
The variance is calculated with respect to the randomness in the local gradients [g++1..... 9t+ k) }. The initial momentum u, is treated constant, as it is synchronized state before local updates begin.
Applying the variance operator:
K-1
Var(u+*) = Var (kKu + (1 B)
Σ
3h gt+ K
k=0
31

---

Published as a conference paper at ICLR 2026
Since u, is constant, Var(3* (u)) = 0. Using the property Var(aX) =a² Var(:(X))
K-1
Var(u++k) = (1-
Σ
gttkk
Given the assumption that the gradients g+++ are independent, the variance of their weighted sum is the weighted sum of their variances, where weights are squared:
K-1
Var Σ 8 ge+ k
Var( g(*K+))
Σ
k=0
Assuming each local gradient has variance o.:
K-1
K-1 Σ
K
Var
8 9 -k
8 m
o
(32gth
A==
k==
k=0
The summation is a finite geometric series, Σx pe
L-na I-r
With r = 3 and n = K:
K-1
1 1 ((33)K
-22K
Σ
(Bagh
1 1-33 1 - 83
0=
Substituting this back into the expression for Var(ue+*))=
Var(u K) (1 - -3)3000 1 - 32K
B²
By factoring the denominator 1- -== B3(1 B), we can simplify the expression:
1- 3 1- B-32K
Var(u = (1 3)20 om
1 3 (1- 52K jom
(1 - (1 + 3)
This completes the derivation.
F.2 MUTUAL INFORMATION
The mutual information T(U++K;; U2) is derived by modeling the momentum states multivariate Gaussian random vectors. The model for the update process is:
U+K 3 U, + T
The following assumptions are made:
1. The initial momentum U, a Gaussian random vector with zero mean and covariance Σv,
i.e., U, ~ E.,)
2. The accumulated local gradient noise L is Gaussian random vector with zero mean and covariance Σr, i.e., L NO Σr.)
3. Ug and L are statistically independent.
The mutual information between two random vectors X and Y is defined as I(X;Y) h(Y)- (X), where /(-) is the differential entropy. For a d-dimensional Gaussian vector 2 ~ NN, E), the entropy is h(Z)= 1/2 log (2.......
First, we determine the distribution of Ue+K- As a linear combination of independent Gaussian vectors, is also Gaussian.
Mean: [[[U++]] E[8K U, + L] 3 [[U B[L] = 0.
* Covariance: Cov(U++K) = Cov + L). Due to the independence of U, and L:
EU Cov(3* + Cov(T.) 32K Ev, + Er.
32

---

Published as a conference paper at ICLR 2026
Thus, ULtK NO, B2k Ev. ΣL).
The entropy of UttK is:
h(U++R) log det (2TE(B2k Dv,
Eu, + Er))
Next, we determine the conditional entropy (u Uz). The distribution of UL+K conditioned on a specific value U2 ut is:
UU 4s N(B
S ue, ΣL)
The entropy of this conditional distribution is:
h(U++ UU, =4) log det (et(2
Since this expression does not depend on the specific value uy the conditional entropy h(U++x|U) (U,) is the same.
Now, we compute the mutual information:
I(U(+K:U:) = h(U1+K)) -h(Ue+x(UU)
KUU+K; U1) log det (2 EV, + EL)) log (2te
2
2
Using the logarithmic property loga log b log(a/b):
I(U+4KiUi) = log /det(2Te(82* Eu, + Ex,)
del (2Te
det(A) The constant factors (2me)² cancel out. Using the determinant property det(B) = det(AB-1)
I(U+KK:U)) log det ((32K Ev. + +E
Distributing Σz inside the determinant:
1
I(U+4K; U.) _ log det (B2K Ev, Ev, + 2T.1
1 I(UL+K;U1) = log det (I + 32K EV, 2v,
This completes the derivation.
G EXTENDED RELATED WORK
Strategies for Coamunncationn-Efliieeee Distributed Training. A substantial body of research aims to curtail communication overhead in distributed training, primarily by either reducing the frequency of synchronizations compressing the data transmitted per round. The first approach, often termed periodic or local SGD, involves performing multiple local optimization steps between global aggregations. This strategy has been extensively analyzed in both IID and non-lID contexts (see Kairouz et al. (2021) for a survey and Lin et al. (2018a)). In the realm of foundation-model pre-training, methods like DiLoCo (Charles al., 2025) have shown that infrequent synchromization can, with careful tuning, achieve performance comparable to or better than standard data parallelism, with scaling laws characterizing its behavior across model sizes (Charles al., 2025). This paradigm has also been adapted for federated-style pre-training (Sani et al., 2025) and variants with overlapping or eager updates (Douillard et al., 2025; Kale al., 2025). The second strategy involves compressing communication payloads. Techniques range from randomized quantization (QSGD) (Alistarh et al., 2017) and sparse updates tailored for non-IID data (STC, ZeroFL) (Sattler et al., 2019; Qiu et al., 2022) to one-bit aggregation (signSGD-MV) (Bernstein et al., 2018). In practice, these two strategies are often combined; for instance, FedPAQ integrates local training with quantization and partial participation to provide strong theoretical guarantees (Reisizadeh al., 2020).
33

---

Published as a conference paper at ICLR 2026
Multi-Timescale Momentum for Temporal Mismatches. The temporal discrepancy between frequent local updates and infrequent global synchronizations creates a need for optimizers that can integrate information across different timescales. Standard momentum, while beneficial in low-curvature landscapes (Sutskever et al., 2013), imposes a compromise: low decay values are responsive but slow, whereas high decay values are fast but prone oscillations (Lucas et al., 2019). A single exponential moving average (EMA) cannot effectively weight both recent and distant gradients (Pagliardini et al., 2025). Multi-timescale optimizers address this limitation. Quasi- Hyperbolic Momentum (QHM) decouples the current gradient's 's weight from the momentum decay rate (3) (Ma Yarats, 2019), recovering methods like Nesterov and Triple Momentum (Scoy et al., 2018). Aggregated Momentum (AggMo) maintains and averages multiple momentum buffers with distinct 33 values, using faster-decaying terms to passively damp oscillations caused by slower, more aggressive terms (Lucas et al., 2019). Similarly, AdEMAMix mixes fast EMA with an ultra-slow one (c.g., 33- = 0.9999), demonstrating that long-ter gradient memory significantly reduces catastrophic forgetting in language models (Pagliardini et al., 2025). This principle of leveraging multiple timescales is also present in other contexts. Optimizers like Grok asst (Lee al., 2024) and AdMeta (Chen al., 2023b) employ nested EMAs for different purposes, providing orthogonal evidence for the value of long-term momentum. While these methods have shown promise in step-wise synchronous training. their potential to resolve the temporal mismatch in communication -effriiennnffi distributed optimization remains largely unexplored.
Perspectives from Federated Optimization. The field of Federated Learning (FL), particularly in the cross-device setting, offers rich history of methods for managing statistical heterogeneity and communication constraints, which are central challenges. The foundational FedAvg algorithm (McMahan et al., 2017) has inspired numerous successors (see survey by Kairouz et al. (2021)). To counteract client drift caused by non-IID data, FedProx introduces a proximal regularizer for stability (Li et al., 2020), SCA FFOLD employs control variates reduce gradient variance (Karimireddy et al., 2020b), and FedNova normalizes local updates to correct for objective inconsistency (Wang al., 2020). Server-side momentum (FedAvgM) has also been shown to stabilize aggregation under data skew (Hsu et al., 2019). Adaptive methods have been extended to this setting in Adaptive Federated Optimization (FEDOPT), which provides nonconvex guarantees for FedAdam, FedYogi, and FedAdagrad (Kingma & Ba, 2015). Furthermore, Mime adapts centralized algorithms to FL by marrying control variates with server statistics (Karimireddy et al., 2020a). Personalization techniques, such as meta-learning-basee Per-FedAvg (Fallah et al., 2020) and FedL,PP (Lee et al., 2023) or the regularized Ditto (Li et al., 2021), complement these global models by improving per-client utility.
Orthogonal Approaches in Payload Compression and Optimizer Design. Orthogonal to reduc- ing synchronization frequency, another line of work focuses on compressing the communication payload itself, often in combination with periodic training. Foundational methods include quanti- zation, as in QSGD (Alistarh et al., 2017), and sparsification, as in Deep Gradient Compression (Lin et al., 2018b) 8b), with convergence analyses providing theoretical grounding (Alistarh et al., 2018). More recent work like CocktailSGD combines random and top-k sparsification with quantization for aggressive compression during LLM fine-tuning (Wang et al., 2023a). Beyond compressing gradients, some methods compress the optimizer states. For instance, LDAdam performs adaptive updates using low-rank approximations of gradient statistics (Robert al., 2025), while DeMo decouples momentum across workers and communicates only selected components (Peng et al., 2024). Other advanced optimizers aim for stability and efficiency through different mechanisms; for example, Lion uses sign function with interpolated momentum (Chen et al., 2023a), and Sophia employs Hessian-based pre-conditioner to temper step sizes in high-curvature directions (Liu et al., 2024). These approaches are generally compatible with and can be composed with infrequent synchronization strategies.
H ADDITIONAL RESULTS
This section presents the full suite of additional experiments conducted during the review process. We begin with toy-problem stability check (Fig. 11), followed by time-normalized downstream evaluation (Fig. 12), momentum counntttumoountttto ablations N E {1, (Appendix H.2), batch-size and worker-count sweeps (Appendix H.3), streaming communication comparison (Appendix H.4),
34

---

Published as a conference paper at ICLR 2026
A
00
10C
303
301
500
663
Communication Rounels
Stat
17-080
Blax
(a) Distance to optimum vs. steps.
(b) Contour plot of trajectories.
Figure 11: MT-DAO remains stable both fast regimes, as pictured here, and in slow regimes as in Figure 1. This is unlike prior stateful methods like Local Adam which only offer stable convergence for fast 3 values As before, we optimize the non-convex Rosenbrock function f(z1, x2) = (1
x1)² + 100(x22 2² with M 256 workers and IID Gaussian noise (a 2).
(WVG-
ARC-Challenge (leehot)
ARC-Eaey (0-shos)
Score (76)
Store (30)
Score (166
88
EANO SA
8
CLi GLANDNEE
OUL
50
Time (Hours)
(Hours)
Time (Hours)
HellaSwnn (I-ahot)
(MLUU(-shot)
PIQA (a-shon)
Store (%6
22
Score (96)
58
2018
125-0 wi oee $1.66
Gpttewee 21.40
20
30
50
Time (Hous)
'I Time (Hour))
Time (Hous)
Downstream task accuracy versus time on ARC-CHALLENGE, ARC-EASY HELLASWAG, MMLU, and PIQA. Curves compare MT-DAO, Local ADCPT, Nesterov, AdamM- DDP, and QHAdamW-DDP: horizontal reference lines (when shown) indicate external model baselines. Error bars denote too over samples. MI-DAO exceeds AdamW-DDP on the aggregate and on reasoning-heav tasks (ARC-CHALLENGE. HELLASWAGE with smaller but steady gains on MMLU. As training length increases, MT DAO closes the gap to QHAdamW- DDP, consistent with the perplexity results. Nesterov tracks AdamW-DDE and outperforms Local ADOP T, but remains below MT-DAO across metries, normalzzed results are provided in Figure 7.
Figure 12:
gradient-clipping ablations (Appendix H.5), the synchronous-limit K=1 analysis, and Muon experi- ments (Appendix H.7). In total, these comprise 235 new experiments spanning five inner optimizers, model scales from 16M to 360M, and up to 8 workers.
To investigate the stability of MT-DAO under varied momentum parameterizations, we first examine its performance in fast 3 regime. Figure 11 presents the results of this comparison, plotting both the convergence rate in terms distance to the optimum and the optimization trajectories on the function's contour plot.
35

---

Published as a conference paper at ICLR 2026
H.1
DOWNSTREAM EVALUATION
Time-to- T'mee-t-TTrrge Advantage: MT -TODAO achieves downstream target accuracies faster wall-clock than DDP baselines in realistic multi-node setup (4 H100, 100 Gbit), maintaining improving quality while reducing communication.
H.2 ABLATIONS ON THE NUMBER OF MOMENTA NV
In our main work, we focus on MT-DAO with N specifically its Muasi-Hyperbolic (QH) formulation. We argue this provides a "free lunch": it captures instantaneous gradient and one slow momentum) without the memory and communication overhead of storing and synchronizing additional buffers.
dynamics (via the
In this section, we empirically validate this design choice by ablating the number of momenta N. Guided by prior work on multi-momentum optimizers (Lucas et al., 2019; Ma & Yarats, 2019; Pagliardini et al., 2025), we define the optimizer update as convex combination of N states with weights Σ wi = 1.0. We distinguish between two families:
Standard variants: N momentum buffers are maintained. This corresponds to AdEMAMix
(N = 2) or AggMo (N = =).
2. Quasi-Hyperbolic (QH) variants: The fastest momentum buffer is replaced by the current
preconditioned gradient gr. Mathematically, this is equivalent to setting the fastest 3 0. This corresponds to (N 1, used in our main text) or QHAdEMAMiX (N 2).
Experimental Setup. To ensure fair comparisons without the confounding factors of complex scheduling introduced Pagliardini et (2025), we use Adam as the inner optimizer for all variants in this section. We utilize the WSD learning rate scheduler. Due the combinatorial explosion of hyperparameters as N increases, we fix the decay rates to exponentially spaced values following the recommendation of Lucas et al. (2019): 3 1 - 0.1".
Tuning Protocol: We tune the mixing weights and the learning rate y the 16M model for 12, 288 steps. We then transfer the optimal configurations to 125M and 360M scales trained for 40, 960 steps.
Baselines: We compare MT-DAV DAO against DDP baselines for every N to measure the perfor- mance gap.
H.2.1
MT-DAO (N 1) vs. MT-DAO (N 2)
We first compare our default MT-DAO (N = 1) against a standard MT-DAO (N = 2) which maintains two explicit momentum buffers.
Configurations: For N = 1. For N 2, we use a fast momentum (31,1 = 0.9) and slow momentum (31,2 = 0.999).
the gradient and slow momentum with 31 0.999.
Grid Search: We sweep the weight of the slow momentum Welow
{0.5, 0.6, 0.7, 0.8, 0.9, 0.95] and the learning rate {1, 4, 8, 16} x Tome-
Results. Figure 20 and Fig. 21 visualize the tuning results. We find that shifting weight towards slow momentum consistently optimal. Figure compares the convergence of optimal configurations across scales.
We observe critical distinction between the DDP and MT-DAO regimes. In DDP, adding a fast momentum buffer (N = 2) yields slight perplexity gain over N = 1 (0.01 at 16M, 0.02 at 125M and 0.44 at 360M). However. for MT MT-DAO, the 2 variant performs worve than the = variant by approximately 1.5% 125M scale.
Interpretation: In the infrequent communication regime (K- = 32), a fast momentum (3 = 0.9. Tu.s steps) decays almost entirely between synchronization steps (332 0.03). Consequently, the worker's 's fast momentum buffer becomes decorrelated from the global direction, injecting noise. In contrast, the QH formulation (N 1) uses the instantaneous gradient the fast component.
36

---

Published as a conference paper at ICLR 2026
ME-DMD (23-52
MI-MMO (K-1) (9-22
MI-MO (8-1) 26.6
MT-DAA (N..2)
MET-DA (N-2) 9999
SET- TAO (S..2) 9.:6
JdkaW-DDP
AdaaW-DDP 1843
Qliddaas (X-1)-0P (22.4))
Qiidiane (N-11220F (355))
QHiiiam (8-1-DDP (616)
Aderabitt (N. 11-000 (23.41
Admmniite (N 3, 2.4D (9-37]
Adonsitte (N_ 2D 17.95
191
Tiae
ATT-DAA (S. (N..1) (22-5))
RET-DAO (S- 1952
MET-RA (8
(5-2) (33.88)
154
(N-5)
Ael (24.55;
Joleaz DDP (3007)
JokasW-DDP (5.41)
Qullddaree (S 1)-UPP (2L46)
QuiAdaaee (8- 1-DUPP (4.6)
Quthdiazee (Ni D-D (616)
(9:37)
Telne
11
Figure 13: MT-DAO (N 1) vs. N 2 variants. Validation perplexity vs. Wall-clock Time (top)
and Tokens (bottom) for 16M, 125M, and 360M models. Takeaway: The N = 1 Cuaas-Hyyperboli variant matches beats the standard N 2 variant in loken-efficiency while being slightly faster in wall-clock time due to lower communication volume.
Thus, MT-DAO (N = 1) captures the fast momentum.
benefit without the instability of a decaying
Memory and Communication Efficiency of N = 1: MT-DAO with N = (Quasi-Hyperbolic) matches the token-wisg performance of N 2 methods while reducing model-state memory usage and communication volume by 33%.
H.2.2 QUASI-HYPERBOLIC N 2 STANDARD N
To see whether increasing the number of momenta terms beyond N brings about further improvement, we proceed to compare a QH N 2 variant (QHAdEMAM: x) against standard N (AggMo). We repeat the same experimental design as was done in H.2.1, specifically:
Configurations: For QHAdEMAM (N 2), we the instantaneous gradient, medium momentum ($1,1 = 0.99) and a slow momentum (31. = 0.999) term. AggMo (N = 3) instead uses fast momentum, (31,1 = 0.9) medium momentum (3 0.99), and finally a slow (81,3 0.999) momentum term.
Grid Search: For QHAdEMAMi and AggMo, we sweep over the configurations:
{0.05, 0.05, 0.9). (0.05, 0.25, 0.8), (0.05, 0.25, 0.7), (0.05, 0.35, 0.6), (0.05, 0.45,,050000 0.5), (0.1, 0.1, 0.8), (0.1, 0.2, 0.7), (0.1, 0.3, 0.6), (0.1,
In the case of QHAdEMAMi x, the first in the tuple refers to wg. which is applied the gradient, whilst for AggMo this is W1 and this is applied to the fast momentum term. All other elements of the procedure and all other parameeas remain the same, including the optimal base learning rate 1) = 0.01.
We note that in the main text we used a slightly higher W1 = 0.98 as recommended by Ma & Yarats (2019) beyond what we found to be optimal in short runs in order guarantee stability at very long training horizons. However, the goal for this section is to provide fair comparison across N's under similar tuning budgets,
always use the optimum found by 12288-step runs.
Results. Observing Fig. 22 and Fig. 23, we find that similar trends occur as in H.2.1, where both formulations prefer hyperparameters that shift the weight of the terms to the slow momentum.
37

---

Published as a conference paper at ICLR 2026
ME-DAD (N-2QR (2572)
MI-DAO (N-2QQ) (A78))
ME-DAD (N-GAA (8.2))
METAA (N-2)) (5)
MIE-NAO (N-.) (1022)
MTT-AA (N--2) (5:57)
MiaanMis (S-S-0HDDDP
AlenseMaa (N-B/OR-DOP (9.50)
Ausble (X=S)DDPP (24.4))
Auchlo (K-3)EOP (1522)
Auzhii (X-3)20P (8.30)
192
104
Time (llaa))
MEE-AA (N. 2Q13 (2L72)
ME--AD (N (N.2QQ)) (6.76)
MEE-DAD (N-2Q00) (8
(N-3) (9L
MILDA (N-2) (10 53)
(N--2) (8.5)
Adeanekis, (X-2,000DDPP (20.5)
Adezsablaa 2X-2QQ8-2OP (2.5))
Aieasala (N-2 2--200--2PP (8.04)
Appblo SN--P) (ALLN))
Apphli (8 -5-2UP (1.2))
Appilin (S--P) (6.26)
Teen
Ti
Figure 14: MT-DAO QH (N 2) vs. Standard N (K 32). Validation perplexity across scales. The QH N = 2 variant (orange) consistently overlaps or outperforms the N 3 variant while requiring one fewer memory buffer.
When evaluating the well-tuned models in Figure 14, we see that across model scales, the MT-DAO (N 2,QH) formulations perform better than their N counterparts (0.83 at 16M, 0.47 at 125M and 0.11 at 360M), with the gap between them decreasing as the model size increases. When comparing these results with those observed in Fig. 23, we arrive at the following conclusion: the addition of more momentum buffers has diminishing returns for our method as N > 1. As such, this motivates our decision to prioritze MT-DAO with N in our main experiments: provides good tradeoff in terms of performance, whilst minimizing the memory overhead of maintaining additional momentum buffers.
Diminishing Returns of High N: Inereasing N beyond yields marginal token-wise gains that are outweighed by the increased memory, communication, and tuning complexity. The Quasi-Hyperboliic formulation consistently the superior choice for infrequent synchronization.
H.3 ABLATIONS ON THE NUMBER OF WORKERS AND BATCH SIZE
In this section, we investigate the interplay between the global batch size (Bg) and the number of workers (M) for both MT-DAO and Local 1 Adam. We utilize small model configuration since we can easily use batch sizes orders of magnitude larger than necessary, simulating very latce-batch regimes. For Local Adam, we employ the standard Adam W hyperparameters 31 0.9, B32 0.999. For MT-DAC, we utilize our robust configuration with 31 = 0.999, B2 = 0.999 and a convex combination coefficient w1 = 0.98. We vary the global batch size BG E (64, 128, 256, 512) and the worker count M E {1,2, 2, 4, 8), noting that the effective por-worker batch size B is given by Bc/M.
Figure 15 demonstrates that MT-DAO is significantly more robust to increases in the worker count M. As M increases for a fixed global batch size, the per-worker batch size Be/M decreases, injecting higher noise into the local updates. As shown in Figure 15(c), the performance delta generally increases
move down the y-axis (increasing workers), indicating that Local Adam degrades faster than MT-DXX -DD in high- noise regimes. This robustness stems from the multi-timescale design: MI-DAO relies heavily on the slow, shared momentum (governed by W1 0.98) rather than the noisy local gradients.
This empirical behavior aligns with variance analysis derived in Appendix F.I. While increasing the batch size Bg reduces variance linearly, the momentum decay rate 3 controls variance exponen- tially relative the synchronization period K. The variance of the local momentum state u x can
38

---

Published as a conference paper at ICLR 2026
Vabieiis Pupploaty
Valilries Pemploity
of Validation Porploctize
2213
20.20
2:41
31.73
21.2
20:2
21.0
1391
2:35
2-8
3.0
25.6
2:45
2336
31.00
23
3:41
2433
675
2018
20.74
2.43
12.11
aurd
23:23
232
835
125
Okied Sanch Bie
Ukid Barch Fee
Oliball Sarch See
(a) Local Adam.
(b) DAO
(c) PPL Ploce Adinn
PPLMT-DAO
Figure 15: Robustness to Batch Size and Worker Count: Validation perplexity heatmaps for (a) Local Adam and (b) MT-DAS across varying worker counts and global batch sizes. (c) shows the performance gap, where positive values indicate MT-DAO outperforms Local Adam (lower perplexity is better). MT-DAO demonstrates superior stability as the number of workers increases (and per-worker batch size decreases) compared to the baseline.
be approximated as follows for global batch size Bg and number of workers M:
1 3
0²
Var(u+*)
g2k)
1 1+
BG R
Drift Term
Standard optimizers with low 3 (e.g.. 0.9) fail to suppress the drift term (1 - 32K) when K is large. MI-DAO utilizes high 3 (0.999), suppressing this term significantly. Consequently, MT DDD is less sensitive to the linear increase in batch noise Ba/M caused by increasing M, allowing it to scale o²
efficiently even with small per-worker batches.
Performance at Large Batch Sizes. Finally, we observe that MT-DAO maintains a performance advantage even at large global batch sizes (B = 512), more appropriate for a model roughly 45x larger. According to the empirical model of large batch training (McCandlish al., 2018) and more recent works on the topic (Zhang et al., 2025), there exists a critical batch size past which further increases in batch size do not lead to linear improvements in performance-ffo-ccomppu due to a decrease in sample efficiency. McCandlish et al. (2018) argue that this happens once the batch size sufficiently denoises the gradient. Given the much better performance of BG E (64, 128) compared Bg 512, our model is in this regime. However, despite the global batch size being sufficient to reduce gradient noise, Local Adam still struggles due to the timescale mismatch inherent in infrequent communication. MT-DAO stabilizes training by preserving the global optimization direction across rounds, proving that the benefits of multi- timescle tracking extend beyond merely compensating for small-batch noise. We also observe that for larger numbers of workers, the fall-off in performance the global batch size increases is not as sharp for a single worker (equivalent DDP), for example, the 8 worker case, the optimal batch size of MT-DAO 128 rather than 64. This corroborates the findings of Charles et al. (2025), which indicate that communication-efficin training methods can benefit from larger batch sizes compared to DDP.
Robustness Parallelism: MT-DAO consistently outperforms Local Adam as the number of workers increases, effectively mitigating the variance introduced by smaller per-worker batch sizes. It provides stability across diverse batching regimes, from noise-dominated small batches to large-batch settings typical of massive-scale training. We also observe a potential increase in the optimal global batch size for MI-DAO at higher numbers of workers.
H.4 MT-DAO COMPARISON AGAINST STREAMINGDILOG.
In this section, we demonstrate that the benefits of MT-DAO are orthogonal to and synergistic with the streaming techniques introduced by Douillard al. (2025). We introduce St reamming MT-DAO, variant that synchronizes fragments of the model and optimizer states continuously the background, as formalized in Algorithm 5 for Adam. We adopt the fragment-based approach where the model
39

---

Published as a conference paper at ICLR 2026
0 is partitioned into disjoint fragments Fi...., We extend StreamingDiLoCo which only synchronizes parameters (and an outer optimizer state), so that our framework supports N inner momentum states. We decouple the synchronization frequencies of the model parameters (K,,), the N first momenta ({K;) D, and the second momentum (K,). We use averaging as the outer optimizer for all model states by default. The inner update step (1..24) retains the multi-timescale structure of MI DAO. For the St reamming Local Adam baseline, we implement the same algorithm but with N I, standard single-timescale momentum (w/ 1.0).
Agorithmm Streami ng -DD (Fragmented, Overlapped, Multi-Scale)
Require: Model tensors, hyper-parameters
I: Zo E R", fu', AN E (R U- R initial params, N first momenta, sccond momentum
2: 2 Global state (for all Z {z, u.)))
3:
Fi....,FF
partition of lensors into F fragments with offsets & K2, (K, 1 1- K. E
4:
(N+ )
independent syne periods for tensors
5: a, E [0, 1J, blending factor, inner overlap delay
OuterOpt 2 outer optimizer specific tensor type Z E {a, u, t}
Ensure: Z7: [u
Vr-1
7: initialize worker states z0" (w_"", J. =" identically for all workers
8: initialize global state Zo +- Zui for all tensors Z E far w', o
9: for parallel 1... do
10: for ..... T' 1 do
for each fragment f (1...., F} and tensor far, (w"),""]]] J, do K < Kz
12:
select freq. for at, us
13:
if (t tz) mod K == 0 then
check syne schedule
14:
A (f) < z( m 2
15:
hz assnc- send Σ A
broadeast delta
(t - ty - r) mod
then Awe < block -block-receivee (eeekkeeeevee
check overlap schedule wait for data
ă +- OuterOpt (2 (2, A
19:
z + %
archive new global state at index
20:
zin.m a2(5).m + (1 a)
merge
21:
9hth + F(a"" ).P)
clipped stochastic gradient update momenta
22:
for
j = N do
23:
" mm +- u j.m
+ 315)
24:
0i + 320 + (1 (gt"
25:
A.
tor+ (1 Σn wi w1)9%' En wyul
26:
#1 <- It" Am
27:
end parallel for
Streaming Schedule and Offsets: Following Douillard et al. (2025). we assign synchronization offset fy to each fragment. This enforces a round-robin schedule where, at any given step f, only a small subset of the model, specifically, fragments where (t (4) (mod K) = 0. This design flattens the communication spikes into a more uniform stream.
Independent State Frequencies: Streaming MT-DAO manages N + 2 distinct tensor types: the parameters x, N first momenta {u²), and the second momentum v. This opens new design space for distributed optimization: we define independent synchronization periods Kz, (Ky). J, Kr (L.6). This allows for schedules where rapidly changing parameters synced frequently, while stable. long-term momentum states (high ()) are synced on slower timescales.
To hide the latency of these synchronizations, we implement the overlap mechanism proposed by Douillard et al. (2025). The inner optimizer continues stepping for 7 iterations while the outer gradients transmitted and aggregated. The updates are then merged into the live stream with a blending factor (Lines 11-16).
Computation Overlap:
40

---

Published as a conference paper at ICLR 2026
Strarsing MI-3AG (26.135
Strvnning MI-3AG (M55)
Stmazinng MT-310 (638)
Straninng BilaCh (55 R0)
Straning TillCh (to (9)
Smenring TiltaC 18 my)
Sirennin Loadl bolan (26.7)
Stverinn Loce Jdan
Saeniin Loeel Mleu
MI-DAO (2583)
MT-DAO BUS
MT-DAO (825)
-PL Qliddero DX2 (33.91)
Qliddome DE2 (9
QilAdoare DDE (+1K)
Adlass-DOP
AlanW-EDP
Adkontt DGP
W.D.lak Tixe (llaao)
Strvanttg MI-3AA (26135)
Strwastng MT-3AG
271
Strvanting MT-IAA (8:20
Streasiin
Strenting
Steercing
Strominn Locd Man (2L78)
Strominn Local Jeaa (11.1)
Saaamiig Local Auhas (144))
MI-32A (2181)
ME-BAM (AND)
MI-IAD (6225)
Nitare-DDP (2416)
AituN-EDP (1)
Adhuii DDP (8 48)
Figure 16: Streaming MT-DAO vs. StreemiigpiLooo. Validation perplexity vs. Time (top) and Tokens (bottom) for 16M, 125M, and 360M models. Takeaway: Streaming MT-DAO (orange) consistently outperforms streamingDiLco Streamingte IeesamingDiia excessive divergence via its slow inner momentum, leading to superior stability even with partial, strided communication.
and streaming Local Adam. While relies an outer optimizer to reconcile divergent workers, MT-DAO prevents
Experimental Setup. We adopt the configuration from Douillard et al. (2025), using the strided fragment communication pattern. We set the synchronization periods to K2 K₁ K, = 128 (closest power of two to the recommended 100) for reami ng MT-DAO (N- = 1). We use 2 layers per fragment for the 16M model and layers per fragment for the 125M/360M models. We utilize a blending coefficient
0.5 and communication overlap of = 1 as recommended. For CtreamingDiiLe we use the recommended server learning rate of 0.4 with server momentum 0.9. For Streaming MT-DAO, MT-DAO hyperparameters tuned in Appendix H.2 with the batch size and clipping threshold as for all our other experiments with the same model scale. When using the non-streaming variant of MT-DAO we use the standard K = 32 unified sync frequency. For St st ng Local 1. Adam we use W1 1.0 (no quasi hyperbolic term) and 31 0.9, 32 0.999, n 0.001 as tuned for the standard Adam in Appendix H.2.
use the W1 0.95, B1 0.999, 32 0.999. 0.002
Results. Figure 16 shows that St starmmin ng MT-DAO closely mirrors the performance of MT-DAC, with a small gap, across model scales, starting with a 1.3% gap for the small model which shrinks to 1% for the 125M and 360M models. This is in-line with the findings of Douillard et al. (2025) which indicate very similar performance between the streaming and non-streaming variants of their method. St reaming DAO and MI-DAO consistently outperform StreamingD wng.iigCCCCCo. LoCo and streaming ng Local Adam. For example, at the 360M scale, Straaming MT-DAO achieves 8.35 perplexity compared to 8.69 for Streami LoCoo 4% improvement under matched token budgets.
Beyond the effects discussed in Section 5.4, the lower syne frequencies of the streaming regime (K >> 100) also play role in the performance gap. With such infrequent synes, standard fast momentum decays immediately at the start of a round and then optimizes solely for the local trajectory for extended duration, decoupling the local optimization from the global trajectory. St Stremmi inn Loco attempts to correct this via the outer Nesterov optimizer, but this correction is infrequent for large K. In contrast, the slow momentum component of MT-DAO persists across the long streaming intervals, ensuring the local update direction remains anchored to the long-term global trajectory. While our method introduces additional communication requirement for synchronizing
41

---

Published as a conference paper at ICLR 2026
30.0
MT-DAO-NN- Clip (125M) (10.09)
30.0
MT-DAO-NO-CHL (125M) (10.09)
MT-DAO (125M) (9.82)
MT-DAO (125M) (9.82)
27.5
27.5
MTT-DAO-N-- Clip (360M) (8.27)
MT-DAO--No -a-Clppppppp (360M) (8.27)
25.0
Validk.kon Perpleats
MT-DAO (360M) (8.26)
Perplenity
MT-DAO (360M) (8.20)
02.5
78.5
17.5
17.5
10.0
15.0
100
100-
7.5
26
52
00
56
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Wall Clock Time (Hous))
Tolous (Billions)
Figure 17: Sensitivity to Gradient Clipping. Validation perplexity vs. Time (left) and Tokens (right) for 125M and 360M models trained with standard clipping (p = 0.5 for 125M and p = 0.25 for the 360M, dotted lines) versus clipping (p oc, solid lines). While the 16M model diverged without clipping (not shown), the 125M and 360M models converged successfully with minimal performance degradation, demonstrating that MT-DAO's heavy reliance the slow momentum stabilizes updates at scale, we use w1 0.95.
optimizer states, this overhead is effectively mitigated by the streaming architecture. Due to the conmunication-computttion overlap (r) and update interpolation, we observe minimal impact wall-clock training time with our hardware configuration. Furthermore, any potential overhead can be readily offset by increasing the overlap factor т, allowing computation proceed uninterrupted. Although we utilize the default 1, which proves sufficient for these model scales, Douillard et al. (2025) demonstrates that values up
10 remain effective in practice.
Synergy with Streaming: MT-DAO is fully compatible with streaming communication. Its slow momentum compensates for the extended staleness of model fragments in the streaming regime, offering a substantial perplexity improvement over St ream ngD LoCo without additional overhead.
H.5 ABLATION ON THE CLIPPING THREEHIOLD
In standard LLM pre-training with Adam, gradient clipping is considered critical heuristic to prevent training divergence caused by "spiky" gradients numerical instabilities where coordinates of the second moment term / approach zero while gradients are large.
Mechanism of Stability. In Figure 17, compare MT-DAO trained with standard clipping against a version with clipping disabled (p = oo). We observe that for 125M and 360M models, removing clipping results in negligible performance degradation. This stability is intrinsic the MD-DAO design:
1. We only transition from the base optimizer MT-DAO after the warmup phase is complete. By this stage, the gradients have typically stabilized compared to the initial training steps.
2. MT-DAO assigns a large convex coefficient to the slow momentum. This naturally dampens outliers without explicit clipping.
Scale Dependency. We observe that the robustness to unclipped gradients scale-dependent. In our experiments, the 16M parameter model diverged immediately without clipping. However, the 125M and 360M models trained stably.
It is important to clarify that the necessity of clipping is primarily a property of the inner optimizer (Adam) rather than MT-DAO itself. Adam is susceptible to divergence when the preconditioner is ill-conditioned when dealing with rare tokens in heavy-tailed distributions.
42

---

Published as a conference paper at ICLR 2026
Clipping Robustness: MT-DAO naturally dampens gradient spikes via its heavy weight on the slow momentum, removing clipping entirely safe larger model scales (125M+) where gradient norms are naturally lower. For robust training across all scales, we recommend retaining standard clipping.
H.6 ABLATION WITH K
Our primary design objective for MT-DAO is to reduce the performance gap of communication efficient methods relative fully synchronous DDP. In our main experiments (K 32), we demonstrate that MT-DAO can match fully synchronous baselines by using slow momentum to bridge the gap between synchronization steps. In this section, we investigate the limit case of K 1 to isolate the behavior of the algorithm when this specific challenge is removed. As shown in Fig. 18, while DAO performs similarly to DDP, it trails slightly for both and N at the 16M and 125M scales.
60-
60-
MT-DAO (16M) (25.97)
MT-DAO (16M) (25.96)
34-
QHAdanw (16M) DDP (25.56)
54-
AdEMAMix (16M) (25.54)
MT-DAO (125M) (1119)
MT DAO (125M) (11.15
46
Valitlaton Puplects
QI1Admmw (125M)-DDP (10.91)
Valilation Purplectt
AdEMAMix (125M)-DDP (10.82
42-
36-
30
30
24-
18
18-
12-
0000
(000
3000
10020
12000
1.000
16000
18900
20000
2000
30.0
7500
10000
19000
15.00
17000
200000
Steps
Steps
(a) N Perplexity
(b) N Perplexity
-3
4.
(16M) (3
MT-DAO (GGM) [5.06-11
3,5
QUAAi URD (2 (33)
-
QliAAci UURM)-UUP (1 550
(1253) (3.55-04)
MT-DAO (1255) (6.05-044
Secood Mamnetans Li Noan
3.0
4
QlAAcam (125X))-DPP (3.21000)
4.2
-
QUiAcmm (1200))-DDP (2.3600)
3.6-
2.5
Mauweston
3.6-
2,0
1.5
1.23
0.5
0.6
00.0
10000
14000
10000
18000
200000
3000
03.0
8000
10000
120010
1.0000
10000
90000
Steps
Steps
(c) N Second Momentum Norm
(d) N 2 Second Momentum Norm
Figure 18: Comparison at the Synchronous Limit (K = 1). Top: Validation perplexity for MT-DA-
(K 1) versus DDP baselines for N (left) and N 2 (right) variants across 16M and 125M scales. Bottom: The L2 norm of the second momentum state v2 for the corresponding runs. While MT-DAO performs similarly to DDP, trails slightly. The bottom plot shows that the local estimation of the second moment in MT-DAO leads to larger norm than the global DDP estimate over time, resulting in slightly lower effective learning rate.
We emphasize that would not expect MT-DAO to generally outperform DDP at K for two key reasons. First, the noise inherent to Local SGD methods (K > 1) can act as an implicit regularizer (Lin et al., 2020), which may account for the shrinking disappearance of the performance
43

---

Published as a conference paper at ICLR 2026
gap to DDP in the main work, depending the setting. This benefit is strictly lost at K 1. Second, while for Local SGD methods with a simple first-order momentum, = 1 is mathematically equivalent to standard DDP, for state-of-the-aun adaptive optimizers this equivalence breaks down due to the non-linearity of the second moment update. We must distinguish between the local gradient estimate 9;" computed
per-worker batch of size B, and the global gradient estimate Gr - M 2m=19h computed on the effective global batch of size x B. DDP computes the
second moment using the lower-variance global estimate, i.c., (gt)² 2J Σ Em=1 It In contrast, MT-DAO computes the second moment locally using the noisier Di" (derived from batch size B) and subsequently averages these states, effectively computing 1 >m ( )o.²)) By Jensen's 's inequality
applied to the convex square function (element-wise), M ( 2 Em= 9 Since the second moment estimate is a moving average of these squared terms, the MT-DAO estimate is guaranteed to be larger than that of DDP, resulting in larger denominator in the adaptive update and implicitly reducing the average per-coordinate effective learning rate.
Regime of Applicability: MT DDO is specialized for infrequent communication (K > 1). the synchronous limit (K 1). it performs comparably to DDP but trails slightly. This stems from the nonlinearity of adaptive optimizers: by Jensen's inequality, MT-DAC's local estimation yields larger second momentum than DDP's global estimation, implicitly reducing the effective learning rate as demonstrated empirically. Consequently, we recommend DDF as the default choice when fully synchronous training feasible.
H.7 EFFICACY WITH MUON
To demonstrate the universality of our approach, we apply MT-DAO to Muon (Jordan et al., 2024), a recent optimizer that utilizes Newton-Schie iterations for update orthogonalization. The Newton- Schulz preconditioning is typically applied to the momentum term. This means our multi-timescale considerations regarding momentum variance and drift apply directly to the underlying state before preconditioning occurs. While we leave the full theoretical analysis of preconditioned local updates for future work, our framework structurally compatible with these methods. We now demonstrate that MT DDO is empirically effective when using Muon as an inner optimizer.
Experimental Setup. We use the default weight decay of 0.1 and set Nesterov to true, using the default PyTorch implementation of Muon with the match. rms norm learning rate adjustment recommended by Liu et al. (2025). Following standard practice for this optimizer, we employ a split optimization strategy: AdamW trains the embeddings and layer norms, while Muon trains all 2D matrices (Jordan al., 2024). We independently tune the base Adamw learning rate, finding the optimum to be 1c-3, and the Muon learning rate, finding the optimum be 2c-3. We then jointly tune the multiplier we sweep over for the quasi-hyperbolic variants in Fig. 24. We use 3=0.9 for base Muon and as in the rest of our work for the base optimizer and 31, 32 = 0.999, 0.999 for MT-DAO and OHMuon. We use the clipping threshold appropriate for Adam (Table 2).
0.999 for MT-DAO and OHMuon. Adam parameters are fixed at $1, B2 0.9, 0.999
Results. Our results shown in Figure 19 indicate that MT-DAO significantly outperforms base Muon and matches exceeds QHMuon at all model scales. Beyond proving that MT-DAO is effective with Muon, our results are also the first, to the best of our knowledge, to indicate that using an independent weight for the gradient is beneficial for Muon. While the base Muon implementation includes a Nestero rov term, this is equivalent to a quasi-hyyerbollic formulation where the weight of the gradient is directly tied to the momentum Our hyperparameter sweep shown in Fig. 24 indicates that this coupling is likely to be suboptimal when / is very high, since higher weight should be assigned to the gradient to compensate for the low reactivity of the momentum.
Efficacy under Newton-Schulz: Our findings confirm that the same principles that underlie MT-DAO apply effectively within Newton-Schule preconditioning, enabling MT-DAO to match the performance of the base Muon optimizer and match or exceed QHMuon. We also generally recommend QHMuon with a high 3 over standard Muon.
44

---

Published as a conference paper at ICLR 2026
ME-DAO (N--1 (5))2) QJEMioon (21.3)
ME-DA (N--1) (1006)
ME-AA (N..)) (9.00)
QJMMoo DDP (10
QUiMoro TIDP 19 10)
Figure 19: MT-DAO with Muon. Validation perplexity vs. Tokens for 16M (left), 125M (middle), and 360M (right) models. MT-DAO significantly outperforms the base Muon-DDP baseline and matches or exceeds QHMuon-DDe DDP at model scales. This demonstrates that the benefits multi-timescale optimization and independent gradient weighting transfer effectively to matrix-bass optimizers using Newton-Secee -chullz preconditioning.
wis: (0.9.) S's (0.998.)
w/s (0.00.) (2's (0.989))
1.00-03 (p16-31 82, saap-Budl
y 2.00x088 (pp1-31.7) <legrEaad))
Perplexity
45.0
Language
37.5
35.0
Oytinnan
0./012
0.001
0.005
0/0SS
0.00
0,022
0.014
0.016
01000
0000
0100
0.00
01000
0012
0.04
0,066
77
Figure 20: Hyperparameter Tuning Surface: QH-AdamW =N 1). Validation perplexity 16M models. Performance improves as the weight w shifts heavily towards the slow momentum (3 0.999), confirming that the fast component (gradient) primarily reactive correction term. We select the optimum (marked in yellow) for scaling experiments.
45

---

Published as a conference paper at ICLR 2026
(DUE) FURG) OUL
1821 EUR) pIA2
ZONEC LISHOSA MAjOrinN
418
315
214
(2.00 (38
(02 (96,
PONES SESNN SOSAENC
20000 Suparael
Pespledit
Langgnagge
</o 20600 241-1220 inp-tall
(05
0.5-
274
0202
5.00
Figure 21: Hyperparameter Tuning Surface: AdEMAMix (N = 2). Validation perplexity on 16M models. Performance improves the weight shifts heavily towards the slow momentum (82 = 0.95), confirming that the fast momentum (3 = 0.05) acts primarily as a reactive correction term. We select the optimum (marked in yellow) for scaling experiments.
46

---

Published as a conference paper at ICLR 2026
011
(2.0 EER0
1I2A (JD 12.E
JONEC Sppinit A
271
312
211
302
O7 1990
0000 (OE9)
10:000 peaat supatall
Peppledity
015
618-
Langnagge
075
235
200EC Liginst Aayenau
215
0202
6312
0508 3.000
Figure 22: Tuning N QHAdEMAMix. Optimal performance is found with a heavy bias towards the slowest momentum (w2 = 0.9), confirming that long-horizon signal is the primary driver of performance in distributed settings. We select the optimum (marked in yellow) for scaling experiments.
(001 2% (B16)
(B16 de (196)
N. (E (8
Langunge Perplocity
TMMULT
Figure 23: Tuning QHMuon. Using our pre-defined grid for quasi- huusi-hyperbooic..o w, we find that utilizing 0.95 provides the best performance for the QHMuon optimizer.
47

---

Published as a conference paper at ICLR 2026
OUE, SEM-C Ecpeppaaee
CODO)
271
2034
235
025 013 (200)00)
JUG, 20000 potadt separall
183. www.doapaaaall
0065)
Peepledit
818
Langgnage
275
L.T. DG 1930
L0000 bel-11 inp-tall
215
208
0202
5.000
Figure 24: Tuning N AggMo. Optimal performance is found with a heavy bias towards the slowest momentum (wg 0.8), confirming that long-horizon signal is the primary driver of performance distributed settings. We select the optimum (marked in yellow) for scaling experiments.
48

---

Published as a conference paper at ICLR 2026
I LLM USAGE DECLARATION
As declared in the submission form, LLMS were used in this work in order to aid or polish writing and for retrieval and discovery of related work. We used GPT-5 and Gemini 2 PRO primarily to abbreviate or rephrase text or to evaluate the clarity of our writing and provide guidance on areas of improvement. We also used the deep research feature present in both models order discover, but not describe or interpret, additional papers for our extended literature review in Appendix G. Finally, we used both models generate plotting code and as general code assistants.
REPRODUCIBILITY
We provide the complete source code for our mt _ao framework, accompanied by detailed setup instructions, to ensure the reproducibility of our results.
System & Dependencies:
The required environment, including specific versions of Ubuntu, CUDA, and Python, can be installed using the provided scripts. These scripts handle all necessary dependencies.
sh and install env. sh
Data: We provide scripts download and prepare the datasets. The distribution of data across clients for both IID and non-IID settings is managed through declarative YAML configuration files found in mt dao/ conf//oo////////////o dataset/ htt//straaatt streams/.
Execution & Hyperparameters: Example scripts available for launching federated, central- ized, and evaluation runs (e.g., reproduce paper's experiments, users can utilize the provided base launcher scripts and set the specific hyperparameters detailed in the paper. A concrete example this process is available in scripts/ https/icclcmmmmmee
sh, sen__25m_exame
sh). To fully
49


Reviews:
[{"review_id": "kRCqDNo0yw", "reviewer": "Reviewer_uewt", "summary": "This paper presents a unified perspective on momentum-based distributed optimizers. A novel multiple time-scale distributed optimizer (MT-DAO) with convergence guarantees is proposed that maintains momentum at different time scales. Results on language modelling task with 3 different sizes (16M, 125M and 720M) demonstrate that MT-DAO outperforms prior approaches and also match DDP.", "questions": "- The MT-DAO approach is proposed and motivated with multiple fast/slow-moving momenta but experiments show results with N=1, why? \n- Following up on above, the proposed framework looks like a generalization of AdEMAMix optimizer -- how does it directly compare for LLM training in DDP? Is it feasible to match the performance of AdEMAMix optimizer (and other recent matrix-based optimizers such as Muon) with much less communication steps?\n- How does the proposed approach compare with DiLoCo in terms of performance and memory requirement? Moreover, DiLoCo supports complex adaptive rules in the inner optimization and not just limited to momentum, so was wondering if MT-DAO supports that and how it affects theoretical results.\n- How critical is gradient clipping for this approach to work well or can it work without clipped gradients as well? Does DDP baseline also use the same clipping values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clean framework to develop momentum-based distributed optimizers, nice analytical studies.\n- Strong set of language modelling experiments with 3 sizes upto 720M using modern optimization tools including completeP for LR transfer, WSD scheduler, standard data mixtures, etc - great job!", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYqybZRhwc", "reviewer": "Reviewer_BGje", "summary": "The authors hypothesize that the gap between the fully synchronous DDP and sparse comm strategies is due to a time scale mismatch; that is, the optimizer's fast moving momentum decays too quickly between synchronization steps, causing replicas to diverge. To remedy this, they propose a bandwidth-efficient distributed data parallel training method for LLMs, which incorporates multiple fast and slow moving momentums. They show that using this approach they can close the gap between DDP and sparse comm methods, and can preserve mutual information better between model replicas.", "questions": "1. Why are the experiments only conducted on Adopt? In my opinion, this significantly harms your claims.\n2. Infrequent communications mean that each replica is effectively seeing a smaller batch size (compared to the global batch size across replicas). In such cases, it has been previously shown that increasing the betas work robustly [1]. How does your work differ from the insights given in such works.\n3. Can you compare your method with more recent methods such as streaming DiLoCo? I think such comparisons are crucial to validate the efficacy of the proposed work.\n4. When tuning the (beta, w), did you conduct a grid search? How well do these parameters scale across model size? \n5. What are the hyper parameters used in Fig. 5?\n\n[1] - Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important question. The convergence gap between infrequent communication strategies and synchronous DDP settings is a glaring problem. I like the authors approach to the problem which is formulated around the fact that this is due to a time scale mismatch between synchronization intervals.", "weaknesses": "The narrative of the paper is a bit unclear to me. The method is proposed as a generic plugin across many optimizers and the authors even provide algorithms for SGDM and Adam. However, the experiments are only demonstrated on ADOPT. This also harms the claim that this method is useful for optimizers that use multiple first order momentums, however, as far as I am aware, ADOPT only use a single first order moment? I invite the authors to correct me if I am missing something.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JURFpvK7vo", "reviewer": "Reviewer_CDpB", "summary": "This paper proposes MT-DAO, a family of optimizers to stabilize and accelerate training in a communication-limited environment. The authors observed that when the synchronization interval K is large, a relatively large $\\beta$ leads to training instability. They introduced the Quasi-hyperbolic method into training, provided convergence proofs, and experimentally validated its effectiveness.", "questions": "1. Could the choice of batch size impact the results? The paper chooses a fixed global batch size settings from [1], which is not in a Local SGD setting. According to [2], DiLoCo seems to tolerate a larger batch size than DDP for a better wall-clock time performance, which I think is pretty reasonable because it does local updates.\n2. The paper seems not to discuss the choice of the weight parameter w, does it have a clear pattern of range or recommended value?\n\n[1] Benchmarking optimizers for large language model pretraining  \n[2] Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper has a clear structure with well-defined motivation\n- Convergence proofs are provided, and experiments address the questions of interest", "weaknesses": "Most of my concerns are already mentioned in the Limitations of the paper itself.\n\n- The scalability of MT-DAO. Other than the largest model is at the size of 720M, the experiments are conducted under four workers with 1 GPU per each.\n- A possibly weak baseline with ADOPT instead of Adam. Similarly seen in Figure 2, Local SGD converges faster than MT-DAO-SGDM in this toy example.\n- There is a suspicion of overclaiming because the paper only conducted experiments with N=1. When N=1, it degenerates into each worker storing momentum similar to AdEMAMix and combining updates like QHM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oo0u7bpmz6", "reviewer": "Reviewer_KKBR", "summary": "This paper introduces MT-DAO, a novel family of optimizers designed to bridge the performance gap between DDP and adaptive optimizers using low-frequency communication strategies (e.g., Local SGD). The authors identify \"timescale mismatch\" as the root cause of this gap and propose employing multiple slow and fast-moving first momenta to capture update dynamics across different time scales. In large-scale language model pretraining experiments, MT-DAO successfully eliminates the performance disparity with synchronous DDP and outperforms existing low-communication baselines in perplexity. Combining rigorous theoretical analysis, solid empirical validation, and practical significance, this work constitutes a major contribution to distributed deep learning optimization.", "questions": "- The novelty of this paper lies in the introduction of a multi-momentum strategy. However, all experimental results were obtained under MT-DAO with N=1, meaning only a single momentum was used. I wonder whether increasing the value of N would lead to further improvements in model performance.\n- If we set K=1, effectively turning MT-DAO into a DDP optimizer, will its performance surpass the DDP baseline? And does it hold any advantages over other DDP optimizers that adopt multi-momentum strategies, e.g. adEMAMix?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novelty: Identifies the performance gap between low-frequency communication strategies and DDP as stemming from temporal scale mismatch, and innovatively adapts multi-momentum strategies to the distributed setting.  \n- Empirical Rigor: Comprehensive experiments demonstrate MT-DAO’s reliability.\n- Clarity and Readability: The paper is well-structured, clearly articulating motivation, methodology, and results.", "weaknesses": "Please refer to Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)