DES-LOC: Desynced Low Communication Adaptive Optimizers for Foundation Models
Authors: Alex Iacob
Lorenzo Sani
Mher Safaryan
Paris Giampouras
Samuel Horváth
Andrej Jovanovic
Meghdad Kurmanji
Preslav Aleksandrov
William F. Shen
Xinchi Qiu
Nicholas D. Lane
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed Training
Foundation Models
Large Language Models
Optimizers
Communication Efficiency
Federated Learning
Distributed Systems
Optimization Theory
Scaling
Robustness
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose provably convergent local adaptive optimizers with decoupled sync frequencies, empirically reducing communication 170x vs. DDP and 2x v s. Local Adam (prior SOTA), reducing time by 1.3x-2.1x , validated up-to billion-scale models.

Abstract: Scaling foundation model training with Distributed Data Parallel~(DDP) methods is bandwidth-limited.
Existing infrequent communication methods like Local SGD were designed to synchronize model parameters only and cannot be trivially applied to adaptive optimizers due to additional optimizer states.
Heuristic approaches that keep states local or reset them lack guarantees and can be unstable in compute‑efficient batch regimes; conversely, Local Adam synchronizes all states uniformly and is provably convergent but triples communication costs.
We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Our theoretical analysis shows that while parameter synchronization dominates the asymptotic rate in-expectation, high-probability convergence guarantees require at least infrequent synchronization of the second momentum. Furthermore, we prove that more frequent momentum sync permits larger stable step sizes. Experiments on language models of up to 1.7B show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local Adam, enabling 1.3x–2.1x wall‑clock speedups over DDP for 1-13B models on 100Gb/s links. Furthermore, unlike previous heuristic methods, DES-LOC is robust to worker failures offering a scalable, efficient, and fault-tolerant solution for foundation model training.

Published as a conference paper at ICLR 2026
DES-LOC: DESYNCED Low COMMUNICATION ADAP- TIVE OPTIMIZERS FOR FOUNDATION MODELS
Alex Iacob f.1,2
Lorenzo Sani!
Mher Safaryan?
Paris Giampouras
Samuel "
Andrej Jovanovié
Meghdad Kurmanji
Preslav Aleksandrov16
William F, Shen
Xinchi Qiu
Nicholas D. Lane
ABSTRACT
Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize model parameters only and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Heuristic ap- proaches that keep states local reset them lack guarantees and can be unstable in compute-efficient batch regimes; conversely, Local Acam synchronizes all states uniformly and is provably convergent but triples communication costs. We propose Desynced Low Communication Adaptive Optimizers (ES-LOC), a family of opti- mizers assigning independent synchronization periods parameters and momenta, enabling lower communication costs while preserving convergence. Our theoretical analysis shows that while parameter synchronization dominates the asymptotic rate in n-expectation, high-probability convergence guarantees require at least infrequent synchronization of the second momentum. Furthermore, we prove that more fre- quent momentum sync permits larger stable step sizes. Experiments on language models of up to 1.7B show that DES-LOC can communicate 170x less than DDP and 2 less than the previous state-of-the-art Local Adam, enabling 1.3-2.1 x wall-clock speedups over DDP for 1-13B models on 100Gb/s links. Furthermore, unlike previous heuristic methods, DES-LOC is robust to worker failures offering a scalable, efficient, and fault-tolerant solution for foundation model training.
INTRODUCTION
Training foundation models requires distributing optimization across workers for improved memory and compute. However, frequent gradient communication standard Distributed Data Parallelism (DDP) (Li et al., 2020a) increases networking costs and limits scalability. Farly works like Local SGD (Stich, 2019) and FedAvg (McMahan et al., 2017) reduced this overhead by synchronizing infrequently, averaging parameters only after K >> local steps, instead of gradients at every step. However, modern foundation model training, e.g., Large Language Models (Dubey et al., 2024), uses adaptive optimizers (Kingma & Ba, 2015) which require additional momenta.
Some extensions of Local SGD to adaptive optimizers (Sani et al., 2025; Douillard et al., 2023) only average model parameters, which poses challenges. First, they lack convergence guarantees. Second, keeping momenta local (Douillard et al., 2023) accumulates noisy small-batch gradients and provides means initialize workers. This makes them unsuitable for failure-prone environments. Third, re-initializing momenta (Sani et al., 2024; 2025) destabilizes training.
Local Adam (Cheng & Glasgow, 2025) addresses these challenges, proving periodic synchro- nization can converge faster than standard Adam with DDP, and remain robust to the addition of new workers. However, it requires synchronizing momenta alongside model parameters, tripling communication costs compared to Local SGD. Hence, we aim to answer the following question:
faa 30 cam. ac.uk; 'Equal contributions; University of Cambridge; Flower Labs; Institute of Science and Technology Austria; "University of Warwick; "Mohamed bin Zayed University of AI, 'IMEC

---

Published as a conference paper at ICLR 2026
Can independently syncing parameters and momenta improve communication efficiency for adaptive optimizers while maintaining convergence and robustmess?
As result of our inquiry. propose optimizer family, Desyneed Low Communication Adaptive Optimizers (DES-LOC), which sets independent synchronization frequencies for parameters and momenta. This reduces communication overhead by synchronizing momenta less frequently.
Contributions
1. Provable convergence. We prove convergence (see Section 3) for DES-IOC under: non-convex objectives when using SGD with momentum (SGDM), and weakly convex objectives when us- ing Adam. Our theory indicates a higher momentum syne frequency enables larger step sizes. Furthermore, high- nigh-probabiility bounds demand momenta be synced with finite period for (2 1.0.
2. Communication reduction. We empirically show that parameters require more frequent syne than momenta, and that less frequent momentum syne reduces communication costs (2x Loca Adam, 170x vs DDP), leading to 1.3 - 2.1 2.1x x reductions in training time over DDP on our hardware.
3. Scalability to large models. We validate DES-LOC at billion-scale language model training, demonstrating competitive ICL performance against both Local Adam and DDP.
4. Hardware robustness. Unlike previous heuristic methods, DES-LOC avoids persistent local states, enabling to seamlessly integrate new workers to support environments prone to system failures.
DESYNCED Low COMMUNICATION ADAPTIVE OPTIMIZERS (DES- LOC)
We start by characterizing the relation between the rate of change of optimizer states and Local Adam, and how these can be leveraged to lower the communication cost. Consider the Adam update: U4 B1 1 + (1 B1)9t and e f201-1 + (1 82)gi 9t-
For Local Adam, convergence is contingent on B2 satisfying 32 o 3/22 (Cheng Glasgow, 2025) where K is the number of local steps and R the total communication rounds. Large K R, implies 32 - and conversely larger 32 permits higher K or R.
A useful summary measure is the number of steps until a state's 's weight decays to fraction v/ (3) In In Following Pagliardini et al. (2025), we use the half-life To.5 as our primary measure, omitting 8 when clear. For typical values of 3, have To.5 (0.95) 13.5 (Allal et al., 2025), To.s R 692.8 (Kingma & Ba, 2015), and To.s (0.990.9 R 6931 (Taniguchi et al., 2024). Intuitively, larger half-lives imply synchronizing gradients is less sensitive to new gradients; choosing B 0 ignores all previous momenta, whereas 8 + 1 progressively attenuates signal from the current gradient.
longer horizons the optimizer
While the half-life captures the horizon for which an optimizer state remains relevant to model updates, it provides no information on its absolute rate of change. With coordinate-wi clipping. each gradient component satisfies )l S Unrolling Adam's 's recursions over K local steps gives K-1 of 9t+K-1-k and its second moment analogue. Since !9c.al S p and (gg, gc)1 pp² the maximal loo drift of each moment is (see Section F);
the follow relation: 4ч-K 38 4 + (1 8,) k==
||uck - - welloc 2p 10-1--0) 3K)
| - -ellsc 2p2 (1 3
(2)
From the above, large 33 values and small clip bounds p. a common practice in foundation model training (Brown et al., 2020; Scao et al., 2022), limit the absolute changes in optimizer states. We can construct similar reasoning for other optimizers (Sutskever et al., 2013; Taniguchi et al., 2024), and norm-based clipping (Pascanu et al., 2013; Brown et al., 2020). From the above, the half-life of an optimizer state should inform its synchronization frequency. For example, if To.s (0.95) x 13.5 and K 256, synchronization only affects few initial local steps. Over the course of the local training, the impact of the synchronised optimizer state shall decay to given Equations and Conversely, if K 16, synchronization approximately matches the half-life, strongly influencing local updates.
2

---

Published as a conference paper at ICLR 2026
2.1 DES-LOC ALGORITHM
Algorithm DES-LOC
Require:
Model tensors, update functions, hyper-parameters
1: z0 E R., {6-1)==1 E (R")* initial parameter vector, the initial N optimizer states
(UPDATE' LX 13=1 (Rc x R +
R updates optimizer state j from its previous state and gradient.
OPT IRd R x 1R, (R - R update params from all worker models.
4: SERVEROPT R - R update params using abstract outer optimizer
5: E H (R+)*
clipping radius for clip(:, p), learning-rate for each time-step total optimization steps and number workers
T.M N+
7: K, E N4, {K E (N+)* communication periods (steps)
Ensure: 2T; - LN
for each worker m: x +
J.m
local init training loop
9: for = O....,, T- do
10:
for all workers
O,
M in parallel do
11:
9i V (=(=T
stochastic grad per por-doondi clipping
9t <- (l :P)
13:
for j 1 to N do
14:
if mod K2 0 then
syne s
15:
st g.m + UPDATE (m [s/_1) Ø())
16:
else
17:
s!-" UPDATE (//- 9("))
18:
if mod K2 0 then
sync
#/41 <- DEP(SERVEROPT( "6" Ths {s. }==)))))))))))
20:
else
21:
4++ OPT(2 96 The (3)""))))))
Motivated by the above insights, we formalize Desynced Low Communication Adaptive Optimizers as family of optimizers offering the same convergence and robustness as Local Adam but with significantly lower communication costs. Our approach applies generically to adaptive optimizers parameterized by OPT (IRd, Ind R ( + ıd with N optimizer states {s'-
Rd.
each updated by UPDATE (Rd ) Rd. Coordinate- wise clipping is defined as [clip(X, X, p)], sgn(X.) min p}. To ensure that our method is provably convergent, SERVEROPT is that of FedAvg (McMahan et al., 2017). However, our algorithm directly extends to the larger FedOpt (Reddi et al., 2021) framework, which we discuss in Section C.I.
We focus our analysis on SGDM and Adam, As shown in Algorithm 1, DES-LOC synchronizes parameters x R2d and optimizer states {sd } at state-specific intervals K,, (K N Setting N = 2, st = ut, o? = et, and using update rules UPDATE .UDDTTE based on the Adam update rules above yields DES-LOC-Adan (see Algorithm 2).
Toy Example To highlight DES Des-LOC's practical benefit, Fig. illustrates a scenario where DES-LOO and Local Adam converge under noisy gradients, while prior heuristic methods (Douillard al., 2023; Sani et al., 2025; Iacob et al., 2025; Sani al., 2024) fail.
CONVERGENCE GUARANTEES FOR DES- LOC
This section provides theoretical support for the proposed DES-LOC approach. We focus on version of the Adam optimizer that uses only single momentum state. Extensions to the full Adam optimizer with both momenta are available in Section D.1 with gigh-probbbility bounds shown in Section E.
Formally, we consider the following optimization problem:
min f(x) = M
with fx) EgnD D.. [F ( (2; E)]-
In this setup, all M machines collaboratively minimize the objective in (3). Generally, each machine m has access to only dataset D Dm.. which can differ from device to device. This recovers the homogeneous distribution case when all machines have the same dataset Dj D2 DM
assume
3

---

Published as a conference paper at ICLR 2026
10-1
10 -4
10-7
-2
-2
10
10-13
DES-LAC
10
FAVO OPT
80
Communication Rounds
Figure 1: We present: (left) the distance to the optimum and (right) a 2-D contour of a toy problem
where DES-LOC (K₂ 192, K, 192, K, 692) and Local Adam (K Kn)) both converge to the optimum (overlapping). Methods keeping optimizer states local ( fail to converge. Periodically resetting states similarly stalls due to repeated oscillations. We optimize the non-convex function f(1 x2) = (1 x1 + 100(x² x²)² with M = 256 workers and IID Gaussian noise (a = 1.5). show an example of such a toy problem on Non - IID data in Fig. 8.
and minimize the same loss (1) fx() fm) f(x). We assume each machine m computes mini-batch stochastic gradients corresponding to randomly selected samples ~ Dm from dataset Dn. We further use the following technical assumptions on the problem structure.
=
Assumption 1 (Lower bound and smoothness). The overall loss function f Rd - is lower bounded by some f E E and all local loss functions f are L-smooth:
(m) - (mm(y)) S T|[x-1y] for any V E Rdd
Assumption (Unbiased noise with bounded stochastic variance). The stochastic gradient g" of local loss function fm computed by machine m is unbiased and the noise has bounded variance:
E g = f. (x), Bilgg"" v (z))1) S o², for any x E Rdd
Assumption 3 (Bounded heterogencity). For any E RR, the heterogeneity is bounded by
M DM [m G2 + (=))
All three assumptions are standard and widely used in the convergence analysis of optimization algorithms Yu et al. (2019); Karimireddy et al. (2020b); Wang et al. (2021); Yuan et al. (2022). Note that the bounded heterogeneity condition recovers the homogencous case when G2 = 0 and B 1. To facilitate the technical presentation of the analysis, we view model and optimizer state synchronizations through assigning probabilities to each averaging event. Particularly, instead of averaging model parameters every K2 steps (i.e., t mod K2 0), average with probability Pz K. which are statistically equivalent. In the following theorem, we provide convergence rate of SGDM optimizer under such probabilistic and decoupled synchronization:
Theorem 1. Let Assumptions and hold. Then, choosing the step size 17 min ) with
1 min (1- 3, 4L
del 4(1-p=) (1-0)
1o
6 (max(1,B22))
where
13
(4)
the average iterates 2t = Em [x(") of comeeee
converge with the following rate:
+ V < + (I(Z))
Laa 2nd
1+
(5)
We now discuss the convergence result and its implications. The obtained rate (5) is asymptotically optimal for this setup (Arjevani et al., 2023). Notably, the leading term O (+})) is unaffected by the number local steps. Interestingly, probabilities Pa, Pa, and the momentum parameter 3 appear in the higher-order term O(+), ), and thus have limited impact asymptotic convergence speed.
Regarding state synchronization, it is evident from (4) that model synchronization has greater impact convergence due to the dependence O ps
With vanishing Pa, the term becomes
4

---

Published as a conference paper at ICLR 2026
unbounded and breaks the rate. For optimizer states, it seems that momentum averaging can be turned off (pu = 0) without affecting the asymptotic behavior of the rate. Setting Pa = 1 and Pa. = recovers standard mini-batch SGDM (Liu et al., 2020). However, the 1 term also appears in the step- (1-P=)
size restriction (4). As Pu - 0. 1- 1-(1-pu) + 1-3" This imposes the most severe restriction on the væ 1/s is maximized. This theory shows that increasing the frequeney Pa of momentum averaging while not changing the asymptotic sate-allow, for a larger step size, potentially leading to faster convergence in practice. This theory justifies that momentum states can be synchronized less frequently than parameters and that more averaging improves convergence Adam in
learning rate o/o since ?/u 1
by supporting larger step sizes. Furthermore, our high probability analysis of Section E shows that the syne frequency of momenta must be finite for 3g < 1.0.
4 EXPERIMENTAL DESIGN
Our experimental setup addresses the following research questions:
RQ1 Do theoretical rates of change predict the empirical evolution of optimizer states?
RQ2 How does the synchronization frequency of a model/optimizer state impact performance?
RQ3
To what extent can DES-LOC cut communication w.r.t. Local 1 Adam in practical scenarios?
RQ4 How does DES-LOC scale with increasing model size and longer training horizons?
RQ5 How does DES-LOC perform when using Nesterov outer optimizer?
RQ6 Is DES-LOC compatible with non-Adam inner optimizers such Muon?
4.1 EXPERIMENTAL SETUP
Models and data. We train 115M-parameter GPT-style model (arch. in Table 3) with sequence
length 2048. We distinguish worker batch size B. from global -D B, (Sani al., 2025). A 2M token global batch is split across M = 4 workers sampling IID from SMo1LM2 (Allal et al., 2025): 70% F neweb-Edu (Penedo al., 2024), 10% Cosmopedia (Ben Allal al., 2024), 10% Python-Edu, 5% FineMath 4+, and 5% Infi-WebMath 4+. The 135M model trains for 6.4B tokens (2.4x compute-optime (Hoffmann et al., 2022)). For RQ4, we scale to 1.7B model for 40B tokens (2x compute-optimal (Sardana et al., 2024). We show Non-IID data results in Fig. 14.
Optimizers. We use Adam (Kingma & Ba, 2015) (results in Section B) and its variant ADOPT (Taniguchi et al., 2024), which modifies the update to guarantee convergence for any B2- For the 135M model, we grid-search (31, .3, 7) under DDP; the 1.7B model uses hyperparameters from Allal et al. (2025); Taniguchi et al. (2024). Learning rates use the WSD schedule (Hägele et al., 2024; Allal et al., 2025). We favor ADOPT (32 = 0.9999) in high-3 regimes where Adam is unstable. We also ablate the outer optimizer, comparing F FedAvg with a Nesterov optimizer (Reddi et al., 2021; Douillard et al., 2023; Charles et al., 2025) on a 700M model trained on 40B tokens.
Baselines. We compare DES-LOC with: (i) synchronous DDP; (ii) Local Adam/ADOPT; (iii) FAVG+OPT (persistent states (Sani et al., 2025; Douillard et al., 2023)); and (iv) FAVG-OPT (reset states (Sani al., 2024; Iacob et al., 2025)). Persistent-state FedAvg is DES- with Ku, K2 = co, an upper bound on comms efficiency. DDP is an upper bound on ML performance.
Metrics. We evaluate models by (i) perplexity and (ii) wor-worker asymptotic communication cost assuming a bandwidth-optimal
(Sergeev & Balso, 2018) algorithm scaling linearly with model size. For the 1.7B model, we report standard nn-context-leaninng (ICL) bench- marks (Brown et al., 2020). We use a zero-shot setting for ICL tasks unless stated otherwise following Allal et al. (2025) and report the best performing communication-effiiiaa. best-performing overall in bold. To fairly compare optimizer-state changes across decay rates, we measure their relative rates of change as || - sell2/1|
method in blue with the
[|ll//|>x]]] $x s11 For convergence plots, we report final-round means and standard deviations next to labels. We also provide wall-time clock results; we use 4 machines with one H100 for sub-1B models, and 4 machines with H100s each for larger scales. While the links between machines run at 100Gb/s, bandwidth to 60- 70 Gb/s. We report stepwise (see Section B.3.1) 1) and timewise convergence. We also provide analysis the wall-clock time vs bandwidth in Section G.I.
observed overheads limiting the practical
5

---

Published as a conference paper at ICLR 2026
(00 -alkk//
$0.01
[Pra wele/wil
3040
0100
0e-11
2007
4--01
-055
4040
2x 01
2-10
1--10
Communiication Round
Commuiieation
Figure 2: Relative rates of change for first (left) and second (right) momenta across rounds using standard Loca 1 ADOPT (K 64). For ADOPT (32 0.9999), increasing 31 2 0.99 greatly slows the first-momentum rate of change. The second momentum evolves x slower.
5 EVALUATION
Our results show optimizer states change at different rates (Section 5.1), forming a clear synchroniza- tion hierarchy (Section 5.2). DES-LOC reduces communication 2 vs. Local Adam (Section 5.3) while converging robustly with adding workers and scaling effectively to large models (Section 5.4).
5.1 HIGHER 3 OPTIMIZER STATES HAVE SLOWER EMPIRICAL RATES OF CHANGE (RQ1)
Figure shows that relative rates (of change for the two momenta in Local ADOPT/Adam scale with their decay rates under gradient clipping
1). Supported by our theoretical discussions momenta half-lives (Section 2), the second momentum evolves substantially slower than the first at high-32. For Local 1 Adam, the second momentum remains slower even when 33 B1, potentially because gradient variance (Kingma & Ba, 2015) evolves slower than the first momentum.
Takeaway: As discussed in Sections 2 and when 81 82, the second momentum evolves slower than the first, proportional to half-life ratio of the two (B2) In(
(8)) In(22)
5.2 PARAMETERS REQUIRE FREQUENT SYNC, MOMENTA SYNC PROPORTIONAL TO 3 (RQ2)
Figure 3 evaluates the effect of independently varying synchronization periods (K Ku, K.)) for parameters and optimizer states. We consider two baseline periods (K, = 16, 256). chosen based on the fastest state's s half-life (0.95) 13.5). Frequent parameter synchronization (K₂) is crucial for performance, while synchronizing momenta (Ka, K2) significantly impacts training only their half-lives align with the base frequency K₁. Otherwise, synchronization frequency primarily influences communication costs rather than model quality. Adam results can be seen in Section B.
Takeaway: Parameter synchronization frequency (Kz) strongly impacts performance, motivated by the leading term theoretical bounds (Section 3). Momentum synchronization periods matter empirically only when chosen near their half-lives, consistent with Sections 2 and 3.
5.3 DES-LOC BRINGS 2x COMMUNICATION REDUCTIONS OVER LOCAL ADAM (RQ3)
As shown in Figure 4, DES-LOC halves communication versus Local Adam (Cheng Glasgow, 2025) with matching perplexity by syncing momenta less frequently (K, 3K,, K, 6K2), exploiting the eecond-mommntungg speedup over DDP and 1.1 x over Local Adam at K2 = 16 on 4 H100s. At K 256, the speedup over DDP increases to 1.47x and both DES-LOC and Local Adam saturate throughput.
lower sensitivity to syne frequency (Fig. 3). This yields 1.37x
As shown in Figure 4, DES-LOS effectively initializes new workers and outperforms the straightfor- ward approach ad-hoe averaging from checkpoints- insufficiency detailed Section H.
Takeaway: DES-IOC achieves 2x communication reduction over Local Adam by leveraging
insights: optimizer-state sync matters less than parameter sync, and slower-changing states (high 82)
can sync less often. By eventually syncing all optimizer states, DES-IOC matches the robustness Local Adam with = max K,, K,) when adding new workers/respoddngg system failures.

---

Published as a conference paper at ICLR 2026
100
(7740055
Perplesity
Perpieiit
00.0
R. 2072 4250
30.0
23-
1500
2000
2500
3000
Sequential Stepe
(a) DES-LOC vary Ka fixed K. K. 256
(b) DES-LOC vary K2 fixed K2 K. 256
1010
-0001
60.0
Pheplesity
6:27)
Paplaat
3824 (7
00.0
Rul 1854
60.0
1938 (178-1))
Rv (122-44)
4001
20.
300
00
1200
2000
60
500
1000
Seqperse and Stegps
Sequrntad Stago
(c) DES- DOC vary Ku, fixed K2 K. = 16
(d) DES-LOC vary Ku. fixed K2 Kg 256
Figure 3: Model perplexity for DES-LOC LOC (ADOPT, 31 0.95, 0.9999), varying synchro- nization periods independently (others fixed at K2). Parameter synchronization (a) is critical, with degradation higher periods. Second-momentum syne (b) minimally affects performance due its large half-life. First-momentum sync improves perplexity when the freq matches its half-life; contrast (c) and (d). Parameters and second momentum behave similarly across frequencies (Section B)
(630 127) TT-13
31
5031
265 DCC ESAAEE THE2
1003
LOC OT T-ILS
Parphedty
3LLA T-12A
Perpledt
03.01
400
00
00
Time (Hours)
Time (Houms)
(a) DES DES-LOC-ADIAE LOC-ADOPT vs basclines K₂ 16 I1D
(b) DES-LLCC -L0P--OP vs basclines K₂ = 256 IID
000.0
322h 1L
0.8
Liaett Adle (9H600)
Perplesit
Groetenn Norm
PNND-Drr (1.30-000) 080
0110
0.0
PNPS-5F 11.30 63
AdBacc (SMMALL))))
FWSS Ar-by Ary OPS (URIARK
-0.0
3.0
Sequercial
Sequinttial
(c) Perplexity impact of doubling workers, K2 128.
(d) Gradient norms after doubling workers, K., 128.
Figure 4: DES-LCC (K,, K, = 3K, 6K₂) reduces comms by 2x over Local Adam while match- ing its performance and that of heuristic baselines at high (a) and low (b) frequencies (Section 4.1). We show robustness by doubling worker count at step 1536 (c,d), where DES-LOC and Local 1 Adam maintain stable perplexitynoomms outperforming heuristics and ad-hoc optimizer-state averaging.
7

---

Published as a conference paper at ICLR 2026
Eavilin Tudko
Activetinus (1-)
708
DC3 LOC (6-I2ANRR
EGP (UOEAAA))
Depledty
WWW
Local Koon (UUESUU
0100
86
FIA: OFT (14
200
002
000
01
2000
10000
12000
15000
Time (3oum))
Sequentaal Stops
Figure 5: DES-LOC matches Local 1 Adam perplexity (left) for billion-scale model training half the communication cost (K₂ 256, Ku = 3/g, K2 = 6K²). representing a 170x reduction over DDP. Both DES-LOC and Local Adam converge to competitive perplexity at this scale. FAVG+OPT I achieves good performance (Ieft) but suffers activation growth (right) and parameter- norm growth (Section B), potentially due noisy updates, raising concerns for extended training.
Table Our billion-scale model trained with DES-LOS matches surpasses the (ICL) performance of models trained with Local Adam and FAVG+OPT, approaching DDP performance. FAVG+OPT underperforms compared to its perplexity results from Fig. 5.a, indicating that the activation in- creases (Fig. 5.b) from the unstable training procedure may have damaged the model.
Are Challenge
Are Easy
PIQA
HellaSwag
Avg
DES-LOC
31.8
59.0
70.7
51.6
Local Adam
31.9
59.0
70.6
45.8
51.8
FAVG+OPT
30.1
58.0
70.0
44.8
50.7
DDP
33.8
62.5
71.1
47.8
53.8
5.4 DES-LOC Is SUITABLE FOR LARGE-SCALE TRAINING (RQ4)
Figure 5 shows that DES-LOC reliably scales to 1B models and very infrequent communica- tion (K₃ 256). Evaluating the billion-scale models on the TCL tasks (Table 1), DES-LOC competitive with all baselines while reducing communication versus Local Adam and DDP. The heuristic baseline (Sani ct al., 2025) suffers training instabilities (Fig. 5.b) potentially impacting downstream performance (Table 1) and underscoring the advantage of DES- -OC's training stability. We elaborate more on the settings in which we expect DES-LOS to improve stability in Section
DDP (Figure 5). As show by our benchmarks Table 2, these time savings scale with model size and comms frequency. At the 13B scale with K2 16, DES-LOC would save 13 days Local 1 Adam and 73 days over DDP. The advantage over DDP widens for K2 = 256, where communication-effiiii methods maximize throughput. Table shows equivalent results for throughput.
Our method's 's reduced communication costs result in 2.2x training speedup
DES-LOC enables efficient training of large-scale foundation models, especially long training horizons, with downstream ICL performance competitive with DDF. We recommend setting K₂ for sufficient throughput based on bandwidth, then setting K2, by as constant multiples (e.g. 3x, 6x) or based on half-life of their (see Section 1).
Takeaway:
Table 2: Wall-clock time (days) for 1B-13B models to reach 2x compute-optimal tokens at high (K 16) and low (K 256) frequencies with 2M token batch size. At high frequency (K 16). DES-LOC outperforms Local ADAM by over 13 days on the 13B model and is within 3% of FAVG+OPT. At low frequency (K- 256), cuts the 13B's training time > 93 days versus DDP.
Model
Model
K_
16
256
16
256
16
256
DDP (Baseline)
1.41 0.008
1:11 = 0.008
38.74 = 0.161
38.74
0.161
175.50 = 0.478
175.50
0.478
FAVC4OPT
0.80 0.007
0.63
0.006
28.52
0.085
24.01
0.088
100.21
0.544
82.46
0.5.13
uccal Adam.
0.96 0.006
0.64
0.006
31.46
0.080
24.18
0.087
116.10
0.484
83.34
0.500
DES-TOC (K,, K, KK,6KK)
0.81 0.006
0.63
0.006
28.80
0.084
24.06
0.088
102.03
0.537
82.68 1: 0.512
8

---

Published as a conference paper at ICLR 2026
2000
ESP 1-4TAL.22
Peepleity
10
Peeptedity
25.3
EB0 000 15.723 MC
100
1.0
33
00
100
3003
10030
13000
20000
20000
50000
35000
Time (Houn)
Sequantiaa Stepa
Figure 6: Ablation of the outer optimizer for DES-LOC 700M parameter model in medium- frequency communication setting (K₂ = 32), showing (left) convergence in terms of time and (right) in terms of steps. In this regime, a Nesterov outer optimizer provides a further improvement over averaging.
final perplexity is within 1% of the DDP baseline. Using
Figure 7: Training loss comparison between Local Muon (K 32) and DES- Muon (K₂- 32, K₂, 96) across model scales (16M, 125M, 360M). DES-LOC matches the Local Muon baseline across all scales while communicating more than 1.5x fewer bytes. See Section B.7 for full experimental details.
5.5 NESTEROV AS THE OUTER OPTIMIZER (RQ5)
We ablate the outer optimizer for DES-LOC on 700M parameter model, comparing averaging to a Nesterov optimizer with momentum of 0.9, outer learning rate of 1.0 tuned following Charles et al. (2025). The experiment ran on H100s and used a medium-synchronizztion regime (K₂ = 32, K2 = 3k K2 6K₂) where models are initialised from 2048-step DDP checkpoints, following Charles et al. (2025). While our convergence bound is not trivially applicable, our analysis of Eq. (4) suggests higher momentum synchronization frequency (p.)) should permit larger step size (Nj 1/v().
As shown in Figure 6, two key points emerge. First, more frequent synchronization (K₃ = 32) allows DES-LOC come within 1% of the final perplexity of DDP, performing much better than in infrequent settings (Kg = 256). Second, using Nesterov as the outer optimizer improves performance over averaging by z 0.5%, with its performance w.r.t DDP being similar the one reported in Charles et al. (2025, Table 4) for models at this scale. The Nesterov approach preserves the practical benefits of DES- LOC, ensuring effective worker initialization and reducing local optimization noise, which can help prevent issues like exploding activation norms.
Frequent synchronization (Ka 32) allows DES- LOC to reach within 1% of the perplex- ity of DDP. Furthermore, using Nesterov outer optimizer improves performance over averaging, while preserving practical benefits like effective worker initialization and reduced optimization noise.
Takeaway:
We further investigate the benefits of synchronizing optimizer states under a Nesteror outer optimizer relative to purely local-state methods (DiLoCo) the 1B scale in Section B.5.
5.6 MUON AS THE INNER OPTIMIZER (RQ6)
To assess the versatility of DES-LOC beyond fdam-ffamly optimizers, we integrate it with Muon (Jor- dan et al., 2024), single-momentun optimizer that uses Newton-Schulz iterations for orthogonaliza- tion. Since Muon preconditions the momentum term directly rather than tracking second -comentmoment estimates, the relevant synchronization periods reduce to parameters (K₂) and momentum (Kw)). We apply LEC- by synchronizing the Muon momentum buffer less frequently than the parameters, setting K, 3K with K2 32.
As shown in Figure 7, DES-LOC matches the perplexity of the Local Muon baseline across model scales ranging from 16M to 360M parameters. By decoupling the synchronization frequencies, DES-LOC communicates more than 1.5x fewer bytes than the baseline, confirming that the Half- Principle extends to optimizers beyond the Adam family. Full experimental details in Section B.7.

---

Published as a conference paper at ICLR 2026
Takeaway: DES-LOC is compatible with single-momentum optimizers such as Muon that rely Newton-Schulz preconditioning. By reducing the synchronization frequency of the momentum buffer relative to parameters, DES-LOC maintains solution quality while significantly lowering communication volume, demonstrating that our approach is imner-opimizzr--gooiii.
RELATED WORK
In synchronous data-parallel training, workers exchange full gradients parameters every iteration, incurring communication costs linear in model size using Ring-Allnee Alleuuue (Sergeev & Balso, 2018). When hardware is weakly connected widely distributed, communication significantly slows wall-clock training time (Sani et al., 2025) as workers need to wait for synchronization to finish. Federated Averaging (FedAvg) (McMahan et al., 2017) and Local SGD (Stich, 2019) reduce communication by performing K local optimization steps before averaging parameters, decreasing communication rounds by factor of K. Ad-hoc extensions to adaptive optimizers either keep optimizer states local (Douillard et al., 2023; Charles et al., 2025; Liu et al., 2024) or reset them after each syne (Sani et al., 2024; 2025), both lacking robust convergence guarantees.
Adam (Kingma & Ba, 2015) is popular for pre-training as it scales to larger batches than SGD (Kunst- ner et al., 2023; Dubey et al., 2024). It uses moving averages of gradients and their squares, however, its convergence is not guaranteed as it requires 81 / < 1, with large, problem-specife B2 (Reddi et al., 2018; Zhang et al., 2022). Other optimizers also track gradient moments (Sutskever et al., 2013; Chen al., 2023; You et al., 2020; Taniguchi et al., 2024). Local Adam (Cheng & Glasgow, 2025) reduces communication with local steps but requires syncing optimizer states, which triples the communication cost relative to Local SGD/DDP, as syne costs scale with the number of states. For further related work, including compressionn/paarificatio and structured updates, check Section K.
7 CONCLUSION
DES-LOC reconciles communication efficiency with rigorous convergence guarantees in distributed adaptive optimization. By extending theory the independent synchronization of Adam and SGDM optimizer states, we empirically demonstrate convergence alongside 1 170x x and 2 communication reductions over DDP and prior state-of -fte-of-the-arr art methods at billion-scale LLM training, even in envi- ronments prone to system failures. Our findings yield clear guidelines: i) frequently synchronize parameters, and ii) synchronize optimizer states less often, proportional to their half-lives. These insights open avenues for future research, including layer-wise synchronization, adaptive frequencies, compressed updates, as well as emerging applications, such as worldwide cross-data center training and collaborative training. As training workloads scale, we envision DES LOC becoming the standard for efficient, resilient mundationnmoode training in data centers and distributed environments.
ACKNOWLEDGMENTS
All costs for the computational resources used for this work were funded by Flower Labs, and the research conducted by a team of researchers from Flower Labs, The University of Cambridge, The Institute of Science and Technology of Austria, The University of Warwick, and Mohamed bin Zayed University of Artificial Intelligence. Support for university-based researchers came from a variety of sources, but in particular, the following funding organizations are acknowledged: the European Research Council (REDIAL), the Royal Academy of Engincering (DANTE), the Ministry of Education of Romania through the Credit and Scholarship Agency, and the European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cune grant agreement No 101034413.
REFERENCES
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cédric Renggli. The convergence of sparsified gradient methods. In Conference on Neural Information Processing Systems (NeurIPS), 2018.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlícek, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua
10

---

Published as a conference paper at ICLR 2026
Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojum Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model. arXiv preprint arXtv:2502.02737, 2025.
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming. 199( 2):165-214, 2023.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, February 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication-effieie language model training scales reliably and robustly: Scaling laws for diloco. arXiv preprin! arXiv: M.Kiv:503.00799 2025.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc Le. Symbolic discovery of optimization algorithms. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
Ziheng Cheng and Margalit Glasgow. Convergence of distributed adaptive optimization with local updates. In International Conference on Learning Representations (ICLR), 2025.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Samjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240: 240: 13, 2023.
Arthur Douillard, Qixuang Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.08100 2023.
Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Gar- rett, Gabriel Teston, Dave Lacey, Ross Mcllroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc' Aurelio Ranzato, and Paul Barham. Streaming diloco with overlapping communication: Towards a distributed free lunch. CoRR, abs/2501.18511 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller. Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruy Choudhary, Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes,
11

---

Published as a conference paper at ICLR 2026
Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filipp Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07948 2020.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633: 2019.
Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, and Nicholas Donald Lane. DEPT: Decoupled embeddings for pre-training language models. In International Conference on Learning Representations (ICLR), 2025.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https: / //kellerjordan wittuuttuhu...... io wwwotppssss//.oooooooooooooo /mwon/
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Olivei,, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón. Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang He. Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi. Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song. Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao Advances and open problems in federated learning. Found. Trends Mach. Learn., 14(1-2):1-21) 2021.
Satyen Kale, Arthur Douillard, and Yanislav Donchev. Eager updates for overlapped communication and computation in diloco. arXiv preprint arXiv: 2502. 12996, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001. 108361, 2020.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint urXiv:2008.03606 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference Machine Learning (ICMI.), 2020b.
12

---

Published as a conference paper at ICLR 2026
Ahmed Khaled, Satyen Kale, Arthur Douillard, Chi Jin, Rob Fergus, and Manzil Zaheer. Understand- ing outer optimizers in local sgd: Learning rates, momentum, and acceleration. arXiv preprint arXiv: 10439, 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise not the main factor behind the gap between sgd and adam transformers, but sign descent might be. In International Conference on Learning Representations (ICLR), 2023.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. FLUX.11 kontext: Flow matching for in-context image generation and editing in latent space. CoRR, abs/2506.15742 2025.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow., 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dimitris S. Papailiopou- los, and Vivienne Sze (eds.), Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020. misys.org. 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning (ICML), 2021.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations (ICLR), 2018
Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A. Rusu, Jiajun Shen, Arthur Szlam, and Mare' Aurelio Ranzato. Asynchronous local-sgd training for language modeling. arXiv preprint arXiv:2401.09133 2024.
Jingyuan Liu, Jianlin Su, Xingeheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang. Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. CoRR, abs/2502.16695 2025.
Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum, arXiv preprint arXiv:2007.07989, 2020.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. CoRR, abs/1812.061622 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communicationn-eficieenn learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
Matteo Pagliardini, Pierre Ablin, and David Grangier. The adEMAMix optimizer: Better, faster, older. In International Conference on Learning Representations (ICLR), 2025.
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning (ICML), 2013.
Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
13

---

Published as a conference paper at ICLR 2026
Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Demo: Decoupled momentum optimization. arXiv preprint arXiv:2411.19870 2024.
Samyam Rajbhandari, Jeff' Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations
toward training trillion parameter models. In Proceedings the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR), 2018.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. In International Conference on Learning Representations (ICLR), 2025.
Joshua Romero, Junqi Yin, Nouamane Laanait, Bing Xie, M. Todd Young, Sean Treichler, Vitalii Starchenko, Albina Y. Borisevich, Alex Sergeev, and Michael A. Matheson. Accelerating collective communication in data parallel training across deep learning frameworks. In NSDI, pp. 1027-1040. USENIX Association, 2022.
Lorenzo Sani, Alex Iacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William Shen, Preslav Alcksandrov, Xinchi Qiu, and Nicholas D. Lane. The future of large language model pre-training is federated. arXiv preprint arXiv:2405.
2024.
Lorenzo Sani, Alex Iacob, Royson Lee Zeyu Cao, Bill Marino, Yan Gao, Wanru Zhao, Dongqi Cai, Zexi Li, Xinchi Qiu, and Nicholas D. Lane. Photon: Federated Ilm pre-training. In Eighth Conference on Machine Learning and Systems, 2025.
Nikhil Sardana, Jacob P. Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-pptimal: Accounting for inference in language model scaling laws. In International Conference on Machine Learning (ICML). 2024.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:abs/221 1.055000 2022.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint :00i::/802.057000 2018
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im Training multi-billie parameter language models using model parallelism. CoRR, htts/1999.00053, 2019.
Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations (ICLR), 2019.
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 188:1211 127063, 2024.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning (ICML), 2013.
14

---

Published as a conference paper at ICLR 2026
Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong. Go Nagahara, Tomoshi liyama, Masahiro Suzuki, Yusuke Iwasawa, and Yutaka Matsuo. ADOPT: modified adam can converge with any B2 with the optimal rate. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. arXiv preprint Xiv:2007.07488. 2020.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Fichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi. Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi. Sashank Reddi, Peter Richtarik. Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng. Chen Zhu, and Wennan Zhu. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning (ICML), 2023.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. In NeurIPS, 2023.
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
Hao Yu, Rong Jin, and Sen Yang- On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. arXiv preprint 1xXiv.190.008811 2019.
Kun Yuan, Xinmeng Huang, Yiming Chen, Xiaohan Zhang, Yingya Zhang, and Pan Pan. Revisiting optimal convergence rate for smooth and non-convex stochastic decentralized optimization. arXiv preprint arXiv:2210007865 2022.
Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham M. Kakade. How does critical batch size scale in pre-training? In The Thirteenth International Conference on Learning Representations, 2025.
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi- Zhi-Quan Luo. Adam can converge without any modification on update rules. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
15

---

Published as a conference paper at ICLR 2026
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 2023.
16

---

Published as a conference paper at ICLR 2026
Appendix
A Table of Contents
Experimental Details and Optimizer Hyperparameter Sweeps (See Section 4.1)
Architecture Details and Hyperparameters Optimizer Parameters Sweeping Procedure
A.2
18
B Complementary Results to Sections 2.1 and
20
B.I
Toy Prohlem on Non-1 ID Data (See Section 2.1)
20
B.2
RQ2: Independent Sync Frequencies
21
B.3
RQ3: Communication Reduction And Baseline Comparisons
23
B.4
RQ4: Additional Metries Training Instabilities FAVG+OFT (See Fig. 5.b)
26
B.5
DES-LOC Nesterov LoCo at IB Scale
26
B.6
Very low bandwidth experiments
27
B.7
Muon Experimental Details (See Section 5.6)
28
B.8 Experiments on the Flux Vision Model
28
B.9
Throughput at 7B Scale
29
Further Algorithmic Details DES-IOC
C.I Extension ree0ppt
30
C.2 Deterministic Optimizer-specifie Variants of Algorithm ................................
31
DES-LOC-Adam DES -BS-LOC-ADOP-
31
31
D Convergence Analysis DES-LOC-SGDV (in expectation bounds)
32
Extension Adam optimizer
35
D.2 Key Lemmas
40
E Convergence Analysis of DES-LOC- probb-trybability bounds)
44
F Derivation of Eqs. (1) and (2): Maximum Momentum Change With Clipping
46
G Wall-Clock Time Modeling
47
Estimating Total Wall-Clock Time
48
Modeling Results
H Checkpointing vs. Periodic State Synchronization
51
Choosing Synchronization Frequencies
51
J Critical Batch Size and Regime Positioning
52
Extended Related Work
53
L LLM Usage Declaration
54
M Limitations
54
17

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS AND OPTIMIZER HYPERPARAMETER SWEEPS (SEE SECTION 4.1)
Here we provide additional experimental details complementing those in Section 4.1, including: a) model architecture details and hyperparameters independent of optimizer choice (Section A.1), b) our hyperparameter sweep procedure to select optimizer-ppecffic settings (Section A.2), and c) the optimal hyperparameters with those used Section highlighted in bold.
A.1 ARCHITECTURE DETAILS AND HYPERPARAMETERS
Table Model architecture and training parameters. We denote the number of transformer blocks by #Blocks, number of attention heads by #Heads, embedding dimension by dmodel, vocabulary size by [V], and feedforward-layer expansion by Exp. Ratio. All models use positional embeddings (Su et al., 2024), the si lu activation function, and norm-based based gradient clipping with clip-bound p. Global batch size (summed across all workers) is [Bc], and sequence length standard for models at these scales. For model initialization we use V/ddmadeh The total number of steps is denoted by T.
Model Size
Blocks
dmodel
#Heads
Exp. Ratio
ROPE
ACT
Init
p
Seq Len
[6c
T
135M
30
576
50K
9
4
10000
sila
0.04
1.0
2048
1024
1536, 3072
720M
12
2048
50K
16
10000
$110
0.02
1.0
2048
512
38912
1.7
21
50K
10000
silu
0.02
1.0
2018
1021
20480
Table 3 summarizes the architectural details of our models, following established practices for large language models at their respective scales. Unless otherwise stated, we adopt the hyperparameters recommended by Allal al. (2025) for both the 135M and the 1.7B models. We operate batch size of 2M tokens, which is very large for the 135M model at the length of training we perform (Zhang al., 2025) and industry-standard for the 1.7B model (Touvron et al., 2023), we chose to operate at large batch sizes because adaptive optimizers provide benefits primarily in large- batch training regimes (Kunstner al., 2023). Moreover, we intend DES-LOC for use in cross data-center scenarios, where effectively utilizing available accelerators naturally demands large batch sizes and/or model scales. For both model sizes, we train for approximately 2x the compute-optimal token budget (Hoffmann et al., 2022), placing our evaluations within the context of extended-duration foundation model training (Allal et al., 2025). Our chosen token budget conservative due to resource constraints; for comparison, Allal et al. (2025) used 11 trillion tokens which is over x compute-optimal for the 135M model, and 300 for the 1.7B.
We select warmup and decay schedules following recommendations from Zhang et al. (2025); Hägele et al. (2024); Allal et al. (2025). For the 135M model, the warmup period is set to TWARM = 512 steps, corresponding to the roughly 40% of the compute-optimal training tokens recommended by Zhang et al. (2025). For the 1.7B model, we use the recommended TWARM 2048 steps from Allal al. (2025), roughly 10% of total training. The stable-decay period uses the final DECAY = 10% x T steps (Hägele et al., 2024). For shorter runs, such as T = 1536 during heterogeneous data evaluations, ensure
SQRT schedule over
keep the warmup fixed and proportionally scale the decay to parameter updates during the stable learning rate period. The seeds we use for data sampling and for controlling the training algorithms and model are provided in the code accompanying the appendix.
A.2 OPTIMIZER PARAMETERS SWEEPING PROCEDURE
As detailed in Section and verified empirically in Section 5.2, the choice of decay rates 31, 82 strongly influences the effective synchronization frequencies achievable by both DES-LOC and Local Adam. This relationship arises directly from the half-life of optimizer states, given by In(0.5) ln())
T0.5
For Adam, prior studies such as Wortsman et al. (2023) have demonstrated a critical interplay between the learning rate (1)), batch size, and the second-momen momentum decay 3z. Specifically, increasing either
18

---

Published as a conference paper at ICLR 2026
the learning rate batch size typically demands lower 32 maintain training stability and avoid loss spikes. Conversely, higher B2 values constrain the learning rate and batch size. Such dynamics have also been recently observed between the learning rate and the first-momentum decay 31 in Pagliardini et al. (2025). Given that all our experiments use a fixed large batch size of roughly 2 million tokens (appropriate for billion-scale training), we systematically tune the learning rate 1j in response to changes in 31, 32. We try values of 31, 32 based on previous works (Zhang et al., 2025) and follow the theoretical convergence requirement of Zhang et al. (2022) setting 31 S /B2.
Due to computational constraints, we cannot jointly optimize synchronization periods, data distribu- tions, and decay parameters, and instead adopt a structured two-stage tuning approach:
1. Stage 1: Tuning for DDP. Starting from the recommended baseline learning rate (71) from Allal et al. (2025), we conduct a grid search as outlined by Charles et al. (2025):
f..., V2 No: v2 nu: no. V21ju. v22 .j..... We expand this search until perplexity stops improving, identifying an optimal learning rate TonP for each (51,32) 32) configuration.
2. Stage 2: Tuning for Local Adam. We then repeat this procedure for Local Adam, using Vide as the new baseline. To balance generalizability and computational cost, we set the synchronization period an intermediate value of K 64, between high-frequengy (K = 16) and low-frequency (K = 256) scenarios.
Additionally, following Zhang et al. (2025). we omit weight decay (set to zero) to simplify the hyperparameter tuning process, as it directly affects only model parameters, not optimizer states.
For experiments using Nesterov, we follow the hyperparamtere sweeping procedure of Charles et al. (2025), starting with server learning rate of 1.0 and a momentum of 0.9 and only lowering it it fails to converge
A.2.1 OPTIMIZERS' HYPERPARAMETE CONFIGURATIONS
Table 4: Optimal learning rates 17" for $1,33 configurations of ADOPT/Adam. The hyperparameter sweep procedure (see Section A.2) involves incrementally adjusting the learning rate by factors of vē around the initial value from Allal et al. (2025) until performance stops improving.
Optimizer
B1
32
0.9
0.9999
0.0021
0.95
0.9999
0.0021
ADOPT
0.99
0.9999
0.0014
0.995
0.9999
0.0007
0.9
0.95
0.0042
0.95
0.95
0.003
Adam
0.9
0.99
0.003
0.95
0.99
0.003
0.99
0.99
0.0021
Our hyperparameter sweep (Table 4) indicates that the optimal learning rate 17" under the warmup- stable-decay scheduler (Hägele et al., 2024) strongly depends on both optimizer type and the chosen $1. ,82 values. For Adam, optimal learning rates and momonmmmmmntumm decay (B2) align closely with recommendations from Allal et al. (2025), though slightly higher firt-mmoeentm decay (3)) consistently performs better, in agreement with prior findings (Zhang et al., 2025). For ADOPT (default (2), we observe a lower optimal learning rate compared to Adam, but similar best-performing 31 values. We also find that the optimal learning rate does not differ between DDP and Local Adam for given B1, 82 when K 64 and using v2 sweep, higher learning rates either do not provide benefit or diverge while lower learning rates are only necessary when pushing K far closer to the complete training duration.
We find that increasing 31 for ADOPT, and 31. 32 for Adam, leads to rapid performance degradation, particularly or above 0.99. Since the half-life than at 3 = 0.95 (Ty.s x 13.5) to justify the observed performance drop. we select 3 = 0.95 for all experiments, along with the default 32 for ADOPT and B2 0.95 for Adam.
0.99 (Tu.s ss 69) is not sufficiently longer
19

---

Published as a conference paper at ICLR 2026
Takeaway: Increasing an optimizer state's significantly affects performance. Since linear increases in cause only logarithmic changes in half-life 70.3, raising beyond the optimal value degrades performance without substantially improving the achievable synchronization frequency (Section 5.2).
B COMPLEMENTARY RESULTS TO SECTIONS 2.1 AND 5
We now provide additional results supplementing those presented in the main text. Specifically:
1. Section B.1 complements Section 2.1 by including results on the heterogeneous data distribution described in Section 4.1. This highlights DES-LOCK robustness under imperfect sampling or strongly Non- IID federated scenarios (see Kairouz et al., 2021, Sec 3.1).
2. Section B.2.1 complements Fig. by showing the separate impact of varying synchronization
frequencies for parameters and the second momentum when the base frequency is K, 16. It supports our claim that parameters and second momentum exhibit similar behavior across different synchronization regimes, unlike the first momentum.
3. Section B.2.2 extends Fig. 3 by evaluating DES L-C--LC---- Adam. We confirm that the parameter synchronization frequency is the most important, as predicted by our theory. In contrast, the momenta sync frequency is far less impactful, especially for low parameter syne frequencies.
4. Section B.3.2 complements Fig. by showing DES-LCC-ADDE
perplexity against baseline methods on heterogencous data (as defined in Section 4.1). This validates our claim from Contribution 2 regarding
effectiveness heterogeneous datasets.
5. Section B.3.3 presents an ablation study examining alternative low-communication configu- rations of DES LOC, justifying our choice of K,, = -3,, K,, = 6K₁, used Fig.
6. Section B.3.4 repeats the baseline comparison from Fig. for DES-LOC-Adamm demon- strating that DES-LOC achieves similar communication reductions and performance when using Adam instead of ADOPT.
7. Section B.4 provides additional metrics illustrating training instabilities for the FAVG+OPT baseline, including rapidly growing parameter norms, supporting observations in Fig. 5.b.
8. Section B.5 presents the full 1B-scale comparison DES-LOC Nesterov against Di LoCo, demonstrating the benefits of periodic optimizer-state synchronization under a Nesterov outer optimizer.
9. Section B.6 benchmarks DES-LOC under extremely low bandwidth (10 Gbit/s), showing 9.42x wall-clock speedups over DDP.
10. Section B.7 provides full experimental details for the Muon experiments presented in Section 5.6.
Section B.8 evaluates DES-LOC LOC on the Flux vision model, demonstrating generality beyond LLMss
12. Section B.9 measures training throughput at the 7B parameter scale on B200 GPUs.
B.1 Toy PROBLEM ON NON-IID DATA (SEE SECTION 2.1)
Toy Example Non-IID: Fig. 8 simulates the scenario from Section 3, where each worker m optimizes distinct loss fm on heterogeneous data. Both DES-LOC and Local Adam show more stable convergence and get closer to the optimum than heuristic baselines.
20

---

Published as a conference paper at ICLR 2026
Local Adam
DES-LCC
10e
FAVG
-2
10-1
20
80
Commmnicttion Bounds
Loall Dim
PANC
(a) Distance to the optimum.
(b) 2-D contour.
Figure 8: We present a toy problem in a Non-IID setting, where DES-LOC (with synchronization periods K2 192, K, 192, K, 692) and Local Adam (with K K₂) converge superior solution compared to methods that keep optimizer states local (Douillard et al., 2023; Sani et al., 2025) periodically reset them (Sani al., 2024; Iacob al., 2025). Like the IID scenario, resetting optimizer states prevents convergence due to repeated oscillations caused by reinitializations. Additionally, as seen in panel (a) between rounds 15 and 40, methods keeping optimizer states local suffer from larger oscillations further away from the optimum. The function optimized is f(1 x2) = - x1 ) + 100(x2 x²)², and we simulate M 256 workers, each adding Gaussian noise with worker-specific standard deviation Tm ~ 30,
B.2 RQ2: INDEPENDENT SYNC FREQUENCIES
This section provides supplementary results for RQ2, complementing Section 5.2. Section B.2.1 shows that perplexity has similar sensitivity to the first and second momentum synchronization frequencies at both high and low base synchronization frequencies. Additionally, Section B.2 B.2.2 repeats the comparison from Fig. for Poss-LoC-Adan revealing similar trends regarding the importance of the parameters, with reduced importance for the momenta due lower /2.
B.2.1 PARAMETER AND SECOND MOMENTUM AT K₁ 16 (SEE FIG. 3.A.FIG. 3.B)
Figure 9 examines the effects of independently varying synchronization periods (Kgs K2)) for pa- rameters and second momentum under DES-LOC-ADOPT in the high-frequency regime (K, = 16), chosen based the first momentum's half-life (To.5 13.5). Similar to the low-frequency results in Fig. 3.a, parameter synchronization frequency (K₂) strongly influences perplexity, while the second momentum (K,) has minimal impact due its long half-life. This contrasts with the first momentum, whose half-life closely matches the high- -eguffrryeency period.
180
(1L1-14)
Perplesity
60
Perplesity
140
K
(142)1
Kr
(1Afte 37.
500
31000
1200
2000
2000
3002
500
1000
1200
2000
2003
3000
Sequestial Steps
Sequcutial Stope
(a) DES-IOC vary K., fixed K. K. 16
(b) DES-LOC vary K.. fixed K. K. 16
Figure Model perplexity for DES-LOC (ADOPT, 0.95, Bg 0.9999), independently varying synchronization periods at high baseline frequency (K₁ 16). Similar to Fig. 3, parameter synchronization (a) is critical, with performance sharply degrading at higher periods, while second- momentum synchronization (b) has minimal impact due to its large half-life (7o.((82) > K2).
21

---

Published as a conference paper at ICLR 2026
(00 1alx-/////
$0.01
A.b. BALESS
5--11
06-11
1122
4--01
R (UR,
6e-111
2x 01
1 005,000
2--01
F. 106,000
1e-10
Communiication Round
(a) First momentum change rate for Local Adam.
(b) Sccond momentum change rate for Local Adam.
Figure 10: Relative rates of change for first and second momenta across rounds using standard Local Adam (K 64). Increasing 31 substantially reduces the rate of change of the first momentum, while increasing either B1, ,3 decreases the rate of change of the second.
100
[21 E
80
89
Perplecity
255 (1.6003)
Perlocty
10.0
08
(122511-)
60
40
100
43
1024 (16i469)
20
23
10B (BRROLS)
R. 302 (16109))))
2000
2500
3003
1500
2000
2500
Sequeniaal Steps
Sequential Steps
(a) DRSS 1OC vary K3 fixed Kg K2 16
(b) DES-LLOC vary K₂ fixed Kg K2 = 256
Perplecity
120
Perpoxity
(1122-01)
R. .075 (360-13)
20
2000
3000
3003
1500
2000
2503
3000
Sequentiaal
Sequential Stepe
(c) DDES-LOS vary Ku. fixed K2 K. 16
(d) DES-LOC vary Ku. fixed Ke K. 256
100
100
80
89
Perplecity
08
291 (1.3103)
Perpocty
8
f. 1601 (1SPHLL)
1633
16.0
0673
40
8
30
25
1000
2000
2500
3003
1500
2000
2500
30.0
Sequertial Steps
Sequential Stepe
(c) DESSLLC vary Ke,, fixed K2 K2 = 16 (f) DES LOC vary Ke, fixed K2 K. 256
(81 Bg 0.95) when independently varying sync periods (K,, Ku, K.,) while fixing others at baseline K,. Parameter synchronization (a.b) influences performance in both high (K,- 16) and low (K, 256) frequency regimes. Momenta synchronization minimally impacts perplexity due to both states' high adaptivity (low 8), with potentially minor effects during the early stages of training in high-ffrquency regimes (c.e).
Figure 11: Model perplexity for DES-LOC-AdO
Takeaway: In high-frequency synchronization regimes, the importance of parameters and the second momentum remains similar to the low-frequency regime shown in Scction 5.2,
B.2.2 ADAM RESULTS (SEE FIG. 3)
Fig. 10 show the rate of change results for Adam momenta at various 3 values.
Figure 11 provides complementary results Figs. and 9 using DES-LOC-aAe Adam with 31 B3 0.95. Unlike ADCPT, the relatively low 3 result both the first and second momentum quickly adapting the local gradients, reducing the impact of their sync frequency.
22

---

Published as a conference paper at ICLR 2026
Takeaway: For DES-LOC-Adamt parameter synchronization remains critical, consistent with theory, However, due reduced 82, momenta synchronization less impactful since both the numerator and denominator of Adam updates are driven by local worker gradients after few initial steps.
B.3 RQ3: COMMUNICATION REDUCTION AND BASELINE COMPARISONS
This section provides supplementary results for RQ3, complementing Section 5.3. Section B.3.2 shows the perplexity of different configurations providing a 2x communication reduction over Local Adam. Additionally, Section B.3.4 repeats the comparison against baselines from Section 5.3 for DES-LOC-Adeess showing similar communication reductions relative to Local Adam.
B.3.1 STEPWISE PLOTS FOR BASELINE COMPARISON
Figures 12 and 13 show stepwise plots for wall-clock results in the main text, they are the counterparts to Figs. and 5.
101.0
1000
HAST
-000
(2-4115))
500
Derpledtt
.0.0
3001
(a) LOOFT vs baselines K2 16 IID
(b) -O-DEEL--D-DDE baselines K₂ 256 IID
Figure 12: Setting K2 = K, K, 3K and Kg = 6K,. DES-LOC achieves a 2x communication reduction over Local 1. Adam, matching performance high (a) and low (b) frequencies for Local Adam and heuristic baselines (see Section 4. Using stepwise converges shows that DES-LO matches Local Adam on per-step basis.
40.0
9.5
DDP (8.45=0.18)
DES-LOC 08.96+0.22))
30.0
Local Adam (8.96 08.96:00266
9.0
Perplexity
FAVG OPT (7.5940.51)
20.0
8.5
10.0
0.0
0
2500
5000
7500
10000
12500
15000
17500
20000
Sequential Steps
(a) DES-DDC LLC-LDOPT 1B-mode perplexity
Figure 13: RES-LOC matches Local Adam perplexity for billion-scale model training at half the communication cost (K, 256, K, 3/ K, = 6Ka)), representing a 170x reduction over DDP. Plot shows that stepwise convergence matches between Local Adam and DES -LOC).
Takeaway: DES LOC matches the stepwise convergence of Local Adam and approaches the convergence speed of DDP.
23

---

Published as a conference paper at ICLR 2026
B.3.2 DES-LOC ON HETEROGENBOUS DATA (SEE CONTRIBUTION 2)
Figure 14 evaluates the robustness of DES-LOC against baselines under heterogeneous (Nnn-II) data distributions as described in Section 4. We set synchronization periods K = K, K₂ = 3K., and K, 6K₂ to achieve targeted x communication reduction over Local 1 Adam.
100
(ULIGGO 3N)
SES-ACC 2EL JELAPSSA
80
DeS- LDD D2.Wer AN;
Perplesity
Lookk Adaa (50.5460.3)
Perplexity
Local adan (12/2147
00
24PG (LIJERES.TT)
10.0
60
FANG (129227/0)
-0
40
-02
214
400
003
800
1000
1230
1400
230
400
600
900
1000
2010
1400
Seeguentioal Staps
Seqpontial Stagss
(a) DES-DE--EE 10SS-LLC-Adaaa high sync frequency K2 = 16. (b) MOSS-LLC-Adamm low syne frequency K2 = 128.
Figure 14: Comparison of perplexity under Non-IID conditions for DES-LOC, Local Adam (K2 = K, K.), and heuristic baselines (defined in Section 4.1) at high (a) and low (b) synchro- nization frequencies. Due to higher cross-worked variance caused by heterogeneous data, parameters require slightly more frequent synchronization in the low-frequency regime (K, = 128 < 256). Experiments limited
1536 steps (>compute-optimal) for computational feasibility.
Takeaway: DES-LOC effectively converges on heterogeneous data distributions, maintaining the 2x communication reduction observed in homogeneous settings. This aligns with theoretical convergence results for heterogeneous losses (Section 3) and shows applicability in federated scenarios.
B.3.3 DES-LOS Low COMMUNICATION CONFIGURATIONS ABLATION (SEE FIG. 4)
Figure 15 explores alternative synchronization configurations enabling DES-LOC to achieve im- proved communication efficiency over Local Adam. Motivated by theoretical insights (Sections and 3) and empirical evidence (Sections 5.1 and 5.2), we only consider settings where parameter synchronization most frequent (K₃ S min( K., K,)). This constraint follows from experiments in Section 5.2, which show that infrequent parameter synchronization significantly degrades perplexity, while momentum synchronization frequency has smaller impact. For fixed 2x communication reduction over Local Adam, our findings confirm that synchronizing the first momentum more frequently than the second aligns with their respective half-lives and maintains performance close to Local Adam.
Rig (19.611440)
Perplecity
Perplesity
Re, 1x.1x (12.9123:00
100g
2000
2000
3003
1500
2000
2500
30,0
Sequential
Sequential Stepe
(a) DBS-LOC-ADOOT, high syne frequency Ka (b) DES-LOC-ADDPT.. low syne frequency K₂ 256.
16.
Figure 15: Configurations of DES-LOC targeting 2x lower communication than Local Adam (K, K,, = K,), setting K,, K,, as multiples K,- In both high (a) and low-frequency (b) regimes, performance depends on how communication is split between momenta for 31 & f2. Syncing the first momentum less often (K, 6Kg K, = 3K,) degrades performance, wasting communication the slow second momentum. Conversely, syncing it frequently (K,, 3K K, 6K₂) yields performance comparable to Local Adam. Setting K,, = K,, 4K₂ produces intermediate results.
24

---

Published as a conference paper at ICLR 2026
Takeaway: For a given parameter synchronization period K₂ determined by bandwidth constraints, choose momentum synchronization periods Ku, Kr multiples of Kr. When $1 < 82, set Ku. Ke, with K = 3 K and K 6 x Ka providing robust default choices.
B.3.4 ADAM RESULTS (SEE FIG. 4)
We now present results for Poss-LO--an -oca-Aa- with 31 = 32 0.95. DES LOC Adam achieves similar communication reductions over Local Adam and DDP as ADOPT. However, due to the lower 32. the second momentum half-life (0.95) ~ 13.5) is significantly shorter than for ADOPT (To.5 (0.9999 ss 6931). Figure 16 shows that with both momenta evolving at similar rates, the benefit of selecting K,, < K,, diminishes. For consistency and due meaningful empirical differences in rates of change (Section 5.1), we keep K, 3 x K₂ and K = 6 x K2 in subsequent comparisons.
100
Ea Ri &x (1 (1-77)
Ru Re Ge, (35 (64077)
BA
dx.4x (1121197)
Perplecity
(1135:8700)
220
Perplesity
6
16C-
500
1000
1350
2000
2000
3002
500
1000
1500
2000
2500
3000
Sequentiaal Steps
Sequcotial Stope
(a) DES LOS-LLCCAdam, high syne frequency K, = 16.
(b) DES-LOC-Adam,, low syne frequency K2 256.
Figure 16: Configurations of DES LOC targeting 2x lower communication than Local Adam (K, = Kn K.), using Adam (3) 32 0.95). In contrast to DES-LOC-ADOO (where 31 < B2 yields an advantage for K, < K, as shown in Fig. 15), the similar half-lives in Adam make perplexity insensitive to how communication is split between momenta for high (a) and low- frequencies (b).
Figure 17 shows DES-LOC-Adan achieves 2 communication reduction over the prior state-of- the-art Local Adam (Cheng & Glasgow, 2025) without significant perplexity degradation. Due to the much faster evolution of the optimizer states using Adam compared to ADOPT, local worker gradients drive the optimization reducing the benefit of allocating more of the communication budget to the first momentum.
100
(10-20-11331
DOP (UL1119333)
80
DRE-LDG 1J) 3E
83
DES-LUC (169120 17)
Purplesity
Local adea (1L8111
Porploxtty
Local idao (1830-.1
00
PAYGHOPP (11.420971)
60
FANOIDrT (14 SE55-70)
FANG-4FT (S-N
01
4
00
20
2500
Sequential Staps
Sequential Steps
(a)
high sync frequency K2 = 16. (b) DEIDEE--CGE -..00---damm low syne frequency K; 256.
Figure 17: Setting K, = K, K, 3Kg, and K. 6K.. - Adamm achieves 2 commu- nication reduction over Local Adam, matching performance at high (a) and low (b) frequencies for Local Adam and heuristic baselines (see Section 4.1).
Takeaway: BESS-LOC-Adaan achieves similar 2x communication reduction over Local Adam as DE. ADOPT by exploiting the reduced importance of optimizer-state synchronization relative parameters. However, due to the smaller 32 in Adam, there is limited benefit from assigning different synchronization frequencies to the first and second momenta compared to ADOPT.
25

---

Published as a conference paper at ICLR 2026
B.4 RQ4: ADDITIONAL METRICS AND TRAINING INSTABILITIES OF FAVG+OPT (SEE FIG. 5.B)
Figure 18 complements Fig. 5.b by showing parameter and update norms for DES- and baseline methods when training billion-scale models. Both DES-LOC and Loca Adam regularize updates by synchronizing optimizer states, effectively reducing update norms due averaging across workers (triangle inequality). In contrast, the heuristic baseline (Sani et al., 2025) experiences large updates, leading to uncontrolled parameter growth, increased activations (Fig. 5.b), and degraded performance on downstream ICL tasks (Table 1) relative to its perplexity (Fig. 5.a).
Momentem Nora (13)
12000.0
(197188111. 111/0)
Parameter Normu (12)
10000.0
1500
12011
Enleaon (100)
0000.0
1000
Leat Mike (10
FVC on (1at
4000.0
530.
7500
32500
15000
20000
2003
3000
10300
1200)
35000
17500
20000
Suqurtiaal Stegs
Sequential Stage
(a) DES-LOC-ADOO 1B-update norms
(b) SANS-LOC-ADDE 1B B-paramettr norms
Figure 18: Comparison of update (a) and parameter norms (b) for billion-scale models trained with DES-LOC (K, 256, K, 768, K, = 1536), Local Adam (K 256), DDP, and Federated Averaging with persistent optimizer states (FAVG-OPT) Frequent synchronization in Local 1 Adam and DDP consistently reduces update and parameter norms. Similarly, DES -CC achieves comparable reductions at intervals corresponding to multiples of Lem(K,, K,, K,), with smaller intermediate drops. Conversely, FPVG+OPT, which does not synchronize optimizer states, experiences persistently larger and noisier updates, becoming vulnerable to spikes (notably before step 5000). This leads to uncontrolled parameter growth (b).
Table Throughput (batches/sec) for IB-13B models reach 2x compute-optimal tokens high (K 16) and low (K = 256) frequencies with a 2M token batch size. All local methods achieve significant throughput gains over the DDP baseline. At high frequency (K 16). DES-LOC boosts throughput by over 1.7x on the 13B model. At low frequency (K 256), this advantage grows to over 2.1 versus DDP.
1B Model
7B Model
13B Model
Method
K, = 16
K2 = 256
K, 16
K, = 256
K, = 16
K, = 256
DDP (Baseline)
171.9 - 0.94
171.9 0.9
25.1 + 0.10
25.1
11.1 0.03
11.1 0.03
PAVG+OPT
304.9 2.51
385.4 3.7
34.0 0.11
0.15
19.4 0.11
23.5 0.15
Adam.
253.1 + 1.59
350.1 3.6
30.9
0012
16.7
0.11
DES-LOC (K,, K, 3K,, 6K.)
299.2 + 2.39
384.1 = 3.7
33.7
40.4
23.5 0.10
Unlike heuristic methods, which maintain purely local optimizer states leading to un- stable, noisy updates, DE.S-LOC provides stable regularization similar to Local Adam and DDP by periodically synchronizing parameters and momenta, reducing training instabilitics.
Takeaway:
B.5 DES-LOC NE TEROV vs. DILOCO AT 1B SCALE
Having shown that a Nesterov outer optimizer improves DES-LOC (Section 5.5), we ask whether synchronizing optimizer states still helps relative to the local-state (K,, K, oc) Nesterov method D LoCo. Charles al. (2025) has shown that large scale (> 1B parameters) D LoCo match outperform DD2 with Adam. We adopt their outer hyper-params and train 1B model with the same experimental design as Section 5.5 using 4 X 8 H100s for 40,960 steps (N= 4x compute-optimal) with inner Adam, comparing Adam-DDP, Local Adam, DiLoCo, and DES-LOC Nesterov, with K2 = 32 for Local Adam and DiLoCo and K, = 4K, K. = 8K, for DES-LOC Nesterov.
Figure 19 (top left) shows that Nesterov-base methods outperform Adam W: DiLoCo achieves 7.63 + 0.20 validation perplexity, improving Adam-DDP by R 2%, while DE.S PRRS-OO Nesterov
26

---

Published as a conference paper at ICLR 2026
1.0
0.35
Peppoody
Toual
0.6
MAaiMCEEP (7.KK (NSI)
0.4
Locd Make) Ku D0
0.11
30000
20030
3000
(000
0.0
1000
2000
590.6
40300
Sequentaal Stops
Sequerite Stope
Teea Xetroattes Noem Chrrpet)
X10
Peppooiy
00-00
Lacel (15 (isedaa
ERtato (7.k 1111100
Adaott UnP (2 20) ( $7.26
DeSsoOC (T551/22) in 10335
Lond Mice xinesc 1-0014
DALaC' (rAdeORs 1-4132
20
30
50
100000
2000
50000
43300
Time
Seqpicatial Steps
Nesterov on a 1B-parameter model trained for 40,960 steps with an Adam inner optimizer. Top left: train perplexity steps. Top right: worker gradient norms. Bottom left: train perplexity time, bottom right: whole-model output activation norms. Shaded regions show std across workers. DES-LOC Nesterov outperforms Adam, it also outperforms Di LoCo at the cost of more communication. Error bars show variance across workers, accounting for compounding local drift.
Figure 19: Comparison of Adam-DDP, Local Adam, DiLoCo, and
reaches 7.56 + 0.20, a Ri 0.9% gain over Di LoCo; both outperform Local 1 Adam (8.13 + 0.23). Note that this comparison pits Nesterov-based local updates against DDP with standard Adam; as Charles et al. (2025) note, once DDP also uses Nesterov, this gap can shrink or reverse depending on model size and worker count. These results show that synchronizing optimizer states preserves the benefits of Nesterov while retaining the advantages of state averaging.
We analyze the interaction between the optimizer states and the outer optimizer by measuring the gradient norms and activation statistics. In Figure 19 (top and bottom right), for both DiLoCo and DES-LOC Nesterov, gradient norms drop rapidly relative Adam-DDP and Local Adam and remain roughly 2x smaller than DDP thereafter, suggesting that Nesterov may steer optimization toward smoother regions of the loss landscape. State synchronization slightly accelerates the decrease in gradient norm over DiLoCo, after which the curves coincide.
Under DiLoCo, total output activation norms grow monotonically to more than 2x the Adam- DDP values, whereas DES-LOC Nesteros substantially slows this growth, ending 32% below DiLoCo (bottom row). This resembles the stabilization seen for DES-LOG versus F FedAvg baselines without optimizer-state averaging (Fig. 5), where periodic averaging also curbed activation growth. This supports viewing finite synchronization as a regularizer that limits worker drift and optimizer- state noise, yielding better-controlled activations while offering Nesterov's benefits. DES-LOC Nesterov rov does incur additional communication costs relative to DiLoCo at fixed Kg, being R 2% slower than DiLoCo under these bandwidth conditions (Fig. 19 bottom left) and K., K₂ settings while being ~ 8% faster than Local 1 Adam. The extra cost can be made arbitrarily small by increasing the optimizer-star syne periods (K,, K₂): in the limit K,, K, + in both performance and communication, while any finite sync period partly inherits the robustness and fault-tolerance benefits of synchronizing optimizer states, modulated by the chosen I's.
recovers DiLoCo
Takeaway: At the IB scale and long horizons, Nesteros rovverreddd local-update methods (D (DiLoCo, DES-LOC Nesterov) outperform Adam-DDP, consistent with prior scaling-law results. Relative to Di LoCo, DES-LOC Nesterov matches or improves perplexity while substantially reducing gradient and activation norms via periodic optimizer-state synchronization, yielding a tunable point on the
Parelo frontier.
B.6 VERY LOW BANDWIDTH EXPERIMENTS
While perplexity is invariant to network bandwidth, wall-clock time is highly sensitive to it. To practically showcase this, we perform a benchmark with a 1B model to measure time under extremely low bandwidth conditions (10 Ghit/s). This setup simulates scenario with affordable, consumer-
27

---

Published as a conference paper at ICLR 2026
Porplecity
200
Peeplestty
10.0
Mdde W. DnP (12218-30) T.
AMaaRE-TTPP 50
5.0
DeS-LOC-Saaame
3.0
Ditadla 0u12e620) Sisaus
DiLoCa (132200235)
100
0.0
2000
3000
0000
10000
Time
Sequential Stepe
Figure 20: Training efficiency benchmark on a 1B model under restricted bandwidth (10 Gbit/s). Left: Perplexity versus wall-clock time. DES DES-LOC Nesterov effectively decouples training time from bandwidth, finishing in = 9 hours compared to the projected = 3.5 days for DDP (dashed blue). Right: Perplexity versus sequential steps. While step-wise convergence is comparable, the communication overhead of DDP creates a massive bottleneck in the time domain.
grade interconnects rather than data-centers. Due to the extreme gradient synchronization delay inherent DD? this regime, the benchmark was limited 10,240 step horizon to remain feasible.
As shown in Figure 20, DES-LOC Nesterov dramatically reduces training time by = 9.42x compared to DDP, completing the run in 8.99 hours versus 84.73 hours (3.5 days) for DDP, even with the constant overhcads of our unoptimized implementation. Furthermore, DES-LOC Nesterov more efficient than Local Adam, finishing R 7% faster (8.99h vs. 9.62h) while achieving significantly lower perplexity (8.91 vs. 10.19). When compared to the litta-ligtt DiLoCo baseline (8.68 (8.68h), DES DES-LOC Nesterov incurs a time penalty of ss 3.6% due to the additional optimizer state synchronization. However, this yields performance gains, improving final perplexity by = 2.3% (8.91 vs. 9.12) over DiLoCo.
extremely low bandwidth settings (10 Gbit/s), DES IOC Nesterov eliminates the communication bottleneck, reducing training time by 9.42x over DDP. It strikes balance the Pareto frontier: its wall-clock time is in-between those of Local Adam and DiloCo while outperforming them both in perplexity.
Takeaway:
B.7 MUON EXPERIMENTAL DETAILS (SEE SECTION 5.6)
This section provides the full experimental details for the Muon experiments presented in Section 5.6. Although comprehensive theoretical treatment of preconditioned local updates under Newton- Schulz iterations is outside the scope of this work, the DES-LOC design is inherently compatible with such structures, as the momentum buffer follows standard EMA dynamics regardless of the subsequent preconditioning step.
Experimental Details. We utilize the standard PyTorch implementation of Muon with Nesterov momentum enabled and weight decay of 0.1. Following the recommendations of Liu et al. (2025), we apply the match_ match_rrs______ rm adjustment
learning rates. We adopt the conventional split optimization strategy for Muon: Adaml handles embeddings and layer normalizations, while Muon optimizes all 2D matrices (Jordan et al., 2024). The momentum parameter for Muon set to 3= 0.9, while the Adam component retains the 3₁ 0.9, 82 0.999 settings used elsewhere. Gradient clipping thresholds are scaled by model size: 1.0 for 16M, 0.5 for 125M, and 0.25 for 360M. For the Local Muon baseline, all optimizer states (Muon momentum; Adam first/second momenta) synchronize every 32 steps. In contrast, DES-LOC delays state synchronization: the first momentum (for both optimizers) synchronizes every 96 steps (3x reduction), and Adam's 's second momentum synchronizes every 192 steps (6x reduction).
B.8 8 EXPERIMENTS THE FLUX VISION MODEL
To demonstrate the universality of DES LOC across different modalities and architectures beyond standard decoder-only LLMs, we evaluate its performance on Flux (Labs et al., 2025), a Rectified Flow Transformer designed for text-to-imaae generation. This architecture differs significantly from the causal language models evaluated in previous sections, serving as a robust test for the generalizability of our decoupled synchronization approach.
28

---

Published as a conference paper at ICLR 2026
Experimental Setup. We utilize the 280M parameter variant of Flux provided by torchtitan
training with a global batch size 256. The inner optimizer is Adamw with 31 = 0.9, B2 = 0.999. We compare three settings: (1) DDP. (2) Local Adam. with synchronization period of K = 32, and
(3) DES-LOC with a parameter syne period of K₂ = 32. For DES- -LO,, we decouple the momentum synchronization significantly, setting K, 3K, and K, 6K₂ (192 steps).
3.0
Perplexitt
2.0
1.5
DDP (1 000000)0))
0.5
Local Adam (1.82:0.02)
DES LOS (1.82+0.02)
0.0
1000
2000 Sequential Steps
3000
4000
5000
Figure 21: Training loss comparison the 280M parameter Fl Flux model (Rectified Flow Trans- former). DES-LOC (K₂- 32, Ku = 192, K, = 192) effectively matches the convergence trajectory of both the fully synchronous DDP baseline and Local Adam (K' 32).
Our results, visualized Fig. 21, indicate that DES-LOC generally matches the performance of Local Adam and approaches the DDP upper bound.
The efficacy of DES-1LOG extends beyond L.LMs to Rectified Flow Transformers ux). The method generally matches the performance of DDP and Local l. Adam while reducing communi- cation by 2x over Local Adam, demonstrating the universality of the approach. We leave the scaling this result larger vision models for future work.
Takeaway:
B.9 THROUGHPUT AT 7B SCALE
To assess the practical scalability of our method on state-of-the-art hardware and large model scales, measure the training throughput of 7B parameter model distributed across independent NVIDIA B200 GPUs.
Throughput Analysis. As illustrated in Fig. 22, during the local update phases, each GPU operates at the peak efficiency of a fully isolated local run, achieving identical tokens- peenss-pr--eeood throughput single B200 with zero synchronization overhead. Distinct drops in throughput are observed only at the sparse synchronization boundaries (K, 32, K,, = 96, K, 192), where the system pauses aggregate model parameters and optimizer states.
29

---

Published as a conference paper at ICLR 2026
15000.0
Througliut (4px)
12600.0
10000
10000.0
7500.0
5000
3000.0
2500.0
0
50 100
150
200
250
Step
Figure 22: Instantaneous throughput (tokens/sec) for 7B model 8x B200s. DES-LOC maintains peak "local-only" speed for the vast majority of steps, with throughput dips occurring only at synchronization intervals (32, 96, 192). In contrast, standard DDP would incur synchronization penalty at every step, permanently depressing the throughput curve.
Crucially, standard DDP incurs this communication penalty at every single training step, significantly lowering the average tokens/sec. Even with current unoptimized "stop-the-world" implemen- waich explicitly pauses computation to communicate and does not leverage computation- communication werlap-DES-- significantly increases aggregate throughput by amortizing these costs over long local training windows.
Takeaway: On high-performance B200 hardware, DE.S-TOC enables near-linear scaling by keeping workers high-throughput local regime for majority of training. By restricting communication overhead to sparse intervals, delivers significant wall-clock speedups over DDP, even without low-level implementation optimizations like communication overlap.
C FURTHER ALGORITHMIC DETAILS OF DES-LOC
C.I EXTENSION TO FEDOPT
Although Cheng & Glasgow (2025) show provable convergence for adaptive inner optimizers in federated optimization framework, their result rests on the assumption that after period of local work, the new global model is created by averaging the local client models. In relation the larger FedOpt literature (Reddi et al., 2021), the scheme chosen by Cheng & Glasgow (2025) resembles that of FedAvg, or where the server optimizer SGD with the outer learning rate to one (Reddi et al., 2021). Naturally, the question of whether alternate server optimizers than have been used in prior works can also be implemented for Local Adam, and thus DES-LOC, arises.
We argue that indeed DES-LOC's principles can be effectively applied to any FedOpt method and not just FedAvg. While using an alternate server optimizer does not have proven convergence guarantees as yet, show in Algorithm that the choice of the ServerOpt is not constrained from practical point-of-view However, the improvements that DES-LOC provide are related to the local optimization procedure, which is orthogonal to the outer optimizer choice. Choosing the correct, and most effective, outer optimizer is an open research area (Khaled et al., 2025), and we leave the investigations of the interactions between DESS-LOC and outer optimziers future work.
30

---

Published as a conference paper at ICLR 2026
C.2 DETERMINISTIC SPTIMIZER-SPECIETE VARIANTS OF ALGORITHM
Algorithm 2 -0S--LO-AAaann
Require: Model tensors, Hyper-parameter
1:
1..0 R initial parameter vector,seeds for first and second moments
(no) R20 step-size schedule
3:
81, [0, 1) Adam decay factors
4:
P. E Rs0 gradient clipping term, l2 stability term
5:
T.M N4 total iterations, number of workers
6: K2, Ku, Ku E N4 syne periods for parameters, first and second moments
Ensure: 2T, ur-1, VT-1
7: for each worker mr rig = 20, w- = - =
local init (f -1 seeds)
8: for Lo 0....., T
do for all workers
training
9:
M in parallel do
10:
9e" i
stochastic gradient clip to radius sync
11:
9: + clip
12:
mod K, = then
13:
ut" < 31 Em [4 + (1 -1)0
14:
else
ut"" < 31 ui
(1 -1)
mod K, then
syne
17:
0i &- 32 Em [v + (1 82) @ )
18:
else
19:
1t Bg ut + (2t @ Ø(")
20:
d,
1/1
u
bias-coreeted step
n + X2
21:
if mod
K2 (1 then
sync
22:
T141 F. [x:"" d
23:
else
24:
#t+ 6- m m
Algorithm 3 DES L-C--LCC-DDD
Require: Model HennorrsHyyer-ppraameeee
I: Jo, -o.m-1.0-1 U-1 E Rd - initial parameter vector and momenta
2:
(no) R learning rate schedule
31, B2 [0, 1) - decay factors
4:
P. E R gradient clipping term, stability constant
5: T.M N+ total tterations,numbee of workers
6: K2, Km, K. N sync periods for parameters, first and second moments
Ensure: ær; MT-1, -1
7: for each worker m: xr = 20, m_ = =-1
local initialization
8: for t= 0....... T do
9: for all workers m 0.
M in parallel do
10:
9thh (F(""" H - clip(g("
stochastic gradient gradicot clipping
12:
if mod K. 0 then
13:
v/ + B2 [v] + (1 (o." о )
14:
else
15:
v/th + v2 +(1 (2." @ T(m)
16:
if mod Km 0 then
17:
mf + 31 Em [rn]"
(1 B1)
3 max
18:
else
19:
m + 31 m/ + (1 31)
max
20:
d, < qum!
ADOPT update
21:
if L mod K, 0 then
22:
wm Em m mm
23:
else
24:
#1+1 f- x d
31

---

Published as a conference paper at ICLR 2026
D CONVERGENCE ANALYSIS OF DES-LOC-SGDM (IN EXPECTATION BOUNDS)
Here provide non-convex convergence analysis of the proposed approach applied the SGDM optimizer which has single state (N = 1, momentum). The complete description of the algorithm can be found Algorithm 4.
Algorithm 4 DES-LOC-SGDM
Require: Model tensors
1:
E R initial parameter vector
2:
E IR seed for the momentum, initialised to
Require: Hyper-parameters
(1tt [ C R=00 - step-size schedule
4:
3 E [0, 1) - Momentum decay factor
5:
T' N total optimisation iterations
6: M N4 number of workers
Pa K2 Pu = i K.
[0, 1] synchronization probabilities for parameters and momentum
Ensure: ur-1, 11
8: for each worker xg vo. " 0
seeds)
for O.....
training loop
10:
for all workers m = O...., M- in parallel do
11:
8e" + Fm (x ;5 )
stochastic gradient
Bm [Bu (1 }) with probability Pa
12:
ur"
Bu7 + B)s""
with probability 1 Pu
sync
Bmn [x net"), with probability Pz
13:
Tpu"" with probability 1 -
syne
In order to facilitate the technical presentation, we model synchronization frequencies by assigning probabilities to each averaging event. For example, the parameters x²" are synchronized with the probability Px K_ which is statistically equivalent to performing the averaging in every 1 K2 P=, iteration. Similarly, momentum u," synchronization happens with probability p.. 1, K. which can
differ from Pa-
Step (virtual iterates). For each step t 0, denote the average parameters, momentum and gradient as follows:
def Em
[x)h
def Em [u,".
def Em..."
Then these averaged variables follow the "standard" centralized SGDM dynamics:
Ut Bug + (1 -3)g
Øt+1 = Z1 7/4..
Letting _-1 To, define the global virtual iterations follows
def
Ft
Ft-1 120 0.
The key property of this virtual iterates we are going to exploit in the next steps is that they follow averaged gradients, namely for any & we have
E++1
2t
1-3 (f141
Izt)
(st - xt-1)
8
nn
3 14 +
U4
(Ut -1g-1) = -199t-
1 1
32

---

Published as a conference paper at ICLR 2026
Step 2 (smoothness over virtual iterates). Then apply smoothness of the global loss function over these global virtual iterates.
f(z++1 & f(x)) + (Vf(=)) 2641 zx)
21+1
ze/1
L
f(z)) + V ( Z+41 2x) (V((z)) V.((s)); 2111 2x)
|l lell
11
111
In the next step, we separately bound each term appearing in the above bound. Step 3a (one step progress). Bounding term I.
B(Vf(xt): 2++1
2²)
M
BB (xx))
Σ
9t"
-1
Vz(z)),
1
V
fm(()))
M
" " E
(x))
11 R 2
V
(((xtt
1
v Vm (x+
2
M
M
"
V (x
11 E 2
1
Sm (xx)
V vm
M
M
M
M
- 7E
(=
V - V fm (1
-
M
2MM
m==
m=1
M
nL2
M
EE
V
Blaa, 211
M
2M
mE1
m=
Lemma 3
Step 3b (one step progress).
Bounding term II.
M Σ
- V 7411 - 2x)
nB -
fm (2("
M
M
nP
V(((2) vf(x))1 +
7m
2
2p
M
m=1
mpL
M
24
+ E
V fm (xx")
2
2p
M
Lemma 2
m==1
Step (one step progress). Bounding term III.
L
2
L
M
Bl|2+11 - 22122
9t
2
2
M
m=1
T
M
1
Σ
9t
V (
+
E
V
Jm
(x m
2
M
2
M
m=
m=1
2
M
1
M
1 m
V
Vm (x) 11
E
V
(x="
2M
2
M
m=1
m=
T2L
L
M
<
0² +
E
V
(xx")
2M
2
M
172
33

---

Published as a conference paper at ICLR 2026
Step 3abe (one step progress). Combining previous bounds.
Vf(xt) 20+1 z2) +1 L
Ef((++1) - Ef((z)
E (f(t)) 241 (011))241110
- 21) +B
V (2))
|
zell
2
11
111
M
2
nL2
M
s
E Vf())) 112
DF
V
Vim( (x
1t x
2
M
2M
m==
m==1
Lemma 3
M
+ 2
E B1 - xz 2011
+ 71 +
E
(m
(x)
2p
M
Lemma
m=1
2 0²
17 I.
1
M
+
E
vm
(x(TT)
2K
2
M
mml
M
$
V(zx))
7)
1
1
nL
E
V fm (z)
2
2
P
M m==1
npI.
?
M
2 2M 0
B11z
xzll"
+
E
10
Ft
12
2
2M
Lemma 2
Lemma 3
Step 4 (final). Now we average over the iterates and apply the bounds derived in Lemmas 1,2.
((f( - f(zo]]
1 (2(+1) - (2+1))
T
T
T
T-1
17 (111 11- - nn
M
1
f(x))
E
2T
2
P
M
t=0
t=0
m=1
npL²
T-1
n1? 1
T-1 1
M
+ 7
El xxel? 2
Σ
x. 11
0²
2
T
TM
2M
1=0
(=0
m=1
Lemma
Lemma
<
17
T-1
E
1 1-
nL
1
T-1
E
1
Σ
V
(2."") T
1 0² 2M
2T
2
P
M
npL
32232
17
T-1
0²
+ (1 3)2
V (2
(1-3)2M) 3)2
M
2L
12 -1) T
Σ
E V
4n 102 +3G2)
2
t=
T-1
<
(1
- 138) - t=0
2
1)
[ (1-1- 11 11---------
1
3L2
Σ
M
m (x
2
(1 - 3)
T
M
t=
m=1
2
02 + 1/ PL 0 + 2m 220 +
2M
2(1 - 3
34

---

Published as a conference paper at ICLR 2026
Next, we choose 2 and step size 7 such that
12 - 1) -
to bound the first term
2
nL + 2012
(1- 8)2 -
to bound the second term
12 1,
from Lemma 3
2
Note that
def
min
1 - 3,
4L
6 v max (1 B 1)
satisfies all three bounds. Then, with any v) S r/ we get
B[f(=T) - f(zo)]
T-1
Σ
T
4T
t=0
1 x² + pT222.3
+
0² 200
+
2M
2(1 3)2
Noticing that 2g 2o and fr S f(2)) we have
T-1
(((x)) 4(((zo)-17
2
0² +
0² 87) + 30
T t=
n/
M
(1 -1- B)3M
Furthermore, choosing min(no,
} we get the following rate:
T
E (x))
T
1
4(1(20)- ( + 2L
4I228o02
max
1,
+
+
NOTT
VT
MVT
(1- 3)²MT
T
4(f(zo) - -
2Lo²
4(f(z0) 1*)
4L2880Z
8L2 + 3G
VT
MVT
N0T
(1 (1-3)2M
T
(1(50)
Lo²
1 + z
VT
2M
T
D.1 EXTENSION TO ADAM OPTIMIZER
Here we discuss extension of the previous analysis for the Adam optimizer including the second- order momentum the analysis. The addition is similar the first-order momentum while the synchronization probability P. can differ from other probabilities P., and P.- The complete description of the algorithm can be found in Algorithm 5. Instead of bounded heterogeneity Assumption 3, in this analysis we use stronger condition mentioned below:
Assumption 4 (Bounded gradient). For any iterate 2 0 and worker m, the local stochastic gradient is bounded, namely |!,," 2|2 < G.
This condition facilitates the analysis by providing uniform upper bounds for gradients/momennum variables and is commonly used in the analysis of adaptive optimization.
Step (preconditioning and virtual iterates). Let I'me def diag matrix and for each step / U, denote the averaged variables
1/2 "V""""" + x²) be the preconditioning
Ft def Em
[,
us def Em
def
[m]"),
E def
Em [TT]]
gt def
Emm 1g m2
Then
Wb =
-114-1 1 + (1 -1- -B1)ge
Ft dj 3t -,)E.,. [ mm ui
T++1
35

---

Published as a conference paper at ICLR 2026
Algorithm 5 DES LONS-LOC--- LOC-Adam (with probabilistic synchronization)
Require: Model tensors
1: E R initial parameter vector
2: u-1, E _ seeds for first and second moments, initialised to
Require: Hyper-parameters
3:
{m C R>0 step-size schedule
4:
$1, B2 E [0, 1) Adam decay factors
5:
À R20
stability term
6:
Te N4 total optimisation itcrations
M E N number workers
8:
K2 Pa Kú Pv.
Ky
E [0, - syll-syycconnizattion probabilities for parameters and momentums
Ensure: 3T; uT-1, VT-1
9: for each worker m: x0 = Ju, w vo1
Tocal init ( seeds)
10: for = 0. =....., T do
training loop
11:
for all workers = O.... M 1 in parallel do (F
12:
9i"
stochastic gradient
Bm [m[3ru u²= + (1 B))g(") with probability Pv.
4r
Bruf=1 + (1-Bi)9""
with probability Pu
syac
Em [320 + (1 (g) st
with probability Pe
14:
B20/ (1-32) 20 with probability 1 Pu
syno
o
< max(v? m (111)))
AMSGrad Normalization, 6_1
V-1
11
16:
d:
@ us"
bias-correcies update
X2
Bm 7 df"),
with probability ps- with probability Pr
11+1
syaс
Consider the same averaged iterates 2a and virtual iterates z as before:
B1
31 21
1 B1
In particular, 20 zo. Then,
81
Zt+1
2t
(I++1 - 21)
(0t xz-1)
B1
1 B
7
nn3
Em [
+
Em [1 1-
Bi
1
Bi
"
Em
I uz
m1
+
n/31
E, mm t- --=
17.1
Em
I "
B1
1 31
1 B1
=
"
Em
Tm
Bru 1)
n13
E, [T
t-1 )
B1
1- B1
n1
Em
[ 9tth
En (TmLL
1 31
TB
-1BBm[[T 9r]
+ TyE
T (ge
+ ) )]
Em (T
1 )
1
13
-t)Degt
Em m (g: F
+1
Em m
1-1 >mm)) Jum"1
B1
defu
def
where T, def Em [ and Dt"
def 11"h -81011 for which, Eyn Bm[g?""
Ey [9]" gt-
1-B,
36

---

Published as a conference paper at ICLR 2026
Step 2 (smoothness over virtual iterates). Then apply smoothness of the global loss function over these global virtual iterates.
f(2++1)- f(z))
(Vf(z)),
2041 20+1
21) + 1 2el1
L | 2112
n (Vf(2)). Pegi) + MV (z)) Vi) v1)+ +
V (Vf((z) Light + - vf(2),
Up) + (Vf(z),
111
LL +
Figt U,
Vill
2
1V
In the next step, we separately bound each term appearing in the above bound. For clarity, we are also going use (xxe] G and [[V[l S G. However, these conditions can be avoided through linking V f(() term to vf(x)). and f((x)) term to Em V (x(") with the bound for E|||2 - x;" 12].
Step 3a (one step progress). Bounding term I.
(x)) (19o))
= B (V (x=z) I. 19-1 + THE ((](()) (T.-1 - -eeg:))
M
s
v f(x)) M Σ (x!"
B (- rell.
P
V
12
" E
Sm Tm
2
2
M
M
+
vf(x)))
1
Σ
V
(m(")))
G
r.ll
2
M
m=1
M
!
B|| [xx))
1|
E
fm (xmT)
2
2
M
m=l
M
+ 2
EE
fm (Iz) - v (x("
+ G Billi I'ell]
M
m=
M
S
E
E
V Sm (+,
2Co
2
M
m=
+
E
V
Vi (2+) - fx + G
-ivlll
2XM
m=1
?
Σ E
GG 1[ +
17]
Pell,,
2Co
2AM
where indicates the spectral norm for matrices, and we used the following inequalities:
M Σ
M
M
|[ min
Σ
Σ
def
M
M
M
X
X
Co
m=
min
m=1
m=
37

---

Published as a conference paper at ICLR 2026
Step 3b (one step progress). Bounding term II.
g/ M
II (VVf(z), S n
-
Σ
| (g:
9:
)
M
m=1
G/G
11
96" ||.
AM
Step 3c (one step progress). Bounding term III.
n131 G M
III (V)())), Vi) S <
Σ
11TT-11 - -
B1
m=1
n G
Σ m=
1771
1 B1
M
Step 3d (one step progress). Bounding term IV.
L
IV
:eg U. Vill
31
3r2² L
3r7²L
||Tgge1]+
||,,11
1|V/11
2
31 L.2
3
M
3 3II..IGGG
M
Σ
119:
0t" 00'1
Σ
2X2
2X²M m==
2(1 B1)M
m=1
Step 3e (one step progress). Bounding term V.
v n -Vf(2)) Vf(e),
= 7 (V)(() T + DE (E)((()
Vf(2), (T, - rr-1)g))
M
17 LL
wE
v v/(z), M
Σ (m)
E
[[T, - -r-1)99
1-83 31
Pe-1
(VS( - (\(),, V((xe)) .]
w/
(P/(o)) (V)(o))-V(aa))) (z()), 1
12 G2
+ nE
Σ
{mm(""") (.(((E)))
E -
M
(1- 31)
Te
E [()))()))))))))))))))))){))))[
+
X
[V vs(=))]
1
Σ
|I
- (Vmm))
+
LF
[[]T r.-1
M m=1
(1 - -1)A
F
1
[V
VS(>))1 - [
2p
L M
L2
+ R
X
{f(=))1 + 2 M
Σ
|a:.00 - -x,11
+
(1 (1- -1-
[|[T T.-1ll),
m=1
where we used the following uniform bound on [ - v
3 L
7131 L
[v/().)) v=(=Ill
< I 1| - = -
||, - _4-1|| 1
Em m 1ー ""
B1
i1
nB1L Em
< n G
<
T
||
1 B
1- B1 X
38

---

Published as a conference paper at ICLR 2026
Therefore, ignoring the constants, have the following bounds:
V
S 0
+
nP
On)
1
Σ Bii x4 0
P
2X
M
IV S 0
(17²)
M
III & (
Σ
B - -
E
M
==1
M
II S (n) 1
Bllgg 11
M
M
M
I S
7
V
Σ
[1 =
+ (m)
1 M
Σ BilrT r
2Co
To get the O
bound for the averaged gradients (()))] note that we are left choose and show the following bounds:
small value for
A
2Co
M
E1li, = (172),
(extension of Lemma 3)
TM
t=0
m=1
r = 0(1,
(follows from AMSGrad normalization)
M
Σ
Ella
D. = 0(1),
(see below).
M
t= m=l
For the last bound, we can use similar steps as in Lemma 3, namely
[l II]
Pu 0 Pu E11l + (1 - B1)9: (B1u711 + (1 - 31)9""))l []
(1 3.))31 [[]1uut11111 - 11] + (1 - P1) B(1e - ge [l]
(1 --Pa)(1 - -1:) Σ (11 - (1-Pp)33))
- 9f [1]-
T=0
41 -114-1
11,
-B
El Dt"
E
B1
1
B1
31
1
Ellue 11-1
u -1ll
+
[E 11 ut
1 B
31
31 E u,
1-
B1
r=
(1
pa)
8 g1-7((1 - - - 5.
1=1
3
pn) -1-))))
-
0=0
which has the same double geometric sum structure as (7).
39

---

Published as a conference paper at ICLR 2026
D.2 KEY LEMMAS
Lemma 2. For all T 2 1, we have
1/2
17332
T
1 M
Σ 2 xell2
To +
V m (17
(6)
(1 (1-8) 31-
(1 3)
Proof. Since u 0, unrolling the update rule of momentum, for any t 2 0 we get
Ur = Bu (1 -1- B)g: (1
3 gg.
Using this and the definition of the average iterates, have
3
B1) 1 2 -Bn Σ
24 22 1 B (2t 12-1)
3r 8 gr.
Using convexity of squared norm function and letting St have
def
21 3r-
1-get1 1-9
for all 2 0, we
|| xxel2 = = 32
Σ
t
8
r gr
2
S
3² 16
Σ
8
11g 11 2
172 t 1- B
Σ
g
11g 112
8t.
r=0
T=0
r=0
Summing over the iterates yields
T-1
12230 T
Σ B xell?
E1 1|
1 t=
- 8
32
T
T
8
1- 8
88
8T
11
1
B2 T-1
Σ g 1|
(1- (1-
12
M
17 T-1
Σ E
Σ 9
(x
+
Σ
fm (ITT)
(1-
(1 B3
M
1282
17
M
Σ
E 11gh
V (2 mm |1
Σ
E
Σ V Ff (I:T)
11-8)2M
(1 8)2
M
7=0 m=1
m=1
B2
32
To² +
E
V fm (xx)
(1-3PM
(1- 3
M
m=1
Lemma 3. 2f L2 < 1, then
T-1 1 M
Σ
E - - 13 < 1B2 -1)bb
E
+ + + +302)
MT
T ビ=
1= m=1
where
きb 4(1 pp)
(1- (1-3))1-pa))
12
1- p1-
40

---

Published as a conference paper at ICLR 2026
Proof. Let expand the term 2141 =2+1ll using 2/41 probabiliisii update rule:
E E
2
22 ||2
Pr 0+ (1 Pw) E1| nuu (x m744 I|
(1 - -Pz) Bilar Li 17(2" u/ )1
S (1 P=)(1 + s)El| x 112 + (1 Pa (1 + 1/a) u²'1l2
rf(1 pb) + 1/5) Σ ((1 p1) - 11
where 8 will be chosen later. Next we expand the term Bl|u, - 11 using " probabilistic update rule:
M
E| U4
11 = Pa 0 10+( (1 Pu)
(Bu + - -3)0 - -(3um + (1 -8)
M
m=
(1 Bl] (1441 2/th (1 (3) 9;-
< (1 P..) 3a) (ue-1 - - ||1 + (1 - Pw) (1 3) g.th 11
1-1
(1 - Pw)(1 3) Σ ((1 - -
E 1/
==0
1 8 1-1
(1 pw)) F 1,111
3 T=
Denote Qi = =(1 (1 s) and Ø2 = (1 - -P.)3. Combining the previous two bounds, we get
M M
E. x: 11
< 21 (1 1/b) Σ (11-p)(1+ -7 1
Σ
E Ellu, 2 1|2
(7)
M
M
s 17(1 (1+)1/1/)) Σ
((1 P1) + =))²²=-
((1 - P.. Pw) "Ellg, -" 1|?
M
B
T=1
m=1
V=0
M
1 B
T2(1 (1)
Σ
1 1-7 1/
T-"
Ellg 9" ||
8
M
==v
m=
3
11 )1 1/5)
82
E Eligo
M
T=D+
= (2 (1 + 1/2)
1
[-1
02
02
M
Elgg
g gmrl
3
92
M
V=
m=
t-1
M
(1- (1) B)
02
E gg
9.¹h 112
91
Q2
M m=11
0=0
41

---

Published as a conference paper at ICLR 2026
Next, we bound the gradient term above.
M
M
K
B
TH
9ell2
E
91
91
M
M
M
1=1
M
M
2
E
9t
V (m
(0t
Vm (z(")
K
M
m=1
m=1
M
M
1
+
V fm (x)")
V fm (x
M
M
m=
m
M
M
2
(Lemma 4)
El19%
V fm (x, 11
2E
V fm (mx(x()))
M
M m=1
ma
12L
Σ
E1
E
1.n11 +6B2 ()) 6G
M
m=l
12L
20
+
E. 24
+ 6(B² ((=))] 6G x,"'
M
Again, plugging this bound to the previous one, get
xt 1
MT
t=
T
M
E Blaa x
MT
m=1
T
M
<
Elgg gT
T
Q1 Q2
M
0=0
m
91
1-T
42 T-
1
EE|gg 17
T
Q1 Q2
M
t=
m==1
T-1
Q1 o )
92
o2 -T)
M
E 1gr
g 112
T
11 02
1 1- QI
-
M
m=1
T-1
1 M
<
6 0
1 91
42
Elgg, 9. 1|?
T
41 0
1
42
M
7=0
m=1
T-1
Bllgr 1/
(1
(1)(1
92)
M
Now, let us optimize the factor
(1 -1- pa (1- 1/-) (1=))) 3)00011-1111-00000 3) -1p)
(1 (1-p))(1+/)) 1/4) (1-
91)(1 92)
(1 (1- (1 pw (1 s))(1 Pp))))
11 +)) 1- 1-
42

---

Published as a conference paper at ICLR 2026
by choosing optimal value for introduced earlier. By the first order optimality condition, find VI-P. Hence, the minimal value of the factor is
that the optimal value is s
Ps
(1 - (1- - -Pu)
(1 91 92)
=
(1 vI Ps) 1 31-Pa)
(1 P1] VI- Pz)²
(1- 3) - -Po)
vI - pa) P1)) + vI pp))
1 P3,)
(1 P-))((1 + v1 p=.) (1 B1 Pm))
p3
P
4(1 Pa)
(1- B) - -Pu) tb.
12
1 (1 Pu)
Continuing the chain of bounds
T
12
MT
m==1
4
E 1gt 9 112
T
T-
121,2
M
$
E x1
x
+ 6( 1)) + 20 + 6G2
M
m=
T-1
M
<
12
1 1?1
TM
1=0 m=1
+ (B² - 1)p
+ +3G²).
E
T
Assuming 22 $ 1/2 and reordering the first term in the bound, we arrive
Σ
Tt
21"11 < 12m² (B²
-1) T
Σ 0=
+ +3G).
Lemma 4. Under smoothness and bounded heterogeneity assumptions and 3, we have
M
K
2
67
M
fm
Σ
Σ
|0 - 1.rll1 +3(B² (x))) +3G².
M
K
M
1=1
m=1
43

---

Published as a conference paper at ICLR 2026
Proof. The bound follows from simple algebraic manipulations and Jensen's inequality.
M
N (a()))
Vm
(x7)
K
K
M
N Vm - (x)) + ( + Vs(a))
S((x))
K
K
3
Σ
M
[ (x:" (m)
3
V (ax) -/((=)11
K
K
m=1
M
K
3
+
()(())
K
K
m=1
1=1
3L
|,2 xell²
3
3L
Σ
ry x
K
K
K
2L2
Σ
M
|x m
xz
+
3
(m:)) {((+))1
K
K
m=1
m=1
6
M
"| xell 2 + 3G + - 1)]Vv(())
K
m=1
E CONVERGENCE ANALYSIS OF DES-LOC-ADAR (HIGH-PROBABILITT BOUNDS)
For this section, we refer to Algorithm 1 as DES- DES-LOC-OPT KI..., KN). Let us consider the second algorithm DES-LOC-OPT(A, K,..., K) with K lem lem[Kg Kg, KI...., KN)...,,,.. These two algorithms have a property that they both fully synchronize, i.c., all states and current iterates are the same, if = r for some r E N.
Commonly, the analysis of DES-LOC-OPT(AR K, K..., K) proceeds in the following way. In each step. construct an ideal update as if you were running DES-LOC-OPTIC 1....,1) 1) using virtual iterates (see the proof in the prior section for the example of analysis with virtual iterates), and bound the drift from this idealized scenario. For the case of DES-LOCC-PP(A (K, K,... K), the bound typically depends tion. Below, we show that the drift of ( OPT(K, K ... K), since OPT(K,, K1,... KN) synchronize more often. Therefore, the con- vergence rate of OPT(Kz, K1,... Kx) is not worse than the convergence rate for DES-LOC- OPT(K, K..., K) its analysis also applies to OPT(K,, K,.... Kx), i.e., all final upper KI,
the distance of the current iterate from the last full synchroniza-
KN) is not larger than DES-LOS
bounds derived for DES-LOCK -LCC-OFO-PP(K K, K,..., K,...,K) are also valid for OPT(K,. ..,..., Kx). For instance, a typical way estimate drift to have an assumption of type || shall U for all E 21,2..... k.. 8o = x0 = DES LOC-LOC-,,,, K, K..., K), we can simply bound
k}. and n 21, M). where s² is some state on client n at step i and
M the synchronized state. Then, drift is usually expressed as || - sull- For
|| soll
Σ
8
5
18
S KU.
44

---

Published as a conference paper at ICLR 2026
......: KN), obtain the same bound, where for simplicity For DES-LOC-OPI .i..... assume that s is synchronized every Kg steps and & E {Ko + ,,...,,..6.. 2K,}.
k
18%
sol1
8
+ SK,
80
1== K,
M
|1
8
M
1=
k
M
K2
||
M
1= R +1
m=
< KU.
In a more general case, we would apply the above recursively. Such type of adjustments is the only requirement to adapt analysis of DES DES-LOCKS LOC-OPT K,, K...,,KK to obtain the same rate for DES- LOS-LOC-OPT(K, K for the type the analysis described above.
We do not claim any novelty for this analysis. We mainly include these results for completeness, to showcase that our method converges under different settings. The main theoretical results showing that some of the optimizer states can be synchronized less frequently are presented in the prior section above. We would also like to highlight that this result might be relatively weak and not tight since only show that DES- same worst-case convergence, but DES LOC-OPT K ...., K) requires less communication than DES-LOC-OPT ....., KN) under this analysis, which not the case in practice nor in the analyses presented above.
K,... ....., K) and DES DES-LOC-OFTI K2, K1,-.., KN) have the
Finally, detailed inspection of the analysis of DES-LOC-Adan (K, K, ...... Cheng & Glasgow (2025) reveals that this analysis satisfies the above criteria. Thus, we can directly apply their results under the following assumptions and preliminaries.
We aim to optimize a neural network under the loss function f
min fz) == E< ZER
(8)
using M workers, each of which has access to the stochastic gradient of f. VF(x;; with g independently drawn from the data distribution D. We define the auxiliary sequence,
91 I- 3,
ot
if mod K + -1, otherwise.
m
31
(9)
31
31
4+1
1-3,
where, Tz+1 Em [1 We also define 2e+1 Em Em
We make the following standard assumptions.
Assumption 5 (Lower-boundedness), f closed, twice continuously differentiable and
inf fz) =: f(x.) =: f. -00
Assumption 6 (Smoothness). There exists some set Ω C Rd and L > O, such that for any z, y E Ω
[VV((>) V((iII < L|[x 11lx-w1 -vll,
(10)
|V/(=)] < (x)((x))))))))))))))))))())))))) f.).
(11)
Assumption 7 (Bounded a-moment noise). There exists some set C Rdd a 2 and constant vector 2 0 such that for any S
RE (F(x; vf(x)) 300
(12)
Let 0z |0|| max, ||||| (of +... + +z2) 1/2
Assumption 8 (Weak convexity). There exists constant T > 0 such that f is T-weakly convex, i.e., for any x, gd
(Vf(z) Vf(y), - v) 2 - 1 yl?
(13)
f(y) (V)= 2 || v11 v²f(x) 2/= -TIa-
(14)
45

---

Published as a conference paper at ICLR 2026
Based these assumptions, the DES-LOC-Adan variant of Adam converges stated in the following theorem.
Theorem Let the Assumptions 5,6,7, 8, hold = conv (R, (St)), where S2 == for f(2)
f. S 4A}, (Ro) = {ar E R By ||æ- ylll S Ro Ro = SDL* Kiem = lem (a,, Kus K.h. and the same assumptions in Theorem D.3 of (Cheng & Glasgow, 2025), then with probability 2 8, 1OC-Adan yields,
2iu
Kiem
TA
LA
L
(LAr)
LAg K K R
[V =0
Klem R
R
Klen
V
M MKiem R
K R$ lem
Proof. The above corresponds to Theorem D.3 of (Cheng & Glasgow, 2025) for DES-LOC-- Adam (Kiem..... Klem).
Note that for sufficiently large R, the leading term in the rate is:
Kion
(v
LAO2 M KKiem R
[V
(15)
K lem
In both cases, Theorem 5 shows that for the convergence bounds to hold for the high probability analysis of DES-LOC- LOC-LLLC synchronization needs to be finite lem
F DERIVATION OF EQs. (1) AND (2): MAXIMUM MOMENTUM CHANGE WITH CLIPPING
Lemma. Let the gradient each step satisfy l|gelloc I p for some constant p 0. Assume the firt-mmomentum state in Adam is initialized 2 = 0 and updated by
B u + (1 - B1)g.. 31 E 10,
(16)
Then, for all t 2 O, the momentum is bounded and satisfies
| p. and ||u+ uello - 2p (1 11--39 311
AK 2 1.
(17)
Proof.
STEP 1: BOUND ON |luelboo We first show by induction that the momentum is always bounded by
Base Case (t 0): Since u-1 0, we have:
| loc = ||,3u_1 B1)gollo S (1-B1)l990) lox p.
(18)
Inductive Hypothesis (I.H.): Assume ||uelloo S p for some 2
Inductive Step (t + / + 1): Then,
|M = 11314 + (1-11)9t11k
(19)
- B1 Illuuel ooo00 + - 11)
(20)
I B (1 31)p = p.
(21)
Thus, by induction, we have the desired result:
|| S P. 11 0.
(22)
46

---

Published as a conference paper at ICLR 2026
STEP 2: BOUND ON ||u+ uelloo. Now we bound the change in the momentum over K steps
explicitly. Unrolling the recursion, we have:
K = BBf us (1 B1)
Σ
3
19 K-k-
(23)
k=0
Subtracting in from both sides, we obtain:
U++ K 24e (3f 1))4 + +1 (1-31)
Σ
g
(24)
Applying the triangle inequality gives:
|14 Melloo S 111 B a1luulll +11 B1)
Σ
(25)
Using the bounds lluelloc S p and 119ell00 S p. we simplify to:
K-1
||uc+K- -
s (1 5 3 P + (1 31)p
Σ
B
(26)
k=0
The geometric series simplifies as:
K-1
1 1- 3k
Σ
(27)
31
Substituting this back into the expression yields:
| uell S (1- 3k p (1 (1- 2p(1 33
(28)
Thus, the momentum difference satisfies:
|lu+K ueloo S -
AK 2 1.
(29)
Applying the exact same logic to the second momentum u. with 31 replaced by B2 and the bounded gradient squared term [19: O ge ll S p², immediately gives:
SECOND-MOMENT BOUND.
|| - velloo S 2p 3k
(30)
This completes the proof.
G WALL-CLOCK TIME MODELING
Understanding the practical benefits of our proposal beyond the theoretical aspects and empirical convergence curves is crucial. This section addresses the practical implications of adopting our method for training state-of-the-art (SOTA) large language models (LLLMs) in large-scale distributed training infrastructures. The most critical metrics are based on total wall-clock time, communication time, and resource utilization, i.c., how much of the wall-clock time is spent using the compute available instead of waiting for the communication to complete. We provide the following simplified model for estimating total wall-clock time (Section G.1), computation time (Section G.I.I), and communication time (Section G. G.2. that applies to any method based distributed data parallelism (DD?). The notation used here is consistent with that in Algorithm 1. We conclude this section with the results obtained with this modeling and their discussion.
47

---

Published as a conference paper at ICLR 2026
G.1 ESTIMATING TOTAL WALL- TIME
The total wall-clock time for completing an LLM pre-training is based on the number of tokens processed D (dataset size), the model size d (the number of trainable parameters), the number of compute units M (data-paralle//ooall workers), the floating point operations per second 5 that these compute units can perform, the Model FLOPS Utilization (MFU), the average peer-to-peer (P2P) bandwidth B and the latency between compute units. We separate the total wall-clock time discussion into computational time (Section 1.1.1) and communication time (Section G. G.1.).. In our modeling, the total wall-clock time is the sum of computational time and communication time:
Lotal
Eccoppte
comms
(31)
We next derive fcompute and fcomms separately, and then instantiate Lotal for specific training methods.
G.1.1 ESTIMATING COMPUTATION TIME
The total time spent computing T Temmuus: depends on the number of compute units M, their floating point operations per second S, the MFU of the training pipeline, and the total number of FLOPs C that the training pipeline requires. Following the same approach as in Kaplan al. (2020); Hoffmann et al. (2022), the total number of FLOPs required to train an LLM can be estimated as C = 6dD, where d is the number of model parameters and D the total number of tokens (dataset size). Since the MFU can be considered a measure of efficiency, i.e., MFU e [0, 1J. we can estimate the total time spent computing as:
C
6.d.D. D
computt
(32)
MFU S. M
MFU S M
In other words, if the hardware can perform M FLOPs/sec at peak and is utilized at MFU fraction of peak, the training FLOPs C' translate to that many seconds of compute.
In practice, MFU strongly depends on how the pipeline's parallelization locally configured across the workers M. For the sake of fairness in comparisons, we can assume that the per-batch MFU of a tata-parallll worker is the same as the per-batch MFU of a worker in our proposal and other local adaptive methods. Importantly, this holds in cases where either such workers refer to single GPU each worker locally performs more advanced parallelism techniques, such as the ones proposed by Rajbhandari et al. (2020); Zhao et al. (2023).
Resources Utilization and MFU. Theoretically estimating the resource utilization in large-scale training of LLMs is very challenging despite prior knowledge of the number of hardware accelerators (GPUs), their theoretical peak FLOPs, and the total amount of FLOPSs C' required to perform the task available. Following previous well-established proposals (Chowdhery et al., 2023), we leverage MFU and the theoretical peak FLOPs of the hardware accelerators we used in our experiments. Recent systems research (Shoeybi al., 2019) has shown it is possible to reach 50% of peak FLOPPs even for million-parameter models by carefully combining data, tensor, and pipeline parallelism. This emphasizes that our model's assumptions (e.g., each worker sees full d) can be adapted to those scenarios by treating model-paralle -prralel group one worker with higher S and similar MFU. For the sake of a fair comparison, our analysis in this section compares different methods assuming that the local workers operate with the same theoretical peak FLOPs and the same MFU. The results reported in Section G.2 describe how such values were obtained.
G.1.2 ESTIMATING COMMUNICATION TIME
Communication time is the most critical factor when comparing standard data- mata-parallel approaches to our proposal, since the computation time will be the same, given that they train the same model size on the same number of tokens using the same computing infrastructure. At each communication step, the workers W synchronize a set of parameters M, the amount of which depends on the method used. For example, distributed data- aata-parallel synchronization occurs at every batch step on the complete set of gradients produced by the M workers, each exchanging a payload at batch step of Pode. d parameters. In our proposal, the synchronization involves model parameters and optimizer states at different frequencies, making such estimation slightly more complex. Since their time costs simply
48

---

Published as a conference paper at ICLR 2026
add up, treat the parameter syne and momentum sync contributions independently. For instance, parameters are synced every K₂ steps and momenta every Ku, K2 steps, we sum the time for each series of syncs.
Any of such payloads can be exchanged and averaged using bandwidth-eeficienl AllReduce methods, such RingeelReeuuee (Sergeev & Balso, 2018), which scales only with the speed of the slowest P2P link. Given the slowest P2P bandwidth and latency a single communication at timestamp i performed synchronously and in parallel across the M workers, taking total time of:
2P:
1 M
coomms,
+1,
(33)
B
where P3 is the payload size of the communication happening at the timestamp i, which depends on the optimization method adopted as described above.
DDP. In the DDP training approach, each of the T optimization steps to train on tokens requires communicating every step for total training time of:
2d B
-1 -1
fotal,
ccompute + T
1
(34)
M
FedAvg. The approach of the FedAvg method is that of synchronizing with frequency K only the model parameters across the M workers. This, the total training time can be estimated as:
Lotall Fediigg
coompute
+ K
+
(35)
B
M
This optimization procedure will communicate less than DDP when K < T.
Local Adam. Using a local adaptive optimizer such as Cheng Glasgow (2025) with a synchro- nization frequency of K local steps, requires training for a total training time of:
3T K
2d
+1 )
tooall Local Adam
teompute
(36)
M
This means that, as long as 3K T, Local Adam will always take less wall clock time than DDP.
Our Method (DES-LOC), Adopting our proposal LOES-LOC-aE ically, which we shall use interchangeably for the purposes of this analysis) requires synchronizing model parameters x, fist momentum u and second momentum with frequencies kz, K., K,, re- spectively. Assuming each of these sets synchronized independently, we can compose by adding their communication time contribution to the total training wall-clock time, which results:
and MOSS-LC-ADDEE specif-
T
2d B
(otal, /campute
+
+1
(37)
K2
K
K2
M
This means that, as long K. K. take less wall-clock time than Local Adam and DDP.
A
K,
R.
our method will always
Limitations. We critically discuss here the limitations the proposed modeling in order to shed light on their relevance when it comes to deploying such training algorithms in real-world scenarios.
First, our modeling approach adopts constants for several system components, such as computing capabilities and interconnects. In particular, MFU in the real world always oscillates around some average value depending on the operational performance of high-bbndwidth memories (HBMs), DRAM caches, and processing units in the hardware accelerators. At the same time, the P2P bandwidth and latency between accelerators also fluctuate around average values.
Second, most efficient implementations adopted in the field take advantage of the possibility of overlapping communication and computation, reducing the communication time. Notably, overlap- ping communication with computation can drastically reduce effective communication costs, for
49

---

Published as a conference paper at ICLR 2026
example, Py PyTorch's DDP implementation overlap 95% of the communication (Romero et al., 2022). Our model currently assumes synchronous communications, but could incorporate such approaches by reducing the effective i or B impact. One extension could be adding parameter E [0. 1] representing the fraction of communication time that is not overlapped, so total time would recover the fully overlapped ideal (communication is entirely hidden by computation), and a = 1 is the current no-overlap assumption. This would keep the model Tramework-agnostie but allow tuning to specific training setups.
per step is i fcompute Crlcomm- Setring
Techniques in Rajbhandari et al. (2020); Zhao et al. (2023) complement our analysis by reducing memory usage and communication volume, effectively scaling down payload P. or increasing MFU. Our approach focuses on synchronization timing rather than data partitioning; combining our method with fragmented updates (e.g., ZeRO) could further improve wall-clock time.
Despite limitations, our model was designed so that any gap with real-world performance evenly affects all methods analyzed, assuming thoughtful implementation. Thus, results Section G.2 illustrate potential improvements from adopting DES-LOC and our model can help practitioners estimate performance at larger scales.
G.2
MODELING RESULTS
Figures 23 and 24 analyze the wall-clock time, communication overhead, and GPU utilization of DES-LOC compared DDP, Local Adam, and heuristic baselines for training 1.7B model. By setting synchronization periods as K2 256, K2 768, K2 1536, DES-LOC significantly reduces communication and improves GPU utilization relative Local Adam (K 256), closely approaching the efficiency of heuristic methods, especially in bandwidth-constraine settings.
101
DOP
DRS-1.C
Lodal
1040
FAVG-OPP
Wall-Clook time (s)
10a
10F
107
108
10
10
10
10-1
10
10
10a
10a
10a
10a
P2P Bandwidth (Gbits/s)
Figure 23: Estimated wall-clock time for training the 1.7B model with DES-LOC (K₃ 256, K, 768, K, = 1536), compared to Local Adam (K 256), DDP, and Federated Averaging with persis- tent optimizer states (FAVG+OPT, K = 256). At low bandwidth (< 10²), all communication-effiiie methods substantially reduce wall-clock time compared to DDP. DES-LOC closely approaches the maximum efficiency of FAVG+OPT, significantly outperforming Local 1 Adam, which synchronizes maintains stable and convergent training behav-
all optimizer states frequently, Morcover, ior (Fig. 5). At high bandwidth (> 103), DDP becomes competitive or preferable.
50

---

Published as a conference paper at ICLR 2026
124
PAY2
100
C4
12"
10
3V4
201
107
107
107
10
10
10-7
00-
107
100
FSP Bandbotth (1Rrk)))
(a) Communication costs.
(b) Compute utilization.
Figure 24: Communication overhead (a) and GPU utilization (b) for training the 1.7B model with synchronization periods K₂ = 256, Ku = 768, K, = 1536. DES- reduces communication costs by 170x compared to DDP, outperforming the 85x reduction achieved by Local Adam while FAVG+OPT, T, communicating only parameters, achieves a theoretical maximum reduction (256x) ) The improved communication efficiency of DES-LOC translates to higher GPU utilization at low bandwidths (< 10²), significantly improving over DDP and Local 1. Adam.
Takeaway: By synchronizing optimizer states less frequently, DES DES-LOC enhances GPU utilization and total wall-clock time compared to DDP and Local Adam, especially under bandwidth constraints.
H CHECKPOINTING vs. PERIODIC STATE SYNCHRONIZATION
A natural question is whether simply checkpointing local optimizer states suffices for dealing with variable or elastic compute. This approach is inadequate for two reasons. Quality: Initializing new workers from single stored state yields worse convergence shown in Fig. 4.(c) (~15% higher perplexity in our tests in the follow-up round) compared to DES-LOC's averaging. Elasticity: When the worker count changes from N to M, checkpointing lacks a principled mapping, forcing arbitrary state duplication or sub-selection, which either amplifies outliers or discards information.
more principled ad-hoc strategy is averaging the N existing states. To formalize the comparison, let the local states 0g be i.i.d. variables with mean (the ideal global state) and variance (local drift). The statistical risk is the Mean Squared Error, Risk(A) E[( {
The random selection estimator (checkpointing), Prand Gr. has risk equal to the full sample variance:
Risk(prana o²
(38)
The averaging estimator, Pavg 1/2 - 0., reduces this risk by factor of N:
Risk(pavg)
(39)
N
Averaging is thus more robust to the divergence of any single worker. However, even this principled ad-hoc approach underperforms DES-LOC, The crucial distinction is that DES DES-LOC builds periodic averaging into the training loop, treating it core mechanism rather than an external recovery tool. This proactively constrains the variance of local drift (α²) throughout training, ensuring all workers remain in low-variance consensus state and making the system inherently robust to elasticity.
I CHOOSING SYNCHRONIZATION FREQUENCIES
Our results suggest a simple and principled rule-of-thumb for setting the synchronization periods (K) for model parameters and optimizer momentum states, grounded in the dynamics of exponential moving averages (EMAs). This methodology provides actionable defaults for practitioners seeking to balance model convergence with communication efficiency.
The core principle is that the synchronization frequency of any given optimizer state should be based on its empirical half-life- the time horizon over which its EMA "forgets" half of its past information.
51

---

Published as a conference paper at ICLR 2026
This ensures that states synchronized before they drift too far apart, maintaining training stability. For a state with a decay rate approaching I, the half-life can be calcualted as:
ln2
1/2
1 1-B
Based this, propose the following two-step methodology for setting the synchronization periods Kr (for parameters), Ka, (for first moments), and K, (for second moments).
Parameters First (Kg). The synchronization of model parameters is paramount to training quality. The period K₂ should be chosen to match the end-of-training quality of fully- synchronous DDP at a target step budget while still materially reducing communication. In practice, starting points like K2 16 K2 32 effective shown in and Charles et al. (2025). Parameters should always be synchronized at least as frequently any momentum state.
work
Momentum by Half-Life (K K.). For any optimizer momentum state with decay rate 3, its synchronization period K should be set near its calculated half-life, i.c., K f1/2- For common optimizers like Adam ADOPT with well-tuned decay rates 81 and 8g. this simplifies to setting the sync periods for the first and second moments as: K2 = 1-81 K2 1
lu
and
Following this heuristic can yield a minimum 5x reduction in communication cost over DDP for K2 2 16, significantly decreasing wall-clock time while achieving convergence speed and final model quality comparable to DDP.
J CRITICAL BATCH SIZE AND REGIME POSITIONING
To formally contextualize the regimes where DES-LOC. is most beneficial, we begin with the statistical properties of gradient estimation. Let the true gradient over the full data distribution for loss function L: : R + R be G()) (O)) In practice, mini-batch of size B provides an estimate, Ges(
The variance of this estimator scales inversely with the batch size:
cov(Geu (8) 1 x(()
where Σ(0) is the per-example gradient covariance. This relationship establishes fundamental trade-off: smaller per-worker batch sizes B result in higher-varianee or "noisier," gradient estimates.
Analyses of large-scale training have formalized the concept of critical batch size, Bent (McCan- dlish et al., 2018; Zhang et al., 2025). This represents the point at which the benefits of increasing batch size begin diminish.
When the batch size B < Bem, the gradient estimate Gea(0) is noisy, and increasing B yields substantial improvements convergence speed per step.
When B Berit the gradient estimate (o) becomes highly accurate estimate of the true gradient G(O), and further increases to B provide negligible returns.
In modern distributed settings with N workers, the goal often to operate at a compufe-optimal global batch size (G N x B), which is typically near B for the given model and training duration. In massively parallel environments where that the per-wor worker batch size B = G/N becomes small. Consequently, large-scale, compute-optimal training often forces individual workers into a regime where < Beit thereby exposing them to high levels of gradient noise.
large, maintaining an optimal necessitates
For local-update methods (e.g., Local SGD, FedAvg with local optimizers), this high-variance regime particularly challenging. Each worker performs multiple optimization steps using its noisy gradient estimates, causing its local parameter replica 6, to diverge from the other workers. This workrr drift can destabilize training and severely degrade final model quality.
DES-LOC is designed to counteract this divergence precisely in the high-noise, compute-optimal regime. By periodically synchronizing not only the model parameters but also the optimizer states
52

---

Published as a conference paper at ICLR 2026
(e.g., Adam's momentum and variance accumulators), DES-LOC acts powerful consensus- enforcing mechanism. This periodic averaging reduces the variance of the distributed state, effectively dampening the destabilizing effects of high-vate varianeee local gradients and materially improving stability and final model quality. This allows the system to retain the communication savings of local updates without succumbing parametric drift.
While the benefit of desynchronized momentum syncing may shrink in very-large-bater regimes where B Be (as local optimization is inherently more stable), DES- LOC remains highly attractive due to a combination of other robust properties:
Provable Convergence: maintains strong theoretical convergence guarantees under local updates.
Graceful Qualit-Coommuniatton Trade-off: The synchronization frequencies (Kz, Ku, K2) provide an explicit and effective mechanism to navigate the trade-off be- tween communication cost and model performance.
Inherent Elasticity:
The method is fundamentally robust to dynamic changes in the number of workers. The periodic state averaging provides a principled, low-variance mechanism for initializing new workers, scenario where naive checkpointing and state redistribution underperform significantly.
K EXTENDED RELATED WORK
Federated Optimization. The DES-LOC framework, as mentioned in Section C.I, belongs to the broader field of federated optimization. A foundational algorithm in this field is FedAvg (McMahan et al., 2017), which established that a central model can be trained from decentralized data by averaging the model weights from clients that have performed local training steps. These findings were later generalized by Reddi et al. (2021) through the FedOpt framework, which re- frames the training loop as a bi-level optimization, allowing the server to employ an optimization strategy more complex than simple averaging. Consequently, Reddi et al. (2021) demonstrated the instantiation of algorithms like FedAdam, FedYogi, and FedAdagrad, which achieve strong empirical performance and provide nonconvex guarantees (Kingma & Ba, 2015) by substituting the server's averaging step with a corresponding optimizer. In a related approach, Hsu et al. (2019) incorporate server-side momentum to improve the stability of aggregation, particularly when data skewed. A primary challenge in federated learning involves heterogeneous data distributions, where clients hold non-IID data partitions. To address the problems arising from this heterogeneity, algorithms such as FedProx which applies a proximal regularizer for stability (Li et al., 2020b), and SCAFFOLD, which uses control variates for robust convergence (Karimireddy al., 2020b), have been developed. Likewise, FedNova addresses objective function inconsistencies by normalizing local steps (Wang al., 2020). The Mime algorithm aims reduce the gap between federated and centralized convergence through the use of control variates and server statistics (Karimireddy al., 2020a). Lastly, methods such as Per-FedAvg (Fallah al., 2020) and Ditto (Li et al., 2021) concentrate on personalization to enhance fairness and utility with reduced communication.
Compression of payload. The DES-LOC framework lessens the communication overhead in parallel training by reducing the communication frequency of parameter and momentum states compared to standard data parallel approaches. It is important to note, however, that the communicated payloads-the states themselves-can also be compressed, which would further enhance distributed training efficiency. Specifically, quantization methods can represent precision without loss of model performance (Douillard al., 2025: Kale al., 2025). As an alternative, structured compression can express an update in a lower-rank form, either through SVD-like algorithms (Robert et al., 2025) or by only communicating the fast-moving momentum components (Peng et al., 2024). Sparsification techniques can introduce sparse update structures, which allows for better compression via information redundancy (Lin et al., 2018; Alistarh et al., 2018). Because update periodicity and update compression are orthogonal operations, they are frequently applied together to create highly efficient compression schemes without performance degradation (Douillard et al., 2025; Kale et al., 2025; Wang et al., 2023). Therefore, we anticipate that this would be a fully composable enhancement to the DES-LOG LOC framework, which we leave as a direction for future work.
in lower
53

---

Published as a conference paper at ICLR 2026
L LLM USAGE DECLARATION
As noted our submission, large language models (LLMs) were used throughout to assist with various aspects of this work. Specifically, used GPT-5 and Gemini 2.5 Pro to:
Improve the clarity and flow of our writing.
Find relevant related work that would be useful for our extended literature review.
Assist with plotting code and simple code generations.
Beyond the stated uses above, all work, including but not exclusive the interpretation of related work and results, is our own.
M LIMITATIONS
Limitations. First, while our main non-convex convergence result holds for SGDM, for Adam our anal- ysis uses additional assumptions like bounded gradients and homogeneous data distribution. These assumptions are common in non-convex adaptive optimization. Second, our hyperparameter search was extensive yet constrained to smaller models. Lastly, while our analysis uses Adam/AMSGrad, many experiments use modified Adam (ADOPT) (Taniguchi et al., 2024).
REPRODUCIBILITY STATEMENT
We committed to the reproducibility of our work and provide the code, data- cata-processing scripts, and configurations necessary to replicate the results in this paper.
Code and Environment. Our complete source code is available in the supplementary material. All dependencies are open-source and can be installed using the provided scripts (system_setupp sh, install_env: sh), which automate the full environment setup.
Datasets.
The experiments use publicly available, open-source datasets. We provide the 1m_ wwwpcooruu... sh replicate entire data pre-
script convert_hte hf httpreeet_____tttsee__eeeee ____aattttttt__dds________ processing pipeline, from downloading raw corpora to converting them into the required format.
Experimental Protocol. Reproducing our large-scale experiments requires access significant com- putational infrastructure (e.g., multi-GPU servers), as specified in our documentation. All experiments controlled via well-defined configuration system using YAML files. Key hyperparameters and al- gorithmic settings, such as the synchronization frequencies for our method (fl. looclllsteeaalll 1..paramm waaramtter_scheduueeeeeeeeeeeeee scheduler. (cheduler_kwarg] and the data distribution across workers, are explicitly defined. We include example scripts that execute the main experiments reported in the pape when using the approriate hyperparameters reported in Sections and 4, providing clear path to reproduce our findings.
54


Reviews:
[{"review_id": "w2QOrBklOV", "reviewer": "Reviewer_26GM", "summary": "The paper provides a theoretical argument to synchronize local optimizer states (first and second moments) for fedavg/diloco. Specifically, it shows local momentum needs to be synced; however, it can be infrequent wrt outer updates, and this frequency depends on the momentum coefficients. It improves over vanilla diloco and reduces communication wrt syncing states in every outerstep.", "questions": "1. The method doesn't seem specific to adaptive methods. Would it be better to frame it more generally? Would it work with muon-type optimizers?\n2. Is it correct to say local SGD (ie, fedavg/diloco) is sharing only model parameters? It actually shared the parameter differences (pseudo gradients) and uses them with an outer optimizer. Would it make sense to frame the method as parameter-sharing methods need optimizer states to be synced rather than relating them to diloco type methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The theory-backed argument to have infrequent momentum synchronization is insightful and makes sense. The connection to beta (momentum coefficient) and the synchronization frequency is intuitive. \n2. The results show the benefit of synchronizing momemtum terms with two optimizers and show no degradation wrt to local Adam.\n3. Writing is very clear, and the toy example illustrated the need for momentum synchronization.", "weaknesses": "1. The main theorem is for SGD+momentum, however, the method is proposed for adaptive optimizers. This is a limitation as adaptive optimisers are popular and the frequency of synchronization is derived from the theory for SGDM and applied to adam/adopt.\n2. The contribution over local Adam is marginal as local Adam already established that optimizer states needs to be synchronized and also connected the convergence bound to momentum coefficients. Please clarify the main contributions.\n3. Even with momentum sync, the results are worse than DDP. This raises a question of whether diloco-type methods (weight averaging method) would match DDP (gradient averaging)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9N2NpsCFJ", "reviewer": "Reviewer_KN8y", "summary": "The authors propose DES-LOC, a communication-efficient optimization algorithm that reduces the frequency of optimizer state communication compared to local Adam. The authors prove the convergence of DES-LOC in the non-convex case (for SGDM) and for Adam under weakly convex objectives. The author’s key insight over Local Adam is that the first and second moments can be synchronized at a rate on the order of their half-lives, much less frequently than their parameters need to be synchronized. In experiments pre-training language models, the authors show that significantly less than DDP and up to 2x less than Local Adam while achieving similar final performance.", "questions": "- Are the comparisons to the DDP baseline FLOP-matched?\n- Figure 3 (a), (b), (d) have the same rectangular pattern in the loss curve. Why is this the case?\n\n**Suggestions**:\n- I have trouble scrolling through your .pdf because the figures take long to load. Perhaps subsampling the point used for plotting can help.\n- I have trouble distinguishing different loss curves in your plots. Perhaps smoothing could help.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand. \n- I think experimental settings are well structured.\n- From the math in the main paper, the convergence guarantees look sound (I did not check the appendix). \n- The idea of synchronizing optimizer states relative to their decay half-lives is intuitive to me, and I think it's an interesting direction of inquiry for the distributed optimization community.", "weaknesses": "- My main concern is the lack of comparison to existing methods such as DiLoCo [1] and MuLoCo [2]. Neither method requires synchronizing the optimizer states; therefore, they are trivially more communication-efficient than DES-LOC and have been shown to perform competitively to DDP in practice. I am aware of your experiments in Figure 6 showing a DES-LOC method with Nesterov momentum, but from my understanding, these still synchronize optimizer states. \n- Following from my concern above, since the communication efficiency benefits of DES-LOC are already realized by proposed methods in the literature [1,2], I’m not sure how strong the contribution is. Could the authors comment on this?\n- I wonder if synchronizing the optimizer states in DES-LOC can provide a benefit beyond [1,2]. I would be surprised if it does not, and I believe experiments showing this would greatly strengthen your results.\n\n\n[1][DiLoCo: Distributed Low-Communication Training of Language Models]\n\n[2][MuLoCo: Muon is a practical inner optimizer for DiLoCo]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgRsUryqVs", "reviewer": "Reviewer_cfsD", "summary": "This paper introduces DES-LOC, an optimization algorithm designed for decentralized training of foundation models, with a particular focus on large language models. The main objective is to improve communication efficiency during large-scale training in order to reduce wall-clock time.\nThe proposed method builds upon existing approaches such as FedAvg, FedOpt, and more specifically LocalAdam, where workers perform local parameter updates before averaging the weights with a central server. This approach contrasts with classical Distributed Data Parallel (DDP) methods, where synchronization occurs at every training step.\nDES-LOC modifies the LocalAdam algorithm by decoupling the synchronization frequencies of the parameters and the optimizer states. While LocalAdam synchronizes both every K steps, this can lead to unnecessary communication overhead. The authors argue, and empirically demonstrate, that the first and second moment estimates (i.e., the momentum terms) evolve more slowly than the model parameters themselves, allowing for less frequent synchronization of these states.\nIn practice, the authors recommend using default ratios of Ku = 3Kx and Kv = 6Kx, or alternatively determining these based on the half-lives of β₁ and β₂. The paper also provides a convergence guarantee under standard assumptions commonly accepted in the optimization community.\nExtensive experiments on LLM training show that DES-LOC achieves performance comparable to LocalAdam while reducing communication costs by approximately half. Overall, the paper presents a well-motivated and empirically validated contribution to improving efficiency in decentralized large-scale model training.", "questions": "- What would be the practical relevance of DES-LOC for training scenarios where workers are connected through significantly lower-bandwidth networks?\n- Could the proposed desynchronization strategy also be applied to optimizers that use only a single momentum term (e.g., Muon)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well motivated, and the overall writing is clear and easy to follow. The analysis based on the half-life times provides useful intuition for why less frequent parameter updates do not negatively affect model training.\n- I particularly appreciated the qualitative analysis of the upper bound on model convergence (Theorem 1), which effectively shows that DES-LOC enables the use of a larger learning rate in practice.\n- The authors’ claims are well supported by extensive experiments, including analyses of rate of change, convergence behavior in practical LLM training scenarios, and full model training comparisons (in wall-clock days) with DDP and other baselines, which are especially compelling.", "weaknesses": "- While the method is supported by both theoretical analysis and experimental results, its practical novelty is somewhat limited. The main contribution lies in decoupling parameter updates from optimizer updates, which constitutes a relatively minor modification of the existing LocalAdam algorithm.\n- The authors did not conduct experiments under very low-bandwidth conditions. Demonstrating the method’s effectiveness in scenarios with limited network capacity (e.g., 1 Gb/s or 10 Gb/s) would further strengthen its practical relevance.\n- The title of the paper refers to “foundation models,” but the large-scale experiments focus exclusively on LLM training. Although these results are already impressive, the authors could either adjust the paper’s scope to explicitly center on LLMs or include an additional experiment involving the training of a vision foundation model to better match the stated scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRVY1Gme51", "reviewer": "Reviewer_JkEm", "summary": "The paper proposes DES-LOC, a family of desynced low-communication adaptive optimizers that assign independent synchronization periods to model parameters and optimizer momenta (first/second). Theoretical analysis shows (i) convergence for SGDM in non-convex settings and for Adam in weakly convex settings, and (ii) that more frequent momentum sync allows larger stable step sizes, while high-probability bounds require at least infrequent second-moment sync. Empirically (GPT-style LMs up to 1.7B parameters), DES-LOC achieves 170× less communication vs DDP and 2× less vs Local Adam, with 1.3–2.1× wall-clock speedups (aided by a wall-clock model for larger scales) and shows robustness to worker failures.", "questions": "Could you add a $\\ge$7B measured run?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality. Decoupling sync periods for parameters and momenta, grounded in a half-life/step-size argument, moves beyond Local Adam’s uniform sync and earlier heuristics that lacked guarantees or failed under failures.\n2. Quality. Convergence under standard assumptions with a clear theorem for SGDM and high-probability Adam bounds. Analysis explains why more frequent momentum sync permits larger stable steps and why $\\beta_2$ cannot be left unsynced indefinitely.\n3. Quality (empirics). Solid experimental design with explicit RQs, ablations over $K_x, K_u, K_v$ and comparisons showing DES-LOC matches Local Adam’s perplexity with 2× fewer state communications and drastically less than DDP.\n4. Clarity. Clear algorithm block and figures.\n5. Significance. Communication reductions (170× vs DDP; 2× vs Local Adam) and modeled wall-clock speedups directly target a primary FM training bottleneck (bandwidth), with fault-tolerance a notable practical plus.", "weaknesses": "There is no evidence for a larger-scale model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)