Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Authors: Zachary Charles
Gabriel Teston
Lucio M. Dery
J Keith Rush
Nova Fallen
Zachary Garrett
Arthur Szlam
Arthur Douillard
Venue: NeurIPS 2025 spotlight ()
Keywords: distributed training
communication-efficiency
diloco
parallel training
Primary area: deep_learning
TL;DR: We show that DiLoCo, a method for communication-efficient language model training, exhibits reliable scaling law behavior.

Abstract: As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach (DiLoCo) that relaxes synchronization demands without compromising model quality. However, these works do not carefully analyze how DiLoCo's behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under a fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase a more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for a fixed token budget.

Communication-lffiieel Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Zachary Charles Google Research zachcharles@google.coo
Gabriel Teston Google Search teston@google.cm com
Lucio Dery Google DeepMind ldery@google. com
Keith Rush
Nova Fallen
Zachary Garrett Google Research zachgarrett&googl.. com
Google DeepMind krushdgoogle.c com
Google Research
nfallen@google... com com
Arthur Szlam Google DeepMind aszlam@google.com com
Arthur Douillard
Google Deep.indd douillard@google..com com
Abstract
As we scale to more massive machine learning models, the frequent synchroniza- tion demands inherent data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work [1] 24 develops and analyzes an approach (DiLoCo) that relaxes synchronization demands via periodic synchronization. However, these works do not carefully analyze how DiLoCo's ' behavior changes with model size. In this work, we study the scaling law behavior of DiLoCo when training LLMs under fixed compute budget. We focus on how algorithmic factors, including number of model replicas, hyperparameters, and token budget affect training in ways that can be accurately predicted via scaling laws. We find that DiLoCo scales both predictably and robustly with model size. When well-tuned, DiLoCo scales better than data-parallel training with model size, and can outperform data-parallel training even at small model sizes. Our results showcase more general set of benefits of DiLoCo than previously documented, including increased optimal batch sizes, improved downstream generalization with scale, and improved evaluation loss for fixed token budget.
Introduction
Large language models (LLMs) are typically trained via large-batch distributed data-parallel methods. However, bandwidth and communication constraints can become bottlenecks larger scales due to frequent synchronizations, posing critical challenge to further scaling. As remedy, Douillard al. [ propose DiLoCo (Distributed Don-ummuuniaaiin)) a generalization of algorithms like Local SGD 34 54 and FedAvg (36), which enables training of I.LMs in parallel across "islands" of
compute (such as datacenters connected via wwwdwihdwithh networks) by performing parallel training of models with only periodic synchronization. While empirically successful, DiLoCo exists within broader context of work on communication-effice
LLM training methods [58] 43 56
30).
Due to the breadth of related work, we defer more thorough overview to Section A
Unlike communication-reducee
methods such as quantization and sparsification, DiLoCo funda- mentally alters training dynamics 49). While Douillard et al. [ and Jaghouar et al. 24 show
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

that DiLoCo yields comparable evaluation metrics to data data-parallel training at moderate model scales, it unclear how data-parallel training and DiLoCo compare larger model scales. Moreover, DiLoCo has extra hyperparameters not present in data- aata-parallel training that may be computationally prohibitive to tune at large enough scales. This points to the need for Dil.oCo scaling laws. We focus on two specific scaling laws: (1) predictions for evaluation loss as function of model size and (2) predictions for optimal hyperparameter choices for a given model size (which can obviate the need to perform expensive hyperparameter tuning). In both cases, we are explicitly interested in how these compare to analogous scaling laws for data-parallel training.
Throughout we consider the task of pre-training a model of size N on D tokens. We are concerned with predicting, for both data-paralled training and DiLoCo training. and function of N: (1) the evaluation loss L after training, computed on a held-out set, and (2) optimal hyperparameter settings. One path towards scaling laws for Di would be to view them as modifications of scaling laws for data-pat patalparallll training [26] 20). However, the facets of DiLoCo that are key to its communication- efficiency also make such an approach infeasible. First, DiLoCo operates by training M models in parallel, with periodic synchronization every H steps. The values of M and H depend on the ecosystem of compute available (such as bandwidth across datacenters), and are absent in scaling laws for data-parall aata-parallel training. Second, DiLoCo uses bi-level optimization framework; each model replica performs data-pat iata-parallel training, but upon synchronization we apply an "outer" optimization step [II. This means that DiLoCo has "outer" hyperparameters not present in Pata-Parallel training that cannot be inferred from data-pares paralllllllllll hyperparameter scaling laws.
Contributions. We develop scaling laws for data- patta-paralle training and DiLoCo from the ground up. Fixing the number of tokens to be the "Chinchill--pptmmaa" number of tokens (20), we model the evaluation loss and optimal hyperparameters as functions of model size N and (for DiLoCo) the number of replicas M. We empirically estimate these functions using the final evaluation loss attained by models trained with both algorithms for varying hyperparameters (including learning rate, batch size, and "outer" learning rate for DiLoCo), model sizes (varying N over model sizes ranging from 35 million to 2.4 billion parameters), and numbers of DiLoCo replicas M.
Our scaling laws predict that in many settings, the more communication-efiin actually yields better evaluation loss than data-parallel training for the same token budget. Utilizing our scaling laws to predict the hyperparameters for Dil.oCo, we tested these predictions when training models with 4 billion and 10 billion parameters. The scaling laws proved accurate, with DiLoCo with M = 2 replicas outperforming data aata-parallel training as predicted, even while reducing total communication by a factor of over 100.
DiLoCo algorithm
We show that DiLoCo incurs variety of benefits in comparison data-parallel training, including (1) increased optimal batch size, allowing for greater horizontal scalability, (2) greater reductions in evaluation loss as model size increases, and (3) significantly less wall-clock training time. One potentially surprising finding: DiLoCo improves training even when communication is not a bottle- neck. DiLoCo with M (an enhanced version of the Lookahead optimizer
(22) does not reduce communication but achieves lower evaluation loss at all model scales. It is also more robust to larger batch sizes, greatly reducing wall-clock training time.
2 Preliminaries
Table 1: General Notation
Table 2: Algorithm-Specific Notation
Symbol
Meaning
Symbol
DatalParallel
Dil.CCo
0
Model weights
Learning rate
Inner learning rate Outer learning rate
N
Model size
L
Evaluation loss
r
B
Batch size
Global batch size
T'
Training steps
M
DiLoCo replicas
D
Token budget
H
Synchronization cadence
C
Total FLOPs
While we fix H 30 for these scaling laws, we provide extensive ablations the role of H in Section

---

Throughout, we let & denote the model parameters. We let ((t) denote the model at step t. Since DiLoCo operates on M parallel model parameters, we will use subscript notation 6n to denote the m-th model. When there is no subscript, the parameters are assumed to be replicated across all DiLoCo replicas. For batch of data J, we let [(0, (f) denote the loss of f on the batch of data.
DiLoCo [T is a technique designed for training models in the presence of communication constraints. It is motivated by the training of large models across devices that are not all connected by low-latency bandwidth. To avoid incurring latency costs, DiLoCo trains M models in parallel (ideally, training each one with co-located compute connected via low latency bandwidth), only synchronizing the models every H steps. This is similar to the FedOpt algorithm used in federated learning [47]. but with the important difference that the replicas maintain their inner optimizer state across rounds.
Dil applies a bi-level optimization framework across multiple models: each Doo replica has its own model of and there is a global model ((0), At every step, each replica takes an inner optimization step (InnerOpt). Every H steps, each replica computes the A! = ((t-H) 0 the difference between the replica's 's current model and the most recent global model. We average these differences across replicas, resulting in A which we refer to as an outer gradient. We treat this as gradient estimate of the outer mode H and an outer optimization step (OuterOpt) to the outer model ((t-H), This yields updated outer model g(t) which is broadcast all replicas and set as their current inner model. We give full pseudo-code for DiLoCo in Algorithm
Throughout, we perform model training via dis- tributed data-paralled training (Data-Parallel), and DiLoCo. In Pata-Parallel, at each step we distribute a batch of tokens across work- ers. We then compute batch gradient and perform optimization with a learning rate of y. In DiLoCC, each step i, take global batch of tokens of size B consisting of B/S sequences each of length S. We partition the B/S sequences across the M DiLoCo replicas, so each replica receives B/SM sequences, each of length S. Thus, the global token batch size is B, but each DiLoCo replica uses a local to- ken batch size B/M As in Data- Parallel, each replica computes batch gradient and applies an inner optimization step with learning rate y. Unlike Data-Paralle DoO does outer optimization (on outer- cnter-gradient computed in parameter space) every steps with learning rate 77.
Algorithm 1 DiLoCo
Require:
Loss function f(0,x)) batch size B, number of replicas M, synchronization ca- dence H. initial model weights O("), data
shards (Di....,Di))
Require: Optimizers InnerOpt and OuterOpt
1: Vm, 0 ((0)
2: for step t = = To
3: parallel for replica m 1. .. M do
4:
Receive a batch I' ~ Dm of size
B/M
5:
Vos(on ,mm
0 +- InnerOpt(Ofr 19
6:
7:
end parallel for
8:
if t mod H = 0 then
A!f) + g
9:
Att) + M ΣM A (t)
When comparing Data-Parallel and DiLoCo, we use the same model size N and token budget
m=1
m
g(t) OuterOpt (0% (t-H) H) A!)
11:
D. When computing evaluation loss L on some held-out set, for Data-Parallel we use the current
12:
Vm, 0.! + 0(1)
model, and for DiLoCo we use the most recent global model. We summarize algorithm-independde notation in Table and aggorithm-speciffe notation in Table An important comparison is Data- Parallel versus DiLoCo with M While similar, they are not identical as DiLoCo with M = uses outer optimizer step with optimizer OuterOpt, which is often set to SGD with Nesterov momentum
3 Experimental Methodology
Model architecture. We use a Chinchilla-style decoder-only transformer 20) As suggested by Wortsman et al. (60) and Jaghouar al. 24 we use QK-LayerNorm to reduce sensitivity to learning
Note that A (1) is generally not gradient any function, as it can evince con-conservative dynamics [
*This is similar in spirit to the fast- and slow- www.mmumuuu steps in AdEMAMix [41]. but yields different training dynamics since uses gradient estimate computed by linearizing across multiple training steps.
3

---

Table Model details, including size, number of layers, layer dimensions, and token budgets. For the larger models (4B and 10B) we use scaling laws to predict optimal hyperparameters, rather than performing extensive hyperparameter tuning.
Model Scale
Transformer
Attention Heads
QKV
Hidden Dimension
Token Budget
Hyperparameter Sweep
Layers
Dimension
35M
6
512
2,048
70M
90M
12
768
3,072
1.8B
180M
12
16
1,024
4,096
3.6B
330M
15
20
1,280
5,120
6.6B
550M
18
24
1,536
6,144
11B
1.3B
24
32
2,048
8,192
26B
2.4B
30
40
2,560
10,240
48B
4B
36
48
3,072
12,288
80B
X
10B
64
4,096
16,384
200B
X
rate. We also use z-loss regularization 8 We use a vocabulary size of 32,768: 32,000 in-vocabulary words, and extra tokens for BOS and out-of-vocabulary. We pack multiple sequences into each batch, with a sequence length of 2,048 throughout. We pre-train a family of models, varying the number of transformer layers, number of attention heads, QKV dimension, and fced-forward layer hidden dimension (see Table 3). We use the Chinchilla-optine token budget (20) unless otherwise noted. We do extensive hyperparameter sweeps on all models except the two largest (4B and 10B).
Datasets. Unless otherwise noted, we use the train split of the C4 dataset for training 45 We report evaluation metries on C4's held-out validation set. We compute downstream zero-shot evaluation metrics on tasks: HellaSwag [1], Piqa 44, and Arc-Easy [9] In overtraining ablations (Section we use the Dolma dataset instead C4, to avoid doing more than a single epoch of training.
Optimizers. We use Adam W 32 as the optimizer for Data- Pata-Parallel and inner optimizer for DiLoCo, with 31 = 0.9, B2 0.99. Following [59], we use weight decay parameter of T where T' is the number of training steps. We do 1000 steps of warmup followed by cosine learning rate decay to 5% the peak learning rate. We clip (inner) gradients
norm of We do clip outer gradients. For DiLoCo, we use SGD with Nesterov momentum 55 as the outer optimizer, using momentum of 0.9 and constant outer learning rate. Unless otherwise specified, we set H 30.
Implementation. We use modified version of NanoDO 31 that uses DrJAX 49 to parallelize training steps across replicas. We use 6 representation of model weights and gradients.
Idealized wall-clock time. For each experiment, we compute an idealized end-to-end wall-clock time for training. Our model assumes that are training model across multiple datacenters. Within a datacenter, we have bandwithd network. Across datacenters, we have high-, medium-, or low-bandwidth network. For details on the idealized wall-clock time, see Section
Scaling law experiments. We perform comprehensive hyperparameter sweeps for Data-Paralae and DiLoCo on models ranging from 35M 2.4B We sweep over the learning rate 7 using integer powers of /2 and batch size B using powers of 2. For DiLoCo, we train using M 1,2,44 2,4, 8 and sweep the outer learning rate 1) over {0.2, 0.4, 0.6, 0.8, 1.0). We sweep (inner) learning rate and batch size needed until the minimum loss value obtained on an interior point. Using this data, we derive scaling laws to predict evaluation loss and optimal hyperparameters for larger models. We use the predicted hyperparameters train models with 4B and 10B parameters, in order to validate the scaling laws empirically.
4 Empirical Findings
Before discussing our process for fitting the specific scaling laws, we talk about the empirical results and four critical findings that worth highlighting independently.

---

34
32
Reattre Bya Loos (8)
Algorithin
Algerithm
Data Para
DucC H-1
Dilato, Dilats, Dilats,
Luco H-2
28
DuCo Hed
LoCC Hed
Dilato,
24
Model Size
Model
(a) Evaluation loss for various algorithms, as function of model size N.
(b) Percentage difference in evaluation loss, rela- tive to Data-Paralla
Figure 1: We compare Data-Parallel to DiLoCo for varying model sizes N. For all M, DiLoCo improves monotonically with respect to Data-Paralle N increases.
Finding 1: DiLoCo's evaluation loss improves relative to Data-Parallel as N increases. We give the evaluation loss achieved by Pata-Parallel and DiLoCo for each model size N, along with percentage differences relative to Data-Parallel, in Figure [ We see that for all values of M, the percentage difference strictly decreases with N. At N 2.4B, DiLoCo with M = 1 or 2 performs better than Data-Parallel. We validate this by training 4B and 10B models with hyperparameters set via our scaling laws. The results are Table 4 We see that at 4B and 1OB scales, DiLoCo with M 2 continue to do better than Data-Paralle, We discuss the scaling laws in detail in Section o
Table 4: Evaluation results on 4B and 10B models, using hyperparameters predicted by scaling laws. We indicate settings where DiLoCo reaches lower loss than Data-Paralle in bold.
Algorithm
Loss
4B
10B
Data-Parallel
2.224
2.090
DiLoCo, ==1 2.219
(-0.22 %
2.086 (-0.19%)
DiLoCo, M= 2.220
(-0.18% %))
2.086 (-0.19%)
DiLoCo, M 2.230 (+0.18%)
2.096 (+0.29%)
Finding 2: DiLoCo with M = 1 attains lower evaluation loss than Data-Parallel across model scales. DiLoCo with M 1 also achieves higher downstream zero-shot accuracy, as we show in
Figure 2 These plots also show that DiLoCo with M = 1 also exhibits greater stability with respect batch size: doubling or quadrupling the batch size greatly reduced performance of Data-Parallel, but had little effect on DiLoCo, M = 1, as depicted in Figure 2
Finding 3: DiLoCo increases optimal batch size. We plot evaluation loss as function of batch size in Figure 3 Optimal batch size increases when using DiLoCo, and subsequently increases with M. As batch size increases, Data-Paralled becomes worse than DiLoCo with M 2, and eventually, M = 8. We show that the same occurs for downstream tasks in the appendix (see Figure TÓp. This means that Dil.oCo exhibits more horizontal scalability, reducing training time via resource parallelization in addition to reducing communication. To show this, we plot an idealized wall-clock time when training under networks of varying bandwidth in Figure DiLoCo's tolerance for larger batch sizes allows it to achieve comparable loss Data-Paralle significantly faster.
Finding Optimal outer learning rate constant with N. While optimal inner learning rate varies with model size N, the optimal outer learning rate y for DiLoCo is independent of N and depends only on M. shown F for sufficiently large models (N 2 335M), the best for each M is constant. Larger values of M seem to necessitate larger 77. This is consistent with
5

---

135
339
Mote Su4k
Medel SCaA
138
Aggrtder faril
Algether
24
33
2
(a) Evaluation loss.
(b) Zero-shot accuracy on HcllaSwag.
Figure 2: Evaluation loss and downstream accuracy of Pata-Parallel and DiLoCo with M = 1 for varying model and global batch sizes (measured in tokens).
Model Scale
550M
Model Scale 1.30
Model Scale 2.4B
Eral Loos
2.80
2,50
2.36
Algorithm
Data Paral
2.75
DiLoCo,
2.34
DiLoCo, M==2
2.70
DiLoCo,
2.46
DitoCo, M-8
2.65
235
2 2
2
2
2
227
2
2.2
2.00
2
24
2
21D
2
2
Global Batch Size
Global Batchh Size
Global
Figure 3: Evaluation loss of Data-Paralle and DiLoCo function of global batch size (in tokens).
We see similar results for other model sizes, but omit for conciseness.
prior findings that outer learning rate should increase function of number of clients in federated learning settings [6]
5 Ablations
Synchronization cadence. Our experiments above all use synchronization cadence H of 30. We perform an ablation over H for varying M and model sizes N. For each setting, we take the optimal inner learning rate and global batch size from above, and sweep H over {1, 5, 10, 30, 100, 300} and
100M
Buaco Mez
200
550M
550M
kee 1.00
2.4
onoaes a
48
42
22
(sealaed) Kob Time
Joueltee Toa Waldboe
(a) Network with bandwidth of 10
(b) Network with a bandwidth of (c) Network with bandwidth of
gigabits/s and latency 10 - sec- 100 gigabits/s and latency of 10 400 gigabits/s and a latency 10 onds fow-bandwidig
scconds mordium-bandwidthi
scconds (high-bandwidth)
Idealized wall-clock time (see Section when training with across compute nodes connected via high-, medium- and low-bandwidth networks, for varying model sizes. For models up to 2.4B, we also vary global batch size. For 4B and 10B models, we use the batch size predicted by scaling laws (see Section
Figure 4:
and DiLoCo

---

Best Outer Leariing Rate
1.0
8
1
0.8
0.8
0.8
0.8
0.8
0.8
0.9
0.8
0.6
0.8
0.6
0.6
0.6
0.6
0.6
0.7
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.6
0.5
1
0.6
0.4
0.6
0.4
0.4
0.4
0.4
0.3
35M
90M
160M
335M
550M
1.38
2.48
Figure 5: The best outer learning rate for DiLoCo for varying M and model size N. We select the best outer learning rate over (0.2, 0.4, 0.6, 0.8, 1.0), optimizing over inner learning rate y and global batch size B. For sufficiently large models, the best outer learning rate is constant.
1.0
1.0
Outer Learning Rate
0.8
Outer Leaminng Rate
0.8
0.6
0.6
Algoríthm
DiLoCo, M-1
DiLoCo, M-2
DiLoCo,
0.2
0.2
10
30 100
300
2
2
2
2
2
Synchronization
Model Size
(a) Optimal outer learning rate for each syn- chronization cadence. Shaded regions repre- sent the varianee across model sizes.
(b) Optimal outer learning rate for each model size. Shaded regions represent the variance across synchro- nization cadences.
Figure 6: Optimal outer learning rate y as a function of synchronization cadence H and model size N. The optimal increases with H and M (left), but is independent of model size N (right).
n over (0.05, 0.1, 0.6. 1.0.. We first study whether the observation in Section that should be tuned independently of N, holds for other values of H. We give results in the affirmative in Figure 6 We see that across model scales, the optimal learning rate essentially only a function of the number of replicas M and the synchronization cadence H, and is essentially independent of model size N. There some slight variation, though this is likely due to not re-tuning the inner learning rate. Moreover, our results actually show potentially counter-intuitive phenomenon: The optimal outer learning rate increases with H, even though the outer gradients increase in size as H increases. We present additional analyses of synchronization cadence, including how H impacts evaluation loss and compute utilization, Section E.1
Overtraining. Above, we used the Chinchilla-optime amount of tokens for each model size (20]. It is often beneficial to perform overtraining, using more tokens than that [I7]. We conduct ablations on various overtraining multipliers. Given an overtraining multiplier Ç > 1, we train on D = 20NC tokens, so that Ç corresponds the Chinchillaoppimal number tokens. Our results are in Figure 7 Qualitatively, the scaling remains essentially unchanged as we do more overtraining. We did not re-tune any hyperparameter in these experiments. For each model size and algorithm, we simply took the best-perforning hyperparameters from our Chinchilla-optimal experiments on C4. This means that the consistency of DiLoCoo as we overtrain held despite the fact that for > 1, we used larger batch sizes than Data- aata-Parallel. We shows this in detail in Section B.2

---

3.6
34
Aggonttm
Desa Paralel
Eyal Loze
DitaCo, Mal
DiiaCo,
Overtrain Multplier
2.8
4
2.4
244
132
10"H
20th
10th
10PT
1277
Total PLOPs
Figure 7: Evaluation loss function of FLOPs, for varying algorithms and amounts of over-training. For each overtrain multiplier, the curves are all essentially parallel lines.
Scaling Laws
We now discuss the process used to fit scaling laws empirical results. For each algorithm (Data- Parallel or DiLoCo with M E {1, 2, 4, 8)) we ran extensive hyperparameter sweeps on models of size N up to 2.4B. To fit scaling law for loss L as function of N, pick the best hyperparameters for each N, and fit power law to L(N) = A For DiLoCo, we fit two types scaling laws: independent power law for each value of M and joint fits where we fit single scaling law L(N, M) & AN*M' We also fit power laws, both individual and joint, to the optimal learning rate y and batch size B. Due lack of space, we give the parameters of the power laws fit from Here we overview some of the important findings derived from these scaling laws.
data in Section
Interpolation. First, measured whether individual or joint fit scaling laws interpolated to our data in DiLoCo better. We do this via leave-one-out validation. We fit scaling laws for L, 7, and B, but only using data up N 1.3B parameters, leaving out our data on N 2.4B parameters. We then use the scaling law to predict the optimal value for L, Y, and B at N = 2.4B parameters, across different values of M. We measure the residual rys ÿ) [log(u) (y) log 7) between our prediction and the actual y. and average them across M. The results are in Table We see that the joint fit. matches independent for I. and B, but does better predicting 7-
Table 5: Residuals for scaling law predictions at N = 2.4B and varying M. We compare independent and joint scaling laws in predicting loss L, inner learning rate y, and global batch size B. For the average residuals, we highlight which of independent or joint achieved a lower residual.
Fit
L
B
Independent Joint
0.011
0.35
0.00088
M =
0.019
0.14
0.19
Independent
0.0099
0.18
0.44
M =
Joint
0.013
0.29
0.28
Independent Joint
0.012
0.051
0.25
M
0.0082
0.086
0.11
Independent
0.014
0.62
0.076
M 8
Joint
0.0076
0.23
0.19
Independent
0.012
0.30
0.19
Average over M
Joint
0.012
0.19
0.19
Extrapolation. Next, we use the scaling laws for hyperparameters to predict optimal hyperparame- ters for N 4B and 10B, for Data- Mtra-Parlllel and DiLoCo with M (1,2,4) 4). We train using those

---

hyperparameters, and measure how well our scaling laws extrapolated. We compare the evaluation loss of these models in Table We also show how well these aligned with our actual scaling laws for loss in Figure g Our extrapolation reveals few key points. First, the trends discussed in Section 4 continue to hold at larger model scales, where Di with M = 1,2 2 does better than Data-Pate Parallel. Second, as with our interpolation experiments, joint-fit power laws extrapolate better for DiLoCo.
Table 6: Evaluation results on 4B and 10B models, using hyperparameters predicted by individual and joint scaling laws. We highlight DiLoCo evaluation results that were better (ie. lower) than Data-Parallel. We see that while DiLoCo with M = 2 does better than independent or joint fit rules, DiLoCo M = 1 only does better when using joint fit.
with either
Loss
Algorithm
Fit Method
4B
10B
Data-Parallel
Independent
2.224
2.090
Independent
2.229
2.103
DiLoCo, M
Joint
2.219
2.086
Independent
2.218
2.083
DiLoCo, M 2
Joint
2.220
2.086
Independent
2.232
2.098
DiLoCo, M 4
Joint
2.230
2.096
35
20
3.
341
2.4
3.5
24
24
20
Figure 8: Scaling laws for Data-Parallel and DiL.CCo. Pictured are both the loss values to form the scaling law (by training models up to a scale of 2.4B) and loss values attained on larger models (4B and 1ÖB). While we present the individual-fit scaling laws for simplicity (and the ability to visualize Mata-Parallel), the joint fit also predicts loss similarly well.
Other results. Due to a lack of space, we relegate other findings of our scaling laws to Section F There discuss the methodology in more detail, as well as the actual parameters of the power laws. We also discuss fitting other functional forms to the scaling laws, beyond simple power laws.
7 Discussion and Limitations
Our results above all show that like Data-Paralla DiLoCo scales predictably with model size in ways that make it simpler to tune hyperparameters and train models at extremely large scales. Moreover, DiLoCo can offer significant benefits over Data-Parallel, including superior evaluation loss when using single model replica, and increased optimal batch size for any number of model replicas. These benefits are robust to model scale, overtraining amount, and synchronization frequency.
While promising, there are clear limitations and directions for future exploration. First, while we have done our best to include downstream evaluations on our models (Section[D). careful evaluation across domains and downstream tasks is important for validating scaling behavior. Moreover, there are wide variety of datasets used for pre- pre-training. While we have incorporated two widely used datasets of this kind in our analysis, we have not validated robustness across datasets. Second, like Hoffmaan et al. (20), we use dense transformer architectures of varying sizes, and have not validated these results on alternative architectures (such as coitture-of-xxperrr or in other domains, notably multi-modal settings. Third, we use an idealized wall-clock model (Section B). leaving to future work a thorough analysis of runtime in practical systems, and how it varies according to the type of compute. Last,
9

---

while we have attempted to be comprehensive in analyzing the core DiLoCo algorithm, our work does not encompass various related methods, and does not constitute comprehensive algorithmic benchmark. While we expect our findings transfer to related methods, explicit verification of this is left to future work.
References
[1] Kwangjun Ahn and Byron Xu. Dion: communication-effiiinn optimizer for large models. arXiv preprint arXiv:2504.0529,, 2025.
[2] Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disam- biguation. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pages 26-33, 2001.
[3] Matt Beton, Matthew Reed, Seth Howes, Alex Cheema, and Mohamed Baioumy. Improving the efficiency of distributed training using sparse parameter averaging. In ICLR 2025 Workshop on Modularity for Collaborative, Decentralized, and Continual Deep Learning, 2025.
[4] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages
2020.
[5] Zachary Charles and Keith Rush. Iterated vector fields and conservatism, with applications to federated learning. In Sanjoy Dasgupta and Nika Haghtalab, editors, Proceedings of The 33rd International Conference on Algorithmic Learning Theory, volume 167 of Proceedings of Machine Learning Research, pages 130-147. PMLR, 29 Mar-01 Apr 2022. URL https proceedings. wwa.ggssmmr... ppress//v6//chrr//////
html
[6] Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large-cohort training for federated learning. Advances neural information processing systems, 34:20461-20075, 2021.
[7] Zachary Charles, Nicole Mitchell, Krishna Pillutla, Michael Reneer, and Zachary Garrett. Towards federated foundation models: Scalable dataset pipelines for group-structured learning. Advances in Neural Information Processing Systems, 36, 2024.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. arXiv preprint arXiv: 1803:0800330555. 2018.
[10] Haotian Dong, Jingyan Jiang, Rongwei Lu, Jiajun Luo, Jiajun Song, Bowen Li, Ying Shen, and Zhi Wang. Beyond a single ai cluster: A survey of decentralized I1m training. arXiv preprint arXiv:2503. 1023, 1023,
2025.
[11] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. DiLoCo: Distributed low- communication training of language models. arXiv preprint arX1v:2311.08105. 2023.
[12] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Adhiguna Kuncoro, Yani Donchev, Rachita Chhaparia, Ionel Gog, Mare' Aurelio Ranzato, Jiajun Shen, and Arthur Szlam. DiPaCo: Dis- tributed path composition. arXiv preprini arXiv:2403. 10616, 2024.
[13] Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross Mcllroy, Jiajun Shen, et al. Streaming DiLoCo with overlapping communication: Towards distributed free lunch. arXiv preprint arXiv:2501. 18512, 2025.
10

---

[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2111 2024.
[15] FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training.
com wwweboolrreeerrccffffaaa.ca... 2021.
[16] Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, and Edouard Oyallon. WASH: train your ensemble with communication-efficeen weight shuffling, then average. Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2024. http::t//////ii.orrrggg
17517
[17] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Kch, et al. Language models scale reliably with over-training and on downstream tasks, arXiv preprint arXiv:2403.088.0 2024.
[18] Qiaozhi He, Xiaomin Zhuang, and Zhihua Wu. Exploring scaling laws for local SGD in large language model training, arXiv preprint arXiv:2409113196 2024.
[19] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: 004,0,0499.... 2017.
[20] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimgt large language models. arXiv preprint S..Iv:2203315556 2022.
[21] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.0633 2019.
[22] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. GPipe: Easy scaling with micro-batch pipeline parallelism. proceeding of Computer Science> Computer Vision and Pattern Recognition, 2019.
[23] Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. INTELLECT-1 technical report, 2024. URL httpps///axxiv owg. 2bb/2412 01152
[24] Sami Jaghouar, Jack Min Ong, and Johannes Hagemann. OpenDiLoCo: An open-source frame- work for globally distributed now-commuiication training. arXiv preprint arXiv:2407.07852. 2024.
[25] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1-2): 12000 2021.
[26] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. 2020.
[27] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint a4Xiv: 2014.
[28] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. PyTorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006. /5704, 2020.
[29] Aixin Liu, Bei Feng Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
11

---

[30] Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A Rusu, Jiajun Shen, Arthur Szlam, and Mare' Aurelio Ranzato. Asynchronous local-SGD training for language modeling. arXiv preprini arXiv:2401.09133 2024.
[31] Peter J. Liu, Roman Novak, Jaehoon Lee, Mitchell Wortsman, Lechao Xiao, Katie Everett, Alexander A. Alemi, Mark Kurzeja, Pierre Marcenac, Izzeddin Gur, Simon Kornblith, Kelvin Xu, Gamaleldin Elsayed, Ian Fischer, Jeffrey Pennington, Ben Adlam, and Jascha-Sohl Dickstein. NanoDO: A minimal transformer decoder-only language model implementation in JAX., 2024. URL http://githubb com/ gomn/google-ooo deepmind/nanodo
[32] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[33] Alexandra Sasha Luccioni and Joseph D Viviano. What's in the box? preliminary analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:2 2021.
[34] Olvi L Mangasarian and Mikhail V Solodov. Backpropagation convergence via deterministic nonmonotone perturbed minimization. Advances in Neural Information Processing Systems, 1993.
[35] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model 2018.
of batch-batch training. arXiv preprint
[36] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effcee -effiicienn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[37] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358 50376,, 2023.
[38] Hiroki Naganuma, Xinzhi Zhang, Man-Chung Yue, loannis Mitliagkas, Philipp A Witte, Rus- sell J Hewett, and Yin Tat Lee. Pseudo-asynchronous local sgd: Robust and efficient data- parallel training. arXiv preprint ar arXiv:2504. 18454, 2025.
[39] Deepak Narayanan, Aaron Harlap. Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1-15, 2019.
[40] Feng NIu, Benjamin Recht, Christopher Re, and Stephen Wright, HOGWILD!: lock-free approach to parallelizing stochastic gradient descent, 2011. bttps://:///a org/ /bs/ 1106. 5730
[41] Matteo Pagliardini, Pierre Ablin, and David Grangier. The AdEMAMix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03 03, 2024.
[42] Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of work- stations. J. Parallel Distrib. Comput.. 177 10.1016/1.jpcc.2008 .9.0000..900.... URL htttttttt//oi..oot http://.........../. 1016/j jpdc. 2008 002
February 2009. ISSN 0743-7315. doi:
[43] Bowen Peng, Jeffrey Quesnelle, and Diederik P Kingma. Decoupled momentum optimization. arXiv preprini 2XXx:24411.19700 2024.
[44] Alain Petrowski, Gerard Dreyfus, and Claude Girault. Performance analysis of pipelined backpropagation parallel algorithm. IEEE Transactions on Neural Networks, 4(6)9970-981, 1993.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 1414:0116) 2020.
[46] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
12

---

[47] Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL htttps://openreeee wttpa/rriiieieeeeffrruieu net/forum?
1d=LkFG31B13U5
[48] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. (Zero-offload): Democratizing (billion-scale) model training. In 202/ USENIX Annual Technical Conference (USENIX 2/), pages 551-564, 2021.
[49] Keith Rush, Zachary Charles, Zachary Garrett, Sean Augenstein, and Nicole Elyse Mitchell. Dr- JAX: Scalable and differentiable mapreduce primitives in JAX. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ ICML 2024), 2024.
[50] Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, et al. Photon: Federated LLM pre-training. arXiv preprint 1rXiv2241..0200S 2024.
[51] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.000 00044, 2023.
[52] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanan- takool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, al. Mesh- TensorFlow: Deep learning for supercomputers. Advances in neural information processing systems, 31, 2018
[53] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024.
[54] Sebastian U Stich. Local SGD converges fast and communicates little. arXiv preprint 1805v97805.09767, 2018.
[55] Iyy Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International Conference on Machine Learning (ICML), 2013. URL UtLhttts::/ htttps::/pproceedingggg/ooo/ mlr. press/
sarres//288/ v22/ sutsseeree html
[56] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. gradient compression for distributed optimization. Advances in Neural Information Processing Systems (NeurIPS), 2019. https:/tt:
PowerSGD: practical low-rank
org/ absaaaaa/aaaaa 1905/ 13727
[57] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al- Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, al. A field guide to federated optimization. arXiv preprint arXiv:2107.066) 2021.
[58] Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. CocktailSGD: fine-tuning foundation models over 500mbps networks. International Conference on Machine Learning (ICML), 2023. URL htttps://ppeereeiee het//forumnidee222r10111122
[59] Xi Wang and Laurence Aitchison. How set Adam W's weight decay as you scale model and dataset size. arXiv preprint 1.Xi::2440516690 2024.
[60] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishck Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. S.XXi:22309.122. 2023.
[61] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can 2019.
machine really finish your sentence? arXiv preprint
13

---

[62] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: steps forward, 1 step back. Advances in neural information processing systems, 32, 2019.
[63] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch FSDP: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 2023.
[64] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. Advances in neural information processing systems, 23, 2010.
14

---

A Related Work
Distributed training of LLMs. Due to their increasingly large sizes, the advancement of language models has necessitated advancements in distributed training methods. One vein of work uses data-parallel training, and attempts to shard its constituent computations across accelerators in efficient ways. This includes advancements in things like distributed data parallelism [52] 28 ZeRO parallelism [46] 48 fully-sharded data parallelism [[5] [3], and pipeline parallelism 44 22 39].
Conceptual models of the impact of batch size on training time [35 aid in making the most effective trade-offs between training time and compute cost when using data-parallel training methods.
As scale continues to increase, the need to all-reduce gradients between data- data-parallel replicas becomes a bottleneck in training, causing accelerators to 'wait' on this allreduce for an unacceptably long time. This observation has motivated extensive work in pipeline parallelism and scheduling, communicating activations rather than gradients. An alternative line of work keeps the basic data processing pattern of data parallelism while directly minimizing communication requirements.
Three broad families of algorithms exist in that space: (1) sparse updates (including CocktailSGD
588
PowerSGD [56], DeMo [43] and Dion (I), (2) fast asynchronous updates (including Hogwild
40
WASH [6 and Sparta 3 and (3) infrequent updates (including LocalSGD [54], FedOpt
(38] ). These are extremely active areas of work, and we do not attempt to give a comprehensive survey of them all simultaneously. We instead defer the interested
47
DiL.Co [1, and PALSGD
reader to survey of decentralized LLM training [10].
In this work, we focus on the third category. Douillard et al. showed that we can reduce communication costs in LLM training significantly by training multiple models independently with infrequent synchronization. Their method, DiLoCo, massively reduces communication overhead when training LLMs with a moderate numbers of model replicas. It has also shown great promise in training LLMs up 10 billion parameters (2 233. This work has also been extended to asynchronous
overlapped updates 30
13
and low-communication expert sharding [2]
Federated learning. There is an enormous body of work on communication methods for machine learning. In that vein, DiLoCo is closely related algorithms used in federated learning to perform communication-effiiee sively) on edge devices [25). The prototypical algorithm used in federated learning, FedAvg (36), reduces communication costs by training models in parallel, with periodic model averaging. This algorithm has been invented and reinvented throughout machine learning, and is also known as Local SGD [54). parallel SGD [64] and parallel online backpropagation (34]. The use of inner and outer optimization steps (as in DiLoCo, see Algorithm I was first used by Hsu et al. 22 and Reddi et al. (7 for federated learning, focusing on SGD as the inner optimizer and SGDM or Adam [27] as the outer optimizer in order to leverage more sophisticated optimizers in resource-constrained settings. DiLoCo is also closely related to many other federated optimization methods, though the huge amount of work this area makes it impossible to summarize succinetly here. We instead refer the interested reader to the survey of Wang et al. (57), though the field has of course progressed since then. While federated learning is often applied to more moderately sized models, Charles et al. 7
training
training over decentralized data, often (but not exclu-
and Sani et al. [50) show that federated learning can be used to good effect for LLM training.
Scaling laws. Scaling laws work often aims to estimate how empirical generalization error scales with various facets, including model size and training set size. Empirical scaling analyses with power law behavior date have existed for decades (see (2)). Hestness et al. [19] developed power laws for model and dataset size across various tasks and model architectures (including encoder- decoder LSTM models). More recently, scaling laws for transformer- based L.LMs were proposed by Kaplan et al. (6) and Hoffmann et al. (20). who exhibited power law relationships between LLM performance and model size. Sine then, there has been large number of works developing scaling laws for other facets of LLMs, including (among many others) inference costs [1]) data-constrained training [37] and overtraining 17
Scaling laws for DiLoCo were previously studied by He al. 18] who show that DiLoCo with 8 replicas exhibits analogous scaling behavior to Data-Paralloe He et al. [18) use fixed number of replicas, batch size, and outer learning rate, and an unspecified total token budget Our work
At the time of writing, He et al. [18 only say that "Each model trained to achieve adequate convergence".
15

---

expands many aspects of their work and explores others that were not considered, including but not limited to: 10x larger models, varying the number of replicas (including single-replica DiLoCo), varying token budgets and overtraining, parametric function fitting, scaling laws for hyperparameters, and optimal batch size analysis.
B Wall-Clock Time Model
In this section, we present an idealized model for the wall-clock times of Data-Parallel and DiLoCo. We measure the total elapsed time, which means that parallelization (e.g. via increasing the batch size) reduces wall-clock time.
B.1 Computation Time
Here, we mean the time expended by floating point operations in model training, ignoring communi- cation time across nodes (which we treat separately in the section below). We use the idcalized model where the total FLOPs C 6N D. Given some number of chips R. each of which floating point operations per second, the total computation time bounded below by C/RQ. The number of chips R is purely function of model size N and global batch size B. The number of chips does not depend on the algorithm using DiLoCo.
perform Q
or DiLoCo) or number of model replicas when
B.2
Communication Time
The network connectivity is characterized by a bandwidth W and latency e. When performing an all-reduce of N parameters over compute nodes, the lower bound on the amount of traffic and received by at least one of the compute nodes participating in the all-reduce 2N R=1 [42] Such algorithms synchronously but in parallel, in network with bandwidth W and latency between each pair of nodes, the time to complete the all-reduce is at least
called bandvidtt-optimal. Since communication across the nodes is done
2.V W
1 R +
DiLoCo | was designed for settings where models too large to fit in single datacenter, they must be trained across compute islands connected by low bandwidth. To model this, we will assume that we training over R compute nodes (typically, GPUs TPUs). Some of these connected by networks within a datacenter, and others are connected across datacenters. We let Wo, 6o denote the bandwidth and latency of the within-datacenter network, and W E analogously defined for the cross-datacenter network. Typically, Wo 2 W1, S E1.
Data-Paralle:: At every training step T. we have to perform an all-reduce over all R compute nodes. Since some nodes are connected across datacenters, the total communication time is at least
2N W
1
+ + T
R
DiLoCo, = 1: every inner step T, we perform an all-reduce over all devices as in Data-Parallel training. We also do an all-reduce every H steps for the outer optimization. Some of these nodes are connected across datacenters, so the communication time per all-reduce is at least 2N(1 - W- + E1. The total communication time therefore at least
1
+
T
W
R
H
DiLoCo, M We assume that each the M model replicas is trained on compute nodes connected within the same datacenter. In each inner step T, each model replica is trained by R/M devices which need to do an all-reduce. However, no communication occurs between datacenters, so the communication time of each inner step is bounded by 2N MR- Eo-
16

---

Each outer optimization step involves all-reducing over all R devices, connected across datacenters. This incurs a communication time of at least (N - W + 61. Since occurs only every H steps, the total communication time is bounded below by:
2.V
1 Wo
M
2N
T H
+ 60
1
+ E1
R
W1
R
Note that this suggests that as long as H N Wo/W W,, the outer communication steps incur at most half of the total communication cost.
Streaming DiLoCo. We note that the computed cost above applies to the Streaming DiLoCo as well. While the inner step remains the same, the outer step is smoothed such that each fragment .i.....,)),
P E {l...., P} is every steps. However, fragment communication offiet such that some fragment is communicated every H/P steps, resulting in the communication amortizing to the same per-step cost. This is expected as Streaming DiLoCo reduces peak communication over any step, but does not reduce total communication across training.
Overlapping communications. Another contribution of Douillard et al. [ is the ability to overlap communications required for the outer optimizer with computation by using a stale version of the fragment in the outer optimizer, and merging the result of this outer optimization with the locally optimized fragment. This would allow, for example, the communication term to be omitted from the calculation for wall-clock-time, if computation time dominates communication time. This setting different from an algorithmic perspective, so its impact on scaling would need to be examined independently.
B.3
Total Wall-Clock Time
The total wall-clock time sum of the computation and communication times above. To measure the communication time, we must know the number of chips R used for each experiment, the number of FLOPs per chip per second Q, the bandwidth and the latency of the within -attaeddtteeeter and cross- datacenter networks. For computation costs, we use slightly idealized number of chips R based on our experiments, but ensuring that doubling the global batch size would double the number of chips. We base the constant Q on publicly available information about the FLOPs capabilities of the TPU v5e and v6e chip which have peak compute per chip (in bfloat 6) of 197 teraflops and 918 teraflops, respectively. Assuming maximum FLOPs usage of 50%, these chips have actual compute of approximately 100 and 408 teraflops, respectively. When computing idcalized compute time, set Q 300 teraflops, somewhere in-between the two.
For bandwidth and latency, we consider three archetypes of networks:
1. High-bandwidth network with bandwidth Wiigh 400 gigabits per second and latency of Ehigh 10 seconds.
2. (eediumbbandwidli network with bandwidth medt latency of Emed 10-3 seconds.
100 gigabits per second and
3. Low-bandwidth network with bandwidth W 10 gigabits per second and latency of foo = 10 seconds.
We stress that these are not based on any actual systems, and are simply designed as instructive archetypes of networks. For the idealized communication time, we always use the high-bandwidth network for the within-datacenter network, and one of the three for the cross drtteentt network.
C Datasets and Licenses
In this section, discuss the datasets used in our experiments and their respective licenses. We note that the datasets themselves are all referenced in Section
See http:///wwd..googl...ooo////o/o/
17

---

C4 [45 This dataset is cleaned version of Common Crawl's web crawl corpu We used the version accessible via Hugging Face, provided by Allen Af" and made available under the ODC-BY license. Because it is derived from Common Crawl, its usage is also bound by the Common Crawl terms of use. We use the en version of the dataset. We note that while widely used and filtered, the dataset still contains undesirable content including hate speech. See [33] for an analysis of its contents, and discussion of mitigation strategies. We do not release any assets derived from this dataset.
Dolma [53]. This dataset consists of 3 trillion tokens drawn from web content, academic publications, code, books, and related material. We used the v1 version of the dataset accessible via Hugging Face, provided by Allen A and made available under the ODC-BV D license. We note that the usage of this dataset also bound by license agreements and restrictions of its original data sources. See [53] for more details.
HellaSwag [61. We use this dataset for zero-shot evaluation metrics, We used the ver- sion available under the MIT license For details, see https://roowanelleerss com/ hellaswag/
Piqa [4.
We use this dataset for zero-shot evaluation metrics. We used the version available under the Academic Free License v
Arc-Easy (9]. We use the Are-Easy dataset for zero-shot evaluation metrics. We use the version accessible via Hugging Face. provided by Allen A made available under the CC-BY-SA-4.0 license.
D Additional Experimental Results
Mgoter
Figure 9: Evaluation loss of Data-Parallel and DiLoCo as a function of global batch size (in tokens).
Figure 10: sero-shot evaluation accuracy on HellaSwag of Pata-Parallel and DiLoCo as a function of global batch size (in tokens).
011
340
Colgl Cor
Figure 11: Zero-shot evaluation accuracy on Piga of Data-Parallel and DiLoCo as function of global batch size (in tokens).
In this section, we give additional experimental results that expand on those in Section 4 In Figures 9 10
and 12
we present evaluation loss and evaluation accuracy on various downstream zero-shot
httpes//bttpp:/ommmmccsss...
18

---

X013
X013
X010
k05
Figure 12: Zero-shot evaluation accuracy on Arc-Easy of and DiLoCo as a function of global batch size (in tokens).
tasks, as a function of algorithm, model size, and global batch size. The results consistently show that Data-Parallel's evaluation performance degrades quickly as batch size increases. By contrast DiLoCos performance degrades more slowly, or even improves, as batch size increases. We note that Arc-Easy was quite noisy as an evaluation task.
6.55
Hodel Scale
0.43
3339
SSSM
34
2.43
Faraie Diuito,
0.0
closal Botch Size
Global Butch Size
(a) Evaluation loss.
(b) Zero-shot accuracy on HellaSwag.
Evaluation loss and zero-shot accuracy of Data-Paralle and DiLoCo with M 1 for varying model and global batch sizes (measured in tokens). In all settings, DiLoCo with M = 1 does better than Patallaral,,,, and the gap between them increases with batch size.
Figure 13:
0.72
2.43
6.80
2.18
3.43
Algoriti
Ago kimm
5.5
24
212
2
(a) Zero-shot accuracy Arc-Easy.
(b) Zero-shot accuracy on Arc-Easy.
Figure 14: Zero-shot evaluation accuracy of Data-Parallel and DiLoCo with M = for varying model and global batch sizes (measured in tokens), Piga and Are-Easy. In nearly all settings, DiLoCoo with M = does better than Data-Parallel, and the often the gap increases with batch size.
we compare Data-Parallel and DiLoCo with M = terms of their evaluation loss and zero-shot evaluation accuracy HellaSwag, Piqa and Are-Easy. As above, note that DiLoCo with M = 1 has an improved tolerance to larger batch sizes.
Figures I3 and 1
E
Ablations
Here, we expand upon the ablation studies discussed in Section give more detailed experimental analyses, well as idealized wall-clock times under varying networks.
19

---

E.1 Synchronization Cadence
As discussed in the synchronization cadence ablations in Section we perform an ablation over H for varying M and model sizes N. For each setting, we take the optimal inner learn- ing rate and global batch size from above, and sweep H over {1, 5, 10, 30, 100, 300} and r over (0.05, 0,1, 0.2, 0.4, 0.6, 0.8, 1.0).
In addition the results in Section we analyze how H impacts the evaluation loss of DiLoCo in Figure 13| For all models, synchronizing every step (H = 1) performs the worst, but after this point all values of H perform somewhat comparably. For fixed N and M, evaluation loss increases Hl increases. However, this increase is less pronounced for M = 1 and larger models. This yields an important finding: As the model size N increases, can actually perform synchronization across DiLoCo replicas less frequently, while nearly maintaining evaluation performance.
Model Scale
3.4
90M
3.2
550M
Aggorithm
DlooOo, M-1
DLoCo, Mmd
100
300
Synchranization Cadence
Figure 15: Infrequent synchronization works better for larger models. Outside of H which performs the worst, evaluation loss increases with H. However, the rate of increase is less pronounced for DiLoCo with M = 1 and for larger models, suggesting that large models can be synchronized quite infrequently.
Table 7: Simulated compute utilization. We estimate the step time based on the required flops using the rule proposed by Kaplan et (26) and max flop utilization of 60%. We estimate the bandwidth (in Gbit/s) required to reach a level of compute utilization using [13]'s 's simulator. We highlight in light blue 10x reduction of bandwidth, and in dark blue 100x reduction.
Gbit's to reach compute utilization CU =? 50%
Architecture
Size
Step time
Method
80%
90%
95%
99%
Data-Parallel
104.8
184.2
222.3
222.3
390.7
DiLaCo, H
104.8
184.2
222.3
222.3
390.7
Chinchilla
10B
0.8s
DiLoCo, H = 10
16.0
49.4
86.8
152.6
222.3
DiLoCo. Н 50
3.0
11.0
23.3
41.0
126.5
DiLoCo, H 100
1.4
6.2
13.3
23.3
86.8
DiLoCo, H 300
0.5
4.3
41.0
Data-Parallel DiLoCo, H
126.5
222.3
268.3
323.8
323.8
126.5
222.3
268.3
323.8
323.8
Llama3
405B
26s
DiLoCo, H 10
19.3
72.0
126.5
184.2
268.3
DiLoCo, Н 50
3.6
13.3
28.1
59.6
184.2
DiLoCo, H 100
2.0
7.5
16.0
33.9
126.5
DiLoCo, H 300
0.7
6.2
13.3
59.6
Data-Parallel
323.8
$69.0
686.6
686.6
1000.0+
DiLoCo, H
323.8
$69.0
686.6
686.6
1000.0+
DeepSeek-V3
671B
20s
DiLoCo, H 10
49.4
152.6
268.3
390.7
686.6
DiLoCo. H 50
7.5
33,9
72.0
126.5
390.7
DiLoCo, H 100
4.3
16.0
41.0
72.0
268.3
DiLoCo. H
1.7
13.3
28.1
126.5
Compute utilization.
The synchronization cadence of DiLoCo is critical to training large-scale models distributed across the world. Indeed, less frequent synchronization (larger H) diminishes
20

---

30
Uato marates
Oce fendll
(a) 1OB Chinchilla
(b) 405B Llama3
(c) 671B DeepSeek-
Figure 16: DiLoCo greatly increases compute utilization. Here present simulated compute utilization for DiLoCo and Data-Paralls across a range of bandwidth and synchronization cadences H. compute utilization of 0.8 means 80% of the time is spent in computation, and 20% in communication. A higher value is better. We see similar results for other model sizes, but omit for visual clarity.
bandwidth requirements of training. Following required to have a compute utilization
we simulate the amount of bandwidth as large as possible for three types LLMs: a
13
(compute time
jotal bune
10B Chinchilla-style transformer [20] in Fig 16a 405B Llama3 model [14] in Fig 16b and a 67 IB
DeepSeck-v3 MoE [29 in Fig 16c
We also report raw numbers in Table
E.2
Overtraining
We expand on the overtraining ablation study in Section As we showed there, DiLoCo continues to enjoy large optimal batch sizes, compared to Data-Paralles This improves horizontal scalability. decreasing end-to-end wallclock time. Wallclock time also smaller for M 1 due to reduced communication.
To illustrate this, we plot idealized training time of Data- and DiLoCo with M 2 for different overtraining amounts in Figure 17 DiLoCo speeds up overtraining by reducing communication costs, and utilizing larger batch sizes, therefore requiring fewer serial training steps. This suggests that DiLoCo is sigmificant boon in overtraining, we can amortize compute time (which can be quite long for overtraining) via horizontal scalability.
Alger thm
15900l aND Tool boioc
(a) Network with bandwidth
(b) Network with bandwidth of
Network with bandwidth of 400 gigabits/s and latency of 10 sec- onds (high-bandwidth).
gigabits/s and latency of
gigabits/s a latency 10 3 seconds caddlumm-bnniwdthh
10 - seconds (low- bandwihhhi
Figure 17: For Data-Parallel and DiLoCo, M 2, we plot idealized wall-clock time (see Section B for training by Data-Parallel and DiLoCo, M = 2 across compute nodes connected via high- medium-, and bwwbaandwithh networks. For each algorithm and overtraining amount, we display lines represent varying model sizes, from 335M parameters to 2.4B. DiLoCo is faster in all settings, due both its reduced communication and its tolerance larger batch sizes. Even the high- bandwidth setting, the larger batch sizes increase horizontal scalability, reducing end-to-end wallclock time. We see similar results for M > 4 and for smaller models, but omit for visual clarity.
21

---

F Scaling Laws
We now discuss the process we used to fit scaling laws to our empirical results, expanding on the discussion in Section 6 Recall that for each algorithm (Data-Parallel or DiLoCo with M E {1, 2, 4, 8)) we ran extensive hyperparameter sweeps on models of size N up to 2.4B. To fit a scaling law for loss L as function of N, we pick the best hyperparameters (in terms of evaluation loss) for each N, aggregate the loss values, and fit some kind of function for L(N), such as a power law [26]. We will also fit scaling laws to the optimal hyperparameters. We will fit two types of scaling laws for DiLoCo: independent fits for each value of M, and joint fits where we fit single scaling law as a function of N and M simultancously.
F1 Independent scaling laws
Scaling laws for loss. We first fit scaling laws for the loss obtained by Data-Parallel training. We fit
power law to the evaluation loss of Data-Parallel as a function of N via the power law approximation
L(N) R AN Note that this can easily be done via applying linear fit techniques to log(L), and is not sensitive to things like initial values of A, a. The resulting power law is in the first row of Table g
We mirror this above for DiLoCo when doing independent fits. For each value of M, we record the lowest loss value across all hyperparameters. We can then fit power law LM (NV == L(N, M) R
ANo for each M. The results are given in Table
Table 8: Power law approximations for loss L(N) R ANO,
Data-Paralle 18.129
-0.0953
DiLoCo, M
18.363
-0.0961
DiLoCo, M- ==2 2
18.768
-0.0969
DiLoCo, M- 19.762
-0.0992
DiLoCo, M M-S - 21.051
-0.1018
The results show that Data-Parallel and DiLoCo see similar predicted reductions in loss as a function of N. Notably, the fit parameters suggest that DiLoCo, M 1 outperforms Data-Parall at essentially all but the absolute smallest model scales. This mirrors the results discussed in Section
fit scaling laws for learning rate y and batch size B. For DiLoCo, we fit scaling laws for inner learning rate and global batch size B. Given their analogous role in the algorithms, we fit them in the same way. For (inner) learning rate, we use the same approach as fitting scaling laws for loss: for each N (and M, for DiLoCo), we select the best hyperparameters, and fit a power law. The results are in Table 2
Scaling laws for hyperparameters. For Data-Paralled
For (global) batch size, we alter this slightly. As discussed in Section 3 our sweeps use powers of for batch size, in order to saturate compute. However, the optimal batch size may be between these values. To account for this, we first fit a quadratic approximation to the batch size. Specifically, for each value of N we look at the loss that B), and fit a quadratic to this function. We select the minima of those quadratics and fit a power law to them,
function of log2 (B) (when using the best learning rate for
function of N. The results are in Table I0
DiLoCo has a third hyperparameter we could fit scaling laws to: the outer learning rate. However, as shown in Section 4 the optimal outer learning rate is (for sufficiently large models) seemingly constant. Therefore, a scaling law would seemingly not yield any improved predictive performance over simply using the best outer learning rate for each M (sce Figure 5).
F.2 Joint scaling laws
Alternatively, we can fit joint power laws to various facets DiLoCo, using a two-variable power law /(N, M) x AN We do this for loss L,, inner learning rate y and global batch size B. For the first two, we select, for each value of N, M, the best learning rate and loss. For batch size we do the same, but using the quadratic approximations from the section above. We can then fit a joint
22

---

Table Power law approximations for (inner) learning rate ((N) = AN
Table 10: Power law approximations for (global) batch size ( = AND.
A
A
at
Data-Pun Para-Parlllel
16319.2
-0.119
Data-Parallel
462.68
0.281
DiLoCo, M
74620.6
-0.945
DiLoCo,
27.873
0.435
DiLoCo, M 2
3978.82
-0.780
DiLoCo, M
15.749
0.479
DiLoCo, 4512.99
-0.789
DiLoCo, == =444 4.
10.957
0.510
DiLoCo, M 8
618986
-1.102
DiLoCo, M- == 8 38.072
0.455
Table 11: Joint power law approximations f(N, and batch size B of DiLoCo.
ANO M for the loss L, inner learning rate 7
Cx
3
L
19.226
-0.0985
0.0116
?
22256
-0.827
0.2929
B
14.521
0.4695
0.3399
power law via standard linear regression techniques. The resulting power laws are in Table [T] Just with the independent fits, do not attempt to fit scaling laws to the outer learning rate 77, the optimal value is independent of N.
F.3 Measuring goodness-offfit
Now that we have two different ways developing scaling laws for DiLoCo, we can attempt to ask which one yields better predictions. First we do this via leave-one-out validation. Specifically, we use the same methodology as above fit scaling laws for L, y, and B, but only using data up to 1.3B parameters, leaving out our data on N = 2.4B parameters. We then use the scaling law to predict the optimal value for I, and B N 2.4B parameters, across different values of M.
Table 12: Joint fit scaling laws match or beat independent fit. Here we give the residuals for
scaling law predictions for N 2.4B and varying M. We compare the residual of independent and joint fitting strategies in predicting loss I., inner learning rate 7, and global batch size B. For the average residuals, we highlight which of independent or joint achieved lower residual. We see that the joint fit matches independent for L and B, but does better at predicting
Fit
L
Y
B
Independent Joint
0.011
0.35
0.00088
M -1
0.019
0.14
0.19
Independent Joint 0.013
0.0099
0.18
0.44
M = 2
0.29
0.28
Independent Joint 0.0082
0.012
0.051
0.25
M 4
0.086
0.11
Independent Joint
0.014
0.62
0.076
M
0.0076
0.23
0.19
Independent
0.012
0.30
0.19
Average over M
Joint
0.012
0.19
0.19
Given predicted value ý and reference value of y, we compute residual our prediction as the mean absolute error of the logarithm: res(y, Ø) = [log(v) log(ū)]. We use this measure as it works well for all three variables simultaneously, despite the fact that they vary greatly in their scale. For each M {1, 2, 4, 8), we compute the predicted value of the three parameters above, and measure
23

---

the residual relative the actual optimal value at N = 2.4B. We report these values, as well as their average across M. in Table 12
Our results generally show that both approach is generally valid (as there is no clear winner) but also that there is significant variation in residuals between M. That being said, we see that average, while the individual fit is slightly better at predicting the loss and global batch size, the independent fit is significantly better at predicting inner learning rate.
F.4 Extrapolating to larger models
We use the independent and joint fits to predict optimal hyperparameters for DiLoCo with M E 2, 4] at 4B and 10B model scales. Note that for Data-Parallel, we can only use independent fits. We run training on these models with these hyperparameters, using a Chinchilla-opiimal token budget, and compare the results.
and
Table 13: Joint fit hyperparameters extrapolate well to larger models. Here we show the evaluation results on 4B and 10B models, using hyperparameters predicted by individual and joint scaling laws. We highlight DiLoCo evaluation results that were better (ie. lower) than Data-Paralla We see that while DiLoCo with M = 2 does better than aata-Parallel with either independent or joint fit rules, DiLoCo M-
only does better when using joint fit.
Loss
Algorithm
Fit Method
4B
10B
Data-Parallel
Independent
2.224
2.090
Independent
2.229
2.103
DiLoCo, M
Joint
2.219
2.086
Independent
2.218
2.083
DiLoCo, M 2
Joint
2.220
2.086
Independent
2.232
2.098
DiLoCo, M
Joint
2.230
2.096
We see two important facets. First, unlike results above, at 4B and 10B scales we see that DiLoCo with M- 2 actually outperforms both Data-Paralle and DiI,oCo, M = 1, regardless of using individual or joint fit approaches. Second, we see that DiL.Co. M = 1 requires the joint fit to do better than Data-Parallel, Other than in this case, joint and independent fits perform comparably throughout. All in all, the joint fit approach to hyperparameters appears to have a slight edge over individual fit in extrapolating. Combined with its ability to also extrapolate to larger M, we generally recommend the joint fit approach for all hyperparameters.
We now use these loss values to see how they compare to the scaling laws fit above. We generally find that the loss values are predicted very well, within a few percentage points of the loss predicted by the scaling laws. We present the fit scaling law and extrapolated loss values in Figure 18
Di Ma1
Deats Med
3.5
Figure 18: DiLoCo scaling laws extrapolate well to larger models. We present loss scaling laws for Data-Parallel and DiLoCo. Pictured are both the loss values to form the scaling law (by training models up scale of 2.4B) and loss values attained on larger models (4B and 10B). While we present the individual-fit scaling laws for simplicity, the joint fit also predicts loss well.
24

---

E55 Parametric Function Fitting
Various works on scaling laws have often found it useful to functions more complex than power laws to the data [20]. We are particularly interested in parametric forms for our joint scaling laws. While Hoffmann al. 20 use risk decomposition argument to decompose loss as a function N and D, it is not immediately clear how to do decompose [(N, M). To that end, we use an empirical approach. We develop candidate functions, and determine which does best in extrapolative sense. We use the following functional forms:
M) A M
2. L(N, M) R AN M + C
3. L(N, 2 AN*+BM +C
4. M) A + BM +C
The first included for comparison's sake, as it recovers the power law scaling law used above. Fitting more sophisticated functions can be much more sensitive to initial values of parameters, and also sensitive to outlier data. Therefore, when fitting these functions to our loss values above, we use the general strategy proposed by Hoffmann et al. [20].
In detail, let [N,N, M) denote one the functional forms above, where Q represents the set of parameters to be fit (e.g. Q a,,,, 33) for the first). Let Huber, denote the Huber loss with parameter Let N, M denote set of values of N and M considered. For each N, M, we have an empirical loss [,N, M), and some estimate of the loss (Q(N, We then solve the following minimization problem:
min Σ Σ
Hubers log Jo(N, M) log M.N,M).
NEN MEM
We minimize this via L-BFGS, using some initialization Qu for the parameters. We repeat this process for 256 random initializations Qu of the parameters. We hold out all loss values the N 2.4B scale, and select the parameters Q that best fit the held-out data, measured terms of the average residual [log fQ(N, log L(N, m)) over all M.
Table 14: Parametric function fitting improves joint scaling laws. We showcase various parametric approximations to the empirical loss function L(N, M), along with their validation error held- out loss data at the N = 2.4B scale. We see that joint power laws (the first) and classical loss decomposition (the last) significantly worse predicting loss the held-out data.
Parametric form
Average Residual
L(N, M) s AN*MP
0.0044
L(N, M) AN M +
0.0035
L(N, M) AN*+3M +C
0.0025
L(N, M) AN" + BM +C
0.0043
We see that the power law (row 1) and additive decomposition (row 4) are significantly worse at extrapolating loss values than more nuanced parametric forms. We note that the additive decomposi- tion resembles the decomposition of loss as function of model size N and token budget used by Hoffmann et al. [20]. but does not seem to reflect how M affects loss for DiLoCo. We leave it as an open problem to determine what parametric forms better predict loss, and can be explained by theoretical understanding of communication-effici training.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We give extensive empirical evidence explicitly backing each claim made in
the abstract and introduction in Sections 4 S and 6
with further evidence in Appendix D
and F
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We explicitly identify limitations of our work in Section 7 We have attempted to be absolutely transparent about every detail in our empirical work, as way to make clearer the limitations, as well.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include any theoretical results in the work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We explicitly discuss every aspect of our empirical setup in Section 3 Our scaling law methodology is also explicitly detailed in Section F and our wallclock time model is detailed exactly in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
The datasets used for our experiments are available publicly, through standard distribution channels (including Hugging Face). We do not explicitly open-source our code. While would like to make the code available (and are working a modified version of it that can be open-sourced), we could not open-source the code as is due to the use of instiutionn-specifie code for launching experiments across accelerators. However, our code is based entirely on open-source libraries that we actively reference, including NanoDO [31 and DrJAX 40 There is no special behavior in our experiments that is not detailed explicitly in Section Indeed, our results strongly mirror those in other work,
including 24
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https::/nnis.cco)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. http:/ppblcgguudee///ppp wwwpeedeeeoooSSeeieeeeeeeeeeeeee
ssiocolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We specify this exactly in Section 3 We have detailed all possible aspects of training, including datasets, splits, optimizers, hyperparameters, model architecture details, model sizes, vocabulary size, and sequence lengths. Any missing details are unintentional, and would gladly be provided by the authors.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
28

---

While we would gladly produce and report error bars, doing so to within any statistically significant degree would be prohibitively expensive due to the large number of hyperparameters and model sizes used throughout.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detail the exact accelerators used for experiments. We also detail what aspects their computational profile were used for our idcalized wall-clock computations in Section!!
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have evaluated our work against each of the concerns listed in the guide- lines, and believe that it adheres to the provided standard.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
29

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Generally, our work has specific possible societal impact of the sort in the guidelines below. Algorithms like DiLoCo are similar to the example of "optimizing neural networks faster" in the guidelines below. That being said, do wish give one specific societal impact that is relevant: communication-eeficiennt algorithms like DiLoCo enable thte training of models in a decentralized fashion. This means that some assumptions about AI used in things like policy-making may not hold, such as the assumption that LLMs are trained within a single datacenter. For example, Al policy monitoring based on total amount of compute may be more difficult in settings where algorithms like DiLoCo are deployed, as the amount of compute used by each compute node may be quite small, and the nodes themselves may be decentralized. It may also allow smaller groups and institutions to do LLM training at scale. This can be good (democratizing access to LLMs, and enabling greater competition in AI marketplaces) and bad (enabling adversarial actors to scale up training of LLMs), though the specifics will depend on how the algorithms evolve, and by what margin they can reduce communication and computation bottlenecks.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We release no data models part of this work, and do not believe that the work poses any risks in this regard.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
30

---

that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We discuss all datasets used in Section 3 versions, and licenses used in Section C
Justification:
We further detail the datasets,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URI..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. wwt.coo///aaaa dattasts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do not produce, disseminate, or release any new assets as part of this work.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work did not involve any crowdsoureing research with human subjects.
Guidelines:
31

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work did not involve any crowdsoureing research with human subjects, and did not necessitate any IRB approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We do not explicitly discuss this in the main body of the work, but we did not use LLMs for any component of the research itself. The methodology, coding, and initial draft were all completed entirely by humans. LLMs were only used for minor stylistic and grammatical edits after the first draft.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should should not be described.
32


Reviews:
[{"review_id": "Cum2uXmeq3", "reviewer": "Reviewer_am9z", "summary": "This paper investigates the behavioral patterns of the DiLoCo optimization algorithm [1] with respect to its hyperparameters. The authors analyze how DiLoCo scales compared to Distributed Data Parallel (DDP) across a range of settings. Notably, they observe that the performance gap between DiLoCo and DDP narrows for higher levels of DiLoCo distributed settings as model size increases. Additionally, even when the number of unique model replicas is one, DiLoCo's low-synchronization regime still yields benefits over DDP. Overall, the paper provides a detailed empirical characterization of DiLoCo's performance under varying configurations, highlighting scenarios in which it outperforms/underperforms standard DDP", "questions": "Q1.\n> “Moreover, our results actually show a potentially counter-intuitive phenomenon: The optimal outer learning rate increases with $H$, even though the outer gradients increase in size as $H$ increases.”\n\nCould the authors clarify why they believe this occurs? Is it perhaps due to a need to ‘overcompensate’ for parameter drift at higher $H$? If so, do the updates also exhibit higher variance?\n\nQ2. Initially, it was unclear why DiLoCo receives a batch size of $B/M$, while DDP receives $B$. According to [1], DiLoCo would shard the $B$ tokens across $M$ workers, just as DDP would distribute them across its workers. The primary distinction seems to be that DiLoCo synchronizes every $H$>1 steps, while DDP synchronizes every step ($H=1)$.\nFrom reading the appendix, I gather that the total number of workers $R$ was fixed for both DDP and DiLoCo. For DiLoCo, these were divided into $M$ clusters, each internally synchronized via DDP. Thus, $M$ does not represent the number of replicas in the traditional sense; rather, it denotes the number of independently synchronized clusters (or the *unique replicas* among the R workers when $H>1$). Under this interpretation, $M=1$ and $H=1$ would approximate standard DDP.\nIf this is correct, it would be helpful to clarify the definition of $M$ more explicitly in the main text. If not, I would appreciate further clarification, as the current explanation is somewhat confusing.\n\nQ3. \n> and evenly partition it at the sequence level across the $M$ DiLoCo replicas.\nWhy is the partitioning across the sequence level and not batch level for DiLoCo? For pertaining, I can think of cases where this may not be an issue. Maybe I am missing something here, but isn't it possible that this leads to DiLoCo seeing shorter sequences compared to DDP on average?\n\nQ4. It would be valuable to study whether a relationship exists between synchronization cadence and global batch size with respect to evaluation loss. For example, Figure 3 (DDP vs. DiLoCo at large batch sizes) seems to implicitly suggest that a higher $H$ may be preferable at higher global batch sizes. Do the authors observe consistent trends in this regard?\n\nQ5. Similarly, in Figure 15, it's not immediately clear why $H=1$ performs the worst. Is this because the optimizer essentially reduces to solely SGD + Nesterov?\n\nQ6. Are Figures 5 and 6b conveying the same information? If not, it would be helpful to distinguish their respective roles more clearly.\n\n\n[1] https://arxiv.org/abs/2311.08105", "limitations": "Yes, the authors discuss limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nS1. Overall, the paper helps build a solid empirical understanding of the behavior of DiLoCo with respect to its key parameters. It provides useful insights into various aspects of its performance, clarifying how one might expect it to behave in practice compared to standard DDP.\n\nS2. The observation that DiLoCo outperforms DDP at $M=1$ is particularly interesting. This suggests that a combination of distinct local and global optimizers may offer benefits even in low-synchronization regimes.\n\n**Weaknesses**\n\nW1. While I acknowledge that the authors explicitly mention the limitation of not validating robustness across different pretraining datasets or model architectures, I believe it remains a critical point. It is important to assess whether claims surrounding some of the observed trends, such as the gains at low $M$, can be expected to generalize across training settings.\n\nW2. Some sections lack clarity or are not well written (see Questions).\n\nW3. Certain results and observations could benefit from more thorough justification and analysis (see Questions).", "was_revised": true, "final_justification": "I’m increasing my score to 5: my original concerns have been addressed, but I think broader impact may remain somewhat speculative, as pointed out by fellow reviewers.", "rebuttal": "We thank the reviewer for their detailed feedback. We believe that most of the points raised can be directly addressed via minor sentence rewriting. The remaining we have addressed to the best of our ability (and computation) below.\n\n**While the authors explicitly mention the limitation of not validating robustness across different pretraining datasets or model architectures, I believe it remains a critical point. It is important to assess whether claims surrounding some of the observed trends, such as the gains at low M, can be expected to generalize across training settings.**\n\nWhile we agree that generalizability is important, we are confident in our results for a variety of reasons.\n\n1. Our architecture and data were chosen explicitly to aid generalizability. We use the exact architecture style from NanoDO [1]. There have been a wide array of scaling work predicated on NanoDO, that we believe showcase the reliability of our results, including [2-9]. The training setup has generated many insights into transformer pre-training, enhancing our confidence in the results. Our primary training dataset, C4, has also been widely used across a huge variety of language model training settings.\n2. The utility of $M=1$ is intuitive when considering the empirical success of the lookahead optimizer [10]. We present a tuned version that incorporates momentum. Note that if our tuning is done well, it should do at least as well as the lookahead optimizer (which it does) and in fact does better. Given the success of SGDM over SGD, this is in retrospect, not surprising, but not something that has been shown in the literature before.\n3. We saw qualitatively similar results on both the C4 training dataset and the Dolma dataset (which we used for the overtraining ablations).\n4. The amount of compute needed to duplicate our results on a different set of model architectures would be quite intensive. The amount of experimental work that went into this was already quite significant, and to add more to a single conference paper would be a lot.\n5. The relative success of DiLoCo has been seen in many works. While ours adds a new scaling dimension, the results for a fixed model size mirror those in other works on DiLoCo [11, 14, 15].\n6. The formative works on scaling laws [12, 13] used similar architectures and datasets, and have been extremely generalizable.\n\nIn short, we have (short of doubling our computational expenditure) done our best to ensure generalizability of results.\n\n**Q1: Could the authors clarify why optimal outer learning rate increases with H?**\n\nThis is an open question. One potential explanation is that when you increase $H$, you decrease the number of outer optimization steps that occur throughout training. To keep the effective total progress the same, you would then need to compensate by increasing the outer learning rate. Additionally, by increasing $H$ the outer gradients may effectively undergo variance reduction, as they are now aggregations of more training steps, enabling larger learning rates.\n\n**Q2: What is the distinction between DiLoCo when H = M = 1 and DDP?**\n\nWe discuss this in L96-103 and L111-113 in the paper. The only difference when $M = H = 1$ is the use of an outer optimization step that incorporates momentum. If OuterOpt in Algorithm 1 is SGD with learning rate 1 and no momentum, we recover DDP when $H = M = 1$. This is also noted in [11]. We will mention this fact explicitly.\n\n**Q3: Why is the partitioning across the sequence level and not batch level for DiLoCo? For pertaining, I can think of cases where this may not be an issue. Maybe I am missing something here, but isn't it possible that this leads to DiLoCo seeing shorter sequences compared to DDP on average?**\n\nBy “partitioning at the sequence level” we mean that if a batch consists of $B$ sequences of tokens, we give each replica $B/M$ of those sequences. We said “sequence level” to indicate that we are preserving entire sequences, so our partitioning was not at the “token level”. In short, we do exactly the same partitioning that DDP does. We will make sure to clarify this.\n\n**Q4: It would be valuable to study whether a relationship exists between synchronization cadence and global batch size with respect to evaluation loss.**\n\nWe agree that this is an interesting study. Unfortunately, it is not possible to tune synchronization cadence $H$ and global batch size $B$ without also tuning inner learning rate $\\gamma$ and outer learning rate $\\eta$. Simultaneously tuning $H$, $B$, $\\gamma$, and $\\eta$ across model sizes is even more computationally intensive than the experiments in our paper (due to the extra parameter $H$). In the case of our studies on $H$ in Section 5, we did not re-tune $\\gamma$ and $B$ when varying $H$, as otherwise it would have been too expensive to run.\n\n**Q5: In Figure 15, it's not immediately clear why H=1 performs the worst. Is this because the optimizer essentially reduces to solely SGD + Nesterov?**\n\nWe believe that the answer is analogous to why the optimal outer learning rate increases with $H$. When $H=1$, the outer gradients are likely quite noisy (as noisy as the inner gradients), and so the momentum (on top of the inner AdamW optimization already happening) isn’t useful. By contrast, when $H > 1$ you get a form of variance reduction that we conjecture aids optimization.\n\n**Q6. Are Figures 5 and 6b conveying the same information? If not, it would be helpful to distinguish their respective roles more clearly.**\n\nWe discuss this in L177-188. We state explicitly “We first study whether the observation in Section 4, that $\\eta$ should be tuned independently of $N$, holds for other values of $H$”. We are happy to explicitly reference Figure 5 in this line, instead of implicitly. The key addition in Figure 6b is varying $H$.\n\n### References\n[1] Liu et al., “NanoDO: A minimal Transformer decoder-only language model implementation in JAX”, 2023.\n\n[2] Wortsman et al., “Small-scale proxies for large-scale transformer training instabilities,” 2023.\n\n[3] Everett et al., “Scaling Exponents Across Parameterizations and Optimizers,” 2024.\n\n[4] Lechao Xiao, “Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling,” 2024.\n\n[5] Schug et al., “Attention as a Hypernetwork,” 2024.\n\n[6] McKenna et al., “Scaling Laws for Differentially Private Language Models,” 2025.\n\n[7] Marek et al., “Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful,” 2025.\n\n[8] Gyorgy et al., “Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence,” 2025.\n\n[9] Roulet et al., “Loss Functions and Operators Generated by f-Divergences,” 2025.\n\n[10] Zhang et al., “Lookahead Optimizer: k steps forward, 1 step back,” 2019.\n\n[11] Douillard et al, “DiLoCo: Distributed Low-Communication Training of Language Models,” 2023.\n\n[12] Kaplan et al., “Scaling laws for neural language models,” 2020.\n\n[13] Hoffman et al., “Training Compute-Optimal Large Language Models,” 2021.\n\n[14] Jaghouar et al., “OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training,” 2024.\n\n[15] Jaghouar et al., “INTELLECT-1 Technical Report,” 2024."}, {"review_id": "q2C0OArf5A", "reviewer": "Reviewer_Web4", "summary": "This paper focuses on the DiLoCo (Distributed Low-Communication) approach for scaling LLM training in a highly distributed setting. This is an important topic due to desire to scale to increasingly large models while harnessing available compute that may exist in different data centers connected with limited bandwidth network connections.\n\nIn particular, the goal of this paper is to establish scaling laws for DiLoCo, considering the Chinchilla-optimal number of tokens for a particular model size.", "questions": "Q: To what extent do the results reflect more realisitic real-world communication overhead, especially when attempting to train across machines in different data centers?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nThe paper systematically assesses the loss on a held-out set after training, considering HellaSwag, Piqa, and Arc-Easy as zero-shot downstream evaluation tasks.\n\nThe loss is varied over model sizes up to 2.4B and over relevant hyperparameters in DiLoCo, including inner and outer DiLoCo learning rate, number of DiLoCo replicas, and batch size.\n\nThe derived scaling laws were tested empirically for training 4B and 10B models.\n\nThe paper confirms some of the advantages of DiLoCo.\n\n## Weaknesses\n\nUnclear to what extent the scaling law will benefit genuine frontier LLM developers, given some of the simplifications made in this study.\n\nScaling laws have been derived for a number of architectures and training setups, so the main contribution here is to do this for DiLoCo. In this sense, the study is not particularly creative or exciting. Still, such work can be useful.", "was_revised": true, "final_justification": "The work still does not appear particularly creative, but in their response, the authors addressed some of the concerns I had about the utility of this work for potential readers.", "rebuttal": "While we appreciate the positive feedback, we are uncertain what concrete changes the reviewer believes would improve the paper, and would appreciate clarification.\n\n**Unclear to what extent the scaling law will benefit genuine frontier LLM developers, given some of the simplifications made in this study.**\n\nWe do not know what simplifications the reviewer is referring to. Additionally, we wish to note that “benefitting frontier LLM developers” is not the acceptance criteria for work at NeurIPS.\n\nThat being said, we believe the evidence already shows that our work would be useful to such an audience. For example, [1, 2] constitute open-sourced LLMs trained via DiLoCo already released to the public. If someone wants to scale to larger models, we believe that our work helps them identify how to scale with DiLoCo. Similarly, LLM-focused training libraries like `torchft` and `torchtitan` are already collaborating on how to integrate DiLoCo in order to make LLM training more fault tolerant and scalable from a systems perspective [4].\n\n**To what extent do the results reflect more realistic real-world communication overhead, especially when attempting to train across machines in different data centers?**\n\nWe believe our results are extremely useful across a variety of real-world systems. We give 4 specific reasons.\n\n1. Most of our results are completely independent of the actual bandwidth connection across data centers. The fact that we know how to set hyperparameters algorithmically at larger model sizes $N$ and number of datacenters $M$ is helpful for researchers and practitioners writ large.\n2. The simulated wallclock times we give have 3 archetypes, as we discuss in the paper: (1) high-bandwidth (400 gigabits / s), (2) medium-bandwidth (100 gigabits / s), (3), low-bandwidth (10 gigabits / s). In fact, we show that DiLoCo is faster across all three, due to increasing the critical batch size. Thus, a huge range of realistic settings are benefitted by DiLoCo.\n3. Our low-bandwidth is actually an overestimate of the bandwidth used in the DiLoCo training in [1] (see Figure 8, which says the bandwidth they had ranged between 117 and 935 megabits / s). \n4. As discussed in [3], DiLoCo can significantly improve fault tolerance by enabling greater asynchronicity. Moreover, LLM development libraries like `torchtitan` and `torchft` are already investigating the use of DiLoCo to improve fault tolerance in realistic training settings [4]. Thus, there is a huge amount of utility in showing how DiLoCo scales to large models, where fault tolerance is more important.\n\n### References\n\n[1] Jaghouar et al., “OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training,” 2024.\n\n[2] Jaghouar et al., “INTELLECT-1 Technical Report,” 2024.\n\n[3] Liu et al., “Asynchronous Local-SGD Training for Language Modeling,” 2024.\n\n[4] Rice & Huang, “Fault Tolerant Llama: training with 2000 synthetic failures every ~15 seconds and no checkpoints on Crusoe L40S,” PyTorch blog, 2025."}, {"review_id": "Rc7AnySrZi", "reviewer": "Reviewer_Px8m", "summary": "The paper investigate how DiLoCo, a periodic synchronization variant of Local SGD/FedAvg scales when training LLMs under a fixed compute budget. The author adopt the scaling law and conduct empirical studies for evaluation. Key empirical findings are that DiLoCo achieves lower loss than data-parallel training, supports much larger optimal batch sizes, and therefor yields shorter idealized wall-clocak times een on high bandwidth clusters. \n\nEven though the both LLM and scaling laws are timely topic, I think this paper has marginal contribution given the fact that it only studied one particular method, and the fact author adopt the well-established scaling law formulation.", "questions": "- Generality of laws: do the hyper-parameters exponents change for MoE or multilingual corpora?", "limitations": "Please see above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- through empirical sweeps \n\n**Weakness**\n- the paper only studies one method (DiLoCo), without compare the scalaing laws of other method that are also mentioned in the introduction: FL and data-parallel training. It would be better to have clearly study the difference among them. Also, there are other communication-efficient methods that are not covered in the paper. However, this kind of study suits more of a benchmark paper instead of main track paper in my opinion.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' responses and discussions with other reviewers and AC.", "rebuttal": "We would like to correct some misconceptions in the review (especially regarding data-parallel training). Also, we aren’t sure concretely what the reviewer is suggesting could be improved to warrant acceptance, and would appreciate clarification.\n\n**The paper only studies one method (DiLoCo), without compare the scaling laws of other method that are also mentioned in the introduction: FL and data-parallel training.**\n\nWe respectfully disagree with the reviewer's assertion. We did compare DiLoCo with data-parallel training. We do this repeatedly, even re-deriving scaling laws for data-parallel training in our own setting to make sure that our comparisons are as fair as possible.\n\nWe’re also not sure what the reviewer is asking for with respect to federated learning (FL). This is a very broad area, not just one algorithm, that could encompass a huge range of things. We mention FL in our introduction because algorithmically, the two areas often use similar optimization algorithms. But most FL settings require things like private heterogeneous client data and privacy safeguarding mechanisms (such as differential privacy), making them very different from LLM pre-training tasks, and not comparable. Most realistic FL is also restricted to very small models due to the need to keep model training on-device.\n\nIf the reviewer is asking for us to compare DiLoCo to FedOpt, then would like to note that DiLoCo is FedOpt when the inner optimizer is AdamW and the outer optimizer is Nesterov, which [1] already showed convincingly is the best parameterization of FedOpt when training LLMs.\n\n**Also, there are other communication-efficient methods that are not covered.**\n\nOur focus on DiLoCo is intentional. It is demonstrably more effective at reducing communication costs than other LLM training methods [1]. DiLoCo and its variants are also already being employed to train large LLMs in a globally distributed manner, both in papers and in actual model deployments (see [2, 3, 4, 5]). DiLoCo is being increasingly used as a starting point for new methods in communication-efficient LLM training [6, 7, 8]. Finally, unlike methods such as compression which do not fundamentally alter the training dynamics, the reason DiLoCo is singled out is that it does actually change the underlying algorithmic dynamics. While scaling analyses of things like compression can be done, they rely in large part on their algorithmic similarity to uncompressed training.\n\n**This kind of study suits more of a benchmark paper instead of main track paper.**\n\nWe respectfully disagree that scaling law analyses are only suited for benchmark papers. A brief skim of the NeurIPS 2024 papers reveals at least 10 accepted papers working on topics related to scaling laws. Moreover, our work is not simply a benchmarking of methods. It includes novel scaling law methodology (e.g. jointly scaling based on model size and number of model replicas), novel algorithmic insights (e.g. the fact that DiLoCo with a single model replica is a sensible and performant method), and novel axes of comparison (e.g. a focus on horizontal scalability, batch size, and influence of hyperparameters across values of $M$).\n\n**Do the hyper-parameters exponents change for MoE or multilingual corpora?**\n\nWe do not know yet whether the scaling laws in our work change when training MoE models or on multilingual data. However, it is computationally infeasible to study all of these in the same paper. These took years and many separate studies to nail down for data parallel training (e.g. scaling laws for MoEs were published only after the Chinchilla scaling laws). Performing scaling law analyses on all of these axes simultaneously is just not possible in a single conference paper. While these are great follow-up studies, they are things that are made easier by our work, not something we believe can be studied all at once.\n\n### References\n\n[1] Douillard et al., “DiLoCo: Distributed Low-Communication Training of Language Models,” 2023.\n\n[2] Jaghouar et al., “OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training,” 2024.\n\n[3] Jaghouar et al., “INTELLECT-1 Technical Report,” 2024.\n\n[4] Sani et al., “The Future of Large Language Model Pre-training is Federated,” 2024.\n\n[5] Kim et al., “HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training,” 2025.\n\n[6] Therien et al., “MuLoCo: Muon is a practical inner optimizer for DiLoCo,” 2025.\n\n[7] Beton et al., “Improving the Efficiency of Distributed Training using Sparse Parameter Averaging,” 2025.\n\n[8] Qi et al., “DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster,” 2025."}, {"review_id": "EbikNlwz9I", "reviewer": "Reviewer_6yd8", "summary": "This paper studied the scaling law of pretraining using DiLoCo and showed DiLoCo also follows the similar scaling law of standard DDP training along with a few emperical findings.", "questions": "At line 185 the authors mentioned an counter-intuitive observation: \"outer learning rate increases with H\". Do you have any potential explanation for it?", "limitations": "as is mentioned earlier, the efficacy is limited to DiLoCo with low network bw, which is typically inter-datacenter and less practical for majority of ML practioners", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n* The study is comprehensive, covering the emperical similarity to DDP as well as its own unique properties. It also explores the effect of 1st order hyperparameters like outer learning rate of DiLoCo, batch-size, etc. It can serve as a good scaling law handbook for DiLoCo practioners.\n\n* It showed DiLoCo can reach similar PPL as DDP with a much less wall-clock time, due to less communication and larger batchsize\n\nWeakness:\n\n* The work is a re-study of well-known scaling laws like Chinchilla/OpenAI's work, but focused on DiLoCo. The innovation is limited and significance may be limited to DiLoCo practioners\n\n* The wall-clock time saving is only sigfinicant in networks with low bandwidth, e.g., a-b in Figure 4. For network with modern BW, the gain seems mainly from larger batchsize due to outer optimizer, not the sparse synchronization.", "was_revised": true, "final_justification": "Convinced by the rebuttal that this work provide valuable insights to large scale LLM training.", "rebuttal": "We would like to correct some misconceptions in the review (especially regarding prior scaling laws). Also, we aren’t sure concretely what the reviewer is suggesting could be improved to warrant acceptance, and would appreciate clarification.\n\n**The work is a re-study of well-known scaling laws like Chinchilla/OpenAI's work, but focused on DiLoCo.**\n\nWe respectfully disagree. We do not re-derive the scaling laws of [1, 2]. Rather, we see how the performance of DiLoCo scales with respect to model size $N$ and number of model replicas $M$, simultaneously. We compare it to data-parallel training in the process (and do scaling analysis there), but this is because we want to make sure our results are as accurate as possible. The new content is all in the DiLoCo-specific scaling laws, which are not in [1, 2] whatsoever.\n\nMost of our work and insights are simply not possible to analyze in data-parallel settings. Things like how DiLoCo changes critical batch size, how it affects overtraining budgets, and how hyperparameters should scale across $M$ are simply not axes that are pertinent to data-parallel training. We use some of the same ideas as in [1, 2] (such as fitting power laws to empirical data), but there is a huge body of work on scaling laws that builds on top of the scaling laws of [1, 2] in analogous ways. Moreover, we use unique methodology, like the joint fitting of model size $N$ and model replicas $M$ jointly.\n\n**The innovation is limited and significance may be limited to DiLoCo practitioners.**\n\nOur work is primarily concerned with LLM training under communication constraints. This is a growing area of work, and our focus on DiLoCo within that area is intentional. DiLoCo is demonstrably more effective at reducing communication costs than other LLM training methods [1]. DiLoCo and its variants are already being employed to train large LLMs in a globally distributed manner, both in papers and in actual model deployments (see [2, 3, 4]). In short, DiLoCo is an empirically successful method that already has a large amount of work using it at scale and developing new variants (see [6, 7, 8, 9], as well as others). As such, we believe our work is entirely appropriate for this conference.\n\nMore broadly, NeurIPS is a very large conference. There is plenty of research published at NeurIPS which is primarily important to the sub-community that works on it. For example, research on variational inference methods is primarily of interest to the variational inference community. That’s not a limitation of research in that area, it’s just because researchers specialize and have distinct areas of focus.\n\n**The wall-clock time saving is only significant in networks with low bandwidth, e.g., a-b in Figure 4. For network with modern BW, the gain seems mainly from larger batchsize due to outer optimizer, not the sparse synchronization.**\n\nWe’re confused as to why this finding, which we believe is an exciting insight in our work, is listed as a weakness. The fact that DiLoCo helps speed up training even over high bandwidth networks is (1) a novel insight and (2) has important implications for LLM training.\n\nWe also disagree that LLM training should only consider high bandwidth settings. The high bandwidth setting in Figure 4 is not the type of bandwidth that connects data centers in different parts of the world. Figure 4a is much closer to what you would see in such settings. In fact in [4] the bandwidth available was actually lower than our “low-bandwidth” setting. In short: the bandwidth available across countries is not high enough to train large LLMs purely via data-parallel training, which is why DiLoCo is so promising in realistic LLM training across datacenters.\n\n**At line 185 the authors mentioned an counter-intuitive observation: \"outer learning rate increases with H\". Do you have any potential explanation for it?**\n\nThis is still an open question. One potential explanation is that when you increase H, you decrease the number of outer optimization steps that occur throughout training. To keep the effective total progress the same, you would then need to compensate by increasing the outer learning rate. Also, by increasing H the outer gradients may effectively undergo variance reduction, as they are now aggregations of more training steps, and less variance enables larger learning rates.\n\n### References\n\n[1] Kaplan et al., “Scaling laws for neural language models,” 2020.\n\n[2] Hoffman et al., “Training Compute-Optimal Large Language Models,” 2021.\n\n[3] Douillard et al., “DiLoCo: Distributed Low-Communication Training of Language Models,” 2023.\n\n[4] Jaghouar et al., “OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training,” 2024.\n\n[5] Jaghouar et al., “INTELLECT-1 Technical Report,” 2024.\n\n[6] Sani et al., “The Future of Large Language Model Pre-training is Federated,” 2024.\n\n[7] Kim et al., “HALoS: Hierarchical Asynchronous Local SGD over Slow Networks for Geo-Distributed Large Language Model Training,” 2025.\n\n[8] Therien et al., “MuLoCo: Muon is a practical inner optimizer for DiLoCo,” 2025.\n\n[9] Beton et al., “Improving the Efficiency of Distributed Training using Sparse Parameter Averaging,” 2025."}]

Author rebuttal:

Meta-review: The paper focuses on DiLoCo (Distributed Low-Communication), a periodic-synchronization optimization algorithm for scaling LLM training, and aims to fill a critical gap: systematic empirical characterization of its hyperparameter behavior and scaling laws relative to standard Distributed Data Parallel (DDP).

Reviewer Px8M and 6yd8 recognized a few weaknesses, but in short sentences and with low confidence. Reviewer am9z and Web4 give more positive feedback.

Post-rebuttal, the authors effectively addressed all critical reviewer concerns by clarifying the scope, improving readability, and justifying their focus on DiLoCo’s depth over breadth. While the paper is not theoretically groundbreaking, its empirical rigor and actionable guidance for LLM training at scale make it a valuable addition to the conference. Considering the overall positive scores, I recommend acceptance.
Decision: Accept (spotlight)