Streaming DiLoCo with overlapping communication
Authors: Arthur Douillard
Yani Donchev
J Keith Rush
Satyen Kale
Zachary Charles
Gabriel Teston
Zachary Garrett
Jiajun Shen
Ross McIlroy
David Lacey
Alexandre Rame
Arthur Szlam
MarcAurelio Ranzato
Paul R Barham
Venue: COLM 2025 (2025)
Keywords: distributed training
large-scale
llm
TL;DR: Distributed training where only a subset of the outer gradients is communicated

Abstract: Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of data exchange. Recently, algorithms like DiLoCo have relaxed the constraint that all devices need co-location: accelerators can be grouped into ``workers'', where synchronizations between workers need only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwith across workers. We show experimentally that by properly combining these modifications we can distribute training of billion-scale parameters and attain models of similar quality as before, but reducing required bandwidth by a factor of up to two orders of magnitude.

Published as a conference paper at COLM 2025
Streaming DiLoCo with overlapping communication
Arthur Douillard & Yani Donchev, Keith Rush, Dave Lacey, Ross Mcllroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc'Aurelis Ranzato & Paul Barham Google DeepMind
Satyen Kale' Zachary Charles & Zachary Garrett Google Research
Gabriel Teston
Google
Abstract
Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need be co-located using low-latency high- bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo (Douillard et al., 2024a) have relaxed such co-location constraint: accelerators can be grouped into "workers", where synchronizations between workers only occur infrequently. This in turn means that workers can afford being con- nected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations re- quire all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across work- ers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach sim- ilar quality as before, but reducing required bandwidth by two orders of magnitude.
1 Introduction
Scaling deep learning has led to significant leaps in capability (Team et al., 2024; OpenAl et al., 2024; Grattafiori et al., 2024). Although neural architectures have evolved over the past decade, the standard approach to optimization remains essentially unchanged from the days of Alexnet (Krizhevsky et al., 2012). Practitioners use minibatch stochastic gradient descent, with backpropagation through the model's layers to compute the gradients. As in Alexnet, which already combined two hardware accelerators for parallel training, models are trained with multiple hardware accelerators.
However, modern training runs, for example for large language models (LLM), may use tens of thousands of accelerators, and this number increases year after year. Building and maintaining a data-center that can co-locate that many accelerators expensive and leads to increasingly complex engineering challenges. Beyond the physical infrastructure, orchestrating the passage of gradients, parameters and intermediate states between these devices at each optimization step, while keeping all devices fully utilized is technically challenging from software engineering perspective. Furthermore, the more devices that
AAthhr and Yani contributed equally.
'Currently at Apple.

---

Published as a conference paper at COLM 2025
used for each synchronous training step, the more chances there are that one of them fails, risking halting training, or introducing subtle numerical issues.
Recent publications (Douillard et al., 2024a; Jaghouar et al., 2024a), building work by McMahan et al. (2017), have demonstrated that the co-location requirements of all accelerators be loosened. These methods allow accelerators are grouped into several "workers" with fast bandwidth intra-worker but with slow bandwidth inter-workers. The basic approach is to allow each worker to continue training for many minibatches, independently of the other workers; and then synchronize the parameters of the workers after number of these "inner" steps. The synchronization, in its simplest form (McMahan et al., 2017), is to average the parameters of the workers (Wortsman et al., 2022); but more sophisticated methods (Huo et al., 2020; Reddi al., 2021) use the workers' parameters to form pseudo-gradiens to update the shared parameters. The details of the formulation of the weight synchronization is important for machine- learning efficiency; see Section A.1.
training when
However, in these approaches, the synchronization typically requires an all-reduce operation which fully synchronizes the model parameters on some step. This all-reduce results in two main issues: 1) a large peak bandwidth, and 2) a blocking of the workers while they wait to receive updated weights, In this work, dubbed Streaming DiLoCo, propose three modifications to these approaches to practically reduce the peak bandwidth and mitigate worker-blocking without loss of learning efficiency:
Contribution 1: Synchronization.
We synchronize subsets of parameters on schedule, rather than all parameters at once. This contribution reduces the peak required bandwidth.
Contribution 2: Overlapping.
We overlap worker computation and communication of synchromizations. This contri- bution increases the tolerated latency of communication.
Contribution 3: Quantization.
We compress the outer gradients to four bits per parameters without loss of perfor- mance. This contribution reduces the total amount of bits exchanged.
We show experimentally that our model, Streaming DiLoCo, is strictly superior to the original DiLoCo (Douillard et al., 2024a), and achieves similar performance to the bandwidth- costly data-parallelism. Since we attain the same quality at negligible bandwidth, we consider our approach an important stepping stone towards form of distributed free lunch.
2 Model
We recommend the reader unaware of the previous literature on FedOpt (Reddi et al., 2021) and DiLoCo (Douillard et al., 2024a) read our provided context in the appendix in subsection A.1.
For all algorithms, we denote the model parameters as 0. We use the superscript notation g(t) to indicate the parameters at a given step I, and the subscript notation On to denote
particular shard of the DiLoCo replica. For example, e!!) indicates the parameters of DiLoCo replica at step If no subscript is used, the parameters replicated across DiLoCo replicas. Note that it is possible for parameters not be replicated and yet to be of the same value.
2

---

Published as a conference paper at COLM 2025
Parallel Inner Optimization
Parallel Inner Optimization
Model randomly initialized
Optiiizttion
Optimization
Figure 1: Streaming DiLoCo: each replica trains independently for dozen of inner opti- mization steps, and then synchronize single fragment during outer optimization. In this figure, there are M = 4 replicas with p (1,2,3) fragments. Each fragment can be made of several transformer layers. Note that this figure only showcases the streaming partial updates (subsection 2.1) and not the quantized communication overlapping (subsection 2.2 and 2.3).
2.1 Streaming partial updates
Instead of communicating the full outer gradient vector (A Vm E {1,...M)) every H steps, we propose share only fragment p of (AM)p more frequently, as highlighted in Figure 1. There is huge possible space of choices for these fragments and specification of "more frequently" here we consider the simple partition of our network into fragments made of several transformer blocks. Specifically, we study two fragment patterns, as shown in Fig- ure in the appendix: 1) sequential where each fragment comprises consecutive transformer blocks and 2) strided where each fragment is composed of interleaved transformer blocks. We will demonstrate in subsection 3.3 that the algorithm robust to the particular choice of fragments. Since the stride version offers in practice slightly better compute utilization (less time spent communicating instead of computing), we will use as the default choice in our experiments. As we increase model scale, the fragment definition (e.g., how many transformer blocks comprise fragment) is maintained, which means that larger models have more fragments.
The resulting algorithm in shown is Algorithm (and contrasted with the original version shown in Algorithm 2), where only fragment p of the replica is shared. We denote fragment with new lower script, thus elly is the parameters of fragment p of the replica at step time t.
The Streaming DiLoCo's inner optimization (L3-5 of Algorithm 1) is identical to DiLoCo's (L3-5 of Algorithm 2). However, the outer optimization (L12) is done per fragment. Tf fragment p satisfies the condition t H then it is synchronized. In this way, each fragment will always do H steps before being synchronized, but overall the model is synchronizing some fragment more frequently than every H steps. For example, with H = 100 and = 2 fragments, the first fragment will be synchronized at step
where tp time offset fragment-dependent,
0); the second fragment will be synchronized step 150, 250, ... (tp=2 = 50). While in practice, given an equal H, streaming DiLoCo communicates more often than DiLoCo, the peak communication is reduced by a factor of Irl/L with |p] the size of a fragment in layers and L the total number of layers.
100, 200, ... (fp=1
2.2 Overlapping communication with computation
To further maximize the time spent doing computation v.s. communication, we propose to overlap the communication of the outer gradient fragment with the inner optimization computation; the overlap happens with strictly positive T in Algorithm 1, lines 10-12. At
For example, an extreme version might be to send a constant bitstream of random choices (according to some optimization-useful distribution) of parameters.
3

---

Published as a conference paper at COLM 2025
Algorithm 1 Streaming DiloCo
Require: M replicas
Require: Number of inner steps H
Require: Fragments E (1,..., . P) with their respective synchronization offset 'p
Require: Model replicas (0()) (
Require: Inner overlap delay H
Require: Data shards (D., }
Require: Optimizers InnerOpt and OuterOpt
1: parallel for replica m = 1... M do
2: for step = 1... do
3: x~ ~ ~Dm
4: C+ fxx )
(t) InnerOpt 1) cc
5:
6:
if 3p s.t. f- tpH 0 then
Am.p +- 0 ( 0m.
7:
8:
Ap t- send M Em= (Amp)) }]
9:
if 3p s.t. t tp - TH == then
block block-receive/A. (t-r)
10:
a(t) + outorOpt (0) - Ap (1-1))
e f- u mip + (1 - a 0000
12:
the beginning of outer step t + instead of waiting for the communication of the fragment block-receive, we immediately start the new round of optimization. After T inner steps (L3-5), we block-wait for the exchanged fragment (L10), apply the outer optimizer on the mr,pt (tt(r))) equivalent
previously synchronized fragment (0 T 17) and merge it with the currently optimized
communication between replicas, a = 0 discards any updates done in the first 7 steps on the fragment p, and a = 0.5 does uniform average between the local fragment parameters and the globally shared one.
fragment with a mixing factor
2.3 Low-precision outer gradients
The previous methods, streaming and overlapping communication with computation, reduce peak bandwidth and wall-clock time, respectively. To reduce total amount of bits exchanged, we use lower-precision in the outer gradients exchanged by workers (while still using FP32 for computing gradients), up to float with 4 bits (1 sign bit, 3 exponent bits, and mantissa bit) called E3M0 (Agrawal al., 2024). Across wide variety experiments, we found no sign of performance regression when employing such low precision numbers during communication, even at the billion scale. This compression is applied when sending each replica's unreduced outer gradients to miminize the amount bits communicated (L8 of Algorithm 1), but once received by a replica, importantly, the accumulation is done in FP32 for stability.
3 Experiments
We run experiments demonstrating the compute utilization benefits of our approach in a bandwidth and compute simulation in Section 3.1. In Section 3.2 we show in practice the model learning outcomes. Finally, in Section 3.3 we show results with variations of the three main contributions of the paper, ablating their relative importance.
4

---

Published as a conference paper at COLM 2025
Euta Rough!
FRO Risalt
01835
FRD Ritsal
20830
Steaming
Srcat big DCC LeeeR Bull NeT25t9)
018-1
Figure 2: Simulation of a schedule interleaving forward passes (in blue), backward passes w.r.t. activations and parameters (resp. in light and dark green), and (outer) gradient reduction (in purple).
(a) 1B parameters model.
(b) 10B parameters model
(c) 100B parameters model
Figure 3: Compute Utilization simulated across a range of bandwidth. A compute uti- lization of 0.8 means 80% of the time is spent in computation, and 20% in communication Our best method reaches a compute utilization of 95% for models 1B, 10B, and 100B with bandwidth roughly constant between and 5 Gbit/s. Data-Parallel on the other hand requires 100, 200, and 300Gbit/s.
3.1 Compute utilization
To highlight the impact of our contributions in a controlled setting, we built a simulator to estimate the compute utilization of each method: how much time is spent doing com- putation v.s. communication. The simulation is a DAG with four different types of nodes as seen in Figure 2: forward in blue, backward w.r.t activations and parameters in green, (outer or not, for resp. DiLoCo and Data-Parallel) gradients reduction in purple. Each node represent single layer. Therefore, the total number of nodes, for a single step, is 4 x I- (because we don't need the backward w.r.t activations the first layer). The overall training is represented by DAG made of such nodes. We use this simulator to estimate the compute utilization of each method: how much time is spent doing computation v.s. communication. Ideally this number is close to 1.0: No time is spent waiting for communication. It is more useful than just reporting the reduction in data transferred because our overlapping method (subsection 2.2) reduces the latency while keeping the amount of data exchanged constant.
In Figure 2, we report:
Data-Parallel: the baseline which communicates gradients of the full model at every
step;
DiLoCo: which communicating outer gradients of the full model once in while (in this example, every 2 steps); Streaming DiLoCo: which communicates outer gradients only for subset of the
model (here the fragment size is single layer and there are two fragments) every
H = 2 steps;
Streaming DiLoCo with overlapping communication and computation: This is similar to the above but gradients sent across workers are only needed after steps (in this example 1).
5

---

Published as a conference paper at COLM 2025
Method
GiB/s
10 GiB/s
100 GiB/s
1000 GiB/s
Data-Parallel
21 days
2 days
10 hours
hours
DiLoCo
12 hours
6 hours
hours
5 hours
Streaming DiLoCo with overlapped FP4 com.
6 hours
5 hours
hours
5 hours
Table 1: Wallclock time for Data-Parallel, DiLoCo, and our method for 1B model spending 0.1s of computation per step and doing 186,250 steps under various bandwidth profile..
DNLCCO
Soccaming BluoCo evertapodd com,
3.4
Stowaring Bkuate ovuntapoud 1914 com, 1=-000
32
0.50
0.45
belwouns
2.6
0:30
Data Farslel
2.4
DilaGs No30
Greamingg Di with exereepeed It:10
Streeminn LoCo overlapped H--30
1.5627
R.4677
4w38 LAk18 4.7M19 1.9620 training flops
2621
15:17
0,4017
4:13
10c20
2021
traninng
(a) Evaluation loss on C4.
(b) HellaSwag accuracy.
Figure 4: Scaling models from 35M to 4B on C4.
The simulated compute utilization (CU) depends on some factors, listed as columns in Table 5. For the model scales 10B and 100B, we estimate step time (pure compute) based on the flops profile, a reasonable MFU (40%), and hardware theoretical flops per seconds. We simulate training of each method, across three scales (1B, 10B, and 100B) under various bandwidth profiles Figure 3.
We make several observations: 1) Streaming DiLoCo (in green) improves the CU of DiLoCo (in orange) despite exchanging as much data, because it reduces the latency by splitting the communication the outer gradients across fragments. 2) only overlapping communication with computation can reach full 100% compute utilization. 3) the required bandwidth can become lower as the model scale gets larger when overlapping communication with computation, because longer compute step time (forward & backward) will provide more time to perform the synchronization across workers.
The last point may seem counter-intuitive at first glance, but is the main advantage of our method, exploiting the square-cube law of distributed training (Ryabinin et al., 2023) where computation scales worse than communication (O(n²) vs O(n²) for a square matrix n x n). We provide in the appendix, in Figure 16, the simulated compute utilization for 100B model across various step time.
Remark: simulation vs reality. Of course this is only simulation of what we expect to happen in practice. Such simulation not perfect because for instance we consider only the bandwidth between datacenters and not the local bandwidth between devices. We believe however, that this still a useful tool? to estimate device utilization. We show in Table the real wallclock under various bandwdith for the 1B model only.
Remark:
pipeline parallelism. We focus here solely distributing the data axis and don't include pipeline parallelism. Our goal, achievable with the former, is to decouple instances of the model to facilitate fault tolerance. The latter isn't best suited for that purpose.
2https://en https////eeenwwiieeiii.oooo wikkpeddia.orr//wik//Bonnniii paradox

---

Published as a conference paper at COLM 2025
Method
loken Budget
Hours w/ Ghrlsss
Hours GGhilk/ss
leniiytte exchanged
Iiaal
HellsSwing
Fasy
35B
0.67
100
2.57
62.19
67.35
60:42
Dats-Paralled
10:8
49.75
44313
64:11
DlLoCo
250
0,67
0,88
1,30
2,66
42.05
67.46
38.42
with wwrtlppee TP4
1038
27
3.5
4.42
2.51
09.88
08.06
64/13
communication
2008
6.5
825
11:05
2.45
54.24
71.38
41.92
Table 2: Overtraining Data-Parallel and our method on Dolma with 1 billion parameters model.
3.2 LLM Scaling Experiments
We perform experiments with Chinchilla architecture (Hoffmann et al., 2022). Fol- lowing Wortsman et al. (2023) and Jaghouar et al. (2024a), we use QKNorm (Henry et al., 2020) and Z-loss (Chowdhery et al., 2023) with factor of 1e - 4 to stabilize training. We report in Table 3 the architecture hyperparameters and token budget at each scale. Unlike recommended in Post-LLcal SGD (Lin et al., 2020), we train all our models from scratch. The main hyperparameter of DiLoCo is its outer learning rate; we tuned it to be optimal at small scale at 0.4, and kept fixed across all scales. Likewise, for the simplicity, and to show that Streaming DiLoCo is drop-in replacement of DiLoCo, we used the same outer learning rate, without further hyperparameters tuning.
Except mentioned otherwise, we use the C4 dataset (Raffel et al., 2020) and train models from 35 million to billion parameters, all with a sequence length of 1,024. Each scale is trained with the chinchilla-optimal number of steps. We use 2 DiLoCo replicas, each of them performing FSDP (Zhao et al., 2023) across their respective closely located devices.
Our code is based on NanoDO codebase (Liu et al., 2024b) and uses DrJax (Rush et al., 2024) to parallelize inner steps across replicas.
3.2.1 Scaling
We perform scaling experiments on C4, with models ranging from 35 millions parameters to 1 billion parameters, all with sequence length of 1,024. For Data-Parallel and Streaming DiLoCo with H = 100, we also provide results on a 4 billion parameter model. At each scale, we use the Chinchilla-optimal number of steps. We highlight in Figure the evaluation loss (lower is better) and HellaSwag (Zellers et al., 2019) accuracy (higher is better).
First, we observe in Figure 4 that Data-Parallel (in blue), DiLoCo with H 30 inner steps (in orange), and Streaming DiLoCo with H = 30 (green) perform all similarly across both loss (at 1B parameters, respectively 2.49, 2.49, and 2.48) and accuracy metrics (resp. 46.6%, 46.5%, and 46.6%). Streaming DiLoCo with more inner steps H = 100 (in red) has slightly worse performance initially but use significantly less bandwidth and the loss improves proportionally better we scale: scaling law slope for Data-Parallel is -0.13149 while- 0.13539 for Streaming DiLoCo. We report in the appendix Table 6 all metrics, and include two more downstream tasks: Piqa (Bisk et al., 2020) and Arc-Easy (Clark et al., 2018). Moreover, Table 7 considers an increased number of DiLoCo replicas.
3.2.2 Overtraining on Dolma
The previous experiments were performed on C4 dataset using the chinchilla-optimal number of tokens. Using 1 billion parameter model, this yields a token budget of 25 billion. However, language models are now usually overtrained (Gadre et al., 2024). Therefore we perform comparison of Data-Parallel baseline vs our full model (streaming DiLoCo with overlapped FP4 communication) on the Dolma dataset (Soldaini et al., 2024) with 1 billion parameter model and with a token budget of 25, 100, and 250 billions tokens (resp. 1,9e20, 7.6e20, and 1.9e21 flops) using a sequence length of 2,048. In that larger, more realistic setting, we set the number of inner steps between synchronization to H 100 to further minimize communication.
We report results in Table 2, and note that both our method and the baseline perform simi- larly w.r. loss and accuracy on downstream tasks (HellaSwag, Piqa, Arc-Easy). In addition
7

---

Published as a conference paper at COLM 2025
2.74
pha-c (dfocant 5
2.73
FecDropout
Dore
2.85
Tap
2.71
2.80
2.70
kp-k
srop rate +50%
2.50
2.70
Ipdl
Nr.550%
fb32
30 overlapped
0.2
0.4
0.6
6.8
Figure 5: Varying the number of over- lapped inner steps for a = {0,0.5), A larger 7 requires significantly lower bandwidth, see also Figure 13.
Figure 6: Compressing the outer gra- dients with either value dropping (Fed- Dropout, Dare) using lower-precision floating point numbers.
of being neutral in term of ML performance: 1) the amount of bits exchanged between non- colocated devices over the course of training 400x higher for Data-Parallel; 2) the peak bandwidth (amount of bits exchanged at given moment) is reduced by and 3) while Data-Parallel ideally hopes for a 0 second latency when communicating, our overlapping scheme allows latency long full forward /backward pass, which is several seconds at large scale. For those reasons, we believe our work is step towards truly "distributed free lunch".
num Jayeaayes- 24 8x; fragment size-3
3.3 Ablations
To ablate the importance of each component of Streaming DiLoCo, we perform all our ablations model of size 500 million parameters using the C4 dataset with the chinchilla- optimal number of steps and a token budget of 11 billions.
We split our ablations section in three parts, corresponding to the three improvements brought in this paper: namely 1) Streaming synchronization, 2) overlapping communication with computation, and 3) finally quantized communication.
Number of synced layers per fragment ablate in Figure in the appendix the fragment size, i.., how many transformer blocks are included in a fragment. Based on this analysis, we choose fixed fragment size of 3 layers, striking a desirable trade-off between ML performance and reduction of peak bandwidth (for which the smaller the fragments the better). We also consider whether to have a sequential or strided pattern (see illustration in Figure 8 for reference). We choose the latter for several reasons: 1) ML performance is slightly better for the fragment size we consider, 2) deeper networks, with a small fragment size (e.g. 3), should benefit more from striding by spreading out up-to-date synchronized layers across the full depth of the network. Finally, 3) slightly improves the compute utilization (see Figure 10) by allowing better overlapping schedule, as clearly seen in Figure in the appendix.
Overlapping We first vary the number of inner steps, T we use to overlap communication with computation (see subsection 2.2). Figure 5 shows results varying from to 20, with a (discarding any intermediary inner updates) and a = 0.5 (uniform merging). We can see that the degradation in evaluation loss is negligible up an overlap of 10 inner steps (< 0.2%). By checking the compute utilization of 100B model in Figure 13, we observe little gain in compute time after an overlap inner steps. Therefore, we advise practitioners to limit the overlap to 5 inner steps. In main experiments, we used step for simplicity.
Compressing outer gradients We ablate in Figure 6 two ways of compressing the outer gradients: either by setting to zero some values (FedDropout (Wen et al., 2022), Dare (Yu et al., 2024), and Top-K selection) by lowering the precision. In all cases, accumulate
8

---

Published as a conference paper at COLM 2025
the outer update in float32. We report the evaluation loss on C4. Interestingly, lowering the precision, float32 to float4 does not affect the performance, while setting some values to zero is significantly worse, particularly when zero-ing out at random. We also considered Ties- Mres-negging's s pruning method (Yadav et al., 2023) but preliminary experiments showed it also underperformed; however this approach might become advantageous with larger number of replicas M.
4 Related Works
Federated learning / local SGD. While model merging (Wortsman et al., 2022) proposes to combine several models once, FedAvg (McMahan et al., 2017) and Local SGD (Stich, 2019) do it multiple times with the goal of reducing bandwidth requirements: they operate by performing local training (typically via SGD) across workers for some number of steps before doing some kind of synchronization of worker parameters, or aggregation of parameters across workers. In their original forms, both FedAvg and Local SGD simply averaged the parameters across workers. As shown by Reddi et al. (2021), the synchronization is more effective when each worker calculates "model delta", and these are aggregated over workers produce a pseudo-gradient (Reddi al., 2021; Ilharco al., 2022) or outer gradient, which is then fed to first-order optimizer. This yields a bi-level optimization scheme with inner optimizers and an outer optimizer, referred to as FedOpt by Reddi et al. (2021), who propose using SGD as the inner optimizer and adaptive methods like Adam (Kingma & Ba, 2014) as the outer optimizer in resource-constraines FL settings.
Distributed training for LLMs. The increased requirements of training large language models (LLMs) hastened the need for distributed methods, for both inference (Borzunov et al., 2023) and training (Presser, 2020; Diskin et al., 2021; Ryabinin et al., 2021). More re- cently, Di (Douillard et al., 2024a) proposed particular instantiation of FedOpt (Reddi et al., 2021) with AdamW (Loshchilov & Hutter, 2019) as inner optimizer and Nesterov (Sutskever et al., 2013) as outer optimizer (Huo et al., 2020). This simple formulation proved to be effective for distributed training with LLMs, where the number of replicas is small (i100) and without replica sampling, closer to cross-silo federated learning (Kairouz et al., 2021). The FedOpt algorithm was also shown to be effective in training LLMs in settings that looked more like cross-device federated learning (Charles et al., 2024). The empirical success of DiLoCo has been reproduced multiple times (Jaghouar et al., 2024b; Sani et al., 2024b) and has been successfully scaled up to 10 billion parameter models (Jaghouar 2024a). Related, a simple change on how the outer Nesterov accumulates outer gradients proved handle well asynchronicity between workers of different speeds (Liu al., 2024a). DiLoCo adds a new axis of paralellism to distributed training (Shoeybi et al., 2020), and is compatible (Jaghouar et al., 2024a) with other existing axes like FSDP (Zhao al., 2023), or even another level of federated learning (Sani et al., 2024a). CO2 (Sun et al., 2024) proposes also overlap communication, however using stale communication and for a full outer step. We saw in our work that overlapping only for a few inner steps reap most of the bandwidth reduction benefits while keep the performance stable.
al.,
Partial communication. Communicating subset of the network often used in federated learning to provide personalized models per user, see FedPart (Arivazhagan et al., 2019). DiPaCo (Douillard et al., 2024b) recently proposed a distributed mixture-of-experts where subsets of the model is synchronized with subsets of the replicas, according to a sharing style of algorithm during training.
pattern that is optimized with Expectation-Maxiiiizatee WASH (Fournier et al., 2024) and later Sparta (Baioumy & Cheema, 2025) propose to frequently exchange random subset of the neurons. Finally FedPart (Wang et al., 2024), developed at the same time as Streaming DiLoCo, also proposes to share per-layer fragments. However, they argue that for a given communication round, non-shared fragments should not undergo inner optimization, strategy which we show slows down convergence. Note that all partial communication methods can be seen as a form of sfructured (outer) gradients compression.

---

Published as a conference paper at COLM 2025
Gradient compression. Data-Parallel (with gradients) and Federated learning (with outer gradients) often share similar methods to compress the communication (Lin et al., 2018): from randomly dropping values (Wen et al., 2022), to combining multiple compression schemes (e.g- dropping, top-k, low-precision) (Wang et al., 2023), to use low-rank com- pression (Vogels et al., 2019; Zhao et al., 2024), or recently to keep only the fast moving components with DCT but communicates via an all-gather collective instead of an all-reduce (Peng et al., 2024).
Conclusion and Future Work
In this paper, we introduced three improvements over DiLoCo: we synchronize a only subset of the parameters at time, overlap the communication of this synchronization over several computation steps, and we compress the outer gradients to communicate to low- precision with only four bits. All these innovations combined together leads to training with similar ML-perrormance as a classical Data-Parallel training, while using 400 less bandwidth, reducing the peak bandwidth compared to DiLoCo's bursts of communication, and allowing communication to have an ideal non-zero latency by overlapping with computation.
In sum, we can reach a similar compute utilization as the widely used Data-Paralle using two orders of magnitude less Gbit/s bandwidth, while performing comparably in term of training loss and downstream evaluation accuracies as Data-Paralle
Next. In our view, the ubiquity of co-located Data-Parallel training is likely due to the hardware lottery (Hooker, 2020), when "a research idea wins because it is suited to the available software and hardware and not because the idea is superior allernative research directions". Data- Parallel training has been extensively studied, tuned, and scaled (Kaplan et al., 2020), and is hard to beat the fisdom-of-thhecroow of thousands of researchers. In contrast, the federated learning literature has mainly studied smaller scale models, primarily due to its focus on edge devices. There are huge opportunities for bringing the ideas from the federated learning literature to the new world of large scale training for LLMs. A critical next work is to study how new distributed methods like ours should be tuned and scaled across multiple axes (e.g. model size, overtraining factor, number of replicas). In particular, how to scale efficiently the number of DiLoCo replicas given an equivalent token budget is most needed.
More generally, reducing the communication problem to a minor obstacle allows new classes of co-designed architectures and training paradigms (for example Douillard et al. (2024b)) maximizing available compute (Sutton, 2019): we hope to see the training of modular constellations of small models loosely connected (Dean, 2021) across heterogeneous devices, using compute arbitrage spread world-wide.
References
Aditya Agrawal, Matthew Hedlund, and Blake Hechtman. exmy: A data type and technique for arbitrary bit precision quantization. arXio preprint library, 2024. URL https://arxiv.. https://arxiv.orr//bb////// 13938.
Samuel K. Ainsworth, Jonathan Hayase and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arv preprint library, 2023. URL https:// urxiv.org/abs/2220 04836.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choud- hary. Federated learning with personalization layers, 2019. URL https: org//bs// 1912.00818.
Mohamed Baioumy and Alex Cheema. Sparta. exog.sxotbbs.net, 2025. //blog. coollas.net/day-1i
URL https:
10

---

Published as a conference paper at COLM 2025
Olivier Beaumont, Lionel Eyraud-Dubois Alena Shilova, and Xunyi Zhao. Weight offloading strategies. HAL repository, 2022. URL https://inria, hal. sciennc/hal-0035877777
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020. URL https: /wwxiv.xvvvvvvv///v///..//./vv org/ http/////11111/// 1911 11641.
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL Short Papers), 2023. 2209. 01 1888
URL https://arxil org/abs.
Zachary Charles, Nicole Mitchell, Krishna Pillutla, Michael Reneer, and Zachary Garrett. Towards federated foundation models: Scalable dataset pipelines for group-structured learning. Advances in Neural Information Processing Systems, 2024. URL httpss///aaxxiii //rg///rr/vv.o abs/2307. 09619.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri- donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Mor- eira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling lan- guage modeling with pathways. Journal of Machine Learning Research, 2023. URL html.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, you
and Oyvind Tafjord. Think you solved question answering? try arc, the ai2 reasoning challenge, 2018, URL https: wwtpk/iw.wrr////////eee/ httLttss//rxiiioorr//a////////
05457v1.
Jeff Dean. Pathways: A next-generation ai architecture. Google's blog, 2021. URL pathways-neex--geneaatee
https://blog, google/teecnooogg//iiiii architecture/.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan,
11

---

Published as a conference paper at COLM 2025
Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report. arXiv preprint library, 2024. URL hnttps://aree httpxiwwwwrg//b//////a/ 19437.
Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitry Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ília Kobelev, Yacine Jernite, Thomas Wolf, and Gennady Pekhimenko. Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems (NeurlPS), 2021. URL https: org/ /a////bs///../////////// 10207.
Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Ad- higuna Kuncoro, Marc' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. DiLoCo: Dis- tributed Low-communication training of language models. International Conference on Machine Learning (ICML) Workshop, 2024a. URL https:
08105.
Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Adhiguna Kuncoro, Yani Donchev, Rachita Chhaparia, lonel Gog, Marc' Aurelio Ranzato, Jiajun Shen, and Arthur Szlam. Dipaco: Distributed path composition. arXiv preprint library, 2024b. URL hwtps:://///ttt http:///axvvvv.g/aaeee/e wrggabs////////// 2403.10616 10616.
Louis Fournier, Adel Nabli, Masih Aminbeidokhti, Marco Pedersoli, Eugene Belilovsky, and Edouard Oyallon. Wash: Train your ensemble with communication-efiiien weight shuffling, then average. Advances in Neural Information Processing Systems (NeurIPS) Workshop, 2024. URL
17517.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning (ICML), 2020. URL
wttp://wwww...raa//////..0 05671.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. arXiv preprint library, 2024. URL https: wwwpxvv.org//baee 240.. 08540.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chem, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imaool Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar,
12

---

Published as a conference paper at COLM 2025
Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Praijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierr Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, A Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal
13

---

Published as a conference paper at COLM 2025
Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekk, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https://arxivv http:://arxxv.orr/bb//2000 org/
Summer
217883.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https: http::///xxxvvvoooee org/abs/ /20s..201.0. 04245.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. Advances in Neural Information Processing Systems (NeurIPS), 2022. URL //apx: org/ htrrg//bb/222.... 15556.
Sara Hooker. The hardware lottery. Communication for the ACM, 2020. URL https://
06489.
Zhouyuan Huo, Qian Yang, Bin Gu, and Lawrence Carin. Heng Huang. Faster on-device training using new federated momentum algorithm. Xiv preprint library, 2020. URL /ttpaabb//222.2222
https://arxiv.or https://arxxx.orr//bb///0
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Ha- jishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keib- linger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann. Intellect-1 technical report. ar arviv preprint library, 2024a. URL https://arxiv.or//bb///11
Sami Jaghouar, Jack Min Ong, and Johannes Hagemann. Opendiloco: An open-source framework for globally distributed low-communication training. arx preprint library, 2024b. URL https://arxtts org/abs/2407. .78852.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. Proceedings of the International Conference on Learning Representations (ICI.R,, 2023. URL httpps://axxxs org/ htrgg/abb2211. 09849.
14

---

Published as a conference paper at COLM 2025
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. arvio preprint library, 2023. 08403.
URL https://arxiv http:://aaxxiv..oaabbb...ss
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cum- mings, et al. Advances and open problems in federated learning. Foundations and trends in machine learning, 2021. URL / h/tg://b/////// /9922. 04977.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ar Xiv preprint library, 2020. URL htttp:://axxi..or//bbssss
08361.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimizaton Proceedings of the International Conference Learning Representations (ICLR), 2014. URL https://
arxiv. orggi..rrgeee aaaaa///14 /142. 111.//111.899...
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Sys- tems (NeurIPS), 2012. URL https://papers. nips. cc/paper wtwppppaee..lllee/peeaeeeeeeeeee files/paper /20 201 2/hash/ Abbee4aeeecceD-bbbaratttt.tt.a html.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merge Embarrassingly parallel training of expert language models. Xiv preprint library, 2022. URL https://arxit: org/abs/2200 03306.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't large mini- batches, use local sgd. Proceadings of the International Conference on Learning Representations (ICLR), 2020. URL https://arxiv. wwtprxiwww.og..b////// wtwg/bs///////// 07217.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training, 2018. URL. https: aarxiv.orgg/abs11712.. 01887.
Bo Liu, Rachita Chhaparia, Arthur Douillard, Satyen Kale, Andrei A. Rusu, Jiajun Shen, Arthur Szlam, and Marc' Aurelio Ranzato. Asynchronous local-sgd training for language modeling. International Conference on Machine Learning (ICML) Workshop, 2024a. URL https://arxivv www./orgggsse.... haabb///2410000 091335
Peter J. Liu, Roman Novak, Jaehoon Lee, Mitchell Wortsman, Lechao Xiao, Katie Everett, Alexander A. Alemi, Mark Kurzeja, Pierre Marcenac, Izzeddin Gur, Simon Kornblith, Kelvin Xu, Gamaleldin Elsayed, Ian Fischer, Jeffrey Pennington, Ben Adlam, and Jascha- Sohl Dickstein. Nanodo: A minimal transformer decoder-only language model imple- mentation in JAX., 2024b. URL http: //gi thub. com www.goog dedeeeeiid/ddndndddddd
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Proceedings of the International Conference on Learning Representations (ICLR), 2019, URL https://axxxiv.orr abs/1711.05100
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, Advances in Neural Information Processing Systems (NeurIPS), 2021. URL https: http://axxvvvrrrroee org/abs/ 19832..
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effiiiene learning of deep networks from decentralized data. International Conference on Artificial Intelligence and Statistics (AISTATS), 2017. URL https: ///axxi.orgg/ab///// 05629.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.org wwwp:ivv..wg/////000000 11687.
15

---

Published as a conference paper at COLM 2025
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethotf, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL http:s////xxxv.r//aaaassss 2270003774.......
Bowen Peng, Jeffrey Quesnelle, and Diederík P. Kingma. Demo: Decoupled momentum optimization. arxiu preprint library, 2024. URL
19870
Shawn Presser. Swarm training, 2020. URL hww.h:tpp:/bbbt v01a.pdf.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with
16

---

Published as a conference paper at COLM 2025
unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL https: //aaxiv.org/aasstee.ee atte//1b//191.. 10688.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution gen- eralization. International Conference on Machine Learning (ICML), 2023a. URL https: waaxxiv.rrg/ab///2222 2tt.aa///21......... 10445.
Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural In- formation Processing Systems (NeurIPS), 2023b. URL htttps://axxxv.orrr/bb/////
04488.
Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonij, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution gen- eralization. Advances in Neural Information Processing Systems (NeurIPS), 2023c. URL https://arxiv.oot wwtp://.rgg//b/////e/eoo 221. 09739.
Sylvestre-Alvise Rebuffi, Francesco Croce, and Sven Gowal. Revisiting adapters with adversarial training. Proceedings of the International Conference Learning Representations (ICLR), 2022. URL https: http:///aaxxvv..o/abbbbsss/ /21/////11.. 04886.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization. Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxiv. wrg/obs///////eoooooooooooeeo/// 2atss/203..00 00295..
Keith Rush, Zachary Charles, Zachary Garrett, Sean Augenstein, and Nicole Mitchell. Drjax: Scalable and differentiable mapreduce primitives in jax. International Conference on Machine Learning (ICML) Workshop, 2024. URL https:://rxxtt wttpv/..rgg/bb////////// 07128.
Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit sgd: communication-efffcceeee decentralized training on heterogeneous unre- liable devices. Advances in Neural Information Processing Systems (NeurIPS), 2021. URL
httpss://axxi wwwpii...ggg/b////a/a aww.///ssa... 03239.
Max Ryabinín, Tim Dettmers, Michael Diskin, and Alexander Borzunov. Swarm parallelism: Training large models can be surprisingly communication-efficieent International Conference on Machine Learning (ICML), 2023.
11913.
Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, and Nicholas D. Lane. Photon: Federated Ilm pre-training. ar Xrv preprint library, 2024a. URL https: https:://arxi..oog/bb/////11
02908.
Lorenzo Sani, Alex lacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William F. Shen, Preslav Aleksandrov, Xinchi Qiu, and Nicholas D. Lane. The future of large language model pre-training is federated. arviv preprint library, 2024b. URL https://arxias wrt.//b//////////// http:://arri..orr/dd///2200000 10853.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint library, 2017. URL https://arxiv. wrgg 065388.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint library, 2020. URL https: //arxii////xvv..////// wwwp:ewww......rooaae
08053.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh
17

---

Published as a conference paper at COLM 2025
Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research, 2024. URL
Sebastian U. Stich. Local SGD converges fast and communicates little. Proceedings of the International Conference on Learning Representations (ICLR), 2019. URL https:t/arxxi...ot abs/1805.09785
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. arXiv preprint library, 2023. URL hts //arxiv.org/ abs. 03053.
Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. Co2: Efficient distributed training with full communicttion-ompuatiio overlap. Proceedings of the International Conference Learning Representations (ICLR), 2024.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International Conference on Machine Learning (ICML), 2013. URL httpps://proceee http::////occeees..... mmr. wwt.mm....rreses//.m.m wttpesss////sutseeerrrrr/ sutskever11.. html.
Rich Sutton. The bitter lesson. incompleteiddes..nt,, 2019.
URL http://
www. htt.teiddess.net/nnnne/////////e
html.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Érica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tzze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakssmmiarayannn, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, Hyunjeong Choe, Agoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd AI Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonza- lez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jecisky,Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, llya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankara- narayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomênech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang. Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Gana- pathy, Smit Sanghavi, Ajay Kannan, Ming- Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang. Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanbum,, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le
18

---

Published as a conference paper at COLM 2025
Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogoziñiska, Vi- taliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Aniruch Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Am- playo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yüin Chang, Paul Komarek, Ross Mcllroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Co- manescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornoon, Fangxiaoyu Feng, Woj- ciech Stokowiec, Ce Zheng, Phoebe Thacker, Çaglar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, An- drew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam So- bell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen,
19

---

Published as a conference paper at COLM 2025
Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Älgmyr, Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominík Rabiej. Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" " Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja,
Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichen- baum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoom, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey i
Cideron, Edouard Leurent, Mahmoud Alnahlawi, lonut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño,
20

---

Published as a conference paper at COLM 2025
Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiński,, Ashwin Sreevatsa, Jennifer Prendki, Goedeckemeyy Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Twuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Ka- plan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Min- nie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MosammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Car- penter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Be- nigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Cantamaria--ernande Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhav- ishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucñnska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson,
David Soergel, Adrian
Sonam Goenka,
21

---

Published as a conference paper at COLM 2025
Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas,
Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston,, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikuus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirsshall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhar- gava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pela- gatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hass- abis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024. URL https: http:///axxx...gabb/////
11805.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gra- dient compression for distributed optimization. Advances in Neural Information Processing Systems (NeurlPS), 2019. URL
13727.
Haolin Wang, Xuefeng Liu, Jianwei Niu, Wenkai Guo, and Shaojie Tang. Why go full? ele- vating federated learning through partial network updates. Advances in Neural Information Processing Systems (NeurIPS), 2024. URL http:://aaxivvvooaa///////a 11559.
Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: fine-tuning foundation models over 500mbps networks. International Conference on Machine Learning (ICML), 2023. URL
Dingzhu Wen, Ki-Jun Jeon, and Kaibin Huang. Federated dropout- simple approach for enabling federated learning on resource constrained devices. IEEE Wireless Communica- fions Letters, 2022. URL
http://aaiii..orr//////////// 15258.
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Raste- gari. Learning neural network subspaces. International Conference on Machine Learning (ICML), 2021. URL
10472.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International Conference on Machine Learning (ICML), 2022. URL https::/arxxt wrtg 2203. 05482.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornbith. Small-scale proxies for large-scale transformer training instabilities. ar Xio preprint library, 2023. URL https://arxii: www.ariwvi.ooggbss.eaoee www.orrg//bb//9//.// 14322.
22

---

Published as a conference paper at COLM 2025
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems (NeurIPS), 2023. URL
http://aaxvvv..g//aaa/ 2306.0170.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are su- per mario: Absorbing abilities from homologous models as a free lunch. International Conference on Machine Learning (ICML), 2024. URL https: htttp:://aaxii..or//a///3333// 03899.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL Short Papers), 2019. URL https:///rxitn wwwiiivoogg/bbbbe org/abs/
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. International Conference on Machine Learning (ICML), 2024. URL https: //arxiv wwtrrii..og//aaaaeee org/abs/ 2403. 03507.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. URL https://arxiv.oo/abbs////// 11277.
A Supplementary Materials
A.1 Context: FedOpt and DiLoCo
FedOpt (Reddi et al., 2021) is generic framework to perform federated learning with bi-level optimization. M local replicas perform H steps of inner independent optimizations on a different subset of the data (L3 to L5 in Algorithm 2). Every H steps, each replica
computes an outer gradient A = g!!-H) -e" (L7), a delta in the parameters space, and communicates to all other replicas. This communication can be performed through central parameter server or through direct communication of each worker to the others (e.g. with ring all-reduce), and results in each worker obtaining A' = 1/M ΣM A'n (L7-9). This outer gradient is applied on set of outer parameters, the previously synchronized parameters 8 e((-H), with an outer optimizer (L10). The full algorithm shown in Algorithm
The costly communication between non-colocated devices happens during the averaging of outer gradients, in L8-9 of Algorithm 2. is as costly as in DatalParallei, but instead of being executed at every step, it is done every H (e.g. one hundred) steps, thus amortizing the communication cost.
DiLoCo is successful instantiation of FedOpt applied to language models where the inner optimizer is Adam (Kingma & Ba, 2014) and the outer optimizer is SGD with Nesterov momentum (Sutskever et al., 2013). In this work, which focuses on distributed optimization, and unlike in the federated learning literature (as discussed in FedOpt), the workers aren't sampled; but instead all workers will be present at each step.
A.2 Discussions
Memory overhead. In an SPMD3 model, the memory overhead of the Data-Parallel base- line is the parameters (1x) + Adam state (2x). (Streaming or not) DiLoCo's memory overhead is the parameters (1x), the Adam state (2x), the outer global parameters (1 and the outer Nesterov state (1)). Thus, our method requires 66% (5/3) more memory compared to Data-Parallel. However, in the case of Streaming DiLoCo, only subset of the outer parameters and outer optimizer state is needed at given time. Therefore, this
Phttps://en wikipeddia.orr/vikk///inng./ program, multiple data
23

---

Published as a conference paper at COLM 2025
Algorithm 2 FedOpt DiLoCo
Require: M replicas
Require: Synchronization frequency H
Require: Model replicas {0}- ,...,, ((-11)
M
Require: Data shards (Di,..., ]
Require: Optimizers InnerOpt and OuterOpt
1: parallel for replica m 1... ..... M do
2: for step ! = 1... do
3: x~ DDm
t f(x, Of ))
4:
O InnerOpt (0 1) V
5:
if tH then
Am (t)
-- 0!! H)
(t)
7:
mm
8:
(lf) async-send
M ΣM (4
9:
block-receive
e + OuterOpt (e!! (1-H) ,A())
overhead can be alleviated by offloading the additional bits onto CPU memory (Beaumont et al., 2022). The memory overhead to hold in HBM at any point in time is the size of fragment |pl times two, for the outer parameters and outer optimizer state. For a 100 billion parameter model for instance, with [pl three layers and with total of 108 layers, that amounts to a 2% increase of memory (additional 20 GB to 1,117 GB*). This extra memory is used when there no activations gradients in live memory, and thus should fit in HBM without any problem.
Furthermore, the communication schedule is deterministic and known prior to training. Thus, we can start the transfer from RAM to HBM of a fragment (and its associated outer optimizer state) while finishing the previous (inner) gradients passes. Given that only small subset is required at a given time, the memory transfer cost is negligible. With an H100 with PCIe5, characterized by 2 TB/s of bandwidth speed, and without any sharding, this transfer done in less than 10 milliseconds.
Future work. In our view, the ubiquity of co-located Data-Parallel training is likely due to the hardware lottery (Hooker, 2020), when "a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions". Data-Parallel training has been extensively studied, tuned, and scaled (Kaplan et al., 2020), and it is hard to beat the wisdom toftom--o--hhhrrrowee of thousands of researchers. In contrast, the federated learning literature has mainly studied smaller scale models, primarily due to its focus on edge devices. There are huge opportunities for bringing the ideas from the federated learning literature to the new world of large scale training for L..Ms. A critical next work is to study how new distributed methods like ours should be tuned and scaled across multiple axes (e.g. model size, overtraining factor, number of replicas). In particular, how to scale efficiently the number of DiLoCo replicas given equivalent token budget is most needed.
More generally, reducing the communication problem to minor obstacle allows new classes of co-designed architectures and training paradigms (for example Douillard et al. (2024b)) maximizing available compute (Sutton, 2019): we hope to see the training of modular
4 4The size in GB of the parameters & inner Adam optimizer state is the number of parameters x x 4 (FP32). The size in GB of the additional fragment and its outer Nesterov optimizer state is the number of parameters x x 4 x 3
5htpps: //wp///wwwww////// nvidia. com/en dat/ddta-ceee centerrreee///ee/e//
24

---

Published as a conference paper at COLM 2025
constellations of small models loosely connected (Dean, 2021) across heterogeneous devices, using compute arbitrage spread world-wide.
Discussion on model merging. Model merging, a subfield within the broader study of linear mode connectivity, explores the potential of linearly interpolating between parameters of multiple models to synthesize unified model that inherits the strengths of its constituents (Matena & Raffel, 2021; Wortsman et al., 2021). A key finding in this domain is the existence of low-loss pathways within the parameter space (Frankle al., 2020; Neyshabur et al., 2020) that connect independently trained models, effectively circumventing the anticipated loss barriers. For instance, Wortsman et al. (2022) demonstrated that averaging the parameters of models fine-tuned from common pre-trained initialization on diverse tasks (Ramé et al., 2023a) or with varying hyperparameters (Wortsman et al., 2022) yields performant merged model. This approach, initially demonstrated in computer vision, has been successfully extended to natural language processing (Li et al., 2022), reinforcement learning with human feedback (Ramé al., 2023b), noisy label learning (Rebuffi et al., 2022), and out-of- distribution generalization (Ramé et al., 2023c). Recent research has further investigated alternative strategies for mitigating loss barriers in model merging, including techniques based on parameter space transformations and other model surgery methods aiming to resolve merging conflicts (Jordan et al., 2023; Stoica et al., 2023; Jin al., 2023; Ainsworth et al., 2023; Yadav et al., 2023; Yu et al., 2024).
A.3 Implementation details
DrJax. For training we use a modified version of the NanoDO codebase (Liu al., 2024b) that uses Drjax (Rush et al., 2024) to parallelize inner steps across replicas. The inner optimization done with an annotated variant jax. vmap for the optimization step, with parameters having extra leading axis for the DiLoCo replicas. The outer optimization is implemented with an all-reduce, without any central parameter server.
Architecture hyperparameters We detail the architecture across model scales in Table 3. The token budget per scale is computed from the chinchilla-optimal amount of flops (Hoff- mann et al., 2022).
Model scale
Hidden dim
Num layers
Num heads
Token budget
35M
2,048
8
700M
100M
3,072
12
1.5B
200M
4,096
12
16
3.5B
300M
5,120
15
20
6B
500M
6,144
18
24
11B
1B
8,192
24
32
25B
4B
12,288
36
48
83B
Table Architecture hyperparameters: we consider model from 35M 4B with the following hyperameters and chinchilla-optimal token budget. For all model scale, the vocabulary size 32,000.
A.4
Additional details
A.5 Streaming DiLoCo.
Pattern. We illustrate in Figure 8 the two considered patterns in subsection 2.1: namely sequential and strided. Each has slightly different ML performance (see ??) but also compute utilization. It's better seen the schedules of Figure 9, where can that strided pattern can more often overlap communication with the next step computation, resulting in slightly better utilization (see Figure 10.
25

---

Published as a conference paper at COLM 2025
2,681
Sequential pattem
Strided
2.680
Sequential
2.679
Evall Loss
2.676
2.677
2.676
1
3
6
18
Number of layers per fragment
Figure 7: The fragment's size will determine the peak bandwidth but also the learning dynamics. We choose practice 3 layers per fragment across all model scales.
Comparison to FedPart. Streaming DiLoCo bears similarity with concurrent work dubbed FedPart (Wang et al., 2024), where subset of the model is also exchanged at each round. However, FedPart argues that non-shared layers should be be frozen during inner optimiza- tion. We believe this is rather flops-inefficient: For an 18 layer model, with 3 layers per fragment, 15 layers (83%) are frozen at any point in time despite doing forward/backwerr /backward computation. We ran comparison of Streaming DiLoCo with and without the frozen pattern proposed by FedPart, reaching respectively on the C4 eval loss 3.2145 and 2.6749. Freezing the 18-3 3 15 layers that won't be synchronized the given round therefore results in 20% increase of the evaluation loss. These results confirm our intuition that while freezing layers may help merging, this incurs significant flop inefficiency, which might not be acceptable in training compute-bound settings (which are typical in current large scale training of LLMs).
Number of replicas We perform our main experiments with 2 replicas for simplicity but showcase Figure 11 an ablation of DiLoCo vs Streaming DiLoCo where the number of replicas M vary from 2 to 8. Contrarely to (Douillard et al., 2024a), we keep the total token budget constant. In Figure 11a, we keep the global batch size constant, and thus reduce the local per-replica batch size). In Figure 11b, we keep the local per-replica batch size constant, and thus increase the global batch size but also reduce the total number of steps.
Number of inner steps The number of inner steps H, has an engineering effect and learning effect: larger H means less frequent synchronization and thus less required bandwidth. On the other hand, too small H produce noisy small-normed outer gradients and too high H will see replicas drifting apart. Therefore, some middle ground needs to be found. We ablate in Figure 12 and find that while Streaming DiLoCo has similar behavior as DiLoCo when increases, it is more robust to low values of H.
Which parameters to evaluate We considered multiple subset of the parameters to use for evaluation: 1) the arbitrarily chosen first replica (01), 2) an average of all replicas (M ΣM- 0m)) or 3) the globally shared outer parameters (0). Note that the latter is made of fragments that were synchronized at different points time. We show the performance of each subset in Table 4: The difference here between these methods is small, but the outer parameters yield slightly better performance.
A.6 Communication overlapping
Impact of overlapping on compute utilization. We show in Figure 13, various degrees of communication overlapping and their impact on compute utilization. While more is obviously better, there are clear diminishing returns starting from 5 inner steps, which is also where the MI. performance starts to degrade.
26

---

Published as a conference paper at COLM 2025
Embeddings
Embeddings
Layer 1
Layer
Layer 2
Layer 2
Layer 3
Layer 3
Layers
Layer 4
Layer 4
Layer 5
Layer 5
Layer 6
Layer 6
Layer 7
Layer
Layer 8
Layer 8
Sequential
Strided
time
Figure 8: Streaming pattern: sequential (left) and strided (right). Colors denotes the fragment. different fragment synchronized each time.
Overlapping with some slack between workers If workers use heterogeneous device types (e.g. TPUv5e TPUv6e) or are just placed in different environments, their execution speed might vary and it would be inefficient to force them to synchronize at the same optimization step. In this case, we could grant workers with some slack. For instance, in 2 DiLoCo replicas setting, both workers could send their respective outer gradients at the same step (but not necessarily at the same time) and they could receive the update at different step (Liu et al., 2024a). We accomplish this by simply using different per worker (Ti and 72) as shown in line 7-9 in Algorithm We show in Figure 14 the evaluation 1 and varying T2. Similarly to Figure 5, the loss degradation is limited under a delay of up to 5 inner steps. This result suggests that Streaming DiLoCo is rather robust and could support training of large models several heterogeneous workers.
loss when
27

---

Published as a conference paper at COLM 2025
Figure 9: Simulation of schedule interleaving forward passes (in blue), backward passes w.r.f. activations and weights (resp. in light and dark green), and (outer) gradient reduction (in purple) for Streaming DiLoCo, respectively with sequential and strided pattern.
Data-Parallel
Streamling DILoCo (sequential)
Streaming DiLuCo (strided)
0.8
Compute utllization
0.4
10-3
10a
10a
10a
WAN bandwidth [Gbit/s]
Figure 10: Compute utilization profile of sequential vs strided pattern for 100 billion parameters model.
A.7 Quantized communication
Loss and accuracy. We show in Figure 15 the impact of compressed outer gradient com- munication both the loss C4 and accuracy on HellaSwag.
A.8 Compute Utilization
Compute utilization. We report in Table 5 the amount of Gbit/s required, per method, to reach a certain level of compute utilization. See Figure for a figure view of this table. For DiLoCo and Streaming DiLoco (and variants thereof), we use H 100 inner steps. For Streaming DiLoCo (and variants thereof), we use a fixed fragment size of 3 layers; therefore, deeper networks have more fragments: for 1B, 10B, and 100B model scales, it is respectively 8, 16, and 36 fragments. Also, respectively per model scales, fragment is synchronized every 11, 5, and 2 steps. While the synchronization seems to be more frequent for deeper networks, from the perspective of particular fragment, it is synchronized roughly every H = 100 steps. To estimate the compute utilization in Table 5 and Figure 3, the time spent per step doing computation (forward & backward) is critical: we report respectively 0.1s, 0.8s, and 4.9s based on each model scale flops profile, a reasonable amount of chips, and MFU of 60%.
Compute utilization with various speeds. Varying the time spent per step to do pure putation (forward & backward) affects the compute utilization: e.g. for a fixed bandwidth and thus fixed communication time, longer step time, will improve compute utilization.
28

---

Published as a conference paper at COLM 2025
Streaming DiLoCo
local BSZ 16
2.74
DiLoCo
Eval Loss on C4
2.72
local BSZ = 32
2.70
2.68
local BSZ 64
2
4
8
Number of replicas
(a) Keeping the global batch size constant, and thus decreasing the local per-replica batch size.
M=8
Streaming DiLoCo
2.74
DiLoCo
Eval Loss on C4
2.72
M=4 4
2.70
M=2 2
2.68
256
512
1024
Batch Size
(b) Keeping the local per-replica batch size constant, and thus increasing the global batch size.
Figure 11: Scaling the number of DiLoCo replicas M from M = 2 to M 4. For all experiments, the token budget is kept constant.
Parameters evaluated
Eval Loss
HellaSwag
First replica
2.77
37.77
Replicas average
2.68
37.72
Outer parameters
2.67
37.78
Table Which parameters to evaluate?: Evaluating the outer parameters, where each fragment has been synchronized at a different moment in time, yields better performance than any inner parameters.
We report in Figure 16, simulated compute utilization when using, at 100B model scale, compute step time of 1 second, 5 seconds, and 10 seconds.
Compute Utilization on Llama and DeepSeek. We estimate in Figure 17, the compute utilization of our method baselines top of Llama405 (Grattafiori et al., 2024) and
29

---

Published as a conference paper at COLM 2025
2.73
Streaming DiLoCo
DiLoCo
2.72
Eval Loss on C4
2.71
2.70
2.69
2.68
2.67
10
30
50
100
200
500
Number of inner steps
Figure 12: Varying the number of inner steps H for DiLoCo and Streaming DiLoCo while keeping the total number of steps constants. A lower H means more communication rounds to be done.
0.8
Compute utilization
0.6
0.4
Streaming
Streaming with overlapped communication for inner step Streaming DiLoCo with overlapped communication for 2 inner step Streaming DiooCo with overlapped communication for 5 inner step Streaming LoooCo with overlapped communication for 10 inner step Streaming DiLoCo with overlapped communication for inner step
0.2
10-3
10a
10a
10a
10a
WAN bandwidth [Gbit]s]
Figure 13: Estimated compute utilization for 100B model when increasing t, the number of inner steps which overlap with communication.
DeepSeek-VV V3 (DeepSeek-AI al., 2024). For each, we estimate their step time from the respective paper: 26.9 seconds for Llama (first stage of pretraining) and 20.1 seconds for DeepSeek, using the most charitable estimation everytime. Notably, for DeepSeek-V V3 (Figure 17b), only 35 billion parameters are activated per token due to their MoE architecture (Shazeer et al., 2017). However, the total 671 billion parameters are synchronized between replicas, massively increasing the amount of bits to transfer. In that case, in simulation, our method (in red)) can be close to 100% compute utilization with Gbits per second vs Tbit per second for
30

---

Published as a conference paper at COLM 2025
com. overlap (T1 [2 = x)
2.6900
com. overlap (I1 = 1, [2 = x)
2.6875
Eval loss on C4
2.6850
2.6825
2.6800
2.6775
2.6750
0 1
2 3 4
10
15
20
Number of overlapped inner steps
Figure 14: Varying the number of overlapped inner steps T2 for the second worker while
keeping T = 1. For all data points, a = 0.5. Training is very robust for values of T2 less than 5.
Gbit/s reach compute utilization CU =?
Model
layers
Step time
Method
90%
95%
99%
Data-Parallel
86.8
152.6
184.2
222.3
569.0
Vanilla DiLoCo
1.4
13.3
23.3
86.8
1B
24
0.1s
Streaming DiLoCo
14
5.2
9.1
16.0
28.1
Streaming DiLoCo w/ overlapped com.
1.4
9.1
11.0
Streaming DiLoCo w/ overlapped FP4 com.
0.4
0.9
1.7
2.0
3,0
Data-Parallel
104.8
222.3
222.3
268.3
471.5
Vanilla DiLoCo
1.7
7.5
16.0
33,9
104.8
10B
48
0.8g
Stresming DiLuCo
1.7
9.1
13.3
19.3
Streaming DiLoCo w/ overlapped com.
1.7
3.6
5.2
6.2
7.5
Streaming DiLoCo w// overlapped com.
1.4
1.7
Data-Parallel
184.2
323.8
390.7
390.7
471.5
Vanilla DiLoCo
3.0
23.3
49.4
184.2
100B
108
4.9s
Streamiin DiLeCo
2.4
6.2
9.1
11.0
19.3
Streaming DiLoCo w/ overlapped com.
1.7
5.2
5.2
Streaming DiLoCo w/ overlapped FP4 com.
0.5
0.9
1.1
1.1
1.4
Table 5: Simulation: we estimate the step time (pure compute) of 10B and 100B based on the required flops using Kaplan et al. (2020) rule and using a MFU of 60%. For all DiLoCo and Streaming DiLoCo-varias we use H = 100. For all Streaming DiLoCo-variants, we use a fragment size of 3 layers.
A.9 Additional numbers
Scaling performance. We report in Table 6, the evaluation loss on C4 and accuracy on Hel- laSwag (Zellers et al., 2019), Piqa (Bisk et al., 2020), and Arc-Easy (Clark et al., 2018), for four different methods across model scales. See subsubsection 3.2.1 for the initial discussion. Performance across scales are roughly similar among all considered methods, with usually a slight advantage for Data-Parallel. We found in practice this advantage to disappear when doing more realistic overtraining with larger token budget in subsubsection 3.2.2.
31

---

Published as a conference paper at COLM 2025
drop rate =87.5%
2.90
low-precision FedDropout
Dare
2.85
Top-k
Eval Loss on C4
drop rate =75%
2.80
Top-k 87.5%
2.75
drop rate =50%
Top-k 75%
2.70
fp4
fp8
Top-k 50%
fp32
0.4
0.8
1.0
Compression
(a) C4 evaluation loss
fp8
bf16
0.38
fp4
Top-k 50%
fp32
0.37
Top-k 75%
0.36
drop rate =50%
HellaSwag
Top-k 87.5%
0.35
0.34
drop rate =75%
0.33
low-precisian
0.32
FedDropout
drop rate =87.5%
Dare
0.31
Top-k
0.2
0.4
0.6 Compression
0.8
1.0
(b) HellaSwag accuracy
Figure 15: Compressing the outer gradients with either value dropping (FedDropout, Dare) or using lower-precision floating point numbers.
Scaling with variable number of replicas. Contrarely Data-Parallel, changing the number of replicas for DiLoCo is not mathematically equivalent due to the local training, happening independentely for each replicas. We display in Table 7, a scaling from 35 million parameters to 1 billion parameters the C4 dataset of our method, Streaming DiLoCo with overlapped FP4 communication, with different number of replicas M {2, 4] and different
frequencies of synchronization H (30, 100}. Likewise, in Table 8, showcase token budget overtraining at 1 billion parameters on the Dolma dataset.
A.10 Miscellaneous
Outer gradients' cosine similarity, We observe in Figure 18 the cosine similarity per scale between each replica's outer gradients for respectively all parameters but the embeddings (Figure 18a) and only the embeddings (Figure 18b). For both, the cosine similarity starts from slightly correlated (=s 0.1), spends of the training time to be close to orthogonal (z< 0.0),
32

---

Published as a conference paper at COLM 2025
(a) 1s step time
(b) 5s step time
(c) 10s step time
Figure 16: Compute Utilization for a 100 billion parameters when the step time (pure compute) is second, seconds, and 10 seconds.
Diudo
Strandin
Stateel
(a) Llama405B.
(b) DeepSeek-V2 V3 (671B total, 35B activated).
Figure 17: Compute Utilization simulated across range of bandwidth for Llama405 and DeepSeek-VSs V3, using step time estimated from respective papers.
and ends slightly inversely correlated (= -011) as we reach the fluctuation phase. Note also that the larger the model size, the lower is overall the cosine similarity.
We also plot in Figure 19 the cosine similarity per scale and per transformer layer. Notably, the first transformer layer at each scale has significantly higher similarity, at every model scales.
33

---

Published as a conference paper at COLM 2025
Model size
Flops
Method
Eval Loss
HellaSwag
Piga
Are Easy
Data-Parallel
24.62
57.89
29.65
35M
1.5e17
DiLoCo H=30
3,54
24.53
38.11
29.65
Streaming DiLoCo with overlapped FP4 cum, H-30
24.46
57.67
Streaming DiLoCo with overlapped com, H-100
3.56
24.80
37.89
29.12
Data-Parallel
26.94
DiLoCo H=30
3,21
26.59
60.50
29.12
100M
9.4e17
Streaming DiLoCo with overlapped FP4 cum, H-30
Streaming DiLoCo with overlapped FP4 com,, H=100
3.22
26.68
60.39
31.93
Data-Parallel
29.86
63.71
200M
4e18
DiLoCo H=30
2.98
29.71
62,30
33,68
Streaming Dil with overlapped com, H-30
29.67
Streaming DiLoCo with overlapped FP4 com,, H-100
3.00
29.27
62.13
34.21
Data-Parallel
33.46
64.69
34,91
300M
1.4e19
Streaming DiLoCo with overlapped FP4 com,, H-30
DiLoCo H=30
2.81
33.87
33.66
64.74
63.49
34.74
Streaming Di with overlapped FP4 com, H-100
2.83
33.00
63.71
34.39
Data-Parallel
2.67
38.68
66.49
500M
4.7e19
DiLuCo H=30
2.68
38.37
65.61
36.32
Streaming DiLoCo with overlapped FP4 com, H-30
38.10
66.21
34.91
Streaming DiLoCo with overlapped com., H-100
2.69
37.40
65.51
34.74
Data-Parallel
2.49
46.60
68.93
1.9e20
Streaming DiLoCo with overlapped FP4 com, H=30
Di H=30
2.49
2.48
46.56
46.60
68.82
69.04
39.12
36.84
Streaming DiLoCo with overlapped com., H=100
2.50
46.00
68.82
38.42
Data-Parallel
2.25
59.56
72.42
43.51
oio H+30
2621
Streaming DiLoCo with overlapped FP4 com, H=30
Streaming Di with overlapped FP4 com., H-100
2.26
59.02
72.52
43.16
Table 6: Scaling from 35 million parameters to 4 billion parameters using chinchilla- optimal number of flps/tokens We train on the C4 dataset, and report the evaluation loss on its validation set.
0.4
0.4
33M
SSM
100M
Coarre simiarity bewwee outer gradarrs
100M
0.3
200M
200M
350M
300M
beaeen
0.0
0.0
-0.1
-0.1
0.2
0.2
fraction
fraction
(a) All fragments but the embedding
(b) Embedding fragment
Figure 18: Cosine similarity between the outer gradients across scales.
35M
100M
200M
350M
500M
0.2
Coshhe similarity
0.1
-0.1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Training traction
Figure 19:
Cosine similarity between the outer gradients across scales. Each line transformer layer, with darker colors being earlier layers and lighter colors later layers.
34

---

Published as a conference paper at COLM 2025
Model size
Flops
M
H
Eval Loss +
HellaSwag
Piqa t
Arc Easy
2
30
3.53
24.46
57.67
30.53
4
30
3.60
24.50
56.09
28.60
35M
1.5e17
2
100
3.56
24.80
57.89
29.12
4
100
3.64
24.67
56.75
26.84
2
30
3.21
26.97
59.58
31.40
30
3.25
26.24
59.74
32.63
100M
9.4e17
2
100
3.22
26.68
60.39
31.93
100
3.29
26.54
60.34
29.82
2
30
2.98
29.67
61.92
34.39
30
3.02
29.09
62.89
35.44
200M
4e18
2
100
3.00
29.27
62.13
34.21
100
3.05
28.53
61.10
33.51
30
2.81
33.66
63.49
35.09
4
30
2.84
32.54
64.42
34.74
300M
1.4e19
2
100
2.83
33.00
63.71
34.39
4
100
2.87
32.02
64.25
35.44
2
30
2.67
38.10
66.21
34.91
4
30
2.70
36.95
65.72
35.26
500M
4.7e19
2
100
2.69
37.40
65.51
34.74
4
100
2.73
36.02
66.27
35.09
2
30
2.48
46.60
69.04
39.12
30
2.50
45.25
67.95
39.12
1B
1.9e20
2 100
2.50
46.00
68.82
38.42
100
2.53
44.74
68.34
38.25
Table 7: Scaling from 35 million parameters to billion parameters Streaming DiLoCo with overlapped FP4 communication and with two different synchronization frequencies H {30, 100) and number of DiLoCo replicas M (2.
Method
Token Budget
Terabytes exchanged |
Eval Loss |
HellaSwag
Piqa f
Are Easy
25B
441
42.09
67.35
40.42
Data-Parallel
100B
1,767
2.52
49.78
69.15
250B
4,418
2.45
53.86
70.45
44.21
25B
1.10
2.66
42.08
67.46
38.42
Our method, M=2
100B
4.42
2.51
49.98
69.96
44.03
250B
11.05
2.45
54.24
71.38
41.92
25B
0.55
2.73
38.93
66.92
39.64
Our method, M=4
100B
2.21
2.54
48.35
69.42
40.52
250B
5.52
2.47
52.20
70.29
42.45
Table 8: Overtraining on the Dolma dataset with a billion parameters model, and with an increasing token budgets (25B, 100B, and 250B). We report here for our model both with M = 2 and M 4 DiLoCo replicas. With twice more replicas, the global batch size doubled, and twice less steps are done. It also thus roughly twice faster, but come with slightly worse performance. Our method is the final model: Streaming DiLoCo with overlapped FP4 communication.
35


Reviews:
[{"review_id": "CqeN05ivNs", "reviewer": "Reviewer_Aw6r", "summary": "DiLoCo is a distributed training algorithm inspired by federated learning that reduces communication overhead and relaxes all-to-all high-bandwidth connectivity requirements. It enables training across isolated device clusters (\"islands\") by performing local updates on data shards and infrequent inter-island synchronization via an outer optimizer.\n\nStreaming DiLoCo further optimizes communication efficiency by reducing peak bandwidth demands, consequently minimizing latency. It achieves this by asynchronously synchronizing only subsets of model parameters, rather than the entire set at once, effectively overlapping computation with communication. Additionally, outer gradients are quantized to further minimize data transfer.\n\nThe optimizations used are reminiscent of asynchronous pipeline + data parallel training methods and gradient compression algorithms, which have been used in a new context of DiLoCo.\n\nEvaluation shows that peak bandwidth demand is reduced by orders of magnitude without loss in quality of ML models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well written.\n- Strong ablation studies. \n- Discussion on strided fragment patterns for partial updates is novel and interesting.", "reasons_to_reject": "- The proposed optimizations seem largely derivative of prior work on asynchronous data-parallel training and gradient compression.\n- Evaluation with pure data parallel (instead of pipeline + data parallel, with and without DiLoCo) is unrealistic, especially for large (~>1B parameters) models, and doesn't give an accurate description of potential gains of Streaming DiLoCo.\n- Comparison of wall-clock time for DiLoCo and Streaming DiLoCo would have been useful.  Given that DiLoCo already achieves a 500x reduction in communication by synchronizing outer gradients only once every H=500 steps, it is difficult to appreciate the impact of Streaming DiLoCo.\n- The paper lacks a clear explanation for why model quality does not degrade, despite the reduced synchronization and additional approximations. \n\n\n\nNits: \n- Please improve the quality of figures - especially figures 2 and 3.", "questions_to_authors": "- Evaluation of training efficiency with only data parallel approach is unrealistic.  With large models (with billions, or even 100s-of-millions of parameters), one almost never uses pure data parallel.  Using pipeline parallel training is very common.  From section 3.1, Streaming DiLoCo \"reduces the latency by splitting the communication of outer gradients across fragments\".  This is in fact, what pipeline parallel achieves - parameters are split across partitions, and each data parallel set needs to synchronize only a subset of gradients.  How does compute utilization, bandwidth requirements and latency for training models using Streaming DiLoCo compare against pipeline + data parallel approaches with and without DiLoCo?\n- DiLoCo already claims to reduce communication overhead by 500x without loss in model accuracy. Since outer gradients are only synchronized once every 500 steps, what is the impact of proposed optimizations on compute utilization and training latency (in comparison to DiLoCo)?\n- The proposed approach is similar in essence to asynchronous data parallel training.  These approaches are known to degrade model quality in large language models, and the reason why they have not been adopted in main stream ML development.  What makes the Streaming DiLoCo techniques different from these approaches?  Would the same observations translate to models other than those shown in the paper?\n\nNits: \n- How was latency of data transfer calculated in Figure 2?  I.e., what was the bandwidth provisioned?", "extra_scores": {}}, {"review_id": "UIhyuHVs5Q", "reviewer": "Reviewer_xJA5", "summary": "This paper presents Streaming DiLoCo, a novel enhancement of the DiLoCo framework for distributed training of large language models (LLMs). It addresses key bottlenecks in cross-worker communication by introducing three main contributions: (1) Streaming synchronization of parameter subsets instead of full models, (2) Overlapping communication and computation, and (3) Quantization of exchanged gradients to four-bit representations. These techniques collectively reduce peak bandwidth, wall-clock training time, and overall communication cost, while maintaining training quality. Experimental evaluations, including scaling laws and overtraining on large datasets, demonstrate that Streaming DiLoCo achieves comparable performance to standard data-parallel training with significantly less bandwidth.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Comprehensive experiments: The paper includes scaling studies, ablations, bandwidth simulations, and performance on downstream tasks such as HellaSwag, PiQA, and ARC-Easy.\n\nStrong empirical results: The proposed method demonstrates a 400× reduction in communication cost without sacrificing accuracy, making it highly relevant for scenarios with constrained interconnect bandwidth.\n\nSolid engineering implications: The method is compatible with existing frameworks like DiLoCo, FSDP, and federated setups, making adoption easier in practice.", "reasons_to_reject": "1. Comparison with CO2 (Communication-Computation Overlap): While the paper makes a strong case for overlapping communication and computation, it omits discussion of \"CO2: Efficient Distributed Training with Full Communication-Computation Overlap\"(ICLR 24). A direct comparison or at least a discussion would significantly strengthen the paper’s positioning and highlight its specific advantages.\n\n2. Bandwidth and Hardware Profiling: While the paper includes simulation-based compute utilization plots, real-world hardware experiments (e.g., GPU/TPU wall-clock comparisons) would make the results more tangible. I would suggest including latency and throughput numbers measured on actual hardware, especially under variable bandwidth settings, to validate the simulated results.\n\n3. Impact on convergence in highly asynchronous regimes: The experiments primarily use 2 replicas. It's unclear how the method scales with increasing asynchrony or heterogeneity. I would suggest Provide additional experiments or discussion on how Streaming DiLoCo behaves with larger numbers of non-synchronous workers, or when using heterogeneous devices.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CzhaHtuk8R", "reviewer": "Reviewer_YXr7", "summary": "This paper proposes \"Streaming DiLoCo,\" an enhanced distributed training algorithm for LLMs building upon DiLoCo/FedOpt. The goal is to reduce the high peak and total bandwidth requirements and worker blocking associated with synchronization in distributed training, especially between non-co-located workers. The authors introduce three main improvements: (1) synchronizing only subsets (fragments) of parameters sequentially instead of all at once, (2) overlapping the communication of these fragments with ongoing worker computation, and (3) applying low-precision quantization (4-bit E3M0) to the communicated gradients (outer gradients/model deltas). Through simulations and scaling experiments up to 4B parameters, they demonstrate that this approach can maintain comparable model quality to Data Parallelism and standard DiLoCo while drastically reducing required bandwidth (by up to two orders of magnitude) and improving compute utilization under bandwidth constraints.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses a critical and practical challenge in scaling LLM training: the high cost and complexity of communication infrastructure.\n- Proposes a synergistic combination of three well-motivated techniques (parameter streaming, overlap, quantization) to tackle the problem.\n- Provides both simulation results for compute utilization analysis under varying bandwidths and empirical LLM training results to validate model quality.\n- Achieves significant claimed reductions in bandwidth requirements (peak and total) while maintaining performance, potentially enabling training across less ideal network conditions.\n- Builds logically on recent work (DiLoCo, FedOpt) and offers clear advantages over the baseline DiLoCo.", "reasons_to_reject": "- The combination of streaming, overlapping, and quantization introduces significant system complexity; the practical implementation overheads and tuning difficulties might be understated.\n- Sensitivity to hyperparameters like H (sync frequency), τ (overlap steps), fragment size/pattern, and outer learning rate seems potentially high and may require careful tuning per setup.\n- While comparisons are made to Data Parallelism and DiLoCo, a broader comparison to other advanced gradient compression or communication scheduling techniques could strengthen the claims.\n- The compute utilization results rely on simulation, which makes simplifying assumptions (acknowledged by authors) and may not perfectly reflect real-world network dynamics and system bottlenecks.\n- Lack of citing and comparing with similar works like CO2, SlowMo.", "questions_to_authors": "- Could you elaborate on the practical implementation challenges and potential system overheads (e.g., scheduling complexity, memory management for buffering overlapped gradients) introduced by combining all three proposed techniques?\n- How sensitive is the final model quality to the choice of quantization format (E3M0)? Were other low-bit formats explored, and how did they perform?\n- Figure 5 suggests robustness to overlap (τ) up to 10 steps. How does this interact with H? Is there an optimal ratio, and does the optimal τ change significantly with model scale or fragment size?\n- The term \"distributed free lunch\" is evocative but potentially strong. Could you clarify the trade-offs? Is the \"cost\" primarily in implementation complexity and hyperparameter tuning rather than pure ML performance or bandwidth?", "extra_scores": {}}, {"review_id": "WQly77ehl8", "reviewer": "Reviewer_CTbm", "summary": "Streaming diloco is an improvement on top of diloco in a few aspect: model subset synchronization, overlapped computation and communication, and quantizing the gradients.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Achieves similar performance to DiLoCo training on multiple eval benchmarks, while reducing networking bandwidth usage by order of magnitude, and improved GPU utilization with overlapping techniques\n- A reasonable simulation for the estimated gain on model training with different scales. Although the authors do not have such resources, it helps clarify how much gain is expected and the approach to analyze the training time.", "reasons_to_reject": "- the set of model architectures is limited. It remains to be seen whether such techniques work across other llm architecture variants such as moe models. Both model size and architecture may affect the scale of the gradients during training, quantization techniques may require further experimentation to show the effectiveness. \n- it is unclear how sensitive the hyper-parameter selection is when streaming diloco is used. Ideally it should be a robust algorithm without too much extra tuning.", "questions_to_authors": "-", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Streaming DiLoCo, an enhancement to the DiLoCo distributed training framework that reduces communication overhead through three key modifications: streaming parameter synchronization, computation-communication overlap, and gradient quantization. The work addresses a critical practical challenge in decentralized LLM training by dramatically reducing peak bandwidth requirements (up to 100×) while maintaining model quality comparable to standard approaches. The proposed method represents a well-engineered combination of existing techniques applied thoughtfully to the DiLoCo context, with strong empirical validation showing significant communication cost reductions without accuracy degradation. The comprehensive experimental evaluation includes scaling studies, ablation analyses, and downstream task performance assessment, demonstrating the practical value of the approach for bandwidth-constrained distributed training scenarios.

Pros: Simple improvements on DiLoCo that work well; addresses critical bandwidth bottlenecks in distributed training; achieves substantial communication cost reductions (up to 400×) while preserving model quality; comprehensive experimental evaluation with strong ablation studies.

Cons: Limited evaluation to relatively small models (≤4B parameters) and primarily 2-worker setups; lacks comparison with related communication-optimization methods. The work could be improved by adding analysis of scalability to larger numbers of asynchronous workers and going into the details of practical implementation.
Decision: Accept