ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
Authors: Adel Nabli
Louis Fournier
Pierre ERBACHER
Louis Serrano
Eugene Belilovsky
Edouard Oyallon
Venue: NeurIPS 2025 poster ()
Keywords: distributed training; communication overlapping; sharded optimizers
Primary area: deep_learning
TL;DR: ACCO is a new and principled optimization techniques with provable guarantees for Sharded Distributed LLM Training

Abstract: Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel setups introduces communication overhead that grows with the number of workers, limiting parallelization efficiency. Local optimization algorithms reduce communications but incur high memory costs as they prevent optimizer state sharding, hindering scalability. To address this, we propose $\textbf{AC}$cumulate while $\textbf{CO}$mmunicate ($\texttt{ACCO}$), a memory-efficient optimization algorithm for distributed LLM training. By synchronizing delayed gradients while computing new ones, $\texttt{ACCO}$ reduces GPU idle time and supports heterogeneous hardware. To mitigate the convergence issues caused by delayed updates, we introduce a novel technique ensuring training dynamics align with standard distributed optimization. Compared to ZeRO-1, our approach is significantly faster and scales effectively across heterogeneous hardware.

ACCO: Accumulate While You Communicate for Communicatio--Oeelapee Sharded LLM Training
Adel Nabli!:2
Louis Fournier Pierre Erbacher
Louis Serrano
Eugene Belilovsky? Edouard Oyallon?
SSoroonne Université, CNRS, ISIR, Paris France
Mila Quebec Al Institute, Concordia University, Montréal Québee edouard, oyallon@enrs....
Abstract
Training LLMs relies on distributed implementations using multiple GPUs compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel setups introduces communication overhead that grows with the number of workers, limiting parallelization efficiency. Local optimiza- tion algorithms reduce communications but incur high memory costs as they prevent optimizer state sharding, hindering scalability. To address this, we pro- pose ACcumulate while COmmunicate (ACCO), a memory- optimization algorithm for distributed LLM training. By synchronizing delayed gradients while computing new ones, ACCO reduces GPU idle time and supports heterogeneous hardware. To mitigate the convergence issues caused by delayed updates, intro- duce a novel technique ensuring training dynamics align with standard distributed optimization. Compared ZeRO-1, our approach significantly faster and scales effectively across heterogeneous hardware.
Introduction
Training Large Language Models (LLMs) with billions of parameters requires thousands of GPUs operating in parallel [71]. This process typically relies on distributed backpropagation (31 and gradient-based optimizers such Adam (27] or AdamW (36). However, distributed optimization at this scale is both memory- and communication-iitensiiee In standard data-parallel training, memory bottlenecks arise primarily from the optimizer's s internal states, especially under mixed-preeision training. Techniques like ZeRO (55) mitigate this by sharding states across workers. Due to limited GPU memory and the large size of modern models, large-scale LLM training frameworks must rely on such sharded partitioning (64 58 2) In addition to memory constraints, communication overhead becomes a dominant performance bottleneck, as synchronizing gradients and optimizer states across GPUs can exceed the time spent actual computation (50), problem expected to persist even with future hardware advances [53]. The impact of communication further amplified by the cluster's interconnect topology: effective sharding requires high-bandwidth links, and heterogeneous hardware or slow interconnects introduce straggler effects that slow down the entire system [1] 41
To mitigate these issues, various
caanuuuication-ffffciee distributed optimization algorithms have been proposed, particularly in settings with limited bandwidth such as Federated Learning. Local- update methods (66, 77 40 29 [1] reduce communication by splitting training into inner loops (local steps) and outer loops (synchronization steps). While this approach reduces frequency of communication, it introduces additional hyperparameters compared to standard training, scales poorly with the number of workers, and significantly increases memory requirements. For instance, the state- of-the-art CO2 method (69 requires memory overhead equivalent to four model copies-minn more
Equua contributions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Compute estimates ((()))
Compute #i ((())
Comm. butter
butfer comm. buffer
GPU slow
acc.
V (0,)
Vi Vi(0.)))
stream
Sharded Opt. step
All-Gather
Reduce- Scatter
Shardled Opt. step
All-Gather
comm. stream
Scatter ((8,)
V(A V(8.).
9, - B6_1
GPU fast
stream
(2(P,)
Va(O.)
(a(O.)
Va(A.)
V. v.(O...)
Va(Oeez)
(a(O..1)
V (e(
vallers)
Reduce- Scatter
Sharded
All-Gather
Roduce- Scatter
Sharded
All-Gather
comm, stream
Opt. stop
Opt. step
((0))
V(O)
V(9,)
8e + Be-s
0243
Time
Figure I: ACCO with slow and a fast worker running in parallel, showing no idle time on both and hiding communications. The delayed update is compensated by splitting the mini-batch in two, leading to two steps in our timeline. The first uses half of the mini-batch to estimate "next step" parameters, and the second uses the full
update them.
than standard distributed Adam and even further from its sharded variants (55) Some local-update methods overlap communication with computation to hide latency, by executing both concurrently However, this is challenging in sharded setups, since local updates typically require full optimizer states to be materialized, which defeats the purpose of memory or communication savings. A notable exception is ZeRO-Offload [60], which introduces the Delayed Parameter Update (DPU) mechanism: it runs gradient computation on GPUs while concurrently performing parameter updates on CPUs. Yet this approach suffers from one-step staleness- gradientss (62] are applied to outdated parameters- wwhich harms convergence 89 Thus it struggles to match the performance or training dynamics of standard LLM training.
774
63
69
In this work, we introduce ACcumulate while you COmmunicate (ACCO), a new optimization method that unifies the benefits of communication-coopuuttii overlap with memory-efficient training, matching the training dynamics of Adam DDP without new hyperparameters to tune. Specifically (1) ACCO allows overlapping gradient computation and parameter updates in the memory-efficient sharded optimizer settings, (2) It climinates the need for outer loops, reducing memory and tuning overhead. (3) Crucially, it compensates for the one-step delay introduced by parallel execution of communication and computation by using a novel accumulation mechanism, which avoids the convergence degradation observed in DPU-style updates.
(4) Unlike prior approaches, ACCO is compatible with sharded training frameworks and requires no warm-up up phase. (5) In the case of SGD, we prove that ACCO achieves the standard convergence rate, and (6) we confirm it empirically: ACCO consistently preserves convergence quality across both homogeneous and heterogeneous envi- ronments, with training curves that mirror those of AdamW-just as our theory predicts. (7) Our experiments on diverse I.IM pretraining and fine-tuning tasks show that ACCO delivers substantial wall-clock speedups and effective communication hiding compared to ZeRO-without compromising training stability memory efficiency. The code to reproduce all our experiments is available at https:// tttp:://github.... thub. com/ edouaddoyalaaaoo acco
2 Related work
Local optimization methods for reducing communications. Local optimization methods per- form several local model updates between periodic averaging. With the SGD optimizer, these algorithms predate the deep learning era 90 39. and their convergence properties are still investi- gated nowadays [87 66 76 42). Due to their practical and efficient communication scheme, they have since been used for the Distributed Training of Deep Neural Networks (DNNs) with methods
such as EASGD 82 SlowMo 77 or Post-local SGD
34 50 and are ubiquitous in Federated Learning 40 29 32]. broadening the choice of optimizers beyond SGD [59 25 10]. They have also
recently been applied in LLM training [7] Overlapping communications over consecutive steps of local computations allows to hide communication bottlenecks, resulting algorithms such as Overlap local-SGD 74. COCO-SGD (3] or CO2 69 Moreover, with heterogeneous hardware, they can adapt their local computation rate to their hardware capacity [33] 38 Yet, this comes at the price of additional memory requirements: due to their local nature M. not only do these

---

Table 1: Comparisons of characteristies and memory consumption. V: number of parameters in the model. N: number of workers. K: memory multiplier the optimizer (Adam or AdamW). For SlowMo [75] and CO2 [6). no mention of mixed precision training made. We assumee they use it and that their additional terms are stored in half precision. While no additional momentum is required for our method, still need negligible communication buffer compared the optimizers' states.
Overlap comm/comp
Hetero. hardware
No outer
Convergence
Memory per replicas (K,N, V - 7.55)
Method
loop
Rates
DDP
31
X
X
GB (2+-2 K ) =
(242+K)y
ZeRO-1
15
X
X
ZeRO-2
55
X
X
/
(2 24 V 16 GB
(2 2
ZeRO-3 155
X
SlowMo
75
X
(2+2+2x2+ 150 GB
DiLoCo
14
X
?
++42+x2++KUL 150 GB
CO2 69
X
(2+2+ K 180 GB
DPU 60
x
x
(2+2+2+ K / = 46 GB
R 1 46 GB
WP 8
x
(2+2+2+
ACCO (Ours)
(2+2422 ) 46 GB
methods prevent the use of sharded optimizers such as ZeRO (55. but they also introduce additional control variables [75 42 (69], hindering their scalability as shown in Tab. Moreover, catering for heterogeneous hardware is not straightforward, as using different numbers local updates leads to models shifting at different speeds, requiring extra to counter this effect (38). On the contrary, ACCO does not lead to such disparities. Since all devices share the same parameters, the device speeds difference only affects the mini-batch size computation. Similarly, doing multiple local optimizer updates makes the approach incompatible with optimizer sharding.
Overlapping communications and computations. For the asynchronous approaches, some ap- proaches overlap gradient and communication steps, either explicitly [51, or by modeling them with independent stochastic processes 45 efficiency. Thus, they introduce additional variables and do not consider sharding the optimizer states. Moreover, they do not study optimizers other than SGD, and extending their beneficial properties to adaptive methods commonly used for DNN training such as Adam is still an ongoing research topic 4. Delays being intrinsic to distributed asynchronous optimization, there a rich literature studying them. In the case of distributed SGD in a parameter server setting, while early analysis showed convergence rates depending on the maximal delay [1 67), recent lines of work improved these dependencies [28 77 19,, proving that asynchronous SGD beats standard mini-batch SGD even with unbounded delays (6). However, they only study plain SGD, which is hardly used for DNN training. In this context, some work focused on the interplay between SGD with momentum and delays 43 81], while delay compensation schemes such as re-scaling updates [86] 78 or buffering them 49 were proposed for Federated Learning. But still, they only study versions of SGD and not adaptive methods commonly used for LLMs training such as Adam (7 or Adam W [36]. Closer to work, DPU was introduced as a memory- effecieet way to train LLMs by running the optimizer on the CPU while gradients are computed on the GPU 660 B3 inducing a one-step delay between the gradients computed and the corresponding optimizer step. To mitigate they advise starting training by warming up for several steps with standard DDP. Perhaps surprisingly, we find in our experiments that this one-step delay has noticeable influence on the convergence of LLMs training, even when using warmup steps. Contrary to DPU, we remove the need for them, with no impact on the convergence of our training. Moreover, as it is not its purpose, DPU still runs communications in the gradient computation stream, and is thus impacted both by the communication overhead of scaling and hardware heterogeneity. Finally, in pipeline parallelism, gradient delays also affect computation, and weight prediction methods have been proposed to mitigate their effect g 79 30 80).
However, none of these works focus on memory
Memory-efficient distributed training of LLMs. The activation memory overhead required for training Transformers 772 can be mitigated for an extra computational cost by reconstructing the input with reversible architectures (62 23 37], or recomputing the activations via checkpointing [9. Efficient LLM training also combines parallelism methods. Classical data parallelism (DP) [22
3

---

suffers both from high communication volume and linear increase in memory due to the model replicas. ZeRO-DP [66] and Fully-Sharded DP [55 avoid this issue by sharding the model states (i.e., the optimizer states, gradients, and parameters) between workers. This comes at the cost of further increasing the synchronization between workers and the communication volume, which
can be mitigated by compression [73], memory trade-offs [33, or delayed gradients [88]. The memory can be even more reduced using expensive CPU-GPU communications to unload states on the CPU 160, 577. On the other hand, model parallelism partitions the DNN components for parallelization, either with tensor parallelism [64] by slicing a layer's computation on several workers, with pipeline parallelism, which divides model into sets of layers trained in parallel mini-batch slices. Popularized by [22). this method leaves some workers idling and an inefficient memory overhead [18]. Allowing delay in the gradients avoids worker idleness [47 [8 but exacerbates the
memory overhead, which can be partially mitigated with gradient accumulation [48 [9] and activation
checkpointing 26 35). Combining these frameworks results in the effective 3D) parallelism (65]
3 Method
3.1
Background and Notations
We now present our framework as well as relevant prior methods for overlapping communication and computation from the perspective of an individual worker i (I,...,,N) NV. The goal is to minimize a differentiable objective function f -" R. All workers are initialized with identical parameters ((0) Rd, and each iteration L, worker i has access to a stochastic function F Rd x E R, where E is a sample space derived from its local data shard. The gradient estimates are assumed to be unbiased: E[V c = (0) for & ~ E. This setup allows for flexible, even time-varying, batch sizes depending on each machine's 's speed. However, for simplicity. we assume each worker computes a fixed-size minibatch of N, samples per iteration. The resulting local gradient estimate is given by
N Σ (x),
(.,( A N
where & = E.N.)
We also consider generic optimizer, such as Adam or AdamM (common in LLM training), denoted by Opt. Applying the optimizer may require synchronizing internal states (e-g., moments, learning rates) across workers, which introduces communication barrier. This synchronization can be particularly costly in settings involving optimizer sharding, and may substantially limit GPU throughput.
Distributed Data Parallelism (DDP). In a DDP setting, gradient computation and optimization steps are performed sequentially as follows, for a sequence of {c(0), f::
9
V F. (0 ,("), g(++1) =Opt (0()) ΣE
N;
(1)
(DDP)
where gradients are averaged across all N workers. As illustrated in this formulation, each step is fully synchronous and must follow a strict order, which limits the potential for overlapping communication and computation. common strategy address this limitation
introduce two parameter buffers, denoted and 8, where one is used for computation and the other for communication. Building on this insight, we next describe the main techniques used to achieve such overlap. and then ACCO.
Delayed Parameter Update (DPU). We describe the original DPU 60), and in our re- implementation, we run gradient communications in the same stream as the optimizer step, in parallel to the gradient computations. To prevent GPU from being idle at step t, gradients are accumulated over as many mini-batches as necessary until the communication process finishes. Then, DPU repeat the following, where each line can be run parallel:
(++1)
V FF(O()), g(0))
R
(DPU)
N g(++1) g(t) g(++1) Opt (BL)) EN L
9
Remark that, except at the first step = 0, the gradients used by Opt are computed parameters g(t) g(t-1) which differ from g(t), the ones we apply them to. This is inherently due to the parallel

---

nature of our execution, and what we denote by "delayed update". Sec. 4.3 shows that this has drastic impacts on the convergence, despite being only delayed by one time step. We hypothesize that, although the delay is limited to = 1 and the dependency on delay is known to be linear [67] the training dynamics differ significantly from those of the standard setring.
Weight Prediction (WP). Proposed by the work of [8] on mitigating pipeline delays, simple estimation strategy is to reuse the most recently received gradients and apply a second optimizer step. Compared to DPU, this modifies the update rule for @(++1), leading to:
((+1) VFF(000,
9
(WP)
g(++1) Opt (BLO) ΣN
Σ, N. (1)
g(t+1) = 0pt (glot), EN Σ, N. 9i
(0)
While this enables overlap by decoupling communication and computation, there is no formal guarantee that it leads to favorable convergence. We empirically evaluate this method in Sec. and observe that its training dynamics deviate from the DDP unliik ACCO.
3.2
ACCO: a structured approach to Commuicatiin-CCompptttii overlap.
V aacc stream
v:
V.
V:
V
V
GPU
comm.Jopt. stream
6,
82
8,
Init.
Step
Step
Stop 3
Step 4
Figure 2: ACCO's 's two-stage mechanism (1)-(2) to compensate the delayed updates via overlapping.
ACCO. ACCO splits the computation of the mini-batch gradients into two successive stages, where the first half of the mini-batch used estimate (i++1) while ((++1) is calculated using the entire mini-batch. This is further motivated by the fact that gradient accumulation can be used to reach the extremely large batch sizes required to train LLMs [84, and if gradients computed sequentially on a worker, we can leverage this to produce our estimate. Thus, the two stages, as in Fig. 2 are
= (((0)) c(0)) (1) g(++1) Opt (()), EN
((+)) (F((0((+1) E(0)
o00) (2)
(((+1) 1 Opt (000, EN GEIDE,,
(1) + +z()))
E, N,
(ACCO)
We next describe the different components, whose streams can be run in parallel:
The computation stream uses the second half of the mini-batch to compute the gradients
9 with respect to parameters g(t) while the communication stream estimates what would be the next steps parameters ((++1) using the estimated gradients g
(2) The computation stream uses the first half of the mini-batch to estimate what would be the gradients of (t+1) of the next parameters g(++1) using estimated parameters @(++1) while the communication stream computes ((+++) using the full mini-batch. Note that starts from the same version of the parameters g(t) as in step (1). The first half g!¹) was estimated at step (2) of the last round, while (1) compute the second half (()
Theoretical analysis of ACCO. We state main results (SGD, for simplicity), with complete proofs provided in the appendix. The key idea underlying all proofs is that, for any minimizer 0" of f and any 17 > O. the following function serves Lyapunov function for dynamics
V(0,0) 0) & f(0) - f(0)) + ()) - f(())) + L 0| 0.
Proposition 3.1 (GD). Let :
Rdd - TR. be L-smooth and 0r arg min f. For S 21 define
0,11 = 02 - (V)(O)) + VI(O)). 0 = 111=0--V/(0)) v/V)
5

---

Then, for any T 2 1, and initializations 0b. 8o E Rd, we have
P-1 [IT((0)]+++I11(0)))) - T/T (f(0))+ f( - ((0) -1.0 - -6012)
T 1=0
Proposition 3.2 (SGD). Under the same assumption as above, suppose Bg 0g and we perform: 0++1 = 8, [g: + +()) B.t1 = 0₂ n-g
where gr and It are unbiased, conditionally independent estimators of V ((t.) and V f respec- rively, with bounded variance o². Then, for n S + and any T 2
2L
T Σ E [IV
S nT (f(0o) - f(0)) + 80² Lm-
16
We note that these rates recover the standard convergence guarantees of GD and SGD, unlike those in (75). Indeed, unlike DPU or WP, ACCO does not rely analysis and faster convergence, as reflected in our proof strategy. One can interpret DPU (with SGD as the optimizer Opt) as a parallel version of Delayed- SeD (D-SGD) with stepstep delay. While this setup has been shown to preserve asymptotic convergence rates in convex settings- susssssuch as quadratics [3 and strongly or quasi-convex functions [68 our
approximation, which leads to cleaner
our experiments (Sec. 4.3 show that, practice, this delay significantly degrades performance when training L.LMs with AdamW. In contrast, ACCO completely avoids delayed gradients, eliminating the impact of staleness. We note that the batch size ACCO corresponds to the number of samples processed between two successive updates of Although uses a pair of stochastic gradients. V FF((() g(+) and VF((()) g()), both are computed synchronously with respect to g(t) (see Fig. We confirm this advantage in Sec.
where ACCO yields training curves nearly indistinguishable from those of DDP (see Figs. 6
4 Experiments
GPU,
GPU)
In this section, we present our experiments. Sec-
ZrR04 Bascline
tion 4.2 details the shared experimental setup.
In Sec. 4.3 we demonstrate the shortcomings of DPU and Wp-initially discussed in Sec. which motivate the design of ACCO. This ini- tial analysis focuses on small language mod- els and datasets, using TinyStories [66] as testbed. Sec. [4.4] shows that ACCO scales ef- fectively by training
GPT-
Parameters
Gradients
Comm. butler
Optimiser State
Neo [6] on OppenWeTTee (21). Sec. [4. pushes
further with instruction tuning of a 2.7B GPT-
Figure 3: Memory requirements of ACCO vs DDP and ZeRO-1, see Tar quantitative details.
Neo model, emphasizing communication bottle- necks and the benefits of ACCO. Finally, Sec. (..6/4.6coopaare ACCO and DDP on heterogeneous hardware, where ACCD lets faster GPUs accumulate updates while waiting unlike DDP resulting in faster gradient computation.
8
4.1 Empirical motivation for ACCO
We first illustrate that the time spent communicating gradients can quickly trump the one used for com- puting them when using standard AdamW DDP to train LLMs. For that, we measure the time neces- sary to perform a full backward pass on Llama-2 model [71] with 7B parameters hosted on a single GPU, using batch size maxing out its memory. We compare this to the time necessary to compute an All-Reduce on those gradients with NCCL, varying the number of distributed workers. We experiment
(t
V comp /comm.
computation
All-Raduce V
8 16 24 32
64
128
workers
Figure 4: Time (per worker) spent comput- ing and averaging gradients of a Llama-2 7B model for different numbers of GPUs.

---

on our local cluster of NVIDIA A100-80GB GPUs with 8 GPUs per node and an Omni-PAth inter- connection network at 100 Gb/s for inter-node connections, intra-node connections being done with NVLink 300 GB/s. Each worker is hosted on single GPU. Fig. 4 shows that the communication time outside of a GPU node in our cluster to average the gradients across workers can take more than 4x the one spent on the whole forward and backward step. As DDP only partially hides communications during the backward [31], this means that our GPUs remain idle the majority of the time when we use more than 24 distributed workers, motivating the need for methods leveraging this time to compute.
Method
Method
ACCO
ACCO
Training loss
DDP
Training loss
DDP
DPU
DrU-wa 500
DPU wart-pp 40
DPU wartuup 500
10000
20000
30000
40000
20000
40000
60000
80000
minibatch
minihatch
(a) Training with the specified amount in 16
(b) Training for twiee the specified amount.
Figure 5: Impact of the delayed update and warmup steps.
4.2 Experimental setup
Our experiments are performed on the GPU cluster described in Sec. 4.1 A detailed pseudo-code for ACCO can be found in App. B.2 Our code is in PyTorch [52]. and we verified that our implementation produces two different streams running in parallel for the computations and communications using NVIDIA's Nsight System to profile it, shown in Fig. 13 We trained all our models with AdamW (36), using mixed precision: our model parameters, gradient accumulation and communication buffers in bfloat16 (44] while sharded optimizer states in single precision (see Fig. 33 As nowadays all distributed frameworks training LLMs at scale use a form of partitioning due to GPU memory constraints [58] 2), our main baseline is PyTorch's 's DDP 31 with 7eRO-1 (55) to shard the optimizer's state. As justified in Tab. T local optimization methods cannot be realistically considered for memory reasons. To compare in good faith DPU ACCO in terms of wall-clock time, also implemented our own version of DPU, as the implementation [61] solves different problem than ours. The original ZeRO does not overlap computation and communication it designed to host the optimizer on the CPU, and is slower than ZeRO due to CPU and GPU memory transfers [60]
4.3 ACCO on TinyStories
We experiment with small language models on the TinyStories dataset [16), closely following their con- figuration and training hyperparameters. We use a 36M -ar-mmrreeterr GPT-Nco- based [6] decoder-only
Method
ACCO
Training loss
DDP
WP
transformer and train BPE tokenizer TinyStories to match their 10k vocabulary. All experiments are
with 8 workers single node.
Impact of delayed updates. First, we investigate the impact of using delayed updates, re- re-purposing DPU [60] described in Sec. 3 We run three vari- ants of this algorithm:
10000
20000
30000
40000
# minibatch
(1) with no warmup. (2) with 40 warmup steps of non-delayed optimization step before switching DPU (done in (60), and (3) with 500 steps of warmup. We report in Fig. [oour training losses on 8 distributed workers averaged over runs. Using delayed updates greatly hurts convergence, especially when no or too few warmup steps are performed. Surprisingly, the number of warmup steps given in 660 does not work here, hinting that a sensitive hyper-parameter to tune for each use-case. If we train for twice as long as specified in [16), then the DPU training curve approaches the bascline one, without totally catching it.
Figure 6: ACCO vs. WP on TinyStories
7

---

Contrary to this, the training curve of our algorithm ACCO perfectly matches DDP, as advocated by our theory.
Compensation via WP. To mitigate the detrimental impact of using delayed updates, we test a first approach to mitigate it: WP as described in Sec. 3 This method applies two consecutive optimizer steps, re-using twice the same mini-batch. The first step produces the usual updated parameters, while the second predicts the parameters of the next step so that gradients can be computed on this estimate rather than on stale version of the model. In Fig. 66, we compare the training curves of this delay-compenssttee baseline at all times, WP displays worse behavior, especially at the beginning of the training. Thus, we dismiss this method and keep ours for the remaining experiments.
method to ours. We remark that, while ACCO perfectly matches the DDP
10
Method
10
Method
ACCO
ACCO
Training loss
Training loss
DDP
DDP
0.0
0.5
1.0
1.5
2.0
0
1000
2000
3000
4000
5000
I minibatch
1c6
Time (s)
(a) Evolution of the loss over the whole training.
(b) Focus on the first part of the training w.r. time.
Figure 7: Training curves for ACCO and DDP with 32 workers trained for 50B tokens.
4.4 Training GPT-Neo on Open Web
To assess how ACCO scales with larger models and more data, we pre-trained a model equivalent to GPT-2 [54] with both ACCO and DDP with a ZeRO optimizer. Specifically, we used the GPT-Neo
architecture 6 with 125 million parameters and the WebText dataset [21], which contains
40 GB of text. We used the GPT-Neo tokenizer, pre-trained on the Pile dataset (00] 'he models were trained on sequences of 1024 tokens, with documents concatenated using end-of-sequence tokens. To assess the impact of using different hardware, the experiment was repeated on 2 different clusters. The first was conducted on 8 H100-PCIe 80GB on a single node. The second was on 32 A100-80G GPU distributed on 4 nodes. We maxed out the memory of our GPUs with local mini-batch size of 24. To reach a sufficiently large overall batch size, we used I step of gradient accumulation for DDP, and none for ACCD as method naturally accumulates over 1 step. resulting for the first and second experiments in respectively 400K and 1.5M tokens per effective batch for report additional experimental details, and notice that training with ACCO allows for a 25% speedup on this pre-training task, which is additionally illustrated in Fig. 7 We also report that our implementation of ACCD adaptively scheduled 315 supplementary accumulation steps over the whole training to prevent GPUs from idling while waiting for communications.
both ACCO and DDP. In Tab.
Further details and results for the H100 exper- iment can be found in App. A Tab. 2 reports the perplexity of trained language models with both methods. We evaluate the perplexity of
Table 2: Perplexity of our trained LLMs
Method
LAMBADA (ppl J) OpenWebText (ppl ()
ACCO 1x8
47.1
24.2
DDP 1x8
47.5
24.3
language models on LAMBADA [51] and test split of Open WebText, and report similar results for both methods.
ACCO 4x8
45.5
22.5
DDP 4x8
44.1
21.7
4.5
ACCO for instruction fine-tuning
In the former sections, we compared ACCO against DDP with ZeRO in the pre-training stage. To further validate our algorithm, we consider the GPT-Neo 2.7B model (6] pre-trained on the Pile dataset (20) and finetuned it on the Alpaca dataset 770 containing 52k pairs of instructtin/answer. We fine-tuned the model using two configurations: A 100-800 on a single node, and 8 A 1000-80 distributed equally across nodes. Samples padded to match the longest sequence in the mini-
8

---

batch. We fixed the mini-batch size at 4, leading to a total batch size of 128 for all methods. For DDP and DPU, we used gradient accumulation of 4. while for ACCO, a gradient accumulation of 2 to account for the ACCO accumulation described in Sec. | The learning rate was set to 2 10-5, and with warmup 50 steps for DPU. In this setting, padding to the longest sequence in the mini-batch induces more variability in the number of tokens per mini-batch. This results in more variability in the computational load for each worker, leading to increased wait times for synchronization. We observe in Fig. 8 that ACCO achieves a low validation loss faster than DDP in both settings. Notably, the difference between ACCO DDP becomes more pronounced when workers are distributed across multiple nodes. Additionally, communication times, further demonstrating the efficiency of ACCO in mitigating communication bottlenecks. This advantage translates to 87% speedup for ACCO (see Tab. 3). highlighting the significant impact of communication bottlenecks on standard methods.
shown in Tab. larger models and optimizers result in longer
Method
Method
Validation loss
1.20
ACCO
Validation loss
1.3
ACCO
DDP
DDP
DPU
1.15
1.10
200
400
600
2000
4000
6000
Time (s)
Time (s)
Figure 8: Validation curve with 8 workers on node (left), and workers/node on 2 nodes (right).
4.6 Experiment Using Heterogeneous Devices
To witness the impact of using het- erogeneous devices, we run ACCO and compare it DDP in a four work- setting, with one of the GPU four times slower than the other three. The training setting is the same in Sec.
Table 3: Pre-training (PT) and finetuning (FT) time speedup
with ACCO against DDP on various setups with GPT-Neo.
Stage
Model
GPUs
#tokens
ZeRO-1
ACCO
1x8
6B
4h41m
4h25m
PT
125M
4x8
50B
14h41m
10h55m
4.3 As we experiment on a A100 GPUs cluster, erogeneity the hardware using the time s.mepp () python command. First, measure the time that standard forward-backward step takes, and make one of the four GPUs idle for three times this amount after each pass. In this context, DDP is only fast the slowest worker: 3 out of the 4 workers idle 3/4 of the time. With ACCO, the other workers accumulate during the time they wait for the slow one to finish. Thus, ACCO allows compute gradients for large batch sizes faster than standard baselines, resulting significantly smaller wall clock time, as shown in Fig.
1x8
80M
43min
25min
FT
2.7B
simulate the het-
2x4
80M
3h46m
29min
Method
ACCO
# V acc./ round
300
Training loss
3
DDP
/V
6
2
(1m))
V
200
(ms)
100
m
(m/)
2
17
500
1000
1500
3
Time (8)
GPU rank
Figure 9: Training curves with slow worker (4x slower).

---

Conclusion
ACCO is a novel algorithm that addresses both memory and communication bottlenecks in distributed LLM training, with provable guarantees which match standard SGD. By overlapping gradient computation and communication while partitioning optimizer states, ACCO reduces communication overhead in memory-efficient way. A new two-stage compensation mechanism corrects for delayed updates, ensuring convergence comparable to standard optinizers-withou requiring warmup. Empirical results show significant speedups across pre-training and fine-tuning tasks, particularly in multi-node and heterogeneous environments.
Acknowledgements
This work was supported by Project ANR-21-CE223-0000 ADONIS, EMERG-ADONIS from Alliance SU, PEPR IA (grant SHARP ANR-23-PEIA-0008)) and the Sorbonne Center for Artificial Intelligence (SCAI) of Sorbonne University (IDEX SUPER II-DDEX-0004), It was granted access the Al resources of IDRIS under the allocation 2023-A015 10145526 made by GENCI. EB acknowledges funding from FRQNT New Scholar and computational support from CFI.
References
[1] Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In J. Shawe- Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
[2] Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit. Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien. Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling PyTorch, 9 2023.
[3] Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Aryeh Kontorovich and Gergely Neu, editors, Proceedings of the 31st International Conference on Algorithmic Learning Theory, volume 117 of Proceedings of Machine Learning Research, pages 111-132. PMLR, 08 Feb-11 Feb 2020.
[4] By Mahmoud Assran, Arda Aytekin, Hamid Reza Feyzmahdavian, Mikael Johansson, and Michael G. Rabbat. Advances in asynchronous parallel and distributed optimization. Proceed- ings of the IEEE, 108(1 2):113222031 2020.
[5] Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 344-353, PMLR, 09-15 Jun 2019.
[6] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflww, March 2021.
[7] Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication -fffrcccnntf language model training scales reliably and robustly: Scaling laws for diloco. arXiv preprint arXiv:2503.0977 099999 2025.
[8] Chi-Chung Chen, Chia-L.in Yang, and Hsiang- Yun Cheng. Efficient and robust parallel dnn training through model parallelism multi-gpu platform, 2019.
[9] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016.
[10] Xiangyi Chen, Xiaoyun Li, and P. Li. Toward communication efficient adaptive gradient method. Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, 2020.
[11] Yangrui Chen, Cong Xie, Meng Ma, Juncheng Gu, Yanghua Peng, Haibin Lin, Chuan Wu, and Yibo Zhu. Sapipe: Staleness-aware pipeline for data parallel dnn training. Advances Neural Information Processing Systems, 35:17981-17999 2022.
10

---

[12] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoe Le, and Andrew Ng. Large scale distributed deep networks. In F. Pereira, C.J. Burges, I.. Botton, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[13] Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitriy Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, Thomas Wolf, and Gennady Pekhimenko. Distributed deep learning in open collaborations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[14] Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training language models. arXiv preprint arXiv:2311.08/00 2023.
[15] Sanghamitra Dutta, Jianyu Wang, and Gauri Joshi. Slow and stale gradients can win the race.
IEEE Journal Selected Areas in Information Theory, 2(3): 231):0022-1024 2021.
[16] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
[17] Mathieu Even, Raphaël Berthier, Francis Bach, Nicolas Flammarion, Hadrien Hendrikx, Pierre Gaillard, Laurent Massoulié, and Adrien Taylor. A continuized view nesterov acceleration for stochastic gradient descent and randomized gossip. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[18] Shiqing Fan, Yi Rong, Chen Meng. Zongyan Cao, Siyu Wang. Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. Dapple: A pipelined data parallel approach for training large models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 43 1445, 2021.
[19] Hamid Reza Feyzmahdavian and Mikael Johansson. Asynchronous iterations in optimization: New sequence results and sharper algorithmic guarantees. Journal of Machine Learning Research, 24(158): 2023.
[20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000
2020.
[21] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. http://Skylion007, git..g..t............ io hoo/ppenneeTTextCrpppue 2019.
[22] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[23] Jömn-Heriik Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference Learning Representations, 2018.
[24] Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang. Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of 6 for deep learning training, 2019.
[25] Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. ArXiv, abs/2008.03660 2020.
[26] Chiheon Kim, Heungsub Lee, Myungryong Jeong, Woonhyuk Baek, Boogeon Yoon, Ildoo Kim, Sungbin Lim, and Sungwoong Kim. torchgpipe: On-the-fly pipeline parallelism for training giant models, 2020.
[27] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). San Diega, CA. USA, 2015.
[28] Anastasia Koloskova, Sehastian U. Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous sgd for distributed and federated learning. In Proceedings of the 36th International

---

Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc.
[29] Jakub Konecný. H. B. McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. ArXiv, abx/1610.02527. 2016.
[30] Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, and Urs Köster. Pipelined backpropagation at scale: Training large models without batches, 2021.
[31] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: experiences on accelerating data parallel training. Proc. VLDB Endow. 13(12)3005-3318, aug 2020.
[32] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization for heterogeneous networks. In ICML Workshop on Adaptive & Multitask Learning: Algorithms & Systems, 2019.
[33] Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, and Alexander Schwing. Pipe-sgd: A decentralized pipelined sgd framework for distributed deep net training.
Advances in Neurul Information Processing Systems, 31, 2018.
[34] Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't 't use large mini-batches, use local sgd. In International Conference on Learning Representations, 2020.
[35] Yuliang Liu, Shenggui Li, Jiarui Fang, Yanjun Shao, Boyuan Yao, and Yang You. Colossal-auto: Unified automation of parallelization and activation checkpoint for large-scale models, 2023.
[36] Ula Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[37] Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuuu Yuan Wu, Bo Xiong, Christoph Feichten- hofer, and Jitendra Malik. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10830-10840, 2022.
[38] Artavazd Maranjyan, Mher Safaryan, and Peter Richtárik. Gradskip: Communication- accelerated local gradient methods with better computational complexity, 2022.
[39] Ryan McDonald, Keith Hall, and Gideon Mann. Distributed training strategics for the structured perceptron. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT '10, page USA, 2010. Association for Computational Linguistics.
[40] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-EEicc
Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273-1282 1282. PMLR, 20-22 Apr 2017.
[41] Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[42] Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! arXiv preprint arXiv:2202.09357. 2022.
[43] Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher Ré. Asynchrony begets mo- mentum, with an application to deep learning. In 2016 54th Annual Allerton Conference Communication, Control, and Computing (Allerton), page 997-1004, IEEE Press, 2016.
[44] Adel Nabli, Eugene Belilovsky, and Edouard Oyallon. $ltextbf (Att///tttt celerating asynchronous communication in decentralized deep learning. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Ac-
[45] Adel Nabli and Edouard Oyallon. DADAO: Decoupled accelerated decentralized asynchronous optimization. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 25604-25626. PMLR. 23-29 Jul 2023.
12

---

[46] Giorgi Nadiradze. Ilia Markov, Bapi Chatterjee, Vyacheslav Kungurtsev, and Dan Alistarh. Elastic consistency; A practical consistency model for distributed stochastic gradient descent. 2021.
[47] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1-15, 2019.
[48] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory- efficient pipeline-paraaln parallel dnn training. In International Conference on Machine Learning, pages 7937-7947. PMLR, 2021.
[49] John Nguyen, Kshitiz Malik, Hongyuan Zhan, Ashkan Yousefpour, Mike Rabbat, Mani Malek, and Dzmitry Huba. Federated learning with buffered asynchronous aggregation. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th Inter- national Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages
PMLR, 28 30 Mar 2022.
[50] Jose Javier Gonzalez Ortiz, Jonathan Frankle, Mike Rabbat, Ari Morcos, and Nicolas Ballas. Trade-offs of local sgd scale: An empirical study. In NeurlPS 2020 OptML Workshop, 2021.
[51] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534, Berlin, Germany, August 2016. Association for Computational Linguistics.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. 7ach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: an imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
[53] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D. Sinclair. Computation vs. communication scaling for future transformers on future hardware, 2023.
[54] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[55] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimiza- tions toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
[56] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models, 2020.
[57] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning, 2021.
[58] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20 page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery.
[59] Ja Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
[60] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang. Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scal model
training. 2021.
[61] Jie Ren, Samyam Rajbhandari, Reza Ruwase, Shuangyan Yang, Minjia Zhang,
Yazdani Dong Li, and Yuxiong He.
Aminabadi,
Olatunji
2022.
13

---

[62] Stephane Rivaud, Louis Fournier, Thomas Pumir, Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. PETRA: Parallel end-to-end training with reversible architectures. In The Thirteenth International Conference on Learning Representations, 2025.
[63] Shuheng Shen, Linli Xu, Jingchang Liu, Xianfeng Liang, and Yifei Cheng. Faster distributed deep net training: computation and communication decoupled stochastic gradient descent. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, page 4582-4589, AAAI Press, 2019.
[64] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:I909, 08053, 2019.
[65] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron to train megatron-tumm -uring nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.119900 2022.
[66] Sebastian U. Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019.
[67] Sebastian U. Stich and Sai Praneeth Karimireddy. The error-feedback framework: better rates for sgd with delayed gradients and compressed updates. Journal of Machine Learning Research, 21 21(1), jan 2020.
[68] Sebastian U. Stich and Sai Praneeth Karimireddy, The error-feedback framework: better rates for sgd with delayed gradients and compressed updates. J. Mach. Learn. Res., 21(1), January 2020.
[69] Weigao Sun, Zhen Qin, Weixuan Sun, Shidi Li, Dong Li, Xuyang Shen, Yu Qiao, and Yiran Zhong. CO2: Efficient distributed training with full communication-com comptttttiooommtiiiionoioooo overlap. In The Twelfth International Conference on Learning Representations, 2024.
[70] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instructioo-following llama model. https://gi thggtthu..o.. wwt.///tattlaaa/ttaaaaaae ord_ alapaaaapaaaaapppele 2023.
[71] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cymthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[73] Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He, Zero++: Extremely efficient collective commu- nication for giant model training, 2023.
[74] Jianyu Wang, Hao Liang, and Gauri Joshi. Overlap local-sgd: An algorithmic approach to hide communication delays in distributed sgd. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, May 2020.
[75] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communication-effcieen distributed sgd with slow momentum, In International Conference on Learning Representations, 2020.
14

---

[76] Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai. Brian Bullins. Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10334-10343. PMLR, 13-18 Jul 2020.
[77] Xuyang Wu, Sindri Magnusson, Hamid Reza Feyzmahdavian, and Mikael Johansson. Delay- adaptive step-sizes for asynchronous learning. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th Inter- national Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 24093 221 13. PMLR, 17-23 Jul 2022.
[78] Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. In NeurIPS 2020 OptML Workshop, 2020.
[79] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dan training. In A. Smola, A. Dimakis, and I. Stoica, editors, Proceedings of Machine Learning and Systems, volume 3, pages 269-296,
2021.
[80] Bowen Yang. Jian Zhang, Jonathan Li, Christopher Ré, Christopher R. Aberger, and Christo- pher De Sa. Pipemare: Asynchronous pipeline parallel dnn training, 2020.
[81] Jian Zhang and loannis Mitliagkas. Yellowfin and the art of momentum tuning. In A. Talwalkar, V. Smith, and M. Zaharia, editors, Proceedings of Machine Learning and Systems, volume 1, pages 289-308, 2019.
[82] Sixin Zhang, Anna Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 685-693, Cambridge, MA, USA, 2015. MIT Press.
[83] Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. Mics: Near-linear scaling for training gigantic model on public cloud, 2022.
[84] Wayne Xin Zhao. Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023.
[85] Yanli Zhao, Andrew Gu, Rohan Varma, Liang I Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
[86] Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tic- Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 41 4120-4129. JMLR.org, 2017.
[87] Fan Zhou and Guojing Cong. On the convergence properties of k-step averaging stochastic gradient descent algorithm for nonconvex optimization. In Proceedings of the Twenty-Seventh International Joint Conference Artificial Intelligence, IJCAI-18, pages 3219-3227. Interna- tional Joint Conferences on Artificial Intelligence Organization, 7 2018.
[88] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Accumulated decoupled learning: Mitigating gradient staleness in inter-layer model parallelization. arXiv preprint arXiv:20/2.03747, 2020.
[89] Huiping Zhuang, Yi Wang, Qinglai Liu, and Zhiping Lin. Fully decoupled neural network learning using delayed gradients. IEEE transactions on neural networks and learning systems, 33(10))6013-6022, 2021.
[90] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. In J. Lafferty, C. Williams, J. Shawe Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The introduction and abstract clearly state the goals and contributions of
ACCO, which are substantiated through both theoretical analysis (Section 3 and experiments (Sections 4.6)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
For instance, Table discusses the tradeoff's of each method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
3. Theory assumptions and proof'
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Ys]]
Justification: Section3Jroovder the assumptions and sketches of the proofs; full convergence
analysis and Lyapunov function details are referenced and further elaborated in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
16

---

Proofs can appear in the main paper the supplemental material; if in supplemental material, include short proof sketch in the main text.
Theorems and Lemmas relied upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
describes the full environment and configurations, including model architecture, hardware specs, optimizer, precision formats, and batch sizes. Details are further supported by Appendix sections and figures.
Justification: Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the contribution is dataset and/or model, the authors should describe the steps taken to make results reproducible or verifiable.
Depending the contribution, reproducibility may require full architecture details, commands, or hosted access; see NeurlPS policy for options.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The paper states that the code will released a public open-source reposi- tory at publication time (end of Introduction). All datasets used are publicly available.
Justification:
Guidelines:
See NeurIPS code/data submission guidelines
CodeSubmi ssiospolicy..
Include exact commands and environment when feasible; anonymize at submission time if applicable.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
and 4.5 specify training configurations, datasets, opti- mizer, batch sizes, accumulation strategies, and evaluation metrics.
Justification: Sections 4.2
Guidelines:
Core details should appear in the paper; full details can be in code, appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ys]]
Justification: For TinyStories experiments, results are averaged over 3 runs (see Section 4.3) trends are visualized. For large-scale runs, we show consistent convergence behavior.
Guidelines:
Clearly state what variability factors error bars capture and how they computed. Clarify whether bars are standard deviation or standard error; use asymmetric bars when appropriate.
17

---

8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
specifies compute resources including GPU model, memory, interconnect bandwidths, and training durations (see also Table
Justification: Section 4.2
Guidelines:
Indicate worker type (CPU//PU), cluster/cloud details, memory/storage. Provide per-run and total compute estimates when feasible.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss///eerrrs..c./bbbb//////i
Answer: [Yes]
Justification:
The research uses only publicly available datasets, complies with best practices for reproducibility, and does not raise ethical concerns.
Guidelines:
If "No", explain the special circumstances requiring deviation; preserve anonymity as required.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This work proposes training algorithm and does not directly deploy an application with societal impacts; we discuss potential implications where relevant.
Guidelines:
NA No, explain why impacts out of scope.
If applicable, discuss misuse risks and possible mitigations.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release high-risk assets; standard benchmarks and models are reused under permissible terms.
Guidelines:
If releasing high-risk assets, describe safeguards (access controls, filters, usage guide- lines).
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets (OpenWebText,, Alpaca, TinyStories) and baselines (e.g., GPT- Neo, ZeRO) are cited with sources/authors and licenses where applicable (Sections 4.3 4.4
and references).
Guidelines:
Cite originals, state versions/URLs and include license names (c.g., CC-BY 4.0).
18

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new datasets or models are released in this work.
Guidelines:
If releasing assets, include documentation, license, limitations, consent, and anonymiza- tion at submission time if needed.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve human subjects or crowdsourcing.
Guidelines:
If central to the paper, include as much detail as possible in the main text; compensation should meet local minimum wage.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether IRB approvals (or equivalent) were obtained?
Question:
Answer: [NA]
Justification:
No human subjects are involved.
Guidelines:
obtained, clearly state IRB (or equivalent) approval; keep anonymity for initial submissions.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research?
Question:
Answer: [Yes]
Justification: The paper centers LLM training/fie-ttningg: usage appears throughout (e.g., GPT-Neo 125M/2.7B in Sections 4.5
Guidelines:
See LLM
A Experimental Details and Further Results
A.1 Pre-training on TinyStories
we used the configuration available the Huggingface Hub trained our own 10k vocabulary tokenizer on the dataset. We also report in Fig. the results of our study the impact of halving the batch size for DPU by not performing any gradient accumulation (i.e., performing an optimizer's step at each communication).
For experiments in Sec. 4.3
We
TTny Stories Available http:///wgggaucccccccccoeeooooss
19

---

Method
ACCO
DDP
Training loss
DPU-warmup
DPU-warmup 40
5000
10000
15000
20000
25000
30000 35000
# minibatch
Figure 10: Comparison between running DPU 8 GPUs with 2 steps of gradient accumulation on each (to match the standard batch-size and DPU with only gradient accumulation step. Doing so allows double the number of optimizer's step per minibatch, but divides the effective batch size by 2. This leads to faster convergence early in the training, but worse training loss in the end.
A.2 Proofs of Sec.
first prove the convergence of ACCO in the Gradient Descent case.
Proposition A.1 (Gradient Descent Case), Let f TR.. R. be an L--soooh function, and consider the iterates defined by
01+1 0, (T((0.)+(((0) -
=
initialized at 0o, On E IRd. Assume that J admits global minimizer 0* E arg min S. Then, for any and step size 11 S t, the following bound holds:
8 < 1/T (1(0b) (100)+((000) -(00)-2((00) + 00ll²)
Proof. We define the Lyapunov potential:
V(0,0)== - ( - -(0)) + 7b
- (100)-(00)) -1110 -012
20

---

Using the L-smoothness of f. we apply the standard descent lemma:
f(0c41) - f(6,) < 2 (())) ( + vf(()) Ind² 8
- -V](()) - v
Ini
(0000))+++>>/00/ + V((.))
+ 8
L1 ! + Lni) |( -SS ( + (-2 LnF
V((.))
2
4
ll + Inp²
[S(0,)] + Lip
|V
2
8
8
1 + 4
Lnf
1
(IV 2 + -V -V - -V(())))(()))))))
+
2
2
! +
Ln
1 + 11²) 2
4 [())(1))1
-
2
8
2
L
(
- + Ln²) )) |V
+
8 2
2 2
1
1 +
[V v((E))1
2
2
(-
31
Ln 1V5(0)11 + (-2 + LUd
I [
4
4
LP)
11
IVS()))) - v0
8
37
1VS(0)11 (-2 + Lnd²)
| (
4
4
4
+ 11 Lni²)
1.2100, 0,112
For the second point update, again using smoothness:
1(0+11) - f(0.) < VT(..) T (0, - 0, - -V)((()) + L
00 -0 1/ f(())) |
2
0, 0. 2 + ( - 7) / (0)) ||
L
|0, - 0. - - /
+
2n
2
2n
0
8.
|| + 7 2 |( +
0r
8. n/ r
2m
2
2m
The change in the regularization term satisfies, using T-smoothness:
110--.
1 4 VS(0.) - -
-|00) -.,]== 2/ s (L29 x(LÁr-
0.-
-1) 1) 10. 0.11
Thus, for the sake of readability 0 < 1 $ . we have that:
f(0+41)-f(0) < 8 | "
||= + 88.
L 0. 8, 2
21

---

and
f(0x-1) - f(0,) - 2n
100 0c11
and
3 -11 1100--011
102+1 02+2112-11 -
S
Combining the three terms (two function values and one regularizer), the total Lyapunov change is:
V(O++1, 0411) V(O,, 0.) - "
1
[V
1 8 V (
||
+
L
L 3L
4
0.
0, ||
8
2
1 [Vf[elI
1
V (f
- 8"
Summing over t to and noting V(Or. 0r) 2 O, we conclude:
- -V0, _ (1(00) (100)+ -2((0) 4|00 - B0ll3)
1)
Dividing both sides by gives the claimed result.
We now prove the convergence of ACCO the Stochastic Gradient Descent case.
Proposition A.2 (Stochastic Gradient Descent Case with Bounded Variance). Let f Rd + be an L-smooth function, and let Oo 0o E Idd be the initialization. Suppose we perform the updates:
7 (g. 9.). 0.+1 = -
0.+1 0 2
where 9t and ğt are unbiased stochastic gradients of f. conditionally independent, at O₂ and 0 respectively:
E[g. Oe, 0.] IV(()),
Elge 0c, 0.] =VS(O.),
and assume the variance is bounded as
E [lge ((())1 S o²,
E 1 s s².
Then, for any > and step size S
1 27, it holds that
Σ
[ + [VS())))
S
16 nT (S(0)) f(0))) + 80? Lap-
Proof.
Using the approach in the full gradient case and L-smoothness, we get:
Ln²
E ((0++1) 0, < f(..) (V)))) E[g. [Fgg. + Øe O2, 0.) +
E
[1g. + 8:l12 0z, 02
8
E f(0++1) 8,, 0,,0 < f())) + (..) T (0: - 8, = 6, 0,)) +
L
[10. 0, nāell 1?? 10 06
2
We also expand the expected change in the quadratic term:
[10e+1 - Oet1ll 12 0r, 0, = E
1
1 (gt
ūt) + (0, 0.)1 1 or, 0.
v? E [10x 3el12 02,00 0. + n/0,
= = 0111 +
0 E 9: ge 0z, 0
4
22

---

Further simplufications give
E
10: 0.
Bc. 82 110 0, (o-vs((())) + +n Var(āe 0e),
[19: Tell 10, = =VV](0)+++ + Var(g: 0,) + Var(ā: 0c) E2
E2
[19. Tell 0. = - Var(ge 0,) + Var(ā 0z)
Using the bounded variance a we can follow the same derivation as in the full-gradient case, with extra o² terms appearing.
This yields:
E V(0+41, 01+1) 01.08 0.] - -V(e, 0,) - - [V 8"
1
1
f(b:))
Ln
I2²1}
Lnj
+ 0
+
2
4
Taking expectations and summing over to T 1, then rearranging and using 1/T' S 1 and
Oo = 0g we obtain:
T [IV
16 (f(Oo) - f(0)) + 80 Lm.
n/T
Note that it would be possible to derive bounds for non-increasing step size, as (00 (f(0) f(())) + 7d (f()) f(0r) + L|l0 - 0]| satisfies V(e, 0, n) s V(0, 0, n) for S
A.3 Pre-training on Open Tex
For all pre-training experiments on WebText, the configuration used to instantiate the GPTNeo 125M is available the Huggingface Hut?] We only changed the imax_position_enile embeddingg" parameter from 2048 to 1024. More details are displayed in Tab. 4 We used the Open WebText dataset available Huggingface"] We also report in Fig. 11 further results for the pre-training H100 GPUs.
A.4 Instruction Fine-Tuning
For all fine-tuning experiments, we used the pre-trained GPT-neo 2.7B available on the Huggingface
Hut and the associated tokenizer. We used the Alpaca dataset available Huggingface" More details are displayed in Tab. 5 We also report in Fig. 12 further results on the experiment described in
Sec. 4.5
23

---

10
Method
ACCO
Training loss
DDP
6x 10
4 10"
3 10
50000
100000 150000 200000 250000 minibatch
Figure 11: Training loss during training on OpenWebText with 8 H100 GPUs and 6B tokens.
Table 4: Training hyperparameters for ACCO and DDP configurations.
Hyperparameter
H100
32 A100
mini-batch_se
24
24
lngrad_accumulan
ACCO: -DDP:
ACCO: -DDP:
sequence_les
1024
1024
#tokens_batcr batch
400K
1.5M
optimizer
Adam W
AdamW
learning rate weight decay adam_betal adam adam_beta2 nb jb_steps_tor tot
6e-4
6e-4
0.1
0.1
0.9
0.9
0.95
0.95
50000
50000
scheduler
cosine
cosine 0
warmmp_sstep
Method
Validation loss
1.20
- ACCO
DDP
DPU
1.15
1.10
5000
10000
15000
20000
# minibatch
Figure 12: Validation curve with 8 workers single node.
24

---

BBOO
Cuda kemels on accumalation stream
Communication
streams
accumulation stream
redacs the gro Counter
Peccuso scattr the gradlets
accumulation stream
Communication
accumulation stream
sine saler
Cone optinar
number gradirt
maci
toffer to
butte
Figure 13: Nsight system profile of our implementation of ACCO: our two steams do run in parallel. In this Figure, the computation take more time than the communication because only profiled small scale experiments with 8 workers, and small number of parameters (36M as we profiled on our TinyStories [16] setting). This changes when using larger models and more workers, seen in 4.1
25

---

Table 5: Finetuning hyperparameters for ACCO, DDP and DPU configurations.
Hyperparameter
ACCO
DDP
DPU
mini-batch size
n g_grad_ accumulation
2
4
4
total batch_size
128
128
128
optimizer learning_ran rate weight_decay adam_betal adam_betal nb steps tot scheduler
Adam W
Adam W
Adam W
2e-5
2e-5
2e-5
0.0
0.0
0.0
0.9
0.9
0.95
0.95
0.95
50000
50000
50000
cosine
cosine
cosine
n warmup_ steps
50
B
Implementation Details
B.1 Profiling Results
B.2 Algorithm Pseudo-Code
We present our algorithm for time-varying batch size N!
2GPT-neo 125M Configuration Available http:///hhgiiiigeecccoooooooolsp
blob/main/com conffogjsoooooooooo
Tee Dataset
Available
at:
http:///uugiigseeoccrrsssssssss"
opensebtext
"GPT-neo 2.7B Available at: httpa///waggnniseeeooo//eeeeeeet
AAlpaca Dataset Available at: http::/wwggnnggccco///tttttttttt
26

---

B.3 Slurm script to reproduce our results
Listing SLURM and ACCO fine-tuning configuration
#SBATCH nodes=2 Request 2 nodes
#SBATCH ggge--ggui GPU per node
#SBATCH --naass--per nor-noee1 # task per node
acco-ft: false
batch_size: n.grad_accumulationn: 4
learning_rate: 1e-5
weight_decay: 0.0
adam_betal:
adam adam_beta2: 0.95
nb_ nb_atepa_tots tot: 50000
aicalloader_mmm.vvvreess.
datalloader..inn.eemoryy True
catualouder__orristtet_..eeeee True
Label_smooching_factoo
max_length: 512
scheduler_name: warmup: 'cosine'
save: False pe__mixed_precissie True
w.armup_steps::
bwseline_ False True for DDP
method_name: 'acco' 'ddp' for DDP
Add for DDP
eval: True
eval_step: 10
Lun_expe__et False
const_len__bt False
finetune: True
27

---

Algorithm Training with ACCO in parallel for worker
1: Input: Model with differentiable loss F, number models N, initial parameters @ training steps T, dataset shards D,-
2: Initialize: gradients gi (-1) = FF((O)) ((0) and number of gradients N!-1)
3: # Computation stream
4: while < do
5: Stage 1.
6: while not Seady_for_ttagee do
6 Dj
7:
9i + Di + VF(O() )
8:
N(00 NoO) +1
9:
g(++1) Buffer;
Buffer; + (N "9:"
(4)
11:
12: Stage 2.
13: while not Maady_for_Stere
do
((t) D;
14:
9. + (t)
+ VF(Ot++)) e(())
15:
N((0) N((0) +1
16:
17:
1-t 1+1
18:
g(++1) Buffer;
Buffer; (N())
20:
21: # Communication stream
22: while True do
Stage
24: (N 1.
(()) + Buffer;
N All (e)
Shard; E.g, (1)
Reduce_Scatterr()])
Shard; (i((+)) <- ShardedOpt Shard, (0(0). SSaadd
E Σ, N, 1T
27:
Buffer; + All All_Gather(Shard, (0(++1))
28:
NZ())
29:
0
30: Ready_ For_Stage___ + True Sady_for_Stagen
False
32:
Stage 2.
(N (0) Buffer;
33:
N + (t) + All_llReeueal (e]_ceduce + N,OO
34:
Shard, EEg. + +3.")) + Reduce_Scatter( (1)
+ Σ, )!
35:
E. 2. Σ. N
))
36:
Shard, (0(++1)) +- ShardedOpt
Shard, (0(1). Shard,
37:
Buffer, All All_Gather(Shard, (o(++1)))
N(01
38:
39:
Bady_foo_Stagee 1 True
40:
2 <- False
28


Reviews:
[{"review_id": "SQPmrxam9X", "reviewer": "Reviewer_2aEN", "summary": "This paper proposes ACCO, a novel distributed optimization algorithm designed to overlap gradient computation and communication while remaining compatible with sharded optimizers in LLM training. Unlike previous methods such as DPU or CO2 that either incur significant memory overhead or suffer from stale updates, ACCO introduces a two-stage mini-batch accumulation mechanism that compensates for update delays and avoids convergence degradation. The approach is proven to preserve standard convergence rates theoretically (for SGD) and empirically matches the performance of standard AdamW + DDP baselines. Experiments across multiple model sizes, tasks (pretraining and fine-tuning), and hardware settings (homogeneous and heterogeneous GPUs) demonstrate significant speedups (up to 87%) over ZeRO-1/DDP without loss of convergence quality.", "questions": "1. Why is the baseline comparison limited to ZeRO-1? Since ZeRO-2 and ZeRO-3 are more commonly used in large-scale deployments, a comparison or at least a discussion would strengthen the claims of practical relevance.\n\n2. Although ACCO splits the mini-batch into two halves and does not repeat computation for the same data, the method still incurs a small overhead due to sequential gradient estimation on two partial batches. Have the authors quantified this overhead compared to standard DDP, especially in terms of GPU utilization and FLOPs per sample? Is the slight increase in computation justified by the communication savings at larger scales?\n\n3. Since ACCO computes gradients on slightly stale parameters, how sensitive is convergence to the quality of this estimate, especially under noisy or non-iid data?", "limitations": "Yes. The paper includes theoretical and empirical discussions on the impact of delayed gradients, memory trade-offs, and heterogeneous device behavior.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1.The idea of using a two-stage gradient estimation to compensate for delayed updates is both elegant and effective, especially in sharded setups.\n\n2. ACCO addresses a real and increasingly important bottleneck in LLM training — communication overhead and GPU idle time, particularly in multi-node and heterogeneous settings.\n\n3. The convergence guarantees (under standard assumptions) are well presented, matching vanilla SGD’s behavior.\n\n4. The paper provides comprehensive experiments, including ablation studies (DPU, WP), large-scale benchmarks (GPT-Neo), and heterogeneous hardware simulations.\n\n**Weaknesses**\n\n1. While the authors adopt ZeRO-1 as the primary baseline, more advanced and widely adopted strategies like ZeRO-2 and ZeRO-3 offer deeper memory savings and communication optimizations through more aggressive state and gradient partitioning. The absence of comparison with these stronger baselines makes it difficult to assess the practical relevance and relative advantage of ACCO in modern LLM training pipelines.\n\n2. The theoretical analysis provided in the paper—particularly the convergence guarantees—is grounded entirely in the context of SGD. However, all empirical results rely on AdamW, an adaptive optimizer that has significantly different dynamics. This mismatch raises concerns about whether the theoretical claims extend to the practical setting, especially given that prior work has shown that delay compensation can behave differently under adaptive optimizers.", "was_revised": true, "final_justification": "I appreciate the author's detailed response, which has satisfactorily resolved my concerns. As such, I will retain my original score.", "rebuttal": "We thank the reviewer for recognizing that our work tackles an **important problem** and for highlighting the strengths of our submission, notably the **\"comprehensive experiments\"** and **\"convergence guarantees.\"** We now address the points raised for improvement in detail:\n\n---\n\n**Q1/W1: “Why not ZeRO-2 or ZeRO-3 with ACCO?”**\n\nWe appreciate the reviewer’s insightful suggestion. Our choice to focus on `ZeRO-1` was driven by clarity and simplicity in isolating the core effects of ACCO. Importantly, integrating ACCO with `ZeRO-2` or `ZeRO-3` does not affect the final training accuracy but alters the memory footprint, communication latency, and training time. ACCO is fully compatible with these variants: `ZeRO-2`/`ZeRO-3` shard the optimizer and gradient states across devices, and in this setup, ACCO operates by using a sharded model and gradients on the computation stream. While memory benefits are preserved, the computation stream incurs additional communication overhead to synchronize updates, which introduces slowdown.\n\nTo reflect this, we will revise **Table 1** to explicitly report memory usage under `ZeRO-2` and `ZeRO-3` by dividing the parameter and gradient buffer memory by the number of devices, consistent with the optimizer state sharding.\n\nWe have conducted preliminary experiments on CIFAR-10 using two GPUs and observed a similar runtime slowdown when applying ACCO or not with `ZeRO-2`/`ZeRO-3` on the computation stream. These results confirm our expectation. Nonetheless, we agree that optimizing these variants in conjunction with ACCO is a promising research direction and plan to investigate efficient implementations as future work.\n\n---\n\n**W2: “Only the proofs for Adam in ACCO?”**\n\nWe thank the reviewer for raising this important point. It is a deliberate choice to restrict our analysis to SGD, grounded in clarity and scope. The convergence of adaptive optimizers like Adam is a subtle and complex topic; indeed, works such as *“A Simple Convergence Proof of Adam”* (2020, Deffossez et al.) focus entirely on that challenge and are quite involved despite the name.\n\nOur goal in this paper is to introduce a practical and effective algorithm. Accordingly, our analysis of ACCO with SGD serves as a sanity check, providing intuition and formal reassurance that ACCO preserves training dynamics. Extending these guarantees to Adam is out of scope for this paper, but is an exciting direction for future work.\n\nThat said, our empirical results show that ACCO reproduces Adam’s behavior extremely well (see **Figure 5** and **7**). This is strong evidence that ACCO’s core mechanism remains stable and effective even with adaptive optimizers. Additionally, to reinforce the generality of our method, we include results using SGD on CIFAR-10 with the DLA model and cosine scheduler, where `ACCO+SGD` achieves **95.3% accuracy**, matching vanilla SGD.\n\nWe will clarify this design choice more explicitly in the final version of our manuscript.\n\n---\n\n**Q2: “Maximise GPU throughput while halving batch size?”**\n\nWe fully agree with the reviewer’s assessment. In terms of computation: each sample is processed only once, so FLOPs per sample are effectively unchanged compared to DDP. While ACCO introduces a small overhead by sequentially computing gradients on two partial batches, this overhead is not a concern as long as GPU utilization remains saturated.\n\n---\n\n**Q3: “How does ACCO handle noisy or non-i.i.d. gradients?”**\n\nThis is an excellent theoretical point. The robustness of ACCO to noise and non-i.i.d. data is explicitly addressed in **Proposition 3.2**, where the variance term \\\\( \\sigma^2 \\\\) captures stochastic noise. Non-i.i.d. effects introduce correlations in gradient noise, which appear in the covariance structure of the same bound. Consequently, ACCO’s sensitivity to noise is in line with standard SGD under similar assumptions.\n\nEmpirically, we observe that ACCO outperforms delayed SGD (DPU) and matches the performance of standard SGD even in regimes with increased gradient variance (see **Section 4.6** and **Figure 9**), providing strong practical validation of our theoretical insights."}, {"review_id": "NGa9OyGXu7", "reviewer": "Reviewer_S8XN", "summary": "This paper introduces ACCO (Accumulate while you COmmunicate), a novel distributed optimization algorithm designed to accelerate the training of Large Language Models (LLMs) by efficiently tackling communication bottlenecks. In standard data-parallel training (e.g., DDP with ZeRO), synchronizing gradients across GPUs often becomes the main performance bottleneck, leaving GPUs idle. Existing methods to hide this latency, like local-update algorithms, often incur significant memory overhead, preventing the use of memory-saving techniques like optimizer state sharding (e.g., ZeRO), or they suffer from convergence issues due to stale gradients. ACCO aims to unify the best of both worlds: memory efficiency from sharded optimizers and communication hiding from computation-communication overlap. Its core mechanism involves two parallel CUDA streams: Computation Stream: Continuously computes gradients on new mini-batches of data. Communication/Optimizer Stream: Concurrently handles the communication (gradient synchronization) and optimizer update steps from the previous set of gradients. To address the convergence degradation caused by using stale gradients (a common issue in methods like Delayed Parameter Update or DPU), ACCO introduces a novel two-stage update process. The authors provide theoretical convergence proofs for ACCO with SGD, showing it recovers standard convergence rates. Empirically, they demonstrate ACCO's effectiveness across various LLM pre-training and fine-tuning tasks. They show that ACCO significantly outperforms DDP+ZeRO in terms of wall-clock time, especially in multi-node and heterogeneous hardware settings, without compromising model convergence or final performance.", "questions": "1. The current ACCO design is tailored to compensate for a one-step delay, which is typical when overlapping communication and computation. How would the framework adapt to scenarios with larger or more variable delays, which can occur in less reliable network environments or with more complex parallelization strategies?\n2. The paper focuses on data parallelism (DDP). How does ACCO interact with model parallelism techniques like tensor parallelism and pipeline parallelism? In a fully 3D-parallel setup, the communication patterns are more complex. Would ACCO's two-stream model still be as effective?\n3. The second stage computes gradients on the full mini-batch. Is there any redundancy with the first stage's computation on half the mini-batch? Could the gradients from the first half be reused in the second stage to save computation, or does the change in parameters from θ to θ_tilde necessitate a full re-computation?\n4. The authors claim ACCO does not introduce new hyperparameters to tune, which is a major advantage. However, does the change in the update mechanism (using estimated parameters) affect the optimal settings for existing hyperparameters like learning rate or weight decay compared to a standard DDP setup?\n5. The paper mentions ACCO is not designed for failure-prone environments (e.g., worker failures). This is a fair limitation, but it's worth noting for practitioners considering it for very large-scale, long-running training jobs where fault tolerance is critical.", "limitations": "see questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper addresses a well-known and critical bottleneck in large-scale model training: communication overhead. The proposed solution is elegant, practical, and directly targets the GPU idle time caused by gradient synchronization.\n2. The two-stage \"estimate-and-update\" process is a clever and novel way to mitigate the convergence issues caused by stale gradients. Unlike prior work (DPU, WP), which either ignores staleness or uses simple heuristics, ACCO's approach is more principled and, as shown empirically, highly effective at matching the baseline DDP training curves.\n3. ACCO successfully combines memory-efficient sharded training (like ZeRO) with communication-computation overlap. This is a significant contribution, as previous methods typically forced a trade-off between the two (e.g., local SGD methods require full optimizer states, precluding sharding). Table 1 provides an excellent summary of this advantage.\n4. The experiments are thorough and well-designed. They cover pre-training and fine-tuning, different model scales (36M, 125M, 2.7B), and various hardware configurations. The heterogeneous hardware experiment (Section 4.6, Figure 9) is particularly compelling, as it clearly demonstrates ACCO's ability to handle stragglers by allowing faster workers to accumulate more gradients, a key advantage over synchronous methods like DDP.\n5. The paper includes both theoretical convergence analysis (for SGD) to provide formal guarantees and extensive empirical results with a more complex optimizer (AdamW) to demonstrate practical utility. The inclusion of system profiler outputs (Figure 13) and detailed pseudo-code further strengthens the paper's reproducibility and practical value.\n\nWeaknesses\n1. The two-stage update mechanism requires splitting the mini-batch and performing two forward/backward passes within the logical timeline of a single update. While this is the key to its success, it adds complexity to the training loop logic compared to standard DDP. The paper claims this is motivated by gradient accumulation, but the implementation detail could be a minor hurdle for adoption.\n2. The convergence proofs are provided for SGD, whereas the experiments and the primary use case for LLMs involve adaptive optimizers like AdamW. While the authors acknowledge this and the empirical results are strong, a theoretical analysis for adaptive optimizers would make the paper's claims even more robust.\n3. ACCO effectively uses different batch sizes for the \"estimate\" (half mini-batch) and \"update\" (full mini-batch) steps. This could have subtle interactions with learning rate scheduling and batch normalization (if used). While the experiments show it works well, a deeper discussion of these potential interactions would be beneficial.", "was_revised": true, "final_justification": "I still decide to maintain my original score.", "rebuttal": "We sincerely thank the reviewers for their thoughtful and constructive feedback, as well as for recognizing the significance of ACCO as a **clever and novel** work in addressing **well-known and critical** communication **bottlenecks** in large-scale model training.\n\n---\n\n**W1: Complex implementation**\n\nWe believe that the implementation does not add extensive complexity and can also be modularized in future software libraries. We propose a simple code on CIFAR10 to convince the reviewer of the simplicity to use ACCO where `net` and `net_` would be two CNNs that mirror \\\\(\\theta\\\\) and \\\\(\\tilde\\theta\\\\) and `optimizer_` and `optimizer` would apply ACCO updates:\n\n```python\nfor batch_idx, (inputs, targets) in enumerate(trainloader):\n    inputs, targets = inputs.to(device, non_blocking=True), targets.to(device, non_blocking=True)\n\n    if batch_idx % 2 == 1:\n        with torch.cuda.stream(stream1):\n            outputs = net(inputs)\n            loss = criterion(outputs, targets)\n            stream1.wait_event(event1)\n            loss.backward()\n        with torch.cuda.stream(stream2):\n            copy_model_grads(net_, net) # g_i is initialized with tilde g_i\n            event1.record()\n            optimizer_.step()\n    else:\n        with torch.cuda.stream(stream2):\n            zero_grad(net_) # set tilde g_i to 0\n            outputs = net_(inputs)\n            loss = criterion(outputs, targets)\n            loss.backward()\n        with torch.cuda.stream(stream1):\n            optimizer.step()\n\n    stream1.synchronize()\n    stream2.synchronize()\n```\n\nWe’ve run an experiment on CIFAR-10 using the DLA model, SGD and the cosine scheduler — the model reaches **95.3%** both with `ACCO+SGD` and vanilla `SGD`. However, we agree that a highly optimized implementation would be necessary to perform a PR in PyTorch, but this is beyond the case of this paper.\n\n---\n\n**W2: \"Proofs for adaptive optimizers?\"**\n\nWe agree that extending the theoretical analysis to include adaptive optimizers like AdamW would further strengthen the paper’s contributions, and we view this as an important and valuable direction for future work, particularly in the context of local SGD and large-scale distributed training. However, such proofs are much more involved and we view our SGD proof rather like a sanity check. See our answer to **Reviewer 2aEN**.\n\n---\n\n**W3: “Batch size might influence the learning rate or BatchNorm”**\n\nRegarding the learning rate, it's important to highlight that one of the primary goals of ACCO is to avoid unnecessary grid search or hyperparameter tuning. Therefore, we intentionally did not fine-tune this hyperparameter during our experiments. Furthermore, we believe the effective batch size is the one happening between two successive updates of \\\\(\\theta\\\\). We would be happy to add a clarification in the appendix explaining that the **equivalent batch size in ACCO corresponds to the number of samples processed between two successive updates of \\\\(\\theta\\\\)**, and that training dynamics are adjusted accordingly.\n\nAs for Batch Normalization, if it posed an issue, a potential solution would be to share the buffers and adapt the update rules to reflect the two-stage nature of ACCO. Therefore, we conducted additional experiments on CIFAR-10 using architectures such as DLA, which rely on BatchNorm, and observed no degradation in performance. For models like LLMs, this should not be an issue. Specifically, both `ACCO+SGD` and standard `SGD` achieved an **accuracy of 95.3%**. To support reproducibility, we had included a SLURM script in Appendix B.3 for reviewers who may wish to run ACCO or a baseline method of their choice.\n\n---\n\n**Q1: \"Beyond one-step delays?\"**\n\n ACCO is currently designed to handle a one-step delay, which aligns with typical scenarios involving overlap between communication and computation. However, extending ACCO to handle larger or more variable delays - as might occur in less stable network environments or more complex parallelization schemes - is an interesting direction for future work. Since ACCO is derived from an ADMM-style reformulation involving auxiliary variables \\\\(\\theta=\\tilde\\theta\\\\), one natural extension would be to introduce multiple auxiliary variables to model longer delays. We plan to explore this line of research in future work.\n\n---\n\n**Q2: \"3D parallelism?\"**\n\nACCO integrates naturally with model parallelism approaches such as tensor and pipeline parallelism. Since ACCO modifies only the optimizer step - not the forward or backward passes - it remains agnostic to how gradients are computed. Therefore, tensor and pipeline parallelism can be implemented as usual, and ACCO simply applies its update rule afterward. In a fully 3D-parallel setup, ACCO’s two-stream design remains applicable and effective, as it operates independently of the specific gradient computation strategy. See the code above.\n\n---\n\n**Q3: \"Redundancy?\"**\n\n There is no redundant computation between the first and second stages of ACCO. The gradients computed in the first stage (on half the mini-batch) are accumulated and reused. The second stage computes gradients on the remaining half using the updated parameters (\\\\(\\tilde\\theta\\\\)), and these are added to the previously accumulated gradients. Thus, the gradient computation is distributed across two steps, but no work is wasted. See the code above.\n\n---\n\n**Q4: \"Does ACCO change optimal hyper-parameters?\"**\n\n In all our experiments, we first tuned the hyperparameters for the standard DDP baseline to ensure optimal performance, and then used the exact same settings for ACCO. We observed strong results without needing to retune. While it's possible that ACCO could benefit from additional hyperparameter tuning and potentially outperform the baseline further, this was not the focus of our work. A key advantage of ACCO is that it introduces no new hyperparameters, simplifying adoption in practice. See **Table 4** in the Appendix for a list of the hyperparameters that we tuned only on DDP and line 46 from Section 1.\n\n---\n\n**Q5: \"Failure prone environment?\"**\n\n This is a valid and important point. ACCO does not introduce any new fault tolerance mechanisms beyond what is already provided by the underlying DDP infrastructure. It does not exacerbate failure modes, but it also does not specifically improve resilience to worker failures. We will clarify this limitation in the final version of the paper to guide practitioners who may consider ACCO for large-scale, long-running training jobs. Thank you for pointing this out."}, {"review_id": "sKpKZKzuH7", "reviewer": "Reviewer_Mv7t", "summary": "The paper presents ACCO, a method which enables improved communication/computation overlap during gradient accumulation by splitting the gradient computation and accumulation of a mini-batch into two stages, incurring a gradient delay of one step. A convergence analysis is provided, showing similar convergence to standard SGD methods, and experimental results show similar loss curves to standard methods while achieving improved performance.", "questions": "Please see the above strengths and weaknesses for details. I highlight here some key questions.\n\n1. Is large-scale training of LLMs actually communication-bound? Can you provide some evidence for this?\n2. How would ACCO handle larger models which require 3D parallelism?\n3. Why does Figure 4 report such poor performance for allreduces? If you tune to achieve better performance, does ACCO still have a runtime benefit?\n4. Related to (3), what speedup does ACCO yield when using a higher-bandwidth network?\n5. What is the runtime performance on the 8 H100 run?\n6. Is the difference in loss statistically significant?", "limitations": "I note question 2 of the checklist (\"Limitations\") refers to a discussion of limitations at the \"end of Section 4.6\"; such a discussion does not appear to be present.\n\nHowever, there are no major concerns about negative societal impact.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is targeting an important problem with potential for significant impact: improving the efficiency of large-scale LLM training.\n2. The approach in the paper of partially splitting the mini-batch is interesting and, to my knowledge, has not been used in prior work focused on stale gradients.\n3. The paper includes both a theoretical convergence analysis and experimental validation. The experimental validation includes both loss curves and runtime results.\n\nWeaknesses:\n1. The paper does not make a strong case for the practical need for reducing communication overhead in large-scale LLM training. In my experience, gradient communication (as addressed by ACCO) is _not_ a bottleneck in a well-tuned setup. Consider, e.g., Gemini Team, \"Gemini: A Family of Highly Capable Multimodal Models\" (https://arxiv.org/abs/2312.11805), section 3, where synchronous training achieves ~97% goodput. A case could be made for this approach benefiting systems with lower network bandwidth, but the paper does not push this, nor is such a setting plausible for training large models.\n2. The paper is not clear about how this method would scale to larger models. It seems predicated on having a significant amount of batch parallelism available, which is not common given training large models is memory capacity-constrained. The paper would benefit from a discussion of how ACCO generalizes to cases where 3D parallelism is employed (e.g., consider the limit of employing pipeline parallelism with a microbatch size of 1).\n3. Unclear presentation of ACCO algorithm: While I can get a high-level idea from the main paper, and Algorithm 1 in the supplementary paper provides more detail, simply reading the main paper leaves many things unclear. A few things that would help clarify the algorithm:\n    - Use consistent terminology for the streams. Figure 2 uses \"$\\nabla$ acc. stream\" and \"comm./opt. stream\", whereas the text in Section 3.2 refers to a \"gradient computation stream\", a \"communication stream\", and a \"computation stream\". Algorithm 1 refers to a \"computation CUDA stream\" and a \"communication CUDA stream\".\n    - There are some other confusing terminology differences, e.g., the use of $\\nabla$ in Figure 2 versus $g_i$ in the text, and the use of subscripts versus superscripts.\n    - The equations given are not clear about how the mini-batch is split. I assume $\\tilde{\\xi}$ is supposed to be the \"first half\" of the mini-batch? But the splitting is overall unclear.\n    - Figure 2 would benefit from showing a precise and complete computational graph for the algorithm.\n4. Figure 3 is misleading without an explicit note that the optimizer state is in a different precision than the other quantities, assuming height is meant to represent data storage volume.\n5. It is unclear whether baselines (particularly DDP) are well-tuned. It is unclear in Figure 4 whether the allreduce time presented is for a single allreduce on the entire gradient buffer (for a 7B parameter model and gradients in bf16, a 14 GB buffer) or with the usual approach of gradient bucketing with overlap, but the text implies the former. In this case, the network performance achieved seems to be very low: a bus bandwidth of about 3.35 GB/s with 24 workers ($(14 \\mathrm{GB}) / (8 \\mathrm{s}) \\cdot (2 \\cdot 23 / 24)$), which is about a quarter of the stated network peak (100 Gbit/s). A well-tuned implementation should achieve somewhere between 2-3 s for this allreduce, i.e., with good overlapping there should be little communication overhead. This poor baseline calls into question all the subsequent performance results.\n6. The inter-node network used is very slow, only 100 Gbit/s, which would be unusual for LLM training. I would expect a network with 8-16x more off-node bandwidth (e.g., 200 Gbit/s per GPU). Such a setting would be closer to the _intra_-node bandwidth evaluated here, where communication overhead is not a significant issue.\n7. The paper presents convergence results for 8 H100s on a single node. However, runtime results are not presented (as they are in Figure 7(b)). What is the performance benefit of ACCA in this setting?\n8. The paper does not make a convincing case for the statistical significance of its results, which is critical in settings like delayed gradients where increased variance in training can be introduced. While some curves seem to have some shading around them, it is not clear if this is just illustrating the range of data from a single run, or illustrating a deviation with respect to multiple random seeds.\n9. The paper's discussion of communication/computation overlap in Section 2 does not include a discussion on standard gradient bucketing / communication overlap in synchronous data-parallel training. This pervades the introduction as well. The paper would benefit from being more precise about exactly what sort of overlap it is aiming to achieve, and what regimes it differentiates itself from others.\n10. Section 4.4 states that ACCO achieves a 25% speedup in pretraining, referring to Figure 7, but this speedup is not apparent in the figure.\n11. Minor: The introduction states that training curves \"mirror\" AdamW— I would not use the term \"mirror\" since images are flipped in a mirror. Perhaps \"match\" would be a better term.", "was_revised": true, "final_justification": "The authors' rebuttal clarified several issues in the paper, and I think the paper will be better after incorporating those changes (e.g., consistent terminology, improved presentation, other issues). However, the paper and rebuttal do not convince me that ACCO offers a meaningful speedup to data-parallel communication compared to a well-tuned baseline, and it therefore does not seem to offer significant upside given the added complexity.\n\nThere may be benefit for ACCO or similar methods in very low-bandwidth settings (e.g., federated learning or training initiatives like Prime Intellect's), but the paper does not experimentally evaluate this.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and detailed feedback. We appreciate the time and effort taken to engage with our submission which **“targets an important problem”**. Below, we address each point raised, structured according to the review:\n\n---\n\n**W1. “Lack of practical motivation for reducing communication overhead”**\n\nWhile we agree that some clusters achieve high throughput (e.g., ~97% as reported in Gemini [arXiv:2312.11805]), this is not the case in all data centers, and indeed many datacenters in high-end academic, government, and other settings do not have the most expensive connectivity. Indeed, this work was performed on a relatively high-end and large-scale academic cluster.\n\nWe also note that with methods like ACCO becoming prevalent, even the largest-scale industry players can potentially save on some of the expensive infrastructure. ACCO is designed to be simple, optimizer-agnostic, and portable - requiring no infrastructure-specific tuning or assumptions. As the reviewer notes, our method would be especially beneficial for lower-bandwidth settings, and we appreciate the suggestion to better highlight this use case. We will revise the introduction and Section 2 to explicitly frame ACCO as a low-infrastructure-cost alternative suited for less-optimized clusters than Google.\n\n---\n\n**W2: “Scalability to larger models and 3D parallelism”**\n\nWe thank the reviewer for raising this important point. ACCO modifies only the optimizer step and operates independently of how gradients are computed. Therefore, it is compatible with tensor and pipeline parallelism, and does not interfere with 3D parallel setups. ACCO can be integrated post-gradient computation, making it orthogonal to parallelization strategies. In fact, it may be seen as two models alternating computation on shared threads, and thus naturally extends to larger scales. That said, optimized 3D parallel implementations would benefit from further engineering integration, which we consider an exciting future direction.\n\n---\n\n**W3: “Unclear presentation of the algorithm”**\n\nWe acknowledge this and will revise the paper for clarity. Specifically:  \n- We will consistently use *\"computation stream\"* and *\"communication stream\"* throughout the paper.  \n- We will replace \\\\(\\nabla_i\\\\) with \\\\(g_i\\\\) and \\\\(\\tilde\\nabla_i\\\\) with \\\\(\\tilde g_i\\\\) to unify notation.  \n- You're correct in your assumption. The beginning of Section 3.1 clearly defines each worker’s local batch as \\\\(N_i\\\\), \\\\(\\xi_k\\\\) as a sample and \\\\(\\nabla F_i\\\\) as the local gradient estimate on node i. \\\\(\\tilde g_i\\\\) are computed over separate halves (each of size \\\\(N_i\\\\)), yielding an effective batch of size \\\\(2N_i\\\\). We will clarify this language further.  \n- **Figure 2** is illustrative of how gradient computation overlaps with communication; it is not a full computational graph. ACCO uses standard PyTorch gradient flows - no custom computational graph is introduced.\n\n---\n\n**W4: “Figure 3 misleading due to optimizer precision”**\n\nFigure 3 is adapted from Figure 1 of (*ZeRO: Memory Optimizations Toward Training Trillion Parameter Models*, 2020, Rajbhandari et al.)\n. We will revise the caption to clearly note that the optimizer state is stored in higher precision, and ensure visual clarity by adapting the figure in the final version and referring to the Table 1 of our paper which clarifies this.\n\n---\n\n**W5: “DDP baselines may be poorly tuned; Figure 4 unclear”**\n\nWe appreciate this feedback. We tuned DDP hyperparameters first (see Table 4 of the Appendix for a list of the hyperparameters that we tuned only on DDP, and line 46 from Section 1), and then applied ACCO using the same settings.\n\nFigure 4 illustrates the case where communication is not fully optimized - this was intentional, to highlight potential overlap gains in typical academic setups. While we agree that well-optimized implementations can mask communication overhead, our contribution is algorithmic, not engineering-based. ACCO remains applicable in such contexts and is complementary to techniques like gradient bucketing. Our aim is not to outperform state-of-the-art communication kernels, but to provide a simple algorithmic handle for improving overlap where tuning is limited or ineffective.\n\n---\n\n**W6: “Slow network (100 Gbit/s) is not representative”**\n\nWe agree this is lower than some of the most high-end infrastructure. However, it reflects what is available in many settings - from startups, government institutions, academic settings, and even a broad range of cloud providers. Our results aim to show that even in these constrained environments, ACCO can provide significant speedup. We will better contextualize this in the revision.\n\n---\n\n**W7: “No runtime results for H100s”**\n\nDue to excellent intra-node bandwidth on H100s (which is available on our cluster), we do not expect ACCO to show a runtime improvement. Our experiments on H100 are intended to demonstrate convergence and functional correctness, not performance gains. We will clarify this in the experimental section.\n\n---\n\n**W8. “Statistical significance of results”**\n\nFor experiments in Section 4.3, where we expected variance, we averaged over 3 runs. We did not observe significant differences in training behavior between ACCO and DDP. While a broader statistical analysis is valuable, the cost of large-scale LLM training is prohibitive. That ACCO matches DDP so closely in loss curves across all settings provides strong evidence of its stability.\n\n---\n\n**W9. “Lack of comparison to standard gradient bucketing overlap”**\n\nGradient bucketing and ACCO are complementary: bucketed all-reduce can still benefit from being scheduled one step earlier via ACCO. We agree this relationship could be more clearly explained and will add the following clarifying sentencesin Section 2 to explicitly distinguish ACCO's goal from intra-step overlapping techniques: \"ACCO is complementary to standard gradient bucketing: while bucketing overlaps communication within a backward pass, ACCO overlaps across steps by shifting communication earlier. Both techniques can be combined for additional benefit.\"\n\n---\n\n**W10: “Claim of 25% speedup unclear in Figure 7\"**\n\nThank you for this observation. Indeed, the training accuracy reached by DDP at iteration 5000 is matched by ACCO around iteration 2800 (better than announced). We will clarify this in Section 4.4.\n\n---\n\n**W11: “Terminology: ‘mirror’ → ‘match’”**\n\nWe will update \"mirror\" to \"match\" for clarity.\n\n---\n\n**Q1: \"Is large-scale LLM training communication-bound?\"**\n\nIn the highest-end internal industry clusters, communication may not be the bottleneck until cross-data center training is needed. But in a wide range of existing clusters and settings that we are aware of, LLM training is indeed communication-bound: these include startups working on cloud providers, government research clusters, some of the highest-end national academic clusters, or even the growing number of actors trying to train LLMs over machines connected through the internet (*INTELLECT-1 Technical Report, 2024, Sami Jaghouar et al.*).\n\nThese environments — including ours — commonly suffer from costly inter-node all-reduce, especially as model size grows. **Figure 4** illustrates this overhead, and ACCO offers a simple way to mitigate it without infrastructure changes. We also emphasize that expensive communication infrastructure is built in the highest-end datacenters exactly to overcome these issues. Work towards communication-efficient learning can thus benefit the next generation of highest-end HPC clusters.\n\n---\n\n**Q2: \"How would ACCO handle 3D parallelism?\"** \n\nACCO applies after gradient computation and does not interfere with model partitioning. It is compatible with tensor, pipeline, and expert parallelism. A careful implementation would be required for full integration at scale, which we view as promising future work.\n\n---\n\n**Q3: \"Why does Figure 4 show poor all-reduce performance? Would ACCO still help after tuning?\"**  \n\nThe figure reflects an untuned or modestly tuned academic setup. With tuning, some overlap is achieved, but not fully — particularly for early layers. ACCO still helps by shifting communication one step earlier, thus improving overlap even in optimized scenarios.\n\n---\n\n**Q4: \"What speedup does ACCO yield on higher-bandwidth networks?\"**  \n\nIf communication cost is negligible and the optimizer step can be fully overlapped, then ACCO provides no timing benefit. However, ACCO still maintains correctness and convergence, and can be safely deployed without penalty.\n\n---\n\n**Q5: \"What is the runtime performance on the 8×H100 setup?\"**  \n\nWe observed similar runtime between ACCO and DDP due to the efficient intra-connect. Our goal with this experiment was to verify convergence, not performance.\n\n---\n\n**Q6: \"Is the loss difference statistically significant?\"**  \n\nNo. For training loss, curves are nearly identical. Minor variation in test accuracy falls within expected noise. We have not observed divergence or instability in any experiment.\n\n---\n\n**Minor Corrections and Clarifications**\n\n- The mention of a limitations discussion at the end of Section 4.6 was mistakenly omitted. We will add a proper limitations section in the final version.  \n- We will revise **Figure 3**, fix terminology inconsistencies, and improve the clarity of all **figures and equations** as requested and described above."}, {"review_id": "mCDZA7DYG0", "reviewer": "Reviewer_HYGz", "summary": "The authors propose a distributed optimization algorithm, ACCO, which uses an incompletely-accumulated gradient to approximate the full gradient of the current mini-batch in order to begin computing the gradient on the next mini-batch while the full optimization step is still taking place. This exploits the fact that the mini-batch sizes used for training large language models are typically large enough that the gradients from half of the mini-batch are sufficient for parameter updates that are similar enough to the parameter updates on the full batch. Furthermore, it allows for idle time to be eliminated even when some workers are slower than others by computing more gradients on workers which are faster. The ACCO algorithm is comprehensively compared with existing methods to demonstrate its advantages. The authors empirically validate the claim that training with ACCO can give nearly the same results as training with Distributed Data Parallel (DDP), an algorithm which does not rely on any approximations. They also prove that ACCO achieves the standard convergence rates for gradient descent and SGD that an exact algorithm would enjoy.", "questions": "How is the training data initially split between the workers? Is the data loaded as needed, or does each worker start with a full copy of the data they will use? In the case of heterogeneous hardware, how is the difference in the number of gradients computed by different workers accounted for in the allocation of data between them?\n\nSo far, ACCO has only been tested on LLM training tasks, so it has not yet been established that the technique will show the same performance when applied to other types of models, e.g. diffusion models. I would recommend adding an experiment where a different model type is trained to compare ACCO with DDP.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Quality:** The authors are careful to consider the many requirements for a distributed optimization algorithm to be practical, and show that ACCO succeeds in satisfying them. Their design choices are justified both empirically and theoretically. Empirically, they demonstrate that naively predicting future parameters using stale gradient information is insufficient for closing the performance gap between the DPU method and the exact DDP algorithm, while with ACCO the performance difference is negligible. Overall, I believe this is a high-quality contribution.\n\n**Clarity:** The only clarity issues I observed came from formatting issues (addressed in the formatting concerns section) and typos, and are relatively minor.  For the typos, on line 241 I believe that \"ZeRO\" should really be “ZeRO-1”. In the caption of Fig. 13 the word \"steams\" should be replaced with \"streams\", and in the second sentence the word \"take\" should be replaced with \"takes\".\n\n**Significance:** The authors are able to effectively balance multiple objectives (overlapping communication and computation, accounting for heterogeneous hardware, and requiring a minimal amount of memory) all without compromising on performance compared to DDP. This represents a major advancement over existing algorithms.\n\n**Originality:** This paper builds upon the idea of overlapping gradient computation with communication introduced in earlier works, but with the key novelty of using a subset of the mini-batch gradients to estimate the next set of parameters so that the computation of gradients for the next mini-batch can begin before the full optimization step and its communication overhead is completed.", "was_revised": true, "final_justification": "In my view, the authors have done a good job of responding to the critiques of their paper. Framing ACCO as a way to save on infrastructure costs serves as a suitable justification for the focus on reducing communication overhead. The authors have made a good argument for the simplicity of their method given that it does not introduce new hyperparameters that must be tuned. They have provided empirical justification for the validity of using ACCO with ADAM given that a theoretical justification is too complex to be achieved at the moment. I am not familiar with the DiLoco algorithm, but according to the authors it is not reasonable to make a direct comparison with it.  Overall, this leads me to maintain my original score of 5.", "rebuttal": "We thank the reviewer for their thoughtful and encouraging feedback. We're especially grateful for your recognition of ACCO’s strengths as a **practical**, **major advancement**, and **high-quality contribution**, and address your suggestions below.\n\n---\n\n**Minor formatting issues**\n\nThank you for pointing these out. We will correct all typos and formatting issues as requested. We will also add back the guidelines of the checklist.\n\n---\n\n**Minor formatting issues in the Appendix**\n\nWe will make all requested changes, including adding the missing reference text for Figure 13 and Algorithm 1, and relocating the figure to appear between Sections B.1 and B.2.\n\n---\n\n**Q1: “How are data loaded and handled in heterogeneous hardware?”**\n\nWe follow standard practice by partitioning the dataset evenly across workers. In heterogeneous environments, this data allocation could be adapted to reflect differences in compute capacity: thus, the dataset is loaded as needed. For heterogeneous hardware, it would make sense to indeed allocate the data accordingly to the worker speed. While we do not explore this in the current work, we agree it is an important consideration and appreciate the suggestion.\n\n---\n\n**Q2: “Experiments beyond LLMs?”**\n\nWe have tested ACCO on CIFAR-10 using the DLA model, SGD, and the cosine scheduler — the model reaches **95.3%** both with `ACCO+SGD` and vanilla `SGD`, without any hyperparameter tuning. We plan to release the CIFAR-10 code to support further experimentation. Thanks for the suggestion."}, {"review_id": "pgunKxfaHj", "reviewer": "Reviewer_E3rM", "summary": "This work focus on distributed training using a fast workers of smaller batch size to estimate the \"future weights\" while the slow worker finishes gradient computation of the rest of the batch. This creates an opportunity to overlap communication and computation leaving no GPU idles time. In effect, the model is updated by gradients from the \"estimated weights\" and the gradient of \"real weight\", which in empirical studies show no obvious degradation from baseline DDP.", "questions": "The paper shows theoretical results only for GD and SGD. I wonder how this half-delayed gradient would impact the Adam's convergence, especially the how it would impact the estimation of first and second order momentums?\n\nSince all experiments are done with Adam, without this the paper seems to be incomplete.", "limitations": "Limitation has been discussed and addressed", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n* Wall-clock measurement and solid experiments on LLM pretraining\n* The idea is intuitive and well-motivated\n\n# Weaknesses\n* **Only comparing with naive baseline DPU.** DiLoco despite of seemingly taking more memory on paper, the extra memory can be easily offloaded, since it's only used every few hundred steps, and the cost can be negligible. A more recent version on Streaming DiLoco [1] avoids that issue completely and hiding latency by overlapping completely.\n\n* **The description of algorithm is very confusing.** might be easier to just do with a chart or pesudo code blocks. Here is an example:\n   \"DPU repeat the following, where each line can be run in parallel ...\" There is a read from $\\tilde{\\theta}^{(t)}$ for gradient computation and then there is an assignment to $\\tilde{\\theta}^{(t+1)}$ in the second line, if they are using the same buffer, it would cause a race condition. Similarly, the same thing happens to $g$. This makes me question the baseline's correctness. \n\n* **Not enough ablation studies.**\nWhat happens if it has different mini-batch size? This method intuitively should be highly sensitive to variation in mini-batch size, due to using only half of the batch to estimate the \"future model weights\". What happens, as quite a common thing nowadays, the per node batch size could be very small for large model training, in the extreme case, the batch size in the fast stream would go down to 1, which then would introduce large variance.\n\n[1] Douillard, Arthur, et al. \"Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch.\" arXiv preprint arXiv:2501.18512 (2025).\n\n\n\nMinor issue:\nThe all-reduce expression is unnecessarily complex. Just write out all-reduce wouldn't have been clearer", "was_revised": true, "final_justification": "Overall, I believe this work is still borderline, due to following reasons: 1) the lack of comparison with state of the art methods (only comparing with methods that are almost half a decade old which lack large scale verifications despite of enough time has passed), not really providing assurance on update steps being close to the baseline (only showing semi-close training curve for specific cases, though showing a comparison on toy during rebuttal) and 3) writing and annotations are quite confusing enough for non-system background audience, which are most of our community in Neurips.\n\n I don't mind if it gets accepted, given some of our peers seem to really think highly of this work. But my opinion remains that has not clearly breach the threshold for acceptance by a large margin.", "rebuttal": "We thank the reviewer for their comments. We’re glad that the **motivation and intuition** behind ACCO came through clearly, and we appreciate your recognition of the **\"solid\"** empirical results and wall-clock improvements demonstrated in our LLM pretraining experiments. We address your points and suggestions in detail below.\n\n---\n\n**W1: “Only comparing to DPU”**\n\nWe would like to clarify that in Section 4.3, we also compare ACCO to the **Weight Prediction** method introduced in [8, from ACCO], so the claim of comparison only to DPU is not accurate.\n\nRegarding Streaming DILOCO: it serves a different purpose and is designed for settings with sufficient communication bandwidth to allow full rematerialization of optimizer states — as discussed in **lines 35–39 of the introduction**. In contrast, ACCO specifically targets bandwidth-limited environments where such rematerialization is not feasible.\n\nThat said, if rematerialization were allowed, ACCO could potentially be combined with DILOCO (which is a local SGD method). Exploring such a hybrid is an interesting direction, but it falls outside the scope of this work.\n\n---\n\n**W2: “The description of the algorithm is very confusing”**\n\nWe respectfully disagree, particularly regarding the DPU variant. As written in the algorithm section, gradients are computed using \\\\(\\theta^{t-1}\\\\), while parameter updates are applied to \\\\(\\theta^t\\\\). This is precisely what those equations convey.\n\nThe buffers alternate between steps, and the implementation is consistent with the written description. For further clarity, we have included the code with the submission and have also provided a **code snippet in our response to Reviewer S8XN**.\n\n---\n\n**W3: “What happens if local batch sizes are smaller?”**\n\nAs shown in **Equation (ACCO), line 164,** local gradients are aggregated via AllReduce across all workers, so updates occur based on the global batch, not per-worker mini-batches. Thus, even if a local batch is small (e.g., size 1), the overall update still reflects the aggregated contribution from all workers.\n\nWe address scenarios with imbalanced local batch sizes in Section 4.6. Our current model assumes that nodes process batches at a similar speed, leading to balanced partitions and stable training. While one worker computing less (due to heterogeneity or imbalance) is possible, we find ACCO remains stable under such conditions in practice.\n\nRegarding the comment that the method may be “highly sensitive to variation in mini-batch size,” we would appreciate clarification. If this refers to intra-node batch imbalance, AllReduce ensures global aggregation still proceeds correctly. If it refers to variability over time, our assumption of stationary GPU behavior ensures consistent gradient dynamics under fixed batch size.\n\n---\n\n**“Minor issue: The all-reduce expression is unnecessarily complex”**\n\nWe intentionally provide the full all-reduce expression because we are not simply averaging gradients \\\\(g_i\\\\), but rather computing \\\\(g_i + \\tilde{g}_i\\\\). This distinction is **essential to the algorithm**, and precision in notation is important — a point we believe aligns with the reviewer’s earlier concern in W2.\n\nSimplifying the expression would risk omitting key aspects of the computation.\n\n---\n\n**Q: “ADAM analysis?”**\n\nDeriving theoretical convergence results for Adam is more complex than for SGD, which is why we chose to focus on the latter to build intuition and provide a clear framework. While the analysis for Adam is beyond the scope of this work, we emphasize that `ACCO+Adam` empirically matches standard Adam training behavior -  as demonstrated in **Figures 5–9** - which strongly supports its effectiveness in practice. We appreciate the reviewer’s interest and view theoretical extensions to adaptive optimizers as valuable future work.\n\nConcerning the running buffers, it’ll probably add some variance for very small batch sizes (not common for LLMs training), but should lead to similar convergence for reasonable batch sizes."}]

Author rebuttal:

Meta-review: This paper presents ACCO, a distributed optimization algorithm that uses a two-stage mini-batch accumulation mechanism that compensates for update delays through parallel CUDA streams, achieving theoretical convergence guarantees for SGD and empirical performance matching AdamW + DDP baselines. The work demonstrates significant speedups of up to 87% over ZeRO-1/DDP, addressing an increasingly critical bottleneck in distributed training.

Reviewers consistently concur that the paper's primary strengths lies in its elegant two-stage gradient estimation approach that effectively mitigates convergence issues inherent in overlapped computation-communication schemes. The reviewers also note a key limitation of the mismatch between theoretical analysis (focused on SGD) and practical implementation (using AdamW), thus the raising questions about the extensibility of theoretical guarantees to realistic training scenarios. 

Other concerns regarding the two-stage mechanism introducing non-trivial complexity requiring dual forward/backward passes, and the paper lacks deeper discussion of interactions with critical hyperparameters like batch sizes and learning rate scheduling were addressed during rebuttal. 

Despite limitations, the novel approach addresses an important problem with solid theoretical foundation and demonstrates clear practical benefits. Since the identified limitations can be addressed without undermining the core contribution of this work, I recommend an acceptance for this work.
Decision: Accept (poster)