DUO: No Compromise to Accuracy Degradation
Authors: Jinda Jia
Cong Xie
Hanlin Lu
Fanjiang Ye
Hao Feng
Daoce Wang
Haibin Lin
Zhi Zhang
Xin Liu
Venue: NeurIPS 2025 poster ()
Keywords: Gradient Compression
Communication Computation Overlapping
Efficient Training
Distributed Training
Primary area: infrastructure
TL;DR: Achieving High Accuracy in Distributed Training Even with Aggressive Gradient Compression

Abstract: Distributed training often suffers from high communication overhead due to large-scale gradient synchronization. Although gradient compression—particularly at 4-bit or even lower precision—significantly reduces transfer volume, it typically results in sacrifice in precision and degradation of the final model accuracy. 

In this work, we introduce DUO, a distributed training framework designed to mitigate accuracy degradation incurred by gradient compression without involving additional overhead. DUO achieves this by inserting an additional high-precision gradient synchronization step into a previously computation-only phase, so that its communication is fully hidden by computation.

We provide a comprehensive theoretical proof of convergence for DUO and validate its effectiveness through extensive pre-training experiments on GPT models. Our results indicate that DUO effectively restores accuracy when using 4-bit gradient compression, achieving performance comparable to uncompressed training. Remarkably, DUO maintains minimal accuracy degradation even under extreme compression scenarios, including 1-bit gradients or complete omission of the low-precision gradient communication step (0-bit transmission).

DUO: No Compromise to Accuracy Degradation
Jinda Cong X ; Ye', Hao Feng', Hanlin Lu?, Daoce Wang', Haibin Lin", Zhi Zhang² and Xin Liu?
"Indiana University. "ByteDance Inc.
(jindjia, fanjye haofeng, daocwang) edu {cong.xix xie, hanlin.lu, haibin. lin, zhangzhi joshua, liuxin. ai abye
Abstract
Distributed training often suffers from high communication overhead due to large- scale gradient synchronization. Although gradient cunpresionn-particulatt at 4-bit even lower precision precision-signiffcinn results sacrifice in precision and degradation of the final model accuracy.
reduces transfer volume, it typically
In this work, we introduce DUO (Double Update Overlap), distributed train- ing framework designed mitigate accuracy degradation caused by gradient compression without introducing additional overhead. DUO achieves this by in- serting an additional high-precision gradient synchronization step into previously computation-onl phase, so that its communication is fully hidden by computation.
We provide comprehensive theoretical proof of convergence for DUO and validate effectiveness through extensive pre-training experiments on GPT models. Our results indicate that DUO effectively restores accuracy when using 4-bit gradi- ent compression, achieving performance comparable to uncompressed training. Remarkably, DUO maintains minimal accuracy degradation even under extreme compression scenarios, including 1-bit gradients or complete omission of the low-precision gradient communication step (0-bit transmission).
Introduction
In recent years, transformer-based language models 33.00 26 have demonstrated superior capabilities in learning and predicting sequence tokens based on contextual information. Thanks to the attention mechanism, traditional NLP tasks such as question answering. language translation and sentiment analysis have achieved remarkable improvements. Moreover, leveraging the attention mechanism inherent to transformers, researchers have extended their applications to multimodal domains, including image interpretation and video generation. Empirical evidence indicates that larger transformer bbasedd models can provide more powerful generative capability and consequently improve output accuracy.
As model sizes continue grow, distributed training has become essential for scaling deep learning, enabling the use of many GPUs to handle large datasets and complex architectures. However, it also introduces frequent GPU-10-GPU communication for synchronizing gradients and model parameters. These communications must complete before the next training step can begin, creating dependencies that stall GPU computation, thereby reducing overall throughput. As result, the efficiency gains from multi-GPU parallelism often undermined by communication overhead.
Recent work has also explored optimizer-level strategies to improve stability and reduce communica- tion 5 7 however, these approaches are orthogonal to ours, they modify the optimizer and require retraining or retuning, while DUO preserves the existing training configuration and can be plugged in without modification.
Equual Contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Compression is fundamental technique for reducing data size and has been widely applied distributed training to alleviate both communication and memory overhead. Existing approaches include lossless methods (e-g.. 2
3.2-
3.0
2.1
Validatie
2.8
78000
'75000
80000
2.6
11 34) that preserve exact information, as well as lossy methods (e.g., [24] 27 29
32 33 ) which trade precision for efficiency. Among them, quantization has become the most widely used strategy in large-scale LLM training due to its low latency, compatibility with existing pipelines, and ease of integration. Recent work has successfully applied guantization to reduce communication overhead, such as ZeRO++ [28]. SPD4Bit [14], QSDP [6 and FP8 training 18]
2.2
DUO-03
SDP4Bit- G1-
20000
40000
60000
80000
Deeration
Figure Training validation loss for GPT-350M
However, despite its efficiency, aggressive quantization often harms model accuracy-particulaar during pre-training. Prior empirical studies [18 16 14, (8) show that gradients are highly sensitive to precision loss, making gradient compression especially challenging. While methods such as error feedback 15
13 22 aim to mitigate quantization error and recover lost accuracy, they perform poorly when applied to large language models (LLMs) pre-training under aggressive compression (e.g., 1-bit precision). As shown in Figure the variant SDP4Bit GI-EF -EF--E------- which uses 1-bit gradient compression with error feedback enabled exhibits a significant accuracy gap relative to the baseline and fails to converge. In contrast, DUO achieves accuracy comparable to the baseline.
These observations reveal a fundamental trade-off between communication efficiency and model accuracy, which raises an important question: can we compress gradients without sacrificing accuracy or throughput? In this work, we answer this question affirmatively by proposing DUO, high-accuracy caaaumunniaaton-ffffieeee. training framework.
Our contributions follows:
We introduce the Fast-Slow Reduction algorithm, designed to mitigate accuracy degradation due to gradient compression.
training framework that integrates the Fast-Slow algorithm with system-level design achieve high accuracy with negligible overhead.
We propose DUO, a communication-effiiiiee
We establish a theoretical guarantee for the convergence of SGD with Fast-Slow algorithm, demon- strating that it maintains the same convergence rate standard SGD.
We integrate DUO into Megatron-LM and conduct extensive experiments to evaluate both its accuracy and efficiency. Results show that DUO achieves accuracy comparable to the uncompressed baseline, even under extreme communication compression settings (1-bit and 0-bit).
2 Background and Motivation
Megatron-LM is widely used framework for training large language models (LLMs). It employs three key strategies Sharded Data Parallelism [36], gradient accumulation, and communication overlapping all of which are closely related to the design of our proposed DUO framework. To provide context, we briefly introduce these strategies in the following sections.
2.1 Sharded Data Parallelism
Training LLMs such as GPT D and LLaMA 25 necessitates distributed training strategies to handle their extensive parameter sizes. Unlike naive Data Parallelism, Sharded Data Parallelism (ShardedDP) partitions high-precision optimizer states across all GPUs, which reduces the memory usage of the optimizer states by factor of where represents the number of GPUs. As result, ShardedDP changes the Naive Data Parallel communication pattern from AllReduce to a ReduceScatter plus AllGather, summarized in the following equations:
W AllGather (w[1)....,
9p V0, Dp)
(2)
d ReduceScatterrgn.... gn)
(3)
w [p] OptimizerStep (wpp) g(p))
At the beginning of each training iteration, each worker p holds its respective partition of the parameters w[p] and the corresponding optimizer states. The training iteration proceeds as follows:

---

① An AllGather 8 operation synchronizes the model parameters across all workers. After this synchronization, each worker holds full replica the model parameters w.
Each worker computes gradients Op based on its subset of data Dy.
Gradients synchronized using ReduceScatter [9] operation. This results in each worker receiving a partition g[p] of the averaged gradients. In this paper, our main contribution is to optimize this ReduceScatter operation for speed without sacrificing accuracy.
Each worker performs an optimizer step update its local parameter partition w(p] using the received gradient partition T(p], as well as the corresponding optimizer states.
Table I: Notations of Sharded Data Parallelism. index of worker or the corresponding shard/partition
t
index of iteration
parameter weights
Wt
weights in iteration t
w(p)
pth shard of weights
gradients
9t.p
gradient produced by worker p iteration f
g
averaged gradients
compressed gradients
glp]
pth shard of averaged gradients
2.2 Gradient Accumulation
Gradient accumulation is widely used for training LLMs with large batch sizes. Pre-training of LLMS typically requires massive batch sizes to improve training stability and efficiency. However, computing the full batch at once leads excessive GPU memory consumption.
To address this, gradient accumulation divides large batch into multiple micro-batches. Each micro-batch is processed sequentially, with gradients locally accumulated in memory. Gradient synchronization and weight updates occur only after all micro-batches within a batch have been synchronization.
processed, reducing the frequency of communication-intens.
gradient accumulation reduces communication frequency but introduces communication-idle gaps between synchronization steps. In this paper, DUO exploits these idle periods to overlap high-precision gradients communication that improve parameter and optimizer state accuracy.
As shown in Figure 2
2.3 Communication Overhead Optimization Strategy
As illustrated in the first diagram of Figure 2 and discussed in Section 2.1 Naive Sharded Data Parallelism requires two communication operations per iteration: an AllGather of weights and ReduceScatter of gradients. During these communication phases, the GPU must wait for the communication to complete, remaining mostly idle and leading to significant computational waste during training.
Overlapping and compression 33
31 4 are two commonly used methods for reducing communication overhead. We introduce these strategies in progressive manner.
20
Overlapping. To alleviate communication overhead, previous works 19 36 4 22 have explored overlapping strategies to better utilize computation heavy phases.
Computation-Commnnittt
The key idea that gradient and weight communications need not be performed all at once; instead, they can be partitioned into multiple steps and overlapped with nearby computations.
As shown in second diagram of Figure overlapping strategies divide computation into buckets. During the forward pass, the system performs computation on the current bucket while concurrently gathering weights for the next one. Similarly, during the backward pass, gradient computation for the current bucket overlaps with the gradient reduction of previous buckets.
However, modern GPUs offer significantly higher compute throughput than network bandwidth, making communication the dominant bottleneck. Moreover, communication can only overlap with the computation of the final micro-batch, leaving the remaining (n 1)/n fraction of computation time underutilized. As a result, even with overlapping techniques, a substantial portion communication remains non-overlapped.
Communication Compression. To further reduce the communication overhead, compression is often applied as an additional step in communication--fficies training. Before transmission,

---

Naive
fn
Overlapping
Overlapping & Compression
Gradient Accumulation
mb
mb
DUO
mb
Weights Gather
Gradients Reduction
DUO's Reduction
Fwd-Bwd Computation
Optimizer Step
Optimizer Rollback
Figure 2: An overview of distributed training with step-by-step integration of communication reduction strategies.
data is first compressed into smaller size, significantly reducing communication volume. Prior have demonstrated that quantization is an effective method for L.M training, where high-precision weights/gradients are scaled and cast into lower-precision formats. However, achieving high compression ratio without degrading model accuracy remains a major challenge.
works [6 28 14
2.4 Motivation
By leveraging grouped quantization and weight-difference communication pattern, communication compression method such as SDP4Bit [14] successfully reduces communication to 4-bit precision and is regarded state-of-the-art communication-effieentt training framework for Sharded Data Parallelism. However, it still suffers from accuracy degradation when gradients are compressed low precision (e.g., 4-bit), despite attempts such as Hadamard transforms mitigate quantization error. We attribute this performance drop to the high sensitivity of gradients to quantization errors.
Motivated by this limitation, propose DUO designed to preserve both high accuracy and throughput. DUO achieves accuracy comparable to full-precision baselines, transmission. Additionally, it supports a broader class of compressors such as sparsification, low-rank projection, sign-based compression offering greater flexibility for future gradient compression techniques. A formal definition of supported compressors is provided in Definition
communication-efiice
training framework
under ultra-low-bie bit (I-bit) gradient compression
O-bit gradient
The DUO Framework
DUO is system algorithm co-design framework composed of three components: the base Flst algorithm (Section 3.I communication overlapping strategy (Section(3.2), and memory optimization strategy (Section 3.3) Together, these components enable DUO achieve both high training efficiency and high accuracy. We introduce these three components in the following sections.
3.1 Fast-Slow Reduction Algorithm
To improve training accuracy, we leverage the high-precisiee gradient information that typically dis- carded in naive gradient-compression strategies. As highlighted in the red part of Algorithm| [ DUO caches the full-precision gradient g" before gradient compression (Line 10). This gradient replica will be used for performing high-precision ReduceScatter (Line 5), producing non-compressed averaged "slow" gradient that compensates for compression error later stage (Line 13-15), as
explained below.

---

Algorithm Distributed training with DUO
Algorithm 2 SGD + Fast-Slow Update
Require:
parameter weight (main copy): w, weight
1:
Initialize main parameter weights
for forward-backward: wi, weight for commu-
wo t0, 9t-2 = 0 for ==
nication: w'. weight difference: d, gradient: y.
2:
for all iteration t E [7] do
worker: p. pth shard of weight gradient: [p],
3: Update main weights with full-precision gra-
gradient produced by worker pi g". gradients
dient from previous iteration (slow update):
with compression 9. averaged gradient: 9. cur-
rent iteration: t
Wy <- W 79h-2
1: function ForwardPass
4:
Compute gradient: -t V Vi Se
2: de(p) [p] wi[p] 1it-1
5: Compress gradient: Øt-1 Cg (9t
3: [plp) <- QuantizeWeighsDiiffaa
6:
Update temporary weights (fast update):
w/ < w/ -19-1
4: de < AllGather du (p])
5: Start AsyncReduceSSaater([__]
7: Compress weight difference:
= = ū
6: the 4- 1-11111 d,
7:
output" +- ForwardPass( (wx, impul")
Update weights for
8:
function wackwardPass
Wi 6 W 1- + A.
9:
gi < Gradient( Wie, oulpull))
9: end for
10:
SaveReplica(g))
11:
O? ComprexGGradient(g""
G-(p) < ReduceScatterr(ã")
gt-1 [p] 4- Wait AsyncRedueeeSattter Finish
14:
Recover wp-1pp]
15:
wx[p] Optimizer(@: wp wp-1(p)
16:
w Optimizer(0: p)., wp(p))
To correct the compression error in the "fast" gradients, DUO introduces an optimizer rollback and reset mechanism (Line 14). At each iteration, DUO rolls back the optimizer restore the main parameters and optimizer states to the state where the compressed gradients are not applied. After that, an additional optimizer step is conducted with the non-compressed averaged gradients (Line 15). By doing so, the optimizer update calculated based on compressed gradients in the last iteration is dropped from the main parameters and the optimizer states, and replaced by the optimizer update based on the non-compressed gradients, resulting in more accurate parameters and states.
The remaining steps of Algorithm (non-red part) follow those of SDP4Bit. The difference between the latest main parameters and the previous model parameters of the local shards on each worker are calculated and compressed (Line 2-3), which is then gathered (Line 4) and used for updating the unsharded model parameters (Line 6) for forward-backwart computation (Line 7 and 9). The "fast" gradients are compressed, synchronized, and used for producing a temporary version of the main parameters on the local shard (Line 11, 12, and 16).
3.2 Overlapping ReduceScatter with Computation
Although Fast-Slow Reduction improves model training accuracy by incorporating high-precision gradients, also introduces an additional ReduceScatter communication step. Fortunately, this ReduceScatter operation can be executed asynchronously alongside the computation.
As illustrated in Algorithm 1 the "slow" gradients of cached after they are generated during backpropagation. These full-precision gradients are then averaged via ReduceScatter and used in an additional optimizer step in the next iteration (Line 15). Because the averaged "slow" gradients are not immediately required, the ReduceScatter can be executed asynchronously and overlapped with the computation during the interval between SaveReplica (Line 10 of iteration t) and optimizer step (Line 15 of iteration t + 1).
Because the original AllGather and ReduceScatter operations must complete before the next iteration begins, they take precedence scheduling. To avoid interference, DUO defers the execution of the high-precision ReduceScaller until after the AllGather completes (Line 5).
As discussed in Section existing overlapping strategies only utilize the computation of the final micro-batch, leaving the rest of the computation time non-overlapped. Thanks to the Fast-
5

---

Slow algorithm, the involved ReduceScatter can overlap with the computation of all micro-batches, providing significantly greater opportunities for communication overlap.
3.3 Memory Footprint Optimization
Implementing the Fast-Slow Reduction algorithm requires a high-precision gradient replica for reduction. This replica is continuously used during the reduction phase. Moreover, during high- precision gradient reduction, the GPU continues performing computations. To integrate Fast-Slow Reduction efficiently without interfering with computation or memory usage, DUO employs gradient-offloane strategy. Gradients are first copied from device host memory, where the high-precision reduction is later performed. The averaged gradients are then copied back to the device before executing the optimizer step.
Since CPU memory typically has low bandwidth, naive device-to-host (D2H) copy before gradi- ent compression would be inefficient, leading to long computation idle times. To achieve near- zero-overhead offloading, DUO employs three complementary techniques:
1) D2H Offlead Bucket
Overlapping
Bucket-Wise Offloading, As illustrated in Sec- tion modern LLM training commonly em- plays an overlapping strategy to reduce communi- cation overhead. The core idea partition com- putation tasks into buckets, where each bucket contains set of parameters. Gradient reduction begins as soon as the computation for bucket is completed. Building on this overlapping strategy, we align our gradient offloading granularity with computation task, introducing Bucket-We Wise Offlodding. In this approach, the gradients for each bucket are offloaded immediately after the corresponding computation finishes, ensuring an carly start D2H transmission.
3) GPU Memory Bridge
Compressed Reduction
Copy
Copy
Computation
Figure Step-by-step optimization strategy for device to host copy of gradients. First is a naive D2H copy after computation, after adapting with finer reduction and GPU bridge, the D2H can be fully overlapped.
To accommodate the bucket-wise offloading strategy, we also reorganized the optimizer step to operate the bucket level. Additionally, we reuse the global gradient norm obtained from the low-precision optimizer step in the previous iteration to ensure consistency of gradient clipping between the two optimizer steps.
Finer Overlapping with Compressed Gradient Reduction. A compressed gradient reduction (fast reduction) is performed to execute the first-stage optimizer step. As illustrated in the blue region of the second diagram in Figure 3 a typical gradient reduction consists of four phases: (1) compression, (2) point-to-por point (P2P) communication, (3) decompression, and (4) local reduction. We observe that only the final phase overwrites the previous gradient values. Therefore, the device-to-host (D2H) transfer can be overlapped with the subsequent compressed gradient reduction a finer granularity. Therefore, the device-to- -eee-ttooosss host (D2H) transfer can be overlapped with the first three phases of the next compressed gradient reduction compression, P2P communication, and decompression therrr reducing transfer overhead. A synchronization barrier is inserted before the local reduction phase to ensure that the D2H transfer has completed.
GPU Memory Bridge. Even though we successfully exploit a larger overlapping space to reduce overhead, we observe that the D2H transfer remains too slow be fully overlapped without blocking the reduction operation. As a result, this bottleneck affects overall training speed.
To address this issue, DUO introduces GPU Memory Bridge, which temporarily caches gradients in a small GPU memory buffer before transferring them to the host memory on the CPU. This strategy acts intermediate buffer, effectively serving bridge between the GPU and CPU to mitigate its impact on the subsequent reduction.
As illustrated in Figure GPU Memory Bridge extends the available overlapping space with the next bucket's computation, further improving training efficiency. Notably, we set the memory bridge buffer size equal to the bucket size and restrict its usage single buffer any given time to achieve

---

minimal additional footprint increase. As shown in Table because the bucket sizes are small, this memory increase is negligible.
Theoretical Analysis
In this section, we establish theoretical guarantees of convergence for DUO. To simplify the results, we prove the convergence of Algorithm 2 which is equivalent to Algorithm [ [with stochastic
gradient descent (SGD) as the optimizer, and hiding the details of sharded DP.
4.1 Recursive Error Correction
To make it easier to understand the intuition of the Fast-Slow Reduction algorithm, we present the following equations to show how the compression errors from the weight difference compression and gradient compression are recursively corrected in Algorithm 2
Theorem 4.1 (Recursive Error Correction). The uncompressed weight difference can be recursively
expanded as: A, = w// - = - (199-2 2 - -19t-2) 2) + (At-1 - -,-1), where (gt-2 - 9t-2) corrects the error caused by gradient compression, and (A.-1) İ.-1) corrects the error caused by weight difference compression.
Theorem 4.1
provides an important intuition for the convergence proof of Algorithm It is indicated that the eтror of fast-slow update the parameter weights is composed of two parts: the recursively corrected error gradient compression, and the recursively corrected error of the weight difference. With the help of this recursive eπrоr correction, back to the first step, and establish the corresponding upper bound.
unroll the compression error from the last step
4.2 Convergence Analysis
In our algorithm, use SGD with gradient and parameter compression solve the following optimization problem: f min fww,, where f(w) is the objective function, w E IRd is the model parameter.
Note that we use arbitrary (potentially biased) compressors for both gradient reduction and parameter weight collection, while SDP4Bit [14] only supports unbiased compressors for the gradients. We formally define the compressors as follows.
Definition 4.2. [é-approximate compressor [5]] An operator C Rd Rd is 6-approximate compressor for E (0, 1] if [[C(v) - o/|2 < (1-6)||u Vo E Rd.
We establish analysis under the following assumptions.
Assumption 4.3. For distributed training with n workers, we define the compressed gradient as Ør - (g.) = 1.2 Eppe(t) (gt,),,, where St 1, Eppen) Pt.pr and St.p the stochastic gradient from the pth worker in iteration. We assume that C. is oy-approximate compressor of the average gradient and C.,, is a approxxxiee compressor for weight difference compression.
Assumption 4.4. (Smoothness) We assume that f(x) is L-smooth: [v) V s L|| vll, Vx,y E Rd which implies f(y - f(x) S (Vf(x). y - x) + flly 2||2.
Assumption 4.5. For any stochastic gradient V (). where is an independent random sam-
plc, we assume unbiasedness B fw:; [)] = v f(w), and bounded variance E (w:
V ff(w)] w| pl ( a² (23). Assumption 3).
We derive the following error bounds on the convergence of Algorithm as follows. All proofs can be found in Appendix A
Theorem 4.6 (Convergence error bound). For arbitrary non-convex function under Assump-
tion 4.3
4.4 and 4.5 with the by apprrxxxie compressor Cg for gradient compression and
the ow approximmate compressor Cw for weight difference compression, taking learning rate S
96aL. 96al 114 Algorithm converges to a critical point with the following error bound:
[[V/[(m)]]]] (f(wo) - fr) V6144aI3 + 1)2
T
Toom
384Lo² (4agL + 8.)) (f(wo) - f³)
VT6
7

---

where or 10(1 18
Theorem 4.6 shows that our proposed algorithm has the same convergence rate as ordinary SGD for general non-convex functions.
Experiments
Our experiments are organized into two parts: accuracy and performance. Together, these evaluations show that DUO improves training accuracy while maintaining nearly identical training speed in Sharded Data Parallelism with gradient compression- and can even increase throughput by enabling more aggressive compression.
5.1 Environment Setup
Model and Dataset. To validate the accuracy improvements of the proposed DUO method, conduct pre-training experiments on GPT models with sizes ranging from 125M to 6.7B. The hyperparameter settings follow those of the OPT model [35] ensuring that models of the same size use identical configurations. For reproducibility, we provide the detailed configurations in Table (Appendix B).
We use The Pile 12] as our training dataset due to its open-source availability and general applicability, for 80 000 iterations, processing 80 billion tokens in total. Accuracy is evaluated via validation-loss comparison across all settings.
All pre-training tasks
To effectively evaluate the proposed method, we integrate DUO into Megatron- LLM, one of the most commonly used open-source L.MM training frameworks.
Baseline and notation. We use SDP4Bit-the state-of-the-ar low-bit compression strategy combined with Sharded Data Parallelism (ShardedDP) for LLM pre-training-ar our baseline. SDP4Bit applies 4-bit randomized symmetric linear quantization to both weight differences and gradients. In our experiments, we retain the same 4-bit weight compression mechanism in SDP4Bit but vary the gradient quantization across different bit widths. DUO uses 4-bit gradient compression by default. Suffixes such as denote gradient bit-widths (e.g., DUO-G1 for 1-bit gradients).
Hardware environment. The experiments are conducted on three environments:
4-node setup with A A100 GPUs:
Each node is equipped with 4 NVIDIA A A100SXXM440GGB GPUs connected via NVLink. Nodes are connected by 100 Ghps Slingshot links.
8-node setup with H20 GPUs: Each node is equipped with 8 NVIDIA H20 GPUs connected via NVLink. Nodes are connected by 400 Gbps InfiniBand links.
4-node setup with single A100 GPU: Each node has single NVIDIA A100-SXM4-40GB GPU. Nodes connected by 100 Gbps Ethernet links.
Table 2: Final validation loss J for various compression settings.
Grad Bits
Strategy
125M
350M
1.3B
6.7B
32
Full Precision
2.2716
2.0582
1.8854
1.7527
SDP4Bit
2.2757
2,0629
1.8944
1.7570
4
DUO
2.2727
2.0592
1.8907
1.7535
SDP4Bit
2.4204
2.1843
2.0489
1.8434
1
DUO
2.2712
2.0686
1.8943
1.7572
DUO
2.2761
2.0628
1.8941
1.7550
5.2
E2E Accuracy Evaluation
We evaluate the accuracy of DUO with pre-training tasks. To demonstrate that DUO can achieve high end-to-end training accuracy, train GPT models of various sizes from scratch.
As shown Table 2 we assess DUO's performance under different gradient compression ratios.
The results indicate noticeable accuracy gap between SDP4Bit and full full-precision training. This

---

gap becomes more significant as using a more aggressive way for gradient compression or model being larger. In contrast, DUO preserves accuracy even for larger models and ultra low gradient communication bits, maintaining final loss comparable to the full-precision training.
To further assess DUO under high-compression scenarios, we extend the SDP4Bit setting by reducing the gradient bit-width to I or even 0. Note that "O-bit" does not mean that DUO completely removes gradient communication. Instead, the low-precision gradient communication of the "fast" update is removed, but DUO still performs high-precision (slow) ReduceScatter.
As shown in Table 2 these extreme compressions results in substantial accuracy drop for SDP4Bit compared to full-precine frec-pin training. In contrast, DUO maintains high end-to-end accuracy, demon- strating robustness under severe communication constraints.
5.3 E2E Performance Test
We evaluate DUO in practical training tasks, focusing its end-to-end (R2E) speed and memory consumption. We use TELOPS as the primary metric to measure E2E speed.
High-bandwidth environment. For fair comparison, both DUO and SDP4Bit 4-bit gradient quantization; detailed configuration are Table (Appendix C). As shown in Table 3 despite intro- ducing additional high-precision reduction step, DUO achieves training throughput comparable to SDP4Bit. We attribute this to DUO's carefully designed overlapping strategy, which effectively hides communication latency behind computation.
DUO's s memory footprint closely matches that of SDP4Bit.. This demonstrating the effectiveness of DUO's high-precision gradient-offloading strategy. The slight increase in memory usage is due to the GPU memory bridge used mitigate device-to-host transfer overhead. Since the bridge buffer size matches the bucket size and is manually configurable, this overhead does not negatively impact overall performance.
Table E2E performance comparison between DUO and SDP4Bit on two hardware environments: 8x8 H20 nodes and 8x4 A 100 nodes. Throughput (TFLOPS + std) and GPU memory peak (MB).
H20 (8x8 H20 Nodes)
A100 (8x4 A100 Nodes)
Model
Method
Throughput (TFLOPS)
Mem (MB)
Throughput (TFLOPS)
Mem (MB)
DUO SDP4Bit
70.82 = 0.63
15880
117.53 + 0.49
21270
1.3B
71.02 0.63
15832
117.61 2.35
20934
DUO
78.71 +: 0.92
25600
125.31 +. 0.37
35298
2.7B
SDP4Bit
79.15 0.80
25640
126.42 0.82
34422
DUO
84.09 2.04
16162
122.67 + 3.76
26480
6.7B
SDP4Bit
85.61 + 1.57
15692
121.05 1.89
25684
DUO
93.02 1.08
27596
132.88 + 0.95
27332
13B
SDP4Bit
94.68 + 0.77
27220
136.73 + 1.35
27142
DUO
100.85 1.29
36540
116.37 1.28
27378
18B
SDP4Bit
102.93 = 1.04
35478
122.91 + 1.16
27088
bandwahdwidth environment. We also evaluate DUO in low-bandwidth cloud environment under various network settings (10 Gbps and Gbps) and gradient compression levels (from 4 bits 0),
As shown in Table 4 DUO consistently achieves end-to-end throughput comparable to SDP4Bit and outperforms the uncompressed baseline. Moreover, DUO-G1 and DUO-GO achieve higher throughput than SDP4Bit, thanks to reduced gradient communication overhead.
Table 4: End-to-end throughput (TFLOPS) of different methods under constrained network band- widths using the GPT-350M model with a gradient accumulation step of 32.
Bandwidth
DUO-G4
DUO-G1
DUO-GO
SDP4Bit
Bascline
10 Gbps
130.81 + 3.56
132.67 2.94
133. 4.29
130.52 + 2.50
108.89 = 2.25
5 Gbps
129.35 2.96
131.25 3.16
131.97 3.06
129.84 + 2.97
98.26 1.32

---

5.4 Ablation Study
Overlapping capability of DUO's reduction. DUO strategically overlaps it's newly involved communication with subsequent computations. To hide communication overhead, each iteration's computation time must exceed DUO's high-precision gradient communication time.
The computation- computatioo--o-commmmiici to-communication ratio depends on network bandwidth, mandient-accumulann steps and sequence length. As detailed in Section 2.2 and further elaborated in Section 3.2, increasing gradient accumulation steps significantly enhances computation time per iteration:
ComputationTimeePrlter AccumulationSteps x MicrobatchCommptationniii
(5)
Longer sequence lengths likewise increase computation intensity. Modern LLMs typically use long sequences lengths (e.g., LLaMA 3: 128K, Qwen2: 32K), which substantially increase computation intensity.
We evaluate various combinations of bandwidth, sequence length, and accumulation steps to further demonstrate DUO's overlap effectiveness. As shown Table both larger accumulation steps and longer sequence lengths contribute to a higher computation- cottiioo-t--commmiiaatitit ratio, resulting in better overlap even under low bandwidth conditions.
Table 5: Computation Communication Overlapping Performance. Breakdown results across bandwidths (10/5/2 Gbps), sequence lengths (2048/4096), and accumulation steps (8/16/32).
Bandwidth
Soq.
Ace.
Step
Method
Throughput (TFLOPs)
Commn Iter (mss)
Comp (ms)
Overlap
32
DUO
122.6
2135
6042
32
SDP4Bit
123.0
6042
4096
16
16
SDP4Bit
DUO
122.6
123.1
3030
3030
DUO
86.7
2019
1533
SDP4Bit
117.5
32
DUO
83.5
2215
3024
/
32
SDP4Bit
84.4
3024
2048
16
16
SDP4Bit
DUO
71.2
78.8
2125
1545
DUO
39.0
2136
771
SDX4Bit
32
DUO
128.9
9033
6441
4096
16
16
SDP4Bit
DUO
119.6
123.4
3137
30400
3040
DUO
59.5
3287
1564
SDP4Bit
112.7
32
DUO
80.54
2964
3012
32
SDP4Bit
83.4
3012
2048
16
16
SDP4Bit
DUO
77.3
50.2
1520
1520
DUO
22.7
3378
791
SDX4Bit
791
32
DUO
101.6
7973
6038
16
DUO
49.8
8198
3045
1096
16
SDP4Bit
105.0
3045
DUO
24.7
8273
1558
SDP4Bit
32
DUO
41.1
7967
3048
32
SDP4Bit
72.7
3048
2048
16
16
SDP4Bit
DUO
20.1
59.9
8179
1566
1566
DUO
10.0
8126
839
SDP4Bit
43.8
Conclusion
In this paper, we introduce DUO, a high- sigh-precision, communication-efficin training framework. DUO incorporates Fast-Slow, a gradient compression error reduction algorithm, along with a carefully designed optimization strategy. By integrating these components, DUO improves training accuracy to level comparable to uncompressed training, even at extreme compression ratio of 0 or bit.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Noushin Azami, Alex Fallin, and Martin Burtscher. Efficient lossless compression of scientific floating-poin point data cpus and gpus. ASPLOS '25, page 395-409 409, New York, NY, USA, 2025. Association for Computing Machinery.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Overlap
Chao Yang. Centauri: Enabling Efficient Scheduling for CoaamunicationCCmputaae in Large Model Training via Communication Partitioning. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 178-191, 2024.
[5] Lizhang Chen, Jonathan Li, and Qiang Liu. Muon optimizes under spectral norm constraints. arXiv preprint arXi: 15054, 2025.
[6] Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As Iyapunov predicts. International Conference on Learning Representations, Spotlight, 2024.
[7] Lizhang Chen, Bo Liu, Lemeng Wu, Kaizhao Liang, Jiaxu Zhu, Chen Liang, Raghuraman Krishnamoorthi, and Qiang Liu. Communication efficient distributed training with distributed lion. Advances in Neural Information Processing Systems, 37:18388-18415, 2024.
[8] NVIDIA Corporation. Neel collective operation Allgather. https: nvidia com/ deeplearning http://wwwwuereeeeeeee/////////s http://wwae.ppprroooooo.ttttttae htw.:ttiiionn..mm.mlllggaaaaaa cessed: 2025-05-13.
Ac-
[9] NVIDIA Corporation.
Neel collective operations:
Reducescatter.
https
//oocs. /doos.nvidia.... iwiiii.....................i.ii. com hwtp///rpppeeggg//eooooo htcp:/cccccccccceeeeeeeee http:///n/nnc//eeeuuuuu//////eee
htm ttml##rrduuessctter Accessed: 2025-05-13.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Ming-Wei
[11] Jarek Duda. Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding, 2014.
[12] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
[13] Kaja Gruntkowska, Alexander Tyurin, and Peter Richtárik. EF2 1-P and friends: Improved theo- retical communication complexity for distributed optimization with bidirectional compression. In ICML, pages 11761-11807 PMLR, 2023.
[14] Jinda Jia, Cong Xie, Hanlin Lu, Daoce Wang, Hao Feng. Chengming Zhang, Baixi Sun, Haibin Lin, Zhi Zhang, Xin Liu, and Dingwen Tao. SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training. In NewrIPS, 2024.
[15] Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich, and Martin Jaggi. Error Feedback Fixes SignSGD and other Gradient Compression Schemes. In ICML, 2019.
[16] Ilia Markov, Adrian Vladu, Qi Guo, and Dan Alistarh. Quantized distributed training of large models with convergence guarantees. ArXiv, abs/2302.02399,, 2023,

---

[17] Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D Sinclair. T3: Transparent tracking triggering for fine-grained overlap of compute collectives. In Pro- ceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 1 146-1164, 2024.
[18] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang. Ze Liu, Yifan Xiong, Ziyue Yang. Bolin Ni, Jingcheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310. 183 13, 2023.
[19] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC, pages 1-16. IEEE, 2020.
[20] Saeed Rashidi, Matthew Denton, Srinivas Sridharan, Sudarshan Srinivasan, Amoghavarsha Suresh, Jade Nie, and Tushar Krishna. Enabling compute-commmnicatioo overlap in distributed deep learning training platforms. 2021 ACM/IEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 540-553. IEEE, 2021.
[21] Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. EF21: A new, simpler, theoretically better, and practically faster error feedback. In NeurlPS, pages 4384-4396, 2021.
[22] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020.
[23] Sebastian U. Stich and Sai Praneeth Karimireddy. The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication. ArXiv, 0bss/190900530 2019.
[24] Baixi Sun, Weijin Liu, J. Gregory Pauloski, Jiamnan Tian, Jinda Jía, Daoce Wang, Boyuan Zhang, Mingkai Zheng, Sheng Di, Sian Jin, Zhao Zhang, Xiaodong Yu, Kamil A. Iskra, Pete Beckman, Guangming Tan, and Dingwen Tao. Compso: Optimizing gradient compression for distributed training with second-orde optimizers. PPoPP '25, page 122-224, New York, NY, USA, 2025. Association for Computing Machinery.
[25] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[26] A Vaswani. 2017.
Attention is all you need. Advances in Neural Information Processing Systems,
[27] Daoce Wang, Pascal Grosset, Jesus Pulido, Jiannan Tian, Tushar M Athawale, Jinda Jia, Baixi Sun, Boyuan Zhang, Sian Jin, Kai Zhao, al. Stz: A high quality and high speed streaming lossy compression framework for scientific data. arXiv preprint arXiv:2509.0162 2025.
[28) Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He. ZeRO++: Extremely efficient collective communication for large model training. In ICLR, 2024.
[29] Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: fine-tuning foundation models over 500mbps networks. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
[30] Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In International Conference on Machine Learning, pages 36058-36076. PMLR, 2023.
[31] Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, et al. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 30-106,77 2022.
12

---

[32] Boyuan Zhang, Yafan Huang, Sheng Di, Fengguang Song, Guanpeng Li, and Franck Cappello. Pushing the limits of gpu lossy compression: A hierarchical delta approach. In Proceedings of the 39th ACM International Conference on Supercomputing. ICS '25, page 654-669, New York, NY, USA, 2025. Association for Computing Machinery.
[33] Boyuan Zhang, Jiannan Tian, Sheng Di, Xiaodong Yu, Yunhe Feng. Xin Liang, Dingwen Tao, and Franck Cappello. Fz-gpu: A fast and high-ratio lossy compressor for scientific computing applications on gpus. In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing, pages
2023.
[34] Boyuan Zhang, Jiannan Tian, Sheng Di, Xiaodong Yu, Martin Swany, Dingwen Tao, and Franck Cappello. Gpulz: Optimizing Izss lossless compression for multi-byte data on modern gpus. In Proceedings of the 37th ACM International Conference Supercomputing. ICS '23, page 348-359, New York, NY, USA, 2023. Association for Computing Machinery.
[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models, 2022.
[36] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences scaling fully sharded data parallel. arXiv preprint arXiv:2304. 12777, 2023.
13

---

A Proofs
We the following lemma (simplified from 223 Lemma 14) without proof.
Lemma A.l. For every non-negative sequence (ri and any parameters a 0, c there exists constant S a. such that
0, T 2 0,
rz+1 - + oq
1 10 1T-111
1T41
arO
2
T+1 E "
T+1
T+1
VT+1
Theorem 4.1 (Recursive Error Correction). The uncompressed weight difference can be recursively
expanded as: A, = w// - Wt-1 19:-1 (19:-2 1): + (1t-1- -1-1), where (g:-2 9c-2)
corrects the error caused by gradient compression, and (Ar-1) Dc-1) corrects the error caused by weight difference compression.
Proof. For the uncompressed weight difference A, w/ W we can expand it as follows.
A, W/ W1-1
W/ 179t- 11411
= Wt i1 179--1
We further expand the term w; 1 follows.
Wt W1-1
W + Ai-1)
W 1
79(-2
i1-2 A,-1 + (Á,-1 A1-1))
W
79t-
w1/11 + (Á.-1 ---1))
Wt-1 7/
Ww-1-- - カタ1-2 22 + (Á--1- - Ac-1))
W - 79:-2
Combining the above equations together, we have
A
Wt W1-1 7/9t-1
Wi-1 - 1191-2 (wi-1 -1-1-1994 119--2 + (A.-1 - A 7
= -999t2 (-nit-- + (Á,-1 - A - 1//1-1
-1/1-1 (1191-2 (1:-22 + (A1-1-Âr- -Â.-1).
Theorem 4.6 (Convergence error bound). For arbitrary non-convex function under Assump-
tion 4.3 4.4 and 4.5 with the apprrrixeee compressor Cg for grudient compression and
the ow apprrxxiiaaee compressor Cu for weight difference compression, taking learning rate n S 8w 360TA(pt1)2 Algorithm converges to a critical point with the following emror bound:
[[] [[=))(11 < (f(wo) - -8) (6
E
T
T:o
/ /384L0² (4amL + 8md (f(wo) -f))
VTo
where 10(1-6,)+8

---

Proof. Using smoothness (Assumption 4.4), we have
f(w++1) S f (ww) - vy (V)(w)) 9--1) 1 L
119
Conditional on wt, taking expectation on the random sample ---1 (denoted by 1Ex), we have
Be[f(we+1]]
L²L
< f(w.) - 17 (V] ((i--)))
E. 0c119-111
1224
= -(w)) - 17 (w:) ((;=,))))
Eg Bxc1199-1111 VI(ūx=1) ([(
T²L
- ((w)) - 77 VV
Ee [llg--1 VI 1)11 [V
2
L 11
[f 1| 178
$ f(w)) VV
2
f(w,) 77 VVV[[ - f((w) vf(w:))
1² TP -[ Vf(()) V + 1
s((w) - /V - (w)
V)())))
L + 1) VI (1 +pp 100+111177 \) + T
2
T/I00
= =((w) - - -
2
-V[V/((= -V((m,))))))()))) + +1
1) -V(w))
Lo?
$ 11 (1-
2
+ 7 [Vf(w:))1 " [ vf(w;)]1
D (a, S 1/10112 11161
+ I + 1) V[ -Vf(m)11
= -[(w) - 2
2 r1(p n 1V + +-=+++=+ + (Vf(0))) - - fwx))1
+ LO?
Again using smoothness, and taking 7) S TfGe+T which leads to -n -nL(p + 1)) s - and
+ L + 1) S in we have
Eçlf
1VV ((ob-1))[(((])]]) Lo²
< f(w)) "1 || fw:)
7
4
2
Ln
Lo
$ f(w)
7 11:f(()))
||,, ū1 12
4
2
Now we establish the upper bound of ||w; w as follows.
First, using we W1-1 79:-2 and úi w Å,-1, have the following equations:
= Wt-1 119t-2 - 1 Å
= Wi-1 - 119t-2 - ú Cu (w/ - 22
Wt-
199t- 2 - 1 W Cur - 110t - (
Wi-1 1/:-2 1r (wp-1-11 - 1/9-- 2x-2) )) + +1g -10.22 2 -19
15

---

For Va, b > 0, we have
We - -12-11
= || 1 - 119h-2 - W1-2 Co(wu-1 - 119h-2 - 2x-2) + 1/9h 719t-
$ |||w;=1 11p--2 1h 2 (wp-1-1 11/t-2 (1,22 llnğt- 179t-211
$ (1+ [+a)]]lw-1- - 7/h-2 2 Cu (wr-1 119:-2 11 21=2)
+11+1/o||100-- 191-211
zy
V ar
a=²
+ 2a
Va,
vB
- - - 7 - 2| + (1 + 1/a)
79h-2 - 79t 2
(1++a) - -1 + - 2
+ (1 1/0)ll19-22-990
< <L4-0|11--0000---000)----11-
+ -2
2(1 a) 8w) - 02-211
+ (1+ (1+11/)n"(0-22----00
Also, taking 6-approximation w.r.t. the gradient compressor Cp. we have
||w - -
[|(0--1000021-0000---0(-11100000 -
$ {[+=]|--6)][++]]]]]-----o-) +2L-=o|4-66]|44///7//000011 1/bj]
+
(1 172 - 15))
+ (1/==
5,)1 11
+ [2(1+a)(11 (1) 11
Further, taking expectation w.r.t. the random sample 4- (denoted by Eş), and denoting a 2(1 + a)(1 51 + 1/b) + (1+ -6) + 2(1 + a1 8w) 1/b)(1 6g), we have
Eg [[|w] - 11
< (1 + (1 - (1 + b)]lw[]- - /.-2 2 + 2 [ + E. [l10--2 ( [?]
< (1 + a1 - (1 + b) b)]lwr+1 - 21 2 + onj (p + (( + ani²0
Then, taking
6. b 5. we have 21-005]
212-0w..
Eg lw: 11
< (1 8 w 01 + [) + an
< (1 / /4)]|//-1
|1-1 - /. 2 20n + 1) ((,=-i))
+ 2an2 (p + 1)IlV
ff 1)1 2 + an?
< (1 8mr/4/ / 4)]l - - + 20722 (p + V ([( + 200/ (p + |) - 2
+ an?
b smoothness
< (1 bm./8)]lu
-w; 2 22 2077 + 1) (w. anj²
> using S V 16((0+1)L3>
by
i.c. 20n2 + 1)L² < dn
16

---

where a could be simplified as
a
+ a) )w 1/b) + (1 1/a) 62)) + 2(1 + a) 8 (1 1/6) -g))
(2 6w)(4= 6w) (2 8w)(1 6q) (2 (4 (4- 5w - 6g)
8w
8m
8m
10(1 6g) +
8m
We simplify the notation by denoting E[-] Bgg [-l and unroll the sequence back to t = 2, which yields:
E ||w 11
88/88) 1
+ +an anja=]]
W1
Wo
w1
wo
$ <panf(p 1) Σ (1 8 [1= any o²
8 (8)
Sarja
s 220m + 1) 8m, (1---//))
Et=ol
(1 Bu St-
<
1- (1
Then, staking E [[lw> 113] and taking total expectation, have
T
E [[ - W. [|2]
T
8Tan 02
22n + 1) Σ
(1 8m.
+
6
Im2 mm
+00
8TaNPO2
2 +1
(1 18)
8
16anp +1)
8TamP02
(V
8w
8
16anz 1)
T
8Tam202
8
8
16amz (p+1) 1)
8T
[V((w))
8
8m
17

---

Putting all the ingredients together and taking total expectation, we have
E (f(++l
3nL²
TipLo
[w)(w
Σ
E
[v((w))11
+
Σ E l0, wr-1|??] +
2
t=1
[[f(w:)]
"
3nL²
16an (p 1)
V /V [1)
8Tanf
8w
8w
Lo²
Tnj Lo
2
T
67 [V(())]
To2
[wf(w:)
E
8w
1=1
using
V (6T.
S.
i.c.
3n 16an (p+1)
Re-arranging the terms, we have
67 T.02
TT2
E
[[V s [[f(wo)) [[[(wT)]] +
+
8w
2
D note that w wo
8E[f(wo) f(TT))
48an² L² 2
+ 4nLo²
T
Tn
8m
Finally, using Lemma A.1
we have
T
1
E
V 1(
T
1=1
SEIf (wo) - f(wr)]
+ 2
+ 4nTo²
Tn
(f(w)) fr) (114ALI((p11) 1)2
384Lo² (4amL + dw) (f(wo) - f³)
TVo
VTow
taking 7 S / Mall( S min Tb[p+]j / 16alp+1) v 36L2a(e+1)
B Training Configuration for the E2E Accuracy Test
Table E2E accuracy task training parameters.
Model Size
Learning Rate
Betas
Epsilon
Weight Decay
Batch Size
125M
6c-4
0.9, 0.95
lc-8
0.1
512
350M
3e-4
0.9, 0.95
le-8
0.1
512
1.3B
2c-4
0.9, 0.95
Ic-8
0.1
512
6.7B
1.2c-4
0.9, 0.95
le-8
0.1
512
18

---

C
Training Configuration for the Throughput Test
Table 7: Training configuration for the throughput test on A 100 and H20 systems.
System
Parameter
1.3B
2.7B
6.7B
13B
18B
TP Size
4
4
4
4
4
PP Size
1
2
DP Size
8
8
8
4
4
A100 (8x4)
Grad Accum Step
32
32
32
32
32
Microbatch Size
4
4
1
Seq. Length
4096
4096
4096
4096
2048
TP Size
8
8
8
8
Size
1
1
DP Size
8
8
8
8
8
H20 (8x8)
Grad Accum Step
32
32
32
32
Microbatch Size
4
4
1
Seq. Length
4096
4096
4096
4096
4096
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately shows the contribution in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification: In Sections 2.3 and 3.2, we discuss that the proposed method is only efficient when the newly introduced communication can effectively overlapped with computation. Additionally, in Section 5.1, we clearly specify the dataset used in our experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The full set of assumptions could be found Section 4.2, and complet proof are given in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
21

---

Justification: We list the models and datasets used in our experiments in Section 5.1, and provide detailed training hyperparameters in Appendices B and C.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The dataset used in our experiments is open-sourced and publicly available on Hugging Face. We also provide our source code along with detailed instructions to reproduce the main training results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.cm cc/
Souides//ooeSuumissttt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
22

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: The experiments hyperparameters for both accuracy and performance test can be found in Appendices B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviations for all performance tests to indicate variability and ensure statistical transparency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
23

---

Justification:
The detailed experiment environments can be found in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the code of ethics of NerulPS and don't think this work have any negative social impact or potential harmful consequences.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The goal of this work is to provide a higher training accuracy when compression is being used, don't publish new datasets new models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any new dataset or new model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package or dataset, and included the license in the source code package.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the license in the source code package, which is aligned with the code packages that our work is based on.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
25

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
This research does not involve LLMs as any important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non sta-sadddard components.
Please refer to LLM policy http:///wrrrpsscooooooooo//e//// for what should or should not be described.
26


Reviews:
[{"review_id": "RGsHU5HwDY", "reviewer": "Reviewer_vxA1", "summary": "Large scale distributed training uses two techniques to overcome communication bottlenecks: gradient compression and overlapping the computation and communication. However, aggresively employing them simulatenously (like the prior SDP4bit algorithm) can potentially lead to degradation of accuracy of the trained model. This paper modifies SDP4bit with an additional \"slow\" background communication step where a more high precision synchronization happens, relying on potentially unused bandwidth in the network.", "questions": "1. What are the current SOTA large scale training methods?  \n2. How does DUO compare against them empirically?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper is easy to read and well written.  \n- The modification to SDP4bit is well motivated and the experiment setup comparing to SDP4bit is quite thorough. The improvement results are convincing as well and are performed on realistic large scale training.  \n\nWekanesses  \n\n- This paer lacks comparision to any other method apart from SDP4bit. It justifies this by claiming SDP4bit is the SOTA, but I do not see evidence for that claim. Even now well establised methods like [PowerSGD](https://arxiv.org/abs/1905.13727) as used to train [DALLE2](https://proceedings.mlr.press/v139/ramesh21a.html) and [CocktailSGD](https://proceedings.mlr.press/v202/wang23t.html) have not been compared, let alone the current SOTA methods.  \n- The related work discussion is also very sparse - there is no discussion of recent papers on large scale training e.g. [DiLoCo and variants](https://arxiv.org/abs/2502.12996).", "was_revised": true, "final_justification": "The reviewer responses have sufficiently addressed my concerns.", "rebuttal": "### **W1 - This paer lacks comparision to any other method apart from SDP4bit. It justifies this by claiming SDP4bit is the SOTA, but I do not see evidence for that claim. Even now well establised methods like PowerSGD as used to train DALLE2 and CocktailSGD have not been compared, let alone the current SOTA methods.**\n\nWe respectfully clarify that SDP4Bit is currently the strongest publicly available compressor designed specifically for Sharded Data Parallelism (SDP)—a paradigm widely adopted in large-scale LLM training frameworks such as Megatron-LM, DeepSpeed-ZeRO, and PyTorch-FSDP.\n\nWithin the scope of communication compression for **Sharded Data Parallelism (SDP)**, only a few peer-reviewed methods exist—namely, ZeRO++, QSDP, and SDP4Bit. Among these, SDP4Bit has already conducted comprehensive comparisons against ZeRO++ and QSDP and has demonstrated superior performance. Based on this, we selected SDP4Bit as our primary baselines.\n\nOur proposed **DUO** is a **system–algorithm co-design**, whose core contribution is to **compensate the accuracy loss of any compressor** under SDP. It is compatible with **arbitrary compressors**, unlike ZeRO++/QSDP/SDP4Bit which rely on unbiased compression. Thus, DUO is orthogonal to the choice of compressor, and the focus of our paper is not to benchmark compressors per se, but to demonstrate how **DUO improves training quality**.\n\nRegarding PowerSGD, it is not compatible with SDP. PowerSGD assumes layer-wise AllReduce of full gradients, while Megatron-LM SDP uses flattened, sharded gradients with ReduceScatter, making such operations infeasible. Notably, DALLE-2 used PowerSGD only for inter-replica communication in **DDP**, not in the context of **sharded training**.\n\nAs for **CocktailSGD**, it is optimized for **extremely low-bandwidth fine-tuning scenarios**, with compression ratios exceeding 100×, and has **not been validated for accuracy-sensitive, large-scale pretraining**. \nMoreover, Its experiments are conducted under DDP and pipeline parallelism, not Sharded Data Parallelism (SDP). Its use of **Top‑K sparsification** is **not directly compatible with ReduceScatter-based communication**, and may introduce **additional challenges for system-level optimization**. \n\n---\n\n### **W2 - The related work discussion is also very sparse - there is no discussion of recent papers on large scale training e.g. DiLoCo and variants.**\n\nWe thank the reviewer’s advice and will expand our **Related Work** section accordingly to clarify this distinction and cite **DiLoCo** and related variants.\n\nIn particular, **DiLoCo** and similar **LocalSGD-based methods** are **orthogonal to DUO**. The combination of **Distributed Data Parallelism (DDP)** and **Sharded Data Parallelism (SDP)** has already been explored in prior work, such as **HSDP** [1]. **DUO and DiLoCo** can, in fact, be combined in **HSDP-style setups** to further reduce communication overhead. In this setting:\n- **DiLoCo** reduces **inter-replica communication** by lowering synchronization frequency\n- **DUO** reduces **intra-replica communication** through error-compensated compression\n\nTo elaborate: **LocalSGD-style methods** aim to reduce communication by allowing several local update steps before synchronizing. These typically assume each worker holds the entire model (or identical copies), which differs from **SDP**, where model parameters and optimizer states are **sharded across workers** and communication relies on **ReduceScatter/AllGather**.\n\n[1] HSDP: Accelerating Large-scale Model Training via Efficient Sharded Data Parallelism\n\n---\n\n### **Q1 - What are the current SOTA large scale training methods?**\n\nWe assume the “**training methods**” referred to here concerns **training frameworks**. Widely adopted systems include **Megatron-LM**, **DeepSpeed-ZeRO**, and **PyTorch-FSDP**, all of which implement **Sharded Data Parallelism (SDP)**. SDP is one of the most effective and widely used mechanisms for large-scale model training. It works by **partitioning the optimizer states**—often the most memory-intensive component of model training—so that each worker holds only a **shard** of the states and updates its corresponding parameters. Synchronization across workers is performed via **ReduceScatter** and **AllGather** operations.\n\nFor **communication compression tailored to SDP**, the main methods are **ZeRO++**, **QSDP**, and **SDP4Bit**. Among them, **SDP4Bit** has already provided thorough comparisons against ZeRO++ and QSDP, and currently stands as the **strongest open-source baseline** for gradient compression under SDP.\n\nOther compression techniques such as **PowerSGD**, **QSGD**, and **SignSGD** are mostly designed and evaluated under **naive DDP settings**. Most of them lack validation in **large-scale LLM pretraining tasks**, particularly when used with **Adam optimizers** and **Sharded Data Parallel (SDP)** setups, making them less applicable to our setting. Though PowerSGD is not compatible with SDP, QSGD and SignSGD are compatible. Note that the combination of QSGD or SignSGD with SDP is conceptually the same as ZeRO++ or QSDP.\n\n---\n\n### **Q2 - How does DUO compare against them empirically?**\n\nAs explained in Q1, SDP‑4Bit is already a strong representative of current SOTA compressors for SDP. We choose to apply DUO on top of SDP4Bit, thereby demonstrating how DUO can improve final model accuracy while maintaining its high compression ratio.\nFor the compressors, we choose the quantization as our compression methods to keep our evaluation consistent to the previous works such as QSDP and ZeRO++. Note that the focus of our paper is not to benchmark compressors."}, {"review_id": "GEqfOR2tU8", "reviewer": "Reviewer_Gttj", "summary": "This paper introduces DUO (Double Update Overlap), a distributed training framework that aims to mitigate potential accuracy degradation caused by gradient compression and without incurring additional overhead. \n\nSpecifically, DUO inserts an additional high-precision gradient synchronization step during the computation phase, aiming for the communication to be fully hidden by computation.\n\nThe paper also provides theoretical proof of convergence for DUO under smoothness and bounded variance assumptions and validates its effectiveness through pre-training experiments on GPT models.", "questions": "Can the authors elaborate on the value of compressed versus accurate gradient transmission in DUO? Namely, why is compression even required if the 0-bit approach seems to offer near-baseline accuracy?", "limitations": "The paper states that the proposed method is effective only when communication can be overlapped with computation.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Pros:\n\n•\tUsage of computation-communication overlap to mask communication delays\n\n•\tMaintains minimal accuracy degradation under extreme compression (e.g., 1 bit per parameter) in the tested scenarios\n\n•\tOptimizes memory usage through high-precision gradient offloading\n\n\nCons:\n\n•\tUnclear novelty - the idea of using computation phases to mask communication delays is standard practice\n\n•\tRelated work and evaluation leave more to be desired - numerous strong compression baselines and systems are not considered (e.g., see [1] and references within)\n\n• Increased Communication Overhead - While accurate transmission is achieved during the computation phase, it still increases the communication overhead; thus, the comparison to other compression techniques should take this additional budget into account.\n\n•\tThe comparison to error feedback is not convincing, and the presented results seem to go against the previous conclusion (e.g., [2]).\n\n[1] Xu, Hang, et al. \"Compressed communication for distributed deep learning: Survey and quantitative evaluation.\" (2020).\n\n[2] Karimireddy, Sai Praneeth, et al. \"Error feedback fixes signsgd and other gradient compression schemes.\" International Conference on Machine Learning. PMLR, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "### **C1 - Unclear novelty - the idea of using computation phases to mask communication delays is standard practice**\n\nOur proposed **DUO** is a **system–algorithm co-design**, and this integration is central to our contribution—it cannot be decomposed into independent components. The co-design consists of two parts:  \n(1) A novel algorithm (Sec. 3.1) called **Fast–Slow**, which mitigates compression-induced errors; and  \n(2) A **system-level optimization** (Sec. 3.2–3.3) that enables effective overlap of communication introduced by this algorithm.\n\nThe idea of using computation to mask communication is indeed well-known, but in our work, it is not a generic overlap strategy—it is **specifically tailored to the Fast–Slow algorithm**, and their **combination** forms the core of DUO’s contribution. More importantly, our overlapping mechanism is fundamentally different from conventional strategies used in existing frameworks.\n\nAs explained in **Sec. 2.3** and **Figure 2**, while frameworks like **ZeRO**, **Megatron‑LM**, and **FSDP** implement conventional overlap (e.g., pre‑AllGather + post‑ReduceScatter), they only hide communication for the **final micro-batch** in each gradient accumulation window (that is 1/n of total backward computation). When accumulation has **n** steps, these methods overlap at most **1/n** of the backward pass, leaving the remaining **(n – 1)/n** exposed to communication delays.\n\nIn contrast, our **Fast–Slow algorithm** introduces a new **high-precision synchronization step** that can be **delayed to the next iteration**. This allows DUO to overlap communication with the **(n – 1)/n** portion of the backward pass that conventional frameworks leave uncovered. As a result, **DUO significantly improves overlap efficiency** beyond what existing strategies achieve.\n\n---\n\n### **C2 - Related work and evaluation leave more to be desired - numerous strong compression baselines and systems are not considered (e.g., see [1] and references within)**\n\n**DUO** is designed specifically for **Sharded Data Parallel (SDP)** training—used in frameworks such as **DeepSpeed**, **Megatron‑LM**, and **PyTorch‑FSDP**—which have become the standard for large-scale LLM training. To date, only a few works have focused on **communication compression in SDP settings**, namely **ZeRO++** [1], **QSDP** [2], and **SDP4Bit** [3].\n\nIn contrast, many methods cited in the referenced survey (e.g., [4]) are evaluated under **naive DDP with SGD optimizers**. These settings are not aligned with current **LLM pretraining practices**, which rely on **SDP and AdamW**. As such, applying those baselines would not lead to a fair or meaningful comparison in our context.\n\nThat said, we appreciate the reviewer’s suggestion and will include a discussion of these additional methods in the **Related Work** section to clarify how **DUO** differs.\n\nFinally, we emphasize that our focus is **not on proposing a new compressor**, but rather on **enhancing existing ones via error compensation**. Therefore, a broad empirical study across all compressors is beyond the scope of this work.\n\n[1] ZeRO++: Extremely Efficient Collective Communication for Giant Model Training  \n[2] Quantized Distributed Training of Large Models with Convergence Guarantees  \n[3] SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training  \n[4] Compressed Communication for Distributed Deep Learning: Survey and Quantitative Evaluation\n\n---\n\n### **C3 - Increased Communication Overhead - While accurate transmission is achieved during the computation phase, it still increases the communication overhead; thus, the comparison to other compression techniques should take this additional budget into account.**\n\nWe appreciate the reviewer’s concern about communication overhead. However, **DUO is a system–algorithm co-design**, and its communication should not be considered in isolation. As detailed in Sec. 2.3 and Sec. 3.2, the extra communication introduced by DUO is **fully hidden using our custom overlapping strategy**, which leverages **computation phases that conventional methods leave idle**. As a result, **the end-to-end training time remains unchanged**.\n\nSpecifically, DUO introduces one full‑precision gradient ReduceScatter per iteration (equal in size to the model), but this is **carefully overlapped with the earlier stages of the next iteration’s backward computation**. As shown in Figure 2 and Table 7 (Appendix D), this communication fits entirely within the previously unutilized **(n−1)/n portion** of the computation window, which standard methods cannot overlap.\n\nIn practical large-scale training, what ultimately matters is **wall-clock throughput and final model quality**, not raw byte count. In modern clusters, **network bandwidth is typically provisioned to remain idle if not fully used**. DUO effectively converts this **unused bandwidth into a “free” accuracy gain**, delivering better convergence without increasing total training time.\n\n---\n\n### **C4 - The comparison to error feedback is not convincing, and the presented results seem to go against the previous conclusion (e.g., [2]).**\n\nWe believe the apparent contradiction arises from a difference in **training regimes and model type/scale**. The method in [1] is based on **SignSGD**, which combines SGD with 1-bit compression. This setup differs fundamentally from our setting, which uses **Adam optimizers for large-scale LLM pretraining**.\n\nMoreover, the experiments in [1] are conducted on **small-scale computer vision models**, using a different model architecture, and dataset. The empirical results from [1]—which are valid in their domain—**may not generalize to large language models** trained with **AdamW optimizer**.\n\nAs such, **the conclusions drawn from [1] are not transfer meaningfully to our context**.\n\n[1] Karimireddy, Sai Praneeth, et al. \"Error feedback fixes signsgd and other gradient compression schemes.\" International Conference on Machine Learning. PMLR, 2019.\n\n\n---\n\n### **Q1 - Can the authors elaborate on the value of compressed versus accurate gradient transmission in DUO? Namely, why is compression even required if the 0-bit approach seems to offer near-baseline accuracy?**\n\nFor the 0-bit situation, it actually means **0 bits for data transmission**. This compression setting can be seen as **compressing the global gradients** (obtained after reduce-scatter) into the **local gradients before reduce-scatter**. In ordinary DDP training, the global gradients are the result of applying all-reduce to the local gradients. Since we use local gradients directly without performing reduce-scatter—thus **no data transmission**—this is referred to as “0-bit.” The low-precision optimizer uses these local gradients, **not some 0-bit empty tensor**.\n\nTo answer the question of why compression is still required, we provide a more detailed explanation of DUO’s **two types of gradient communication**. In the full DUO setting, we maintain two complementary gradient channels, as shown in **Algorithm 1 (Lines 5 and 12)**:  \n1) **Timely but compressed communication**  \n2) **Accurate but one‑step‑stale asynchronous communication**\n\nThis co-design allows DUO to deliver **fresh gradients each step** via the compressed channel, and then **correct them exactly one step later** via the accurate channel. This leads to performance **very close to full-precision training** (as shown in **Table 2**).\n\nThe **0‑bit variant** disables the compressed communication channel entirely and relies solely on the **delayed accurate channel**. This reduces communication cost further—making it suitable for **extremely bandwidth-constrained environments**—but typically comes with a **small drop in final accuracy** compared to full DUO.\n\nIn summary, **communication for the compressed channel is still valuable when bandwidth allows**, as it improves convergence. The **0‑bit mode** is a **fallback for more extreme settings**."}, {"review_id": "XQ3B82K9sS", "reviewer": "Reviewer_GiNG", "summary": "This paper introduces DUO (Double Update Overlap), a communication-efficient distributed training framework designed to mitigate the accuracy degradation caused by gradient compression in LLM training. The key idea is leveraging otherwise idle communication time to incorporate an additional high-precision gradient communication phase that overlaps with computation. The authors propose the Fast-Slow Algorithm, which mitigates accuracy degradation by incorporating a high-precision gradient to influence optimizer state updates without compromising throughput. This paper provides a theoretical proof of convergence for DUO and validate its effectiveness on GPT-2 model training. Results show that DUO can maintain the accuracy of 4-bit gradient compression to levels comparable with uncompressed training, even with 1-bit gradient compression it still can provide high accuracy.", "questions": "Questions:\n\nTypically, gradient compression with biased compressors requires error feedback. How does DUO converge without error feedback? And could the authors provide a more detailed comparison and discussion between DUO and error feedback?\nIs DUO also compatible with other compressors, such as top-k sparsification?\nThe 0-bit (no gradient transmission) is confusing, could the authors provide some more detailed explanation?\nCan DUO’s 1-bit (or 0-bit) training maintain its current performance on different datasets and larger-scale models, such as those with 18B models?", "limitations": "The authors do not provide a discussion of the limitations of their work; see Weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper conducts experiments on GPT models with sizes up to 6.7B. DUO achieves accuracy comparable to uncompressed training, even under 1-bit gradient compression or even 0-bit. Notably, in the 1-bit setting, DUO surpasses SDP4Bit in terms of the loss curve while introducing only minimal additional memory overhead. The 0-bit setting is also a very interesting idea.\nThe paper provides a system-algorithm co-design, leveraging otherwise idle communication time to incorporate high-precision gradients to reduce the communication overhead without compromising the training accuracy.\nThe integration of algorithmic innovation (Fast-Slow Algorithm) with system optimizations (memory footprint optimization, computation overlapping) makes DUO a practical system for distributed LLM training. The paper also provides substantial implementation details, making the approach more likely to be reproducible.\nTheoretical analysis of convergence is also provided for smooth but non-convex settings.\n\nWeaknesses:\n\nThe paper focuses on gradient compression, but there is very limited comparison or discussion related to error feedback methods, theoretically and empirically.\nThe compression techniques used in the experiments are limited to simple quantization.\nThe 0-bit (no gradient transmission) is confusing, which lacks detailed explanation.\nWhile the end-to-end throughput experiments are scaled up to 18B models, the largest model size for the validation loss experiments is up to only 6.7B, which is relatively small.", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness**\n\nSince the questions are very similar, we integrate our response directly into the question section.\n\n\n### **Question - Typically, gradient compression with biased compressors requires error feedback. How does DUO converge without error feedback? And could the authors provide a more detailed comparison and discussion between DUO and error feedback? Is DUO also compatible with other compressors, such as top-k sparsification? The 0-bit (no gradient transmission) is confusing, could the authors provide some more detailed explanation? Can DUO’s 1-bit (or 0-bit) training maintain its current performance on different datasets and larger-scale models, such as those with 18B models?**\n\nDUO employs an alternative approach to error feedback by using a **recursive error correction mechanism** to compensate for errors introduced by gradient and weight-difference compression, rather than relying on traditional error feedback methods explicitly. As detailed in **Theorem 4.1**, DUO recursively corrects errors arising from compressed gradients and weight differences through a **delayed, high-precision gradient synchronization step**. The **theoretical convergence guarantee** provided in **Theorem 4.6** further ensures that DUO converges at a comparable rate to standard SGD, even without traditional error feedback.\n\nEmpirically, as shown in **Figure 1**, **Error Feedback performs suboptimally under large-scale training**. In our view, the failure of classic **Error Feedback (EF)** mechanisms under aggressive compression may stem from their **incompatibility with the Adam optimizer and large-scale LLM training**. Prior EF work (e.g., [1] using SignSGD) focused mainly on **SGD-style optimizers** and relatively small CV models, where EF achieved strong results. \nHowever, a **theoretical analysis directly comparing** the relationship between DUO and error feedback mechanisms remains an **interesting future research direction**.\n\nRegarding compressor compatibility: **DUO is compressor-agnostic**, as long as the compressor is compatible with **Sharded Data Parallelism (SDP)**. This includes quantization methods like **SDP4Bit**, as well as potential compatibility with **sparsity-based approaches** (e.g., Top‑k sparsification), though we leave this as **promising future work**.\n\nWe acknowledge that the **0-bit mode** can be confusing and will revise the manuscript to clarify this further. Specifically:\n\nIn the **0-bit setting**, DUO transmits **zero bits of gradient data** in the compressed communication channel. This can be viewed as replacing the global (Reduce-Scattered) gradient with the **local pre-communication gradient**. Since **no communication is performed**, this is termed “0-bit.” The optimizer still receives a **meaningful local gradient**—it is not an empty tensor.\n\nIn the full DUO configuration, we maintain **two gradient channels** (see **Algorithm 1, Lines 5 and 12**):\n1) **Timely but compressed communication**  \n2) **Accurate but one-step-stale asynchronous communication**\n\nThis co-design delivers **fresh but compressed gradients** at each step and provides **exact correction one step later**. The **0-bit variant** disables the compressed communication channel entirely, relying only on the **delayed accurate channel**. This reduces communication even further—**ideal for highly bandwidth-constrained environments**—but typically comes with a **small drop in final accuracy**.\n\nAs for scalability: we have tested DUO across a range of **GPT model sizes**, and we have **not observed any degradation trend** in performance as model size increases. While our largest accuracy experiments are currently up to **6.7B**, we are actively working on extending this to **18B-scale pretraining** and believe DUO will continue to perform well under **larger models and datasets**."}, {"review_id": "ptqffaiB17", "reviewer": "Reviewer_vD9N", "summary": "This paper proposes a distributed training framework that alleviates the accuracy loss caused by aggressive gradient compression. By combining AllGather and ReduceScatter and exploiting the idle periods between synchronization steps to overlap communication of additional high-precision gradients, the accuracy degradation due to gradient compression is effectively reduced. Convergence of the proposed algorithm is established, and experiments on GPT models (125M-18B parameters) pretrained on The Pile validate its effectiveness. Overall, the topic is interesting and timely.", "questions": "The framework requires saving full-precision gradients, copying them out-of-band (GPU→CPU) via a GPU memory bridge, and scheduling asynchronous reductions, which seems to be fairly complex. Could the authors comment on whether it can be readily integrated into existing codebases?", "limitations": "The authors have discussed some limitations in the checklist.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Distributed pre-training of large language models is of paramount interest nowadays, and the compensation for the degradation in accuracy due to gradient compression is an important research direction.\n\n2. Convergence of the proposed algorithm is established from a theoretical perspective, and experiments on various sizes of GPT models are conducted, which are quite comprehensive.\n\n3. Overall, the paper is well-written.\n\nWeaknesses: \n\nI do not identify major weaknesses in this paper. That being said, the proposed strategy depends on having idle compute time between communication phases. In settings with a few micro-batches, there would be little idle time to hide the extra full-precision ReduceScatter. It would be helpful to discuss how DUO performs when accumulation is limited in the main text.", "was_revised": true, "final_justification": "The authors have addressed my concerns and questions, and I will keep my positive score.", "rebuttal": "# Answer to Weakness\n\nThank you for highlighting this point. Indeed, a key component of **DUO's system optimization** is the overlapping of **full-precision gradient communication** with the **backward computation phase**. For this to be fully effective, the compute time must exceed the time required for the added communication introduced by the **Fast–Slow algorithm**.\n\nDUO may be **less effective** in scenarios where **computation is short relative to communication**, such as:\n- **Very short sequence lengths**\n- **Small gradient accumulation steps**\n- **Severely bandwidth-constrained environments**\n\nIn such settings, there may not be sufficient idle compute time to fully hide the additional communication, and DUO’s overhead may become partially exposed, reducing its efficiency.\n\nWe analyze these trade-offs in detail in **Appendix D**, which maps out the regimes where overlap starts to break down. As a **practical remedy**, increasing the **gradient accumulation step size** extends the compute window and restores DUO’s ability to fully overlap communication and reclaim bandwidth.\n\nWe will clarify this limitation in the **main text**, and thank the reviewer again for the thoughtful suggestion.\n\n# Answer to Question\n\nYes, we have already integrated **DUO** into **Megatron-LM**, one of the three most widely used frameworks for large-scale LLM training. Our implementation demonstrates that **DUO can be incorporated with minimal modifications** to the existing training loop, despite involving GPU–CPU transfers and asynchronous scheduling.\n\nWe will **open-source the code upon acceptance**, enabling easy adoption and integration by the broader community."}, {"review_id": "XbcT59e8rC", "reviewer": "Reviewer_o9Va", "summary": "This paper introduces DUO (Double Update Overlap), a distributed training framework designed to mitigate accuracy degradation caused by gradient compression, particularly at low precision (e.g., 4-bit or 1-bit). DUO achieves this by inserting an additional high-precision gradient synchronization step into a computation-only phase, fully hiding its communication overhead. The framework leverages a fast-slow reduction algorithm that caches full precision gradients for a later high-precision update. The main advantage is that such a new algorithm is amenable to communication overlapping. \n\nThe authors also incorporate several interesting memory optimization strategies (Bucket-Wise Offloading, Finer Overlapping and GPU Memory Bridge). Theoretical convergence guarantees are provided followed by extensive experimental validation on LLM models, showing that DUO restores accuracy comparable to uncompressed training even under extreme compression (1-bit and 0-bit gradients), while maintaining comparable throughput and memory usage to state-of-the-art methods like SDP4Bit.", "questions": "Few quick questions to the authors:\n- Could the authors provide more insight into the computational cost of the optimizer rollback and reset mechanism (Line 14 of Algorithm 1) and if it contributes to any minor overhead not captured in the E2E throughput or experiment descriptions?\n\n- The concept of 0-bit transmission seems intriguing to me as well as bit confusing. Can the authors offer more explanation on why DUO performs the way it does in this setting? What brings the robustness? \n\n- Have the authors thought about how DUO can be applied to other important distributed training paradigms such as Federated learning, Semi-Sync Training (e.g. DiLoCo) etc? Curious to hear any thoughts on these.\n\n- Are there any specific types of models or training scenarios where DUO might not be as effective, or where its benefits might be less pronounced? Have the authors envisioned such setups?", "limitations": "The paper only provides a novel algorithm, doesn't release any dataset or model. Therefore I think this question does not fully apply here. The authors have also added this justification.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nOverall, I feel this paper is a strong empirical contribution, solving a key challenge in distributed training.\nBelow I highlight a few of them:\n- The paper tackles the key challenge in memory-efficient (low-bit) LLM training these days - which is the significant drop in accuracy. This has been a long-standing (and largely unsolved) bottleneck. Techniques  provided in this paper seem very beneficial to me to tackling these challenges..\n- I found the core idea of using a high-precision \"slow\" gradient update alongside a low-precision \"fast\" update, and cleverly overlapping the slow gradient communication innovative.\n- The experimental results presented are strong. DUO demonstrates impressive accuracy restoration across various GPT model sizes and extreme compression ratios (including 0-bit gradients), almost matching full-precision baselines. This is a very compelling result.\n- The authors further optimize their solutions by incorporating more practical systems optimization solutions such: Bucket-Wise Offloading, Finer Overlapping, GPU Memory Bridge - these are great contributions that are well-explained and address practical implementation challenges.\n- I like the fact that the authors use Megatron-LM as the reference open-source integration. This makes their approach well tested and can increases their adoption potentially.\n\nWeaknesses:\n- While effective, the DUO framework, particularly the Fast-Slow Reduction algorithm with its optimizer rollback and reset mechanism (Line 14-15 of Algorithm 1) appear quite intricate. This complexity might pose a challenge for widespread adoption or implementation outside of highly specialized teams. Wondering if the authors have thought about this? Are there other ways this could be simplified?\n- There has been a major pillar of work in compressed communication for distributed learning which focusses on Error Feedback mechanisms. While the paper mentions error feedback (EF) methods and shows SDP4Bit-G1-EF failing to converge, a more in-depth comparison or discussion on why DUO succeeds where EF struggles, especially for LLMs under aggressive compression, could strengthen the argument.\n- While extensive experiments on GPT models are valuable since dense models form the bulk of LLMs used these days, a brief discussion on the potential applicability or challenges of DUO to other deep learning architectures (e.g. sparse MoE type models) could be beneficial. Not sure if this has been discussed in the paper and I missed it.", "was_revised": true, "final_justification": "== Update after rebuttal ==\n\nRaised from 5 to 6.", "rebuttal": "**W1 - While effective, the DUO framework, particularly the Fast-Slow Reduction algorithm with its optimizer rollback and reset mechanism (Line 14-15 of Algorithm 1) appear quite intricate. This complexity might pose a challenge for widespread adoption or implementation outside of highly specialized teams. Wondering if the authors have thought about this? Are there other ways this could be simplified?**\n\nThe rollback step is **unavoidable** if we want the high‑precision correction to apply to **exactly the same optimizer states used** in the low‑precision update. After the compressed update advances the optimizer state, we must restore that state to its pre‑update values before applying the accurate version.\n\nFortunately, because **Adam‑style optimizers** update their statistics element‑wise, this rollback is **analytically reversible** and introduces **negligible overhead** (see also Q1). We have also explored implementation simplifications: this function can be implemented using frameworks like *Torch Compiler* or *Triton*, but for ease of adoption, we have implemented a **self‑contained CUDA kernel** that fuses the rollback and accurate re-apply into a single call.\n\nThis kernel will be open-sourced upon acceptance, allowing practitioners to integrate it into existing Adam/AdamW pipelines with just a **two-line wrapper**, requiring no major refactoring.\n\nIn short, while the rollback procedure is essential to ensure Fast–Slow’s **accuracy guarantee**, we have packaged it for **practical use**, so that non-specialized teams can adopt DUO with **minimal engineering burden**.\n\n\n**W2 - There has been a major pillar of work in compressed communication for distributed learning which focusses on Error Feedback mechanisms. While the paper mentions error feedback (EF) methods and shows SDP4Bit-G1-EF failing to converge, a more in-depth comparison or discussion on why DUO succeeds where EF struggles, especially for LLMs under aggressive compression, could strengthen the argument.**\n\nIn our view, the failure of traditional **Error Feedback (EF)** mechanisms under aggressive compression may stem from their incompatibility with the **Adam optimizer** and **large scale Lange Model**. Prior EF work [1] (e.g., using SignSGD) is based on **SGD-style optimizers** on **relatively small CV models**, where EF has shown strong empirical results. However, EF appears to perform poorly when combined with Adam or AdamW for large scale LLM pretraining.\n\nThat said, this remains a hypothesis, and we acknowledge that a rigorous theoretical and empirical study is needed to fully understand why EF struggles in this context. Our **DUO framework**, in contrast, is explicitly designed to operate within the **Adam + SDP** setting and includes **system-level integration** to ensure high-precision correction is applied effectively without optimizer-state inconsistency.\n\nWe thank the reviewer for highlighting this point, and we agree that a deeper investigation into EF’s limitations under modern LLM training regimes would be valuable future work.\n\n[1] Karimireddy, Sai Praneeth, et al. \"Error feedback fixes signsgd and other gradient compression schemes.\" *International Conference on Machine Learning*. PMLR, 2019.\n\n**W3 - While extensive experiments on GPT models are valuable since dense models form the bulk of LLMs used these days, a brief discussion on the potential applicability or challenges of DUO to other deep learning architectures (e.g. sparse MoE type models) could be beneficial. Not sure if this has been discussed in the paper and I missed it.**\n\nThank you for raising this point. We did not conduct a comprehensive evaluation on MoE-style models due to limited compute resources, but we agree this is an important direction.\n\nThat said, DUO is not restricted to dense, fully deterministic Transformer blocks. The prerequisite is that the training stack employs **Sharded Data Parallelism (SDP)** using ReduceScatter/AllGather for gradient synchronization—this is the default in Megatron-LM and similar frameworks.\n\nUnder this communication pattern, DUO is directly applicable to sparse expert architectures, including Mixture-of-Experts (MoE) models and MoE-style variants of LLaMA, Qwen, and other recent large models.\n\n**Q1- Could the authors provide more insight into the computational cost of the optimizer rollback and reset mechanism (Line 14 of Algorithm 1) and if it contributes to any minor overhead not captured in the E2E throughput or experiment descriptions?**\n\n  The computational overhead of the optimizer rollback is very similar to that of a standard optimizer step—both are **lightweight, element-wise operations** with no inter-element dependencies. As such, their cost is **negligible** compared to the overall training step.\n\n  To provide concrete numbers, we profiled the optimizer rollback during GPT-350M training with 8 gradient accumulation steps and a sequence length of 2048. In this setting, the forward-backward computation took approximately **771 ms per iteration**, while the rollback step required only **2.01 ms**. \n\nThis overhead is therefore negligible relative to the total end-to-end training time. We will clarify this point in the final version and include profiling results to support our claim.\n\n**Q2 - The concept of 0-bit transmission seems intriguing to me as well as bit confusing. Can the authors offer more explanation on why DUO performs the way it does in this setting? What brings the robustness?**\n\nIn the full **DUO** setting, we maintain two complementary gradient communication channels, as shown in Algorithm 1 (Lines 5 and 12):  \n1) **Timely but compressed communication**  \n2) **Accurate but one‑step‑stale asynchronous communication**\n\nWhen both channels are active (e.g., compressing the first gradient stream to 4 bits), the model receives **fresh gradient information every step** and an **exact correction one step later**. Resulting in final accuracy that nearly matches the full-precision baseline (as shown in Table 2).\n\nHowever, if the cluster’s network is so bandwidth-constrained that even 4‑bit communication becomes a bottleneck, the compressed communication channel can be disabled—resulting in the **“0-bit” transmission**. In this variant, DUO relies solely on the delayed but accurate gradient updates. This still leads to **acceptable convergence**, although typically with a **slight drop in accuracy** compared to the full DUO setup.\n\nAs discussed in **Appendix D**, the optimal balance between compressed and accurate channels depends on factors such as **model size, available bandwidth, sequence length**, and the **specific compressor used**. Because no single setting is universally best, DUO is designed to expose both communication channels, allowing practitioners to **tune the trade-off** between communication budget and final accuracy based on their system constraints.\n\n**Q3 - Have the authors thought about how DUO can be applied to other important distributed training paradigms such as Federated learning, Semi-Sync Training (e.g. DiLoCo) etc? Curious to hear any thoughts on these.**\n\nYes, **DUO** can be applied in conjunction with other training paradigms such as **Federated Learning** or **LocalSGD-style training** (e.g., **DiLoCo**).\n\nIn particular, DiLoCo and similar LocalSGD-based methods are **orthogonal to DUO**, as LocalSGD primarily targets **DDP-style training**, while DUO is designed for **Sharded Data Parallelism (SDP)**. The combination of Data Parallelism (DDP) and Sharded Data Parallelism (SDP) has already been explored in prior work such as **HSDP** [1]. In that setup, combining DUO with DiLoCo enables a two-pronged strategy:  \n- DiLoCo reduces **inter-replica communication** by lowering synchronization frequency  \n- DUO reduces **intra-replica communication** through error-compensated compression\n\nTo elaborate: LocalSGD-style methods aim to reduce communication by allowing several local update steps before synchronization. These typically assume that each worker holds the entire model (or identical copies), which differs fundamentally from SDP, where model parameters and optimizer states are **sharded across workers**, and communication relies on **ReduceScatter/AllGather**.\n\nHowever, with the right system design—such as that used in HSDP—**LocalSGD and DUO can be combined effectively**, addressing distinct communication bottlenecks without conflict. \n\n[1] HSDP: Accelerating Large-scale Model Training via Efficient Sharded Data Parallelism\n\n**Q4 - Are there any specific types of models or training scenarios where DUO might not be as effective, or where its benefits might be less pronounced? Have the authors envisioned such setups?**\n\nWe have not yet conducted a broad investigation across different model architectures, but we acknowledge that **DUO’s effectiveness depends on the system's ability to fully hide its extra full‑precision ReduceScatter within the compute window**.\n\nIn particular, DUO may be **less effective** in scenarios where **computation is short relative to communication**, such as:\n- **Very short sequence lengths**\n- **Small gradient accumulation steps**\n- **Severely bandwidth-constrained environments**\n\nIn such cases, there may not be sufficient computation time to overlap the added communication, and DUO’s traffic may become visible, diminishing its efficiency. These trade-offs are analyzed in detail in **Appendix D**, which maps out the regimes where overlap begins to break down.\n\nIn practice, a straightforward mitigation is to **increase the gradient accumulation step size**, which lengthens the compute window and restores DUO’s ability to overlap and reclaim bandwidth.\n\nWe thank the reviewer for raising this point and refer to **Appendix D** for further discussion."}]

Author rebuttal:

Meta-review: To overlap compute and communication in large-scale and distributed settings, the authors have proposed to leverage idle synchronization intervals to mask communication of high-precision gradients and compensate for the performance drop due to compression. 

The paper was thoroughly discussed by our reviewers and has mixed reviews. Most reviewers are already happy with the paper and rebuttal. Reviewer Gttj is mainly concerned regarding the novelty of making communication delays and missing major compression baselines.

The authors' rebuttal has further clarified the key novelty compared to pre‑AllGather and post‑ReduceScatter methods in terms of overlap ratio of backward pass. 

Overall, I think this is a good paper. I would suggest that authors address all revisons that are promised within the camera-ready version.
Decision: Accept (poster)