Title: DUO: No Compromise to Accuracy Degradation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gradient Compression

Communication Computation Overlapping

Efficient Training

Distributed Training
Abstract: Distributed training often suffers from high communication overhead due to large-scale gradient synchronization. Although gradient compression—particularly at 4-bit or even lower precision—significantly reduces transfer volume, it typically results in sacrifice in precision and degradation of the final model accuracy. 

In this work, we introduce DUO, a distributed training framework designed to mitigate accuracy degradation incurred by gradient compression without involving additional overhead. DUO achieves this by inserting an additional high-precision gradient synchronization step into a previously computation-only phase, so that its communication is fully hidden by computation.

We provide a comprehensive theoretical proof of convergence for DUO and validate its effectiveness through extensive pre-training experiments on GPT models. Our results indicate that DUO effectively restores accuracy when using 4-bit gradient compression, achieving performance comparable to uncompressed training. Remarkably, DUO maintains minimal accuracy degradation even under extreme compression scenarios, including 1-bit gradients or complete omission of the low-precision gradient communication step (0-bit transmission).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DUO (Double Update Overlap), a distributed training framework designed to mitigate accuracy degradation caused by gradient compression, particularly at low precision (e.g., 4-bit or 1-bit). DUO achieves this by inserting an additional high-precision gradient synchronization step into a computation-only phase, fully hiding its communication overhead. The framework leverages a fast-slow reduction algorithm that caches full precision gradients for a later high-precision update. The main advantage is that such a new algorithm is amenable to communication overlapping. \n\nThe authors also incorporate several interesting memory optimization strategies (Bucket-Wise Offloading, Finer Overlapping and GPU Memory Bridge). Theoretical convergence guarantees are provided followed by extensive experimental validation on LLM models, showing that DUO restores accuracy comparable to uncompressed training even under extreme compression (1-bit and 0-bit gradients), while maintaining comparable throughput and memory usage to state-of-the-art methods like SDP4Bit.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nOverall, I feel this paper is a strong empirical contribution, solving a key challenge in distributed training.\nBelow I highlight a few of them:\n- The paper tackles the key challenge in memory-efficient (low-bit) LLM training these days - which is the significant drop in accuracy. This has been a long-standing (and largely unsolved) bottleneck. Techniques  provided in this paper seem very beneficial to me to tackling these challenges..\n- I found the core idea of using a high-precision \"slow\" gradient update alongside a low-precision \"fast\" update, and cleverly overlapping the slow gradient communication innovative.\n- The experimental results presented are strong. DUO demonstrates impressive accuracy restoration across various GPT model sizes and extreme compression ratios (including 0-bit gradients), almost matching full-precision baselines. This is a very compelling result.\n- The authors further optimize their solutions by incorporating more practical systems optimization solutions such: Bucket-Wise Offloading, Finer Overlapping, GPU Memory Bridge - these are great contributions that are well-explained and address practical implementation challenges.\n- I like the fact that the authors use Megatron-LM as the reference open-source integration. This makes their approach well tested and can increases their adoption potentially.\n\nWeaknesses:\n- While effective, the DUO framework, particularly the Fast-Slow Reduction algorithm with its optimizer rollback and reset mechanism (Line 14-15 of Algorithm 1) appear quite intricate. This complexity might pose a challenge for widespread adoption or implementation outside of highly specialized teams. Wondering if the authors have thought about this? Are there other ways this could be simplified?\n- There has been a major pillar of work in compressed communication for distributed learning which focusses on Error Feedback mechanisms. While the paper mentions error feedback (EF) methods and shows SDP4Bit-G1-EF failing to converge, a more in-depth comparison or discussion on why DUO succeeds where EF struggles, especially for LLMs under aggressive compression, could strengthen the argument.\n- While extensive experiments on GPT models are valuable since dense models form the bulk of LLMs used these days, a brief discussion on the potential applicability or challenges of DUO to other deep learning architectures (e.g. sparse MoE type models) could be beneficial. Not sure if this has been discussed in the paper and I missed it.\n\nQUESTIONS:\nFew quick questions to the authors:\n- Could the authors provide more insight into the computational cost of the optimizer rollback and reset mechanism (Line 14 of Algorithm 1) and if it contributes to any minor overhead not captured in the E2E throughput or experiment descriptions?\n\n- The concept of 0-bit transmission seems intriguing to me as well as bit confusing. Can the authors offer more explanation on why DUO performs the way it does in this setting? What brings the robustness? \n\n- Have the authors thought about how DUO can be applied to other important distributed training paradigms such as Federated learning, Semi-Sync Training (e.g. DiLoCo) etc? Curious to hear any thoughts on these.\n\n- Are there any specific types of models or training scenarios where DUO might not be as effective, or where its benefits might be less pronounced? Have the authors envisioned such setups?\n\nLIMITATIONS:\nThe paper only provides a novel algorithm, doesn't release any dataset or model. Therefore I think this question does not fully apply here. The authors have also added this justification.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a distributed training framework that alleviates the accuracy loss caused by aggressive gradient compression. By combining AllGather and ReduceScatter and exploiting the idle periods between synchronization steps to overlap communication of additional high-precision gradients, the accuracy degradation due to gradient compression is effectively reduced. Convergence of the proposed algorithm is established, and experiments on GPT models (125M-18B parameters) pretrained on The Pile validate its effectiveness. Overall, the topic is interesting and timely.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Distributed pre-training of large language models is of paramount interest nowadays, and the compensation for the degradation in accuracy due to gradient compression is an important research direction.\n\n2. Convergence of the proposed algorithm is established from a theoretical perspective, and experiments on various sizes of GPT models are conducted, which are quite comprehensive.\n\n3. Overall, the paper is well-written.\n\nWeaknesses: \n\nI do not identify major weaknesses in this paper. That being said, the proposed strategy depends on having idle compute time between communication phases. In settings with a few micro-batches, there would be little idle time to hide the extra full-precision ReduceScatter. It would be helpful to discuss how DUO performs when accumulation is limited in the main text.\n\nQUESTIONS:\nThe framework requires saving full-precision gradients, copying them out-of-band (GPU→CPU) via a GPU memory bridge, and scheduling asynchronous reductions, which seems to be fairly complex. Could the authors comment on whether it can be readily integrated into existing codebases?\n\nLIMITATIONS:\nThe authors have discussed some limitations in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DUO (Double Update Overlap), a communication-efficient distributed training framework designed to mitigate the accuracy degradation caused by gradient compression in LLM training. The key idea is leveraging otherwise idle communication time to incorporate an additional high-precision gradient communication phase that overlaps with computation. The authors propose the Fast-Slow Algorithm, which mitigates accuracy degradation by incorporating a high-precision gradient to influence optimizer state updates without compromising throughput. This paper provides a theoretical proof of convergence for DUO and validate its effectiveness on GPT-2 model training. Results show that DUO can maintain the accuracy of 4-bit gradient compression to levels comparable with uncompressed training, even with 1-bit gradient compression it still can provide high accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper conducts experiments on GPT models with sizes up to 6.7B. DUO achieves accuracy comparable to uncompressed training, even under 1-bit gradient compression or even 0-bit. Notably, in the 1-bit setting, DUO surpasses SDP4Bit in terms of the loss curve while introducing only minimal additional memory overhead. The 0-bit setting is also a very interesting idea.\nThe paper provides a system-algorithm co-design, leveraging otherwise idle communication time to incorporate high-precision gradients to reduce the communication overhead without compromising the training accuracy.\nThe integration of algorithmic innovation (Fast-Slow Algorithm) with system optimizations (memory footprint optimization, computation overlapping) makes DUO a practical system for distributed LLM training. The paper also provides substantial implementation details, making the approach more likely to be reproducible.\nTheoretical analysis of convergence is also provided for smooth but non-convex settings.\n\nWeaknesses:\n\nThe paper focuses on gradient compression, but there is very limited comparison or discussion related to error feedback methods, theoretically and empirically.\nThe compression techniques used in the experiments are limited to simple quantization.\nThe 0-bit (no gradient transmission) is confusing, which lacks detailed explanation.\nWhile the end-to-end throughput experiments are scaled up to 18B models, the largest model size for the validation loss experiments is up to only 6.7B, which is relatively small.\n\nQUESTIONS:\nQuestions:\n\nTypically, gradient compression with biased compressors requires error feedback. How does DUO converge without error feedback? And could the authors provide a more detailed comparison and discussion between DUO and error feedback?\nIs DUO also compatible with other compressors, such as top-k sparsification?\nThe 0-bit (no gradient transmission) is confusing, could the authors provide some more detailed explanation?\nCan DUO’s 1-bit (or 0-bit) training maintain its current performance on different datasets and larger-scale models, such as those with 18B models?\n\nLIMITATIONS:\nThe authors do not provide a discussion of the limitations of their work; see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DUO (Double Update Overlap), a distributed training framework that aims to mitigate potential accuracy degradation caused by gradient compression and without incurring additional overhead. \n\nSpecifically, DUO inserts an additional high-precision gradient synchronization step during the computation phase, aiming for the communication to be fully hidden by computation.\n\nThe paper also provides theoretical proof of convergence for DUO under smoothness and bounded variance assumptions and validates its effectiveness through pre-training experiments on GPT models.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n•\tUsage of computation-communication overlap to mask communication delays\n\n•\tMaintains minimal accuracy degradation under extreme compression (e.g., 1 bit per parameter) in the tested scenarios\n\n•\tOptimizes memory usage through high-precision gradient offloading\n\n\nCons:\n\n•\tUnclear novelty - the idea of using computation phases to mask communication delays is standard practice\n\n•\tRelated work and evaluation leave more to be desired - numerous strong compression baselines and systems are not considered (e.g., see [1] and references within)\n\n• Increased Communication Overhead - While accurate transmission is achieved during the computation phase, it still increases the communication overhead; thus, the comparison to other compression techniques should take this additional budget into account.\n\n•\tThe comparison to error feedback is not convincing, and the presented results seem to go against the previous conclusion (e.g., [2]).\n\n[1] Xu, Hang, et al. \"Compressed communication for distributed deep learning: Survey and quantitative evaluation.\" (2020).\n\n[2] Karimireddy, Sai Praneeth, et al. \"Error feedback fixes signsgd and other gradient compression schemes.\" International Conference on Machine Learning. PMLR, 2019.\n\nQUESTIONS:\nCan the authors elaborate on the value of compressed versus accurate gradient transmission in DUO? Namely, why is compression even required if the 0-bit approach seems to offer near-baseline accuracy?\n\nLIMITATIONS:\nThe paper states that the proposed method is effective only when communication can be overlapped with computation.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-03"}

comments: SUMMARY:
Large scale distributed training uses two techniques to overcome communication bottlenecks: gradient compression and overlapping the computation and communication. However, aggresively employing them simulatenously (like the prior SDP4bit algorithm) can potentially lead to degradation of accuracy of the trained model. This paper modifies SDP4bit with an additional "slow" background communication step where a more high precision synchronization happens, relying on potentially unused bandwidth in the network.

STRENGTHS AND WEAKNESSES:
Strengths

- The paper is easy to read and well written.  
- The modification to SDP4bit is well motivated and the experiment setup comparing to SDP4bit is quite thorough. The improvement results are convincing as well and are performed on realistic large scale training.  

Wekanesses  

- This paer lacks comparision to any other method apart from SDP4bit. It justifies this by claiming SDP4bit is the SOTA, but I do not see evidence for that claim. Even now well establised methods like [PowerSGD](https://arxiv.org/abs/1905.13727) as used to train [DALLE2](https://proceedings.mlr.press/v139/ramesh21a.html) and [CocktailSGD](https://proceedings.mlr.press/v202/wang23t.html) have not been compared, let alone the current SOTA methods.  
- The related work discussion is also very sparse - there is no discussion of recent papers on large scale training e.g. [DiLoCo and variants](https://arxiv.org/abs/2502.12996).

QUESTIONS:
1. What are the current SOTA large scale training methods?  
2. How does DUO compare against them empirically?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-10