Title: MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Fault tolerance

Memory efficiency

Computation efficiency

Distributed training
Abstract: As distributed optimization scales to meet the demands of Large Language Model (LLM) training, hardware failures become increasingly non-negligible. Existing fault-tolerant training methods often introduce significant computational or memory overhead, demanding additional resources. To address this challenge, we propose **Me**mory- and **C**omputation- **e**fficient **F**ault-tolerant **O**ptimization (**MeCeFO**), a novel algorithm that ensures robust training with minimal overhead. When a computing node fails, MeCeFO seamlessly transfers its training task to a neighboring node while employing memory- and computation-efficient algorithmic optimizations to minimize the extra workload imposed on the neighboring node handling both tasks. MeCeFO leverages three key algorithmic designs: (i) Skip-connection, which drops the multi-head attention (MHA) module during backpropagation for memory- and computation-efficient approximation; (ii) Recomputation, which reduces activation memory in feedforward networks (FFNs); and (iii) Low-rank gradient approximation, enabling efficient estimation of FFN weight matrix gradients. Theoretically, MeCeFO matches the convergence rate of conventional distributed training, with a rate of $\mathcal{O}(1/\sqrt{nT})$, where $n$ is the data parallelism size and $T$ is the number of iterations. Empirically, MeCeFO maintains robust performance under high failure rates, incurring only a 4.18\% drop in throughput, demonstrating $5.0\times$ to $6.7\times$ greater resilience than previous SOTA approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new strategy of training LLMs (distributed optimization) in the presence of hardware failures. The strategy is called Memory- and Computation- efficient Fault-tolerant Optimization (MeCeFO) which uses some interesting algorithmic innovations, like skip connections, recomputation, and low-rank gradient approximation. They have provided a theoretical guarantee and also included experimental results on LLAMA 350M, 1B, and 7B, comparing with two baselines: (1) Bamboo (2) Oobleck. The proposed strategy shows improvements over these methods (under simulated node failures).\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\n\n1. This is a very interesting and relevant problem. They have motivated the issue of hardware failures quite well, citing practical challenges faced by Alibaba and Meta. \n\n2. The proposed strategy seems quite interesting, and combines three types of algorithmic innovations, diving deep into the nuances of matrix operations.\n\n3. Experiments are provided over several LLMs for pre-training, which is not an easy feat (also requires significant computational resources). They have compared with two baselines and demonstrate considerable improvements. Ablation studies are included in the Appendix.\n\nWEAKNESSES\n\n1. The theoretical guarantee is much appreciated, but it seems to follow from standard assumptions and convergence guarantees of momentum SGD in distributed settings. Could you highlight which steps of the proof are altered under this scheme? Is it only the approximate gradient step? Could you highlight if and how the theoretical guarantee depends on the nature of the failure, e.g., if the system is more/less failure-prone, shouldn't it affect the theoretical guarantee?\n\n2. It is not very clear if and how the system failure is modelled into the mathematical analysis. Do you assume a particular statistical model for the failure? In the experiments, a constant frequency failure model has been considered. \n\nPrior works in fault-tolerant computing have also considered exponential runtimes, motivated by the straggling effect (causes node delays which could also be an interesting motivational use-case for this paper). Some of these references could be included in the motivation/related works:\n\n[1] Tail at Scale: https://research.google/pubs/the-tail-at-scale/\n\n[2] Using Straggler Replication to Reduce Latency in Large-scale Parallel Computing: https://dl.acm.org/doi/abs/10.1145/2847220.2847223\n\n[3] Short-Dot: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products : https://proceedings.neurips.cc/paper_files/paper/2016/file/aace49c7d80767cffec0e513ae886df0-Paper.pdf\n\n3. I guess the hardware failures are simulated - under a constant frequency failure model. It would be interesting future work to study other kind of failures, e.g., by a Poisson process. \n\n4. Curious if it possible to see any kind of real failures rather than simulated ones, either from delays (straggling) or failures? Could you comment on whether the clusters were dedicatedly set aside during these experiments, or were other jobs also being scheduled on them during this time (potentially leading to natural delays/failures)?\n\n5. Prior works in fault-tolerant computing have studied alternate approaches [4,5] to perform reliable training of machine learning models under stragglers/hardware failures - by using replication or error-correction to have extra computations so that one can proceed even if some nodes fail/straggle. This direction of research would be a relevant complementary approach for the proposed technique [4,5], and could be discussed in related works if not compared with.\n\n[4] CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors: https://arxiv.org/abs/1903.01042\n\n[5] Gradient Coding: Avoiding Stragglers in Distributed Learning: https://proceedings.mlr.press/v70/tandon17a/tandon17a.pdf\n\nQUESTIONS:\nCould you highlight which steps of the proof are altered under this scheme? Is it only the approximate gradient step? \n\nCould you highlight if and how the theoretical guarantee depends on the nature of the failure, e.g., if the system is more/less failure-prone, shouldn't it affect the theoretical guarantee?\n\nLIMITATIONS:\nOne sentence on limitations is included. But, this could be expanded to include some of the points mentioned in Weakness.\n\n-- Other kinds of failure models beyond constant frequency\n\n-- Simulated vs Real failures/delays\n\n-- Other kinds of approaches in fault-tolerant computing that are based on replication/ECCs\n\n-- Theoretical guarantees", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposed a novel training algorithm MeCoFO designed to improve the fault tolerance of LLM training while minimizing additional memory and computation overhead.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. This paper targets the important problem of hardware failures in large-scale LLM training. 
2. MeCoFO shows impressively low overhead.
3. Instead of traditionally recovering through checkpoint etc. MeCeFO utilized the noise robust nature of SGD.
4. The author gives theoretically proof that MeCeFO achieves a convergence rate of $O(1/\sqrt{nT})$ under mild assumptions, demonstrating the robustness of MeCeFO algorithm.

Weakness:
1. The author mentioned that "MeCeFO is primarily tailored for specific hybrid parallel strategies and transformer-based architectures" which leads to a concern if this algorithm functions on other parallel strategies.
2. They also note that they haven't explores full system-level optimizations in combination with their method. I appreciate this transparency, and don't see any glaring additional limitations.

QUESTIONS:
1. One claim is that MeCeFO avoids having to reserve half the GPU memory for potential failures. Can the authors provide more insight or data on memory usage?
2. The experiments simulate up to a high frequency failure scenario (a failure every 30 minutes, with 2-hour recovery). If failures were even more frequent or recovery slower, how would that affect training?
3. It would be great to see MeCeFO applied to other training regimes, such as fine-tuning or different model types, just to confirm its broad utility.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new fault tolerance optimization technique for LLM training named MeCeFO. This approach effectively handle the failed node recomputation sceanrios by employing 1) selective skip reconnection 2) activation recomputation 3) low rank gradient approximation to reduce the computational and memory overhead when neighbour is need to recompute the result for failed node. The paper supports this approach by providing convergence analysis and empirical evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nstrength\n\n- Novelty. The author presents a novel method for resource-efficient recomputation in fault-tolerant training, based on the observation of the impact of placing skip connections at the MHA component and utilizing approximation techniques to reduce gradient computation complexity. This approximation technique is novel for weighing the tradeoff between accuracy and computational complexity.\n- Robust Evaluation and Convergence Analysis. The paper provides a robust evaluation with different failure settings and various model size of pre-training to demonstrate the effectiveness of their approach. They also provide a convergence analysis for the proposed approach, showing that it achieves convergence rates comparable to those of SGD.\n- Well-motivated optimization. The paper’s selection of techniques to minimize computational and memory overhead is well-reasoned and provides a good middle ground for the fault-tolerant theme they identified - approximation is sometimes good enough, and exact retracing of computation might not be necessary. I believe that in the paper, the approximation theme is well-motivated and effectively utilized.\n\nweakness\n\n- The author discusses hybrid parallelism, focusing on addressing the failure node through DP group recomputation. However, the TP group is likely the ‘minimal group for nodes’ under the training strategy, i.e., nodes → TP group → PP group → DP group. In this case, a failed node would be present in a TP group. If I understand the proposed approach correctly, the entire TP group / PP group needs to be recomputed as they would be counted as one ‘DP shard’ of the DP group, which results in recomputation of ‘healthy node’. Could the author discuss what would happen in this case, specifically whether the healthy node needs recomputation or can cache the result, and whether their approach could be directly applied to the TP group?\n- In the empirical evaluation, the node recovery time seems to be a bit confusing - could the author explain the rationale for choosing node recovery time that is up to an hour scale, and the recovery time differs in each setting (rather than, for example, a constant recovery time of say, 2h). In addition, based on previous literature, I wonder if the low frequency setting reporting in the evaluation is not representative, and shorter failure period results (i.e., 5 minutes, 15 minutes failure interval) should also be considered.\n- In practice, different failure causes likely result in different failure recovery times and frequencies - some might simply be a flip of a network connection, whereas others require hardware replacement. The uniform failure rate/failure mode assumed in the evaluation is an idealized situation considering what would happen at a real, large-scale LLM pre-training setting. In addition, sometimes due to the failure cause(e.g, failed network switch), more than one node would fail with a locality. Could the author discuss how their approach would be adapted to these various failure modes and failure rates, and how the worst-case and/best-case scenarios may affect this approach?\n\nQUESTIONS:\n- In line 236, I believe the author justified assumption 3 through empirical observation only on the error rate. However, the empirical observation is only based on a simple observation of llama 1B pre-training. Could the author elaborate a bit more on whether this observation is enough / further ablation on a larger model / more setting is required to justify this assumption fully?\n- By choosing the NDB strategy among the DP group, the author assumed that the failure domain should be constrained within one DP group, ensuring that there would always be a node in the same DP group to recover the training progress. I wonder what would be the case if the failure domain is the entire DP group (i.e., if one switch corresponding to that DP group failed). Could the author discuss the impact of the failure domain in more detail?\n- The evaluation and discussion primarily focused on llama-like dense architecture. I am curious about whether these techniques can be applied to MoE models, and if so, whether the sparsity in MoE models can facilitate additional resource savings and recomputation. Additionally, I am interested in determining the optimal placement of the router computation within the failure domain. Also, if the skip connection is suitable for MHA only or is it good on MLA, etc.\n\nLIMITATIONS:\nThe author adequately addresses most of the work's limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MeCeFO, a novel fault-tolerant optimization algorithm designed to ensure robust training with minimal overhead. When a computing node fails, MeCeFO efficiently transfers the workload to a neighboring node while applying memory- and computation-saving techniques. It incorporates three key algorithmic innovations: skip-connection to approximate attention during backpropagation, recomputation to reduce activation memory, and low-rank gradient approximation for efficient gradient estimation. Theoretically, MeCeFO achieves convergence comparable to conventional distributed training, while empirically demonstrating strong resilience under frequent failure conditions.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper proposes MeCeFO, a novel fault-tolerant optimization algorithm that enhances efficiency in distributed training.\n2. This paper provides theoretical guarantees showing MeCeFO achieves convergence rates comparable to standard distributed SGD under mild assumptions.\n3. This paper demonstrates MeCeFO’s superior resilience to frequent failures, maintaining high training throughput and robust model performance.\n\n\nWeakness:\n1. This paper states that \"The convergence depends on the quality of final weights, not the precise trajectory,\" which is not validated by the experiments. The experiments verify \"the quality of final weights\" only through final perplexity, without evaluating downstream transfer capabilities (e.g., zero-shot or few-shot task performance), training stability, or generative quality metrics (such as BLEU).\n\n2. A key implicit assumption behind the MeCeFO mechanism is that failures are sparse and “uniformly random,” with roughly equal failure probabilities across nodes and the ability for neighbors to rotate load. In real systems, failures tend to cluster on certain nodes (i.e., non-i.i.d.), causing the NDB to repeatedly fallback on the same “victim” nodes. The paper does not analyze how this asymmetric load might induce training bias or failure. This raises concerns about the robustness and generalizability of the proposed method.\n\n3. Section 4 provides an upper-bound theoretical analysis on the convergence impact of skip/backward approximation, but it lacks worst-case analysis and does not explore whether these errors could be amplified during early training stages or critical layers. For fault tolerance mechanisms, a key question is whether the \"worst-case deviation is controllable.\" Currently, the theoretical discussion focuses mainly on average-case scenarios, which cannot guarantee system stability.\n\nQUESTIONS:\n1. This paper states that \"The convergence depends on the quality of final weights, not the precise trajectory.\" Can you validate it by experiments?\n\n2. In real systems, failures tend to cluster on certain nodes (i.e., non-i.i.d.), causing the NDB to repeatedly fallback on the same “victim” nodes. Please analyze how this asymmetric load might induce training bias or failure. \n\n3. Section 4 provides an upper-bound theoretical analysis on the convergence impact of skip/backward approximation. Can you provide worst-case analysis and explore whether these errors could be amplified during early training stages or critical layers?\n\nLIMITATIONS:\nNil", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-20"}