MeCeFO: Enhancing LLM Training Robustness via Fault-Tolerant Optimization
Authors: Rizhen Hu
Yutong He
Ran Yan
Mou Sun
Binhang Yuan
Kun Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Fault tolerance
Memory efficiency
Computation efficiency
Distributed training
Primary area: infrastructure
TL;DR: Propose a fault-tolerant algorithm with minimal memory and computation overhead.

Abstract: As distributed optimization scales to meet the demands of Large Language Model (LLM) training, hardware failures become increasingly non-negligible. Existing fault-tolerant training methods often introduce significant computational or memory overhead, demanding additional resources. To address this challenge, we propose **Me**mory- and **C**omputation- **e**fficient **F**ault-tolerant **O**ptimization (**MeCeFO**), a novel algorithm that ensures robust training with minimal overhead. When a computing node fails, MeCeFO seamlessly transfers its training task to a neighboring node while employing memory- and computation-efficient algorithmic optimizations to minimize the extra workload imposed on the neighboring node handling both tasks. MeCeFO leverages three key algorithmic designs: (i) Skip-connection, which drops the multi-head attention (MHA) module during backpropagation for memory- and computation-efficient approximation; (ii) Recomputation, which reduces activation memory in feedforward networks (FFNs); and (iii) Low-rank gradient approximation, enabling efficient estimation of FFN weight matrix gradients. Theoretically, MeCeFO matches the convergence rate of conventional distributed training, with a rate of $\mathcal{O}(1/\sqrt{nT})$, where $n$ is the data parallelism size and $T$ is the number of iterations. Empirically, MeCeFO maintains robust performance under high failure rates, incurring only a 4.18\% drop in throughput, demonstrating $5.0\times$ to $6.7\times$ greater resilience than previous SOTA approaches.

MeCeFO: Enhancing LLM Training Robustness via Fault-Toleram Optimization
Rizhen Hu" Peking University rzhu250stu, pku, edu. on
Yutong He Peking University yutonghe@pku. edu.
Ran Yan HKUST
ryanaf@connees ust
Mou Sun
Binhang Yuan HKUST
Kun Yuan
Zhejiang Lab 123sssmmm@@mail.co com
Peking University
biyuan@ust hk
kunyuan@pku. edu, cn
Abstract
As distributed optimization scales to meet the demands of Large Language Model (LLM) training, hardware failures become increasingly non-negligible. Existing fault-tolerant training methods often introduce significant computational or memory overhead, demanding additional resources. To address this challenge, we propose Memory- and Computation- efficient Fault-tolerant Optimization (MeCeFO), a novel algorithm that ensures robust training with minimal overhead. When com- puting node fails, MeCeFO seamlessly transfers its training task to neighboring node while employing memory- and computation computation-efficient efficient algorithmic optimiza- tions to minimize the extra workload imposed on the neighboring node handling both tasks. McCeFO leverages three key algorithmic designs: (i) Skip- Skip-connection, which drops the multi-head attention (MHA) module during backpropagation for memory- and computation-effiiin reduces activation memory in feedforward networks (FFNs): and (iii) Low-rank gradient approximation, enabling efficient estimation of FFN weight matrix gra- dients. Theoretically, MeCeFO matches the convergence rate of conventional distributed training, with rate of O(IVirT)) where is the data parallelism size and T is the number of iterations. Empirically, McCeFO maintains robust performance under high failure rates, incurring only 4.18% drop in throughput, demonstrating 5.0x to 6.7x greater resilience than previous SOTA approaches. Codes available at htttpps: //t:t/giiit http://gttuuu.ccoooeeeeelloooo// com/ppkumelon/MeeeFO
approximation; (ii) Recomputation, which
1 Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, including machine translation, reasoning, planning, coding, elc., driving their widespread adoption. According to the Chinchilla scaling law [22), model performance scales with the number of model parameters, training tokens, and iterations, necessitating larger model architectures, longer training durations, and, crucially, massive distributed compute resources. For example, Meta's LLaMA 405B I7 was trained on 16,000 H100 GPUs for 54 days. Training clusters are continually scaling up. 100,000 GPUs. At this scale, hardware failures
with leading frontier AI efforts approaching become aisitaabll-lbblll reports downtime percentage of 19. for handling failures [2] and Meta reports frequency of hours per failure average due confirmed hardware issues
The
Equual contribution.
Cooreespondigg author. Kun Yuan is also affiliated with Al for Science Institute, Beijing, China.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

potential hardware failures lead to critical challenges for distributed training, degrading GPU utility and training throughput. Such failures present critical challenges for distributed training, reducing GPU utilization and training throughpur. The frequency of failures tends to increase with cluster size, making them especially problematic large-scale systems. This has led to the rise of robust training, which employs fault-tolerant techniques to mitigate the impact of hardware failures.
Existing fault-tolerant approaches in large-scale training focus on system optimizations, including checkpointing [5] 40) 36 56, 63 2) rescheduling 2 [25]. and redundant computing [50]. Check-
pointing methods periodically save training states to allow recovery from the most recent checkpoint after a failure. However, beyond the additional memory and computation overhead, replacing failed devices with spares and reloading checkpoints is coneeconsuming peessummnn-poiinn a significant challenge in sure--scale training clusters, where failure frequency is high. Rescheduling techniques aim to avoid re- covery delays by dynamically reassigning training tasks based on available resources. Nevertheless, a reduced pool of devices still leads to degraded throughput. Redundant computing improves robustness by replicating tasks across multiple devices, but this significantly lowers effective GPU utilization, even when no failures occur. Overall, existing fault-tolerant methods suffer from inefficiencies, the redundancies required for robustness can substantially degrade training throughput.
It is important to note that the fault-tolerant approaches discussed above are fundamentally algorithm- agnostic; that is, their primary goal is faithfully execute a given training algorithm step-by-step. irrespective of any encountered failures. However, we contend that the ultimate objective of model training is not necessarily replicate an exact sequence of computations but rather obtain model parameters that generalize effectively on the intended tasks. Consequently, intermediate results- and even the final outcomes- need not strictly align with those of a fault-free execution scenario. Instead, the critical factor is the performance of the trained model itself. Notably, optimization methods such as stochastic gradient descent (SGD) and Adam (27] inherently exhibit robustness to variations and noise in gradient computations, further supporting this viewpoint. This observation implies that rigid adherence to the original training trajectory might be overly restrictive. Relaxing this constraint could potentially enhance the overall efficiency of fault-tolerant training. This motivates key question:
Can we design fault-tolerant optimization algorithms that are more memory- and compute- efficiens by strategically sacrificing computation precision, while still achieving strong model performance?
In response to this question, we propose McCeFO, fault-tolerant optimization algorithm for train- ing transformer-based LLMs that reduces the overhead of fault tolerance through memory- and computation-efffcient strategies. Specifically, MeCeFO adopts a neighbor-do-both (NDB) strategy, in which a failed node's 's training task is handled by neighboring node, which is then responsible for executing both its own task and the failed node's. To alleviate the additional memory overhead on the neighbor node, we introduce skip-connection technique for the multi-head attention (MHA) module and recomputation strategy for the feedforward network (FFNN. The associated computation overhead is addressed through the combination of skip-connections and a low-rank gradient approxi- mation technique, which compensates for the extra cost introduced by recomputation. Theoretically, we establish a convergence rate of (1/VVTT)) for McCeFO, matching that of standard distributed stochastic gradient descent (SGD). Empirically, our experiments demonstrate that MeCeFO incurs only a 4.18% drop in throughput while maintaining comparable model performance when pre- trainini LLaMA-7B under high- frequrryy failures achieving 5.0x to 6.7x greater resilience than previous state-of-the methods. Our contributions include:
We propose McCeFO, a novel fault- nult-toleraan optimization algorithm with improved efficiency.
We theoretically prove that McCeFO achieves a convergence rate of O(I/VnI) under mild assumptions, matching that of standard distributed SGD.
We empirically validate MeCeFO across various settings, demonstrating its ability to sustain high training throughput and strong model performance even under frequent failures. In particular, when pre-training I.LMMA-BB under high-frequency failure scenarios, MeCeFO incurs only a 4. 4.18% drop in
5.0x to 6.7x greater resilience than previous state-of-ther off-the-rt methods.

---

2 Preliminaries and related Works
Transformer models. This paper primarily focuses on decoder-only transformer models 153 which
are widely adopted in modern large language model designs, including LLaMA [5] 52 GPT
42 43
DeepSeek 4 30 BI, etc. In general, transformer models consist of multiple transformer
blocks, each containing a multi-head attention (MHA) layer followed by a feedforward network (FFN), with both layers equipped with normalization and residual connections (20]. A typical MHA module includes four weight matrices: Wq. Wi. Wv. and W.. The FFN module is usually a shallow
MLP; for example, in LLaMA models, the FFN comprises three weight matrices: Wgate W up- and
Wdwwn. Popular choices of normalization include LayerNorm [3 and RMSNorm [61]. Positional information is typically encoded using positional encodings such as RoPE [48).
Hybrid parallelism. There are several parallel computing patterns used in efficient distributed training, such as data parallelism (DP), pipeline parallelism (PP), tensor parallelism (TP), and sequence parallelism (SP), among others. In this work, we focus primarily on the hybrid parallelism setting that combines data parallelism and pipeline parallelism a popular strategy for training large-scale deep learning models [66 37 38 47). Specifically, devices are first grouped into different DP ranks, each responsible for processing different subset of the training data. Within each DP rank, the devices are further organized into a pipeline to handle different layers of the model.
Memory consumption. The memory footprint during training consists of four primary components: (i) model weights, (ii) weight gradients, (ii) optimizer states, and (iv) activations. To maximize hardware utilization and training throughput, practitioners often select large mini-batch sizes, which can cause activations to dominate the overall memory consumption, making a critical bottleneck in large-scale training, particularly for deep networks with nigh-dimensional intermediate features.
Computation consumption. The majority of neural network training computation occurs in dense matrix multiplications within linear layers. Each linear transformation y = W æ involves three key operations: (i) forward propagation (Fprop), (ii) weight gradient computation (Wgrad), and (iii) activation gradient computation (Dgrad) during backpropagation. These operations typically require cquivalent amount of computation, resulting in a 1:2 ratio of forward backward pass computation.
Fault-tolerant algorithms. Most existing approaches ensure training robustness through check- pointing. For example, [4)] restarts from checkpoints when adjusting resource configurations; [15] resumes training from quantized checkpoints; (36) reduces checkpointing overhead by algorithmi- cally tuning the checkpoint frequency and leveraging pipelining: [56] accelerates checkpointing using NVMe optimizations and write parallelism: [63] restarts from the last saved checkpoint in response to hardware failures or loss divergences; and 2 introduces job morphing dynamically reconfigure training jobs after restarting from checkpoints with the remaining resources. To avoid the recovery overhead of checkpointing, researchers have also proposed fault-tolerant approaches that do not rely on it. Bamboo [50] employs redundant computation to ensure information availability during failures, while Oobleck (25) precomputes pipeline templates and dynamically adjusts them in response to failures. To the best of our knowledge, this work proposes the first fault-tolerant optimization algorithm that integrates memory- and computation-effin -efficient training techniques to improve efficiency.
Efficient training. As memory consumption becomes major bottleneck in training large- models, researchers have developed training algorithms that reduce memory usage. Adapter- based fine-tune only parameter-efficient additional modules. LoRA (24 and its
methods such 23
4T
variants 332
62
34
reparameterize dense weight matrices using low-rank adapters to reduce
memory costs. 39
proposes randomly activating different layers during training, while [19 combines
low-rank and sparse structures for further memory savings. GaLore [64] and its variants [21 8
45 65
apply low-rank gradient approximations to reduce the memory footprint of optimizer states. 26 60
351 compress activations to save memory. Most memory-effiiient training methods can also enhance training throughput by enabling larger batch sizes (65) Another line of work improves throughput by directly reducing the number of floating -piint operations (FLOPs). For example, DropBP [57] randomly skips connections during backpropagation, while 1 33 reduce computational cost
through approximate matrix multiplication. A major drawback of these memory- and computation- efficient training algorithms that the resulting models often exhibit a noticeable performance gap compared to standard training. In contrast, McCeFO applies efficiency techniques only locally and
3

---

seletively-sspecffcalll when handling failures-allowing allowing it to benefit from efficiency gains without compromising model performance.
Method
Current fault-tolerant methods are suboptimal for deep learning as they prioritize exact step-by-step computation requirement inherited from general distributed computing that is unnecessarily stringent for model training. Unlike rigorous distributed computing tasks, deep learning operates within the framework of distributed stochastic optimization:
1 = T2 Σ f((w), where fi(w) == EgwD, (F
min WERd
f(w)
Here, w collects all the weight parameters in the model, n denotes the number of data-parallel (DP) ranks, DD the local data distribution the i-th rank, and F the per-sample loss function. The key insight is that the training objective targets expected loss minimization rather than exact intermediate computations. Crucially: (i) Optimizer robustness: First-order stochastic optimizers (e.g., SGD, Adam) are inherently tolerant to gradient noise; (ii) Path independence: The convergence depends on the quality of final weights, not the precise trajectory. This reveals fundamental redundancy main- taining exact gradient information for training robustness. Leveraging these observations, MeCeFO strategically relaxes exact computation requirements and incorporates memory- and computation- efficient mechanisms to achieve: (i) reduced memory overhead during fault recovery; (ii) lower computational redundancy; and (iii) maintained convergence guarantees.
3.1 McCeFO overview
Neighbor-do-both strategy. Upon detecting a hardware failure, MeCeFO initiates an efficient failover protocol which the neighbor node within the same DP rank assumes responsibility for both its original workload and the failed node's computational tasks. In the following, we use failed node and neighbor node to refer to the node occurring hardware failures and the node that takes charge of the failed node's 's workload, respectively. Although the failed node's memory (including model weights and optimizer states) becomes inaccessible to the rest of the training network, this information is not entirely lost due to the inherent memory redundancy in data parallelism, which maintains identical backups across other DP ranks. In MeCeFO, the neighbor node directly retrieves the required information -nnclddin the failed node's model weights and optimizer states from the corresponding device responsible for the same layers another DP group.
However, naively implementing the neighbor-do-both (NDB) mechanism significantly degrades training efficiency. The neighbor node must maintain doubled memory footprint and computational load during failures, creating two key bottlenecks: (i) Memory inefficiency: Each GPU must reserve half of its total memory capacity to accommodate the additional layers during failure scenarios. This not only reduces memory utilization but also forces smaller macro-batch sizes to avoid oft-of-memory (OOM) errors, directly impacting computational throughput; (ii) Pipeline imbalance: Processing doubled computational loads increases execution time proportionally. This creates pipeline bubbles that propagate within the data parallel (DP) rank, forcing other devices to remain idle while waiting for the overloaded node to complete its computations.
To address these challenges, we develop specialized computation- and memory-efficien techniques. Our solution incorporates three key innovations: (i) Skip-connection: We reduce both computational load (Wgrad and Dgrad) and activation memory requirements in the MHA modules through strate- gic skip-connections; (ii) Selective activation recomputation: For feed-forward network (FFN) modules, we implement an efficient recomputation strategy that maintains only critical activation checkpoints, dramatically reducing memory demands; (iii) Low-rank gradient approximation: We introduce a novel approximation technique for FFN weight gradients that significantly decreases com- putational complexity of the Wgrad operations, effectively compensating for the overhead introduced by recomputation. Alg. provides a general view the proposed McCeFO algorithm.
Remark. MeCeFO is designed as complementary component within broader fault-tolerant frame- works, contributing to the construction of more resilient large-scale training systems. Rather than endorsing a specific solution, highlight representative examples to illustrate the feasibility and

---

Algorithm MeCeFO Algorithm
Input: Initial model weights w(() = (We,# 12
(e (akaaovoogattt)uuuddooo projection matrix update
frequency T.
Output: Final model weights w (T)
1: Initialize NDB steps f 0 for rank layer &
2: for global step / 0 to 7 - do
3: for DP ranks
1 n do in parallel
4: Check node availability and rearrange tasks according to the NDB strategy;
5: for layer in failed nodes do
on failure
6: Neighbor node fetches w and optimizer states from other DP ranks;
7:
Reset local step 1,,e 0: end for
8:
9:
for layer in recovered nodes do
on recovery
Original node fetches w and optimizer states from the neighbor node: end for
10:
11:
12:
for layer l = to L do
13:
Execute forward pass MeCeFO_Forward
forward pass
14:
end for
15:
for layer é = L to 1 do
Execute backward pass MeCeFO_ MeCkeoo_ackkwrdd W/ 4.));
b backward pass
17:
Get averaged gradients ( according
M
b gradient averaging
Execute optimizer step to get w according to gradient G
optimizer step
19:
end for
end for
21:
end for
Way Pipeline Parallelism
Way Pipeline Parallelism
Way Data Parllellsm
Way Data Parallelism hism
Figure I: Overview the MeCeFO framework. During both forward (left) and backward (right) propagation, the workload of a failed node is offloaded to a neighboring node within the same data parallel (DP) group.
flexibility of integration: MeCeFO effectively addresses isolated node failures; 18 has proposed hierarchical detection mechanisms that target switch-level and interconnect failures distributed environments; (58) has considered broader challenges, such as node freezing, communication dis- ruptions, and software errors. By combining McCeFO with such system-level techniques, one can construct more comprehensive and reliable fault-tolerant infrastructure for distributed training.
3.2 Key technique I: skip-connection
MeCeFO's skip- conneecontttoon technique draws inspiration from DropBP [57] While DropBP randomly skips connections in both the MHA and FFN modules with varying probabilities, MeCeFO employs deterministic strategy that consistently skips the MHA module connections and maintains connectivity for the FFN module, illustrated in Fig. 2 This design choice stems from our empirical observation (Fig. 33 that skipping MHA connections introduces significantly less training disruption than alterna- tive choices. When neighbor nodes skip MHA in the backward pass, gradient contributions come
5

---

Forward Propagation
Layer Norm
MHA
Layer Norm
FFN
541 1FM Dioper
No Drop
Propagation
Layer
MHA
Layer Norm
FFN
Drop
Projection
Toising Korations (11000)
Figure 2: The skip-connection technique MIIA layers. only skip the MHA's connection in the backward propagation.
Figure 3: Ablation of module skipping in LLaMA- 130M pre-training.
exclusively from unaffected DP ranks. Formally, the averaged gradients are computed as:
G
Σ
G
REN,
where # {q. k. v, v,o3. 2,...... nn represents active DP ranks where the device respon- sible for training weight matrix We in the é-th layer is neither failed nor serving as a neighbor node of failed one, and Gi.,+ represents the stochastic gradient regarding We, computed by DP i. When all DP ranks are unaffected, i.e., Ne. = (1,2, n). ① reduces to the standard format:
G Σ G
Memory efficiency. The skip-connection technique eliminates the need for neighbor nodes to store activations in MHA modules, significantly reducing memory overhead when handling doubled tasks.
Computation efficiency. The skip-connection design eliminates the need for neighbor nodes to compute Wgrra and Dgrad in MHA modules, significantly reducing the computation costs.
3.3 Key technique II: selective activation recomputation
Unlike MHA modules, applying kxip-connections FFN modules proves problematic for two key reasons: (i) FEN-skip-connections introduce substantial approximation errors input activation gradients, severely degrading backpropagation quality; (ii) In failure-prone scenarios, reduced participation of DP ranks for FFN weight updates exacerbates data heterogeneity issues, leading non-negligible gradient bias. To maintain training stability while preserving memory efficiency, we instead employ activation recomputation for FFN modules. Specifically, neighbor nodes only maintain the input each FFN modules and recompute all other activations during back propagation.
Memory efficiency. The recomputation technique eliminates the need for neighbor nodes to store intermediate activations in FFN modules, significantly reducing the memory overhead.
Computation overhead. The recomputation technique introduces additional computational costs for neighbor nodes. Specifically, each FFN module requires one additional forward pass (Rcomp). This recomputation cost equivalent to a standard Fprop operation. The total overhead amounts to approximately one third of the baseline FFN computation cost in normal training scenarios.
3.4
Key technique III: low-rank gradient approximation
Although the memory cost for both the MHA module and the FFN module has been significantly reduced by techniques and II, only MHA's computation cost has been reduced, and FFN's com- putation has been increased by 1/3. To mitigate this issue, we propose the following low-rank gradient approximation technique to compensate for the recomputation overhead. Specifically, for linear layer y Wa w in the FFN module with W m=x, we conduct singular value decom- position (SVD) of W yielding W = UEVT Let V1 Vi::: r] collect the top-r right singular vectors-firs first columns of V we we have the following approximation:
Gw Gy Gyæ TVV T
& Gy TV)V
Here, Gy represents the gradient of activation y. We only recompute matrix Vi every iteration to further reduce the SVD overhead.

---

Algorithm 2 MeCeFO Forward Pass
def MeCeFO_ BaaeFFoFowwrrdWWW (W)
if node not taking doubled workload then
b standard node
2: Execute standard MHA and FFN forward pass with all activations maintained;
3: else
neighbor node
4: Execute standard MHA and FFN with only input activations to FFN maintained;
5: end if
Algorithm 3 McCeFO Backward Pass
def [acckFo_ac ( (t)
:t,,e):
1: if node no taking doubled workload then
P standard node
2: Execute standard MHA and FFN backward pass yielding gradients G
3: else
neighbor node
4: if
t 0 (mod 7) then
compute projection matrix every T iterations # E {gate, up, down);
5:
( t = U
2)
(i
6:
else
7:
#
(((1-)
i # E {gate, up, down};
reuse projection
8:
Recompute FFN activations using the maintained input activations:
Apply (2) to approximate ( # via Vi(t) # # E (gate, up, down);
9:
10: Skip the MHA connection and propagate activation gradients to previous layers;
11: Update local step fi, + t,
12:
end if
13:
end if
Memory overhead. The additional memory for V Rexr is negligible when min n).
Computation efficiency. Let b denote the batch size times sequence length. Compared with the original FLOPs 2bmn to compute Gw Gyæ applying the low-rank gradient approximation technique requires only (2brn+ + 2brm + 2rmn) FLOPs. When < min(b, m, n). the approximated Wgrad operation is computationally negligible, approximately compensating for the Rcomp overhead.
4 Convergence analysis
Assumption
and L-smoothness). We assume function f Rd + R satisfies
inf f(w) -0, and [V/((w) V s L|ww - 12|12, V w, w' Rd.
wERd
Assumption 2 (Stochastic gradient). We assume the stochastic gradient oracles satisfy
Ee-D, (w::): = f((),, and [....[I/V(uu - ff( 0 for 21, , and some a >
Assumption 3 (Gradient error). We assume that the following inequalities hold for MeCeFO's approximated gradient (!) during the optimization process 0, T 1:
g 1) - - S 5(1
(1--6) 13.
BEIDD 0 V ||² S 61 ((
where d (0, 1] and g. (t) represents the fault-free averaged stochastic gradient at step t.
Remark. Assumptions and are standard assumptions commonly used in convergence analysis.
To validate Assumption we empirically observe the relative errors through our experiments. Specifically, observe the single-batch relative error ([g gfd 1 ( and full-batch relative

---

LLaMA-1B pre-training
LLaMA-1B pre-training
ePDF
cPDF
Density
2.5
Density
0.00 0,55 9.10 0.15 0.20 8,2 0.39 0.39 0.40
044 035 0.55
0.60
0.000 0.625 0.000 0.075 0.100 8.52 8,350 8.17 8.200 0.22 0.25 0.275 0.50
Relative Error
Relative Error
Figure 4: Single-batch relative error of pre-training ILLMA-1B on the C4 dataset.
Figure 5: Full-hatch relative of pre-training ILaMA-1B on the C4 dataset.
Table I: Configuration of Different Failure Scenarios
Scenario Name
Failure Interval
Node Recovery Time
Low Frequency Failure
Every 2 hours
Every 4 hours
Medium
Frequency Failure
Every hour
Every 3 hours
High Frequency Failure
Every 0.5 hours
Every 2 hours
error R D.
( while pre-training LLaMA- 1B. As illustrated in
Fig. and these errors are consistently smaller than 0.6, justifying the application of Assumption 3
Below present the convergence results of MeCeFO.
Theorem Under Assumptions ! if momentum parameter 31 E ( (1-66(24-126), 1) and learning
rate < min(1/(2L), (8(1 3
()/((2))), MeCeFO (with momentum SGD) converges as
T T
8A
8A,
24(1 ((1--1)o'
E ( 19
(T + 1) 8(1 (T) +1) +
ôn
where A == ((w(()) inf f(w), and A m ()) (Proof is in Appendix A
Corollary 1. Under Assumptions 3 if we choose = (2L + 5(1-8))-
and B1 = 1
+ (8/2(T+1)02 a(TA+t51))
-1
MeCeFO (with momentum SGD) converges as
(V (1A 80 86/2n +1)
(1 =0
LA 1
T+1
86/2(T
matching standard distributed SGD's 's convergence rate 0
VnT ++)
5
Experimental results
In this section, we empirically evaluate McCeFO across various scenarios to assess training performance. Ablation results and additional details are provided in Appendix Cland
5.1
Experimental setup
Cluster setup. We conducted experiments on a 32-GPU cluster composed of four nodes, each with eight NVIDIA A100 GPUs. Intra-node communication leveraged NVLink (600 GB/s), and inter-node communication used InfiniBand (200 GB/s).
Baselines. our experiments, we compare McCeFO against two state-of-ther fault-tolerant training methods, Bamboo (50 and Oobleck (25). using their publicly available implementations.
Workloads. We pre-train LLaMA 511 models of various sizes on the C4 4k dataset, using different global batch sizes and training iterations for each configuration. Specifically,

---

Table 2: Throughput Performance and Degradation under Different Fault Frequencies
Throughput (tokens/s)
Throughput Drop (%)
Model
System
Fault
Low Freq.
Mid Freq.
High Freq.
Low Freq.
Mid Freq.
High Freq.
Bamboo
438.06k
428.90k
421.45k
407.22k
2.09
3.79
7.04
LLaMA-350M
Oobleck
703. 73k
674.15k
662.93k
632.40k
4.20
5.80
10.14
MeCeFO
1199.23k
1197_39K
1193.25k
1186.35k
0.15
0.50
1.07
Bamboo
153.754 75k
146.91k
144.66k
141.13k
4.45
5,91
8.21
LLaMA-IB
Ooblcck
291.05k
276.05k
268.29k
250.68k
5.16
7.82
13.87
MeCeFO
471.19K
464.79k
461.23k
457.13k
1,36
2.11
2,98
Bamboo
12.41k
11.45k
10.74k
9.82k
13.42
20.84
LLaMA-7B
Oobleck
66.95k
57.05k
51.63k
48K
14.78
22.87
28.09
MeCeFO
111.12k
108.15k
107.70k
106.47k
2.67
3.08
4.18
Table 3: Validation Perplexities of LLaMA Models Pre-trained by MeCeFO under Different Fault Frequencies
Model
No Fault
Low-frequency Fault
Medium-frequency Fault High-frequeney Fault
LLaMA-350M
18.74
18.75
18.88
19.04
LLaMA-IB
15.49
15.51
15.61
15.83
LLaMA-7B 14.92
14.97
15.04
15.16
LLaMA-350M: Trained for 6,000 iterations with global batch size of 8,192.
LLaMA-IB: Trained for 20,000 iterations with a global batch size of 4,096.
LLaMA-7B: Trained for 60,000 iterations with a global batch size 1,024.
Failure Scenario. We simulate three distinct failure scenarios, each defined by a specific failure frequency and recovery time of corresponding nodes. The detailed configurations summarized in T These scenarios impose varying levels stress on the system, from stable (low-frequenc) to moderately disrupted (medium-freede frequency), and highly volatile high-frrequncy). This design allows us to evaluate the system's 's fault tolerance and recovery behavior under different failure intensities.
5.2 Training throughput under failures
Comparisons across frameworks under fault--rree conditions and varying fault frequencies highlight the strong performance MeCeFO. The throughput of each method is summarized in Table 2
In the fault-free setting, MeCeFO maintains a throughput of 1,199.23k tokens/s with LLaMA-350M, dropping only slightly 1,186k tokens/s under high-frequency faults mere 1.07% degradation. Similar robustness is observed for LLaMA-1B (2.98% degradation) and LLaMA-7B (4.18%).
In contrast, Bamboo experiences 2.76x 13.9x throughput drop compared to MeCeFO across different settings. Due to its reliance on redundant computation, Bamboo also suffers from low throughput even under fault-free conditions. For instance, LLaMA-350M pre-training, it achieves only 438.06k tokens/s-Substantiin substantially lower than both Oobleck (703.73k tokens/s) and MeCeFO (1199.23K tokens/s). As a result, while Bamboo's relative throughput degradation under faults may appear modest, its heavy resource overhead fundamentally limits overall performance.
Oobleck, focused system-leve optimizations, exhibits significant throughput degradation fault frequency increases, ranging from 3.71 x to 28.0x worse than MeCeFO. For LLaMA-350M, the degradation reaches 10. 14% under high-frequency faults and escalates to 28.09% for LLaMA-7B.
These results support our perspective that strictly adhering to conventional optimization algorithms in fault-tolerant training can be unnecessarily restrictive. By relaxing this constraint and incorporating memory- and computation-efficiee learning techniques, is possible to significantly enhance training efficiency, highlighting that efficient fault-tolerant design goes beyond purely system-level solutions.
5.3 Training performance under failures
To evaluate MeCeFO's ' impact on training convergence, we measured the validation perplexity of LLaMA-350M, LLaMA- IB, and LLaMA-7B trained with McCeFO under different failure scenarios.

---

Table Zero-shot evaluation scores of LLaMA-IB Pre-trained by MeCeFO under Different Fault Frequencies
Fault Frequencies
BoolQ[
ARC-Easy [10]
PIQA [5]
TruthfulQA-MC2 (9
Avg.
No Fault
0.579
0.459
0.682
0.427
0.537
Low Freq.
0.594
0,455
0.674
0,451
0.544
Mid Freq.
0.571
0.446
0.678
0.425
0.530
High Freq.
0.587
0.454
0.684
0.417
0.536
Table 5: Fine-tuning Results on Pre-trained LLaMA-IB IB under Corresponding Fault Frequencies
Fault Frequencies
CoLA
STS-B
MRPC
RTE
SST2
MNLI
QNLI
QQP
Avg.
No Fault
46.93
89.21
89.12
62.61
92.36
81.82
88.61
89.83
80.06
Low Freq.
46.86
89.14
88.92
62.59
92.31
81.78
88.58
90.07
80.03
Mid Freq.
47.21
89.14
88.84
63.18
92.25
81.80
88.61
90.02
80.13
High Freq. 46.67
89.16
88.87
62.58
92.30
81.71
88.66
89.94
79.99
To further assess downstream capabilities, we evaluated LLaMA-IB models treitrained with MeCeFO on several zero-shot tasks and conducted fine-tuning experiments on the GLUE 54 benchmark under corresponding failure scenarios.
Pre-training performance. As shown in Table the increase in perplexity caused by MeCeFO's efficient training strategies under failure conditions is minimal. Under high-frequency faults, the perplexity for LLaMA-350M increases slightly from 18.74 to 19.04 (1.60%); for LLaMA-IB, from 15.49 to 15.83 (2.19%) and for LLaMA-7B, from 14.92 15.16 (1.61%) Under medium- and low- frequency fault scenarios, the increases are even smaller- less than 0.80% and 0.34%, respectively.
Zero-shot performance. As shown in Table the pre-trained LLaMA-1B models maintain robust zero-shot performance across all failure scenarios. Compared to fault-free baseline (0.537 average), the average scores are 0.544 under low-frequency faults, 0.530 under mid-frequency faults, and 0.536 under high- high-frequency faults. Notably, under low-frequency faults, MeCeFO even yields slight improvements on BoolQ (0.594 vs. 0.579) and TruthfulQA-MCS (0.451 vs. 0.427), leading to the highest overall average. These results demonstrate that McCeFO preserves, and in some cases enhances, the downstream generalization ability of the model despite frequent failures.
Fine-tuning performance. As shown in Table LLaMA-1B models pre-trained with MeCeFO under different failure scenarios achieve downstream performance on GLUE that is nearly identical the fault-free baseline. The average score of the baseline model (80.06) well preserved: 80.03 under low-freed frwfrreuency faults, 80.13 under mid- frequfrueenccy faults, and 79.99 under high- high-frequency faults. In particular, the mid-frequency fault model slightly surpasses the baseline on CoLA (47.21 vs. 46.93) and RTE (63.18 vs. 62.61), leading to the highest overall average.
These findings confirm that McCeFO effectively maintains training performance. Its ability sustain comparable perplexity metrics even under high-frequency fault conditions demonstrates robust fault tolerance without significant compromise to final model quality.
6 Conclusions and limitations
We propose MeCeFO, fault-tolerant training algorithm that achieves high efficiency through three core techniques: (i) skip-connection, (ii) selective activation recomputation, and (iii) low-rank gradient approximation. Theoretically, MeCeFO retains convergence rate of O VnT), matching that of standard distributed SGD. Empirically, MeCeFO incurs only a 4.18% throughput degradation when pre- pre-training LLaMA-7B under high-frequency failures while maintaining comparable model performance, contrast, existing SOTA methods that strictly adhere to fault-free assumptions suffer 5.0x to 6.7x greater throughput degradation. Our study has several limitations, including the use of a per-iteration failure setting, limited access to large-scale pront clusters for experiments, and the reliance of our theoretical results on Assumption which plan to address in future work.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (No. 2024YFA1012002) and National Natural Science Foundation of China (No. 124B2017, 92370121, 12301392, W2441021).
References
[1] Menachem Adelman, Kfir Levy, Ido Hakimi, and Mark Silberstein. Faster neural network training with approximate tensor operations. Advances in Neural Information Processing Systems, 34:27877-27889, 2021.
[2] Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472 487, 2022.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.044500 2016.
[4] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint Stti::2001.029555 2024.
[5] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[7] Guanduo Chen, Yutong He, Yipeng Hu, Kun Yuan, and Binhang Yuan. Ce-lora: Computation- efficient lora fine-tuning for language models, arXiv preprint (rXXi:2502201378, 2025.
[8] Xi Chen, Kaituo Feng. Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of lims under low-rank constraint? arXiv preprint arXiv:2410001623, 2024.
[9] Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint
2019.
[10] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint /003./880000555 2018.
[11] Jeffrey Dean and Luiz André Barroso. The tail at scale. Communications of the ACM, 56(2):74- 80, 2013.
[12] Jianbo Dong, Bin Luo, Jun Zhang, Pengcheng Zhang, Fei Feng. Yikai Zhu, Ang Liu, Zian Chen, Yi Shi, Hairong Jiao, et al. Boosting large-scale parallel training efficiency with c4: A communication-diie approach. arXiv preprint arXiv:2406 04594, 2024.
[13] Sanghamitra Dutta, Ziqian Bai, Tze Meng Low, and Pulkit Grover. Codenet: Training large scale neural networks in presence of soft-errors. arXiv preprint arXiv: 2019.
[14] Sanghamitra Dutta, Viveck Cadambe, and Pulkit Grover. Short-dot: Computing large linear transforms distributedly using coded short dot products. Advances In Neural Information Processing Systems, 29, 2016.
[15] Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Murali Annavaram, Krishnakumar Nair, and Misha Smelyanskiy. Check-n-run: A checkpointing system for training recommendation models. CoRR, 2020.
11

---

[16] Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al. Dapple: A pipelined data parallel upproach for training large models, In Proceedings of the 26th ACM SIGPIAN Symposium on Principles and Practice of Parallel Programming, pages 431-445, 2021.
[17]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[18] Bhagvan Krishna Gupta, Ankit Mundra, and Nitin Rakesh. Failure detection and recovery in hierarchical network using fin approach. arXiv preprint arXiv:1401.888 2014.
[19] Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. Sitrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. arXiv preprint arXiv:2406.022144 2024.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[21] Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410.1 12289, 2024.
[22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203./5556. 2022.
[23] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790-2799 PMLR, 2019.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):,, 2022.
[25] Insu Jang, Zhenning Yang. Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. Oobleck: Resilient distributed training of large models using pipeline templates. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 382-395, 2023.
[26] Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, and Zhangyang Wang. Back razor: Memory-efficien transfer learning by self-sparsified backpropagation. Advances in neural information processing systems, 35:29248 222611,926111 2022.
[27] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 142.69,4 2014.
[28] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates. arXiv preprint 2:XX::3307.05655 2023.
[29] Stephanic Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falsehoods. arXiv preprint 210.022009.07988, 2021.
[30] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang. Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong. economical, and efficient mixture-of-experts language model. arXiv preprint 44440440 2024.
[31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[32] Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: deigpo-decomposed low-rank adaptation. In forty-first International Conference on Machine Learning, 2024.
12

---

[33] Zirui Liu, Guanchu Wang, Shaochen Henry Zhong, Zhaozhuo Xu, Daochen Zha, Ruixiang Ryan Tang, Zhimeng Stephen Jiang, Kaixiong Zhou, Vipin Chaudhary, Shuai Xu, et al. Winner-take- all column row sampling for memory efficient adaptation of language model. Advances in Neural Information Processing Systems, 16:3402-34244 2023.
[34] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-121072. 2024.
[35] Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. Velora: Memory efficient training using rank-1 sub-token projections. arXiv preprint arXiv:2405.1799 2024.
[36] Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. Checkfreq: Frequent, fine- grained dnn checkpointing. In 19th USENIX Conference on File and Storage Technologies (FAST 21), pages 203-216, 2021.
[37] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pages 1-15, 2019.
[38] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-Im Im. In Proceedings of the international conference for high performance computing, networking, storage and analysis. pages 1-15, 2021.
[39] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: lay- erwise importance sampling for memory memory-efficien efficient large language model tie-ttnnig. Advances 2024.
in Neural Information Processing Systems,
[40] Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. Optimus: an efficient dynamic resource scheduler for deep learning clusters. In Proceedings of the Thirteenth EuroSys Conference, pages 1-14, 2018
[41] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: À framework for adapting transformers. arXiv preprini arXiv:2007.0777) 2020.
[42] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):166) 2020.
[45] Thomas Robert, Mher Safaryan, lonut-Vias Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from oow-dimensional gradient statistics. arXiv preprint 1tt.::2410.1610000 2024.
[46] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 0ttl::/70000655500 006538, 2017.
[47] Shaden Smith, Mostofa Patwary. Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint 2rXiv:2201.11990 2022.
[48] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomputing. 167063, 2024.
13

---

[49] Rashish Tandon, Qi Lei, Alexandros G Dimakis, and Nikos Karampatziakis. Gradient coding: Avoiding stragglers in distributed learning. In International Conference on Machine Learning. pages 3368 -376. PMLR, 2017.
[50] John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. Bamboo: Making preemptible instances resilient for affordable training of large (DNNs}. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 497-513, 2023.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficcent foundation language models. arXiv preprint arXiv:2302.13977 2023.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 2023.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 2018
[55] Da Wang, Gauri Joshi, and Gregory Wornell. Using straggler replication to reduce latency in large-scale parallel computing. ACM SIGMETRICS Performance Evaluation Review, 43(3):7- 11, 2015.
[56] Guanhua Wang, Olatunji Ruwase, Bing Xie, and Yuxiong He. Fastpersist: Accelerating model checkpointing in deep learning. arXiv preprint arXiv:2406. 13768, 2024.
[57] Sunghyeon Woo, Baescong Park, Byeongwook Kim, Minjung Jo, Se Jung Kwon, Dongsuk Jeon, and Dongsoo Lee. Dropbp: accelerating fine-tuning of large language models by dropping backward propagation. arXiv preprint arXiv:2402. 178/2, 2024.
[58] Baodong Wu, Lei Xia, Qingping Li, Kangyu Li, Xu Chen, Yongqiang Guo, Tieyao Xiang, Yuheng Chen, and Shigang Li. Transom: An efficient fault-tolerant system for training llms. arXiv preprint arXiv:23 10046, 2023.
[59] Ran Yan, Youhe Jiang. Xiaonan Nie, Fangcheng Fu, Bin Cui, and Binhang Yuan. Hexiscale: Accommodating large language model training over heterogeneous environment, 2025.
[60] Zhiyuan Yu, Li Shen, Liang Ding, Xinmei Tian, Yixin Chen, and Dacheng Tao. Sheared back- propagation for fine-tuning foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5883-5892, 2024.
[61] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[62] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 10512 2023.
[63] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0/066 2022.
[64] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient I1m training by gradient low-rank projection. arXiv preprint 2:Xiv:2403.0307. 2024.
[65] Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sik- memory, adamw-level performance, arXiv preprimt arXiv:2412.05270, 2024.
14

---

A Proof of Theorem 1
First, we specify the update rules of MeCeFO with momentum SGD as follows:
m (t) m (t-1) 1) + (1 - -1)g
w (t+1) =w 1/m (t)
where g( is McCeFO's averaged weight gradient, m! 0, B1 E (0,1) is the momentum parameter, 1) is the learning rate.
Next, we present several key lemmas.
Lemma (Descent lemma). Under Assumption 7 it holds that
f(w((t1) {f(w(()-
L
[|
"0 " (
- 11 -m)
2n 2 V [Vf((w())
7
(3)
Proof of Lemmal
7 By L-smoothness of f (Assumption [. we have
L f(w((+)) sf(w(()) + (Vf())()), w w())) ww 22
(4)
For the inner product, we have
(V (((nll)) ()), (n(+1) (1))))
mn())
(t)
w (1+1) - (1) ) + Vf(w((()) ml()
2 w (++1) 20
2
1
|| Iw
(t+1)
(0 + !7
3 (/((((())) - (4) V f(w (1) )1)
(5)
21
where the last equality uses ((-11) - (t -ym (t) Applying ( to
yields
Lemma 2 (Momentum--raane -rrddient gap). Under Assumptions
andd 3 it holds that
E Bill m
((((()))]
T
t=U
2
4L2
E Blllw(()) (1-1-1113
(T) +1) 8(1 - 6112 T
(1-2) 2 (7 - 631) (7-63:)
6(1 31)0²
)1 +
(1)
(6)
T +
where A == m0) (0) V (1)
Proof of Lemma
According to the update rules, we have
(t)) (f(
-1]]] (m (t-1) {)) + (1 (1- (
()(
[ (m (t-1) ()(w + (1 (t)w())))
+ (1 --) " - Elg [| ]|
For the first term, applying Jensen's 's inequality yields
[||l (t-1 ()(w())) (1 B, (F) - ((n []]]]
{lmmll (t-1) v)(w + (1 - 31 31) - V f (1 [
<8 Bl]lm (1-1) v)(w(())) + + () -. 11
(8)
15

---

where the last inequality uses Assumption 3 By Young's inequality, we have
- 1 6(1
-11)
E m - V)(w(())
E m (t-1)
V (2(0((11)))1
2
El| ((I(s(((()) (
(9)
8(1 51)
For the second term, applying Cauchy's inequality yields
g - 1
<3Bllg - g 0 |11 + 3Ellg (1) - V (f(w - - E 113)
30
<3(1 - 3)B|ll!! 0 +
+ 3(1 (151-88B30 8) (0)
(6 - 36)0²
<6(1 /)l|ll / 1133]
(10)
where the second inequality uses Assumptions and 3 the last inequality uses Assumption Applying (8 (T)) to (7) and using Assumption
we obtain
18 m Vi(w 112]
1 - (1-1:1 [ (1- 1122) ) [[]lm((-1) ( + 8(1 - 31)
2L
B(llw()) (1-1))
(6 - 36) 31)
+ +1 () (7) 63, ) /
+
(11)
Summing
from 1 1 to T yields 6
Now we are ready to prove Theorem We restate Theorem as follows.
Theorem 2 (Convergence McCeFO), Under Assumptions 3 if 31 E (1 - 6/(24 126), 1) and
S (i)(L, ((8(1 (8L3), MeCeFO (with momentum SGD) converges as
T T +
8A 6n + 1)
8A,
+ 24(1 -1)02
E (9)l))))))
+
(12)
8(1 - 31) (T+)) 8
ōn
where A (w)( inf f(w), and A |m (0)
V f f(
Proof of Theorem
Summing 3 in Lemma I for t = 0, 1, ... T and taking expectation, we have
inf WERd
fww f(w (9))
E [ (1) 11
1
E Blll
(t+1)
w (1
2m
2
2
E E[| 1))
(13)
Applying Lemma 2 to 13) and noting that 31 E (1 - 5/(24 125), 1) implies (1 (1- (11)522)(7-0 -33))) < 1 8/4, we obtain
Elll f( 11
(0
<-
8
1
L
27L
1
B[llw ((+1)
w (11)
21
5(1
81)3
T+1
t=
1=0
8A
841
24(1
31)02
(14)
(n +1)
8(1
(T) +1)
6n
Noting that 7 < min(1/(2L), v (8(1 1 (1--8)3)(8L3)) implies 1/(4n) 2 L/2 and 1/(4n)
(2nL))/(((11 8)) 12) is direct result of
16

---

Table Performance and Memory Usage Comparison of Models with Varying Batch Sizes. "OOM"" denotes an Out of Memory error. A hyphen (-) indicates data not available.
Batch Size 256
Batch Size 512
Method
Throughput (tokens/s)
Memory (GB)
Throughput (tokens/s)
Memory (GB)
MeCeFOmrl
19.11k
76.13
OOM
MeCeFOrl
30.23k
54.65
OOM
McCeFO1
26.41k
38.48
23.86k
70.71
MeCeFO
28.06k
39.21
27.19k
73.25
MeCeFO w/o Fault
28.12k
41.52
30.04k
76.05
B Discussions
Experimental setup of failure scenarios. Although the experimental setup assumes uniformly random failures, real-world failure patterns are often asymmetric or localized. From a theoretical perspective, MeCeFO remains robust in such settings, since fallback operations continue to balance data exposure across pipelines without introducing systematic bias. support this claim, we further simulated persistent failures closely matched those under uniform random failures (see Appendix C.2). In addition, the failure- to-recovery ratios reported in Table Rable[[highligh the increasing challenges of repair and maintenance in larger computing systems; further discussion on the implications of these ratios can be found in Appendix
fixed subset of GPUs and observed validation perplexities that
Extension to other parallel strategies. While our main discussions focus on the DP+PP setting, the connections,
design of MeCeFO is inherently local to each node, its three core mechanisms selective activation recomputation, and low-rank gradient approximation are applied independently at the node level. This locality allows MeCeFO to naturally extend to TP scenarios. In the event of failed node within TP group, the workload can be redistributed across sibling TP ranks within the same PP stage, avoiding the recomputation of the entire group. When TPP 2, the resulting overhead per node is strictly less than 1x, which makes it feasible to adopt conservative fallback strategies for better eиrоr control. Furthermore, the mechanisms in McCeFO are tunable: skip connections may be applied to only a subset of sub-modules, gradient checkpointing can retain additional activations to reduce recomputation depth, and low-rank gradient approximation can employ higher ranks for improved fidelity. These adaptations ensure that McCeFO remains compatible with TP while providing flexibility in balancing efficiency and accuracy.
addition to hard-fault tolerance, there exists a complementary line of work on mitigating soft errors and stragglers. For instance, replication and redundancy mechanisms have been proposed to prevent undetected computational errors that may corrupt outputs [13] and to alleviate the performance impact of slow workers in distributed systems [49]. Our method takes a different perspective by tolerating bounded training errors in exchange for reduced computational redundancy, thereby improving efficiency while preserving robustness to hard faults. We view this perspective as complementary to existing approaches, and it may inspire future extensions of MeCeFO toward soft-error resilience straggler mitigation.
Transfer potential to soft-error scenarios.
C
Ablation studies and additional results
C.1 Ablation on key techniques in MeCeFO
We conducted ablation experiments assess the contribution of each technique training efficiency. These experiments were carried out on a server with 8 A100 GPUs using pipeline parallelism to train the LLaMA-7B model. "MeCeFO w/o Fault" denotes baseline training without node failures, while all other setups involved a single node failure during training. "MeCeFO" refers to the full proposed fault-tolerant algorithm. To evaluate individual components, we designed the following variants:
MeCeFOmrl: MeCeFO without skip-connection, selective activation recomputation and low-rank gradient approximation (key techniques I, II and III).
17

---

Table 7: Validation perplexities LLaMA-IB trained with MeCeFO under asymmetric (static subset) vs.
symmetric (uniform random) failures.
Failure Setting
No Fault
Low Freq.
Mid Freq.
High Freq.
Asymmetric 15.49 15.49
15.54 15.51
15.62 15.61
15.75
Symmetric
15.83
Table 8: Configurations and validation perplexities of LLaMA-IB pre-trained with McCeFO.
Scenario Name
Failure Interval
Node Recovery Time Perplexity
High Frequency Failure
Every 30 minutes
Every 120 minutes
15.83
Higher Frequency Failure
Every 10 minutes
Every 40 minutes
15.81
MeCeFO without selective acivation recomputation and low-rank gradient approxima- tion (key techniques II and III..
MeCeFOrl:
MeCeFOI: MeCeFO without low-rank gradient approximation (key technique III..
According to Table 6 removing all techniques (MeCeFOmrl) leads sharp increase in memory footprint from 41 B1.52GB to 76. 13BI for the neighbor node when resuming training with a batch size of 256. At the same time, throughput drops significantly from 28. 12k tokens/s to 19.11k tokens/s. With a batch size of 512, this configuration triggers an 00M (Out of Memory) error. These results indicate that using the NDB strategy alone is impractical.
For the MeCeFOrl variant, an DOM error still occurred a batch size of 512, indicating that dropping only MHA activations is insufficient to alleviate the memory pressure caused by the doubled workload.
In the MeCeFO1 variant, the throughput at a batch size of 256 decreased from 18.12k tokens/s to 26.41k k tokens/s. This suggests that although recomputing FFN activations helps reduce memory usage, the added computational overhead negatively impacts throughput.
Finally, the full MeCeFO algorithm, integrating all optimization components, achieved a throughput of 28.06k tokens/s and memory footprint of PGB under batch size of 256 in single-node failure scenario-closed approaching the performance of fault-free training.
These experimental results confirm that each component of the McCeFO scheme plays critical role in either memory optimization or computational efficiency. Their synergistic integration enables the system sustain high throughput and effectively prevent memory overflows, even under fault conditions with reduced computational resources.
C.2 A Ablation on asymmetric failures
We conducted an ablation study simulating persistent non-uniform failures. Specifically, we randomly selected GPUs to fail repeatedly throughout the entire training process, while the remaining GPUs remained fully operational. All other experimental settings were identical those used in the main study. The validation perplexities are summarized in Table 7 where the asymmetric setting closely matches the symmetric (uniform) failure case.
The results indicate that even the presence of persistent and localized failures, MeCeFO maintains robustness without significant degradation in training quality.
C.3 Ablation on failure scenarios
In fact, is the ratio between failure and recovery rates rrther than their absolute values- -tat is more relevant to training performance under failures, as it determines the steady-state proportion of healthy nodes and thus overall system behavior and algorithmic robustness. effect, we pre-trained the LLaMA-1B model with MeCeFO under a new failure scenario named Higher Frequency Failure where failures occur every 10 minutes and recoveries take 40 minutes, i.c. both events are more frequent while preserving the same ratio in the fighffrequency setting.
examine this
18

---

Table Equivalent failure and recovery rates under different scenarios.
Freq. Per GPU Per Hour
Freq. Per Real Node Per Hour
Scenario
Sim. Cluster
#Real Nodes Per GPU
Fail Freq.
Recov. Freq.
Fail Freq.
Recov. Freq.
Low Freq.
32 GPUs
1/64 1/32
1/128
N
1/164N)
1/(128N) 1/(192N)
Mid Freq.
32 GPUs
1/96
2N
1/(64N)
High Freq.
32 GPUs
1/16
1/64
4N
1/(64N)
1/(256N)
Table 10: Validation perplexities of MLA+MoF-11. 1.2B pre-trained with McCeFO.
Model
No Fault
Low Freq.
Mid Freq.
High Freq.
MLA+MoE-1.2B
16.17
16.22
16.37
16.43
The resulting validation perplexity 15.81, which is nearly identical to the 15,83 obtained in the original high-frequency scenario (see Table S
Remark. To further align our experimental setup with realistic large-scale deployments, we inten- tionally amplify failure and recovery events on 32-GPU cluster to emulate systems with hundreds or even thousands of nodes. In this abstraction, each simulated GPU corresponds to N > 1 real nodes, each with a much lower per-node failure or recovery rate. As summarized in Table [ this setup ensures that the equivalent failure frequency per real node remains consistent across different scenarios, while the equivalent recovery frequency per real node decreases, reflecting increasing difficulty of repair and maintenance in larger-scale clusters. A summary of this mapping between simulated and equivalent real-node clusters is provided in Table 9
C.4 Results on other model structures
We further evaluate MeCeFO Attention (MLA) [3I and Mixture-of-Expert (MoE) [46), with total of 1 1.2B parameters and 0. active parameters. As shown in Table I0, the validation perplexities of this model trained with McCeFO remain consistently comparable across all failure scenarios.
Deepseek V3-style model that integrates Multi-Head Latent
C.5 Results on extended validation of Assumption
Deeper models may exhibit longer error propagation paths. To assess the generality of our assumption beyond the IB case, conducted additional experiments 7B model with 32 transformer layers, constrained by our current computational resources. The results shown in Fig. 6 and | reveal similar trends in gradient approximation error, suggesting that Assumption remains valid at larger scales.
D Experimental specifications
This section provides detailed descriptions of experimental setup, covering algorithm implemen- tation, model specifications, and training configurations.
Implementation. We implement MeCeFO on top of the HexiScale framework 59
which itself
builds upon Megatron-LMM 38
Parallel strategies. We use DP] = 4 and IP] throughout all experiments.
Model specifications. Table II presents the detailed configurations of LLaMA-350M, LLaMA-IBB and LLaMA-7B, including hidden dimensions, FFN intermediate dimensions, number of attention heads, and layers. A maximum sequence length of 256 is used across all experiments.
Training configurations. Across all scenarios, we use the AdamW optimizer with 01 0.9. 32 = 0.999, peight_decap
0.01, and e = x 10-8, A learning rate warmup is applied over the first 10% of training iterations, followed by a cosine annealing schedule that decays the learning rate to 10% of its initial value. For MeCeFO, the SVD frequency is set to 100. The number of training steps, batch sizes, and initial learning rates are listed Table TT and are tuned exclusively for optimizing baseline fault-free training performance.
19

---

LLaMA-7B pre-training
LLaMA-7B pre-training
cPDF
10.
cPDF
Density
Density
0.05
0.30
0.15
0:20
0.25
0.30
0.55
0.40
0.45
0.59
0.000
0,829 0,550 0.075 0.100 8:22 0.35 0.175 0.20 8.25 0.25 Relative Error
Relative Error
Figure 6: Single-batch relative error of pre-training LLaMA-7B the C4 dataset.
Figure 7: Full-hatch relative error of pre-training ILaMA-7B on the C4 datasct.
Table 1I: Architecture and Hyperparameters of Different LLaMA Models.
Model
Hidden
Intermediate
Heads
Layers
Steps
Batch Size
Learning Rate
LLaMA-350M
1024
2736
16
24
6k
8192
8 x 10-4
LLaMA-1B 1B
2048
5461
32
24
20k
4096
6 10 -4
LLaMA-7B
4096
11008
32
32
60k
1024
4 x 10
Failure Modeling in Fault-Tolerann Computing. Our work is related to fault-tolerant computing and reliability in distributed training. Prior studies have often modeled system failures using expo- nential or shiftedd-eppmenntall distributions motivated by straggler effects [1] 55 14). In contrast, focus is primarily sudden hardware failures (e.g., node crashes), which approximate as memoryless events. This motivates the adoption of a Poisson process assumption, under which each node is assigned a constant failure probability per iteration. Importantly, one adopts stricter time-based Poisson model, then methods with lower throughput (i.e., longer iteration times) would experience higher effective failure probabilities. Since baseline methods tend suffer more severe throughput degradation under failure than MeCeFO, such a model would further amplify the relative advantage of our approach.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of the work in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
The paper provides the full set of assumptions in Sec. and complete and correct proof in Sec. A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses all information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
This paper provides open access to the codes in https:///itthii com/ Akumellon/MeCeFO The dataset used are all publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//bdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and testing details necessary understand the results, including the optimizers, hyperparameters, batch sizes, etc.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper reports the average performance without error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This paper specifies compute resources in Sec. and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwurrr...ccccaaaaccccctt
Answer: [Yes]
Justification: The research conducted this paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper mainly focuses on robust training strategies, which does not have direct social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose the concerned risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
This paper explicitly mentioned and properly credited the used models and datasets.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps//eec.ooooeeeeee////2////// for what should or should be described.
27


Reviews:
[{"review_id": "j2yOzxUS9Q", "reviewer": "Reviewer_cmnw", "summary": "This paper introduces MeCeFO, a novel fault-tolerant optimization algorithm designed to ensure robust training with minimal overhead. When a computing node fails, MeCeFO efficiently transfers the workload to a neighboring node while applying memory- and computation-saving techniques. It incorporates three key algorithmic innovations: skip-connection to approximate attention during backpropagation, recomputation to reduce activation memory, and low-rank gradient approximation for efficient gradient estimation. Theoretically, MeCeFO achieves convergence comparable to conventional distributed training, while empirically demonstrating strong resilience under frequent failure conditions.", "questions": "1. This paper states that \"The convergence depends on the quality of final weights, not the precise trajectory.\" Can you validate it by experiments?\n\n2. In real systems, failures tend to cluster on certain nodes (i.e., non-i.i.d.), causing the NDB to repeatedly fallback on the same “victim” nodes. Please analyze how this asymmetric load might induce training bias or failure. \n\n3. Section 4 provides an upper-bound theoretical analysis on the convergence impact of skip/backward approximation. Can you provide worst-case analysis and explore whether these errors could be amplified during early training stages or critical layers?", "limitations": "Nil", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. This paper proposes MeCeFO, a novel fault-tolerant optimization algorithm that enhances efficiency in distributed training.\n2. This paper provides theoretical guarantees showing MeCeFO achieves convergence rates comparable to standard distributed SGD under mild assumptions.\n3. This paper demonstrates MeCeFO’s superior resilience to frequent failures, maintaining high training throughput and robust model performance.\n\n\nWeakness:\n1. This paper states that \"The convergence depends on the quality of final weights, not the precise trajectory,\" which is not validated by the experiments. The experiments verify \"the quality of final weights\" only through final perplexity, without evaluating downstream transfer capabilities (e.g., zero-shot or few-shot task performance), training stability, or generative quality metrics (such as BLEU).\n\n2. A key implicit assumption behind the MeCeFO mechanism is that failures are sparse and “uniformly random,” with roughly equal failure probabilities across nodes and the ability for neighbors to rotate load. In real systems, failures tend to cluster on certain nodes (i.e., non-i.i.d.), causing the NDB to repeatedly fallback on the same “victim” nodes. The paper does not analyze how this asymmetric load might induce training bias or failure. This raises concerns about the robustness and generalizability of the proposed method.\n\n3. Section 4 provides an upper-bound theoretical analysis on the convergence impact of skip/backward approximation, but it lacks worst-case analysis and does not explore whether these errors could be amplified during early training stages or critical layers. For fault tolerance mechanisms, a key question is whether the \"worst-case deviation is controllable.\" Currently, the theoretical discussion focuses mainly on average-case scenarios, which cannot guarantee system stability.", "was_revised": true, "final_justification": "I am satisfied with rebuttal.", "rebuttal": "We thank the reviewer for acknowledging our theoretical and experimental results, as well as the detailed comments and suggestions. All questions have been clarified as best as we can, and we are glad to address any further comments or questions.\n \n---   \n> **Weakness 1 (Question 1). The paper states that convergence depends on the quality of final weights rather than precise trajectory. Evaluation regarding downstream transfer capabilities, training stability or generative quality metrics are expected.**\n\nWe appreciate the reviewer’s concern and acknowledge that our statement — “The convergence depends on the quality of final weights, not the precise trajectory” — may have caused some confusion. Our intended message is that, for the goal of training a performant model, the specific optimization path is less critical as long as the final checkpoint achieves strong performance. In this context, our proposed training modifications are justified as long as they lead to a model of comparable or better quality.\n\nWe agree that final validation perplexity alone may not fully capture the quality of a model, and that broader evaluation—including training stability, downstream task performance, and generative quality—is important. Regarding training stability, we did not observe any issues such as gradient spikes or divergence in either MeCeFO or standard training, though a more systematic study of robustness is a valuable future direction.\n\nFor downstream evaluation, which is more straightforward to assess, we have conducted additional experiments. The following zero-shot evaluation results on BoolQ (Nautral Language Understanding), ARC-Easy (Reasoning), PIQA (Commensense Understanding) and TruthfulQA (Truthfulness) show that LLaMA-1B-N/L/M/H (LLaMA-1B trained with MeCeFO under no-fault/low-frequency/medium-frequency/high-frequency fault conditions) have similar performance:\n    \n**Table 1: Zero-shot evaluation results using MeCeFO-pretrained LLaMA-1B models**\n| Model   | BoolQ     | ARC-Easy  | PIQA      | TruthfulQA-MC2 | Avg.      |\n| ---------- | --------- | --------- | --------- | -------------- | --------- |\n| LLama-1B-N | 0.579     | **0.459** | 0.682     | 0.427          | 0.537     |\n| LLama-1B-L | **0.594** | 0.455     | 0.674     | **0.451**      | **0.544** |\n| LLama-1B-M | 0.571     | 0.446     | 0.678     | 0.425          | 0.530     |\n| LLama-1B-H | 0.587     | 0.454     | **0.684** | 0.417          | 0.536     |\n    \nWe also fine-tuned LLaMA-1B-N/L/M/H checkpoints—originally pre-trained with MeCeFO under no-fault, low-, medium-, and high-frequency failure scenarios, respectively—on the GLUE benchmark under the same corresponding failure settings. As shown in Table 2, MeCeFO-trained checkpoints maintained strong performance across all tasks, confirming that the benefits of MeCeFO extend to fine-tuning:\n\n**Table 2: Fine-tuning results on the GLUE benchmark using MeCeFO-pretrained LLaMA-1B models**\n\n|            | CoLA      | STS-B     | MRPC      | RTE       | SST2      | MNLI      | QNLI      | QQP       | Avg       |\n| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| LLama-1B-N | 46.93     | **89.21** | **89.12** | 62.61     | **92.36** | **81.82** | 88.61     | 89.83     | 80.06     |\n| LLama-1B-L | 46.86     | 89.14     | 88.92     | 62.59     | 92.31     | 81.78     | 88.58     | **90.07** | 80.03     |\n| LLama-1B-M | **47.21** | 89.14     | 88.84     | **63.18** | 92.25     | 81.80     | 88.61     | 90.02     | **80.13** |\n| LLama-1B-H | 46.67     | 89.16     | 88.87     | 62.58     | 92.30     | 81.71     | **88.66** | 89.94     | 79.99     |\n\nResults show that models trained with MeCeFO during both pre-training and fine-tuning achieve downstream task accuracies comparable to those trained with standard methods, supporting the claim that our approach does not compromise generalization or transfer ability.\n\n---   \n> **Weakness 2 (Question 2). MeCeFO mainly considers sparse and \"uniformly random\" failures. In real systems, failures tend to cluster on certain nodes, implying asymmetric load, raising concerns about MeCeFO's robustness and generalizability.**\n    \nWe thank the reviewer for raising this important concern. We agree that real-world failures can exhibit asymmetric and non-i.i.d. patterns, and understanding how MeCeFO performs under such conditions is crucial. While our current design assumes uniformly random failures for simplicity, we argue that MeCeFO remains robust even when this assumption is relaxed. Below, we provide both theoretical reasoning and supporting experimental evidence.\n\nSpecifically, we consider two illustrative failure patterns across two data-parallel pipelines, $P_1: N_{11} \\rightarrow N_{12}$ and $P_2: N_{21} \\rightarrow N_{22}$, where $N_{ij}$ denotes the node in $P_{i}$'s $j$-th pipeline stage. \n(a) failures alternate between $N_{11}$ and $N_{21}$, causing NDB to fallback alternately to $N_{12}$ and $N_{22}$;\n(b) $N_{11}$ always fails while $N_{21}$ remains healthy, resulting in repeated fallback to $N_{12}$ only.\n\nIn both cases, during even iterations (e.g., $t = 0, 2, 4, \\dots$), the fallback and non-fallback pipelines are consistent. During odd iterations, although the fallback targets differ, each step still involves one fallback and one normal pipeline. Assuming the training data is evenly and randomly distributed across DP ranks, the model continues to encounter similarly distributed data across both fallback and normal paths over time. As a result, no significant training bias is introduced, and the overall training dynamics remain stable.\n\nTo validate this analysis, we conducted an additional ablation study simulating persistent, non-uniform failure. Specifically, we randomly selected 5 GPUs and caused only these GPUs to fail throughout the entire training process, while others remained fully operational. All other experimental settings were kept the same as in the main study. The validation perplexities under this skewed failure setting are shown below, and closely match the results under uniform failure:\n\n| Setting  | No Fault | Low-frequency Fault | Medium-frequency Fault | High-frequency Fault |\n| -------- | -------- | ------------------- | ---------------------- | -------------------- |\n| LLaMA-1B (Asymmetric failure) | 15.49    | 15.54               | 15.62                  | 15.75                |\n| LLaMA-1B (Symmetric failure) | 15.49 | 15.51 | 15.61 | 15.83 |\n\nThese results suggest that even in the presence of static and localized failures, MeCeFO maintains strong robustness and does not suffer from significant degradation in performance.\n\nThat said, we acknowledge that a more systematic evaluation of highly skewed or clustered failure patterns is a valuable direction for future work, and we appreciate the reviewer for highlighting this important aspect.\n\n---   \n> **Weakness 3 (Question 3). The upper-bound theoretical analysis in section 4 focuses mainly on average-case rather than worst-case and does not explore whether these errors could be amplified during early training stages or critical layers.**\n\nWe appreciate the reviewer’s concern regarding the potential amplification of gradient approximation errors, particularly during early training stages or in critical layers. This is indeed an important aspect of fault-tolerant training.\n\nOur theoretical analysis explicitly addresses this issue through **Assumption 3**, which states that at each iteration $t$, the approximate gradient computed by MeCeFO for weights $w^{(t)}$ must maintain a bounded relative error—no worse than a multiplicative factor of $(1 - \\delta)$—compared to the true gradient from standard backpropagation. This assumption applies both to the stochastic and deterministic gradients.\n\nImportantly, our main convergence results—Theorem 1 and Corollary 1—hold even in the **worst-case scenario** where the relative error reaches this upper bound $(1 - \\delta)$ at every iteration. In other words, the theoretical guarantees are **not merely average-case**, but are designed to ensure convergence under worst-case bounded degradation, as long as Assumption 3 holds.\n\nTo support this assumption empirically, we provide error analyses in Figures 4 and 5, which show that in practice, including scenarios where the errors could occur in early training stages or critical layers, the observed gradient deviations stay well within the required bound. This suggests that even under stress conditions, MeCeFO maintains stability and robustness throughout training.\n\n---\nWe thank the reviewer again for the careful comments and valuable suggestions. We hope these responses can clarify the reviewer's questions and are more than happy to address any further comments or questions."}, {"review_id": "MhbBRljPRS", "reviewer": "Reviewer_hbK1", "summary": "This paper proposes a new fault tolerance optimization technique for LLM training named MeCeFO. This approach effectively handle the failed node recomputation sceanrios by employing 1) selective skip reconnection 2) activation recomputation 3) low rank gradient approximation to reduce the computational and memory overhead when neighbour is need to recompute the result for failed node. The paper supports this approach by providing convergence analysis and empirical evaluation.", "questions": "- In line 236, I believe the author justified assumption 3 through empirical observation only on the error rate. However, the empirical observation is only based on a simple observation of llama 1B pre-training. Could the author elaborate a bit more on whether this observation is enough / further ablation on a larger model / more setting is required to justify this assumption fully?\n- By choosing the NDB strategy among the DP group, the author assumed that the failure domain should be constrained within one DP group, ensuring that there would always be a node in the same DP group to recover the training progress. I wonder what would be the case if the failure domain is the entire DP group (i.e., if one switch corresponding to that DP group failed). Could the author discuss the impact of the failure domain in more detail?\n- The evaluation and discussion primarily focused on llama-like dense architecture. I am curious about whether these techniques can be applied to MoE models, and if so, whether the sparsity in MoE models can facilitate additional resource savings and recomputation. Additionally, I am interested in determining the optimal placement of the router computation within the failure domain. Also, if the skip connection is suitable for MHA only or is it good on MLA, etc.", "limitations": "The author adequately addresses most of the work's limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength\n\n- Novelty. The author presents a novel method for resource-efficient recomputation in fault-tolerant training, based on the observation of the impact of placing skip connections at the MHA component and utilizing approximation techniques to reduce gradient computation complexity. This approximation technique is novel for weighing the tradeoff between accuracy and computational complexity.\n- Robust Evaluation and Convergence Analysis. The paper provides a robust evaluation with different failure settings and various model size of pre-training to demonstrate the effectiveness of their approach. They also provide a convergence analysis for the proposed approach, showing that it achieves convergence rates comparable to those of SGD.\n- Well-motivated optimization. The paper’s selection of techniques to minimize computational and memory overhead is well-reasoned and provides a good middle ground for the fault-tolerant theme they identified - approximation is sometimes good enough, and exact retracing of computation might not be necessary. I believe that in the paper, the approximation theme is well-motivated and effectively utilized.\n\nweakness\n\n- The author discusses hybrid parallelism, focusing on addressing the failure node through DP group recomputation. However, the TP group is likely the ‘minimal group for nodes’ under the training strategy, i.e., nodes → TP group → PP group → DP group. In this case, a failed node would be present in a TP group. If I understand the proposed approach correctly, the entire TP group / PP group needs to be recomputed as they would be counted as one ‘DP shard’ of the DP group, which results in recomputation of ‘healthy node’. Could the author discuss what would happen in this case, specifically whether the healthy node needs recomputation or can cache the result, and whether their approach could be directly applied to the TP group?\n- In the empirical evaluation, the node recovery time seems to be a bit confusing - could the author explain the rationale for choosing node recovery time that is up to an hour scale, and the recovery time differs in each setting (rather than, for example, a constant recovery time of say, 2h). In addition, based on previous literature, I wonder if the low frequency setting reporting in the evaluation is not representative, and shorter failure period results (i.e., 5 minutes, 15 minutes failure interval) should also be considered.\n- In practice, different failure causes likely result in different failure recovery times and frequencies - some might simply be a flip of a network connection, whereas others require hardware replacement. The uniform failure rate/failure mode assumed in the evaluation is an idealized situation considering what would happen at a real, large-scale LLM pre-training setting. In addition, sometimes due to the failure cause(e.g, failed network switch), more than one node would fail with a locality. Could the author discuss how their approach would be adapted to these various failure modes and failure rates, and how the worst-case and/best-case scenarios may affect this approach?", "was_revised": true, "final_justification": "The author has resolved most of my concerns regarding the paper. With the recommended modification that I suggested, I am satisfied to give the score (5) on the paper.", "rebuttal": "We thank the reviewer for acknowledging our novelty, theoretical and experimental results, as well as the detailed comments and suggestions. All questions have been clarified as best as we can, and we are glad to address any further comments or questions.\n\n---\n> **Weakness 1. While this work focuses on DP+PP parallelism, TP group is likely the minimal group for nodes is real settings. Could the author discuss what would happen in the TP setting?**\n\nWhile our main discussions focus on the DP+PP setting, we emphasize that **all three core mechanisms of MeCeFO—skip connections, selective activation recomputation, and low-rank gradient approximation—are applied locally at the individual node level**, and thus MeCeFO can **naturally extend to TP scenarios**.\n\nIn the presence of a failed node within a TP group, a practical adaptation is to **redistribute its workload across sibling TP ranks within the same PP stage**. This avoids recomputing the entire TP group and instead only requires additional computation on a small subset of neighboring nodes. When TP > 2, this redistribution introduces **less than 1× overhead per node**, which allows us to adopt a more **conservative fallback strategy** to control the error.\n\nMoreover, MeCeFO is inherently **flexible and tunable**. Although in the standard configuration, our techniques reduce approximately half the workload per fallback, they can be easily adjusted to align with the parallelism structure and accuracy-performance tradeoff in TP settings:\n\n* **Skip connections** can selectively bypass only a fraction of sub-modules,\n* **Gradient checkpointing** can retain more activations and reduce recomputation depth,\n* **Low-rank gradient approximation** can adopt higher ranks for improved fidelity.\n\n---\n> **Weakness 2. Why does the node recovery time differ in each setting? In addition, evaluation on shorter failure period (i.e., 5/15 minutes) are expected.**\n\nIn our setup, the relatively **short failure intervals and variable recovery durations per node** serve as a mechanism to **simulate the aggregate failure dynamics of large-scale distributed systems using a small-scale cluster**. As discussed in [1] (section 7.2), such an experimental setting can cover a wide spectrum of environments.  Therefore, our increasing the per-node failure frequency is to simulate a larger distributed system where node failures happen more frequently, within a fixed number of hardware resources, rather than assuming every single-node is easier to fail. \n\nAccordingly, the choice of recovery time follows a similar rationale. In real systems, the failure and recovery rates are typically linked through shared infrastructure and operational constraints. Hence, we design the **recovery-to-failure rate ratio** to reflect this relationship, with recovery rates increasing **sublinearly** relative to failure rates. This models realistic scenarios in ultra-large clusters, where repair and maintenance resources may become bottlenecks under high failure pressure.\n\nWe argue that it is this **ratio between failure and recovery rates**—rather than their absolute values—that is most critical, as it determines the **steady-state proportion of healthy nodes** and thus the overall system behavior and algorithm robustness. To further support this point, we include an additional experiment, where failures occur **once every 10 minutes**, and recovery takes **40 minutes**. Under this setting, we pre-trained the LLaMA-1B model using MeCeFO and obtained a **validation perplexity of 15.81**, which is consistent with the performance (15.83) observed in our high-frequency corruption experiments.\n\n> [1] Jang, I., Yang, Z., Zhang, Z., Jin, X., & Chowdhury, M. (2023, October). Oobleck: Resilient distributed training of large models using pipeline templates. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 382-395).\n\n---\n> **Weakness 3 (Question 2). In practice, different failure causes likely result in different failure recovery times and frequencies. The uniform failure rate/failure mode is an idealized situation. In addition, failures like failed network switch may cause more than one node to fail. Could the authors discuss how MeCeFO would be adapted to these scenarios?**\n\nIn this work, MeCeFO is **explicitly scoped to address one of the most common case: single-node failures**, which, according to large-scale system studies [2], account for approximately **82.5%** of observed failure events in production-scale LLM training clusters. This single-node failures setting is the standard practice [1,3] in the field of fault-tolerant training.\n\nWe acknowledge that more complex scenarios, such as **localized multi-node failures**, are indeed important in real-world systems. While MeCeFO is specifically designed to handle **single-node failures**, which represent the majority of observed failures, it can be naturally integrated into a **hierarchical robustness framework**—serving as the **node-level component** within a broader fault-tolerance strategy.\n\nIn summary, while MeCeFO currently targets single-node failure resilience, it is **designed to be modular and composable**, making it compatible with broader strategies for tolerating diverse failure types. \n\n> [2] Dong, J., Luo, B., Zhang, J., Zhang, P., Feng, F., Zhu, Y., ... & Fu, B. (2025, March). Enhancing Large-Scale AI Training Efficiency: The C4 Solution for Real-Time Anomaly Detection and Communication Optimization. In *2025 IEEE International Symposium on High Performance Computer Architecture (HPCA)* (pp. 1246–1258). IEEE.\\\n> [3] Thorpe J, Zhao P, Eyolfson J, et al. Bamboo: Making preemptible instances resilient for affordable training of large DNNs[C]. 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 2023: 497-513.\n\n---\n> **Question 1. The empirical justification of Assumption 3 is only based on pre-training LLaMA 1B. Further ablation on a larger model / more setting is expected.**\n\nRegarding the scalability of this assumption, we offer the following clarifications:\n- **Model size and depth**: It is true that deeper models may exhibit longer error propagation paths. To assess the generality of our assumption beyond the 1B case, we conducted **additional experiments** on a **7B model with 32 transformer layers**, constrained by our current computational resources. The results shown in Table 1 and 2 below reveal similar trends in gradient approximation error, suggesting that Assumption 3 remains valid at larger scales.\n    \nTable 1: The single-batch relative error of pre-training LLaMA-7B on the C4 dataset under high-frequency failure scenarios. (Corresponds to Figure 4)\n\n|Relative Error Interval|[0.000,0.050)|[0.050,0.100)|[0.100,0.150)|[0.150,0.200)|[0.200,0.250)|[0.250,0.300)|[0.300,0.350)|[0.350,0.400)|[0.400,0.450)|\n|-|-|-|-|-|-|-|-|-|-|\n|Density|3.87|2.27|4.33|2.67|2.87|1.67|1.67|0.53|0.13|\n\nTable 2: The full-batch relative error of pre-training LLaMA-7B on the C4 dataset under high-frequency failure scenarios. (Corresponds to Figure 5)\n\n|Relative Error Interval|[0.000,0.025)|[0.025,0.050)|[0.050,0.075)|[0.075,0.100)|[0.100,0.125)|[0.125,0.150)|[0.150,0.175)|[0.175,0.200)|[0.200,0.225)|\n|-|-|-|-|-|-|-|-|-|-|\n|Density| 10.13|6.00|3.47|6.27|8.80|0.27|1.60|2.93 |0.53|\n\n- **Data parallel (DP) averaging effect**: In practice, hardware failures are often sparse. Due to **gradient averaging across DP ranks**, the impact of a single node’s gradient deviation is diluted. As the number of DP replicas increases, the overall effect of any one faulty node diminishes. This effect enhances the robustness of our method in **larger-scale deployments**, where partial failures are more frequent, but individually less impactful.\n\nTaken together, while more comprehensive validation under a broader set of model scales and failure modes remains a valuable direction for future work, our current empirical evidence and theoretical reasoning suggest that Assumption 3 holds **under realistic and practically relevant conditions**.\n\n---\n> **Question 3. The evaluation is limited to LLaMA-like dense architecture. Evaluation on MoE models and MLA is expected. It is interesting whether MoE's sparsity can facilitate resource saving and how to determine the router computation upon failure.**\n\nWe sincerely thank the reviewer for this insightful and forward-looking question.\n\n- **Applicability to MoE:** MoE models share a modular structure similar to dense models, consisting of feedforward blocks and attention layers. Despite sparsity at the expert-selection level, **under balanced routing, each node remains significantly utilized**, meaning MeCeFO’s memory and computation reduction strategies—such as skip connections and low-rank approximations—can still provide benefits. In cases of **load imbalance**, where some experts are underutilized, there might be further opportunity to reduce fallback aggressiveness, potentially improving accuracy.\n- **Router computation and failure domains:** Routing computations in MoE generally occur during the forward pass and are relatively inexpensive compared to expert MLP computations. We therefore anticipate that **router computations could remain unchanged during fallback**, as MeCeFO already preserves the forward path. \n- **Application to MLA:** We have experimented with a Deepseek V3-style model incorporating Multi-Head Latent Attention and Mixture-of-Experts, totaling 1.2B parameters with 0.1B active parameters. This model was trained under various fault scenarios—no fault, low-frequency, medium-frequency, and high-frequency faults—with corresponding validation perplexities as follows:\n\n|Model|No Fault|Low-frequency|Medium-frequency|High-frequency|\n|-|-|-|-|-|\n|MLA+MoE|16.17|16.22|16.37|16.43|\n\n---\nWe thank the reviewer again for the careful comments and valuable suggestions. We hope these responses can clarify the reviewer's questions and are more than happy to address any further comments or questions."}, {"review_id": "KACb3hIPhv", "reviewer": "Reviewer_16Gg", "summary": "This paper proposed a novel training algorithm MeCoFO designed to improve the fault tolerance of LLM training while minimizing additional memory and computation overhead.", "questions": "1. One claim is that MeCeFO avoids having to reserve half the GPU memory for potential failures. Can the authors provide more insight or data on memory usage?\n2. The experiments simulate up to a high frequency failure scenario (a failure every 30 minutes, with 2-hour recovery). If failures were even more frequent or recovery slower, how would that affect training?\n3. It would be great to see MeCeFO applied to other training regimes, such as fine-tuning or different model types, just to confirm its broad utility.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: \n1. This paper targets the important problem of hardware failures in large-scale LLM training. \n2. MeCoFO shows impressively low overhead.\n3. Instead of traditionally recovering through checkpoint etc. MeCeFO utilized the noise robust nature of SGD.\n4. The author gives theoretically proof that MeCeFO achieves a convergence rate of $O(1/\\sqrt{nT})$ under mild assumptions, demonstrating the robustness of MeCeFO algorithm.\n\nWeakness:\n1. The author mentioned that \"MeCeFO is primarily tailored for specific hybrid parallel strategies and transformer-based architectures\" which leads to a concern if this algorithm functions on other parallel strategies.\n2. They also note that they haven't explores full system-level optimizations in combination with their method. I appreciate this transparency, and don't see any glaring additional limitations.", "was_revised": true, "final_justification": "Thank you for your detailed response. \n\nIt has resolved all my concerns, and I have decided to maintain my rating score, and increase my confidence score to 4.", "rebuttal": "We thank the reviewer for acknowledging our theoretical and experimental results, as well as the detailed comments and suggestions. All questions have been clarified as best as we can, and we are glad to address any further comments or questions.\n\n---\n> **Weakness 1. MeCeFO is tailored for specific hybrid parallel strategies. Could the authors discuss if this algorithm functions on other parallel strategies?**\n\nWe appreciate the reviewer’s concern regarding the generality of MeCeFO under different parallel strategies. As noted in our Limitations section, our current implementation and evaluation are focused on a specific hybrid parallel setup commonly used in large-scale transformer training.\n\nThat said, **MeCeFO is fundamentally designed to reduce per-node computational and memory overhead**, thereby allowing surviving nodes to take on additional workload during failure recovery. This core principle is **independent of the particular parallel strategy** and remains valid under other setups, including tensor parallelism (TP).\n\nWhile we have not implemented MeCeFO under TP settings, we believe the extension is natural. Specifically, the three key techniques—**Skip Connection Rescheduling**, **Selective Gradient Checkpointing**, and **Adaptive Low-Rank Approximation**—are all applied locally within a node and can be adjusted to reflect the level of workload rebalancing required by different parallel granularities.\n\nFor example:\n\n- Under TP, when a failed node must be covered by sibling TP ranks in the same pipeline stage, the increase in per-node overhead is generally moderate (e.g., <2× when TP > 2). MeCeFO can thus employ more **conservative configurations** of the three techniques to ensure minimal impact on accuracy.\n- The ratio of skipped submodules in **Skip Connection Rescheduling**, the extent of recomputation in **Selective Gradient Checkpointing**, and the target rank in **Adaptive Low-Rank Approximation** can all be dynamically tuned to match the exact overhead introduced by the specific recovery pattern.\n\nTherefore, while our prototype targets a commonly used configuration, we believe **MeCeFO naturally extends to other parallel strategies**, and exploring such extensions is a valuable direction for future work.\n\n---\n> **Weakness 2. The authors haven't explored full system-level optimizations. Could they discuss the implications of this limitation?**\n\nWhile our current work has **demonstrated the algorithmic effectiveness of MeCeFO**, we have also **implemented a practical system prototype** with **measured throughput under fault scenarios**, showing that MeCeFO can be deployed and used in real training settings.\n\nHowever, this is not yet a full system-level implementation. In particular, **our fault scenarios are simulated rather than triggered by controllable hardware-level failures**. Achieving a complete system integration—where failures are **automatically detected, injected, and handled at the infrastructure level**—would require **low-level access to the cluster scheduler or hardware stack**, which is beyond our current resource scope.\n\nIn this sense, while our work covers both algorithm design and usable implementation, **a fully integrated system with real-time fault injection and recovery logic lies slightly beyond the scope of this paper**, and we consider it an important direction for future work.\n\n---\n> **Question 1. MeCeFO claims to avoid reserving half the GPU memory. Could the authors provide more data on memory usage?**\n\nAs discussed in **Section 3**, MeCeFO reduces memory usage primarily through two key techniques:\n\n- **Skip Connection for Fault Recovery:** By skipping the update of attention parameters during the recovery phase, we save both the **activation storage** and the **optimizer states** for these parameters. In standard Transformer blocks, the multi-head attention (MHA) module includes Q/K/V/O projections totaling roughly $4h^2$ parameters, while the feedforward network (FFN) has about $8h^2$. Thus, the MHA typically takes up around **1/3 of the total parameters per block**, and skipping their updates leads to significant memory savings during training.\n- **Selective Gradient Checkpointing:** We apply gradient checkpointing to **only the FFN module**, which avoids storing intermediate activations for FFN and instead recomputes them during backpropagation. This choice balances memory savings and recomputation overhead, and further **reduces activation memory** substantially.\n\nTogether, these two techniques allow MeCeFO to avoid the need for **pre-reserving up to half of GPU memory** for recovery.\n\nWe refer the reviewer to **Table 4** (reproduced below) for quantitative memory usage comparisons:\n\n|Method|Throughput/(Tokens$\\cdot$s$^{-1}$) (bsz=256)|Memory/GB (bsz=256)|Throughput/(Tokens$\\cdot$s$^{-1}$) (bsz=512)| Memory/GB (bsz=512)|\n|-|-|-|-|-|\n|MeCeFOmrl|19.11k|76.13|-|OOM|\n|MeCeFOrl|30.23k|54.65|-|OOM|\n|MeCeFOl|26.41k| 38.48|23.86k|70.71|\n|**MeCeFO**|28.06k|39.21|27.19k|73.25|\n|MeCeFO w/o Fault|28.12k|41.52|30.04k|76.05|\n\n**Interpretation**:\n\n* **MeCeFOmrl** represents the baseline with no memory-saving techniques.\n* **MeCeFOrl** applies only the skip connection, already reducing memory by \\~22GB.\n* **MeCeFOl** adds **selective gradient checkpointing** on top of the skip connection, achieving even more memory reduction (from 76.13GB → 38.48GB) for batch size 256, and remains runnable at batch size 512.\n* **MeCeFO** combines all three techniques and keeps memory usage close to the fault-free baseline, thus **avoids the need to statically reserve half of the memory**, instead dynamically adapting with significantly **lower memory footprint** and maintaining high throughput.\n\n---\n> **Question 2. How would even more frequent failures or slower recovery affect training?**\n\nThank you for the insightful question. In general, the **training efficiency under failures is primarily determined by the expected fraction of failed nodes at any given time**. This fraction is governed by the **ratio between failure rate and recovery rate**:\n\n* If failures become **more frequent** or **recovery becomes slower**, the expected fraction of failed nodes increases. This leads to **larger degradation** in training performance for all methods.\n* Conversely, if failures are less frequent or recovery is faster, the average number of unavailable nodes at any time is lower, and **performance degradation is mitigated**.\n\nThis relationship is also evident in our experiments: from *low-frequency* to *high-frequency* settings, the **failure-to-recovery ratio increases**, and we observe **increasing gaps between MeCeFO and other baselines** in terms of training throughput. This confirms that:\n\n- **All methods degrade** as the failure-to-recovery ratio increases.\n- **MeCeFO degrades much more gracefully**, maintaining significantly better training speed under harsher conditions.\n\nTo further support this, we conducted an additional experiment using MeCeFO to pre-train LLaMA-1B under a failure frequency of once every 10 minutes and a recovery time of 40 minutes. This setting yielded a **validation perplexity of 15.81**, closely matching the **15.83** observed under the 30-minute failure, 2-hour recovery setting. These results reinforce our claim that **training performance is primarily influenced by the failure-to-recovery ratio**, rather than the absolute failure rate alone.\n\nIn the **extreme case** where all nodes are permanently failed (e.g., recovery never happens and no new nodes are added), **no distributed algorithm can make progress**, including MeCeFO. However, within practical ranges of node unavailability, **MeCeFO exhibits strong robustness**, and its **relative advantage grows as fault conditions worsen**.\n\n---\n> **Question 3. Could MeCeFO be applied to other training regimes (e.g., fine-tuning) or different model types?**\n\nWe appreciate the suggestion. To assess the broader applicability of MeCeFO, we conducted two additional evaluations covering both fine-tuning and alternative model architectures.\n\n- **Fine-tuning on GLUE:** We fine-tuned LLaMA-1B-N/L/M/H checkpoints—originally pre-trained with MeCeFO under no-fault, low-, medium-, and high-frequency failure scenarios, respectively—on the GLUE benchmark under the same corresponding failure settings. As shown in Table 1, MeCeFO-trained checkpoints maintained strong performance across all tasks, confirming that the benefits of MeCeFO extend to fine-tuning:\n\n**Table 1: Fine-tuning results on the GLUE benchmark using MeCeFO-pretrained LLaMA-1B models**\n\n| |CoLA|STS-B|MRPC|RTE|SST2|MNLI|QNLI|QQP|Avg|\n|-|-|-|-|-|-|-|-|-|-|\n|LLama-1B-N|46.93|**89.21**|**89.12**|62.61|**92.36**|**81.82**|88.61|89.83|80.06|\n|LLama-1B-L|46.86|89.14|88.92|62.59| 92.31|81.78|88.58|**90.07**|80.03|\n|LLama-1B-M|**47.21**|89.14|88.84|**63.18**|92.25|81.80|88.61|90.02|**80.13**|\n|LLama-1B-H|46.67|89.16|88.87|62.58|92.30|81.71|**88.66**|89.94|79.99|\n\n- **Support for other model architectures:** To evaluate MeCeFO’s compatibility with different model types, we implemented a DeepSeek V3-style architecture featuring Multi-Head Latent Attention (MLA) and a Mixture-of-Experts (MoE) design. This model has 1.2B total parameters, with 0.1B active per token. We pre-trained it under the same failure scenarios, and observed consistently robust performance, as illustrated in Table 2.\n\n**Table 2: Validation perplexities of MoE models trained with MeCeFO under various failure frequencies**\n\n|Model|No Fault|Low-frequency Fault|Medium-frequency Fault|High-frequency Fault|\n|-|-|-|-|-|\n|MLA+MoE|16.17|16.22|16.37|16.43|\n\nThese results demonstrate that **MeCeFO remains effective beyond pre-training**, extending to both **fine-tuning tasks and structurally different models**.\n\n---\nWe thank the reviewer again for the careful comments and valuable suggestions. We hope these responses can clarify the reviewer's questions and are more than happy to address any further comments or questions."}, {"review_id": "axIi0G0JGZ", "reviewer": "Reviewer_Ebnf", "summary": "The paper proposes a new strategy of training LLMs (distributed optimization) in the presence of hardware failures. The strategy is called Memory- and Computation- efficient Fault-tolerant Optimization (MeCeFO) which uses some interesting algorithmic innovations, like skip connections, recomputation, and low-rank gradient approximation. They have provided a theoretical guarantee and also included experimental results on LLAMA 350M, 1B, and 7B, comparing with two baselines: (1) Bamboo (2) Oobleck. The proposed strategy shows improvements over these methods (under simulated node failures).", "questions": "Could you highlight which steps of the proof are altered under this scheme? Is it only the approximate gradient step? \n\nCould you highlight if and how the theoretical guarantee depends on the nature of the failure, e.g., if the system is more/less failure-prone, shouldn't it affect the theoretical guarantee?", "limitations": "One sentence on limitations is included. But, this could be expanded to include some of the points mentioned in Weakness.\n\n-- Other kinds of failure models beyond constant frequency\n\n-- Simulated vs Real failures/delays\n\n-- Other kinds of approaches in fault-tolerant computing that are based on replication/ECCs\n\n-- Theoretical guarantees", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "STRENGTHS\n\n1. This is a very interesting and relevant problem. They have motivated the issue of hardware failures quite well, citing practical challenges faced by Alibaba and Meta. \n\n2. The proposed strategy seems quite interesting, and combines three types of algorithmic innovations, diving deep into the nuances of matrix operations.\n\n3. Experiments are provided over several LLMs for pre-training, which is not an easy feat (also requires significant computational resources). They have compared with two baselines and demonstrate considerable improvements. Ablation studies are included in the Appendix.\n\nWEAKNESSES\n\n1. The theoretical guarantee is much appreciated, but it seems to follow from standard assumptions and convergence guarantees of momentum SGD in distributed settings. Could you highlight which steps of the proof are altered under this scheme? Is it only the approximate gradient step? Could you highlight if and how the theoretical guarantee depends on the nature of the failure, e.g., if the system is more/less failure-prone, shouldn't it affect the theoretical guarantee?\n\n2. It is not very clear if and how the system failure is modelled into the mathematical analysis. Do you assume a particular statistical model for the failure? In the experiments, a constant frequency failure model has been considered. \n\nPrior works in fault-tolerant computing have also considered exponential runtimes, motivated by the straggling effect (causes node delays which could also be an interesting motivational use-case for this paper). Some of these references could be included in the motivation/related works:\n\n[1] Tail at Scale: https://research.google/pubs/the-tail-at-scale/\n\n[2] Using Straggler Replication to Reduce Latency in Large-scale Parallel Computing: https://dl.acm.org/doi/abs/10.1145/2847220.2847223\n\n[3] Short-Dot: Computing Large Linear Transforms Distributedly Using Coded Short Dot Products : https://proceedings.neurips.cc/paper_files/paper/2016/file/aace49c7d80767cffec0e513ae886df0-Paper.pdf\n\n3. I guess the hardware failures are simulated - under a constant frequency failure model. It would be interesting future work to study other kind of failures, e.g., by a Poisson process. \n\n4. Curious if it possible to see any kind of real failures rather than simulated ones, either from delays (straggling) or failures? Could you comment on whether the clusters were dedicatedly set aside during these experiments, or were other jobs also being scheduled on them during this time (potentially leading to natural delays/failures)?\n\n5. Prior works in fault-tolerant computing have studied alternate approaches [4,5] to perform reliable training of machine learning models under stragglers/hardware failures - by using replication or error-correction to have extra computations so that one can proceed even if some nodes fail/straggle. This direction of research would be a relevant complementary approach for the proposed technique [4,5], and could be discussed in related works if not compared with.\n\n[4] CodeNet: Training Large Scale Neural Networks in Presence of Soft-Errors: https://arxiv.org/abs/1903.01042\n\n[5] Gradient Coding: Avoiding Stragglers in Distributed Learning: https://proceedings.mlr.press/v70/tandon17a/tandon17a.pdf", "was_revised": true, "final_justification": "The authors have addressed my comments. I have increased my rating.", "rebuttal": "We thank the reviewer for acknowledging our method and experimental results, as well as the detailed comments and suggestions. All questions have been clarified as best as we can, and we are glad to address any further comments or questions.\n\n---\n> **Weakness 1 (Question 1 and 2). Could the authors highlight which steps of the proof are altered under the scheme and how the theoretical guarantees depend on the failure?**\n\nThank you for the thoughtful question. The main modifications in the theoretical analysis occur in **Lemma 2**, specifically in equations **(8)** and **(10)**, where we apply **Assumption 3** to upper-bound the bias introduced by the approximate gradient.\n\n* If there were **no failure** or **no gradient approximation**, the analysis would reduce to the standard momentum SGD proof (e.g., by setting $\\delta = 1$).\n* In our case, $\\delta$ captures the **quality of the gradient approximation**, which is **directly affected by the failure rate**. A higher failure rate leads to **a larger proportion of approximated gradients**, which results in a **smaller value of $\\delta$** under Assumption 3.\n\nThis dependency is reflected in **Corollary 1**, where the convergence rate is explicitly influenced by $\\delta$: the **smaller the $\\delta$**, the **slower the convergence**. Thus, the **impact of failures on theoretical convergence** is clearly accounted for through this parameter.\n\n---\n> **Weakness 2. It is not clear how the system failure is modeled into mathematical analysis. Prior works [1-3] in fault-tolerant computing have considered exponential runtimes motivated by the straggling effect, which could be included in the motivation/related works.**\n\nThank you for the valuable references. For mathematical analysis, the failure is modelled through $\\delta$ in Assumption 3. For experimental design, due to limited computational resources, we did not collect real-world failure traces to fit a detailed statistical model. Our focus in this work is primarily on **sudden hardware failures**, such as node crashes, rather than **latency or straggler effects**.\n\nWe assume that such hardware failures are approximately **memoryless**, which motivates our use of a **Poisson process** as an underlying assumption. Given that iteration times are relatively stable under our setup, this further justifies modeling each node with a **constant failure probability** per iteration.\n\nMoreover, we assign the same failure/recovery probability to each node for two main reasons:\n\n1. The failure simulation is intended to estimate the behavior of large-scale training under uniform stress, rather than to capture heterogeneous failure behavior.\n2. Under the **i.i.d. data distribution** assumption, the specific identity of the failed node (i.e., which data-parallel worker it belongs to) does not substantially alter the training dynamics.\n\nWe agree that more refined models like **shifted exponential distributions**, as used in [2], are highly relevant and could enrich our framework, especially in extending MeCeFO to cover straggler-style delays. We thank the reviewer for suggesting this direction and will incorporate these discussions and references into the revised version of the paper.\n\n---\n> **Weakness 3. It would be interesting future work to study other kind of failures, e.g., by a Poisson process.**\n\nThank you for the insightful comment. As we mentioned earlier, if the iteration time is approximately constant—which is the case in our experiments due to stable training throughput—then our **constant per-iteration failure probability** is essentially equivalent to a **Poisson process** in continuous time.\n\nIf one were to adopt a stricter Poisson process model where failures are measured per unit time rather than per iteration, then methods with **lower throughput (longer iteration time)** would experience **higher failure probabilities**. The baseline methods suffer from **more severe throughput degradation** compared to MeCeFO under failure, this would effectively make their performance worse under a true Poisson process. In contrast, MeCeFO would benefit relatively more, potentially **amplifying its advantage** in such a setting.\n\nWe appreciate the reviewer for raising this interesting point, which we believe strengthens the case for MeCeFO under realistic failure conditions. We will include this discussion in the revised version of the paper.\n\n---\n> **Weakness 4. Curious if it possible to see any kind of real failures rather than simulated ones. Whether the clusters were dedicatedly set aside during these experiments, or were other jobs also being scheduled on them during this time?**\n\nThank you for the question. Due to hardware resource limitations, our study primarily focuses on **simulated failures** rather than real hardware failures or delays, following a common setup in the literature [6]. While we recognize the importance of evaluating real-world failure scenarios (e.g., stragglers or transient faults), it is challenging to observe and control such events reliably without access to large-scale dedicated infrastructure.\n\nTo ensure the **validity of our throughput measurements**, we ran all experiments on **dedicated nodes**, without any other concurrent jobs scheduled on the same machines. This allowed us to isolate the effects of simulated failures and ensure consistency across runs.\n\nWe agree that incorporating real system-level failures or straggling behaviors would be a valuable direction for future work.\n\n> [6] Jang, I., Yang, Z., Zhang, Z., Jin, X., & Chowdhury, M. (2023, October). Oobleck: Resilient distributed training of large models using pipeline templates. In Proceedings of the 29th Symposium on Operating Systems Principles (pp. 382-395).\n\n---\n> **Weakness 5. Prior works have studied alternate approaches [4,5] to perform reliable training of machine learning models by using replication or error-correction to have extra computations. This direction of research would be a relevant complementary approach and could be discussed in related works.**\n\nWe thank the reviewer for the valuable suggestion. We plan to add the following discussion to the paper:\n\n> In addition to addressing **hard errors** such as node dropouts or hardware failures, there has been substantial work on **soft-error tolerance** in distributed training. For example, [4] focuses on preventing undetected errors that may cause nodes to produce garbage outputs, while [5] addresses the straggler problem by using redundancy to mitigate slow workers in distributed setups.\n> We believe that the core idea of our work—**tolerating a bounded amount of training error in exchange for reduced computational redundancy**—offers a novel perspective for improving robustness to hard faults. This approach may also inspire future research into complementary domains such as soft-error resilience or straggler mitigation.\n\n---\n> **Limitation. The limitation could be expanded to include some of the points mentioned in Weakness.**\n\nWe thank the reviewer for the constructive suggestion. In the revised version, we will expand the discussion of limitations to explicitly include the following points:\n\n- **Failure modeling**: Our current model assumes a constant per-iteration failure probability, which aligns with a Poisson process under the assumption of stable throughput. However, it does not capture more complex or dynamic failure patterns such as straggling delays or correlated failures.\n- **Simulated vs. real failures**: Due to limited hardware access, our experiments simulate failure events rather than observing real hardware failures or delays. Investigating MeCeFO’s behavior under real-world system failures would be an important future direction.\n- **Complementarity with redundancy-based approaches**: Our method is orthogonal to prior fault-tolerant strategies based on replication or error-correcting codes. While those methods aim to recover exact results through redundancy, our approach tolerates a small amount of error to preserve throughput, making it particularly suitable for large-scale or LLM training tasks.\n- **Theoretical guarantees**: Our analysis relies on Assumption 3 (bounded deviation of failure-influenced gradients), which may not always hold in more complex or adversarial settings. Extending the theory to relax this assumption is also an important future direction.\n\nWe will include these points explicitly in the limitations section of the paper.\n\n---\nWe thank the reviewer again for the careful comments and valuable suggestions. We hope these responses can clarify the reviewer's questions and are more than happy to address any further comments or questions."}]

Author rebuttal:

Meta-review: **Summary**:
The paper introduces MeCeFO, a novel fault-tolerant optimization technique for large language model (LLM) training, designed to handle node failures efficiently. It employs selective skip reconnection, activation recomputation, and low-rank gradient approximation to minimize computational and memory overhead during failure recovery. The paper provides theoretical convergence guarantees and empirical evaluations on LLAMA models (350M, 1B, and 7B), comparing MeCeFO against baselines like Bamboo and Oobleck.

**Strengths**:

1. Reviewers unanimously praise the novelty of MeCeFO, particularly its use of skip connections at the multi-head attention (MHA) component and low-rank gradient approximations to balance accuracy and computational efficiency. The approach creatively leverages the noise-robust nature of SGD, offering a fresh perspective on fault-tolerant training.
2. The paper provides a robust convergence analysis, demonstrating that MeCeFO achieves convergence rates comparable to standard distributed SGD under mild assumptions. This theoretical backing strengthens confidence in the algorithm’s reliability.
3. The experimental results are compelling, showing MeCeFO’s superior resilience to frequent failures and maintaining high training throughput across various model sizes and failure settings. The inclusion of ablation studies in the appendix further supports the robustness of the approach.
4. The optimization techniques are well-reasoned, effectively addressing the trade-off between computational overhead and fault tolerance. The approximation-based strategy is particularly noted for its practical relevance in large-scale training.

**Weaknesses**:

1. Reviewers highlight that the evaluation focuses primarily on final perplexity without assessing downstream transfer capabilities (e.g., zero-shot or few-shot performance) or generative quality metrics like BLEU. This raises concerns about the generalizability of the final weights.
2. The assumption of sparse, uniformly random failures is criticized as unrealistic, as real-world failures often cluster on specific nodes, potentially leading to training bias or instability. The paper lacks analysis of how MeCeFO handles non-i.i.d. failure patterns or asymmetric loads.
3. While the convergence analysis is robust for average-case scenarios, it lacks worst-case analysis, particularly regarding error amplification in early training or critical layers. And it is questioned how the theoretical guarantee accounts for varying failure rates or statistical failure models.
4. MeCeFO is tailored to specific hybrid parallel strategies (e.g., DP group recomputation), but its applicability to tensor parallelism (TP) groups is unclear. Reviewers note that recomputation of entire TP/PP groups may involve healthy nodes, potentially increasing overhead.
5. The empirical evaluation assumes idealized failure models (constant frequency) and lacks discussion of varied failure causes, such as network flips or hardware replacements, which could affect recovery times and frequencies. The choice of node recovery times (up to an hour) is not well-justified, and shorter failure intervals (e.g., 5–15 minutes) are not explored.

**After Rebuttal,** all reviewers give positive scores of this paper and most of concerns are addressed. Thus, I recommend Accept.

However, the paper is not recommended for a spotlight or oral presentation due to significant concerns about the scope of its evaluation and assumptions. The algorithm modifies the training process substantially, yet the experiments lack large-scale validation to confirm its effectiveness in real-world, diverse failure scenarios. The absence of downstream task evaluations and the reliance on idealized failure models limit confidence in its generalizability. Additionally, the lack of worst-case theoretical analysis and clarity on handling varied parallelism strategies (e.g., TP groups) suggests that further investigation is needed to establish its robustness across practical settings.
Decision: Accept (poster)