SpareTrain: Fault-Tolerant LLM Training via Low-Cost Dual Modular Redundancy
Authors: Rihae Park
Yeonjae Kim
Seung Yul Lee
Yeonhong Park
Jae W. Lee
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
distributed training
silent data corruption
fault-tolerance
activation checkpointing
parallelism
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: SpareTrain is a novel framework that achieves full DMR protection for LLM training with negligible overhead, by repurposing activation checkpointing and exploiting idle GPU time to preserve throughput.

Abstract: Dual Modular Redundancy (DMR) is a highly effective mechanism for detecting silent data corruption (SDC)—a critical reliability concern in large language model (LLM) training—by executing each operation twice. However, its high computation overhead has prevented practical deployment at scale. In this paper, we present SpareTrain, an LLM training system that achieves complete DMR with minimal overhead by repurposing the activation checkpointing mechanism and exploiting idle GPU time. Evaluations on up to 32 H200 GPUs show that SpareTrain improves throughput by 12–35\% over naive DMR, corresponding to only 3–14\% overhead compared to unprotected training, while maintaining full DMR error detection capabilities.

Published as a conference paper at ICLR 2026
SPARETRAIN: FAULT-TOLERANT LLM TRAINING VIA Low-CoST DUAL MODULAR REDUNDANCY
Rihae Park, Yeonjae Kim, Seung Yul Lee, Yeonhong Park, Jae W. Lee Seoul National University (rihae, duswol 1 120, tri criomphantl, pateett,,111191 111196, jaesleawweeee e (eessu.ac... ac.kr
ABSTRACT
Dual Modular Redundancy (DMR) is highly effective mechanism for detect- ing silent data corruption (SDC)-a critical reliability concern in large language model (L.M)) training-by executing each operation twice. However, its high computation overhead has prevented practical deployment at scale. In this per, we present SpareTrain, an L.M training system that achieves complete DMR with minimal overhead by repurposing the activation checkpointing mechanism and exploiting idle GPU time. Evaluations on up to 32 H200 GPUs show that SrareTrain improves throughput by 12 35% over naïve DMR, corresponding only 3-14% overhead compared to unprotected training, while maintaining full DMR error detection capabilities.
INTRODUCTION
Large language models (LLMs) are transforming daily life (Grattafiori et al., 2024 Achiam et al.,
2023
Yang et al. 2024). Behind this remarkable success lies the tremendous training cost. Training [.LMs requires massive compute clusters and often takes several months to complete over large-
scale datasets (Grattatoori et al. 2024 Achiam et al. 2023 Yang et al. 2024 Isaev et al. 2023
Laurençon et al. (022).
Recently, reliability has become serious concern for LLM training. as even single bit-flip can derail months-long runs. Particularly, addressing Silent Data Corruption (SDC)-erve not caught by intrinsic hardware mechanisms such as ECC or CRC has emerged as a critical challenge due to the difficulty of detecting them (He et al. 2023 Ma et al. 2025 Bonderson 2021).
Dual Modular Redundancy (DMR), which executes the same computations twice and compares the outputs, is the most effective way to detect SDCs (Reinhardt & Mukherjee| 2000 Jeon & An- navaram 2012 Dixit et al., 2021 Ma et al.) 2025). Despite its completeness, the use of DMR in many practical settings limited due its overhead. DMR effectively doubles the computational cost. This leads to the key research question that this paper addresses: how can we minimize the L.MM training throughput loss caused by DMR without compromising its SDC detection capability?
To answer this research question, this paper investigates two key strategies: Piggyback-DMR (P- DMR) and Deferred-DMR (D-DMR), P-DMR piggybacks DMR onto the inherent redundancc intro- duced by activation checkpointing- de facto standard for memory savings in LLM training (Isaev) al. 2023 Liang al, 2025a MosaicML] 2023 Narayanan al.) 2021 D-DMR leverages the fact that GPU compute idle cycles constitute a significant portion of training time, primarily due to the overheads of parallelism strategies used in distributed training. By moving redundant computa-
tions for DMR into these idle cycles, it becomes possible to minimize the latency increase due to DMR.
Building these two strategies, we propose SpareTraai,, which systematically combines P-DMR and D-DMR to maximize latency savings while carefully managing their potential side effect: mem- ory overhead. In our evaluation on Llama-3-70B, Mistral-Large, and Llama-4-Scout (70B-123B parameters) using up to 32 H200 GPUs under various memory setups. SpareTrain improves the throughput of a DMR-protected LLM training system by up 35%, 29%, and 16%, respectively, while fully preserving detection capability. These gains translate to only 3-14% overhead compared to baseline training without SDC protection.

---

Published as a conference paper at ICLR 2026
2
BACKGROUND
2.1 SILENT DATA CORRUPTIONS IN LLM TRAINING
Silent Data Corruptions (SDCs) are hardware-induced errors that evade detection, allowing cor- rupted values to propagate silently and compromise application correctness. These errors have be- come increasingly problematic due to factors such as the growing fragility of hardware components from aggressive technology scaling. Processors such as CPUs and GPUs are more vulnerable to SDCs than memory systems because they have limited protection mechanisms, making them the primary source of SDCs in modern computing systems (He et al. 2023 Mitra et al. 2025 Dixit al. 2021 2022).
Although SDCs are rare, they pose a significant risk to LLM training in datacenters, where runs can span months across hundreds of thousands of GPUs (Bonderson, 202T). Recent technical reports and industry papers indicate that when they occur, SDCs can significantly degrade model accuracy and destabilize training convergence (He et al. 2023 Laurençon et al. 2022 Grattatiori al. 2024
Team et al. 2023). While SDC-induced gradient noise within a single optimizer step may appear negligible, large-scale studies show that its accumulation
time can cause parameter divergence
and convergence to yield suboptimal models (Ma et al. (025). Other studies further challenge the conventional belief that magnntue errors are harmless, showing that even minor SDCs can irreversibly degrade model quality (He et al. 2023
2.2 SOFTWARE TECHNIQUES FOR DETECTING SDCs
Software-leve SDC detection spans spectrum of approaches, trading detection guarantees against computational efficiency. At one end are expensive exact techniques; at the other are lightweight methods that lower overhead the cost of incomplete coverage.
Dual Modular Redundancy. Dual Modular Redundancy (DMR) detects errors by executing each operation twice-once as the primary execution and once as the checker execution-ant comparing their results. A key advantage of DMR is its generality-it can be applied to any type of operation, whether linear or non-linear. Under the standard assumption that identical faults in both executions are negligibly probable, DMR achieves complete error detection. However, DMR also has a critical limitation: it incurs substantial computational overhead approximately 100% since each opera- tion must be performed twice.
For decades, efficient DMR implemennations have been extensively studied across hardware, com- piler, and software layers (Austin 1999 Reinhardt & Mukherjee 20000 Mukherjee et al., 2002
Yim et al. [201 Wang et al. 2007 Wadden et al. | 2014 Reis et al. 2005 Didehban & Shrivastava
2016 Oh et al. 2002 Jeon & Annavaram 2012, Abdel-Majeed et al. 2015). However, none of
these works target large-scale LLM training. despite its growing importance for reliability. Industry reports have mentioned the use of like techniques for enhancing reliability, underscoring its
potential relevance in this domain (Team et al., 2023 Ma et al. (202).. Nevertheless, likely due to the high overheads, no deployment has achieved full coverage DMR.
Algorithm-Basee -Based Fault Tolerance. For linear algebra kernels such as General Matrix Multiplica- tion (GEMM), Algorithm-Based Fault Tolerance (ABFT) (Huang & Abraham 1984) offers more efficient alternative by embedding checksum- based invariants into the computation. ABFT can de- tect and, in some cases, correct errors with far lower overhead than DMR, assuming at most one fault occurs per GEMM invocation. However, when applied in conjunction with seduced-precision formats such as FP16, BF16, and FP8-which have become the de facto standard technique LLM training- the limited numerical precision can compromise the accuracy of ABFT, potentially lead- ing to false negatives (Ma et al. 2025). As result, despite its high computational overhead, DMR remains the more robust and reliable option for building SDC-free LLM training systems.
Approximate Approaches. A number of recent works propose even lighter- non-exact meth- ods tailored to ML workloads. For example, He et al. (2023) flags extreme gradient outliers as po- tential SDCs, and other approaches train small neural networks as SDC detectors (Ma et al. 2024). A more specialized line of work focuses on LLMs, such as Liang et al. (2025b). which detects and corrects anomalous outputs with just 7% end-to-end overhead by targeting only computation in specific layers like attention. While these methods demonstrate promising results, their selective
2

---

Published as a conference paper at ICLR 2026
coverage and probabilistic nature limit their applicability production environments that require strong reliability guarantees.
Among software-level techniques, DMR remains the only method that guarantec complete error detection for arbitrary LLM training workloads. However, its perceived high computational cost has led to systematic exclusion from practical large-scale training considerations. This motivates our work, which aims to make DMR practical without sacrificing its detection guarantees.
3 STRATEGIES FOR REDUCING DMR OVERHEAD IN LLM TRAINING
naive application of DMR would roughly double each operation's 's execution time due to checker execution overhead. However, large-scale LLM training exhibits characteristics that can mitigate this cost. In this section, we identify two complementary strategies: 1) Piggyback-DMe (P-DMR), which exploits redundancy already present in activation checkpointing, and 2) Deferred-DMR (D-DMR), which utilizes GPU idle periods to hide checker execution overheads.
3.1 P-DMR: LEVERAGING REDUNDANCY IN ACTIVATION CHECKPOINTING
(a) No Activation Checkpointing (AC)
(a) Naive DMR
Forward
Backward
Forward
Backward
Compute
MEM Compute
Each operation doubles the computation time.
MEM
(b) With Activation Checkpointing (AC)
(b) Apply P-DMR to AC Segments
Backward
Backward
MEM Compute
AC
AC
MEM Compute
Primory Exicuutio
Checker Evedution
Segment
Seament
Bockward Seginiet
Wockward AC Soginiens
Outpuc
Output Camparison
Camplatiao
Figure 1: Activation Checkpointing (AC)
Figure 2: Naive DMR vs. Piggyback-DMB
Activation Checkpointing. Activation Checkpointing (AC) a popular technique for reducing memory overhead from intermediate activations. Figure [ illustrates its concept by comparing execution without AC (a) and with AC (b). With AC, consecutive forward operations are grouped into AC segments (e.g., F1 F4,, F6 F-FF8), where only their inputs-known as checkpoints (e.g., A0, A5)-are stored, while the other intermediate activations are discarded and recomputed during the backward pass. AC can be applied fully, where all operations form single segment, or selectively, where multiple small segments are created and some operations (c.g., B5) remain outside any segment. How segments
Opportunity:
formed depends factors such as the memory budget and model structure, but AC has become a de facto standard in LLM training for its substantial memory savings (Isaev al. 2023 Liang et al. 2025a MosaicML
2023 Narayanan et (021).
Proposed Strategy: Piggyback-DMR (P-DMR). Under AC, all operations within segment are re- executed during the backward pass, except for the last operation whose outputs are typically already retained. Piggyback-DMP (P-DMR) leverages these recomputations as natural checker executions: the forward pass serves as the primary exccution, while the backward ccomputation-aaugeeee with one extra run of the last operation-serve serves as the checker exccution. The resulting outputs are then compared with those from the forward pass, completing DMR coverage for the entire segment. Figure [ contrasts naïve DMR and P-DMR under the same AC segment configuration as Figure []] with additional computation for DMR shown yellow. In naive DMR, every operation requires its own checker execution, doubling the computation time. In contrast, P-DMR adds only one extra execution of the last operation per AC segment- F8 or F4 in the backward pass cosp-eeingn the checker execution for the segment. Since the outputs of each segment's last operation are already stored in the forward pass in most cases, retaining them for comparison incurs no additional memory overhead. Detailed discussion is provided in Appendix
3

---

Published as a conference paper at ICLR 2026
3.2
D-DMR: LEVERAGING GPU IDLE TIME
100%
2.16
Misc
Stage 0
F2
F3
B1
B2
83
84
4-04-
Stage
F4
C PP Bubblc
50.97
Stage
F4
56 26.22
Inter Comm.
Stage 3
F1
B1
F2
82
F3
F4
84
40%
PP Bubéle
PP Comm.
Block
Forward
Block
Block 3
PP Comm.
Comm.
20%
0%
21.09
8.82
15.65
C Forward Compute
Backward Compute
Compute
Comm.
(a)
(b)
Comm.
Computte
Figure 4: Training time break-
down for (a) Mistral-Larg Large and
Figure 3: Compute and communication timeline of LLM training.
(b) Llama-4-Scout
Opportunity: GPU idle time. Training LLMs requires large-scale GPU clusters that employ mul- tiple parallelism strategies, including data parallelism (DP), pipeline parallelism (PP), tensor paral- lelism (TP), and expert parallelism (EP) for Mixture of Experts (MoE) models. While essential for scaling, these strategies introduce substantial GPU compute idle time from two main sources (Isaev ct al., 2023 Liu al. 2023b Pati al. 2023 Wang al. 2024 Feng et al. 2025). The first is communication overhead, both intra- and inter-node. TP and EP involve frequent, high-volume communication, they typically confined within single node, relying on high-bandwidth intra- node interconnects (Jin et al. 2025 Liu et al.) 2025). DP and PP often span across nodes. Second, pipeline stage dependency in PP forces GPUs wait for preceding stages, creating idle periods known as PP bubbles. Such communication patterns and PP bubbles are illustrated with an example of LLM training with four PP stages Figure
We quantify idle time by analyzing the training time breakdown of two representative LLM archi- tectures: Mistral-Large (a dense model) and Llama-4-Scout (an MoE model), as shown Figure Both models are trained on clusters of H200 nodes (8 GPUs per node). Mistral-Large is configured with PP=3, TP=8, and batch size=64 on three nodes, while Llama-4-Scout configured with PP=4, TP=8 for non-expert layers, EP=8 for experts, and batch size=64 on four nodes. Both models exhibit substantial idle time (68% and 77%, respectively), which corresponds to the combined fraction of communication and PP bubbles.
Deferred-DMe. (D-DMR). Deferred-DMRR (D-DMR) leverages GPU idle time to hide checker execution overhead. Instead of rumning checker executions immediately after pri- mary executions (as in naïve DMR), D-DMR defers them to later idle periods. While this reduces throughput loss, it introduces memory overhead since input and output pairs must be retained until checker executions, requiring careful balance between performance and memory.
Proposed Strategy:
In D-DMR, only communication-indune lize due to their infrequency and uneven distribution (see Figure 3). Thus, D-DMR defers checker executions by overlapping them with communication windows, and further distinguishes between D-DMRinter- which overlaps with inter-node communication (c.g., PP), and D-DMR which overlaps with intra-node communication (e.g.. TP, EP).
idle time is exploited, PP bubbles difficult uti-
SPARETRAIN OVERVIEW
Training job Configurations
Offline Phase
Online Phase
Phosel
Phase2
Phase3 Fine- grained D-DMR
Coanse grained D-DMR
Static Planner
co
P-DMR
Profiler
Execution Engine
Generator
Model
Phose 1
Phoses (Saticc)
Dynamic Planner
Cluster Paral elism
Phose2
Static DMR plan
model
Profiled results
Phose3 DDyomicc
MoË model
Memory budget
Static
Dynamic
Figure 6: Three phases of
Figure Spare overview.
DMR planning.
4

---

Published as a conference paper at ICLR 2026
We present SpareTrrain, novel framework for lightweight, SDC-resilient LLM training that enables full DMR with minimal overhead. The central design question for SpareTrain is determining how to utilize P-DMR and D-DMR to minimize DMR cost, which refer to DMR plan. A DMR plan assigns each operation to one of four sets: P-DMR, D-DMR D-DMM or Naive-DMR, corresponding to the sets of operations verified by P-DMR, D-DMR D-DMR and na've DMR, respectively. For operations assigned to D-DMR and D-DMRintros the planner must also determine the specific idle window for placing the checker execution.
Figure S presents high-level overview of SpareTrrann Notably, SpareTrrain consists of both an of- fline and an online component, each with its own planner: a static planner and dynamic planner, both responsible for generating DMR plans.
In the offline phase, a profiler executes a few iterations of the training job under the given config- uration to collect information required for planning. This includes (1) the activation checkpointing (AC) configuration, i.e., how operations have been segmented into AC segments, (2) the execution time and memory usage of each operation, (3) the timing and duration of GPU idle windows, and (4) per-device memory usage patterns. The static planner then uses this information to generate a plan and rewrite the training job code accordingly.
During runtime, the execution engine runs the generated code. If mismatch detected on any device during DMR checks, the entire iteration is rolled back and re-executed. If there are opera- tions that cannot be planned offline (c.g., MoE layers), the dynamic planner monitors computation and communication at runtime to dynamically determine and execute DMR plans, opportunistically applying D-DMR.
5 DMR PLANNING
Figure 6 shows high-level overview of the DMR planning process in SpareTrain, which forms its core mechanism. DMR planning in SpareTrrain is composed of three phases. In the first phase, P-DMR is planned by assigning operations to P-DMR. The second phase performs coarse- coarse-grained assignment for D-DMR, grouping operations and assigning them D-DMR The third phase then handles the remaining operations with fine-grained D-DMR, assigning each operation individ- ually to either D-DMRRitter DD-DMR Any operations not assigned in these phases naturally fall back to Naive-DMR. For non-MoE models, where all operations are fixed, all three phases are performed by the static planner. For MoE models, however, Phase is carried out by the dynamic planner to handle iteration-dependen variability.
Forward
Backward
AGC Segment
(a) w/o DMR
Compute Comm.
Ar Segmentt
Ao
A
R:
[b]
Compute
0
BA,
naive DMR
Comm.
8.
(a) P-DMR
Compute
A
B
A
ow:
aW2
aw;
aw,
Execution Time
Condition for use of P-DMR:
w/o Checkpointing:
As - Au, AA -A,, ow, aW, (14)
1
2x ExRecompute Additional Comm. for final op
+ 2x > E
w/ Checkpointing: Ag. Av. dAv. OA aw, ow, (8)
Figure 7: Exception case of AC segment where naive DMR outperforms P-DMR.
Figure 8: Concept of Activation Gradient Checkpointing (AGC).
5.1 PIASE I: PLANNING FOR P-DMR BY STATIC PLANNER
General Rule. By default, the static planner applies P-DMR for all operations recomputed by AC. P-DMR usually beneficial since the only extra cost per AC segment is the last operation (e.g., F8, F4 in Figure bb).
Exception Case. In certain cases, naive DMR is preferable to P-DMR. This occurs when the last operation of an AC segment requires expensive communication to gather its input operands. With
5

---

Published as a conference paper at ICLR 2026
naive DMR, since the two redundant executions occur back-to-back, the input operands need to be gathered only once and can then be reused. In contrast, P-DMR requires replaying the communica- tion because the two executions are separated in time.
Figure illustrates such an exception case, comparing execution without DMR (a), naive DMR (b), and P-DMR (c) for AC segment including operations A and B, where B involves communi- cation denoted as B- The additional overhead of naïve DMR is 2 x A + B. P-DMR eliminates the duplication of A but requires replaying Be, resulting in B. B. When Be 2 x A, naïve DMR is preferable, so the planner selects it. While this example segment is small with only operation A being recomputed, general, A can represent the collection of all operations that are recomputed.
For MoE Models. For AC segments consisting only of static operations, the same procedure applies as in non-MoE models. If dynamic operations (e.g., MoE layers) are included in an AC segment, exact cost calculation and comparison may not be feasible, so the planner defaults to P-DMR.
5.2 PHASE 2: COARSE-GRAINED PLANNING FOR D-DMR BY STATIC PLANNER
AativationGrradien Checkpointing (AGC). To make D-DMR efficient, the planner first applies a coarse goarse-grained strategy: it groups multiple connected operations in the computation graph into single segment and defers them together. In this way, only the segment's 's boundary tensors (inputs/outputs) are needed to replay the entire segment and verify results. We call this Activa- tion-Grudient Checkpointing (AGC), by analogy activation checkpointing (AC), but extended to the backward pass, where both activations and gradients are checkpointed.
Figure 8 illustrates the idea simple graph with four forward ops (F*) and their corresponding backward (B*) and weight-gradin (W") operations. Under operation-level deferral (every operation deferred individually), the inputs and outputs of each operation must be retained- tensors in total. In contrast, treating the same set as one AGC segment requires retaining only the boundary tensors- in total. During the checker execution, the segment replayed from the stored inputs, and the computed outputs are compared with the stored outputs, completing DMR.
PP Comm
Block1
Block2
Block3
PP Comm
Constraints
Protected Region by P-DMR
15
Memory Headroom (per Block) Time Slack (per Block) 5
Segment
Segment #1
Segment #3
Ag A
Ao
A
Step
Step 3
Step 4
0A1
aAq
Segment Candidates
Execution
0.A
Memory
Gain
Time
W44
8>6 6
12
9
Step
#2
5
4
3
aw;
aw2
aw4
dW;
Figure 9; AGC segment selection process.
AGC-Based D-DMR Planning. AGC greatly reduces D-DMR's memory overhead but introduces additional complexity in configuring AGC segments, and D-DMR windows are typically too short to accommodate sizeable segments. Therefore, the static planner selects only one AGC seg- ment per transformer block and assigns it to D-DMR scheduling it in the PP communication window immediately after the corresponding backward stage.
Accordingly, the goal of this phase is to select the AGC segment that maximizes time saving for checker executions, subject to two constraints: (i) memory headroom, defined as the available budget beyond the peak usage from the stage through the following PP communication, and (ii) time slack, the inter-node communication window available to the block. While a larger segment offers greater savings, it also increases the likelihood of violating memory or time slack constraints, necessitating a careful selection.
The selection process consists of four steps: Step enumerates all possible segments within a block; Step 2 filters out segments that exceed the memory headroom constraint; Step eliminates those that violate the time slack constraint; and Step 4 selects, from the remaining candidates, the segment

---

Published as a conference paper at ICLR 2026
with the greatest gain. Here, gain is defined as the checker execution time saved: the replay time of the segment minus any extra costs (e.g., operations already protected by P-DMR, or additional intra-segment communication).
Figure illustrates the overall selection process. Unit memory and execution time are assumed for all tensors and operations. The stage has total memory headroom of 18 and upcoming PP commu- nication time of 15. Dividing evenly across three blocks gives a memory headroom of 18 = 6 and per block. Three candidate segments are shown for brevity. Segment #1
time slack of 15 violates the memory constraint (requires 8 tensors: Ao, As, 0 A O A4, lates the time slack constraint (replay time 6: F3, B3, W3, F4, B4, W4). Segment #2 is feasible and thus compared against other feasible candidates; among them, the one yielding the largest gain is selected. For Segment #2, the replay time is 4 with an extra cost of from re- re-protecting F1, yielding gain of 3 (= 4 - 1). An example selection result from our evaluation is provided in Appendix B
Segment #3 vio-
For MoE Models. AGC segments are selected only from the static parts of the computation graph, and memory headroom is conservatively set based on the maximum usage observed during profiling.
5.3 PHASE 3: TINE-GRAINEL PLANNING FOR D-DMR BY STATIC OR DYNAMIC PLANNER
Phase 3 assigns the operations not covered by Phases 1-2 in fine- fine-grained manner (i.e., per- operation). These remaining operations are considered for deferral into unused communication win- dows, including intra-node communication time within each stage inter-node (PP) communication time immediately following each stage that remains after Phase 2. In this phase, operation-level D- DMR is managed through D-DMR queue. How operations are enqueued and scheduled differs between non-MoE and MoE models.
For Non-MoE Models. For non-MoE models, the static planner executes this phase. It scans op- erations in order of execution. When not-yet-used communication window (scheduling point) is reached, the following steps are performed: Step tentatively defers all unassigned operations since the last scheduling point and checks whether the resulting D-DMR queue exceeds the memory bud- get. Step keeps all operations deferred if within budget; otherwise, demotes shortest-runtime op- erations to DMRR until the budget is met. Step chooses the subset that best fits the current window, assigns those to D-DMR (or D-DMRinter). and continues to the next scheduling point.
Scheduling point 1 (WI)
Scheduling point (W2)
Initial state: D-DMR queue = 0 (empty).
Initial state: D-DMR queue = (4).
D-DMR queue (A, B).
D-DMR queue = (A C, C.D.EPP.
Step 1
Step
Memory check: qqueee] (pass).
Memory check: [queue] 55 (fall).
Compute
A
B
D
E
Computte
A
B
B W17
D
F
Comm.
Comm
Keep deferred.
Damate the shortest-runtinne ops (C, E) to naive DMR.
Step 2
queue ()
Step
D-DMR queue (4, D,
Compute
^
B
E
F
Compute
A
B
B
C
F
Comm
Assign the best-fit subset (8) to W1. W1, D-DMR queue (4).
Assign the best-fit subset (4, to w2. After w2, D-DMR queue (F).
Step3
Step
Compute
A
D
E
Compute
B
C
Comm.
W1
Comm.
WX
Figure 10: Fine-grained (i.e., per-operation) planning for D-DMR of static DMR planner.
Figure I0 illustrates this process, assuming memory budget equal to the input/output tensor size of three operations, with all operations (A-F) initially unassigned and all communication windows (W1 and W2) unused. At the first scheduling point (W1), no demotion occurs since the memory budget is not violated, and is assigned to W1. At the second scheduling point (W2), the memory budget is exceeded, so C and E are demoted to Natve DMR, and A and D are assigned to W2.
For MoE Models. For MoE models, the dynamic planner- while less efficient than the static plan- (see Appendix C.1 for details)-perrfrmm Phase 3 at runtime. When executing primary execu-

---

Published as a conference paper at ICLR 2026
tions of operations that are not yet assigned, the dynamic planner enqueues their inputs and outputs into the D-DMR queue. Upon encountering a communication window, the planner uses it as fol- lows: for static windows (e.g., TP), it selects best-fit subset from the queue and launches those operations; for dynamic windows (e.g. HP), it dequeues and executes operations one by one until the window closes the queue empties. However, the queue cannot grow indefinitely. The dynamic planner continuously monitors memory usage, and if usage nears the limit, it immediately dequeues and executes checker operations (i.c., falls back to Naive-DMR) satisfy the memory constraint.
6 EVALUATION
We implement SpareTraie built on PyTorch 2.9, extending its training stack with new error detection capabilities to support DMR verification during training. Section 6.1 details the evaluation training setup; Section 6.2 presents end-to-end throughput results; and Section 6.3 quantifies the contribution of each component via ablation studies.
In addition, Appendix
describes our implementation decision enabling effective D-DMR provides further experimental setup details. As part of our extended experiments, Appendix E. presents a sensitivity study on sequence length; Appendix F.22 presents an evaluation of Spare' Train combined with asyncTT,, which also leverages GPU idle time in TP; and Appendix E.3 explores alternative parallelism strategies for MoE models.
C.2
and Appendix D
6.1 SETUP
Hardware Configurations. We conduct experiments on a GPU cluster with up to four nodes. Each node is equipped with 8 NVIDIA H200-SXM5 GPUs (141GB memory per GPU). Inter-node com- munication uses InfiniBand (per node: 8 x 400Gb/s links) with GPUDirect RDMA enabled, while intra-node GPU-GPU communication leverages NVSwitch with 900GB/s bandwidth.
To evaluate SpareTraine under varying memory budgets (i.e., different GPU memory capacities), we emulate 80GB and 94GB configurations by capping each GPU's s available memory. These settings correspond to the two NVIDIA H100-SXM5 variants, another widely adopted GPU line for LLM training. Because H100 and H200 share the same compute capabilities and cache hierarchy and differ primarily in memory specifications, this emulation reasonably reflects practical deployments on H100 GPUs.
Model and Training Configurationss We evaluate SpareTrain on both dense and MoE models. For dense models we use Llama-3-70B and Mistral-Large (123B). For the MoE model we use Llama- 4-Scout (109B). Dense models are trained using combination of TP and PP. For the MoE model, expert layers use EP and non-expert layers use TP. TP and EP are confined within a node and set to degree eight (matching the eight GPUs per node), while PP spans across nodes. To determine practical setups, we sweep the PP degree (up to four) and AC degree, selecting the configuration that achieves the highest throughput for each model. All training runs use the TorchTitan frame- work (Liang et al., 2025a) with muxxed-precisiog training (Micikevicius et al. 2018), and enable torch. . compile across all setups.
6.2 TRAINING THROUGHPUT
Figure presents training throughput, measured in tokens per second, of No-DMR (a vanilla train- ing system without DMR), Naive Only (naive DMR applied all operations), and Spare across different memory capacities and batch sizes. We highlight key observations from the results below.
Dense Models (Llama-3-70B and Mistral-Large). For Llama-3-70B, averaged over the evaluated batch sizes, Spare'Train achieves 32%, 31 %,, and 33% higher throughput than Naive-Only at 80, 94, and 141GB, respectively, while trailing No-DMR by 3.3%, 13.8%, and 10.9%. The relatively smaller gap at 80GB arises because this configuration requires higher PP degree (PP-3, compared to PP-2 at 94/141GB) due to memory constraints. The increased PP degree expands communication windows, which in turn allows for more aggressive D-DMR The same trend is observed for Mistral-Large. Averaged over the evaluated batch sizes, SpareTrain improves throughput by 26%, 21% %, and 23% over Naive-Only, while incurring slowdowns of 6. 1,, 11.0%, and 9.8% relative
8

---

Published as a conference paper at ICLR 2026
No-DMR
Naive-Only
SpareTrain
Llma-33-708
Mistral-Large
Llama-4-Scout
80GB
Tokens/Sec
Tokens/ s/Sec
200
300
161
227
181
240
192
252
264
Tokens/Sec Tokens/ SSec
100
122
92
125
130
129
100
Tokens/Sec
OOM
100
PP=3
PP=4
0
Tokens/Ses Sec
400
367
200
163
Tokens/Se /Sec
400
94GB
200
300
299
240
324
100
121
139
120
144
200
-244
292
102
184
100
PP=2
PP=3
PP=d
16
141GB
Tokens/Sec
400
361
439
358
445
Tokens
150
100
149
4....
130
165
166
166
139
Tokens/Se
200
300
292
257
342
276
330
284
200
286
296
50
100
PP=2
PP=3
PP=4
0
32
32
64
Batch Size
Batch Size
Batch Size
Figure 11: Training throughput of No-DMR, Natve-Only and SpareTrain.
No-DMR respectively. In summary. SpareTrain accelerates DMR-protected training by roughly 30%. Put differently, it enables full DMR with only about a 10% slowdown.
MoE Model (Llama-4-Scout), For Llama-4-Scout, averaged over the evaluated batch sizes, Spare- Train improves throughput over Nalve-Only by 11% and 14% at 94GB and 141GB, while incurring slowdowns of 9.8% and 8. relative No-DMR, respectively. The 80GB setup is excluded due to out-of-memory (OOM) errors. The performance gaps among Naive-Only, SpareTrain, and No- DMR are relatively smaller for MoE models than for dense models. This is because the relative DMR overhead is lower: MoE models incur higher communication costs relative to computation, thus DMR contributes less to the overall runtime. Nevertheless, Spare still effectively reduces DMR overhead.
6.3 ABLATION STUDY
Figure 12 demonstrates how each phase
I Phase
M Phase 1+2
Phase 1+2+3
of DMR planning contributes to training throughput improvement, For the dense model (Mistral-Large), we progressively enable the three phases of the static DMR planner, and for we instead
Mistral-Large (80G, PP-4]
Llama-4 4-Scout (141G, PP=4)
Improvement (S]]
Throughout
20
30
22
2911
23.
233
17.
28.2
16
13.7
16.5
20
the MоE model enable Phases 1 and 2 of the static planner fol- lowed by the dynamic planner. All three phases significantly contribute
16
32
64
16
32
64
Batch
Batch
improving training throughput. This implies that both P-DMR and D-DMR play critical roles, and that within D. DMR, both fine- grained and coarse- -raained planning are essential.
Figure 12: Ablation study of the three phases of DMR planning in SpareTrain.
7 CONCLUSION
In this paper. we present SpareTrain, novel approach to achieving exact silent data corruption de- tection in LLM training with minimal overhead. By leveraging the inherent redundancy in activation checkpointing and GPU under-utilization, our system demonstrates that DMR can be practically applied without prohibitive costs. As LLMs continue scale to larger models, Spare Train enables reliable training while maintaining computational efficiency, bridging the gap between theoretical fault tolerance mechanisms and practical deployment requirements.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Samsung Electronies Co., Ltd. (Device Solutions) and the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemmeett (MSIT) (RS-2025-02217656) Jae W. Lee is the corresponding author.
REFERENCES
Mohammad Abdel-Majeed, Waleed Dweik, Hyeran Jeon, and Murali Annavaram. Warped-re: Low- cost enror detection and correction in gpus. In 2015 45th Annual IEEE/IFIP International Con- ference on Dependable Systems and Networks, pp. 331-342. IEEE, 2015.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Todd M Austin. Diva: A reliable substrate for deep submicron microarchitecttue design. In MICRO
32. Proceedings of the 32nd Annual ACM/IEEE International Symposium Microarchitecture, pp. 196-207, IEEE, 1999.
Rich Bonderson. Training in turmoil: Silent data corruption in systems at scale. In International Test Conference Silicon Lifecycle Management Workshop, 2021.
Moslem Didehban and Aviral Shrivastava. nzde: A compiler technique for near zero silent data corruption. In Proceedings of the 53rd Anmual Design Automation Conference, pp. 1-6, 2016.
Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. Silent data corruptions at scale. arXiv preprini arXiv:2102. 2021.
Harish Dattatraya Dixit, Laura Boyle, Gautham Vunnam, Sneha Pendharkar, Matt Beadon, and Sri- ram Sankar. Detecting silent data corruptions the wild. arXiv preprint arXiv:2203.08989, 2022.
Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng. Haibin Lin, and Minlan Yu. Optimus: Ac- celerating large-scale multi-modal 11m training by bubble exploitation. In 2025 USENIX Annual Technical Conference (USENIX ATC 25), pp. 161-177, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Horace He and Shangdi Yu. Transcending runtime-memon tradeoffs in checkpointing by being fusion aware. Proceedings of Machine Learning and Systems, 5:414-427, 2023.
Yi He, Mike Hutton, Steven Chan, Robert De Gruijl, Rama Govindaraju, Nishant Patil, and Yan- jing Li. Understanding and mitigating hardware failures deep learning training systems. In Proceedings of the 50th Annual International Symposium Computer Architecture, pp. 1-16, 2023.
Kuang-Hua Huang and Jacob A Abraham. Algorithm-based fault tolerance for matrix operations. IEEE transactions on computers, 100(6):518-522 1984.
Mikhail Isaev, Nic McDonald, Larry Dennison, and Richard Vuduc. Calculon: a methodology and tool for high-level co-design of systems and large language models. In Proceedings of the In- lernational Conference for High Performance Computing, Networking, Storage and Analysis, pp- 1-14, 2023.
Abhinav Jangda, Jun Huang. Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. Breaking the computation and communication abstraction barrier in distributed machine learning workloads. In Proceedings of the 27th ACM International Conference on Architectural Support for P Programming Languages and Operating Systems, pp. 402-416, 2022.
10

---

Published as a conference paper at ICLR 2026
Hyeran Jeon and Murali Annavaram. Warped-dmit Light- enror detection for gpgpu. In 2012 45th Anmual IEEE/ACM International Symposium on Microarchitecture, pp. 37-47. IEEE, 2012.
Chao Jin, Ziheng Jiang, Zhihao Bai, Zheng Zhong, Juncai Liu, Xiang Li, Ningxin Zheng, Xi Wang, Cong Xie, Qi Huang, et al. Megascale-moe' Large-scale commumication--ffiikii training of mixture-of-exppene models in production. arXiv preprint wrXiv:2505.11422 2025.
Vijay Anand Korthikanti, Jared Casper. Sangkug Lym, Lawrence McAfee, Michael Andersch, Mo- hammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341 2023.
Patrick Labatut. Activation checkpointing techniques.
https: htaps::/pyttorc..rr//bbbb act eestivation-coe hngp.rawi...oocccccccooo.nnnnnen ques/ 2025. Accessed: 2025-08-06.
Hugo Laurençon, Thomas Saulnier, Quentin Wilezynski, and Thomas Wolf.
Train-
ing bloom with megatron- deepppped.
https://huggingffce... co. co/log///
b wow.mmaattttoodeeeeeeoee
2022.
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang. Sanket Purandare, Gokul Nadathur, and Stratos Idreos. Torchtitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth International Conference on Learning Representations, 2025a. URL https: //openzeview.net.. http://neee.//oommmomoddsoe
Yuhang Liang, Xinyi Li, Jie Ren, Ang Li, Bo Fang, and Jieyang Chen. Attnchecker: Highly- optimized fault tolerant attention for large language model training. In Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pp- 252-266, 2025b.
Dennis Liu, Zijie Yan, Xin Yao, Tong Liu, Vijay Korthikanti, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Jianbin Chang, et al. Moe parallel folding: Heterogeneous parallelism mappings for efficient large-scale scale moe model training with megatron core. arXiv preprint arXiv:2504 14960, 2025.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.01880 2023a.
Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. In Proceedings the International Con- ference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13, 2023b.
Dongning Ma, Fred Lin, Alban Desmaison, Joel Coburn, Daniel Moore, Sriram Sankar, and Xun Jiao. Dr. dna: Combating silent data corruptions deep learning using distribution of neuron activations. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 239-252, 2024.
Jeffrey Ma, Hengzhi Pei, Leonard Lausen, and George Karypis. Understanding silent data corruption in Ilm training. arXiv preprint arXiv:2502.12344 2025.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
Subhasish Mitra, Subho Banerjee. Martin Dixon, Rama Govindaraju, Peter Hochschild, Eric X Liu, Bharath Parthasarathy, and Parthasarathy Ranganathan. Silent data corruption by 10x test escapes threatens reliable computing. arXiv preprint arXiv:2508.01786 2025.
MosaicML. Benchmark measurements for mpt models trained mosaicml plat- 1m-
htttps://githus wtw.://oooommooa//// www.:///moosiiciioooooooo t Craiin hew.ii//bnnchaaaama rk i rwiig//EADE.,........
form.
README. .md 2023.
Shubhendu S Mukherjee, Michael Kontz, and Steven K Reinhardt. Detailed design and evaluation of redundant multithreading alternaties.. ACM SIGARCH Computer Architecture News, 30(2): 99-110, 2002.
11

---

Published as a conference paper at ICLR 2026
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Effi cient large-scale language model training gpu clusters using megatron-Im. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1-15, 2021.
Nahmsuk Oh, Philip P Shirvani, and Edward McCluskey. Error detection by duplicated instructions in super-scalar processors. IEEE Transactions on Reliability, 51(1):63-75, 2002.
Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D Sinclair. Com- putation vs. communication scaling for future transformers on future hardware. arXiv preprint arXiv:2302.02825, 2023.
PyTorch Team. Cuda semantics memory management.
http://dossspppt..ee oor org/ docs botos/staablenotes//ooooo html #memory management, 2025, Accessed: 2025-09-
17.
Steven K Reinhardt and Shubhendu S Mukherjee. Transient fault detection via simultaneous multi- threading. In Proceedings of the 27th annual international symposium on Computer architecture, pp. 25-36, 2000.
George A Reis, Jonathan Chang, Neil Vachharajani, Ram Rangan, and David I August. Swift: Software implemented fault tolerance. In International symposium on Code generation and opti- mization, pp. 243-2442444 IEEE, 2005.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Jack Wadden, Alexander Lyashevsky, Sudhanva Gurumurthi, Vilas Sridharan, and Kevin Skadron. Real-world design and evaluation of mompiler-managed gpu redundant multithreading. ACM SIGARCH Computer Architecture News, 42(3):73-84, 2014.
Cheng Wang, Ho-seop Kim, Youfeng Wu, and Victor Ying. Compiler-managed software-based. redundant multi-threading for transient fault detection. In International Symposium on Code Generation and Optimization (CGO'07), pp. 244-258. IEEE, 2007.
Guanhua Wang, Chengming Zhang, Zheyu Shen, Ang Li, and Olatunji Ruwase. Domino: Elimi- nating communication in Ilm training via generic tensor slicing and overlapping. arXiv preprint arXiv:2409. 15241, 2024.
Shibo Wang. Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hechtman, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, et al. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume I, pp. 93-106, 2022.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15 155, 2024.
Keun Soo Yim, Cuong Pham, Mushfiq Saleheen, Zbigniew Kalbarczyk, and Ravishankar Iyer. Hauberk: Lightweight silent data corruption error detector for gpgpu. 201 JEEE International Parallel & Distributed Processing Symposium, pp. 287-300, IEEE, 2011.
12

---

Published as a conference paper at ICLR 2026
A NEAR-ZERO MEMORY OVERHEAD OF P-DMR
P-DMR requires storing the outputs of each AC segment's primary execution for later comparison. In practice, this rarely introduces additional memory overhead. If an AC segment is followed by another AC segment within the same stage, its outputs are already needed checkpoint inputs for the next segment, Conversely, if the AC segment is followed by operations not covered by activation checkpointing. the outputs are naturally preserved in memory as inputs to those operations (since those operations retain activations for the backward pass). Thus, in most cases, P-DMR simply reuses tensors that are already in memory.
The only exception arises when the last operation of AC segment is also the last operation of a pipeline stage. Since no subsequent operation consumes these outputs, they must explicitly retained until the backward pass recomputes them for comparison. The worst-case additional reten- tion depends on the pipeline schedule and pipeline degree: when a stage processes multiple forward microbatches consecutively before their backward passes, it must keep the last outputs of all those microbatches concurrently.
Stage 0
F1
F2
F3
F4
B1
82
83
B4
Stage 1
82
F1
Stage 3
74.
B1
B2
82
F4
84
Backward Black 7
Block
Block
Black
Block
Block 6
MEM compute
Compute
Seggment.co
Segment
Additionna heckpainting
MEM
Outpur Companison
Output Comparison
Outpur
Compartson
Figure 13: Illustration of P-DMR memory retention under 1FIB pipeline schedule when a 12- block transformer model is partitioned across four pipeline stages (PP 4). The figure highlights additional memory required by P-DMR, where only the final forward stage outputs can introduce any additional memory.
Figure 13 illustrates this scenario 12-block transformer model partitioned into four pipeline stages (three blocks per stage) under a 1FIB schedule, For simplicity, we assume AC is applied at the granularity of each block. Regardless of the specific AC segmentation, the only tensor that can introduce additional memory overhead is the last output of each stage (c.g., A8 for Stage 2). In this example, Stage processes four consecutive forward microbatches (FI-F4), requiring additional memory to store four sets of outputs until their corresponding backward passes.
Formally, the extra per-GPU retention is proportional the product of bytes per element, quence length, hidden dimension, the number of stages per node, the maximum number of con- secutive forwards, and the microbatch size, all divided by the TP degree. For example, on Llama- 3-70B (sequence length 8192, hidden 8192, microbatch size 1, TP=8), a IFIB schedule with pipeline degree PP=3 adds at most 96MB per GPU. Even with Interleaved- IF (two stages per node), the overhead is most 192MB per GPU-negligibh compared to modern GPU capacities (80-141 GB) did not affect the actual peak memory usage.
In the backward pass, P-DMR may marginally extend the lifetime of certain tensors. As illustrated in Figure [3 A7 would be freed immediately after the backward of block in standard AC, but must be retained until the backward of block 7 to enable comparison. This effect does not accumulate across AC segments and persists only for a short duration, making the additional retention negligible in practice.
13

---

Published as a conference paper at ICLR 2026
B AGC SEGMENT SELECTION RESULT
In Phase 2, the AGC segment selection results for Llama-3-70B are summarized Table covering device memory budgets of 80GB, 94GB, and 141GB.
Table 1: Phase evaluation results for Llama-3-700 training, showing the actual constraints and the selected AGC segments. Memory headroom and Memory cost are denoted in MB, while PP slack, Execution time, and Time saved are denoted in ms.
Constraints (Per block)
Selected Segment (Per block)
Memory
Batch Size
PP
PP Stage
Budget
Degree
Blocks
Memory headroom
Memory cost
Execution time
Time
PP slack
saved
706
26.602
15.627
7.860
80GB
32
PP=3
27
853
13.229
104
13.178
5,441
2
27
380
25.020
136
13.360
5.659
0
40
257
10.845
120
6,939
3,484
94GB
PP=2
1
40
84
10.507
80
6.617
3.747
0
40
612
10.774
400
8.166
4.348
141GB
PP=2
40
715
10.398
400
8.146
4.372
Comparing the 80GB setting with the larger budgets, larger memory budget allows lower PP degree, which reduces per-block PP slack because inter-node communication is divided among more transformer blocks per stage. For example, each stage hosts 26-27 blocks PP=3 but about 40 blocks at PP-2; consequently, the available PP communication slack must be distributed across more blocks, shrinking the per-block slack. This, in turn, leads the planner to favor smaller segments when the PP degree is lower.
Between the 94GB and 141GB settings, the pipeline degree remains the same, so the per-block PP slack is nearly identical. However, the additional memory headroom at 141GB allows for larger segments be selected, whereas the 94GB case is restricted smaller ones.
Overall, 41-57% of the segment execution time yields actual time saved, since intra-segment com- munication and redundant operations already protected by P-DMR still need to be re-executed. Nev- ertheless, by selecting efficient segments that respect both memory headroom and PP slack, the planner effectively utilizes idle communication windows by deferring DMR work off the critical path.
C OVERLAPPING COMMUNICATION AND COMPUTATION KERNELS
C.I D-DMR IN PHASE 3: STATIC VS. DYNAMIC PLANNING
When performing D-DMR main inefficiencies arise under dynamic planning, where tain communication durations are not known in advance.
The first is the launch overhead in dynamic communication. In static communication, operations ex- ecuted under D-DMR can leverage predetermined communication durations to launch all best- fit operations from the queue simultaneously at the start of communication. However, in dynamic communication, operations under D-DMR are launched one by one until communication com- pletes, incurring repeated overhead from kernel launches and communication completion checks. Figure 14 illustrates this difference between static and dynamic communication.
Second, naïve DMR in dynamic planning requires longer memory retention than static planning. In static planning, operations designated for naïve DMR are predetermined as Naive-DMMR during the offline phase, allowing them to be executed twice back-to-back on the critical path. In dynamic planning, operations
opportunistically placed in the D-DMR queue first, and later executed with naive DMR on the critical path whenever memory resources are tight or the communication window is not long enough to allow overlap. Although the critical path execution remains the same, memory retention is prolonged because operations cannot be processed immediately due to the uncertainty of whether each operation will eventually be executed with naïve DMR.
14

---

Published as a conference paper at ICLR 2026
(a) Static Intra-node Communication
CPU
Lounch
Compute
A
B
C
TP
TP
Comm
All Gather
Reduce Scatter
(b) Dynamic Intra-node Communication
CPU
woit
wait
walt
wait
wait
wait
Signal
Compute
A
B
D
E
F
Comm
All-to-All
Figure 14: Overlap of checker executions with intra-node communication in (a) static and (b) dy- namic communication.
C.2 INTRA-NODE COMMUNICATION VIA NCCL VS. P2P
Intra-node collective communication typically performed using collective libraries such as NCCL. However, NCCL collective operations implemented device kemells that consume SM (the GPU's compute cores), leading to contention with concurrent compute kernels during overlap. As result, although computation-coommnicatii overlap appears feasible in theory, the performance gains are often minimal in practice.
In contrast, peer-to-peet (P2P) transfers operate through NVLink using dedicated copy engines (DMA) and do not consume SM resources. This substantially reduces resource contention and en- ables stable overlap. PyTorch's commetric_memory which implements P2P-based collective com- munication, may be slightly slower than highly optimized NCCL kernels, but its clear ability to overlap leads higher overall throughput than sequential execution computation and communi- cation. Therefore, when SpareTrain exploits intra-node communication windows for D-DMR we rely on PyTorch's symmetric memory instead of NCCL to achieve effective overlap.
D EXPERIMENTAL SETUP DETAILS
D.1 HARDWARE CONFIGURATION AND MEMORY CAPACITY EMULATION
Detailed hardware specifications shown in Table 2 We evaluate SpareTrain under varying ory budgets (or GPU clusters with different memory capacities). To emulate 80 GB and 94 GB con- figurations, cap each GPU's usable memory by setting upper bound all reserved memory (allocated and unallocated) managed by PyTorch's Cachi ngal locator
(Py Torch Team 2025
Table Hardware specifications of single evaluation node.
Category
Specification
CPU
x Intel Xcon Platinum 8580 (128 cores)
System Memory GPU
2,048 GB DDR5-5600 x NVIDIA H200-SXMS
GPU Memory
141 GB HBM3e per GPU
Inter-node Network
x 400 Gb/s InfiniBand (RDMA)
mode (GPU P2P)
NVSwitch, 900 GB/s
15

---

Published as a conference paper at ICLR 2026
D.2 CONFIGURING PIPELINE PARALLELISM AND ACTIVATION CHECKPOINTING DEGREES
While full activation checkpointing stores only layer inputs and recomputes all intermediate ac- tivations, selective strategies provide finer control for better memory-computttion trade-off's (He
& Yu 2023; Korthikanti et al. 2023; Labatut, 2025). PyTorch's torch, compi supports auto- mated selective checkpointing through the Memory Budget API (Labatut] 2025). Given ac ac_gedget
[0, 1]-where O enforces full checkpointing and 1 disables it-the compiler automatically finds pareto-optimal checkpointing plans.
Table Training performance of Llama-3-70B on 16 H200 GPUs (14IGB) under varying au_budget settings.
ac_budget
0.0
0.25
0.5
0.75
1.0
Tokens/Sec
376.3
421.9
438.3
439.0
436.5
Peak Memory (GB)
82.71
101.47
107.17
107.17
129.39
The performance implications of activation checkpointing, however, are nontrivial. While tighter memory budgets enforce high degree of checkpointing, relaxing the budget does not always yield higher throughput. For example, Table 3 shows training performance of Llama-3-70B on 16 H200 GPUs under PP 2, TP 8: although budget 1.0 consumes the most memory (129.39 GB), throughput peaks at ac ac_budget 0.75, which uses less memory (107.17 GB). This high- lights that the optimal degree of checkpointing cannot be derived analytically and must be identified empirically through sweeps over ac ac_budget.
Similarly, pipeline parallelism degree (PP) creates trade-offs: higher PP degree reduces per-device memory pressure but introduces communication overhead. The combined effect of PP degree and activation checkpointing degree means optimal configurations require joint empirical evaluation of both parameters to achieve best throughput under given hardware constraints.
To ensure fair comparison, establish strong baselines by sweeping all feasible PP degrees (up to 4 nodes) and ac budget values (0, 0.25, 0.5, 0.75, 1.0] for baseline (No-DMR), selecting the highest-thooughuut configuration. For Spare'rrain, we the same PP degree but independently sweep ac budget values, since ac_budget directly determines both the extent of P-DMR coverage and the available memory headroom for D-DMR execution. This sweep also required for vanilla training, so our method incurs no extra search overhead.
In practice, using the minimum PP degree that fits the model consistently outperformed configura- tions with higher PP degrees and relaxed ac budget values; therefore, the selected PP degrees are shown in Figure TT Section For ac budget, however, the optimal value can differ: for Llama- 3-70B with a 14TGB memory budget and batch size of 64, the vanilla baseline achieved its best performance with ac_budget 1.0 (no activation checkpointing), whereas SpareTrain performed optimally at acduee = 0.5, which provides better utilization of both P-DMR and D-DMR by reserving additional memory headroom.
E EXTENDED EVALUATION
E.1 EFFECT OF SEQUENCE LENGTH ON SPARETRAIN
Figure 15 shows the impact of sequence length (4K, 8K, and 16K tokens per sample) on the training throughput of SpareTrain, Experiments were conducted on Mistral-Large (PP-4, TP-8) using 32 GPUs, each with 94GB of memory. Averaged over batch sizes, the throughput of SpareTrain is lower than that of No-DMR by 4.6%, 6. 1,, and 3.1% at 4K, 8K, and 16K, respectively. These results indicate that the relative overhead of SpareTrain remains small and consistent, suggesting that its effectiveness is largely unaffected by sequence length.
E.2 SPARETRAIN PERFORMANCE UNDER COMMUNICATION COMMUNICATION-COMUUTATTTOOO OVERLAP
To mitigate the communication overhead, numerous techniques have been proposed to overlap TP communication with computation (Wang al. 2022 Jangda al. 2022) The key idea in these
16

---

Published as a conference paper at ICLR 2026
No-DMR
SpareTrain
seq len = 4k
seq len 8k
seq len 16k
Tokens/Ses Sec
150
134
Tokens/Seo
150
122
138
Tokens/Se Sec
150
115
124
128
100
116
128
133
100
125
129
100
111
119
126
50
50
50
PP=4
PP=4
PP=4
0
16
32
64
32
16
64
Batch Size
Batch Size
Batch Size
Figure 15: Training throughput of No-DMR and SpareTrrain across varying sequence length.
approaches is to decompose both communication and computation into finer-grained steps so that they can proceed in a pipelined fashion. representative implementation PyTorch's asyne tensor parallelism (asyneTP), which overlaps TP communication with its dependent GEMM computa- tion. Specifically, asyncIP breaks down collective operations (e.g., all-gather, reduce-scatter) into chunked send/recv steps (implemented via P2P communication with symmetric.memory and splits the subsequent matrix multiplication into sub-matmul kernels. This design allows each sub-matmul to run in parallel with the transfer of the next communication chunk, achieving effective overlap.
We evaluate SpareTrain under this overlap mechanism by training Mistral-Large (PP = 4, 80GB memory per GPU) with аsyncTP enabled. Table reports throughput for No-DMR and Spare- Train under different batch sizes, with results in parentheses showing the corresponding runs with- asyпcTP. On average, SpareTraan incurs only 4.2% slowdown relative No-DMR with asyncT?.
Table 4: Training throughput comparison between No-DMR and SpareTrrain with asyn enabled under different batch sizes, evaluated Mistral-Large Values in parentheses show the correspond- ing results without asyпcТP,
Batch Size
16
32
64
No-DMR
123.6 (121.8)
132.1 (130.0)
140.5 (137.5)
SpareTrrain
118.5 (112.6)
126.2 (124.5)
134.9 (128.6)
Slowdown
4.2% (7.6%)
4.5% (4.3%)
4.0% (6.5%)
Notably, the slowdown of Spare'rrain can be smaller with asyпcTP enabled. Without asyncTP, only Spare Train suffers from communication performance degradation due to switching from NCCL to P2P-based communication for D-DMR operations, while the No-DMR continues using opti- mized NCCL collectives. However, asyncTP internally replaces NCCL collectives with P2P-based communication via symmetric.mente .memory for both systems, eliminating this communication imple- mentation disparity. Although P2P communication takes slightly longer than NCCL in communica- tion time, this change does not benefit No-DMR; however, SpareTraine can leverage it as additional slack for D-DMR operations.
Therefore, in some cases, enabling asyneTP can even reduce the relative performance gap between Spare Train and the No-DMR. Overall, these results demonstrate that SpareTrain remains compatible with existing overlap mechanisms.
E.3 ALTERNATIVE PARALLELISM FOR MoE MODEL.
Context Parallelism (CP) shards the input sequence along the sequence dimension across GPUs (Liu et al. 2023a). By splitting long sequences, CP reduces per-GPU memory usage and enables train- ing with larger sequence lengths. However, unlike pnuuence-invariann operations (e.g., feed-forward networks, normalization), self-attention requires the full sequence context, which incurs additional communication of key/value tensors across GPUs.
17

---

Published as a conference paper at ICLR 2026
For MoE training, the commonly recommended configuration combines EP for MoE layers with either TP or CP for non-MoE layers (Liu et al. 2025 Jin et al. 2025). Our main evaluation adopted TP for non-MoE layers, it provided better throughput. To validate the generality of our approach, we also conducted experiments using CP for non-MoE layers while retaining EP for MoF. layers. set PP=4 across nodes, while non-MoF layers used CP with degree
In these experiments, and MoE layers used EP with degree 8, As in standard practice, these intra-node parallelism di- mensions (CP, TP, and EP) matched the number of GPUs per node mitigate the of frequent communication.
Table reports the throughput under this configuration. The results confirm that SpareTrain main- tains complete DMR coverage with only reasonable slowdowns, demonstrating that our technique effective even when CP replaces TP MoE training.
Table 5: Throughput comparison for MoE training when using CP for non-MoE layers and EP for MoE layers. Results are for Llama-4-Scoue with a 141GB GPU memory budget and PP=4.
Batch Size
4
No-DMR
160.1
229.1 222.5
296.4
SpareTraii
152.1
284.1
Slowdown
5.0%
2.9%
4.2%
E.4 P-DMR COVERAGE AND EXCEPTION CASES UNDER VARYING ACTIVATION CHECKPOINTING CONFIGURATIONS
The coverage of P-DMR depends directly on the degree of activation checkpointing (AC), which determines how many operations are recomputed during the backward pass. As described in Ap- pendix D.2, the ac au.but parameter of torch. compile provides fine-grained operation-level activation checkpointing. We analyze both P-DMR coverage for varying ac budget and the fre- quency of exception cases in the end-to-end training experiments from Section 6
Table shows P-DMR coverage for varying ac buaget values, measured on Llama-3-70B trained on H200 GPUs (141GB) with TP=8. Both rows represent percentages of total forward operation execution time: the first row shows the fraction spent on recomputed operations, and the second row shows the fraction covered by P-DMR. The coverage closely tracks the recomputation ratio. At budget 0.0 (full AC), nearly all forward operations are recomputed and thus verified by P-DMR. As budget increases, P-DMR coverage proportionally reduces.
Table 6: Recomputation ratio and P-DMR coverage for Llama-3-70B under varying ac ac.budget. Values are percentages of total forward operation exccution time.
ac_budget
0.0
0.25
0.5
0.75
1.0
Recomputed Operations (%)
99.1
55.8
22.7
22.7
0.4
Operations Covered by P-DMR (%)
99.1
50.9
21.3
21.3
0.4
Phase 1 applies P-DMR by default to all operations recomputed by AC. However, an exception case occurs when the last operation of an AC segment requires expensive communication to gather its input operands-in such cases, naïve DMR becomes preferable. Table reports the exception case ratio observed during the training runs from Section 6 for Llama-3-70B.
As described in Section 5.1
Overall, exception cases either do not occur or appear at low ratio (under 20%), highlighting the practical effectiveness of P-DMR. The results correlate with ac bu_budget: at 80GB and 94GB mem- ory budgets, sc_budget = 0.0 enforces full activation checkpointing, where each transformer layer forms a single large AC segment, so no exception cases occur. Exception cases appear only at higher memory budgets (c.g., 141GB with acJoudget 0.5), where selective activation checkpointing cre- ates smaller. fine- -rainraindd segments. In these smaller segments, communication costs can more easily outweigh recomputation costs, occasionally triggering the exception condition. Importantly, even
18

---

Published as a conference paper at ICLR 2026
an 18.2% exception ratio, the actual performance impact is smaller still, shown by the gap be- tween recomputed operations and P-DMR coverage in Table 6 segments with low computational cost.
exception cases inherently involve
Table 7: Exception ratio in Phase (P-DMR) for Llama-3-70B across memory budgets.
Memory Budget
80GB
94GB
141GB 18.2% (160/880)
Exception Case Ratio
0.0% (0/80)
0.0% (0/80)
19


Reviews:
[{"review_id": "6lnv1mxSZK", "reviewer": "Reviewer_cTfD", "summary": "Naive DMR doubles computation to detect silent data corruption. This paper SpareTrain overcomes this through two key innovations: Piggyback-DMR using activation checkpointing, and Deferred-DMR Utilizing idle GPU time and perform checker executions asynchronously. This hides redundancy latency behind existing idle periods, minimizing impact on training throughput. SpareTrain includes both offline and online planners that coordinate P-DMR and D-DMR to balance memory, timing, and redundancy coverage.", "questions": "Would love to see a comparison with checkpoint–resume solution. It remains unclear how frequently such errors materially affect final model quality compared to the dominant sources of variance (e.g., data order, learning rate tuning). When an error or hardware failure occurs, training can resume from the last saved state", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "good originality:\nReusing activation checkpoint recomputation is an creative idea and it works well with existing AC APIs. Deferred-DMR using idle GPU time is also intuitive. The combination of static and dynamic DMR planning introduces a new layer of runtime adaptivity for reliability in distributed training. It's a practical system for achieving full Dual Modular Redundancy (DMR)\n\ngood quality:\nThe paper showcased integration with Llama-3, Mistral-Large, and Llama-4-Scout and up to 32 GPUs. It carefully analyzes overhead sources, memory trade-offs, and timing windows, and conducts ablation studies showing contributions of each design phase. Evaluations demonstrate up to 35% faster than naive DMR with full detection\n\ngood Clarity\nCheckpointing vs. P-DMR, DMR scheduling diagrams are very helpful to communicate the timing and memory concepts. Despite system-level complexity, the narrative maintains good readability, with consistent terminology and careful explanation of assumptions and constraints.\n\nfair significance\nSilent data corruption is becoming a critical reliability issue when scaling to hundreds of thousands of GPUs. Making full DMR feasible at low cost has immediate industrial relevance. SpareTrain’s integration with PyTorch and TorchTitan indicates it could be adopted with minimal workflow disruption", "weaknesses": "While the paper’s DMR planner (static + dynamic) is a well-engineered mechanism for allocating operations across Piggyback-DMR and Deferred-DMR, its soundness depends on the assumption that the profiled execution characteristics remain stable throughout training. This assumption is not always valid. The static planner bases its schedule on a short profiling phase that measures activation checkpointing configuration, operation latencies, and communication windows (GPU idle times). However, these metrics can change substantially across iterations due to variations in \n* pipeline bubble patterns: which depend on load imbalance, microbatch size, and runtime scheduling\n* activation checkpointing overhead: which fluctuates with adaptive memory pressure, gradient accumulation steps, or sequence length changes;\n\nIt is less clear what's the net win of full DMR over checkpoint rollbacks. From a cost–benefit standpoint, many production training pipelines might prefer to tolerate occasional checkpoint rollbacks over continuously paying 15% runtime overhead for protection against rare SDCs. Unless the paper provides quantitative evidence that SDCs cause non-recoverable or undetectable model degradation despite checkpointing, the necessity and urgency of system-wide DMR protection remain open to question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nY0DOI41mc", "reviewer": "Reviewer_WUDn", "summary": "This paper introduces SpareTrain, a system designed to provide complete, low-cost fault tolerance for Large Language Model (LLM) training. The primary goal is to detect Silent Data Corruptions (SDCs) using Dual Modular Redundancy (DMR)—which traditionally incurs a prohibitive 100% compute overhead—at a minimal performance cost. The core contribution of SpareTrain is a set of two complementary strategies that exploit existing characteristics of modern LLM training:\n\n1. Piggyback-DMR (P-DMR): This technique cleverly repurposes the inherent computational redundancy from Activation Checkpointing (AC). The forward pass serves as the primary execution, and the re-computation of activations during the backward pass (which AC necessitates for memory savings) is used as the checker execution.\n\n2. Deferred-DMR (D-DMR): This technique leverages the significant GPU idle time present in large-scale distributed training, which arises from communication overhead (e.g., in Pipeline Parallelism (PP) and Tensor Parallelism (TP)) and pipeline bubbles. SpareTrain defers the checker execution of an operation into these idle periods, effectively hiding its computational latency.\n\nThese strategies are orchestrated by a three-phase planner that statically (for dense models) or dynamically (for MoE models) assigns operations to P-DMR, coarse-grained D-DMR (for PP bubbles), or fine-grained D-DMR (for TP/EP communication windows).", "questions": "1. Long-term Viability of D-DMR: As distributed training frameworks get better at hiding communication latency and reducing pipeline bubbles, the GPU idle time that D-DMR relies on will decrease. How do you see the effectiveness of SpareTrain changing in a future system that has near-perfect overlap and minimal idle time? Does the benefit of SpareTrain then rely almost entirely on P-DMR?\n\n2. In your experiments on Llama-3 and Mistral-Large (Section 5.1, Figure 7), how frequently did the planner's exception case trigger? That is, what percentage of AC segments were assigned to Naïve-DMR instead of P-DMR because the condition ($2 \\times \\text{Recompute} < \\text{Additional Comm.}$) was met? This would help in understanding the practical importance of P-DMR.\n\n3. Does Silent Data Corruption significantly hurt model quality? Please give us some evidence.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The core strategies are both insightful and effective. The P-DMR concept is particularly creative. Identifying that activation checkpointing already provides a \"free\" source of redundancy for fault tolerance is a non-trivial and elegant insight. It turns a memory-saving technique into a reliability feature.\n\n2. The paper is exceptionally well-written and easy to follow, especially for a complex systems paper. The concepts of P-DMR and D-DMR are introduced intuitively, and the figures provide clear visual explanations of the baseline, the problem, and the proposed solution's mechanisms.\n\n3. The work is evaluated on relevant, large-scale hardware (H200s) and state-of-the-art models (Llama-3, Mistral-Large, and an MoE model), not just on small-scale toy problems.", "weaknesses": "See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0BcTflReq", "reviewer": "Reviewer_MpZL", "summary": "This paper presents SpareTrain, a system for fault-tolerant LLM training that reduces the cost of dual modular redundancy (DMR). It introduces two mechanisms: Piggyback-DMR (P-DMR), which reuses activation recomputation from checkpointing for validation, and Deferred-DMR (D-DMR), which exploits idle GPU time (e.g., pipeline bubbles) to execute redundant checks. A planner coordinates these modes to balance reliability and efficiency. Experiments on up to 32 H200 NPUs show up to 35% speedup over naïve DMR while maintaining full coverage claims. However, the paper provides limited validation of correctness and scalability.", "questions": "- How does the proposed planner adapt to different parallel configurations (e.g., TP, PP, EP) that directly affect communication cost and idle-window size?\n- How does the checkpointing frequency (n) influence P-DMR coverage and overall verification cost?\n- Is there a quantitative cost model or optimization formulation guiding the planner’s decisions, or are all heuristics empirically tuned?\n- Can the authors provide the distribution of P-DMR, D-DMR, and naïve DMR operators under different configurations to explain the observed throughput changes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Addresses an important reliability problem in large-scale LLM training, where silent hardware errors can silently corrupt long-running training.\n- The idea of reusing existing computation (recompute or idle slots) for redundancy is intuitive and practically relevant, reducing the prohibitive cost of full DMR.\n- The paper demonstrates a non-trivial engineering effort, with experiments on real Ascend NPU clusters up to 8192 devices.\n- The proposed planner that automatically assigns P-DMR, D-DMR, or Full-DMR modes based on profiling data is conceptually elegant and system-aware.\n- Results show that SpareTrain can significantly lower redundancy overhead while maintaining theoretical detection coverage.", "weaknesses": "Although the paper presents a conceptually reasonable design, the overall analytical depth and experimental coverage are limited, leading to weak interpretability of the proposed DMR planner.\n\nFirst, the relationship between the DMR planning mechanism and the underlying parallel configuration (such as tensor, pipeline, and expert parallelism) is never formally analyzed.\nThe effectiveness of both P-DMR and D-DMR clearly depends on these configurations—pipeline depth determines bubble size, tensor-parallel degree affects collective communication cost, and expert parallelism introduces dynamic execution patterns.\nHowever, the paper only illustrates a few heuristic cases and does not provide any mathematical modeling, cost function, or decision boundary describing when a mode switch (e.g., from P-DMR to naïve DMR) should occur.\nAs a result, the planner behaves like a hand-tuned heuristic rather than a principled optimization framework.\n\nSecond, the interaction between checkpointing frequency and P-DMR coverage is completely unexamined.\nSince activation checkpointing recomputes every n layers, the interval n directly controls both recomputation cost and P-DMR reuse potential.\nWithout quantitative analysis of how n influences memory usage, recomputation overlap, or fault detection coverage, it is unclear how the proposed planner would adapt under different checkpoint configurations.\n\nThird, the experimental evaluation is overly high-level.\nIt reports only end-to-end throughput and speedup ratios, without any breakdown of how many operators are assigned to P-DMR, D-DMR, or naïve DMR modes under different parallel setups.\nThere is no analysis of idle-window utilization, checkpoint overlap ratio, or the planner’s sensitivity to communication intensity.\nConsequently, the claimed throughput improvement lacks interpretability and cannot be correlated with the underlying design decisions.\n\nFinally, the planner’s objective and constraints are only described qualitatively (“maximize reuse given time/memory slack”) without any explicit cost model or optimization formulation.\nThis makes it difficult to reason about convergence, scalability, or generalization to unseen workloads.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oOh2uh3Atz", "reviewer": "Reviewer_EkeY", "summary": "The paper addresses silent data corruption detection by integrating Piggyback-DMR and Deferred-DMR into a novel framework called SpareTrain.", "questions": "In Piggyback-DMR, could some internal SDCs be overlooked, since it only checks the input and output of Activation Gradient Checkpointing?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is applied to high-performance GPUs in a large-scale cluster.\n2. The proposed method can be applied to both MoE and non-MoE models.\n3. The proposed method introduces only a 3–14% overhead compared to baseline training without SDC protection.", "weaknesses": "1. D-DMR appears to work only with pipeline parallelism, which may limit its applicability.\n2. It seems that the proposed method relies on high-performance GPUs, where computation time is significantly faster than communication time. For GPUs like the H20, the effectiveness of the proposed method remains uncertain.\n3. We seem to observe a contradiction: we use pipeline parallelism due to memory constraints, yet the proposed method requires additional memory. Does this imply that, with given memory, the baseline could use smaller pipeline parallelism to achieve higher throughput? This seems like a better baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)