DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
Authors: Xinwei Qiang
Hongmin chen
Shixuan Sun
Jingwen Leng
Xin Liu
Minyi Guo
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Attention Mechanism
Deterministic Training
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose DASH, a high-throughput scheduling framework that accelerates the deterministic backward pass of attention for reproducible LLM training by up to 1.28$\times$.

Abstract: Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such as FlashAttention-3, the deterministic backward pass can incur up to a 37.9% throughput reduction relative to its non‑deterministic counterpart, primarily because gradient accumulation operations must be serialized to guarantee numerical consistency. This performance loss stems from suboptimal scheduling of compute and gradient‑reduction phases, leading to significant hardware underutilization.

To address this challenge, we formulate the backward pass of deterministic attention as a scheduling problem on a Directed Acyclic Graph (DAG) and derive schedules that minimize the critical path length. Building on this formulation, we present DASH (Deterministic Attention Scheduling for High-Throughput), which encapsulates two complementary scheduling strategies: (i) Descending Q‑Tile Iteration, a reversed query‑block traversal that shrinks pipeline stalls in causal attention, and (ii) Shift Scheduling, a theoretically optimal schedule within our DAG model that reduces pipeline stalls for both full and causal masks.

Our empirical evaluations on NVIDIA H800 GPUs demonstrate that DASH narrows the performance gap of deterministic attention. The proposed strategies improve the throughput of the attention backward pass by up to 1.28$\times$ compared to the baseline, significantly advancing the efficiency of reproducible LLM training.

Our code is open-sourced at https://github.com/SJTU-Liquid/deterministic-FA3.

Published as a conference paper at ICLR 2026
DASH: DETERMINISTIC ATTENTION SCHEDUL- ING FOR HIGH-THROUGHPUT REPRODUCIBLE LLM TRAINING
Xinwei Qiang--3, Hongmin Chen', Shixuan ;un'; Jingwen Leng', Xin Liu", Minyi Guo
1 School of Computer Science, Shanghai Jiao Tong University
ByteeDance Seed, 9Zhiyuan College, Shanghai Jiao Tong University
II /giangxinwei, sunshixuan, leng-jw, jw, guo-my ( jtu.jtuu..u...
2 [chenhongmin, will, liuxin. ailebytedance..oo com
ABSTRACT
Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such as FlashAttention-3, the deterministic backward pass incur up to a 37.9% throughput reduction relative to its non-deterministic coun- terpart, primarily because gradient accumulation operations must be serialized guarantee numerical consistency. This performance loss stems from suboptimal scheduling of compute and gradient-reduction phases, leading to significant hard- ware underutilization.
To address this challenge. we formulate the backward pass of deterministic at- tention as scheduling problem on Directed Acyclic Graph (DAG) and derive schedules that minimize the critical path length. Building on this formulation, we present DASH(Deterministi) Attention Scheduling for High- Throughttt,,, which encapsulates two complementary scheduling strategies: (i) Descending Q-Tile It- eration, reversed query- traversal that shrinks pipeline stalls in causal atten- tion, and (ii) Shift Scheduling, a theoretically optimal schedule within our DAG model that reduces pipeline stalls for both full and causal masks.
Our empirical evaluations on NVIDIA H800 GPUs demonstrate that DASH nar- rows the performance gap of deterministic attention. The proposed strategies im- prove the throughput of the attention backward pass by up to 1.28% x compared to the baseline, significantly advancing the efficiency of reproducible LLM training.
Our code is open- sourced at https: /tops://qithuse com/ SJTU-Liquid/ determing ni stiitticFFA
INTRODUCTION
The pursuit of consistent and verifiable outcomes is cornerstone of rigorous scientific research and large-scale engincering. In the domain of large language model (LLM) training (Wu et al. 2024)
where experiments span thousands of GPUs (Grattafiori al. 2024 DeepSeek-Al al. 2025)
and incur enormous costs, this principle of reproducibility becomes indispensable. Reproducibility empowers practitioners to diagnose training instabilities, such as loss divergence, and evaluate the impact of architectural modifications. Consequently, deterministic training, which guarantees bitwise identical results across runs, is increasingly adopted standard practice for industry.
The origin of the non-determinism in attention of L.LM training can be traced back to a fundamental yet often overlooked characteristic of computer arithmetic: the non-associativity of floating-poin point (FP) operations (Villa et al., 2009). For instance, (10s + 10-6) 10 evaluates to 0.0 in single- precision, whereas 100 10sh + 10 6 yields the correct 10 10-6, This sensitivity is magnified in the massively parallel environment of GPUs (Shanmugavelu et al. 2024).
"Shixuan Sun is the corresponding author.

---

Published as a conference paper at ICLR 2026
Computation Flash-Mtention
Global Reduction
Performance Drop
Backward
Glbbal Thuffres
5k, 15% 19,
19, 6%,, M%, Ss,
SMI
S0C
534
SM7
Non Determinstic Baseline
100%
Normalized Throughout
02
Reduction Cinter SMs)
80%
73%
78%
77%
Lopil Redhhleee
01
Abtts S01
Reduction with Atomic Operation
60%
400%
0-
20%
Locl Bullen
0%
Reduction with
H0=64
Full
HDw128
Ful
Coussll HD364
HD=128
Causal
Figure 1: Overview of the deterministic FlashAttention. Left: Tiled computation structure of the backward pass, highlighting the local and global reductions. Middle: Comparison between the non-deterninistic (atomic-based) and deterministic (ordered) global reduction. Right: Performance degradation under causal and full attention masks, HD stands for head dimension.
In nigh--peffrmmance attention VVaswani et al. 2017 mechanisms like FlashAttention (Dao et al. 2022). the backward pass computation is para varallelize leized across hundreds of GPU Streaming Multipro- cessors (SMs) (NVIDIA 2022) maximize throughput. Each SM, rumning a Cooperative Thread Array (CTA), accumulates partial contribution to gradient tensors (e.g.. the gradient for the query matrix, dQ). The default high-speed approach allows these CTAs to concurrently update the final gradient in global memory via -on-deterministic atomicAdd operations, shown in Figure [ mid- dle. This creates a non-deterministic accumulation order: the final accumulated value depends on the uncontrolled completion order of the CTAs, leading to bit-wise variations between runs.
To enforce reproducibility, FlashAttention-3 (Shah et al. 2024 provides a deterministic mode. enforces fixed accumulation order by using synchronization barriers to force CTAs perform their additions in a serialized order (c.g., ordered by CTA index). However, this guarantee of consistency imposes significant performance penalty. As illustrated in Figure D right, enabling deterministic mode may lower throughput by up to 37.9%, leading to severe training costs when scaling L.MSs across hundreds of thousands GPUs.
This performance gap is not an inherent consequence of serialization itself. Instead, it stems from direct conflict between the tile scheduling and a rigid, pre-determined accumulation order. As illustrated in the middle of Figure [ the full mask scenario, commonly employed in multi-modal tasks, highlights key inefficiency: the naive schedule creates a bottleneck by forcing reductions to start sequentially. An ideal schedule, however, would parallelize this process, allowing CTAs to begin reduction on different tiles concurrently. Crucially, this reveals that the computation schedule and the accumulation order are tightly coupled and cannot be optimized in isolation.
To address this, we introduce Deterministic Attention Scheduling for High- thgh-throughput (DASH), framework that formulates deterministic attention backward execution as an explicit scheduling optimization problem. We model the deterministic backward pass as a Directed Acyclic Graph (DAG), and formalize the objective minimizing the DAG's critical path length. Based on this model, we design two complementary scheduling strategies. The first, Descending Q-Tile Iteration, heuristic that processes query tiles in reverse order to advance dependency resolution and shrink pipeline bubbles in causal attention. The second strategy, a theoretically optimal algorithm we term Shift Scheduling is provably optimal under our DAG model. It employs phase-shifted assignment of computational tasks to GPU multiprocessors, creating a perfectly staggered execution pattern. This ensures that the workload is perfectly balanced and that the serialized reduction operations proceed without contention while approaching the model's 's theoretical utilization bound.
Our empirical evaluations on NVIDIA H800 GPUs show that DASH significantly narrows the per- formance gap relative to the FlashAttention-3 deterministic baseline. The two strategies deliver up to a 1.28x speedup for the deterministic attention backward pass, significantly improving the efficiency of reproducible LLM training.
In summary, we made the following contributions in this paper:
We identify the misalignment between tile execution and accumulation ordering as the principal source of performance degradation in deterministic attention.
2

---

Published as a conference paper at ICLR 2026
We provide the first DAG-based formalization of deterministic attention backward scheduling, enabling principled optimization of critical path length.
We introduce two complementary scheduling strategies, Descending Q-Tile Iteration and Shift Scheduling, that achieve up to a 1.28x speedup over the FlashAttention-3 deterministic baseline on H800 GPUs
BACKGROUND
2.1 DETERMINISTIC FLASHATTENTION BACKWARD PASS
FlashAttention backward pass: dQ, dK, and dV (Figure | left). During backpropagation, the gradients dK and dV are accumulated across all queries for each key (or value) position, i.e., they are reduced along the Q axis. contrast, dQ requires reduction across all key-value (KV) positions for each query, i.c., along the KV axis. To expose parallelism, the implementation partitions the KV dimension across SMs, allowing dK and JV to be computed within each SM via local reduction. However, this strategy distributes partial contributions to dQ over multiple SMs, necessitating a global reduction to produce the final gradient. conventional implementation performs this reduction using atomic additions (Figure [ middle), which induces run-to-run variation because floating-point addition is non-associative. The resulting numerical nondeterminism undermines strict reproducibility determinism, one must enforce a prescribed accumulation order. FlashAttention-3 achieves this by performing tile-wise sequential accumulation of dQ along the KV dimension.
We first outline the core gradient computations
large-scale training. To guarantee
2.2
GPU ARCHITECTURE
On modern GPUs, the memory hierarchy comprises registers, shared memory, 1.2 cache, and global memory (NVIDIA 2022), reflecting a fundamental capacity-latency trade-off: smaller and faster storage resides closer the compute units. Shared memory private each SM, enabling low- latency intra-SM data reuse, whereas the L2 cache is globally shared, mediating inter-SM data ex- change and coherence. In datacenter-class GPUs, the L.2 cache may be physically segmented, with each segment preferentially serving subset of SMs; latency than local ones. This hierarchical organization materially shapes the attainable performance and the efficiency of memory-bound GPU kernels.
accesses typically incur higher
2.3 DETERMINISM IN OTHER OPERATIONS OF THE TRANSFORMER
Other components, such as GEMMs, attention forward and normalization, also involve reduction op- erations; however, the computational cost of enforcing determinism these cases generally min- imal during typical LLM training. GEMMs may exhibit nondeterministic behavior only when the reduction axis (i.e., the K-dimension) is partitioned across multiple blocks, as in split-K (NVIDIA Corporation] 2025) or stream-K (Osama et al.) (023) parallelization modes. In large-batch LLM training, parallelism along the M and N dimensions is typically sufficient to fully utilize the GPU, rendering split-K or stream-K modes unnecessary; therefore, disabling these modes generally re- sults only a minor reduction in throughput. Similarly, other operations involving reduction, such attention forward passes and normalizations, typically perform reductions within single block, thereby ensuring a deterministic reduction order. Purely elementwise operations, including activa- tion functions and bias additions, are inherently deterministic.
3 DASH: SCHEDULING STRATEGIES FOR DETERMINISTIC ATTENTION
In this section, we introduce optimized scheduling strategies for deterministic attention. Without loss of generality, assume that the number of KV tiles equals the number of SMs, denoted by n. When the actual number KV tiles differs from the number of SMs, we conceptually refine or aggregate attention heads so that all SMs remain fully utilized under the same analytical framework.
3

---

Published as a conference paper at ICLR 2026
Compute CO.D
SMO
C(0,0)
|<0,0)
((0.1)
R(0.1)
Compute
Reduction R
Dependescy
SM1
C(1.1)
Idie
Dependancy
Criicall Path
Critical Path Length 2c
Figure 2: Visualization of the Deterministic Scheduling Problem. The Gantt chart (left) shows naive execution schedule for a problem with two KV-tiles (i-index) and two Q-tiles (j-index). Each task consists of compute phase C(i,j) and a reduction phase R(i, Local reductions enforce contiguous execution on a single SM (e.g., all tasks for i = 0 on SMO). A deterministic global reduction order introduces a cross-SM dependency (red arrow), forcing SMI to idle and creating pipeline bubble. The corresponding DAG (right) abstracts this schedule, where the critical path determines the end-to-end latency.
3.1 PROBLEM FORMULATION
We formalize the deterministic attention backward scheduling problem as an optimization over directed acyclic graph (DAG), as shown in Figure 2 The DAG's structure is constrained jointly by the dataflow of FlashAttention and the architectural characteristics of the target GPU. Our model represents a simplified abstraction of actual GPU execution; its primary purpose is to offer insights into more effective scheduling decisions, rather than accurately predict real execution times. As such, there remain significant differences between our theoretical model and the complexities of real-world GPU behavior.
Graph Construction. Each tile- prooeessinng task is modeled as a linear path of nodes connected by edges that encode two successive phases: the tile's computation and (ii) the subsequent global re- duction. These phase edges are weighted by their respective execution times, which are assumed to be constants. To encode legal accumulation orderings and data dependencies across tiles, we insert zero-weight dependency edges between nodes of different task paths. In this way, edge weights capture quantitative duration, while the topology captures qualitative ordering constraints. The scheduling objective is to minimize the critical-par length of the resulting DAG, thereby reduc- ing end-to-end latency and improving overall execution efficiency.
Optimization Constraint. Data movement across different memory levels incurs substantial over- head, while registers provide the fastest storage in GPUs. To leverage fast register-resident accumu- lation of dk and dV, all operations for given KV tile must run contiguously on a single SM. Consequently, the edges associated with this tile form an unbroken chain, which imposes a key constraint on our optimization.
3.2
ANALYSIS OF
DETERMINISTIC BACKWARD SCHEDULE
SMD
C0
F0
C1
1
C2
c3
13
SMO
c0
/0
c1
C2
r2
c3
r3
F0
|2
SM1
r2
SM2
10
c1
[1]
c2
/2
c3
r3
SM2
c2
F2
r3
SM3
cD
r0
F1
c2
F2
c3
r3
SM3
c3
:3
(a) Schedule for Full Mask
(b) Schedule for Causal Mask
Figure 3: Backward scheduling of FlashAttention-3 for both mask shapes. Each colored segment denotes one block's computation (cost c) followed by reduction (cost r). Idle gaps correspond to pipeline bubbles. For clarity, since assume the number of KV tiles equals the number of SMs, each SM processes exactly one KV tile; thus we omit the KV index the visualization and show only the query index for each block.
Under a full attention mask, the FlashAttention-3 backward schedule achieves reasonable pipeline utilization (Figure 3a) Observable bubbles (SM idle periods) arise only during the startup phase of the first computation stage, before steady-state overlap is established. Let each stage incur computation cost e followed by a reduction cost r. After the initial fill, each attention head sustains sequential (computation reduction) pairs, giving T Tsteady = n--c (c+ where is the number of
4

---

Published as a conference paper at ICLR 2026
SMs. The startup overhead contributes additional (n-1)-r due staggered completion of the first sequence of reductions. Hence, for m heads, Tyull = mmT T = (2--(c+))((211)))) up to negligible control and synchronization overhead.
In contrast, when a causal mask is applied, the data dependencies inherent in the schedule lead to significant inefficiencies. As shown in Figure 3b) this schedule introduces substantial bubble within the execution of each attention head, preventing effective pipelining. The critical path for a single (c (c++) + (n 1).r. Since this inefficient pattern repeats for every head, the total execution time for m heads is approximately Teauso! m. pawsd + Tttortup Rd m
head becomes Thead cousal
(c [ r) + (n 1).
3.3
DESCENDING Q-TILE ITERATION: A ROBUST HEURISTIC FOR CAUSAL MASKS
SMO
c3
r3
c2
F2
d
f1
C0
0
c3
r3
c3
C2
f2
C1
C3
f3
C2
f2
SM2
r3
SM3
c3
r3 c2
f2
r1
cb
Figure 4: Descending (reverse-order) query tile schedule for the causal mask. Reversing the Q-block traversal accelerates dependency resolution. Colors distinguish attention heads in the pipeline.
To mitigate the pipeline bubbles caused by causal masking, we propose a simple yet effective mod- ification: reversing the processing order of the query (Q) blocks. As illustrated in Figure 4 this reversed schedule allows most SMs to begin their computation earlier by resolving dependencies more quickly.
The crucial advantage of this approach its impact on pipeline efficiency for subsequent atten- tion heads. By reversing the order, the short tasks are completed first. freeing up their SMs much earlier. Consequently, the second head can immediately begin to utilize these available re- sources, creating tightly coupled pipeline that almost eliminates the idle gaps between heads. This sustained high utilization across an even number of m heads yields total execution time of:
T Trewersed m-(n+))(etr) + (n 1) r.
3.4 SHIFT SCHEDULING
Proceeding dependency
Reverse dependency
Compute Reduction Dependency
8
Critical Path Length 2r
Critical Path Length 2c + 3r
Figure 5: Illustrative example for Lemma 1 Left: Added dependency (zero-weight) edges pre- serve non-decreasing depth order and do not lengthen the critical path. Right: A backward (depth- decreasing) dependency edge violates the lemma's is condition and increases the critical path.
Although the Descending Q-Tile Iteration significantly improves performance, it is natural to ask whether theoretically optimal schedule exists. To address this, we examine the impact of introduc- ing reduction-induced inter-SM dependencies on the computation DAG's critical path.
Disregarding (for the moment) the accumulation edges required for dQ updates, the graph decom- poses into n independent chains whose total time is minimized when their cumulative workloads are perfectly balanced. In this idealized scenario, all chains are also isomorphic, identical task structure and number of tasks. The core challenge is thus to insert the necessary zero-weigee weight dependency edges without lengthening the original critical path. The lemma below characterizes precisely when this is possible; its proof is deferred to Appendix B for brevity.
they share an
lemma 1.
Ler Go = (V, Eu) be a DAG consisting of single source node s, a single sink node t, and parallel, isomorphic chains connecting
t.. All edge weights in Eg are strictly positive. Let the depth of a node u, denoted depth(u), be the number of edges on the unique from to u within its chain in Go. Let sequence of graphs G Gi...., Gy be generated such that G; (V, F4 (ex)),
5

---

Published as a conference paper at ICLR 2026
Key-Value Tile
KVD
KV2
Query Tile
Q1
SM1
C1
F1
F2
c3
60
Q2
2
0
c2
F2
r3
co
rd
cl
3
1
SM3
c2
SMO
SM1
SM2
SM3
Assigned SM
Figure 6: Optimal full-mask schedule via cyclic shifting. Left: Cyclic visiting order of Q tiles per SM: distinct timestamps (i.e., the value each box) on each row induce a natural, conflict- free reduction sequence for every dQ block. Right: Simulated timeline showing fully balanced utilization without additional bubbles.
where each e; = (u;; v;) is a zero-weight edge. We add the explicit condition that every new graph G2 in the sequence must remain a DAG.
Under this condition, the critical path length of Gk is equal to that of Go if and only if for every added edge e; (u,, for i {1....,k) ..)
the condition depth(u:) S depth(v:) holds.
As illustrated in Figure 5 Lemma dictates that to preserve the original critical path length, any added dependency edge (it, v) must satisfy the condition depth(u) I depth(v). This formal con- straint translates to critical physical limitation: for any given query tile Qj. the tasks involving it cannot be executed in parallel.
A schedule that assigns two tiles contributing to the same dQ,-say (KV, Qÿ) and (KV2, Q2) execute concurrently on different SMs would create resource conflict during their reduction phases. Resolving this conflict requires serializing the reductions, for instance, forcing the reduction for (KV,, Q2) to wait for the one from (KV, Q2) complete, or vice versa. Because the confficting reduction tasks would otherwise start at the same depth in the DAG, this forced serialization in- troduces a dependency edge (u, u) where depth(u) > depth(v)- from the completion of the first reduction to the start of the second. This directly violates the lemma's 's condition and sub-optimally extends the critical path.
Our objective is thus twofold: first, to balance the workload across SMs, and second, to devise conflict-free reduction order that adheres to the lemma's 's constraint.
Optimal Schedule for Full Masks Under full mask, per-KV-tile workloads uniform, allow- ing for immediate balancing. To satisfy the second objective, we employ Shift Scheduling, as illustrated Figure 6 this schedule, SM, processes KV blocks in the order (i, 1....,
1, O...... 1). This cyclical assignment inherently creates conflict-free, sequential ordering for the reductions on any given dQ block, directly satisfying the lemma's condition. As both workload balancing and conflict-free reduction are achieved, this schedule is theoretically optimal.
Symmetric Shift Scheduling for Causal Masks Causal masking induces strongly imbalanced workload: carly KV blocks participate in the full set query interactions, whereas later blocks contribute progressively fewer operations, yielding computation workloads that decrease linearly across the sequence.
We address this by Symmetric Shift Scheduling. Its core is symmetric pairing principle: SMs jointly handle KV blocks and with the second-shortest, and so forth. This pairing equalizes task chain lengths per SM, restoring near-perfect balance.
i, pairing the longest with the shortest, the second-longest
We operationalize symmetric pairing via two-phase schedule. In Phase 1, cyclic shift is applied to the dense lower-left rectangle, efficiently filling the pipeline. Phase addresses the residual triangles using a purely analytical model of workload folding, where tasks from the lower-right are logically mapped to the upper-left's masked slots to form a conceptual square without any data movement. The operational sequence- top-down traversal of the left triangle and bottom-up traversal of the right- is algebraically equivalent to diagonal-initiale initialize shift schedule on this conceptual square. This equivalence is key: it preserves workload balance, ensures contiguous computation for each

---

Published as a conference paper at ICLR 2026
KV block, enforces depth-monotone accumulation to satisfy Lemma 1 and ultimately eliminates all pipeline bubbles.
Summary of Optimal Performance In summary, the proposed scheduling strategies achieve theoretical opti- mality for both scenarios. By perfectly balancing work- loads and eliminating pipeline bubbles, the total execu- tion time for m heads is: Full Mask: Tull = m n
Key Value Tile KV3
KV0
KV1
KV2
KV4
KV5
KV6
KV7
Q0
Phase
5
(c + r); Causal Mask: Teausal.opt 1)
6
Query lile
4 EXPERIMENTS
Q4
In this section, empirically evaluate the performance of our proposed scheduling strategies under full and causal masks. We measure throughput under various se- quence lengths and analyze how architectural factors in- teract with different scheduling choices.
Qs-
1
4
Q6-
2
0
7
SM1
SMB 5M2
SMD
Paired SM Alloc
4.1 EXPERIMENTAL. SETUP
Figure 7: Optimal causal-mask sched- ule using symmetric shift and two-phase workload folding. Phase processes the dense lower-left rectangle; Phase 2 folds the remaining triangles into logical square and traverses it starting from the main diagonal, first covering the upper-left portion before the lower- right, ensuring each KV block exe- cuted contiguously.
experiments are con- equipped with NVIDIA H800 GPUs, CUDA version 12.6 and Triton (Tillet et al. 2019) ver- sion 3.4. All kernels implemented by extending the FlashAttention-3 implementation.
Hardware and Software. ducted
Baseline and Proposed Methods. We compare our methods against the deterministic backward pass of FlashAttention-33, which serves our primary baseline. We also benchmark against the Triton tutorial's implementation for causal attention, its pub- lic version lacks a full-mask counterpart. We omit FlashAttention-2 because prior published benchmarks (Shah et al.) 2024) on Hopper-class GPUs show it consistently outperformed by FlashAttention-3, and thus it no longer constitutes competitive baseline. The methods under our evaluation are:
Descending Q-Tile Iteration (for both masks)
Shift Scheduling (for full masks)
Symmetric Shift Scheduling (for causal masks)
Benchmark Settings Following the methodology of the FlashAttention-3 study, we evaluate per- formance by fixing the total number tokens at 16,384 while varying the sequence length from 512 to 16.384. Similarly, fix the hidden dimension to be 2,048, and test different head dimensions in 64 and 128. All the results are tested using BF16 precision random inputs.
4.2 PERFORMANCE ON FULL ATTENTION MASKS
Figure presents the throughput comparison for the full attention mask scenario. Our Shift Schedul- ing consistently outperforms the FlashAttention-3 baseline across most sequence lengths, demon- strating the effectiveness of our theoretically optimal approach. However, notable exception occurs at the maximum sequence length of 16,384, where its performance slightly degrades relative to the original FlashAttention-3 baseline.
This phenomenon highlights a divergence between our theoretical model and practical hardware execution. Our model assumes zero-cost dependency edges, but in reality, inter-SM communication for synchronizing reduction operations is mediated by the L2 cache. This incurs significant latency, ranging from approximately 200 cycles for accesses the local 1.2 cache segment over 500 cycles

---

Published as a conference paper at ICLR 2026
Busddin
Pertormance (TRLOPS)
Descending Stitt Schoouling
Performence (TRLOPS)
600
Oescending
500
Non dete maiii
400
300
300
200
512
1024
2046
4096
8192
16384
512
1024
2046
40966
8192
16384
Sequence Length (sealen)
Sequence Length (segien)
(a) Full mask, headdim 64.
(b) Full mask, headdim 128
Figure 8: Backward-pass throughput under full attention masks.
for remote segment accesses on H800-class GPUs (Luo et al., 2025 This latency differential is direct consequence of the distributed L.2 cache architecture described Section 2
At a sequence length of 16,384 and a KV block size of 128, the computation for a single head is dis- tributed across 128 blocks, often mapped to 128 SMs. This high degree of parallelism necessitates frequent cross-SM communication to signal task completion. Given the large number of partici- pating SMs, substantial portion of these synchronization signals must traverse the higher-latency links to remote L2 cache segment. The Shift Scheduling, with its more intricate dependency graph compared the simpler, linear dependency munication overhead at extreme parallelism. This increased synchronization cost, dominated by remote 1.2 accesses, ultimately outweighs the computational benefits of the schedule in this specific high-parallelism, long-neqe scenario, leading the observed performance degradation.
the baseline, becomes more sensitive this com-
4.3
PERFORMANCE ON CAUSAL ATTENTION MASKS
600
Perormanc (FFLDRS)
300
Symmerr th Senesuling
Suselive
Descending
500
600
Thtnn
Save Ise
Syrmerr' Shift Scheolling ng
Dusconding
Nav denemmmiidiii
Arfommanc
400
300-
300
200
200
100
100
512
1024
2048
10966
B192
16384
512
1024
2048
4096
8192
16304
Sequence Length (segien)
Sequence Length (seqion)
(a) Causal mask, headdim 64
(b) Causal mask, headdim 128
Figure 9: Backward-pass throughput under causal attention masks.
The performance evaluation for causal attention masks, presented in Figure 9 confirms the efficacy of our proposed methods. Both the Descending Q-Tile Iteration and our theoretically optimal Sym- metric Shift Scheduling demonstrate throughput improvement over the FlashAttention-3 baseline across all tested configurations.
An interesting trade-off emerges when comparing our two proposed methods at different head di- mensions (headdim). At headdim performance, validating the benefits of its superior workload balancing. However, the descending schedule does not perform very well in this case. This because in the Flash Autenionn causal backward kernel, the L2-aware LPT scheduler interleaves multiple heads across SMs. When head- dim 64 and the sequence length short, each head's 's L2 footprint remains small, allowing many heads to reside in cache with only 1-2 tiles in flight per head. Consequently, the causal stalls tar- geted by Descending Q-Tile Iteration are largely masked by cross head interleaving, resulting in only marginal net performance gains.
64, the Symmetric Shift Scheduling achieves the highest
8

---

Published as a conference paper at ICLR 2026
However, at headdim 128, Symmetric Shift Scheduling's performance is surpassed by the sim- pler Descending Q-Tile Iteration. This performance inversion is attributable to a critical interaction between algorithmic complexity and GPU resource limitations, specifically register pressure. The Symmetric Shift Scheduling, while algorithnically optimal, requires a more complex implementa- tion manage the state of the folded task space. This complexity translates to higher register usage per thread to maintain additional loop counters and intermediate states.
When headdim 128, the base register requirement for storing accumulators and other interme- diate values is already substantial. The additional overhead (around 10 registers) from our optimal schedule can push the total register count per thread beyond the hardware's 's physical limit, as shown by Nsight Compute (NVIDIA Corporation, (024). This forces the compiler to generate code that spills registers, offlodding their contents to the much slower local memory. The high latency incurred by these spill-induced memory operations introduces significant execution stalls, which negate the algorithmic benefits of the more balanced workload and lead to degraded performance. In con- trast, the simpler Descending Q-Tile Iteration operates below this critical register pressure threshold, thereby avoiding spilling and achieving better effective performance this high-resource-demandd scenario. Therefore, the two schedules for causal masks are complementary: theoretically optimal under our DAG model, while Descending is the practically preferred choice for large head dimensions on current GPUs.
teteeetttt Symmetric Shift is
In the future, Symmetric Shift's theoretical advantages are expected to be fully realized on newer architectures with greater on-chip resources (such as Blackwell GPUs with TMEM, devices equipped with larger register files), or under kernel designs that are less constrained by register allocation than the present FlashAttention-3 implementation.
4.4
END- TO-END PERFORMANCE
LIMS
Seq
1.06
Models
Bistle Tasciig
Oet
1.300
Speedup
1.675
1.04
Batio
1.425
1.050
1.02
0.50
0.2
1.000
LLMNA3-8
Owen2.5 5-78 Mistrai-Ex78
1.00
SAMHA/T-Hge
523 Sarge
503 Medion
Uiema3 68
Qwen2.5-7
Nistral 6x78
UaDDA-1 ib
suge sos L-rage
SD1.3 5D1.5-Medom Medoum
(a) Specdup for an entire transformer block.
(b) Time breakdown.
Figure 10: End-to-end performance of a transformer block.
To assess the performance gains delivered by DASH during training, we measured the runtime required to process an entire transformer block, accounting for both forward and backward passes.
We evaluated DASH across a range of widely adopted models. For causal mask scenarios, we selected famous LLMs: LLaMA3-80 (Grattafiori et al. (2024). Qwen2.5-7b (Qwen et al. 2025). and Mistral-8 (Jiang et al.) 2024). For full mask scenarios, we included the vision model SAM- huge (Kirillov et al. 2023). ) the diffusion models StableDiiffsiio3.55 (medium and large) (Stability AI 2024),) and the diffusion-basse language model LLaDA- lb (Nie et al., 2025).
For LLMs, we employ batch size of with sequence lengths of 8k, 16k, and 32k. In the case of full mask models, a batch size of 16 is used, with the training sequence length fixed at 4k in accordance with standard architectural configurations. The relative speedup achieved by approach compared to the baseline is illustrated in Figure 10a For causal models, we observe end-to-end performance improvements ranging from 2% to 10%. Full mask models also exhibit speedup of approximately 4%. In summary we achieved an average speedup of around 5%, which aligns with our internal training experience thousands of GPUs. Additionally, Figure TOb) provides detailed breakdown of computation time across different kernel operations, with causal models evaluated at a sequence length of 16k.
4.5 IMPACT OF DETERMINISM ON NUMERICAL STABILITY
Our analysis of backward passes indicates that non-deterministic kernels cause run-to-run gradient deviations of O(10~ *), while deterministic ones guarantee bitwise identical outcomes (Table []. Although small, this variability accumulate, so determinism is key achieving reproducibility.

---

Published as a conference paper at ICLR 2026
Table
Max gradient deviation averaged over 10 identical backward passes; M, max (gr grefl
Masking Scheme
Non-deterministic
Deterministic
Full
2.4 x 10
0
Causal
4.9 x 10-1
0
5
RELATED WORKS
FlashAttention and Kernel-Level I/O Optimization Early optimization of attention focused on mitigating the I/O bottleneck imposed by the quadratic attention matrix. FlashAttention (Dao et al.) 2022) introduced an 1/O-aware tiled and fused kernel that avoids materializing the full attention matnx in HBM. FlashAttention-22 and 3 (Dao 2023 Shah et al., (024) further improved utilization refined work partitioning and leveraged specialized hardware for asynchronous data movement.
Low-Precision Attention Low-precision methods further reduce bandwidth and memory cost. The SageAttention series (Zhang et al. 2025b a systematically explores progressively lower for-
mats while maintaining accuracy.
Inference-Oriented Attention Kernels snférence-specializee kernels include FlashDecoding and FlashDeccdding+((DDe et al.) 2023] Hong et al.) 2024) for autoregressive decoding, PodAtten-
Kamath et al. 2025 for mixed prefillig//ecodingg and DeFT (Yao et al. 2025) and Fast-
Tree (Pan et al. 2025) for tree-structured generation.
Distributed Cyclic Scheduling Our shift-based scheduling is inspired by cyclic (ring- style) phase-shift patterns long used in distributed systems. Distributed attention algo- rithms- Attentttntionn (Liu et al.) 2023), StripedAttention (Brandon et al. (023). and Loong- Train (Gu et al., 2 related cyclic schemes to overlap communication and computation across devices, whereas we apply a shift strategy intra-GPU to co-optimize deterministic accumula- tion and work balance.
Deterministic Implementations Existing deterministic implementations either split dK, dV and dQ calculation into different passes (c.g., Triton tutorials (Tillet et al. 2019 f
forcing a second K/V read or materialize per-tile dQ partials for later consolidation (FlashAtter Attentionn))) adding memory footprint and an extra reduction kernel. These designs trade bandwidth or memory rather than co- optimizing execution and accumulation order, which is the focus of our approach.
Determinism in Inference Determinism for inference has also been examined: He & Lab (2025) attribute non-reproducibility to lack of "batch invariance," where outputs depend on batch size, and design batch-invariant kernels. Their goal differs from ours: target training time run-to-run determinism, where batch configurations are fixed to ensure reproducibility.
CONCLUSION
In this work, we addressed the significant performance penalty associated with the deterministic backward pass in modern attention mechanisms. By formulating the computation problem on a DAG, we introduced DASH, framework featuring two distinct and complementary strategies. The first, Descending Q-Tile Iteration, provides a simple yet remarkably effective heuris- tie that accelerates causal attention. The second, derived from our conflict-free scheduling lemma, represents a theoretically optimal solution.
scheduling
Our empirical evaluation not only demonstrates that DASH significantly narrows the performance gap, improving throughput by up to 1.28x over the baseline, but more importantly, reveals crucial insight: theoretical optimality does not always translate to practical superiority. We iden- tified hardware realities, such as register pressure and inter-SM communication latency. as critical factors that can override the benefits of more complex, algorithmically perfect schedule. By pro- viding suite of solutions catering to different scenarios, DASH enables practitioners to achieve high throughput attention in reproducible L.IM training.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is partially supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (YYB2225XXX1113)
REFERENCES
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers, 2023.
URL
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL
https://arxiv.orgg/ wttp:/rr/aabb//////// 08691
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL WwRLtttps://axxiv.rrgg obbb///b/ 2205.14133
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long- context inference. PyTorch Blog, October 2023, URL https://pytorcch.ort
flash-decoding/
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui
Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu. Shengfeng Ye, Shengfeng Ye, Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao. Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Ximnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xic, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https:://arxin wwt./or.///ooooooooooooooo/o.ota abss 2412.42244 19437
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary,
11

---

Published as a conference paper at ICLR 2026
Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee. Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak. Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh. Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji. Ning Zhang, Olivier Duchemne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic. Roberta Raileanu. Rohan Maheswari. Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei. Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape. Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le. Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche. Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Mareus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena. Keqian Li, Kiran Jagadeesh. Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa,
12

---

hatp://ssw..ooooodaaaaaannnnnnnn Published as a conference paper at ICLR 2026
Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma. Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez. Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhao- duo Wen. Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: /tt:://axxiiv. otg/ abs/ 2407 21783
Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang. Yonggang Wen, Tianwei Zhang, Xin Jin, and Xu- anzhe Liu. Loongtrain: Efficient training of sequene llms with head-context parallelism,
2024. URLhttps::/ /ttp::///arxvvoooo .rwg//rr//aaa abs /2w../406... 18485
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm infer- ence. Thinking Machines Lab: Connectionism, 2025. doi: 10.64433/ml1222500100.
Ke Hong. Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding++: Faster large language model inference on gpus, 2024. URL
https:// attpps://axxiw.orr/ab////// wttp:oor/////....oa 01282
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL
Aditya K. Kamath, Ramya Prabhu, Jayashree Mohan, Simon Peter, Ramachandran Ramjee, and Ashish Panwar. Pod-attention: Unlocking full prefill-decode overlap for faster Ilm inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems, Volume 2, ASPLOS '25, pp. 897- ACM, March 2025. doi: 10.1 08.1144/3677641.77159999 URL htl.hhtp:/// /ttp//xxxx///o doi www.oooorro....... 1145//366141 3715996
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. ftt.tttts:://
http://v.wrrrbb//////3// 02643
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockiie transformers for near- infinite context, 2023. URL htt.tttts///aaiiiiii http://////aaa.ooo////////01./00
13

---

Published as a conference paper at ICLR 2026
Weile Luo, Ruibo Fan, Zeyu Li, Dayou Du, Hongyuan Liu, Qiang Wang, and Xiaowen Chu. Dis- secting the nvidia hopper architecture through microbenchmarking and multiple level analysis, 2025. URL https: httpss://axxivv org/ abs abs/250l11 12084
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025. URL https://
arxiv. wrggabs/ss.oooooo... 200.250229929
NVIDIA. NVIDIA H100 Tensor Core GPU Architecture. Technical report, NVIDIA,
mar 2022. URL https: nvi dia /ttp://ttttettttemeeeeee////////
White paper.
NVIDIA Corporation. NVIDIA Nsight Compute, 2024. URL htt.p:://devee.oerr..vv/
com//ssight-cooutte. Version 2022.4.
NVIDIA Corporation. Split-k gemm. http:://ggtaab...c/////////////
ma http////aaamppeee____pppppeee1ee _emmm 2025.
Muhammad Osama, Duane Merrill, Cris Cecka, Michael Garland, and John D. Owens. Stream- k: Work-centric parallel decomposition for dense matrix-matris multiplication on the gpu. In Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP '23, pp. 429-431, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700156. org 111770111153332288833333
doi: 10. 10.11453557948.35774499
URL https:://oi.
Zaifeng Pan, Yitong Ding. Yue Guan, Zheng Wang. Zhongkai Yu, Xulong Tang, Yida Wang. and Yufei Ding. Fasttree: Optimizing attention kernel and runtime for tree-structured LLM inference. In Eighth Conference on Machine Learning and Systems, 2025. URR.htttps///ppenreeee
net Forum?idd SWVHcH233J
Qwen, :, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: aaw.iv org/ abs/ /21//242222222 15115
Jay Shah, Ganesh Bikshandi. Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision, 2024. URL
https: httpe//////xii.ooo////oooe.// www.//////////oooo/oo///ooeess// 2407. 006608
Sanjif Shanmugavelu, Mathieu Taillefumier, Christopher Culver, Oscar Hernandez, Mark Coletti, and Ada Sedova. Impacts of floating- point non associativiiiy on reproducibility for hpc and deep learning applications, 2024.
05148
Stability AI. Stable diffusion 3.5, 2024. URL Accessed: Nov. 2025.
sd3. AI/sd3.
Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10-19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196 doi: 3329973
0.1144//311508.339999
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:/css//ppcceeinss.eeee
wttp://eepppeeeeeeeeeeaeeeeeeeee /fap/p//p///epppp///111/// fi 911e33 htt.5a/aee2BSeeeeeooooacaaaaaaea
14

---

Published as a conference paper at ICLR 2026
Oreste Villa, Daniel Chavarria-Miranae Miranda, Vidhya Gurumoorthi, Andres Marquez, and Sriram Kr- ishnamoorthy. Effects of floating-Pos point non-associativity on numerical computations on mas- sively multithreaded systems. Pacific Northwest National Laboratory (PNNL), Richland, WA (US), Cray User Group, Inc., Corvallis, OR, United States(US)., 05 2009. URL. https: //www. osti http:///oo/bbbbo////999999
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey, 2024. URL ARLLttts////axiv..o.
abs/2402.. 01364
Jinwei Yao, Kaiqi Chen, Kexun Zhang, Jiaxuan You, Binhang Yuan, Zeke Wang, and Tao Lin. Deft: Decoding with flash tree-attention for efficient tree-structured Ilm inference, 2025, URL
https: org/ wwtpaas.////oooooooot.ee/eeeeest 240/ .
Jintao Zhang, Haofeng Huang, Pengle Zhang. Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2:
Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2025a. URL
attpp:://axixv org/ wwtg//bbat.o
Jintao Zhang, Jia Wei, Haofeng Huang, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageatten- tion: Accurate 8-bit attention for plug-and-play inference acceleration, 2025b. URL https
7/
Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training, 2025c. httppss///axxi...orraaa/////2
15

---

Published as a conference paper at ICLR 2026
A
THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, the authors employed a large language model (LLM) for two primary purposes. First, the LLM was used as a tool to improve the grammar, spelling, and overall clarity of the text. Second, it was used to assist in the initial stages of the literature search. The role of the LLM was strictly that of an assistant. All language suggestions were reviewed and edited by the authors to ensure they accurately reflected the intended scientific meaning. Further- more, any literature identified with the assistance of the LLM was independently retrieved, reviewed, and vetted for relevance and accuracy by the authors. All intellectual contributions, including the conception of the research, methodology, and final conclusions, are the exclusive work of the human authors, who take full responsibility for the final content of this paper.
B PROOF OF LEMMA 1
Proof. Let (x) denote the length of the longest path from the source node s to node x in graph G4. The critical path length of G, CP(G.) = (t).t).
Due to the isomorphic structure of the chains in Go. all nodes at the same depth j have the same longest path length from s. Let's denote this common length LG = LPo(v) for any node v with depth(u)= j. Since all original edge weights in Eg are strictly positive, it follows that for any two depths ji and j2, if ji < ja, then T.j. J.ja. This implies ji S j2 > T.j. S Top2-
The proof proceeds by induction on the number of added edges, k.
Base Case (k=1): We prove the statement for the addition of a single edge e1 = (u, v) to Gu to form Gi.
Sufficient Condition (=) ): Assume depth(u) S depth(r). By the lemma's premise, we are given that adding 6 results in G2 being a DAG. We must show that CP())) = CP(Go).
The longest path to any node x in G1 is given by the recurrence (x) = (w,)E)) (w) weight(w,, J. For node v, this becomes:
(P((v) max( LPoou) 0)
By definition, I.Po(U)= Tedepth(()) and (u) T.depth(u)- The condition depth(u) S depth(w) implies Ldepsh(u) I Leepah(u)- Thus, (v) max(L Ldepth(u)) Ldepen(s) LPo(v). Since the longest path to v is unchanged, and this is the only modification, the longest paths to all successors of v also remain unchanged. Therefore, (z) = LPo(z) for all E V, which implies CP((G)= CGG)..
Necessary Condition (+): Assume CP(G)) = CP(G)) and (as per the lemma's premise) G1 is DAG. We prove the contrapositive: if depth(u) (u) > depth(v). then CP(G)) CP((.)..
Since G₁ is a DAG, adding the edge (u, v) did not create cycle. The longest path to becomes: LP max( (u) + 0) max(Ldepth(o)) (u))th(u))
Since we assume depth(u) > depth(v) and all original edge weights are strictly positive, we have Ledett(()) > (ed This leads to (v) Leepth(a) Ldepth(u) = (P), The longest path to v has strictly increased. This increase propagates to all successors of including the sink t. Therefore, LP (1) > LPo(!), which means CP(G,) > CP(Go). This contradicts our assumption. Thus, the condition depth(u)= depth(e) is necessary.
Inductive Hypothesis (IH): Assume for some k > 1, the lemma holds. That is, given that Gk a DAG, CP((G)) = CPG))) if and only if the condition depth(u,) < depth(u:) held for all i {1...., k). We make the stronger hypothesis that if the condition held, then LP LP((x:) for all nodes x E V.
Inductive Step: We prove the lemma for the addition of the (I + 1)-th edge, ct+1 = (u, v). to Gx to form Gk-1-
16

---

Published as a conference paper at ICLR 2026
Sufficient Condition ( Assume depth(u) depth(v). By the lemma's premise, given that Gk+1 is a DAG. We must show CP(Gk+1) = CP(Gk).
The longest path to in GR-1 is LPk+1 (1)+1())a__))))))( max LPA (u)+ By the IH. since the ditions held for the first & edges, we have = = de)) and LPx(u) = LPo(u)= =
(u) The calculation identical to the base case: LPx+1(v) = max Leepth( Ldephh(w)
(o)) (o). The longest path is unchanged, and by propagation. LP++11 LPx
for all V. This maintains our strong hypothesis and proves CP(C)+1) = -CP(G)
Necessary Condition (-): Assume CP((a-1) = CP(G2) and (as per the lemma's premise) GK+1 is DAG, We prove the contrapositive: if depth(u) > depth(v),, then CP(C CP(G&).
Since Gk+1 DAG, adding (u, v) did not create a cycle. We compute (Px+1
( (nx( +)
Using the IH (CP(G)) = CP))))) implies the conditions held for the first k edges, so LPx(1) (P)(2)) for all z)):
[P)+1(U) max (ooo LPo(u)) max (edeptht)))) Lideph((u)
Since we assume depth(u) (u) > depth(v). we have Leepth(u) de) This leads LPs+1(u) (up Leepth(e) (P)(v) (P)(t). The longest path strictly increases. This in- crease propagates the sink node t, CP(Gx+1) > CP(Gk). This contradicts our assumption. Therefore, the condition necessary.
By the principle of induction, the lemma holds for any k 2 1.
C EXACT ALGORITHM AND MODIFICATIONS
We present the exact algorithm in Algorithm in in this section. The following pseudocode is adapted from the original FlashAttention-3 paper (Shah et al. 2024), with all modifications introduced by DASH explicitly marked.
17

---

Published as a conference paper at ICLR 2026
Algorithm DASH algorithm
Require: Matrices Q, V, O, dO E R Nxd in HBM, logsumexp vector L E R& in HBM, block sizes Be, Br.
1: In preprocessing kernel, compute D = rowsum 0) R (pointwise multiply), write to HBM and divide it into T, blocks Di,..., Dr. of size B, each.
2: Divide Q into T,
blocks Q1,..., Qr, of size B, each, and divide K, V in to B,
N
T. = N blocks K,,..., Kr. and V,...., Vr,, of size Bg x d each.
Be
3: Divide dO into Т, blocks dO,,.. dOr, of size B, x d each, and divide L into T, blocks Li...., LT. of size B, each.
4: Initialize pipeline object to manage barrier synchronization with s-stage circular SMEM buffer.
5: if in producer warpgroup then
6: Deallocate predetermined number of registers.
7: Issue load K2 and V, from HBM to shared memory.
8: Upon completion, commit to notify consumer of the load of K, and V,,
9: for in assigned Q-tile schedule do [DASH]
10: Wait for the (i % s)th stage of the buffer to be consumed.
Issue loads of Q, dO; from HBM shared memory at the (i % th stage of the buffer. 12: Upon completion, commit to notify consumers of the loads of Q; dO,,
13: end for
14: else if in consumer warpgroups then
15: Reallocate predetermined number of registers as function of number of consumer warps.
16: On-chip, Initialize dK;
(.)xx dV, j (0) Bxx..
17:
Wait for K, and V, to be loaded in shared memory.
18:
for in assigned Q-tile schedule do [DASH]
19:
Wait for Q to be loaded in shared memory.
20:
Load Li, D, from HBM on-chip SRAM.
On chip, compute S = Q.K? E B. (SS-GEMM). Commit,
21:
22:
Wait for dO; to be loaded in shared memory.
23:
On chip, compute dp(i) = dO, VJ
R R R (SS-GEMM). Commit.
On chip, wait for s(1) then compute p(O) exp (Sij - L,) E B, B.
24:
On chip, wait for dP()) then compute ds ()) p(() (dP(1) (3) D:) R B..BB.
25:
On chip, compute d dV, + dV, + (P dO; E Boxd (RS-GEMM) Commit.
26:
On chip, compute dk, dk, + ds ( Q Bexd (RS-GEMM). Commit and wait for both dV, and dk,
27:
d On chip, compute dQ. (local) ds! K2 B. (SS-GEMM), and write dQ. (local) to smem. Notify the dQ-writer.
28:
29:
end for
30:
else if in dQ- warp then
31:
for in assigned Q-tile schedule do [DASH]
Wait for dQ (local) to be ready in smem.
32:
Wait until the global order grants this block its turn to reduce. [DASH]
Using a semaphore, atomically add dQ (local) to dQ, in global memory.
34:
35:
Advance the global order. [DASH]
36:
end for
end if
18


Reviews:
[{"review_id": "GUYJ9ThiUh", "reviewer": "Reviewer_RNMq", "summary": "1. The paper builds on identifying the cause of nondeterminism in Flash Attention to be the atomicAdd for the dQ reduction, and identifies pipeline bubbles caused by the default ordering of processing the reduction, showing that it is specially prevalant in the case of causal attention because of the dependency order enforced by the causal mask. \n\n2. The authors propose a simple heuristic of reversing the order of the query blocks on the SMs and interleaving the processing of different heads to reduce the pipeline bubbles. \n\n3. The authors then model the scheduling problem as a critical path identification problem on a DAG, with within SM computations and reductions forming an isomorphic chain, and data dependencies across SMs (dQ transfer) as zero weight dependencies, and using that to inform the schedule to minimize critical path, thereby demonstrating strong gains over baseline deterministic variant of FA3.", "questions": "Besides my primary concern above, I had two other questions:\n\n1. The main issue for why we are unable to capture the long sequence length issue in the scheduling for the non causal mask setup seems to be the assumption of the zero weighted edge. Is that not something that can be factored in (eg: a cost reduction optimization on the DAG with weighted edges ?)\n\n2. [Minor] Given the popularity of FP8, would it be possible to also compare against FA3 on the a similar setup, just get get more confidance on the algorithm's applicability for lower precision training ?\n\nMinor Presentation Nits:\n\nThe authors go from representing a compute reduction blocks in the Gantt chart as a function of (kV index, query index) in Figure 2 to a (query index) representation in Figure 3. It is a bit confusing, so would be good to be consistent.\n\nIt would be good to include the exact algorithm (similar to how it's presented in FA3) for the proposed backward pass", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses an important area of deterministic training that has been gaining a lot of traction especially with large model training, bridging the gap between the non deterministic and the deterministic version of the attention kernel, which serves as one of the fundamental pieces in the commonly used transformer architectures.\n\n2. I especially like the in-depth analysis of when the theoretical model deviates from the on hardware execution results: both for scenarios with long context length for no mask scenario where the inter-SM communication latency starts becoming the bottleneck, and for the large head size analysis with the causal mask.", "weaknesses": "My only concern is that given the motivating backward's schedule analysis presented in Section 3.2, I would have expected the deterministic attention baseline to have been more competitive with shift scheduling for the non causal mask scenario. \n\nSimilarly, I would have expected the descending schedule to have been much better compared to baseline for the causal mask case with head size 64. That does not seem to be the case. Would it be possible for the authors to specify what might be the cause of it, since without that, it's a bit difficult to verify the correctness of the theoretical model, which is the underpinning of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R0WPexqX5o", "reviewer": "Reviewer_DP1T", "summary": "This paper introduces DASH (Deterministic Attention Scheduling for High-Throughput), a framework addressing the performance loss in deterministic backward passes of attention mechanisms, particularly FlashAttention-3. Determinism ensures reproducibility in LLM training but enforces sequential gradient accumulation, limiting GPU utilization.\n\nDASH formalizes the backward pass as a DAG scheduling problem, minimizing critical path length. Two strategies are proposed:\n\n- Descending Q-Tile Iteration (DQTI): reverses query-tile traversal to reduce pipeline bubbles in causal attention.\n\n- Shift Scheduling: a theoretically optimal schedule ensuring conflict-free accumulation, extended as Symmetric Shift Scheduling for causal masks.\n\nExperiments on NVIDIA H800 GPUs show up to 1.28× speedup over deterministic FlashAttention-3. However, for long sequences or large head dimensions, inter-SM communication latency and register pressure diminish the theoretical gains.", "questions": "- Do we really need sequential accumulation to ensure reproducibility? Can we do some sort of parallel reduction like in the binary tree to achieve better efficiency with determinism?\n\n- Are there similar challenges for the forward operation of attention?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel kernel implementation for an important operation in deterministic Transformer training\n- Clear and novel DAG-based formalization of deterministic backward scheduling.\n- The two scheduling strategies are well-motivated, combining theory and practicality.\n- Empirical validation on modern GPUs with thorough analysis of full vs. causal masks.\n- Addresses an important reproducibility issue in large-scale deterministic training.", "weaknesses": "- There is performance degradation for long sequence length with full attention mask (Figure 8)\n\n- The theoretical model ignores some of the GPU implementation considerations, such as inter-SM communication overhead and register requirements (as mentioned in sections 4.2 and 4.3)\n\n- Focuses solely on the backward pass; potential extensions to the forward path are not explored.\n\n- Symmetric Shift Scheduling introduces significant register pressure, limiting practical benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TfPjGgBT4", "reviewer": "Reviewer_rDjX", "summary": "DASH is a kernel implementation for attention that allows deterministic backward passes with less overhead, compared to the default non-reproducible kernels. To achieve determinism reduction operations are ordered, which incurs overhead. To reduce them, DASH employs two techniques. First, it fills the pipeline bubbles caused by the irregular ordering pattern of causal attention with the next attention heads ordered in reverse. Second, it formalizes reduction operations through a DAG of operations, and reorders the deterministic order so the critical path is minimized. DASH is tested on an H800 GPU and compared to a deterministic version of Flash-Attention 3, and achieves higher throughput on a variety of sequence lengths.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "Thank you for submitting your work.\n- I'd like to praise the authors for their presentation. This was a pleasure to read. I was astonished how such a dense topic was explained so well in the text.\n- FlashAttention 3 is a strong baseline, and the NVIDIA H800 GPU serves as a typical setup.\n- The ideas are sound and the reason they occur is intuitive, though less so for shift scheduling.", "weaknesses": "- Though a minor point, I think there is merit to discussing the effect of determinism in other operations of the transformer. Matrix multiplications include reductions as well, correct? Does determinism reduce performance there as well?\n- I think the paper should include a table that shows what the end-to-end relative benefit is for **a whole transformer block**, not just the attention part. While there is value in making attention faster, it is hard to put these gains in perspective without seeing the whole picture. For the transformer block configuration, pick a popular LLM, e.g., Llama3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dMpQBbUpmz", "reviewer": "Reviewer_aww5", "summary": "The paper focuses on improving the efficiency of the deterministic version of Flash-Attention (Backward). In FA, the backward involves an AtomicAdd operator to accumulate the dQ tensor. To make the kernel deterministic, it requires making the accumulation conducted in a fixed order. Existing methods introduce too many bubbles when handling this. The authors reframe this as a DAG scheduling problem and introduce DASH, a framework with two new scheduling strategies. Their methods improve throughput by up to 1.28x over the deterministic FlashAttention baseline by minimizing pipeline stalls and optimizing task execution order.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is clear and intuitive. 1.28x speedup is also quite practical.\n2. The proposed method, Shift Scheduling, is clear to understand and can be easily proved to be optimal.", "weaknesses": "1. \"Optimal\" Method is Sub-Optimal for Most State-of-the-Art Models: The paper's core theoretical contribution, the \"Symmetric Shift Scheduling,\" is shown to be less performant than the simple baseline when headdim>=128 (Figure 9b), a degradation attributed to register pressure. This is not a minor edge case: it is the standard configuration for a vast majority of today's most influential and widely used large language models, such as Llama 3, Qwen 3, Mixtral 8x7B, and DeepSeek, all of which use a head dimension >= 128. This severely limits the practical applicability of what is presented as a key contribution of the work.\n2. Scalability to Long-Context Scenarios is Questionable: the \"Shift Scheduling\" method's performance advantage diminishes with sequence length, and it ultimately becomes slower than the baseline at the maximum tested length of 16,384 tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)