Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
Authors: Jiayi Yuan
Hao Li
Xinheng Ding
Wenya Xie
Yu-Jhe Li
Wentian Zhao
Kun Wan
Jing Shi
Xia Hu
Zirui Liu
Venue: NeurIPS 2025 oral ()
Keywords: Large Language Models (LLMs)
Reproducibility
Numerical precision
Deterministic inference
Primary area: deep_learning
TL;DR: This paper demonstrates that low precision causes non-reproducible LLM inference across different setups, proposing a hybrid-precision method, LayerCast, that computes in FP32 to achieve determinism while saving memory.

Abstract: Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. 
This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9\% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size.
We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. 
This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge.
Our analysis reveals that floating-point precision—while critical for reproducibility—is often neglected in evaluation practices.
Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
Jiayi Yuan'
Hao Li
Xinheng Ding
Wenya Xie Xia Hu'
Yu-Jhe Li Zirui Liu? 'Adobe Inc.
Wentian Zhao
Kun Wan
Jing Shi
Riice University
University of Minnesota Twin Cities
{jy101, xia.),iiii edu (11003703, ding0499, xie00470, zrmiu)0umn. edd {jhel, wezhao, kuwan, (aanngshi@@ddbba Gadobe. com
Abstract
integral across various domains and have Large Language Models (LLMs) Progress, however, rests on the premise demonstrated impressive performance. that benchmark scores both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configura- tion, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pro- nounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under 6 precision with greedy decoding. a reasoning model like DeepSeek-R1 11-eee------iitii --iisttll--Qen----- B can exhibit up to 9% variation in accuracy and 9,000 tokens difference response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating- point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, quantify when and how model outputs diverge. Our analysis reveals that floating-pott precision-wiie critical for reproducibility- practices. Inspired by this, we develop lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://githin com/ hww.//aaoooomaaana http:/ocooo..oooooooooooooooooat
often neglected in evaluation
Introduction
Large Language Models (LLMMs) are increasingly being deployed in everyday scenarios, powering applications from chatbots 2 to automated coding tools (25) and personalized healthcare agents (6. As their impact grows, rigorous benchmarking and evaluation become critical to measure real progress and ensure reliability, safety, and fairness 7 37). There are two commonly used evaluation strategies. The first one is to use greedy decoding by setting the temperature zero, which produces deterministic outputs, and reports the result from a single run. The second uses random sampling with a non-zero temperature and reports performance using the Pass K metric [8, i... the probability that at least one of K independent solution attempts will succeed.
In this paper, highlight commonly overlooked factor in both evaluation settings: numerical precision. First, for the greedy decoding setting, this factor undermines the assumption of determin-
Equual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

BFloat16 24
0.6
(nspreecit ourl Q Oreia Weraa
Question:
Find the grealest reall number that less than SBD for all such rhombi
*Let $AS, SBS, $CS, and $D$ be point the hyperboka:
0.5
Greedy, Sead=42, BS=32, #GPU=4
Accuracy
0.4
Okay, so problem ... perpendicular, but square, for all such rhombi is (Mbbxxed/180N)
0.3
BF16
0.2
Okay, so have this problem perpendicular, Walt, no, hold on
0.1
for
Greedy, Seed=42,, BS-8, #GPU=4
Figure 1: Left: Under BF16 precision and greedy decoding, the model's output can vary significantly depending on factors such GPU count, evaluation batch size, and GPU hardware version. Right: For example, changes in evaluation batch size alone can lead to noticeable differences in responses, which is often ignored and not standardized by evaluation benchmarks.
ism-even with the same prompt and random seed, the generated output can still differ significantly between different hardware and system configurations. Second, for the random sampling setting, the numerical error demands a larger number of runs to control the variance. Surprisingly, we found that the results can be significantly different under the greedy decoding when changing the evaluation batch size, number of GPUs, or GPU versions. Through analysis, we found that the root cause of this problem is the non associaaivee property floating-point arithmetic, meaning (a + b) + c + a + (6 + e) due to finite precision and rounding errors. This issue is particularly problematic for recent reasoning- focused models [3], which generate very long chains of thought. such cases, small numerical errors accumulate during the token generation process, eventually leading to significant differences in output across different runs. As illustrated in Figure [ the model produces significantly different outputs when the number of GPUs, evaluation batch size, or hardware versions change- if the same random seed and greedy decoding are used. This inconsistency makes difficult to reproduce results, posing serious problem for measuring the progress.
We highlight the significance of this issue for several critical reasons. First, many researchers re- port benchmark performance based on a single inference run with a fixed random seed to reduce computational cost. However, as shown in Figure II This practice can lead misleading conclu- sions about model performance. Second, even when performing random sampling with multiple independent runs, the averaged results can still vary significantly due to hardware and system-level nondeterminism. Moreover, when researchers report standard deviations without accounting for this numerical nondeterminism, they risk severely overestimating a model's true uncertainty, since the reported variance reflects mixture of intrinsic model uncertainty and variance introduced by finite numerical precision. When results cannot be exactly reproduced, it becomes difficult to distinguish whether improvements from better methods merely random variation.
To fill this gap, we conduct a comprehensive analysis of how numerical precision and hardware configurations affect the reproducibility of LLMs. Our findings show that inference using the commonly adopted BF16 precision is highly sensitive to variations in hardware and system configurations, such as tensor parallel size, evaluation batch size, and GPU types. These hardware-relatee factors are often beyond users' control and can vary significantly due to resource availability, yet they are often overlooked in current LLM evaluation methods. We observe that increasing the number of mantissa bits in the numerical format-such as using FP16 or FP32 can significantly mitigate this issue. Based on these findings, we propose an optimized inference pipeline that performs all computations in FP32 while retaining model weights in BF16 precision. This approach effectively balances memory efficiency and reproducibility. Specifically, contributions and suggestions can be summarized as follows:
Extensive analysis of how numerical precision affects reproducibility in both greedy decoding and random sampling scenarios. Our finding suggests that due the limited precision, the model performance is highly sensitive to variations in hardware and system configurations, such as tensor parallel size, evaluation batch size, and GPU types.
Practical suggestions for reproducible reasoning to the community. Based on our findings, we suggest (1) If sufficient computational resources are available, please use random sampling (non-zero temperature) with multiple runs. Report the mean accuracy, average answer length, and

---

IEEE 754 Single Precision 32-bit Float (IEEE FP32)
Sign: 1 Bit
Exponent: Bits
Mantissa: 23 Bits
IEEE 754 Half Precision 16-bit Float (IEEE FP16)
Sign: Bit Exponent: Bits
Mantissa: 10 Bits
Value (-1)8ign x x(1 Mantissa) x Exponent bias
Google Brain Float (BFloat16 or BF16)
Sign: Exponent:
Figure 2: Floating-poin format of FP32, FPI6 and BFI6.
eπror bars. Note that with 16-bit precision and small datasets, more trials are needed for stable results in general. (2) If using greedy decoding with a single run, please use FP32 precision to improve the reproducibility of your results.
An optimized FP32 inference pipeline. We propose optimized inference pipeline called LayerCast, which performs all computations in FP32 while retaining model weights BF16 precision. This approach effectively balances memory efficiency and reproducibility. We release it as patch to vLLM and can used with just a few lines of code change.
2 Preliminary and Analysis
2.1 Current Practices on LLM Reproducibility
There are two widely adopted evaluation strategies. The first one greedy decoding, where tempera- ture is set to zero and model always select the token with highest probability. The second strategy employs random sampling with non-zero temperature, and evaluates performance using the Pass metric [8] Below, introduce the commonly adopted experimental setting for the reproducibility.
Greedy decoding
deterministic text generation strategy where the model always selects the token with the highest probability at each step. In theory, this approach should produce identical outputs given the same input and model parameters. However, this paper, we show that even greedy decoding can yield different results across runs due to numerical precision issues.
Random sampling selects output tokens based on the model's probability distribution with non- zero temperature to introduce randomness. The most commonly used evaluation metric is the mean accuracy across multiple independent trials, which is equivalent Pass @
Deterministic libraries and Random Seeds. Random seeds LLM generation control the pseudo- random number generator that selects tokens when using non-greedy decoding. is standard practice to fix random seeds; however, this not always sufficient for ensuring full determinism in greedy decoding. Even with fixed seed, differences in computation order can alter the sequence of the pseudo-random number generator [66]. Meanwhile, frameworks like Py and TensorFlow provide flags for deterministic behavior (e.g., torch. iise_deterministtie These ensure that certain operations avoid algorithms that could produce different results across runs. However, in practice, can still produce nondeterministic results even with these flags.
algori thms (True) D
Despite these efforts, achieving perfect reproducibility in LLM inference remains a challenge. The next subsection delves into crucial factor that often leads to nondeterministic behavior even with greedy decoding and the aforementioned practices in place.
2.2 Numerical Precision, Rounding Errors, and GPU Kernels
Numerical precision is critical for reproducibility (66. Higher precision numbers have less rounding error, which can reduce variability in results. Examples include using FP32 for certain parts of computation-like softmax or attention scores even the model weights are in 16-bit precision.

---

Table 2: Two illustrative cases demonstrate how rounding error, together with the non- associativity of floating-pon addition, can affect numerical results. Example reveals accumulation error at both precisions; Example 2 exhibits a discrepancy only in BF16, while FP32 delivers identical results, illustrating higher-precision numeric types are more tolerant of rounding errors.
Example
Sum Order
FP32
BF16
+ b + c
00111110010010011110000000111110 1001 101
001111
b. 0.1, 0.1.
+ b
1001 1001 110
001111 1001001110
b e
001111111000000000000011 10011100111
001 10000001
a, 0.0016, 0.0027, 1.0
+ b
0011111100000100011 1001 110011
00111 TI11111100000 110000000
This practice can mitigate run-to-run differences, but it doesn't 't provide theoretical guarantees since floating- point arithmetic cannot represent all real numbers exactly. Even with the same precision, dif- ferent hardware implementations or computation or- ders can lead to slightly different results. In this paper, we primarily focus on the importance of numerical precision and provide a detailed analysis of its impact on LLM inference reproducibility.
Table 1: Rounding error when the same true value (1.00012) is represented in three differ- ent numeric formats.
Precision
Decimal
Rounding Error
FP32
1000020043755776
4-4388e-8
FP16
1.0
00.00012
BF16
1.0
-0.00012
Figure 2 shows the format of BF16, FP16 and FP32. Table iillusratts the rounding error that occurs when the same true value is represented in three different numeric formats. As expected, lower precision formats like FP16 and BF16 introduce larger rounding errors compared to FP32.
Moreover, a key aspect of floating-pont -ointt arithmetic that contributes to nondeterminism is its non- associativity. This means that the order in which numbers are added can affect the final result due to accumulated rounding errors. Table 2
provides illustrative examples of how the order of summation can lead to different results in FP32 and BF16. As the examples demonstrate, although rounding follows deterministic rules, non-associativity introduces nondeterminism, which is further amplified by the larger rounding error of BF16. This becomes particularly relevant in the parallel computations performed in GPUs during LLM inference.
In the context of LLMs, even small numerical variations in the logit values can affect the final token selection when the top probabilities are close. Floating- point arithmetic in GPUs exhibits non-associativity, meaning (a + b) + c & + (b + c) due to finite precision and rounding errors. This property directly impacts the computation of attention scores and logits in the transformer architecture, where parallel operations across multiple threads can yield different results based on execution order. Formally, for a reduction operation @ over a set of values {v), U2, ...; Up (such as summing attention scores or computing softmax denominators), the result depends on the specific ordering of operations: D 1/2 = (1) @ (x) @ ... ① (n))); where π is a permutation of indices {1, 11.2...,,?? n determined by thread scheduling.
GPU Kernels add numbers in different orders when changing system configuration. In serv-
ing systems, several factors can change the order in which floating-point operations are executed, including: (1) continuous batching (36), which dynamically modifies the set of requests within a batch; (2) different operator implementations, such as Split-K versus Non-Split-K MatMul 23 (3) operator hyperparameters, like the block size in MatMul or FlashAttention; (4) collective operations in parallel settings, such as AllReduce; and (5) parallelization strategies, such as tensor parallelism (TP), which distribute computation across multiple GPUs Together, all these factors can affect the determinism of LLM inference. To address this issue, Batch-invariant operations [14, including batch-invariant Flex FlexAAteenin Attention, RMSNNrm, and MatMul kernels, are introduced. These kernels guar- antee that inference results remain deterministic regardless of batch size variations. Specifically, this approach achieves determinism by parallelizing computation along the batch dimension, thereby decoupling batch size from the final outputs of individual requests. However, as the name "batch- invariant" suggests, the technique is currently limited to handling variations related only to the batch dimension, making it robust to continuous batching and other mich-size-related changes, but not to other forms of nondeterminism like changing the TP sizes GPU types.
4

---

3 Reproducibility Issues with Limited Numerical Precision
3.1 Experiment Setup
We conduct experiments on four recent LLMss, including two reasoning models: DeepSeek-R1--Dssill!
Qwen-7B, DeepSeek-R1-Distill DDistillLLlama-88 and two non-reasoning models: Qwenn2.5-7BIInstrue
and Llama-3.1-8B--nstrre ---tt-nnttt [3 34 22], across five commonly used LLM evaluation benchmarks: SerCoodeBench-Meddime and LiveCodeBench-Haar [ 18 To verify the generalizability of our findings, we also conduct additional experiments on a larger model (Qwen3-32B) and a diverse reasoning benchmark (GPQA Diamond) covering graduate- level science questions, with results provided in the Appendix. For reasoning models, we set the maximum output token length to 32,768 and for non-reasoning models, we set 2,048. We primarily use vI.LM (0) as the inference backend; to verify that our findings are not backend- specific, we also conduct verification experiments using HuggingFace Transformers (see Appendix for details). In the random sampling experiments, we set temperature to 0.7 and top-p to 0.95. Our Evaluation implementation and prompt setting is adapted from SkyThought-evals 29] more details
AIME'24, 24, MATH500, LiveCodeBench--B
can be found in Appendix C
For each model-task pair, we evaluate under 12 different runtime configurations, representing all combinations of 2 GPU types (NVIDIA L40S and A100), 2 GPU counts (2 and 4), and 3 batch sizes (8, 16, and 32), i..., 2 environments encountered in real-world evaluations. Unlike decoding parameters or random seeds, these hardware-related factors are often beyond users' control and can vary significantly due to resource availability, yet they are often overlooked in current L.M evaluation methods.
3 12 different configurations, to simulate the diversity of deployment
To comprehensively quantify output instability and better analyze the impact of numerical precision on inference variability, we evaluate the reproducibility under both greedy decoding and random sampling settings. Specifically, for greedy decoding, we analyze:
Std Acc (Standard deviation of accuracy): For each numerical precision, we evaluate accuracy under the 12 different runtime configurations and compute their sample standard deviations, which serve as indicators of the stability of LLM inference outputs during greedy inference.
Avg Std @ @OtpuOttuu...mm Length (Average standard deviation of output length): We measure the length of output tokens, compute the sample standard deviation per example across 12 runtime config- urations, and report the mean of standard deviations over the entire dataset. This provides an alternative perspective on the stability of LLM inference outputs during greedy inference.
Div Index (Divergence index): For the same question, ixf two or more responses produce identical token sequences up to certain position, but generate different tokens after that position, we define the index of that position the divergence index. A higher Index indicates greater consistency across responses under different runtime configurations.
Avg Std @ @topp _ttpp_ppoobbbp__et_eeeeeee (Average standard deviation of top-1 token prediction probability):: Before divergence, all responses across different runtime settings produce identical top-1 tokens at every position. However, due to floating-pott point computation errors, the predicted probabilities of these tokens may vary across settings. To quantify this, we compute the standard deviation of the predicted probability for the top-1 token at each position across settings, then average over all positions from O to Div Index and over all examples in dataset. We define this metric as the Average Standard Deviation of Top-1 Token Prediction Probability, which serves as an indicator of the magnitude of numerical variation introduced by floating-poin errors.
For random sampling, we analyze:
@ass@1 1: For each numerical precision, we evaluate performance by running the model multiple times and computing the average accuracy accuracy (commonly referred to Pass @ l). Specifically, we use both 16 and 64 independent runs to compute mean accuracy for AIME'24 and independent runs for MATH500. To assess the stability of random sampling-based decoding, also report the standard deviation of P across 6 runtime configurations, varying GPU count (2 vs. 4) and numerical precision (BF16, FP16, FP32).
3.2 Greedy Decoding Deterministic Output
Contrary common belief, our experiments reveal that greedy decoding does not guarantee deterministic outputs across different hardware and system configurations. However, using
5

---

Table 3: @td@@ Acc of greedy decoding across 12 different settings (GPU types, GPU counts, and batch sizes) under BF16, FP16, and FP32 Numerical Precisions. Reasoning models also exhibit larger variance compared to non-reasoning counterparts. More results can be found in Appendix
AIME'24
MATH500
LiveCodcBench-
BF16
FP16
FP32
BF16
FP16
FP32
BF16
FP16
FP32
DeepSeek-R1 Rii------Dill-eweeee
9.15%
5.74%
1.04%
1.12%
0.12%
1.67%
1.28%
0,37%
DeepSeek-RU Barockk-Ri-iistiie D-sttiil-LLama------
4.60%
6.00%
5.8c-17
1.59%
0.73%
0.23%
2.31%
1.92%
0.29%
Qwwsst.-77--ustruett
1.71%
1.45e-17
1.45e-17
0.83%
0.36%
1.16e-16
0.79%
0.48%
1.16e-16
1.92%
1.30%
0
0.94%
0.34%
0.13%
1.00%
0.67%
0.25%
Table 4:
Standard deviation of output length of greedy decoding across 12 different settings (GPU types, GPU counts, and batch sizes) under BF16, FP16, and FP32 numerical precisions. The output length of reasoning models exhibit large variance. More results can be found in Appendix B
A1ME
MATH500
LiveCode Easy
BF16
FPI6
FP32
BF16
FP16
FP32
BF16
FP16
FP32
DeepSeek-RU PrepSeek-RI-Diiteee RI-Distill-wwnn=
9189.53
5990.32
2774.28
2090.46
138.75
5507.52
4282.78
262.55
DeepSeck-RI -ieees------itiii-----e Liam---
9348.59
7822.43
4015.00
2518.38
146.03
4732.85
3652.16
105.85
Qwen2.5-7B-Instueeee
211.47
48.14 14
52.61
15.37
0
7.79
0.71
0
Llama-3.1-8B. 3.struuttttt
119.21
49.73
124.43
40.57
2.76
31.03
4.70
0.49
FP32 precision helps a lot. Table 3 presents the standard deviation of accuracy (Std @ Acc) across 12 runtime configurations for BF16, FP16 and FP32 precisions. The results demonstrate clear pattern: FP32 consistently achieves near-perfect reproducibility with negligible variance, FP16 shows moder- ate variability, while BF16 exhibits substantial instability. This pattern is particularly pronounced for reasoning models like DeepSeek-RI- -ar--See-RR--isiill-------e RiD-Disiill--wen-7BB where BF16 precision introduces up to 9% standard deviation in accuracy on AIME'24 24, compared to virtually zero variance under FP32. This finding is concerning because it suggests that using different GPU types varying the number of GPUs can prevent you from reproducing others' results, even when using greedy decoding. Beyond accuracy variance, our results in Table 4 reveal that BF16 precision also causes response lengths to vary significantly across different settings. This raises significant concerns for recent work in efficient reasoning and long-to-short research 277). which needs to report response length metrics. Specifically, changing the system configuration can lead to a difference of up to 9,000 tokens in response length. This variability undermines the effectiveness of many existing efficient reasoning approaches. However, our findings show that using FP16 helps reduce output length variance to some extent, while FP32 offers the most consistent and reliable control over this variability. Thus, we suggest if using greedy decoding, please use FP32 precision to improve the reproducibility of the efficient reasoning research.
The performance gap clearly indicates that across different runs, the model generates different tokens. To investigate why this happens, we compared outputs from different runs. Since we are using greedy decoding, the top-1 probability token differs hetween runs at the point of divergence. Figure 3 shows example of token logits when two runs diverge, illustrating how numerical precision errors can flip the order of the top-1 and top-2 token probabilities. Figure 3 aalso shows the histogram of probability differences between top-1 and top-2 tokens under FP32 precision. We observe that for reasoning models, the token probability differences between the top two competing tokens are often minimal.
To further understand how this observation interacts with the numerical errors, Figure fur- ther illustrates the average standard deviation of the probability of prediction of the top-1 token (Avg_Std@)) Moop_PProh at different numerical precision levels. BF16 shows significantly higher variance in t token probabilities compared to FP16 and FP32. This increased variance arises because BF16's limited mantissa bits (7 bits compared to FP16's 10 bits) introduce larger errors in probability computations, increasing the likelihood of token flips when these fluctuations overlap with the small gap between top-1 and top-2 candidate tokens. In contrast, FP32's higher precision (23 mantissa bits) makes runtime variations nearly negligible. Together, these results demonstrate that token selection during greedy decoding is highly sensitive to even small numerical variations because of the minimal probability differences between competing tokens.

---

Deep-eekk-R1- Didi------illl------ee MATH500
250
Answer
Answer 2
Token
Prob.
Token
Prob.
200
know
49. 35.5%%
have 46.65%
have
43.91% %
know 46.64%
need
3.18%
need 3.39%
100
.m 2.47%
'm 2.63%
vve 0.49%
vve 0.52%
0.00
0.01
0.02
0.03
Top1-Top2 Tokan Probability Gap
Figure 3: Left: the top-5 tokens and their predicted probabilities at the divergence index for two different answers to the same question BF16. Right: The gap between the top-two competing tokens probability. We observe the token probability gap are often minimal in reasoning models.
IP16 Deeplesk-tt-- Sampsael-tt--st---eeee -ti---n------11
5216 Domppeeek-t--hhiil -laam---
2216 Queri 5-1B-rsiacc
BF26 Liama- 5--r------0-mmmmmett
FP16 DeeaSrec-R RI-0 -istti-Quee-T1
FP16 DeepSrei-A1- R1----------------- ama-88
FP16 Qwen2 5-7---------uucc
FF16 3-m--31-4B-nstrartt
FP32 Doesöbok RL DEzil Owen 78
#P32. DespSook RL Diszi Lona
FP82 Qwon2.5.78 instuct
FP32 Liama 3.1 Instruct
50
40
20-
F732
1932
F332
FP32
Avr.
20
LCB-Eany
LCB-Medium
LCB-Hard
Figure 4: Avg Std@topl_prob across 12 different settings for 4 models and tasks, under BF16, FP16 and FP32. FP16 shows significantly lower variance compared to BF16. FP32 yields near-zero variance, demonstrating strong robustness to foating-poin rounding errors.
The impact of numerical precision output sta- bility is further evidenced by divergence patterns in greedy decoding. Figure 5 shows the distribu- tion of divergence points (Div (Di__ndex) across pre- cision formats for DeepSeek-RI-Diseee -wen- 7B on MATH500. As precision increases from BFI6 FP32, we observe both fewer divergent examples overall and significant shift in when divergence occurs. With FP32, almost all ex- amples produce identical outputs across config- urations, resulting only 2.2% of the samples diverging. In contrast, with BF16, divergence frequently occurs early in generation despite deterministic setting, with over 90% of examples showing divergence. When divergence happens in higher precision formats, it typically occurs much later in the sequence, limiting its impact on the final output and answer accuracy.
50
96.6%
FP16: 73.0%
2.2%
40
Frequency
30
20:
10
10a Div lid_e (log scale)
Figure 5: Distribution of Div Index for -i--Seck-R--Disil---Oe
on MATH500 un- der BF16, FP16, and FP32. Higher numerical precisions lead to fewer divergent examples and shift of divergence point to later token positions.
These results conclusively demonstrate that numerical precision is critical factor in achieving truly deterministic outputs with greedy decoding, with higher precision formats providing substantially better reproducibility.
3.3 Random Sampling Has Reproducibility Issue, Too
One might argue that while greedy decoding is vulnerable to token-level instability from numerical precision, random sampling might less sensitive due its intrinsic stochasticity. However, our experiments reveal that numerical precision significantly affects the stability and reproducibility
7

---

Table Standard deviation of Pass@] performance (%) under different GPU counts and precisions. We emphasize that the reported values reflect the variability of Pass @ performance across different system configurations (3 batch sizes x GPU counts), not across repeated runs of the same configuration.
MATHS50 (n=4)
A1ME'24 (n=16)
AIME'24 (n=64)
BF16
FP16
FP32
BF16
FP16
FP32
BF16
FP16
FP32
DeepSeek-RI Meatt-iDill-eeeeeee
0.3158
0.1463
0.1021
1.7151
0.8273
1.1785
0.3749
0.5391
0.7377
DeepSeek-RI Parsseek-R--Disiee
0.3602
0.3371
0.1211
1.5124
1.8792
0.8606
0.8774
0.8539
0.5034
Owen2.5-7B-Instiee
0.4663
0.1686
0.0274
0.7056
0.2523
0.1784
0.1382
-lama-3.1-S--Instrue
0.6020
0.1725
0.3293
0.5992
0.2282
0.7759
0.4216
0.2898
0.1296
LiveCodeeench-HHrd
MATH500
MATH500
50
35
35
Avg. stdgttpp prob (x10'')
GPU
Avg. Stdigropp prob(x1 10
BS=16
Avg. Std.ttopl probtx10
29.6
29.1
25.1 29.6
38.6
BS=32
25
25
30
20
20
10:
10
1.0
6.6
5.
47
4.7
L40S
BS=16
Figure 6: Controlled experiments for Avg. Std Std@topl prob under different runtime configurations: (a) 2GPU vs 4GPU (varying batch size); (b) BS=8 vs 16 vs 32 (varying A100 GPU Count): (c) L40S vs A100 (BF16, varying GPU Count). Larger GPU counts and smaller batch sizes tend to increase inference instability.
of sampling-based evaluations as well. When using random sampling with temperature > O, researchers typically report the mean accuracy averaged over multiple runs (commonly referred to as Pass@1 1). We conduct Pass@1 evaluations across two benchmarks-AIMET 16 and 64 independent sampling that Table the reported standard deviation is calculated based on @ performance across system configurations per model (batch sizes and GPU counts), not across repeated runs of the same configuration. Thus, the reported variance largely reflects the impact from limited numerical precision, the inherent variance of models. As shown Table [5] we observe clear trend: numerical precision introduces an additional source of variance beyond the intended randomness, lower-precision precision formats such as BF16 tend to produce higher output variance. See Appendix G
and MATH550-using for AIME' 24 and 4 runs for MATH500. Here emphasize
and lower- for the complete Pass@) results that support these variance statistics.
This pattern largely holds across the models evaluate, especially the MATH500 benchmark. notable exception arises in the AIME'24 results for DeepSeek-R1-Disiie -ii-------siill-ee----- where at n = 64, FP32 exhibits higher variance (0.7377) than BF16 (0.3749). We interpret this result of dataset size and sampling dynamics rather than a contradiction of the overall trend. With only 30 problems in AIME'24, 24, single answer can shift Pass @ by about 3.33%, amplifying statistical noise. Moreover, BF16-which exhibits the highest variance at = 16-also shows the greatest improvement when increasing to n 64. This suggests that instability from reduced numerical precision can be mitigated with sufficient averaging, but remains dominant factor at typical sample sizes.
These findings highlight numerical precision as critical factor in the reproducibility of sampling- based evaluations. Researchers using random sampling with BF16 may need substantially more runs achieve the same statistical confidence with higher-precisie formats, representing computational overhead that is rarely acknowledged current evaluation practices.
3.4 Ablation: How Runtime Configurations Affect Reproducibility
After establishing that numerical precision plays a crucial role L.M reproducibility, we now investigate how specific runtime configurations batch size, number of GPUs, and GPU type-afféee

---

BF16 Tensor
FP32 Tensor
Precision Upcast
(Atuully Stored)
W4
W Korss
W sparsal
W downggn
(Add &)) Norm
[Add Norm
Softmax
X
Wx
QK
W
W Waren
Xut
W Vario
galeapie
Wv
W
padc
Figure 7: Workflow of LayerCast within one transformer block. Here we omit skip connections, position embeddings and activation functions.
output stability across different precision formats. We conduct experiments of greedy decoding setting on DeepSeek-R1 -iitt--illl---ae
as shown in Figure
Our analysis of runtime configurations reveals three key factors affecting token probability variations. First, in Figure 6 (a). configurations with 4 GPUs tend to exhibit higher probability variation than those with 2 GPUs across tested batch sizes (particularly in BF16 precision), potentially due to increased parallel computation introducing more varied doating-poine operation orderings and consequently different rounding errors. Second, Figure 6 (b) suggests that smaller batch sizes counter-intuitively produce higher variance in token probabilities because they may require more sequential processing steps that accumulate rounding errors, while larger batches benefit from parallel computation within optimized CUDA kernels that limit error accumulation. Third, GPU architecture matters: Figure (c) shows A100s generally exhibit slightly higher probability variance than L40S under identical configurations, likely due to differences in hardware-level floating-poin implementations and memory hierarchies. All these effects are most pronounced under BF16 precision with its limited mantissa bits making it especially susceptible to rounding effects. For more results of runtime configurations and tested tasks, please refer to the Appendix F
In summary, our experiments reveal three critical insights about numerical precision and reproducibil- ity in LLM inference. First, the fundamental cause nondeterministic outputs the small gap between competing logits, which makes token selection vuinerable to minute numerical fluctuations. Second, precision format critically impacts stability, with FP32 providing near-perfect determinism, FP16 offering moderate stability, and BF16 exhibiting significant variance despite being commonly used. Third, specific runtime configurations-ppane passspppatcullrlyyy GPU count, batch size, and GPU ar- Chitecture-further affect reproducibility, with these effects most pronounced in lower precision formats. These findings highlight the urgent need for standardized evaluation practices that account for numerical precision effects, especially as LLMs continue to be deployed in increasingly critical applications where reproducibility essential.
4 Near-Perfect Deterministic Reproduction: LayerCast
Given our findings when and why reproducibility breaks, now propose directions to improve reproducibility in LLM inference. The basic solution using FP32 precision, as we've shown in previous sections. However, this approach incurs significant costs: it doubles the memory usage and inference time compared to BF16, making it impractical for many production environments.
We propose a more efficient solution: LayerCast, hybrid precision approach that balances compu- tational stability with memory efficiency. LayerCast works by: (1) Loading the model parameters initially in FP32 precision; (2) Explicitly casting all linear layer weights and biases to BF16 for storage before inference; and (3) As inference runs, upcasting each weight back to FP32 just-in-time for matrix multiplication, one at time.

---

As illustrated in Figure 7 this approach ensures all computations occur in full FP32 precision while storing weights in memory-effcient 16-bit formats. Thus, this model benefits from FP32's stability during computation, while the memory footprint remains closer to that of 16-bit models. This provides determinism comparable to full FP32 inference but with substantially reduced memory requirements, particularly beneficial for the KV cache in long-context scenarios.
Our experimental results strongly support this ap- proach. When examining the standard deviation of accuracy across runs, Layer Cast achieves stability nearly identical to FP32, while BF16 shows much higher variability. As shown in Figure [] the di- vergence index measurements further confirm that LayerCast produces consistent outputs across dif- ferent batch sizes and GPU configurations, with
50
BF16
FP32 LayerCast
40-
Frequency
30
20
divergence rates below 3.4%. From resource per- spective, LayerCast offers substantial benefits over full FP32: memory usage is reduced by 34% (par- ticularly important for KV cache in long-context scenarios). These improvements make LayerCast practical solution for applications requiring both deterministic outputs and reasonable performance. For full result, please refer to Appendix H
10
0
10a
10a
Div Index (log scale)
Figure 8: Distribution of Div Index for --reee----itsssl--ee------ on MATH500 under BF16, FP32, and LayerCast.
5 Related Works
Since the era of traditional deep learning, the reproducibility of models [28 39 33 have remained
complex and challenging problems that have yet to be fully resolved. As large language models (LLMs) have risen to prominence, numerous empirical studies [21] 3 I 19
32
24 38) have
shown that nondeterministic behavior during LLM inference widely observed. Existing study [S) have found a clear negative correlation between output length and inference consistency: As the length of generated text increases, output variation during inference also rises, which explains the phenomenon that reasoning models tend to exhibit greater inference uncertainty.
Multiple factors can affect the reproducibility of LLM inference results, including but not limited to prompt formatting, decoding parameters (such as temperature and top-p thresholds), random seed settings, and hardware and software configurations [17 10 9.. Existing studies 24
21 4 have systematically analyzed the impact of decoding parameters (e.g.. temperature, top-p) on the stability of LLM inference outputs. Hochlchnert al. (16) points out that many reported improvements in LLM performance are, in fact, partially attributable to unfair comparisons and unreported sources of variance in the evaluation process. In practical applications, using FP32 single-precistee point) inference [30] 31 is often empirically believed to enhance the robustness of numerical computations. Feng et al. [2] investigates the impact of numerical precision on the mathematical reasoning ability of LLMs. However, there is currently lack of systematic and quantitative studies specifically analyzing the effects of different numerical formats (e.g., FP32, FP16, BF16) on the reproducibility of L.I.M inference.
floating
6 Conclusion
In this paper, we conducted a comprehensive investigation into the reproducibility challenges in I.M inference caused by numerical precision issues. Our experiments across multiple models, tasks, and hardware configurations revealed that even under supposedly deterninistic greedy decoding, outputs can vary significantly due to Doating-point arithmetic non-associativity. We demonstrated that precision format critically impacts stability, with FP32 providing near-perfect determinism while BF16-despite being widely used-exhibits substantial variance. To address these challenges without incurring the full overhead of FP32, we proposed LayerCast, a practical solution that achieves FPP2-level determinism while maintaining reasonable memory efficiency. Our findings highlight the importance of standardizing evaluation practices to account for numerical precision effects, especially as LLMs are increasingly deployed in critical applications where reproducibility is essential.
10

---

References
[1] Aime 2024 dataset. http:///wnnniggteeeeerrr/aretttr
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[3] Chetan Arora, Ahnaf lbn Sayeed, Sherlock Licorish, Fanyu Wang, and Christoph Treude. Optimizing large language model hyperparameters for code generation. arXiv preprint arXiv:2408. 10577, 2024.
[4] Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe. Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej. Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. Non-determinism of "deterministic" Ilm settings, 2025.
[5] Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408. 046667 2024.
[6] Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasibility of chatgpt in healtheare: an analysis of multiple clinical and research scenarios. Journal of medical systems, 47(1):33, 2023.
[7] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models, ACM transactions on intelligent systems and technology, 15(3):1-45, 2024.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint crX:122077000. 2021.
[9] Huggingface Docs. Reproducibiiltty
0/ htwp:/ggdd....ueeeeeppprrrccbbbd 2025. Accessed: 2025-04-25
[10] PyTorch Docs. randomness html 2025. Accessed: 2025-04-28.
Reproducibility.
https://yyorrd.ooo////////tt///t
[11] Benedetta Donato, Leonardo Mariani, Danicla Micueci, and Oliviero Riganelli. Studying how configurations impact code generation in llms: the case of chatgpt. arXiv preprint arXiv:2502. / 7450, 2025.
[12] Guhao Feng. Kai Yang. Yuntian Gu, Xinyue Ai, Shengjie Luo, Jiacheng Sun, Di He, Zhenguo Li, and Liwei Wang. How numerical precision affectt mathematical reasoning capabilities of llms. arXiv preprint arXiv:2410. 13857, 2024.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[14] Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm inference. Thinking Machines Lab: Connectionism, 2025. nondeterminism -iniiiiis-ii---lll- in-IIm-infferenee!
[15] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 203i::101.00.. 038744 2021.
[16] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.070Ss 2025.
[17] Ingonyama. ing and 1lms: - wwtperrrrooouibbiyyyyyooooooo cha cealeesss in -earmi-gaan 11ms our journey
Solving reproducibility
challenges in deep learn-
Our journey.
https: // /ww. http://wwwwnnwnnaaa...mmmma com/post/
2025. Accessed: 2025-04-28.

---

[18] Naman Jain, King Han, Alex Gu, Wen- Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0799 2024.
[19] Eugene Klishevich, Yegor Denisov-Blanch Simon Obstbaum, Igor Ciobanu, and Michal Kosinski. Measuring determinism in large language models for software code review. arXiv preprini arXiv:2502.2074, 2025.
[20] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 626, 2023.
[21] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21577 2023.
[22] Meta. Introducing Ilama 3.1: Our most capable models to date. https://ai. meta. com/
blog/meta- 11w-a-aaaaaa--------------a---ma 3- 1/ 2024, Accessed: 2025-05-13.
[23] NVIDIA Corporation. Efficient gemm in CUTLASS. NVIDIA Developer Documentation.
[24] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. An empirical study of the non-determinism of chatgpt in code generation. ACM Trunsactions Software Engineering and Methodology, 34(2):1-28, 2025.
[25] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[26] Sanjif Shanmugavelu, Mathieu Taillefumier, Christopher Culver, Oscar Hernandez, Mark Coletti, and Ada Sedova. Impacts of floating-poin non-associativity on reproducibility for hpc and deep learning applications, 2024.
[27] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang. Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint 2xXi::5503.1611 2025.
[28] Cecilia Summers and Michael Dinneen. Nondeterminism and instability in neural network optimization. International Conference on Machine Learning, pages 9913-9922, PMIR,
2021.
[29] NovaSky Team. Sky-tl: Train your own ol preview model within $450. https:// novasky-ai wwt.gg.ttoooggo httppssssss.sstttlllllssl 2025. Accessed: 2025-01-09.
[30] vllm Issue 12343, Why are the vllm and hugging face transformers inference results incon-
hattps://githur http:///ha...aaoaoooooopo/ooaaa/ 2025. Accessed: 2025-04-28.
[31] vllm Issue 12699. Different logprobs for qwn2-vl when rumning on transformers and on vllm. https://github.ccom
2025. Accessed: 2025-04-
28.
[32] Julian Junyan Wang and Victor Xiaoqi Wang. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. arXiv preprint 2rXiv:2503.1669
2025.
[33] Xiangzhe Xu, Hongyu Liu, Guanhong Tao, Zhou Xuan, and Xiangyu Zhang. Checkpointing and deterministic training for deep learning. Proceedings of the 1st International Conference on Al Engineering: Software Engineering for AI, pages 65-76, 2022.
[34] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint 1rX1v:2412.15115. 2024.
12

---

[35] Boyang Yu. Benchmarking large language model volatility. arXiv preprint arXiv:2311.15180, 2023.
[36] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A distributed serving system for (Transformer-Basees generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, 2022.
[37] Jiayi Yuan, Jiamu Zhang, Andrew Wen, and Xia Hu. The science of evaluating foundation models. arXiv preprint orXiv:2502.06670, 2025.
[38] Qiuhai Zeng, Claire Jin, Xinyue Wang, Yuhan Zheng, and Qunhua Li. An analyst- inspector framework for evaluating reproducibility of Ilms in data science. arXiv preprint arX1v:250..16395, 2025.
[39] Donglin Zhuang, Xingyao Zhang, Shuaiwen Song. and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. Proceedings of Machine Learning and Systems, 4:316-336, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are supported by Section 4 and 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: Not released
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwiicccguie//eeeeeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:
Answer: [Yes]
Justification:
The research conform with NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Properly cited and stated
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer:
[NA]
Justification:
Not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not related
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not related
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification:
Not related
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httees////earrppcccc////f/rreers for what should or should not be described.
20

---

A Limitations
Despite the importance of work in highlighting and addressing numerical precision's 's effect reproducibility, there are limitations that are worth noting. First, our experiments focus on a certain set of models and settings. Due to resource constraints, do not conduct large-scale experiments much larger models (>70B) or a wide range GPU architectures or accelerators; these settings might exhibit different levels of numerical behaviors. Additionally, our study primarily addresses numerical precision issues in transformer-based LLMs and may not fully generalize to other architectures or modalities.
B Broader Impacts
This research has several societal impacts. First, by raising the discussion of reproducibility issues in LLM inference, our work promotes greater scientific rigor in Al research, calling for a more reliable comparison of models and techniques. This is particularly important as LLMs are increasingly deployed in critical areas like healthcare, education, and public services, where consistency and repro- ducibility are essential. On the other hand, our findings reveal that true reproducibility may require additional computational resources, potentially exacerbating the already significant environmental effects of LLM development. We hope our work will encouage the Al community to establity better standards of idlluatioo-ultimaale leading to more trustworthy and reliable Al systems.
C Prompt Formats Used in Our Evaluation
Our evaluation implementation and prompt formats adapted from SkyThhough repository. AIME'24 and MATH500 share common prompt format across all models. LiveCodeBench-Fassy LiveCodeBench-Meddue
and LiveCodeBench-Hall models, which varies by the test type of the problems (stdin, functional), The -__BPRROOEM_TEE the placeholder that will be replaced with specific problem texts from benchmarks during prompt- ing.
also have the same prompt format across all
C.1 MATH Benchmarks: 24ME'24 and MATH500
DeepSeek-Ri-Distill--eeenr and -ee-Seel-R--Distll---
< <lbegin_or faaiih_of_eetteeee >>< User >Return your final response within (bboxed)). PROBLEM TEXT /ssisstant/><thinkze
Owen2.5-7B-Instruct
< ilimsstart |systemmnYYo are Qwen, created by Alibaba Cloud. You are
+ a helpful assistant. i|i_edd/>\n</ laokllm_starre >user \nReturn your
+ final response within \lboxed(). -- PROBLEM TEXT <I iim_end] >\n
<| imm_start/ bassistant
Llama-3.1-BB-Instrrue
< begin__ wtwtttff_eo_eeeeeeeeeeeeeeeeee st header __dd| (idl>systen >system< end header id i >n Annnnniinnnnnnggn
Knowledge Date: December 2023 nToday Date: 26 Jul 2024\n\n
id >user< end end htw__heede__re idd >n Annuuuuu your final response within \ {boxed().
beot_id/><\starryy
PROBLEROBBMMME.. TEXTTT_TEET____ </eot_id|>
<lstart_header id >assistant< had__headeriid in
21

---

C.2 Code Generation Benchmarks: LiveCodeeBnnch-asy, LiveCoddeBenc--Medddn, LiveCodeBench-Hard
DeegSSeds-R---isiill---ee------ and DeepStek--R--Disill-llaa---
For problems with "stdin" tests:
< <|begin_ cooegin_of_re of__setenneeeleeeeeeeeeeeeeeeeel | |Uer |Senerrte an executable Python function generated from given prompt. The function should take stdin + as input and print the output. Simply call the function after the definition. PROBLEMM TEXT < (issss/</ksssttttt///hhhaa
For problems with "functional"
< <bbegin_of oft_t_fn_seteeeeeeeeeeeeeee |Generaaeee executable Python function generated from the given prompt. Return the function body without invoking it at the final solution. ...-PROBLEM_ TEXT__TEE_____ - Assistantt <think>\n
Qwen2.5-7B-Instruce
For problems with "stdin" tests:
www.ssttrt//yysmmml/ky.... are Qwen, created by Alibaba Cloud. You are
an + executable Python function generated from the given prompt. The
helpful assistant. hnt.n//onddluiiimmaamttttt//aaee
function should take stdin as input and print the output. Simply
call the function after the definition. PROBLEM_TXXTT
<lim_end/>\n<< in_ssimtttt/____ Dassistant st
For problems with "functional" tests:
<
iim_start/////s |>system/nYou are Quen, created by Alibaba Cloud. You are an + executable Python function generated from the given prompt.
helpful assistant. im start///use//nGeeeeeee
Return the function body without invoking it at the final
solution. P..BLLLE_MEEEC < llm__nnd /and>>immssttttttt im_sart >assistant In
Llama-3.1-8B-Instrus
For problems with "stdin" tests:
<
<bbegi__of_teke
-n >system< /end Baaad__hader__ Betader_id/>>e id| >ww/nnnnnnnnnnnnnggcccmoo
Knowledge Date: December 2023 nnoday Date: 26 Jul 2024\n\an id executable Python function generated from the given prompt. The
+ [oot_idd><sttart__ng
function should take stdin input and print the output. Simply
call the function after the definition. PROBLEE_ TEXT _eet____eeee___idddlll <lstart_header, id wassistants headd_header... seader_id/>Ant id/>\n
For problems with "functional" tests:
< beggin text |><| header id >system< end header id > >n AnAnCutting stadeheaderr__rr__
Knowledge Date: December 2023 nooday Date: 26 Jul 2024\n
InGennertee an executable Python function generated from the given prompt.
>user< end__eader__iddeeeaaaaa
Return the function body wit without invoking it at the final
solution. PROBLEEE_MMMTE TEXEETEEEEEE__EEEEEEEEE </eot_id id|x startksaaerreerdidd Passistann
<lend_ hend__haader_id(>hnn
22

---

BFloat16 MATH500
BFloat16 LiveCodeBench-Eaxy
0.9
0.8
0.8
0.7
0.7
Accuracy
DeepSeek-
Accuracy
0.6
DeepSeek
DeepSenk-A R. -istill-Qeen-8
DeepSeek-
(--eee------iill--eeee Distäl-Qween--
Uama-33J 2.1---31-66 8i-isrruct
Liama-3.1 38-Instruct
0.6
Owen2 3-/B-instruct
0.5
0.4
0.4
BFloat16 WweCodeBench-Medivee
BFloat16 LiveCodeBench--Hard
DeepSenk-R1
0.14
DeepSenk-A R1--iiiii------i--i---- Llama-88
DeepSeek-R1 Distill-Qeeen11 Qwen-/B 1aann-3.-4--insttuur
DeepSeek R1-Distil-Qeen-V Quen--u
0.35
Llama-3.1-88- 1-98-------
Qwen2.5-72 3-1B-instruct
0.12
Qwen2.5 7-B-instrutt
0.30
0.10
0.25
Accuracy
Accuracy
0.08
0.20
0.06
0.15
0.04
0.10
0.05
0.02
0.00
0.00
Figure Accuracy varies significantly across different settings under BFloat16.
D
Supplementary Results on Accuracy Variance under BFloat16
In Section [ we demonstrated the variance of accuracy on BF16 with the same seed and greedy decoding but different batch sizes and number of GPUs in Figure [ To provide a comprehensive understanding of reproducibility challenges across different problem domains, Figure eetendds this analysis to all evaluated datasets. The results reveal clear pattern: while accuracy variance under BF16 precision is relatively modest for MATH500 and LiveCodeBench-Eass it becomes substantially and LiveCodeBench-HLae
more pronounced for the more challenging benchmarks. This dataset-dependent variation suggests that the impact of numerical precision on reproducibility scales with problem complexity.
E
Supplementary Results on Greedy Decoding
Following up 5 and Table Table Section 3.2 present comprehensive results across all experimental settings. We report three key metrics: Std Acc, Div Percent, and Average Div Index.
Table 6 presents the standard deviation of accuracy (Std@ Acc) across 12 runtime configurations and LiveCodeBencch-HHo
under BF16, FP16, and FP32 precisions for LiveCodeBBench-eeeee datascts. Table reports the average standard deviation of output lengths (Avg Std @ Output Length)
23

---

Table 6: @td@@ Ace of greedy decoding across 12 different settings (GPU types, GPU counts, and
batch sizes) under BF16, FP16, and FP32 numerical precisions on LiveCodeBench datasets
LiveCodeeBnnc--Medin
LiveCodeBench-Hart
BF16
FP16
FP32
BF16
FP16
FP32
ReepSeck-RI--istill---eeaae
2.28%
2.21%
0.44%
1.19%
1.98%
0.42%
DeepSeck-RR DeepSeck-Rt-Dittlll R1-Disstil--Lamma---
2.08%
1.84%
0.78%
2.15%
1.90%
0.53%
Qwen2.5-7B-Instee
1.44%
0.24%
0
0.40%
1,45e-17
1,45c-17
0.70%
0.48%
0.44%
0.65%
3,62e-18
3.62e- 18
Table 7: Avg_ @rd_Std@Oiil Sad@@utppt_LLnngee of greedy decoding across 12 different settings (GPU types,
GPU counts, and batch sizes) under BF16, FP16, and FP32 numerical precisions LiveCodeBench datasets
LiveCodeeRench-Mediun
LiveCodeBench-Harr
BF16
FP16
FP32
BF16
FP16
FP32
DeepSeck-RI--Dstill--ueee
8893.02
8026.63
547.50
8036.21
7476.98
646.30
8240.77
7842.29
646.30
7518.49
6827.19
690.86
Qwenn.5--B-Instruee
26.47
6.18
36.01
9.26
60.25
7.49
0.89
78.26
15.95
2.75
across the same runtime configurations and precision settings for these two datasets. In Table we measure the Div Index of each example across 12 runtime configuaations-witeee the O-Div_Inde Index range, the responses corresponding position-aad and report the average Div _ndeex over the entire dataset. Table shows the percentage of examples each dataset exhibiting divergent outputs across the 12 runtime settings.
the same problem have exactly the same token each
These results consistently support our conclusion that during greedy decoding, rounding errors from floating-point operations cause serious reproducibility problems. As numerical precision increases from BF16 to FP32, the divergence phenomenon is substantially reduced, evidenced by lower Std Acc, lower Div Percent, and larger Div Index values.
E.1 Additional Results on Larger Models and Diverse Tasks
To further validate findings the impact of numerical precision reproducibility, conducted additional experiments on a larger model (Qwen3-32B) and a diverse reasoning task (GPQA Dia- mond). These results demonstrate that the reproducibility challenges we identified are not limited to 7-8B models or math/coding tasks.
Larger Model Evaluation (Qwen3-32B): To demonstrate the impact of numerical precision on reproducibility for larger models, we benchmarked Qwen3-32B on the AIME'24 dataset under different runtime configurations covering 2 GPU counts (2 and 4), 2 GPU versions (A100 and L40S), and 3 batch sizes (8, 16, 32). Due to memory constraints, Qwen3-32B cannot be loaded into L40S GPUs with FP32 precision, resulting in 9 feasible configurations for FP32 out of the 12 total configurations. The results in Table I0 confirm that larger models exhibit the same reproducibility challenges identified in 7-8B models. Across all configurations, BF16 shows significant variance with Std @ Acc of 5.02%, while FP32 maintains much better consistency with Std Acc of only 1.11%.
T0
Diverse Task Evaluation (GPQA Diamond): To demonstrate that reproducibility issues extend beyond math and coding tasks, we evaluated DeepSeek- SeeeSeeek---- RI--Ditill--Lamaa on the GPQA Diamond benchmark, which focuses on graduate-level science questions. Table shows the results across 12 different runtime configurations. The results reveal that BF16 exhibits Std@ Acc of 3.03% compared to 1. 1.96% for FP32, confirming that numerical precision affects reproducibility across diverse reasoning domains.
24

---

Table 8: Average Div Index across 12 different settings under BF16, FP16, and FP32 numerical precisions. In the table, a value of "-1" indicates that no divergence occurred for any example in the dataset under the given setting.
DeepSeek-RIe PeepSeek-RI-Distill Qwen-7B
DeepSeck-RI -I--ssll--- Llama-8B
Qwen2.5-7B- Instruct
Llama-3.1-81- Instruct
BF16
45.67
67.37
82.37
22.44
AIME'24
FP16
430.13
430.47
457.41
635.43
FP32
13520
1976.40
BF16
65.33
76.85
103.58
38.07
MATH500
FP16
395.46
463.14
267.59
284.89
FP32
1825.79
1855.17
1509.61
BF16
35.11
47.43
46.98
53.99
LCB-Easy
FP16
292.99
327.53
87.53
133.80
FP32
1143.30
1036.04
108.86
BF16
44.23
47.59
44.23
54.33
LCB-Medium
FP16
267.81
287.83
107.07
172.73
FP32
2291.83
2585.83
125.00
BF16
43.85
31.57
43.64
54.95
LCB-Hard
FP16
216.16
329.38
136.89
221.76
FP32
5807.8
3832.54
352.78
Table Div_Percent across 12 different settings (GPU types, GPU counts, and batch sizes) under BF16, FP16, and FP32 numerical precisions
DeepSeck-RR CrepSeck-R1-Distiine RI-Distil--
ReepSeck-R1--iitee Llama-8B
Qwen2.5-7B- Instruct
Llama-3.1-8B Instruct
Qwen-7B
BF16
100%
100%
53.33%
AIME'24
FP16
100%
100%
73.33%
23.33%
FP32
6.67%
6.67%
16.67%
BF16
99.60%
100%
90.20%
77.60%
MATH500
FP16
86.00%
87.80%
37.60%
36%
FP32
5.80%
6.00%
1.80%
9,20%
BF16
100%
100%
72.53%
96.15%
LCB-Easy
FP16
100%
97. 07.25%
16.48%
41.76%
FP32
10.99%
13.19%
3.85%
100%
100%
89.32%
98.06%
LCB-Medium
FP16
100%
100%
35.44%
48.06%
FP32
19.90%
23.30%
3.40%
100%
100%
95.12%
100%
LCB-Hard
FP16
100%
100%
50.41%
58.54%
FP32
24.39%
30.08%
7.32%
These additional experiments on a larger model and diverse reasoning task further validate our core findings: numerical precision critically impacts reproducibility across model scales and task domains, with BF16 consistently showing higher variance compared to FP32.
E.2 Verification with HuggingFace Transformers Backend
To verify that the reproducibility issues we observed are not specific to the vLLM inference backend, we conducted additional experiments using the standard HuggingFace Transformers implementation without vLLM or PagedAttention. The experimental setup identical to main paper: 2 GPU
25

---

Table 10: Accuracy of Qwen3-32B on AIME'24 under different runtime configurations (GPU types,
GPU counts, and batch sizes) and numerical precisions. OOM indicates out-of of-mmmorymyyy emrors.
GPUs
2 GPUs
Precision
A100-BS8
A100-BS16
A100-BS32
100-BS8
A100-BS16
A100-BS32
BF16
80.00%
83.33%
83.33%
76.67%
83.33%
73.33%
FP32
76.67%
76.67%
76.67%
76.67%
80.00%
76.67%
4 GPUs
2 GPUs
Precision
L40S-BS8
L40S-BS16
L40S-BS32
L40S-BS8
L40S-BS16
L40S-BS32
BF16
73.33%
83.33%
70.00%
83.33%
76.67%
73.33%
FP32
76.67%
76.67%
76.67%
OOM
OOM
OOM
Table 11: Accuracy of DeepSeck-RI RI-Distill-Llamal on GPQA Diamond under different runtime configurations (GPU types, GPU counts, and batch sizes) and numerical precisions.
GPUs
GPUs
Precision
A100-BS8
A100-BS16
A100-BS32
A100-BS8
A100-BS16
A100-BS32
BF16
33.84%
38.38%
41.41%
40.91%
41.92%
41.41
FP32
40.91%
45.45%
43.43%
40.91 %
43.43%
40.91%
GPUs
GPUs
Precision
L40S-BS8
L40S-BS16
L40S-BS32
L40S-BS8
L40S-BS16
L40S-BS32
BF16
43.43%
38.38% 42.42%
44.95%
38.89% 42.93%
43.94%
41.92%
FP32
38.89%
44.95%
41.41 %
44.44%
counts (2 and 4), 2 GPU versions (A100 and L40S), and batch sizes (8, 16, 32), covering 12 settings for each precision.
Table 12 shows the results for two models on AIME'24. The same pattern emerges: FP32 inference fully reproducible with 0% Std@ Ace for both models, while BF16 exhibits Std@ Ace of 2.23% for Swen2.5-7B-Innttee
and 1.51% for Llama-3. I--B-IInstree similar variance what reported with vLLM. These results demonstrate that the reproducibility phenomenon is fundamental to GPU hardware and floating-poin arithmetic, and is observable across different inference backends.
Table 12: Std@ Acc using HuggingFace Transformers backend (without vLLM) AIME'S 24 across 12 different settings, confirming that reproducibility issues are not backend-specific.
Model Dataset
Std @ Ace BF16
Std Ace FP32
D0wen..57BB-Insiee BnInstuutt/ AIME'24 24
2.23%
AIME'24 24
1.51%
Supplementary Results on Ablation Study
provide additional results that isolate specific runtime configurations to examine their individual effects on reproducibility. For greedy decoding experiments, we focus exclusively on BF16 and FP16 precisions, since FP32 precision effectively mitigates pammericalpprecisio--eeelaee errors and thus provides limited insight into reproducibility challenges.
Continuing the discussion from Section 3.4
Tables 13 and 14 discuss the effect of different numbers GPUs since, intuitively, the effect has
dependency on the GPU version, so we consider the situation on L40S and A100, separately. Table 13 reports the si__Stt@topp__poo of L.LM inference responses for the same question using 2 L40S GPUs under three different batch sizes (8, 16, and 32), while Table [4 presents the
26

---

Table 13: Instability (Avg_Std@topl_ (x10-²)) under different numbers of L40S GPUs
DeepSeek- Ri--sssill-
SeepSeck-RI-Distee
Qwen2.5-7B- Instruct
Llama-3. Instruct
Qwen-7B
Llama-8B
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
BF16
29.66
24.52
23.02
28.01
29.40
34.05
22.12
44.54
AIME'24
FP16
5.83
5.19
3.18
3,95
2.94
3,91
2.89
7.30
BF16
32.91
32.67
23.03
27.33
24.91
35.92
42.31
MATH500
FP16
4.48
4.72
3.19
3.55
3.79
3.75
4.80
5.75
BF16
42.80
46.82
28.46
34.80
28.48
30.31
28.05
33.93
LCB-Easy
FP16
6.03
6.55
4.27
4.68
4.29
4,42
3.62
4.45
BF16
38.44
44.52
30.25
39.24
29.68
34.56
27.62
36.34
LCB-Medium
FP16
7.04
7.68
4.52
5.00
4.84
4.74
3.45
4.05
BF16
38.63
48.74
29.76
36.10
34.31
29.60
28.90
34.60
LCB-Hard
FP16
7.36
5.04
5.58
5.03
4.85
3.17
5.61
Table 14: Instability (Avg_ (Avg_Std@topl_ppig (x (x10-1) under different numbers of A100 GPUs
DeepSek--R-Diitiie DeepSeck-RI-De RI-DDstill- Qwen-7B
Qwen2.5-7B- Instruct
Llama-3, Instruct
Llama-8B
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
BF16
32.91
33,90
31.23
25.86
30.64
41.65
53.72
35.56
AIME'24
FP16
7.23
5.79
4.57
3.85
4.23
3.75
5.66
5.02
BF16
34.59
30.88
31.69
25.63
26.59
21.88
43.03
44.33
MATH500
FP16
5.29
4.45
4.40
3.46
4.17
3.08
7.43
6.58
BF16
48.20
43.05
36.76
32.01
28.82
34.64
36.01
35.56
LCB-Easy
FP16
7.55
6.38
6.14
4.52
4.75
4.70
5.36
4.26
BF16
44.86
44.26
42.58
37.60
31.02
35.01
39.96
33,40
LCB-Medium
FP16
8.40
7.37
6.53
5.03
5.12
4.44
4.93
4.07
BF16
46.25
47.95
41.58
37.75
36.88
39.98
41.82
33.43
LCB-Hard
FP16
7.95
6.79
6.63
5.41
5.11
4.76
5.29
5.88
corresponding results using A100 GPUs. As shown in Table 13 increasing the number of GPUs from 2 to 4 generally leads higher wwg__St@@topp_peee This observation suggests that increasing the number of GPUs may introduce greater inference nondeterminism. However, this trend becomes less consistent in the A100 setting as shown in Table 14 where in many cases the 2 GPU results are even slightly higher than those of 4 GPUs. One reason behind it is that A100s do have higher instability our experiments, which may have more influence beyond the number of GPUs (see Figure O (c)).
Table 15 examines the effect of different batch sizes. We report the Avg Std Std@topl_ of LLM inference responses for identical questions under varying GPU counts and GPU types. The results show that larger batch sizes consistently lead to lower svg_Std poo@oopp_prob,,, indicating better reproducibility model outputs. This finding aligns with our hypothesis that higher parallelism reduces error accumulation.
Finally, we compare the effect of the two GPU versions we used (L40S and A100). Table 16 reveals that under the same experimental settings, the Avg set@topl_pprot the A100 GPU is consistently slightly higher than that on the L40S GPU. This conforms to the previous findings that, in our experiments, A100 exhibits slightly higher hardware-induced variability, which may contribute to less stable top-1 token predictions across different runtime configurations.
Tables 17
18 19 and 20 present the Std@ Ace for each ablation study. Unlike Avg_ Avg_Std@top Sr_Sttttoop_pree it's hard observe clear trends or patterns from Std@ Ace. We argue that Lrg_Stt@topl_pn serves more informative metric for ablation studies, as it directly reflects the numerical instability
27

---

Table 15: Instability Awv__Std@toll_proh (x xx= under different batch sizes
DeepSeck-Rl--iiaie Qwen-7B
DerpSeck-RI1 Ri-ieci-------iill---- DiGil--LlaaaaSSS
Qvren2557BBBl
Llama-3.1 3-1--------- fretrrct
BS:8
BS-16
BS-32
BS:8
BS-16
BS-32
BS-5
BS=16
BS=32
BS-6
BS=16
BS=32
AIME'24
BF16
42:35
$7.96
39.21
30.51
36.82
$4.41
35.58
32.00
57.75
$3.85
58.55
47.23
6.66
BB16
37.61
36.91
34.41
32.34
26.93
54.17
50.32
1P16
5.26
5:25
4.79
4.28
4.19
3.08
3.87
3.96
357
6.69
5.59
6.14
LCB-Easy
BF16
56.00
$8.99
$1.39
41.93
41.36
36,80
39.14
38.27
36.21
44.63
45 83
39,00
4.68
L.R.Mediin
BF16
53.20
51.56
48.35
45.21
46.54
41.43
41.11
99.74
39.48
47.32
46.33
43.92
1P16
8.55
8.36
8.11
6.21
6.25
5.46
3.98
5.94
5995
4.92
4.68
4.52
59.04
$4,00
47.51
41.43
FP16
Table 16: Instability Biwwg_Std@oop...c prob (x (x10-4) under different GPU versions
DeepSeck- R1-ncisill-
DeepSeek BorpSeek-R1--ane R1-Distill-
Qwen2.5-7B- Instruct
Llama-3.1-81 Instruct
Qwen-7B
Llama-8B
A A100
L.40S
A100
L40S
A100
1.40S
A100
L400S
BF16
42.43
37.74
33.89
31.48
39.08
34.86
55.99
42.36
AIME'24
FP16
7.22
6,34
4.72
4.23
4.69
4.34
6.86
6.01
BF16
39.12
36.11
33.94
29.32
30.96
30.19
53.43
47.06
MATH500
FP16
5,50
5.09
4.51
3.87
4.22
4,00
6.86
5.58
BF16
58.36
54.88
43.33
36.87
37.45
35.12
44.84
41.29
LCB-Easy
FP16
7.96
7.15
6.28
5.32
5.23
4.82
5.45
4.49
BF16
54.76
48.67
47.79
43.59
38.53
37,90
46. 17
42.22
LCB-Medium
FP16
8.81
8.26
6.67
5.65
5.54
5.59
5.08
4.62
BF16
60.11
54.66
49.39
42.40
47.11
44.72
45.59
40.64
LCB-Hard
FP16
8.38
8.10
7.04
6.18
5.91
5.91
5.59
5.13
introduced by rounding error-i.e., fluctuations in the predicted probabilities of the same token. However, such fluctuations do not always lead token flip. since flip only occurs when the variation at a specific position exceeds the original top-1 and top-2 probability gap, which is inherently stochastic event. Moreover, even when token flip happens for example, replacing "the" with "that" it may not necessarily cause the evaluation outcome to change from correct to incorrect or vice versa, and thus may not affect the final accuracy.
G Supplementary Results on Random Sampling
the random sampling setting, report the complete set of Pass@1 I results in AIME'24 and MATH500 to further evaluate reproducibility under different precision settings. These experiments are conducted across different batch sizes (8, 16, 32), and GPU counts (2 and 4). The standard deviations are calculated along numeric precision types (BF16, FP16, FP32).
In this section, we show more results related the discussion in Section 3.3
Tables 21 22 2.3 and 24 summarize the Pass@1 accuracies and their standard deviation on DeepSeek- Llama-3.1 Insttuuuutt
RI-Distill Qwen- 7B, DeepSeek-RD - Ri-DDisiillLLlmm--8 Qwen2.5-7B-Insteee respectively. In most cases, FP32 yields the lowest standard deviations, reflecting greater stability across configurations. However,
worth noticing that there are (4 out of 12) cases where FP16 results are more stable. This suggests that during random sampling, the two sources of randomness can interleave. We still urge researchers to sample more extensively to obtain more reproducible results.
28

---

Table 17: Instability (Std@ Ace) under different numbers of L40S GPUs
DeepSeek-RLe SrpSSeek-I--istilll
SeepSeek-RI-Distiiee
Qwen2.5-7B- Instruct
Llama-3.1-8B-
Qwen-7B
Llama-8B
Instruct
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
BF16
3.33%
13.47%
5.10%
1.92%
1.7e-17
1.92%
0
3.33%
A4ME'24
FP16
10.72%
5.10%
8.39%
1.92%
1.7e-17
1.7c-17
1.92%
1.92%
BF16
1.10%
1.11%
2,53%
2.16%
0.90%
0.72%
1.70%
0.23%
MATH500
FP16
1.79%
0.64%
0.90%
0.46%
0.40%
0.46%
0.46%
0.23%
BF16
2.54%
1.14%
1.93%
0,64%
0.64%
0.95%
0.83%
1.27%
LCB-Easy
FP16
1.14%
0.32%
1.10%
1.45%
0.32%
0.46%
0.23%
BF16
2.92%
3.03%
1.29%
3.12%
0.74%
1.40%
0.57%
0.84%
LCB-Medium
FP16
3.45%
1.56%
1.71%
1.84%
0.28%
0.28%
0.84%
0.28%
BF16
0.94%
1.69%
2,40%
0.47%
0
0.47%
0.94%
0.47%
LCB-Hard
FP16
2.81%
0.94%
2.15%
3.08%
Table 18: Instability (Std@ Acc) under different numbers of A100 GPUs
PrepSeek-R1-Ditiiee Qwen-7B
DeepSeek-RII RI-Disllll- Llama-8B
Qwen2.5-7B. Instruct
Llama-3.1-8B- Instruct
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
2GPU
4GPU
BF16
5.78%
5.09%
3.85%
8.39%
1.92%
1.92%
1.92%
1.92%
AIME'24
FP16
5.09%
1.92%
5.09%
6.94%
1.7e-17
1.7e-17
0
BF16
0.61%
1.60%
0.70%
1.41%
1.29%
0.20%
0.46%
1.00%
MATH500
FP16
1.44%
0.61%
0.64%
1.11%
0.35%
0.31%
6.8e-17
0.50%
BF16
1.14%
2.40%
2.08%
3.22%
0.84%
0.84%
0.32%
1.14%
LCB-Fasy
FP16
2.22%
1.14%
2.48%
1.26%
0.55%
0.64%
0.64%
0.84%
BF16
2.39%
1.22%
1.84%
1.75%
2.12%
1.22%
0.84%
0.49%
LCB-Medium
FP16
2.57%
2.24%
2.95%
1.28%
0
0.28%
0.49%
0.28%
BF16
0.94%
0.47%
1.63%
0.47%
0.47%
0.47%
0
0.47%
LCB-Hard
FP16
1.88%
0.47%
2.05%
0.82%
0
Table 19: Instability (Std@ Ace) under different batch sizes
DeepSeek RI----R---Diittt Ri-tDiilll--wer 7D
DeepSeeks RI Disill Llama 8D
Qwen2,5-7B Instract
Llamo 3.1 4B--Instrcc
B5-8
BS-8
BS-16
BS-32
BS-32
BS-16
AIME'24
B6
7,20%
8.61%
12.58%
4.19%
5.00%
4,191
1.92%
1,87%
2.72%
1.92%
FP16
7.94%
5.005
4.19%
7,30%
6.38%
3.10%
1.67%
1,67%
BF16
0.76%
1.12%
0.50%
2.07%
1.91%
1.00%
0.60%
0.5%
1.44%
0.44%
FP16
0.77%
1.30%
0.97%
0.43%
0.33%
0.35%
0.35%
LCB--aay
BF16
1.90%
2.12%
0.789
1.80%
1.48%
3.11%
0.9%
1,30%
0.32%
1,224
0.525
1.29%
FP16
1.38%
1.0%%
1.45%
2.00%
0.95%
0.69%
0.5%
0.94%
0.64%
LCE-Mediiue
B116
1.95%
2.79%
2.38%
0.39%
1.65%
0.89%
0.73%
FP16
2,2216
3.23%
0.25%
2.64%
0.79%
0.89%
0.28%
0.24%
0.24%
0.60%
0.40%
0.66%
LCB-Hard
16
0.675%
188
1.00%
2.05%
1.68%
3.15%
0.47%
0.41%
0.41%
0.78%
2.24%
H Supplementary Results on LayerCast
We evaluate the LayerCast method proposed in Section 4 across multiple models and benchmarks. These evaluations span different batch sizes (8, 16. 32), GPU counts (2 and 4), GPU types (A 100 and L40S), and numerical precision settings (BF16, FP32, and LayerCast). All LayerCast expeerraseexperimment follow the same experimental configuration our main studies.
29

---

Table 20: Instability (Std@ Acc) under different GPU versions
DeepSeek-R. BAak--RR-Diiiiii RI-Distill-
DeepSeek-RR RI-Distill
Qwen2.5-7B. Instruct
Llama-3.1-82
Qwen-7B
Llama-8B
Instruct
A100
L40S
A100
L40S
A100
L40S
A100
L40S
BF16
5.44%
11.81%
5.87%
3.44%
1.82%
1.72%
1.72%
2.11%
AIME'24
FP16
3.50%
7.72%
5.96%
6.55%
1.52e-17
1.52e-17
1.72%
BF16
1.14%
1.01%
1.03%
2.11%
0.84%
0.83%
0.72%
1.17%
MATH500
FP16
0.99%
1.21%
0.83%
0.64%
0.33%
0.39%
0.35%
0.34%
BF16
1.69%
1.79%
2.66%
1.42%
0.90%
0.75%
9.59%
1.08%
LCB-Easy
FP16
1.58%
0.96%
1.76%
2.14%
0.67%
0.22%
0.73%
0.45%
BF16
1.76%
2.70%
2.20%
2.16%
1.83%
1.04%
0.62%
0.67%
LCB-Medium
FP16
2.16%
2.41%
2.19%
1.62%
0.35%
0.25%
0.40%
0.57%
BF16
6.65%
1.51%
2.61%
1.58%
0.45%
0.33%
0.33%
0.80%
LCB-Hard
FP16
1.82%
2.14%
1.52%
2.38%
Table 21: Pass I accuracies (%) under 6 system configurations and the standard deviation across
them for PeepSeel-RI-Distil--wweese
A100
4x A100
Stdev
BS-8
BS=16
BS=32
BS=8
BS=16
BS=32
AIME'24, n=16
FP32
53.33
56.25
55.21
53.13
54.17
54.17
1.1784
FPI6
52.71
55,00
52.92
53.33
53.13
53.13
0.8273
BF16
53.33
56.88
55.42
56.67
53.54
53.13
1.7151
AIME'24, 24, n=64
FP32
54.38
54.12
53.39
54.74
$4.38
55.63
0.7377
FP16
54.27
54.06
55.52
54.38
54.11
54.38
0.5391
BF16
54.74
54.17
55.10
54.90
55.16
55.10
0.3749
MATH500, n=4
FP32
90.60
90.80
90.80
90.75
90.90
90.70
0.1021
FP16
90.25
90,20
90.20
90.45
90,50
90.15
0.1463
BF16
90.65
91.15
90.40
90.95
90.85
90.35
0.3158
is important to note that LayerCast's effectiveness is not merely empirical but has strong theoretical foundations. As explain in Section 4 LayerCast performs all computations in FP32 precision, which fundamentally addresses the root cause of non-determinism we identified the accumulation of rounding errors from lower-prees lower-precision arithmetic. The only difference from pure FP32 inference is the just-in-time casting from BF16 storage to FP32 for computation, which a deterministic operation that introduces additional variance. The experiments practical applicability of this approach across different model architectures.
present here serve demonstrate the
Table 25 reports the accuracy and standard deviation for DeepSeek-RR BeanSSeek-1--a RI-Distill-Qwent across five benchmarks. To facilitate comparison, results under FP32 and BF16 are also included. As expected, BF16 exhibits the most instability, with significantly higher standard deviations. LayerCast matches FP32 in terms of stability (often with zero or near-zero std) while preserving memory efficiency.
To further demonstrate LayerCast's general effectiveness across different model architectures, Ta- ble presents results for DeepSeek-Ri- --it--Ri-iiiil-ammmaaa
on the AIME' 24 benchmark across all 12 runtime configurations. These results confirm that LayerCast consistently achieves perfect re- producibility with 0% Std@ Acc (matching FP32) while maintaining memory efficiency benefits, following the same pattern observed with DaaSSee-R---isiil----- significant variance with Std@ Acc 5.87% on 100 configurations and 3. 3.44% L40S configura- tions.
In contrast, BF16 shows
30

---

Table 22: Pass accuracies (%) under 6 system configurations and the standard deviation across them for Deepseek-RI--istill--n 8B.
2x A100
4x A100
Stdev
BS-8
BS=16
BS=32
BS=8
BS=16
BS=32
AIME'24, n=16
FP32
43.33
43.75
41.88
42.92
43.96
42.08
0.8606
FPI6
43.96
45.63
43.75
41.04
40.83
42.08
1.8792
BF16
46.46
45.42
42.71
43.13
43.54
43.13
1.5124
AIME'24, n=64
FP32
43.65
43.39
44.22
44,48
43.33
44.32
0.5034
FP16
43.49
44.22
44.17
42.87
42.60
42.14
0.8539
BF16
43.13
42.76
43.07
41.41
41.98
43.85
0.8774
MATH500, n=4
FP32
83.85
83.95
83.75
83.65
83.95
83.75
0.1211
FP16
84.55
83.95
83.95
84.15
84.25
83.55
0.3371
BF16
84.50
84.80
84.35
84.60
85.20
85.20
0.3602
Table 23: Pass @ accuracies (%) under 6 system configurations and the standard deviation across them for Qwen2.5-7B-Instruut.
A100
4x A100
Stdev
BS-8
BS=16
BS=32
BS=8
BS=16
BS=32
AIME'24, n=16
FP32
11.46
11.46
11.46
11.46
11.46
11.46
0 0.2523
FP16
11.04
11.67
11.46
11.25
11.67
11.25
BF16
11.25
10.21
11.04
9.38
11.04
10.42
0.7056
AIME'24, n=64
FP32
11.25
11.25
11.25
11.25
11.25
11.25
0 0.1382
FP16
10.99
11.25
11.25
11.20
11.41
11.30
BF16
11.20
11.04
11.41
11.04
11.30
10.94
0.1784
MATH500,, n=4
FP32
74.85
74.85
74.85
74.80
74.80
74.80
0.0274
FP16
74.50
74.90
74.80
75.00
74.80
74.75
0.1686
BF16
90.65
91.15
90.40
90.95
90.85
90.35
0.3158
These results corroborate main findings in Section 4 reproducibility on par with FP32 across different model architectures while operating within a lower memory footprint.
demonstrating that LayerCast delivers
31

---

Table 24: Pass accuracies (%) under 6 system configurations and the standard deviation across them for Llama-3.1-80 Instrrutt
2x A100
4x A100
Stdev
BS=8
BS=16
BS=32
BS=8
BS=16
BS=32
AIME'24, n=16
FP32
5.00
5.42
3.75
5,00
5.42
3.75
0.7759
FPI6
5,00
5,00
4.58
5.21
5.00
5.21
0.2282
BF16
3,96
4.38
4.79
5,42
5.21
5,42
0.5992
AIME'24, n=64
FP32
4.01
4.01
4.27
4.01
4.06
4.27
0.1296
FP16
3.91
4.06
4.17
4.22
4.58
4.64
0.2898
BF16
4.01
3.70
4.32
4.01
4.90
3.91
0.4216
MATH500, n=4
FP32
37.00
36.50
36.30
36.80
36.30
36.15
0.3293
FP16
37.10
36.60
37.00
36,90
36.95
37.00
0.1725
BF16
36.25
36.95
36.65
35.35
35.90
36.75
0.6020
Table 25: Accuracy and standard deviation of DeepSeck-RI-Disiill-OUe under different GPU counts, precisions (including LayerCast), and batch sizes across 5 benchmarks.
2x A100
4x A100
Benchmark
Std @ Ace
BS=8
BS=16
BS=32
BS=8
BS=16
BS=32
AIME'24
LayerCast
0.4333
0.4333
0.4333
0.4333
0.4333
0,4333
0
FP32
0.4333
0.4333
0,4333
0,4333
0,4333
0,4333
0
BF16
0.4667
0.4667
0.3667
0.4333
0.5333
0.4667
0.0544
MATH500
LayerCast
0,8680
0.8680
0,8680
0,8660
0.8660
0,8660
0.0011
FP32
0.8680
0.8680
0.8680
0.8680
0.8660
0.8700
0.0013
BF16
0.8700
0.8620
0.8580
0.8540
0.8860
0.8700
0.0114
LCB-Easy
LayerCast
0.7308
0.7308
0.7363
0.7363
0.7418
0.7418
0.0049
FP32
0.7363
0.7363
0.7363
0.7308
0.7363
0.7363
0.0022
BF16
0.7198
0.7418
0.7253
0.7418
0.6978
0.7363
0.0169
LCB-Medium
LayerCast
0.3058
0,3010
0.2961
0,2961
0.3010
0.3058
0.0043
FP32
0.3058
0.3010
0.3058
0.3058
0.3058
0.3010
0.0025
BF16
0.2816
0.2767
0.3204
0.2961
0.2864
0,2718
0.0176
LCB-Hard
LayerCast
0.0488
0.0488
0.0569
0.0569
0.0488
0.0569
0.0044
FP32
0.0488
0.0488
0.0488
0.0569
0.0569
0.0569
0.0044
BF16
0.0569
0.0732
0.0569
0.0650
0.0569
0.0650
0.0066
32

---

Table 26: Accuracy of DeepSeck-RI-Distil-Llammn on AIME'24 under different GPU counts,
GPU types, precisions (including LayerCast), and batch sizes.
2x A100
4x A100
Precision
BS=8
BS=16
BS=32
BS=8
BS=16
BS=32
LayerCast
36.67%
36.67%
36.67%
36.67%
36.67%
36.67%
FP32
36.67%
36.67%
36.67%
36.67%
36.67%
36.67%
BF16
30.00%
30.00%
36.67%
23.33%
40.00%
30.00%
2x L40S
4x L40S
BS=8
BS=16
BS=32
BS=8
BS=16
BS=32
LayerCast
36.67%
36.67%
36.67%
36.67%
36.67%
36.67%
FP32
36.67%
36.67%
36.67%
36.67%
36.67%
36.67%
BF16
30.00%
36.67%
26.67%
33.33%
30.00%
30.00%
33


Reviews:
[{"review_id": "bBiiW1PxIL", "reviewer": "Reviewer_VHiC", "summary": "The paper investigates the importance of numerical precision on the reproducibility of LLM evaluation results. Specifically, it found that inference using the commonly adopted BF16 precision is sensitive to variations in hardware and system configurations, and increasing the number of mantissa bits in the numerical format can mitigate this issue. Hence they proposed an inference pipeline that performances all computation in FP32 while having model weights remain in BF16.", "questions": "Please see weaknesses for the questions. Having greater coverage in the experiments as asked would significantly increase the robustness of this study and better support the claims.", "limitations": "Please see weaknesses for limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The paper addresses an important problem of the reliability and reproducibility of LLM benchmarking, especially for reasoning models. \n- Specifically, it highlights an often overlooked issue of numerical precision in experimental set-up and how it affects reproducibility.\n- The metrics used and analysis in the experiments are reasonable and detailed.\n- The proposed solution is simple and easy to implement.\n\n**Weaknesses**\n- The main weakness of the paper seems to be in its coverage of experimental settings to validate the claims. The experiments have all been run on fairly similar model sizes (7-8B) and 3 model families, on a fairly limited set of math/coding reasoning tasks. Given the claims of the paper that implies the prevalence of this issue, it is important that the paper has more coverage in its experiments to better support its claims. For example, testing on larger models with higher accuracy and stability in responses may result in much smaller variations due to numerical precision. Similarly for other types of non-math reasoning or long generation tasks.\n- It is likely important that the paper also study the impact of the choice of inference backend, i.e., whether the use of vLLM had an impact on the phenomenon that has been observed.", "was_revised": true, "final_justification": "The authors have provided additional experimental results that have addressed my concerns regarding the robustness of the results.", "rebuttal": "### [W1] `Limited coverage of models and tasks` — We are providing new experiments on larger models and diverse tasks.\n\nThank you for the suggestion on greater coverage in experiments. Here we provide additional results on a larger model and a new reasoning dataset to strengthen our claims about the prevalence of reproducibility issues.\n\n**Larger Model Evaluation (Qwen3-32B):**\nTo demonstrate the impact of numerical precision on reproducibility on larger models, we benchmarked Qwen3-32B on AIME24 dataset under 12 different settings as reported in our paper, which cover 2 GPU counts (2 and 4), 2 GPU version (A100 and L40S), and 3 batch sizes (8, 16, 32). Since Qwen3-32B model cannot be loaded into 2 L40S GPUs with FP32, we have 9 feasible configurations for FP32. The benchmark results are shown below:\n\n|  | 4A100-BS8 | 4A100-BS16 | 4A100-BS32 | 4L40S-BS8 | 4L40S-BS16 | 4L40S-BS32 |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| BF16     | 0.8     | 0.8333     | 0.8333     | 0.7333     | 0.8333     | 0.7     |\n| FP32     | 0.7667     | 0.7667     | 0.7667     | 0.7667     | 0.7667     | 0.7667     |\n\n|  | 2A100-BS8 | 2A100-BS16 | 2A100-BS32 | 2L40S-BS8 | 2L40S-BS16 | 2L40S-BS32 |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| BF16     | 0.7667     | 0.8333     | 0.7333     | 0.8333     | 0.7667     | 0.7333     |\n| FP32     | 0.7667     | 0.8     | 0.7667     | OOM     | OOM     | OOM     |\n\n|Std@Acc / AIME24|BF16|FP32|\n|----|----|----|\n|Qwen3-32B|5.02%|1.11%|\n\nThese results confirm that larger models exhibit the same reproducibility challenges we identified in 7-8B models, with BF16 showing significant variance while FP32 maintains consistency.\n\n**Diverse Task Evaluation (GPQA Diamond):**\nFor new tasks other than math/coding, we ran GPQA Diamond benchmark on DeepSeek-R1-Distill-Llama-8B, and the results are shown below. \n\n| Precision | 4A100-BS8 | 4A100-BS16 | 4A100-BS32 | 4L40S-BS8 | 4L40S-BS16 | 4L40S-BS32 |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| BF16     | 0.3384     | 0.3838     | 0.4141     | 0.4343     | 0.3838     | 0.4495     |\n| FP32     | 0.4091     | 0.4545     | 0.4343     | 0.3889     | 0.4242     | 0.4495     |\n\n| Precision | 2A100-BS8 | 2A100-BS16 | 2A100-BS32 | 2L40S-BS8 | 2L40S-BS16 | 2L40S-BS32 |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| BF16     | 0.4091     | 0.4192     | 0.4141     | 0.3889     | 0.4394     | 0.4192     |\n| FP32     | 0.4091     | 0.4343     | 0.4091     | 0.4293     | 0.4141     | 0.4444     |\n\n|Std@Acc / GPQA-Diamond|BF16|FP32|\n|----|----|----|\n|DeepSeek-R1-Distill-Llama-8B|3.03%|1.96%|\n\nThe above results indicate that the reproducibility issue also exists on GPQA Diamond benchmark, supporting the prevalence of this issue. These additional results strengthen our argument that numerical precision is a critical but overlooked factor in LLM evaluation reproducibility across model scales and task types.\n\n\n### [W2] `Is this a backend-specific issue?` — The issue is fundamental to GPU hardware and observable across backends.\n\nWe thank the reviewer for this insightful suggestion. We agree that it is important to ensure our findings are not specific to a single inference backend.\n\nOur paper's central thesis is that the observed non-determinism originates from a fundamental property of modern computing hardware: the non-associative nature of floating-point arithmetic (as detailed in Section 2.2). As we detail in our paper, operations like `(a+b)+c` are not guaranteed to equal `a+(b+c)` due to rounding errors in finite-precision formats like BF16. This issue is not specific to vLLM but is inherent to how parallel computations are performed on GPUs across all major deep learning frameworks.\n\nIndeed, **this phenomenon is observable across different inference backends**. For instance, a similar issue of non-reproducible outputs with greedy decoding has been reported by users of SGLang [1]. Furthermore, the official PyTorch documentation [2] explicitly discusses this source of non-determinism, attributing it to the low-level CUDA kernels.\n\nIn conclusion, while our experiments are implemented using vLLM, the root cause we identify and analyze is at a much lower level of the computing stack, related to hardware-level floating-point computation. Our findings are therefore general and applicable to other inference systems that rely on parallel GPU execution.\n\n[1] SGLang Issue 7916\n\n[2] PyTorch Docs: Randomness"}, {"review_id": "LYvqTfzULB", "reviewer": "Reviewer_EEhu", "summary": "The paper analyzes the effect of the known precision problem for floating point numbers as used in numerical computation and the introduced randomness via non-associativity of operations on LLMs and reasoning models' performance metrics. Different inference settings with different GPUs, GPU count, and batch sizes reveal that the common BF16 data type is prone to large deviations in exhibited performance. The authors propose the use of their LayerCast wrapper that keeps memory footprint low by using BF16 weights (for inference), while using the more stable FP32 for computation.", "questions": "- Isn't FP32 up-conversion during compute already performed for matrix multiplications on certain hardware (BF16 tensor core acceleration)? \n\n- Does your analysis imply that also during training FP32 should be superior? \n\n- How is precision typically handled for the three common aggregation operations in Transformers: Matrix Multiplication, Layer/RMS Normalization and Self Attention (FlashAttention)? \n\n- Could you provide an inverse benchmark setting, where models are exactly the same and should run reproducibly, but people can report results using the local hardware / library setup - revealing differences? This would necessitate adding some code along with the publication.", "limitations": "The limitations of the work are clearly stated.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- provides an empirical study of the actual effects of precision problems, and their non-negligible strength compared to differences between model performances \n- LayerCast mitigates the precision problems without additional memory overhead\n\nWeaknesses:\n- FP32 inference can be far slower on modern hardware\n- BF16 tensor core ops actually use higher precision internally and store results in FP32, so this should partly be mitigated already?\n- There is no clear analysis how this actually does not badly affect training (which happens typically the other way round: FP32 weights, BF16 computation)\n- there is no discussion of actual hardware implementations / algorithms (FlashAttention) wrt the choice of precisions, despite the focus on Attention-based models", "was_revised": true, "final_justification": "While this analysis does not provide fundamentally new insights, it solidly addresses one main concern of scientific work: reproducibility. And it actually shows that for different hardware setups and precision settings, outputs vary by significant margin. While there is still some room for improvement regarding the exact causes of the problem, in principle I rather recommend this work for acceptance.", "rebuttal": "### [W1] `FP32 is too slow` — We agree that this is a trade-off, and we proposed LayerCast to help.\n\nWe agree with the reviewer that FP32 inference carries a higher computational cost, and this is a **trade-off between speed and precision**. Our goal is not to advocate for abandoning 16-bit formats entirely, but to demonstrate that for applications where reproducibility is critical, the cost of numerical non-determinism in BF16 can be unacceptably high, leading to misleading or irreproducible results (Tables 1 & 2).\n\nThis is precisely the motivation behind our proposed solution, LayerCast (Section 4). LayerCast is designed to provide the near-perfect reproducibility of FP32 while mitigating the performance and memory overhead. It achieves this by storing weights in BF16 but performing all computations in FP32, offering a practical compromise for achieving reproducible reasoning without the full cost of FP32 inference.\n\n### [W2 & Q1] `Isn't BF16 compute already mitigated by internal FP32 accumulation?` — This helps but is insufficient for end-to-end reproducibility due to intermediate casting.\n\nThe reviewer rightly notes that modern GPU tensor cores use FP32 for intermediate accumulation during BF16 matrix multiplications, then downcast the results to FP16/BF16 to limit error accumulation. This internal up-conversion does help reduce rounding errors within a single matrix multiplication.\n\nHowever, as our paper demonstrates, **this is not sufficient to ensure end-to-end reproducibility**. A Transformer forward pass consists of many sequential operations. While a single operation (such as GEMM) may use a higher-precision accumulator, its inputs are the outputs of previous operations, and its results are typically cast back to BF16 before being passed to the next layer (for example, when storing in the KV cache or as input to an activation function). As a result, our experimental results show that significant non-determinism remains on BF16 models.\n\n### [W3 & Q2] `What are the implications for training?` — Our findings are specific to inference; training is a fundamentally different scenario that actually benefits from BF16's characteristics.\n\nThank you for raising this important distinction between training and inference. Our analysis focuses specifically on inference reproducibility, and we do not claim that FP32 is superior for training. In fact, there are several key reasons why training is fundamentally different from inference in terms of numerical precision requirements:\n\n1. **Different Objectives:** During inference, deterministic and reproducible outputs are needed for reliable evaluation and deployment; while training benefits from stochasticity as a form of implicit regularization that helps escape local minima and improve generalization\n2. **Architectural Differences:** Training only do full prefill computation—the model processes complete sequences without generating new tokens or using KV cache; while during inference, the model generates tokens autoregressively, where each new token depends on all previous tokens, allowing errors to accumulate through the KV cache. Therefore, training do not have the *divergence* problem.\n3. **Established Best Practices:** Mixed-precision training (FP32 weights, BF16 computation) is the industry standard precisely because it balances: computational efficiency (BF16 operations are faster) and training stability (FP32 master weights).\n\nIn summary, our findings about inference reproducibility do not contradict the well-established benefits of mixed-precision training. The core issue we identify—cascading errors through autoregressive generation—is specific to inference and does not manifest in the teacher-forced training paradigm.\n\n\n### [W4 & Q3] `How is precision handled in key operations like FlashAttention?` — We provide a detailed breakdown and explain why variance accumulates.\n\nThank you for this suggestion. Here we provide a more detailed discussion on how precision is handled in key Transformer operations and why variance still arises. Let's use a typical BF16 inference setting as an example:\n\n- **Matrix Multiplication**: As discussed in [W2], while the inputs and outputs are typically in BF16, modern Tensor Cores perform the actual accumulation in FP32. However, after the computation, the results are usually cast back to BF16 before being passed to subsequent layers, which reintroduces rounding errors and potential loss of precision.\n- **Layer/RMS Normalization**: These normalization operations require the computation of sums and variances, which are sensitive to numerical errors. For this reason, they are often implemented in FP32 for improved numerical stability, even when the rest of the model uses BF16.\n- **Self-Attention (e.g., FlashAttention)**: Optimized attention kernels such as FlashAttention support both BF16 and FP16. For FP32 models, implementations like xFormers may be used. Regardless of the implementation, self-attention fundamentally relies on floating-point matrix multiplications and reductions to compute attention scores. The order and grouping of these operations can vary depending on GPU architecture, parallelization strategy (e.g., number of heads, sequence length), and specific kernel implementation, making them susceptible to the same non-associativity issues described in Section 2.2. Additionally, intermediate attention scores are often stored in lower precision formats before the final softmax and value-weighting steps, introducing further sources of numerical variation.\n\nIn summary, while some components may use higher internal precision, the end-to-end computation involves multiple stages of calculation and data movement where precision can be lost. Our results show that the combination of these small, accumulating errors across the entire model leads to the significant output divergence we observe.\n\n\n### [Q4] `Could you provide an inverse benchmark to reveal hardware differences?` — A good idea! Here we provide some discussions\n\nThank you for this excellent suggestion. We believe the reviewer is proposing an \"inverse benchmark\" where researchers worldwide would run identical model and inference code on their local setups to reveal hardware-induced differences. This is indeed a compelling idea that aligns perfectly with our paper's findings.\n\nIn fact, our experimental framework already serves this purpose to some extent. The code we provide (linked in the abstract) allows researchers to reproduce our experiments on their own hardware configurations. Our results in Table 3, showing significant variance across 12 different hardware/system configurations, essentially demonstrate what such an inverse benchmark would reveal.\n\nHowever, creating a more formalized \"inverse benchmark\" as the reviewer suggests presents several challenges:\n\n1. **Attribution Complexity**: As our ablation studies (Section 3.4) show, the variance arises from complex interactions between multiple factors—GPU type, GPU count, batch size, driver versions, and CUDA kernel implementations. Isolating which specific factor causes a particular difference would require extensive controlled experiments.\n2. **Stochastic Nature**: The hardware-induced differences manifest probabilistically. As shown in Figure 5, not all examples diverge even under BF16, and when they do, the divergence point varies. This makes it difficult to create a simple deterministic test that reliably exposes hardware differences.\n3. **Standardization Challenges**: The combinatorial explosion of hardware configurations (GPU models × driver versions × CUDA versions × framework versions) makes it impractical to maintain a comprehensive mapping of expected outputs for each configuration.\n\nDespite these challenges, we encourage researchers to use our code to test on their own hardware and contribute their findings, which could collectively build the kind of cross-platform reproducibility map the reviewer envisions. This crowdsourced approach may be more practical than a centralized benchmark given the vast diversity of hardware configurations in use today.\n\n\n### Regarding Originality\n\nWe appreciate the reviewer's feedback on originality and would like to clarify our contributions. While the concepts of floating-point non-associativity and non-determinism in deep learning are known, our work provides the **first systematic and quantitative investigation into how the interplay of hardware configurations (GPU type, tensor parallel size) and system settings (batch size) specifically impacts the reproducibility of modern LLM reasoning.**\n\nPrior to our work, performance discrepancies were often anecdotally observed but not rigorously analyzed. Our contributions are:\n1.  We move beyond anecdotal evidence to empirically demonstrate and quantify the problem across multiple models and benchmarks, showing that BF16 can lead to accuracy variations of up to 9% (Table 3).\n2.  We isolate the root cause in the context of LLMs: the interaction between the non-associativity of floating-point math and the minimal probability gaps between competing tokens in reasoning tasks (Figure 3 and 4).\n3.  We propose LayerCast, a practical and lightweight solution that addresses this problem by achieving FP32-level determinism with a memory footprint much closer to 16-bit models.\n\nWe believe this comprehensive analysis, which connects low-level hardware characteristics to high-level model behavior and provides a practical solution, represents a significant and original contribution to making LLM evaluation more reliable and scientific."}, {"review_id": "O8ZLAEOjZ8", "reviewer": "Reviewer_mWEn", "summary": "This paper uncovers a possible reason for pronounced challenges in reproducibility of LLM performance. The authors posit that imprecisions arising from floating point representations being neglected in evaluation benchmarks and practices is a primary cause. They verify this hypothesis through systematic controlled experiments, and identify conditions when model outputs diverge. The authors also develops an inference pipeline termed LayerCast, which is shown to balance memory efficiency with numerical stability.", "questions": "Q1: Experiments demonstrating the effectiveness of LayerCast can be more thorough and examine other models. Even in the Appendix, results are reported only for DeepSeek-R1-Distil-Qwen-7B across the five benchmarks. Could the authors comment on this?", "limitations": "Addressed in the Appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\n(+) Revealing numerical (im)precision as a cause for lack of reproducibility is deceptively simple- in hindsight it should seem obvious. A strength of this paper is the lengths to which the authors have gone to carefully and systematically demonstrate that this is indeed the case. \n\n(+) The illustrative examples presented demonstrating violation of the associative property for FP addition does an excellent job of highlighting the significance of the problem when scaled up to larger models including LLMs. \n\n(+) Experimental evaluations are presented on reasoning and non-reasoning models, across multiple benchmarks and runtime configurations, demonstrating thoroughness. \n\n(+) This work has the potential for long-lasting impact in any field in which reproducibility heavily relies on precision and representations used in compute components. \n\n(+) Beyond identifying the problem, the paper proposes a possible fix where all computations are performed in FP32, while model parameters continue to be stored in BF16. \n\nAreas for Improvement: \n\n(-) Experiments demonstrating the effectiveness of LayerCast can be more thorough and examine other models. Even in the Appendix, results are reported only for DeepSeek-R1-Distil-Qwen-7B across the five benchmarks.", "was_revised": true, "final_justification": "I believe that this is a very well-written and presented paper addressing an interesting and important problem. Examining the authors' response reaffirms my initial views on this work.", "rebuttal": "### [W1 & Q1] `LayerCast experiments need to be more thorough` — We agree and are running new experiments on more models.\n\nWe thank the reviewer for this valuable feedback. We acknowledge that in the initial submission, LayerCast results were primarily demonstrated on DeepSeek-R1-Distill-Qwen-7B across five benchmarks. We agree that evaluating LayerCast on additional models would strengthen our claims about its general effectiveness. In response to this suggestion, we are conducting additional experiments with LayerCast on DeepSeek-R1-Distill-Llama-8B across the same benchmarks. Our preliminary results on AIME24 benchmark show that **LayerCast consistently achieves near-FP32 levels of reproducibility** while maintaining the memory efficiency benefits we described. Full results are to be included in the final version.\n\n| Precision | 2A100-BS8 | 2A100-BS16 | 2A100-BS32 | 4A100-BS8 | 4A100-BS16 | 4A100-BS32 | Std@Acc |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LayerCast     | 0.3667    | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0 |\n| FP32     | 0.3667    | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0 |\n| BF16     | 0.3    | 0.3     | 0.3667     | 0.2333     | 0.4     | 0.3     | 5.87% |\n\n| Precision | 2L40S-BS8 | 2L40S-BS16 | 2L40S-BS32 | 4L40S-BS8 | 4L40S-BS16 | 4L40S-BS32 | Std@Acc |\n| ------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| LayerCast     | 0.3667    | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0 |\n| FP32     | 0.3667    | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0.3667     | 0 |\n| BF16     | 0.3    | 0.3667     | 0.2667     | 0.3333     | 0.3     | 0.3     | 3.44% |\n\nIt's important to note that LayerCast's effectiveness is not merely empirical but has strong theoretical foundations. As we explain in Section 4, LayerCast performs all computations in FP32 precision, which fundamentally addresses the root cause of non-determinism we identified—the accumulation of rounding errors from lower-precision arithmetic. The only difference from pure FP32 inference is the just-in-time casting from BF16 storage to FP32 for computation, which is a deterministic operation that introduces no additional variance. The additional experiments we're conducting serve to demonstrate the practical applicability of this approach across different model architectures."}, {"review_id": "RWFtzFgar7", "reviewer": "Reviewer_3spC", "summary": "The manuscript highlights the significance of numerical precision for precise LLM reasoning reproducibility. Hardware and system-level heterogeneity result in non-associative property floating-point arithmetic. While most of the researches overestimate this model's uncertainty, making it difficult to distinguish real improvements or random variation. BF16 precision behaves higher risk at numerical sensitivity. Authors conduct experiments on 12 different configurators to highlight that up to 9% of variance would occur even with identical experiment setups.", "questions": "1. The relevant results in appendix shows \"A100 exhibits slightly higher hardware-induced variability\". What is the biggest factor that makes two GPU types exhibit different numerical non-deterministic?\n2. The observation \"for reasoning models, the token probability differences between the top two competing tokens are often minimal.\" seems kind of in contrary to the hypothesis in abs \"This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy.\" Do you think reasoning models would suffer more from the numerical non-deterministic? Table 5 seems to have the conclusion that reasoning model has better stability under BF16.\n3. While the LayerCast approach provides a practical compromise between memory efficiency and reproducibility by performing computations in FP32 while storing weights in BF16, similar techniques have been adopted in prior systems such as NVIDIA Transformer Engine and Megatron-LM (e.g., float32_matmul). The paper could benefit from better positioning LayerCast within this line of existing work, and clarifying whether its novelty lies in system implementation, evaluation scale, or reproducibility focus.", "limitations": "Authors provide the limitation statement in the start of Appendix. But I still suggest put it on the tail of the paper to broader its visibility if it does not exceed the page limit.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n+ Extensive analysis of how numerical precision affects reproducibility.\n+ Comprehensive background&motivation illustration, and clear comparison between different numerical precision. The narrative is attractive, motivation is solid and convincing.\n\n**Weaknesses**\n- I personally feel reasonable the claim \"different hardware implementations or computation orders can lead to slightly different results\". However, please add more context to make it more convincing, e.g., more in-depth comparisons or at least references in section 2.2.\n- The experimental results in terms of reasoning model seem counterintuitive.\n- The proposed LayerCase, compute in FP32 while store in BP16, seems a common practice in LLM serving frameworks. Thereby, the design contribution of this manuscript is quite minor.", "was_revised": true, "final_justification": "Authors rebuttal have adequately resolve my concerns. I believe this paper will be helpful to the NeurIPS and MLSys communities.", "rebuttal": "### [W1] `Need more context on the root of phenomenon` — We agree and will add more details and references to strengthen this section.\n\nThank you for this suggestion. We agree that strengthening this context is important. In the current version of Section 2.2, we introduce the concept of non-associativity in floating-point arithmetic and provide a concrete numerical example in Table 2 to illustrate how summation order can change results in both FP32 and BF16.\n\nTo make this more convincing as requested, we will expand this section in the final paper. We will add a more detailed discussion on how parallel reduction algorithms on GPUs can lead to different operation orderings, and we will include references to official documentation, such as the PyTorch guide on numerical accuracy [1], which explicitly mentions: *\"bitwise identical results are not guaranteed across PyTorch releases, individual commits, or different platforms\"*. This will better ground our claims in established computer architecture and deep learning framework principles.\n\n[1] PyTorch Docs: Numerical Accuracy\n\n\n### [W2 & Q2] `Are reasoning models more or less stable?` — Reasoning models ARE more vulnerable; apparent contradictions arise from different evaluation contexts.\n\nThanks for bringing up this question. We first clarify that reasoning models are more susceptible to numerical non-determinism precisely *because* they exhibit minimal probability gaps between competing tokens (as shown in Figure 3). When top candidate tokens have nearly identical probabilities, tiny numerical fluctuations from floating-point errors can change their ranking, causing \"token flips.\" These flips, especially early in long chains of thought, cascade into completely different reasoning paths.\n\nThen, regarding the experimental results. While Table 5 in our Appendix reports larger variance of BF16 on reasoning models, do you reference Table 4 in our submission? In Table 4, Qwen2.5-7B-Instruct seems to have higher performance variance than DeepSeek-R1-Distill-Qwen-7B. Here, we provide more results. The tables below show standard deviation of Pass@1 performance (%).\n\n| MATH500 (n=4) | BF16 | FP16 | FP32 |\n| ------- | -------- | -------- | -------- |\n| DeepSeek-R1-Distill-Qwen-7B     | 0.3158    | 0.1463     | 0.1021     |\n| DeepSeek-R1-Distill-Llama-8B     | 0.3602    | 0.3371     | 0.1211     |\n| Qwen2.5-7B-Instruct     | 0.4663    | 0.1686     | 0.0274     |\n| Llama-3.1-8B-Instruct     | 0.6020    | 0.1725     | 0.3293     |\n\n| AIME24 (n=16) | BF16 | FP16 | FP32 |\n| ------- | -------- | -------- | -------- |\n| DeepSeek-R1-Distill-Qwen-7B     | 1.7151    | 0.8273     | 1.1785     |\n| DeepSeek-R1-Distill-Llama-8B     | 1.5124    | 1.8792     | 0.8606     |\n| Qwen2.5-7B-Instruct     | 0.7056    | 0.2523     | 0     |\n| Llama-3.1-8B-Instruct     | 0.5992    | 0.2282     | 0.7759     |\n\n| AIME24 (n=64) | BF16 | FP16 | FP32 |\n| ------- | -------- | -------- | -------- |\n| DeepSeek-R1-Distill-Qwen-7B     | 0.3749    | 0.5391     | 0.7377     |\n| DeepSeek-R1-Distill-Llama-8B     | 0.8774    | 0.8539     | 0.5034     |\n| Qwen2.5-7B-Instruct     | 0.1784    | 0.1382     | 0     |\n| Llama-3.1-8B-Instruct     | 0.4216    | 0.2898     | 0.1296     |\n\nBased on all results, we clarify (i) why the MATH500 slice shows smaller variability for some reasoning model configurations, and (ii) why we still think that reasoning models are more vulnerable to numeric error.\n\nFirst, the MATH500 evaluation used four trials for pass@1 performance, whereas AIME24 had 16. The relatively fewer trials makes the standard deviation more sensitive to the randomness of random sampling generation: a single flip can move the accuracy by 0.2% while these standard deviation are mostly below 0.5%. The isolated low-variance numbers for MATH500 are from the sampling randomness when there are limited trials, not from an inherent robustness.\n\nSecond, wo found evidence in follow-up experiments on AIME24 with n=64. When we increase n of Pass@1 metric, we witness a significant overall drop of the standard deviation, especially in BF16 settings. This demonstrates how repeated independent trials smooth out the intrinsic randomness during the sampling process. As shown in our results, when n=64, i.d., when more sampling randomness is skimmed and numerical error starts dominating the standard deviation, reasoning models consistently show higher variance. \n\nTo sum up, we think reasoning models display greater numerical non-determinism, and we recognize that, with limited experiments, there could be isolated results showing some discrepancy.\n\n\n### [W3 & Q3] `LayerCast is not novel` — Our novelty lies in framing reproducibility as a scientific crisis and providing an accessible, targeted solution for researchers.\n\nWe appreciate the reviewer placing our work in the context of prior systems. We agree that the concept of mixed-precision computation is not new. However, the novelty of our work and LayerCast lies in the **problem framing, evaluation focus, and practical accessibility for the research community.**\n\nOur main contribution is not the invention of mixed-precision techniques, but rather the **first systematic quantification of the reproducibility crisis in LLM reasoning evaluation** and the presentation of LayerCast as a direct, targeted solution. While frameworks like Transformer Engine use similar concepts for optimizing training/serving throughput, our work focuses on reproducibility for scientific evaluation. We frame numerical non-determinism not as a performance issue, but as a fundamental threat to reliable benchmarking.\n\nOur novelty can be summarized as:\n1.  **Reproducibility Focus:** We are the first to evaluate and propose this technique explicitly as a solution to the *reproducibility problem* in LLM evaluation, supported by extensive experiments quantifying the instability of standard methods (Tables 1, 2, 4).\n2.  **Evaluation Scale:** Our contribution lies in the comprehensive analysis that demonstrates the severity of this problem across different hardware, models, and tasks, proving why such a solution is necessary for reliable research.\n3.  **Accessibility:** We provide LayerCast as a simple, open-source patch to a widely-used inference library (vLLM). This makes it easy for any researcher to adopt and ensure their results are reproducible, democratizing a practice that was previously embedded in large, complex training frameworks.\n\nWe will revise the wording to better highlight our contributions.\n\n\n### [Q1] `What causes different variability across GPU types?` — We hypothesize it's a combination of kernel implementation, thread scheduling, and compiler optimizations.\n\nThis is a very insightful question. While a definitive answer would require proprietary details from the hardware manufacturer, we can offer a hypothesis based on known architectural principles. The difference in variability between an A100 (Ampere architecture) and an L40S (Ada Lovelace architecture) likely stems from a combination of factors at the hardware and software level that influence the order of floating-point operations.\n\nThe most probable factors are:\n1. **Kernel Implementation is different:** Modern low-level kernel libary will use different kernel implementation specialized for different hardware architecture. For example, FlashAttention-3 will use warpgroup-wide WGMMA instruction of Hopper H100 GPUs, which is not available on Ampere architecture like A100. Thus they inherently have different numerical stability [FA3].\n2.  **Thread Scheduling and Parallelism:** The two GPU architectures have different numbers of SMs, cache hierarchies, and memory bandwidths. The CUDA scheduler may therefore produce different parallel execution plans (i.e., different orderings for associative operations) to best utilize the resources of each specific architecture.\n3.  **Compiler and Driver Optimizations:** The NVIDIA driver and CUDA compiler generate architecture-specific microcode. The low-level instructions compiled for an A100 can be different from those for an L40S, even from the same source code, leading to different computation paths."}]

Author rebuttal:

Meta-review: The paper shows that choice of floating point precision can lead to significant changes in inference accuracy reported for reasoning models. 

Strengths: Extensive study showing the extent of variation, important to the interpretation of reasoning results, provide a solution to address reproducibility issues without incurring memory overhead
Weaknesses: one of those papers where in hindsight, it seems obvious that FP precision should impact results

During the rebuttal, authors strengthened their argument by experiments on models of varying sizes and a new task beyond math and code. All reviewers appreciated the insight from this paper and the extensive experiments to support it. I think it is a timely paper and worthy of broader discussion.
Decision: Accept (oral)