Title: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models (LLMs)

Reproducibility

Numerical precision

Deterministic inference
Abstract: Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. 
This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9\% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size.
We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. 
This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge.
Our analysis reveals that floating-point precision—while critical for reproducibility—is often neglected in evaluation practices.
Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript highlights the significance of numerical precision for precise LLM reasoning reproducibility. Hardware and system-level heterogeneity result in non-associative property floating-point arithmetic. While most of the researches overestimate this model's uncertainty, making it difficult to distinguish real improvements or random variation. BF16 precision behaves higher risk at numerical sensitivity. Authors conduct experiments on 12 different configurators to highlight that up to 9% of variance would occur even with identical experiment setups.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n+ Extensive analysis of how numerical precision affects reproducibility.\n+ Comprehensive background&motivation illustration, and clear comparison between different numerical precision. The narrative is attractive, motivation is solid and convincing.\n\n**Weaknesses**\n- I personally feel reasonable the claim \"different hardware implementations or computation orders can lead to slightly different results\". However, please add more context to make it more convincing, e.g., more in-depth comparisons or at least references in section 2.2.\n- The experimental results in terms of reasoning model seem counterintuitive.\n- The proposed LayerCase, compute in FP32 while store in BP16, seems a common practice in LLM serving frameworks. Thereby, the design contribution of this manuscript is quite minor.\n\nQUESTIONS:\n1. The relevant results in appendix shows \"A100 exhibits slightly higher hardware-induced variability\". What is the biggest factor that makes two GPU types exhibit different numerical non-deterministic?\n2. The observation \"for reasoning models, the token probability differences between the top two competing tokens are often minimal.\" seems kind of in contrary to the hypothesis in abs \"This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy.\" Do you think reasoning models would suffer more from the numerical non-deterministic? Table 5 seems to have the conclusion that reasoning model has better stability under BF16.\n3. While the LayerCast approach provides a practical compromise between memory efficiency and reproducibility by performing computations in FP32 while storing weights in BF16, similar techniques have been adopted in prior systems such as NVIDIA Transformer Engine and Megatron-LM (e.g., float32_matmul). The paper could benefit from better positioning LayerCast within this line of existing work, and clarifying whether its novelty lies in system implementation, evaluation scale, or reproducibility focus.\n\nLIMITATIONS:\nAuthors provide the limitation statement in the start of Appendix. But I still suggest put it on the tail of the paper to broader its visibility if it does not exceed the page limit.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uncovers a possible reason for pronounced challenges in reproducibility of LLM performance. The authors posit that imprecisions arising from floating point representations being neglected in evaluation benchmarks and practices is a primary cause. They verify this hypothesis through systematic controlled experiments, and identify conditions when model outputs diverge. The authors also develops an inference pipeline termed LayerCast, which is shown to balance memory efficiency with numerical stability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n(+) Revealing numerical (im)precision as a cause for lack of reproducibility is deceptively simple- in hindsight it should seem obvious. A strength of this paper is the lengths to which the authors have gone to carefully and systematically demonstrate that this is indeed the case. \n\n(+) The illustrative examples presented demonstrating violation of the associative property for FP addition does an excellent job of highlighting the significance of the problem when scaled up to larger models including LLMs. \n\n(+) Experimental evaluations are presented on reasoning and non-reasoning models, across multiple benchmarks and runtime configurations, demonstrating thoroughness. \n\n(+) This work has the potential for long-lasting impact in any field in which reproducibility heavily relies on precision and representations used in compute components. \n\n(+) Beyond identifying the problem, the paper proposes a possible fix where all computations are performed in FP32, while model parameters continue to be stored in BF16. \n\nAreas for Improvement: \n\n(-) Experiments demonstrating the effectiveness of LayerCast can be more thorough and examine other models. Even in the Appendix, results are reported only for DeepSeek-R1-Distil-Qwen-7B across the five benchmarks.\n\nQUESTIONS:\nQ1: Experiments demonstrating the effectiveness of LayerCast can be more thorough and examine other models. Even in the Appendix, results are reported only for DeepSeek-R1-Distil-Qwen-7B across the five benchmarks. Could the authors comment on this?\n\nLIMITATIONS:\nAddressed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the effect of the known precision problem for floating point numbers as used in numerical computation and the introduced randomness via non-associativity of operations on LLMs and reasoning models' performance metrics. Different inference settings with different GPUs, GPU count, and batch sizes reveal that the common BF16 data type is prone to large deviations in exhibited performance. The authors propose the use of their LayerCast wrapper that keeps memory footprint low by using BF16 weights (for inference), while using the more stable FP32 for computation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- provides an empirical study of the actual effects of precision problems, and their non-negligible strength compared to differences between model performances \n- LayerCast mitigates the precision problems without additional memory overhead\n\nWeaknesses:\n- FP32 inference can be far slower on modern hardware\n- BF16 tensor core ops actually use higher precision internally and store results in FP32, so this should partly be mitigated already?\n- There is no clear analysis how this actually does not badly affect training (which happens typically the other way round: FP32 weights, BF16 computation)\n- there is no discussion of actual hardware implementations / algorithms (FlashAttention) wrt the choice of precisions, despite the focus on Attention-based models\n\nQUESTIONS:\n- Isn't FP32 up-conversion during compute already performed for matrix multiplications on certain hardware (BF16 tensor core acceleration)? \n\n- Does your analysis imply that also during training FP32 should be superior? \n\n- How is precision typically handled for the three common aggregation operations in Transformers: Matrix Multiplication, Layer/RMS Normalization and Self Attention (FlashAttention)? \n\n- Could you provide an inverse benchmark setting, where models are exactly the same and should run reproducibly, but people can report results using the local hardware / library setup - revealing differences? This would necessitate adding some code along with the publication.\n\nLIMITATIONS:\nThe limitations of the work are clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the importance of numerical precision on the reproducibility of LLM evaluation results. Specifically, it found that inference using the commonly adopted BF16 precision is sensitive to variations in hardware and system configurations, and increasing the number of mantissa bits in the numerical format can mitigate this issue. Hence they proposed an inference pipeline that performances all computation in FP32 while having model weights remain in BF16.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The paper addresses an important problem of the reliability and reproducibility of LLM benchmarking, especially for reasoning models. \n- Specifically, it highlights an often overlooked issue of numerical precision in experimental set-up and how it affects reproducibility.\n- The metrics used and analysis in the experiments are reasonable and detailed.\n- The proposed solution is simple and easy to implement.\n\n**Weaknesses**\n- The main weakness of the paper seems to be in its coverage of experimental settings to validate the claims. The experiments have all been run on fairly similar model sizes (7-8B) and 3 model families, on a fairly limited set of math/coding reasoning tasks. Given the claims of the paper that implies the prevalence of this issue, it is important that the paper has more coverage in its experiments to better support its claims. For example, testing on larger models with higher accuracy and stability in responses may result in much smaller variations due to numerical precision. Similarly for other types of non-math reasoning or long generation tasks.\n- It is likely important that the paper also study the impact of the choice of inference backend, i.e., whether the use of vLLM had an impact on the phenomenon that has been observed.\n\nQUESTIONS:\nPlease see weaknesses for the questions. Having greater coverage in the experiments as asked would significantly increase the robustness of this study and better support the claims.\n\nLIMITATIONS:\nPlease see weaknesses for limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}