Title: FP4 All the Way: Fully Quantized Training of Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Quantization

Low precision

Training

FP4

Accelerations

Compression
Abstract: We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately $\sqrt{3}$ times the quantization noise, quantized training becomes less effective. Leveraging these insights, we successfully train a 7-billion-parameter model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to a standard BF16 baseline, confirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in https://github.com/Anonymous1252022/fp4-all-the-way.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this work demonstrate end-to-end 4-bit FP training at LLM scale (7B, 200B tokens), extending FQT capabilities beyond existing FP8 approaches. The authors claim their work to be the first line of works demonstrate complete FP4 end to end training. The authors evaluate key design choices (block size, scaling format, rounding modes) and identify NVFP4 (E2M1 data with shared E4M3 scale in blocks of 16) with mixed rounding (stochastic for backward/update, round-to-nearest for forward) as optimal. Lastly, they successfully train a 7B-parameter model on Intel Gaudi2 accelerators to performance comparable to BF16.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a timely work which focuses on lower precision training of LLMs. If generalizable across models, it could have moderate to high impact on an industrial scale.\n\nStrengths:\n1. This authors claim their work to be the first to enable end-to-end 4-bit FP training at LLM scale (7B, 200B tokens), extending FQT capabilities beyond existing FP8 approaches.\n2. The authors systematically explore block sizes (8 to 128), scaling formats (MXFP4 vs NVFP4), and rounding schemes, concluding NVFP4 with block size 16 delivers optimal stability. Also, they identify that rounding strategy using stochastic rounding for gradients and round-to-nearest for forward passes, significantly improves stability, especially for backward and update paths. These insights can be very beneficial for the community to further push ultra low precision training of large scale models.\n3. The paper introduces a theoretically derived and empirically validated threshold for gradient norms droping below √3 × quantization noise, suggesting dynamic precision adaptation.\n4. Demonstrated training of a 7B LLM on Gaudi2 chips with only FP4 showing practical viability. The trained model also achieved downstream results on par with BF16 baselines.\n\nWeakness:\n1. Although the 7B training shown performs at par with BF16 baseline,  it's unclear if FP4 FQT scales to larger models (30B+) or more diverse architectures beyond the 200B token setting. Additionally, as recent work [1] has also proposed an end-to-end FP4 training framework, the authors are encouraged to clarify the distinctions and provide insights into how their approach compares or complements this effort.\n2. Due to the use of shared scaling across blocks, stochastic rounding, and dynamic precision switching, this could likely incurs non-trivial computational cost. Providing a quantitative analysis of these costs would offer clearer insight into the trade-offs between efficiency and precision stability.\n3. The authors are also encouraged to include comparisons to all-nearest or all-stochastic rounding configurations. This would be beneficial in understanding the relative comparison of different schemes for stable training.\n\n[1] Wang, Ruizhe, et al. \"Optimizing Large Language Model Training Using FP4 Quantization.\" arXiv preprint arXiv:2501.17116 (2025).\n\nQUESTIONS:\n1. Please refer to weakness points (1), (2), (3). Including ablations to address these concerns would strengthen their claims.\n2. How does the FP4-trained model compare to FP16 baseline on downstream tasks (e.g., QA, summarization, code generation)? Does full quantization affect generalization across domains or modalities? The authors are encouraged to include a few difficult baselines to demonstrate the at-par performance wrt BF16 baseline.\n3. How sensitive is the training stability to the √3 gradient-to-noise threshold? Also, can the authors elaborate on how and when precision switching is triggered in practice?\n4. How does the method behave under varying batch sizes? Smaller batches often lead to noisier gradients, does this make FP4 more prone to training collapse? Also, how sensitive is the FP4 training pipeline to initialization schemes?\n5. Given that the authors use of stochastic rounding for backward pass, how reproducible are the training runs across seeds or hardware environments? Are there significant variance in convergence / final performance for different random seeds?\n\nLIMITATIONS:\nYes, the authors have discussed limitations of their method.\n\nAs a suggestion, the authors are encouraged to further consider robustness related limitations, especially the potential vulnerability of their approach to noisy gradients or its applicability in vision and multimodal settings, where different modalities may exhibit different gradient distributions or magnitudes that could affect the stability of FP4 training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates the first fully quantized training (FQT) of large language models using only FP4 precision for all matrix operations, including weights, activations, and gradients on NVFP4 format, with mixed rounding modes, round-to-nearest in the forward pass and stochastic rounding in the backward pass, to ensure stable training. They also propose a straightforward gradient noise threshold heuristic, based on a simple quadratic loss, to determine when to switch from FP4 to higher precision during training. This is implemented as a brief mixed-precision \"Quantization-Aware Fine-tuning\" phase using BF16. The study features systematic ablation experiments on quantization format, block size, and rounding strategies, achieving competitive zero-shot performance on multiple downstream tasks. Although currently emulated on Gaudi2 hardware, the method is designed as a blueprint for efficient FP4 training on future hardware such as Blackwell.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. First end-to-end FP4 fully-quantized training at non-toy scale, bridging the gap between FP8 and the state-of-the-art hardware.\n2. Clear exploration and ablations on scale formats, block sizes, and per-operand rounding choices.\n3. A full 7B parameter pre-training run on an open-source corpus with +200B tokens, demonstrating realism beyond toy tasks\n4. The noise threshold provides an actionable heuristic for when to up-switch precision, matching empirical behavior.\n\n\n\n\nWeaknesses:\n1. Lacking an isolated round-to-nearest impact per matrix, similar to Fig. 3. Without it, we cannot decide where SR would beat RtN.\n2. While the theoretical derivation in 4.1 is intuitive and somehow hints why we might have to switch to a higher precision, it lacks proper argument on why this critical measure is the direct cause of instability, i.e. it can be concluded that the worst expected error happens at this critical variance, yet doesn’t imply that the absolute value of $E[\\Delta L]$ is critically high when variance reaches this critical value. The derivation does not bound the absolute loss change $E[\\Delta L]$, nor prove divergence or “instability” at that point.\n3. The 220 B-token run yields a token-to-parameter ratio of ~30×, which scaling-law literature[3,4] deems under-trained and thus less likely to expose late-stage instabilities; training a smaller model on the same token budget would be a stronger stress test.\n4. No measured total BOPS, throughput, or average bit-width across FQT + QAF; a quick estimate suggests roughly ~5.2 bits overall.\n5. Experiments rely on Gaudi2 simulation rather than widely deployed Blackwell hardware, so real-world performance gains remain speculative.\n6. Stochastic-rounding implementation is unspecified; the probability mass function is unclear.\n7. Related-work survey omits two sub-FP8 methods, QuEST[1] and AdaBin[2], which should appear in Table 2.\n8. Training hyperparameters are incomplete; learning rate schedules, batch sizes, and optimizer settings, among others, are missing.\n9. All results are based on a single seed. Table 3 lacks error bars or confidence intervals.\n10. No empirical study on how long QAF must run, nor on LR and optimiser sensitivity in FP4.\n11. No exploration of combining the method with PTQ for additional compression. i.e., applying post-training quantisation methods such as SmoothQuant or GPTQ on top of the FP4-trained model, and inspect the combined effect on accuracy.\n\n\n\n\nMinor text issue:\n1. Possible typo on Fig.1 and Fig.2 x-axis label “proceed” -> “processed”\n2. Line 238: “close completely” -> “completely close”\n3. Typo: “trainng” in Fig. 6 caption.\n\n\n\n\nReferences:\n\n[1] QuEST: Stable Training of LLMs with 1-Bit Weights and Activations, Panferov et al.\n\n[2] AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets, Tu et al.\n\n[3] Scaling Laws for Neural Language Models, Kaplan et al.\n\n[4] Training Compute-Optimal Large Language Models, Hoffmann et al.\n\nQUESTIONS:\nI would like to kindly ask the authors to address the issues stated in the “Weaknesses” section. I would gladly increase my scores if the issues are properly addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores FP4 training of LLMs using NVFP4 format. The authors use stochastic rounding for the backward path, which is instrumental in getting unbiased gradients. Furthermore, the authors provide theoretical and empirical analysis of the quantized SGD convergence. The proposed method also explores switching to higher precision training when quantization noise dominates gradient updates. The paper is well structured, well-written, and provides insightful information.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper is proposing a solid methodology and mathematical derivation. I personally enjoyed seeing mathematical derivation accompanied with empirical evidence,\n\n2. Fig 1 and Fig 2  provide insightful intuition regarding the number formats. Also Fig 4 and Fig 5 are informative and provide empirical evidence for mathematical derivation of critical quantization noise level.\n\n3. Fig 6 shows quantized training loss is similar to the baseline training, showing the effectiveness of this algorithm for Llama 7B model.\n\n4. Experimental evidence with perplexity and zero shot tasks shows strong evidence that training procedure was effective.\n\n**Weaknesses**\n\nThis paper does not have too many weaknesses; here are my suggestions to further improve the quality of the presentation:\n\n1. As mentioned by the authors, the lack of validation in FP4 hardware is a key limitation of this paper.\n\n2.  It would also be insightful to see the results of FP4 training for more models and more evaluation tasks such as EvalPlus and GPQA.\n\n3. Section 4.1 can include quantized SGD convergence analysis to make sure and understand the conditions of quantized SGD convergence.\n\nQUESTIONS:\nAs mentioned in the previous section:\n\n1. More evaluation results such as GPQA, IFEveal ( Open LLM Leaderboard 2 ), EvalPlus will add more insight on how well the quantized model is trained.\n\n2. Did authors use any sort of Maximal Update Parametrization (muP)? What was the effect of quantization on that? [1]\n\n3. Is it possible to derive some mathematical guarantee that quantized SGD converges given that higher precision SGD converges? Example: refer to Section 4 of [2]\n\nReferences:\n\n[1] Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).\n\n[2] Ghaffari, A, et al. \"Is integer arithmetic enough for deep learning training?.\" Advances in Neural Information Processing Systems 35 (2022): 27402-27413.\n\nLIMITATIONS:\nAuthors adequately addressed the limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the first successful demonstration of FP4 training for weights, activations, and gradients. Leveraging the NVFP4 format and a carefully selected rounding strategy (stochastic rounding for backward and round-to-nearest for forward), the authors train a 7B LLaMA2 on 200B tokens using 256 Intel Gaudi2 devices. The trained FP4 model achieves performance on par with BF16 after a brief quantization-aware finetuning (QAF) phase.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper presents the first demonstration of FP4 across all three GEMMs (forward, backward, update) at scale (7B model, 200B tokens).\n- The use of FP4 for both weights and activations has the potential to largely accelerate training.\n- The theoretical analysis of the gradient-to-noise threshold—supported by empirical evidence—offers valuable insight into when higher-precision training should be resumed.\n- It is great to see the open-sourced code.\n\nWeaknesses\n\nWhile large-scale experiments are understandably expensive, it would strengthen the paper to include broader evaluations. \n\n- It would be great to evaluate your methods across different model sizes or architectures.\n- A key limitation is that the proposed FP4 training is only simulated on Gaudi2 hardware, which lacks native FP4 support. As such, the actual speed and memory benefits remain speculative. It would be great if you could provide a solid prediction for the cost reduction. \n- The paper lacks qualitative or in-depth comparison with other recent FP4 training approaches. The baseline is only BF16. \n- Details of the training configuration—such as optimizer type, learning rate schedule, batch size, and initialization—are only briefly mentioned and could be more thoroughly documented for full reproducibility.\n\nQUESTIONS:\nCould the authors clarify the scope and applicability of the proposed FP4 training method? Specifically, are there particular model types, sizes, or tasks for which this approach is especially suitable or advantageous? Or do the authors believe it can generalize effectively to any LLM training scenario?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}