Bridging the Gap Between Promise and Performance for Microscaling FP4
  Quantization
Authors: name: Vage Egiazarian; name: Roberto L. Castro; name: Denis Kuznedelev; name: Andrei Panferov; name: Eldar Kurtic; name: Shubhra Pandit; name: Alexandre Marques; name: Mark Kurtz; name: Saleh Ashkboos; name: Torsten Hoefler; name: Dan Alistarh
Published: 2025-09-27 09:22:21

Abstract: The recent hardware-accelerated microscaling 4-bit floating-point formats
such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to
revolutionize large language model (LLM) inference. Yet, their practical
benefits remain unproven. We present the first comprehensive study of MXFP4 and
NVFP4 for post-training quantization, revealing gaps between their promise and
real-world performance. Our analysis shows that state-of-the-art methods
struggle with FP4, due to two key issues: (1) NVFP4's small group size provably
neutralizes traditional outlier mitigation techniques; (2) MXFP4's power-of-two
scale quantization severely degrades accuracy due to high induced error. To
bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the
classic GPTQ quantization algorithm that tailors the quantization process to
FP4's unique properties, by using block-wise Hadamard transforms and
format-specific optimizations. We support our proposal with a set of
high-performance GPU kernels that enable the MR-GPTQ format with negligible
overhead, by rotation fusion into the weights, and fast online computation of
the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and
2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on
RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches
or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the
point where it nears that of NVFP4. We conclude that, while FP4 is not an
automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock
a new frontier of accuracy-performance trade-offs.