INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization
  Formats
Authors: name: Mengzhao Chen, name: Meng Wu, name: Hui Jin, name: Zhihang Yuan, name: Jing Liu, name: Chaoyi Zhang, name: Yunshui Li, name: Jie Huang, name: Jin Ma, name: Zeyue Xue, name: Zhiheng Liu, name: Xingyan Bin, name: Ping Luo
Organization: ByteDance Seed

Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly
embracing low-precision floating-point (FP) formats to handle the pervasive
activation outliers in Large Language Models (LLMs). Despite this industry
trend, a unified comparison of FP and integer (INT) quantization across varying
granularities has been missing, leaving algorithm and hardware co-design
without clear guidance. This paper fills that gap by systematically
investigating the trade-offs between FP and INT formats. We reveal a critical
performance crossover: while FP excels in coarse-grained quantization, the
comparison at fine-grained (block-wise) levels is more nuanced. Our
comprehensive comparison demonstrates that for popular 8-bit fine-grained
formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart
in both algorithmic accuracy and hardware efficiency. However, for 4-bit
formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we
show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like
Hadamard rotation are applied. We also introduce a symmetric clipping method
that resolves gradient bias in fine-grained low-bit INT training, enabling
nearly lossless performance for MXINT8 training. These findings challenge the
current hardware trajectory, demonstrating that a one-size-fits-all FP approach
is suboptimal and advocating that fine-grained INT formats, particularly
MXINT8, offer a better balance of accuracy, power, and efficiency for future AI
accelerators.