Metis: Training Large Language Models with Advanced Low-Bit Quantization
Authors: name: Hengjie Cao; name: Mengyi Chen; name: Yifeng Yang; name: Ruijun Huang; name: Fang Dong; name: Jixian Zhou; name: Anrui Chen; name: Mingzhi Dong; name: Yujiang Wang; name: Jinlong Hou; name: Yuan Cheng; name: Fan Wu; name: Fan Yang; name: Tun Lu; name: Ning Gu; name: Li Shang
Published: 2025-08-30 08:09:08

Abstract: This work identifies anisotropic parameter distributions as a fundamental
barrier to training large language models (LLMs) with low-bit quantization: a
few dominant singular values create wide numerical ranges that conflict with
the inherent bias of block-wise quantization. This bias disproportionately
preserves high-magnitude values while discarding smaller ones, causing training
instability and low model performance. This work introduces Metis, a training
framework that combines (i) spectral decomposition with random embedding to
efficiently disentangle dominant from long-tail components, compressing broad
distributions into quantization-friendly narrow ranges; (ii) adaptive learning
rates in the spectral domain to amplify underrepresented directions and better
capture diverse features critical for performance; and (iii) a dual-range
regularizer that jointly constrains numerical precision and parameter range
distribution, ensuring stable, unbiased low-bit training. With Metis, FP8
training surpasses FP32 baselines, and FP4 training achieves accuracy
comparable to FP32, paving the way for robust and scalable LLM training under
advanced low-bit quantization. The code implementation for Metis is available
at: https://github.com/typename-yyf/Metis-quantization.