Metis: Training Large Language Models with Advanced Low-Bit Quantization
Authors: name: Hengjie Cao, name: Mengyi Chen, name: Yifeng Yang, name: Ruijun Huang, name: Fang Dong, name: Jixian Zhou, name: Anrui Chen, name: Mingzhi Dong, name: Yujiang Wang, name: Jinlong Hou, name: Yuan Cheng, name: Fan Wu, name: Fan Yang, name: Tun Lu, name: Ning Gu, name: Li Shang

This work identifies anisotropic parameter distributions as a fundamental
barrier to training large language models (LLMs) with low-bit quantization: a
few dominant singular values create wide numerical ranges that conflict with
the inherent bias of block-wise quantization. This bias disproportionately
preserves high-magnitude values while discarding smaller ones, causing training
instability and low model performance. This work introduces Metis, a training
framework that combines (i) spectral decomposition with random embedding to
efficiently disentangle dominant from long-tail components, compressing broad
distributions into quantization-friendly narrow ranges; (ii) adaptive learning
rates in the spectral domain to amplify underrepresented directions and better
capture diverse features critical for performance; and (iii) a dual-range
regularizer that jointly constrains numerical precision and parameter range
distribution, ensuring stable, unbiased low-bit training. With Metis, FP8
training surpasses FP32 baselines, and FP4 training achieves accuracy
comparable to FP32, paving the way for robust and scalable LLM training under
advanced low-bit quantization. The code implementation for Metis is available
at: https://github.com/typename-yyf/Metis-quantization.