ICML 2025

ResQ: Mixed-Precision Quantization of Large Language Models
with Low-Rank Residuals
Utkarsh Saxena 1 Sayeh Sharify 2 Kaushik Roy 1 Xin Wang 2
Abstract
Post-training quantization (PTQ) of large lan-
guage models (LLMs) holds the promise in re-
ducing the prohibitive computational cost at infer-
ence time. Quantization of all weight, activation
and key-value (KV) cache tensors to 4-bit without
significantly degrading generalizability is chal-
lenging, due to the high quantization error caused
by extreme outliers in activations. To tackle this
problem, we propose ResQ, a PTQ method that
pushes further the state-of-the-art. By means of
principal component analysis (PCA), it identifies
a low-rank subspace (in practice 1/8 of the hidden
dimension) in which activation variances are high-
est, and keep the coefficients within this subspace
in high precision, e.g. 8-bit, while quantizing the
rest to 4-bit. Within each subspace, invariant ran-
dom rotation is applied to further suppress outliers.
We show that this is a provably optimal mixed pre-
cision quantization scheme that minimizes error.
With the Llama and Qwen2.5 families of mod-
els, we demonstrate that ResQ outperforms recent
uniform and mixed precision PTQ methods on
a variety of benchmarks, achieving up to 33%
lower perplexity on Wikitext than the next best
method SpinQuant, and upto 5× speedup over
16-bit baseline. Code is available here.1
1. Introduction
Growing capabilities of large language models (LLMs)
come with an increasing computational cost at inference
time. LLM inference has two distinct stages: prefilling,
which processes the input prompt and populates the internal
1Department of Electrical and Computer Engineering, Purdue
University, West Lafayette, USA 2d-Matrix, Santa Clara, USA.
Correspondence to: Utkarsh Saxena <saxenau@purdue.edu>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1https://github.com/utkarsh...