ICML 2025

RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for
Large Language Models
Quan Wei * 1 Chung-Yiu Yau * 2 Hoi-To Wai 2 Yang (Katie) Zhao 1 Dongyeop Kang 3 Youngsuk Park 4
Mingyi Hong 1
Abstract
Supervised fine-tuning is a standard method
for adapting pre-trained large language models
(LLMs) to downstream tasks. Quantization has
been recently studied as a post-training technique
for efficient LLM deployment. To obtain quan-
tized fine-tuned LLMs, conventional pipelines
would first fine-tune the pre-trained models, fol-
lowed by post-training quantization. This often
yields suboptimal performance as it fails to lever-
age the synergy between fine-tuning and quanti-
zation. To effectively realize low-bit quantization
of weights, activations and KV caches in LLMs,
we propose an algorithm named Rotated Straight-
Through-Estimator (RoSTE), which combines
quantization-aware supervised fine-tuning (QA-
SFT) with an adaptive rotation strategy that iden-
tifies an effective rotation configuration to reduce
activation outliers.
We provide theoretical in-
sights on RoSTE by analyzing its prediction error
when applied to an overparameterized least square
quantized training problem. Our findings reveal
that the prediction error is directly proportional to
the quantization error of the converged weights,
which can be effectively managed through an op-
timized rotation configuration. Experiments on
Pythia, Qwen and Llama models of different sizes
demonstrate the effectiveness of RoSTE. Com-
pared to existing post-SFT quantization baselines,
our method consistently achieves superior perfor-
mances across various tasks and different LLM
architectures. Our code is available at https:
//github.com/OptimAI-Lab/RoSTE.
*Equal contribution 1Department of Electrical and Computer
Engineering, University of Minnesota, USA. 2Department of Sys-
tems Engineering and Engineering Management, The Chinese Uni-
versity of Hong Kong, Hong Kong SAR of China. 3Department
of Compute...