RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
Authors: Quan Wei
Chung-Yiu Yau
Hoi To Wai
Yang Zhao
Dongyeop Kang
Youngsuk Park
Mingyi Hong
Venue: ICML 2025 poster (2025)
Keywords: model quantization
quantization-aware training
fine-tuning
large language models
Primary area: deep_learning->large_language_models

Abstract: Supervised fine-tuning is a standard method for adapting pre-trained large language models (LLMs) to downstream tasks. Quantization has been recently studied as a post-training technique for efficient LLM deployment. To obtain quantized fine-tuned LLMs, conventional pipelines would first fine-tune the pre-trained models, followed by post-training quantization. This often yields suboptimal performance as it fails to leverage the synergy between fine-tuning and quantization. To effectively realize low-bit quantization of weights, activations and KV caches in LLMs, we propose an algorithm named Rotated Straight-Through-Estimator (RoSTE), which combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy that identifies an effective rotation configuration to reduce activation outliers. We provide theoretical insights on RoSTE by analyzing its prediction error when applied to an overparameterized least square quantized training problem. Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration. Experiments on Pythia, Qwen and Llama models of different sizes demonstrate the effectiveness of RoSTE. Compared to existing post-SFT quantization baselines, our method consistently achieves superior performances across various tasks and different LLM architectures. Our code is available at https://github.com/OptimAI-Lab/RoSTE.

RoSTE: An Efficient Quantizatinn-Aware Supervised Fine-Tuuing Approach for Large Language Models
Quan Wei" Chung- Yiu Yau * Hoi-To 2 Yang (Katie) Zhao Mingyi Hong'
Dongyeop Kang² Youngsuk Park
Abstract
1. Introduction
Supervised fine-tuning a standard method for adapting pre-trained large language models (LLMs) downstream tasks. Quantization has been recently studied as post-training technique for efficient L.MM deployment. To obtain quan- tized fine-tuned LLMs, conventional pipelines would first fine-tune the pre-trained models, fol- lowed by post-training quantization. This often yields suboptimal performance fails to lever- age the synergy between fine-tuning and quanti- zation. To effectively realize low-bit quantization of weights, activations and KV caches in LLMs, we propose an algorithm named Rotated Straight- Through-Estimator (RoSTE), which combines quantization-awe aware supervised fine-tuning (QA- SFT) with an adaptive rotation strategy that iden- tifies an effective rotation configuration to reduce activation outliers. We provide theoretical in- sights RoSTE by analyzing its prediction error when applied to an overparameterized least square quantized training problem. Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an op- timized rotation configuration. Experiments on Pythia, Qwen and Llama models of different sizes demonstrate the effectiveness of RoSTE. Com- pared to existing Post-SFT quantization baselines, our method consistently achieves superior perfor- mances across various tasks and different LLM architectures. Our code is available at https: //github wtm.uu.oooooooooooooooooo......t cwm impI- Lab Lab/ /ROSTE.
LLMs have shifted significant step toward achieving ar- tificial general intelligence (Bubeck et al., 2023) and ex- hibit remarkable capabilities across different domains, in- cluding text generation (Anil et al., 2023; Touvron et al., 2023; Thoppilan et al., 2022), code generation (Chen et al., 2021; Austin et al., 2021; Li et al., 2022), and mathematical problem-solvin (Cobbe et al., 2021; Trinh et al.. 2024; Wei et al., 2022; Lewkowycz al., 2022). To adapt LLMs to various applications and scenarios, supervised fine-tuning (SFT) is a standard approach, enabling models to leverage diverse training data and align with specific tasks based on pre-trained models.
While fine-tuned models excel in domain-specific tasks, their substantial computational and storage demands present challenges for efficient deployment, particularly in resource- constrained environments (Xu et al., 2024a). To address these limitations, various model compression techniques have been developed, including quantization (Lin et al., 2023; Frantar et al., 2022), pruning (Ma et al., 2023; Sun et al., 2023), distillation (Xu al., 2024b), and low-rank ap- proximation (Wang et al., 2024a: Yuan al., 2023). Among these, quantization is particularly effective for compress- ing LLMs, it significantly reduces memory consumption, inference latency and power usage. Additionally, its compat- ibility with specialized hardware accelerators enhances its practical deployment across diverse devices. Quantization techniques generally fall into two categories: post-training quantization (PTQ) and quantization aware training (QAT). PTQ is well-suited for quick deployment with minimal re- sources but often sacrifices accuracy in low-bit settings. In contrast, QAT achieves effective compression with minimal performance loss but requires retraining the entire LLM on a large corpus, incurring substantial computational costs.
Equual contribution 'Deparmment of Electrical and Computer Engineering, University of Minnesota, USA. 'Deparmeent Sys- tems Engineering and Engineering Management, The Chinese Uni- versity of Hong Kong, Hong Kong SAR of China. 'Department of Computer Science and Engineering, University of Minnesota, USA. "Amazon Web Services, USA. Correspondence to: Yang (Katie) Zhao <yangzhao @ umn.edu'>.
For efficient deployment task-specific LLMs, combining quantization with fine-tuning techniques offers a promising solution. A straightforward approach obtaining quantized fine-tuned LLMs involves a two-step process: first fine-tune the pre-trained models, then apply quantization. However, applying quantization through PTQ in the second step often degrades the performance of the fine- tuned models, while
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Accuracy vs Training Time (Exp1: Pythia 6.98)
Accuracy Training Time (Exp1: Qwen2.5 7B)
Accuracy vs Training Time (Exp2: Liama 3.1 8B)
25
RoSTE
25-
STE
Avg. ROUGE Scooe
20
STE
Avg. ROUGE Score
20
Svg. Accuracy
30
QuaRot
SpinQuant
RoSTE
15
15-
25-
10-
20
STE
10
QuaRat GPTQ
QuaRot
SpinQuant
15
SpinQuant
GPTQ
10-
GPTQ
RTN
RIN
2.0
2.5
3.0
3.5
4.0
2.0
2.5
3.0
3.5
4.0
4.5
5.D 5.5 6.0 6.5 7.0 7.5 8.0 8.5
Total Training Time (Hours)
Total Training Time (Hours)
Total Training Time (Hours)
SFT Bit)
Base (FP16)
S ai)
Base [FP16]
SFT+ (4 (4-5FT bit)
Base (3F16)
QA-SFT (4 bit)
SFT (FP16)
QA-STT (4 bit)
SFT (PP16)
SFT (BF16)
Figure 1. RoSTE surpasses the performance of SOTA quantization methods fine-tuning benchmark. Horizontal axis represents the total amount of hours needed to fine-tune pre-trained LLMS on a server of 8 x A100 NVIDIA GPUs.
QAT introduces additional training phase, substantially increasing computational costs. Treating fine-tuning and quantization as separate steps can lead to suboptimal results, as it fails to exploit the synergy between these processes.
manifold optimization, we propose bilevel optimiza- tion formulation, where upper level subproblem optimizes weight matrices, while lower level subproblem employs a surrogate loss to guide the selection of rotation matrix.
This work presents one of the first studies on quantization- aware supervised fine-tuning (QA-SFT) to obtain effective fine-tuned and quantized LLM through a single training phase. To maximize the hardware capability of modern GPUs, we concentrate on designs utilizing the 4-bit quan- tization of weights, activations, and KV cache in L.MSs. However, low-bit quantization presents a major challenge on models with weight and activation outliers: they expand the quantization range and increase the quantization error, degrading the quantized model accuracy. This partly plains why high-performance data-free QAT methods (Liu et al., 2023) fail at 4-bit activation quantization.
To tackle the bilevel QA-SFT optimization, we propose the RoSTE algorithm which alternates between (i) QAT subroutine incorporating rotation-enabled straight- through-estimator (STH) update, and (ii) a low complexity heuristic for selecting rotation matrices based on the ran- dom Halss--addmard matrix.
We provide a theoretical analysis of the benefits of rotation-enabled quantization in QA-SFT by examin- ing the prediction error resulted from the QAT stage of ROSTE. This analysis directly motivates the use of quanti- zation error-based surrogate loss and justifies the adoption of the low-complexity
rotation.
The first key aspect of our work is to leverage rotation-based quantization in QA-SFT. Our work is inspired by recent findings on rotation-based PTQ methods (Ashkboos al., 2024b; Liu et al., 2024), which demonstrate that applying offline and online rotations to linear projection layers and KV caches in LLMs effectively mitigates weight and activa- tion outliers in post-trained models. However, it is not clear whether one-shot PTQ appraoches should be performed be- fore or after fine-tuning. To address this, we propose a joint training method combining an adaptive selection of ro- tation matrices and QA-SFT. The second key aspect of our work is to utilize a bilevel optimization formulation that simultaneously tackles QA-SFT and selects the rotation matrices based on the weights and activations.
We conduct experiments on fine-tuning Pythia, Qwen and Llama models. demonstrating the effectiveness of ROSTE, An accuracy-vs -rtaiyii-tttainnnttinitt plot in Figure 1 illustrates that RoSTE finds quantized fine-tuned models with improved downstream tasks accuracy at the cost of a marginally longer training time. Overall, we believe this is the first work that develops an efficient quantization algorithm for the SFT process, with theoretical justification and practical effective- ness.
1.1. Related Works
The quest on quantizing modern LLMs for efficient deploy- ment started with weight-only quantization. GPTQ (Frantar et al., 2022) stood out as a robust postprocessing algorithm that matches the layer output between a quantized model and full-precision target model. Soon after, a new trend turned to tackling outlier values in weight matrices and acti- vations due to their incompatibility with quantization (Lin et al., 2023; Lee et al., 2023; Chee et al., 2024; Tseng et al., 2024), pushing the limit of accurately quantized models
This paper proposes the Rotated Straight-Through- Estimator (RoSTE) algorithm that integrates the aforemen- tioned ingredients. Our contributions are summarized as:
We introduce a novel SFT training problem that directly optimizes quantized weights and rotation matrices within a single model architecture. To tame the non-smooth
2

---

RoSTE: An Efficient QA-SFT Approach for LLMS
below 2-bits. While the memory consumption of storing the model parameters is reduced by weight-only quantization, their activations remain in full precision during inference which prohibits the application of long context LLMs on consumer-grade accelerators with limited memory storage.
represent element-wise additionn/uubraatioon between tensor and scalar. The choice of scaling s(X) E R and shifting
x(X) E Z determines the range of which the b-bits integer
tensor represents. For symmetric quantization, we adopt
max (x.)) s(X)
This motivates the development of weight-activation quan- tization. It allows weights and activations be directly multiplied using discrete arithmetic units that accelerate inference and reduce the inference memory requirement. Existing methods can be categorized as follows: (I) mixed- precision quantization (Dettmers et al., 2022; Zhao et al., 2024) that assigns extra bit-widths to outlier values; (2) scaling- based quantization (Xiao et al., 2022; Shao et al., 2023) that employs scaling to balance the representation range between activations and weights: (3) rotated quantiza- tion (Ashkboos al., 2024b; Liu et al., 2024) that utilizes orthogonal transformation to remove activation outliers; (4) knowledge distillation (Liu et al., 2023; Du et al., 2024; Xu et al., 2024c) that re-trains a quantized model to match the behavior of a full- full-precision target model. Among these methods, rotated quantization methods demonstrate superior performance in 4-bit weight-activation quantization.
26-1 -1 C,
2 x = 0,
(2)
clamp, (X) max - m min 1,22bb1-00 1}}
with e (0, 1] a constant clipping factor used scale down the representation range so as to mitigate the impact of outlier values. To take advantage of the representation range in tensor with value distribution skewed away from O, we can adopt asymmetric quantization by choosing
max (X. --imin (X.
min
8 s(X)
G, x(x)
26 26-1
x(X)
clamp, (X) = max min (X, 2b - 1}}
(4)
The above uniform quantization scheme reduces the mem- ory consumption from storing a d-elements 32-bit floating- point tensor X using 32d bits, storing an integer tensor with its shifting and scaling scalars using bd b + 32 bits.
2. Preliminary
In practice, we partition a tensor into quantization groups such that each group has its own scaling and shifting s (X), xX). Further description of quantizer hyperparame- ters used in our work will be provided in Appendix D.
This section provides an overview of two major approaches for achieving efficient quantized L.Mss, which are two key ingredients to the proposed RoSTE algorithm.
Incoherence Processing via Rotation. The precision of uniform quantization degrades as the representation range increases, especially when there are outlier values in the full-precision tensor. To reduce the effects of outlier val- ues, incoherence processing was proposed in (Chee et al., 2024) which pre-multiplies an orthogonal matrix to the full- precision tensor prior to quantization, and post- post-multiplies the transposed orthogonal matrix to recover the original ten- sor. Later in (Ashkboos al., 2024b; Tseng et al., 2024), in- coherence processing by Walsh-Hadamart rotation is shown to be effective in both uniform quantization and vector quan- tization for 4-bits reight-activation quantization in LLMs.
2.1. Post- post-Training Quantization (PTQ)
The main objective of post-training quantization is to find a quantized model that preserves the behavior of the original model. While sophisticated quantizer designs such as vector quantization (Tseng al., 2024; Egiazarian et al., 2024) can maintain rich representation of weight values using S 2 bits average, most existing works are limited to weight- only quantization. In contrast, for computationally efficient designs with quantized weights and activation, we focus on uniform quantization that compresses a full-precision tensor into one floating point scaling factor and a set of bit-width limited integers. This scheme is known for its practical efficiency across different modern hardware (Jacob et al., 2018; Ashkboos et al., 2024b).
We illustrate the idea of incoherence processing by con- structing a multi-layer feedforward neural network with activation quantizer Q₂ and weight quantizer Que The out- put of the i-th linear layer is given by
Formally, the b-bits uniform quantizer can be expressed as
(XN Wt R.) = (Q, (XR. QU (R. WT)) (R))))
(5)
(1
x s (X)
Q(X)= = clamps
+
x(X)
e xxxx
s(X)
where X is the input activation; Wi is the pre-trained weight matrix; R, denotes rotation matrix and is any acti- vation function. Notice that as R.R.T = I, the architecture in (5) is invariant to any choice of rotation matrix R, when both quantizers Qw Qn are error-free, i.e., Q,, (.) are the identity map. In general when Q(z) * xr it has been observed that these rotation matrices suppressed outliers
6-bits integor tensor
(1)
where X is a high-precision floating-poin tensor; 1.1 de-
notes an element- nearest rounding; clamp;(-) projects the values to the range of b-bits representable integers; (b, e
3

---

RoSTE: An Efficient QA-SFT Approach for LLMS
within each quantizer and preserved the pre-trained model behavior during inference. On the downside, they impose extra memory and computation overhead since rotation is performed during inference within the activation quantizer. Thankfully, these overheads do not counteract the benefits of incoherence processing due to the fast Hadamard CUDA kernels (Ashkboos et al., 2024b; Tseng et al., 2024).
tokens preceding Vi.e- The likelihood P(yr, X c m) is the probability of the target token Vi.a given the input x; and prior context yt, as predicted by the model m.
Although it is natural idea to apply QAT on fine-tuning tasks, existing works such as (Dettmers et al., 2023; Xu et al., 2023) (also see (Lee et al., 2024; Bondarenko et al., 2024) for similar ideas applied to training LLMs) only considered quantization aware adaptation utilizing an additional LoRA architecture. Moreover, they focused on direct quantization without incoherence processing whose performance can be sensitive to outliers. This motivates us to consider integrat- ing QAT with incoherence processing for SFT. In particular, we shall reveal the separate roles of weight matrices and rotation matrices from the perspective of optimization in the next section.
2.2. Quantization-Aware Training (QAT)
The main objective of quantization-awaan training (QAT) is to directly optimize a quantized neural network using gradient-based methods. From an optimization perspec- tive, this is challenging as the quantization operator is not differentiable. To this regard, straight-througl estimator (STE) (Courbariaux et al., 2015; Bai et al., 2018) is com- monly adopted remedy which approximates the Jacobian of quantizer by the identity matrix. During the backward calculation, the derivative of quantizer Q in the chain rule is replaced by
3. Proposed Algorithm: RoSTE
This section presents the Rotated Straight-Through Estimator (RoSTE) algorithm through jointly optimizing the rotation matrices and model parameters. To fix the idea, we parameterize the quantized LLM by the weight matrices { (W:) and rotation matrices (R.) Con- sider the abstraction of an LLM with l layers/modules as mQ : X + R/I,, where X' is the set of sequences with vari- able context length, T is the set of vocabulary, we denote
xx()(X)) 0g(X) OX
(6)
OX
for any differentiable function g. This approximation uti- lizes the insight that quantizer behaves like identity function in low resolution, while tolerating a gradient bias since quantization error persists in high resolution. In prac- tice, STE is known to work well in training quantized neural network models (Li et al., 2017; Yin et al., 2019) as well as LLMs (Liu et al., 2023; Panferov et al., 2025).
mo (x: (W,, 1 NN(:: (LIN W., (=)
These QAT techniques are useful for the sceneario when we consider obtaining a quantized LLM that minimizes the fine-tuning objective, as introduced below.
for any x E X, where LIN was defined in (5), and NN de- notes the neural network architecture such as transformers!.
With the above parameterization, ideal strategy is to consider the optimization problem:
2.3. Supervised Fine-Tuning (SFT)
min (W..R. 1 Carr(mal ; [W R. k=))
(8)
Foundation models that were pre-trained on large unstruc- tured text corpus require fine-tuning to adapt their output behavior for specialized applications such coding assis- tants (Chen et al., 2021) and instruction-folloiing conver- sational chatbot (Ouyang et al.. 2022). Towards this goal, Supervised fine-tuning (SFT) resumes the training of a given (pre-trained) model with the data distribution replaced by an application-specific curated dataset (Chung al., 2024).
s.t. R.R. = I, i i- 0..., 1,
which simultaneously selects the weight and rotation ma- trices under quantization and directly optimizes the SFT objective of the quantized-and-roote
model.
However, tackling (8) can be challenging even with ap- proaches such as alternating optimization. This is because, upon fixing the weight matrices, minimizing the objective function w.r.t. the rotation matrices {R,) involves an in- tractable manifold optimization while the objective function is oondiifferentiable due to quantization. Meanwhile, when the rotation matrices are fixed, the minimization problem
In specific, let 'D = ((x,, y,) denote the SFT dataset with N samples. For each i E [N], X; E X is sequence of input prompt and y; = (yiio.-.., Mi,T-1) 1) with yi.t E y is a sequence of preferred output tokens. To fine-tune the model with D, we consider minimizing the following SFT loss:
j==
For simplicity, excluded the self-attention layers from our notation but the same idea applies to all the layers the transformer architecture, as demonstrated in (Liu et al., 2024). See Appendix D for the implementation details.
(mFr(m(.)) ;= E,
(7) Σ log (( (emmmm(m))
where the expectation is taken w.n.t i E NO....,N N N 1} with a uniform distribution, Vi,ce denotes the sequence of
'Our approach can be applied on other fine-tuning objectives as well, e.g., (Chen et al., 2024; et al., 2024).

---

RoSTE: An Efficient QA-SFT Approach for LLMs
Algorithm I RoSTE Algorithm
Intislize pre- trained weights
1: Input: Pre-trained model parameters (W!t), t-1
Ji=0: step
size > 0.
2: Initialize W = [WPht-]
; K 1
3: for & 0.....
Search D quantization error
<
matricas that
4: /* Rotation configuration */
Tilee
Updato weight matinces by STE gradiant stap
5: Find approximate lower level solution
1
The
Outer koop mitimiree quanitatton errr
R² argmin E(W"T, (R.EED) (HIIH
(9)
Raturn fine-luned weights with adeettd rosetions
Innd lop rininizzs BFT oójustiv landiio
where each R. is chosen as either the identity matrix I, i.c., no rotation, or a random Walsh-Haue Hadamard matrix H, generated according to (19).
Figure 2. The RoSTE algorithm alternates between tackling the lower level problem for rotation configuration and the upper level problem of SFT training using rotation-aware STE.
6: /+ QAT Stage via STE +/ do Draw mini-batch of training samples gkT+ C
7:
for /. 0, .,...,
8:
Motivated to solving the ideal formulation (8), our re- fomulatioo (11), the optimal lower level variable aims at assisting the upper level weights that an STE gradient approximation on LSFr w.r.t. (W.) has smaller bias. However, it remains challenging for us to access the opti- mal rotation matrices for every iteration of the upper level minimization as solving the lower level problem can still be computationally expensive. this regard, we propose a lazy lower level approximation where the rotation matrices are updated after T iterations of optimizing the weight matrices.
{O.... N - }} uniformly at random and update
WKT+ WHT++
(10)
stc.
V WCsrr(mols WKI++ ckT+))
9: end for
10: end for
11:
Output: Quantized fine-tuned mo( m WKKT R)-1).
The RoSTE algorithm is now summarized in Algorithm 1 and Fig. 2. The algorithm is akin to alternating optimization and consists of two parts. The first part (cf. line 7-9) pertains to the QAT stage with SFT objective for selecting the weight matrices {W&) 1f-1 under the rotation matrices. Notice that the computation overhead introduced by rotations are in- significant when {R.} are chosen as the Walsh-Haaammard matrices. The second part (cf. line 5) pertains to the se- lection of rotation matrices in the lower level optimization which is a non-smooth problem on the manifold. Compared to (8), the lower level objective function (12) can be easily computed through calculating quantized weights and a few mini-batch forward passes on sample data. Furthermore, as we will show in Sec. 4, the random Walsh- Halls-Haddmaar matrix H yields minimize E(-). As such, we propose to approximate the subproblem by limiting the search space to R, E [H, using a random Hadamard matrix H (Tseng et al., 2024) and perform a (low-complexity) combinatorial search to ob- tain an approximate lower level solution that decides if the rotation matrix should be applied on each layer. Details of this heuristic implementation be found in Appendix D.
w.r.t. the weight matrices (W,) [= is similar to standard QAT; see (Liu et al., 2023).
The above obstacle motivated us to consider an alternative formulation (albeit somewhat hcuristic) that simplifies the search for [=] adapted to the weight matrices. This formulation explicitly separates the process of (quantized) model training and the rotation matrix optimization, and leverages a simpler objective function over the rotation ma- trices. More specifically, we consider:
min LSFr(mg) ; W, R R( (Wi)
(11)
s.1. [RA E argmin (R.) ( E((W,, R3
t-1 (
s.t. =I, I 1=
1,
wawroximaat-butuuiivee
solution to
which a bilevel optimization problem where the lower level optimal rotation matrices (R; f- minimize the
weight-activation quantization error:
E([W,, R. R
Σ
(R W.) (R.W.) R w.l? (12)
12, X., 1I
4. Theoretical Insights of ROSTE
for XıJ representing the input activation of layer on the j-th calibration data sample, e.g.. by drawing subset of
This section aims at providing theoretical insights on the RoSTE algorithm that tackles the bilevel problem (11). In particular, we show that the quantization error (12) is a
size n from D the fine-tuning dataset.
5

---

RoSTE: An Efficient QA-SFT Approach for LLMS
suitable surrogate loss for optimizing the rotation matrices, provided that the weight matrices are optimized using the STE method in Algorithm 1. We remark that the SFT objective on quantized LLMs is complicated and possibly untractable for analysis. To concentrate on the insights pertaining to using rotation in the quantized LLMs, we shall introduce a few approximations. We will use (- I)) to denote inner products of vectors, and ||x|| = (x Kx) to denote K-weighted squared norm of vector for any square matrix K.
The above conditions are mild as Amin is only the smallest non-zero eigenvalue of the Gram matrix G and p exists when the input prompts x, are bounded.
Assumption 4.2 (Interpolation), For any orthogonal matrix R. there exists wi I2 such that yς (Q (Rxg) wht for any ç.
The above assumption requires that the quantized-rotated features (Q=(Rx))) are interpolatable by full- precision model wi- This assumption is closely related to the standard interpolation assumption that appeared in the literature on training werrparameeterized models (Ma et al., 2018; Vaswani et al., 2019). It is worth noticing that Assumption 4.2 does not require the interpolator wi to be in the quantized model parameter space (14).
Our setup follows from the literature on analyzing the con- vergence of SGD for neural networks under the interpolation regime (Ma et al., 2018; Vaswani et al., 2019). To describe it, let us fix the rotation matrices [R. 1 and consider the QAT stage (cf. line 7-9) in the RoSTE algorithm. In- stead of analyzing (srr(ma(-)) directly, consider the quadratic loss function as a simplified objective to draw insights for RoSTE, Moreover, the training dataset consists of samples (x) with a target output token in R such that Y& E y = R.
Define the shorthand notation m := mo(; w R), we observe the following convergence results for the QAT stage during the RoSTE algorithm:
Theorem 4.3. Under Assumptions 1, 4.2 and the step size
rj == Amin// /(p),, the objective value of the quantized model produced by the recursion (15) is bounded by
For any m : X - RITI, we now consider the squared prediction error:
1 e[C(m Q.R.
++1 )1 S (1 (1-H)) +21
-1) [(m]
(18)
[(m(()) = 2 R [(o(m(xe)) - y
(13)
+ (6 + 2p- ) 1 (1 - 11) E [[le(Rw))la)
in lieu of LSFT(-). (.), where o: RI71 + y' maps the probabil- ity distribution over T to a token.
X for any 2 0, where μ = 12p
and e(x) == Qw x.
We further assume that the composite map (-m) is a
See Appendix A for the proof. Our result shows that STE only converges to an inexact solution, which is consistent with previous findings on STE training. For instance, when training models with activation-only quantization, (Yin et al., 2019, Lemma 10) proved that the STE gradient is non- vanishing near local minima. For models with weight-only quantization, (Li et al., 2017, Corollary 1) only showed a convergence guarantee for the full full-precision weights but not the quantized weights. In comparison to the prior findings, our result demonstrates the convergence of prediction error with quantized model.
linear activation- weight quantized model given by
o w, R)) = (Rx) Qw
(14)
where R is a rotation matrix satisfying RRT = and
Qz. Qu Rd + Rd are the quantization functions [see Sec. 2.1]. Let Yr be the sample drawn at iteration t in the inner loop update of line 8, Algorithm 1, we have
wr+1 w² gg.
(15)
gh.... = ((Q (Rx,) (ww')) R.) Qu (Rx,),
where 17 > 0 is the step size and we have used the STE approximation a(Q (Rw))/0(w) as R when computing the stochastic gradient gi.... at w
More specifically, suppose the QAT stage of ROSTE is run with T > 1 inner-loop iterations. Apply- ing the theorem shows that given R, the resultant prediction error of model w" will be bounded by O (1 - E [[ (Rw*) - Rw [I& i.e., a weighted sum of the weight quantization errors during the QAT process. Due to the exponential weighting (1 (1-))" the prediction error is dominated by the weight quantiza- tion error of recent iterates. Crucially, the above analysis shows that the rotation matrices play a pivoting role in the performance of QAT. This inspires us to apply E(-) in (12) to guide us in the selection for optimal rotation matrices, covering the weight quantization error of the rotated weight matrices ||ee [R] W.)
Our next endeavor is to study an upper bound the loss value of quantized model, L(mo( : R)), after running the recursion (15) for T 2 1 steps. Define the Gram matrix of the quantized-rotate features by
G :==Ba E
Qu ]
(16)
and make the following assumptions accordingly: Assumption 4.1 (Gram Matrix). There exists constants Xmiin P >>0 0 such that
G²2 2 Amnin 1U S p. (17)
6

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Randomized Rotation Matrices. Now we demonstrated that the quantization error is crucial to the prediction per- formance with the quantized model, we turn our focus to tackling the lower-level subproblem in (11). Notice that minimizing E() w.r.t. the rotation matrix remains challeng- ing. Instead of directly tackling the manifold optimization, our strategy is to apply the random aalsh-Hadamard matrix (Tseng et al., 2024) design as an approximate-yet-unnn yet-univerall solution. Consider the random rotation matrix:
not improve the prediction performance as the bound (18) only depends on weight quantization error. On the other hand, increasing the bit-width of weight quantization always reduces the prediction error as seen in (18), (21). It is also interesting to see that despite adopting low-bit activation quantizers, increasing the dimension d may still allow us to satisfy the interpolation condition Assumption 4.2, un- der the intuition that kernelized high dimensional features are more likely to be separable (Liang & Rakhlin, 2020). In other words, neural network with high-dimensional hidden representations can tolerate low-bit quantized ac- tivations because the information about Xę retains in the high- high-dimensional discrete vector Qr((Rx)).
R(C)= HDiag(r(())
(19)
where H E Idd is a Walsh-Hadamard matrix (Fino & Algazi, 1976) and r({) E 1, is a random sign vector. Notice that R(() is binary matrix which favors efficient implementation on GPUs.
5. Experiments
We observe the following proposition adapted from (Tseng et al., 2024, Lemma 3.1):
We evaluate the performance of the proposed RoSTE al- gorithm for QA-SFT on two standard sets of open-source models and datasets. For the first experiment (Exp. 1), we fine-tune the pre-trained Pythia 1B/6.9B models (Biderman et al., 2023) and Qwen2.5 0.5B/7B models (Yang et al., 2024) on the Reddit TL:DR Summarization dataset (Huang et al., 2024) with evaluation on the TL;DR test dataset using the ROUGE metric (Lin, 2004). For the second experiment (Exp. 2), we fine-tune the pre-trained Llama 3.1 8B model (Dubey et al., 2024) on the Tulu 3 SFT mixture dataset (Lambert et al., 2024) with real-world downstream task eval- uations (Gao et al., 2021). These tasks include TruthfulQA (Lin et al., 2021), MMLU-Pro (Wang et al., 2024b), Big- HenchHard (Suzgun et al., 2022), AGIEval (Zhong et al., 2023), GSMSK (Cobbe et al., 2021), and MATH (Hendrycks et al., 2020).
Proposition 4.4. Consider a -bits symmetric quantizer 11. For any w Rd.
Qu RR + Rdd [cf. (2), (3) with
with R I, it holds that
d max; w?
(w) - wl? 4(2 1)2²
(20)
with R- -R(() from (19), with probability 1 1- ō we have
log(4d/5)
[(.. - RR()) 2(2bm. 1)2 //|
(21)
See Appendix for the proof.
Observe that the quantization error is Old max; w2) with- rotation, and is [[[) with rotation. Note that the former bound is more sensitive to weight vectors with out- liers. In particular, the worst case prediction error in the QAT stage with R chosen as (19) is strictly better than that for the case with R- (no rotation) if
For the RoSTE algorithm, while we relaxed the lower level as é-variable binary combinatorial problem (9), solving this sub-problem has complexity of O(2)) which is still in- tractable for models like Llama 3.1 8B with = 3 x 32 + 1. As a remedy, we estimate the solution of (9) by comput- ing only E(pATT, (1){-)) and E(Wkr, ( then we determine each layer's 's R. by comparing the quantization error layer-wise. Lastly, we set K = 1 where one-shot rotation configuration adaptation by pre-trained model is found to perform well. We anticipate the performance to further improve with larger K on larger datasets. More implementation details can be found in Appendices C, D.
log( (4d/0
max, WT, )d
RR.
(22)
4
where WR. WI are the respective converged solutions of (15). It demonstrates that applying the random rotation matrix in (19) suffices to reduce the quantization error of weight matrices that contain outlier values. To obtain the best performance, we design the RoSTE algorithm such that at the outer loop, it chooses between H or I (i.e., no rotation) according to the current weight matrices.
Baselines. Besides the proposed ROSTE algorithm, we com- pare the performances of LLMs with quantized weight and activation obtained by two streams of baseline approaches.
The first stream consists of applying PTQ methods open- source supervised fine-tuned models in (Huang et al., 2024; Lambert et al., 2024). We reproduce the PTQ benchmarks using round-to-neared (RTN) quantization, GPTQ (Frantar et al., 2022). QuaRot (Ashkboos et al., 2024b) and Spin- Quant (Liu al., 2024). The second set consists of QAT
Remark 4.5. The analysis in Theorem 4.3 and (22) enables a novel interpretation of the bit-widths in Qn and Qu during STE training. On one hand, it is beneficial to increase the bit-width of activation quantization Qn until Assumption 4.2 satisfied, and further increasing its bit-width would
7

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table Results Exp. 1. Accuracies of the 4-hit quantized Pythia 6.9B and Qwen2.5 7B models fine-tuned using the Reddit TI.;RR dataset. FF 16 and BE 16666 refer to using 16-bit half precision floating points and 16-bit brain floating points formats, respectively, and M4A4XV4 refers to using b-bit quantizations on weights, activation, and KV cache.
Bit-width
Method
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-LSum
ROUGE (Avg.)
Pythia-6.9B
Base
28.81
9,45
22.29
22.91
20.87
FP16
SFT
33.69
12.60
26.27
26.31
24.72
RTN
7.42
0.06
6.53
6.56
5.14
GPTQ
8.16
0.08
7.06
7.60
5.73
QuaRot
11.70
0.23
8.52
9.39
7.46
W4A4KV4
SpinQuant
8.61
0.10
8.10
8.07
6.22
STE
28.91
9.07
22.30
22.33
20.65
RoSTE
32.60
11.54
25.25
25.25
23.66
Qwen2.5-7B
Base
32.72
11.82
25.18
25.42
23.79
BF16
SFT
34.75
13.59
27.56
27.58
25.87
RTN
1.07
0.00
1.01
1.01
0.77
GPTQ
0.72
0.00
0.69
0.69
0,53
QuaRot
7.21
0.10
5.93
5.93
4.79
W4A4KV4
SpinQuant
6,87
0.29
5.97
6.12
4.81
STE
30.86
10.16
23.73
23.73
22.12
RoSTE
34.01
12.89
26.74
26.74
25.10
methods applied on the SFT objective, including STE and RoSTE. The hyperparameters reproducing our experi- ment results can be found the Appendix Table and 5.
Layer 30 MLP. Down, Proj
Layer MUP Down. Pro)
All experiments are conducted cluster of 8 NVIDIA A100 GPUs. Details of the training and evaluation settings can be found in Appendix C. Statistics of the training cost (time, memory) can be found in Appendix G.
2E0CE Charci
10000
5.1. Accuracy of Quantized Models
(a) By STE.
(b) By ROSTE.
For Exp. 1, we present the accuracies of 4-bits (weights & activation) quantized, munettned Pythia 6.9B and Qwen2.5 7B in Table On quantizing Pythia 6.9B, the best baseline STE (without rotation). In comparison, RoSTE produces quantized model that improves the average ROUGE score by +3.01. It recovers the performance of the full- precision SFT model with gap of only -1.06 ROUGE score. Simi- larly on Qwen2.5 7B model, RoSTE improves upon the best baseline by +1.78 ROUGE score, with gap of -0.77 be- low the full-precision SFT model. For Exp. 2, we present the accuracies of 4-bits (weights & activation) quantized, fine-tuned Llama 3.1 8B in Table Observe that RoSTE im- proved the average accuracy by +2.56 over the best baseline SpinQuant, despite gap of -00.47 below the full full-precision fine-tuned model.
Figure 3. Visualizations of input activations layer 30 of verged Llama model trained for QA-SFT using STE and RoSTE.
Lastly, in the appendix, we provide additional results of W4A4K4 and W4A8K4 quantization Pythia 1B in Table 7, Qwen2.5 0.5B in Table 8, and I.lama 3.1 8B in Table 9.
5.2. Ablation Study
We now concentrate the effects of optimal rotation config- uration as practiced in line 5 of Algorithm 1. In Table 3, we compare the performance on Exp. 1 with Pythia IB when the random Walsh-Hadamard rotation matrix is applied with different strategies. Notice that the adaptive rotation strat- egy deployed in RoSTE delivered the best performance.
8

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table Results Exp. Accuracies of the 4-hit quantized Ilama 3.1 8B model fine-tuned on the Tulu 3 SFT mixture dataset. BP
refers to using 16-bit brain floating points format, and 74A4KV4 refers to using 4-bit quantizations on weights, activation, and KV cache.
Bit-width
Method
TruthfulQA
MML.U-Pr
BigBenchHard
AGIEval
GSM8K
Math
Avg.
Base
28.51
19.57
62.26
30.16
56.86
18.20
35.92
BFI6
SFT
31.82
33.07
65.67
34.86
64.89
22.66
42. 16
RTN
23.01
0
0
17.03
1.03
0
6,85
GPTQ
25.34
0.02
2.55
16.48
2.05
0
7.74
QuaRot
27.66
21.53
47.69
29.05
37.91
6.90
28.46
W4A4KV4
SpinQuant
26.19
21.58
49.56
28.50
38.36
10.56
29.13
STE 26.68
9.13
24.58
17.63
22.82
1.90
17.14
RoSTE
26.44
25.12
52.00
30.11
44.50
11.94
31.69
For a more detailed and comprehensive comparison, see Fig. and in the appendix. Furthermore, Fig. shows the trajectory of the quantization error (12) during training. As expected,
Table Effects of rotation matrix strategies for STE training in Exp. 1 with Pythia IB that is W4A4KV4 quantized.
Rotation Strategy
ROUGE (Avg.)
that the quantization error of RoSTE is much lower than that in STE, thus suggesting lower bias in the solution obtained [cf. Theorem 4.3].
No Rotation
22.37
Complete Rotation
13.09
RoSTE (Adaptive Rotation)
23.07
6. Conclusion
This paper proposed the RoSTE algorithm for quantization- aware SFT training with an adaptive rotation strategy. Be- sides achieving satte-of-the-at performance, vide theoretical insights to justify the practical efficacy of RoSTE. To the best of our knowledge, this is the first al- gorithm that leverage adaptive rotation and the fine-tuning objective to produce an accurate quantized model.
0.6
0.5
also pro-
Quantization Error
0.4
STE RoSTE Base Model
A Acknowledgements
1500
2500
M. Hong and Q. Wei are supported in part by NSF grants ECCS-2426064 and CIF-2414372. Q. Wei is also supported by an Amazon Machine Learning Systems Fellowship. C.-Y. Yau and H.-T. Wai are supported by the Shun Hing Institute of Advanced Engineering, CUHK, under Project #MMT-p5- 23.
Training Steps
Figure 4. The evolution of quantization error (12) against the QAT stage iterations.
When no rotation matrix is applied, we observe a drop in ROUGE score by -0.70, and importantly, the complete rotation setting, i.e., applying rotation matrix every mod- ule regardless of whether empirical quantization error is reduced, suffers a drop in ROGUE score by -9.98. This shows that while is beneficial to apply rotation in the STE training, an adaptive strategy such as the one in RoSTE is necessary to guarantee good performance.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Secondly, take a closer look the effects of outlier reduction justify claims the of random Walsh- Hadamard rotation matrices. Fig. 3 compares the distribu- tion of the input activations of fine-tuned models trained by STE and RoSTE at layer 30. We observe that the model produced by RoSTE exhibits no activation outliers, while STE suffers from activation outliers even at convergence.
References
Anil, R., Dai, M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.104033 2023.

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Ashkboos, S., Croci, M. L.., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024a.
Courbariaux, M., Bengio, Y., and David, J.-P. Binarycon- nect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 2015.
Ashkboos, S., MoMtashamii, A., Croci, M. L., Li, B., Jaggi, M.. Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00455, 2024b.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 () 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 15:30318-30333 2022.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2023.
preprint arXiv:2 077322, 2021.
Du, D., Zhang, Y., Cao, S., Guo, J., Cao, T., Chu, X., and Xu, N. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillan arXiv preprint 1rXiv:2402.10631, 2024.
Bai, Y., Wang, Y.-X., and Liberty, Proxquant: Quantized neural networks via proximal operators. arXiv preprint arXiv:1870.00861.0000
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp- 2397-2430. PMLR, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Egiazarian, V., Panferov, A., Kuznedelex, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06/ 18, 2024.
Bondarenko, Y., Del Chiaro, R., and Nagel, M. Low- rank cuantizationnawarn training for llms. arXiv preprint crXiv:2406.06338 2024.
Fino and Algazi. Unified matrix treatment of the fast walsh- hadamard transform. IEEE Transactions on Computers, 100(1 112::::11111146,,1 1976.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.127112 2023.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9, 09.8-9,7 9021..
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Broekman, G., et Evaluating large language models trained on code. arXiv preprint arXiv: 2xX.v:2007.33333 03374, 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint MXXi:2009.03300 2020.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXhv:2401.01335 2024.
Huang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., and Tunstall, L. The n+ implementation details of rihf with ppo: A case study tl; dr summarization. arXiv preprini arXiv:2403. 70011, 2024.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., al. Scaling cntruction-finetune language models. Journal of Machine Learning Research, 25(70) 2024.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-onke inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 10.1118, 2021.
10

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, J. V., Liu, A., Dziri, N., Lyu, S., et al. TŞ" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 2021.
arXiv:2109.055
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y. Shi, Y., Krishnamoorthi, R., and Chandra, V. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. 17888, 2023.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, Owq: Outlier- aware weight quantization for efficient fine-tuning and inference of large language models. arXiv preprint arXiv:2306.00 02200200.222222... 2023.
Liu, Z.,, Zhao, C.,, Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIm quantization with learned rotations. arXiv preprint 2rXiv:2405. 2024.
Lee, J., Park, S., Hong, S., Kim, M., Chang, D.-S., and Choi, J. Improving conversational abilities of quantized large language models via direct preference alignment. arXiv preprint arXiv:2407.0305.. 2024.
Ma, S., Bassily, R., and Belkin, M. The power of interpola- tion: Understanding the effectiveness of sgd in modern aver-parametrized learning. In International Conference Machine Learning, pp.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer. E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
PMLR, 2018.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Li, H., De. S., Xu, Z., Studer, C., Samet, H., and Goldstein, T. Training quantized nets: A deeper understanding. Advances in Neural Information Processing Systems, 30, 2017.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 13:27730-277444 2022.
Li, J., Zeng, S., Wai, H.-T., Li, C., Garcia, A., and Hong, M. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for Ilm align- ment arXiv preprint arXiv:2405. /7888, 2024.
Panferov, A., Chen, J., Tabesh, S., Castro, R. L., Nikdan, M., and Alistarh, D. Quest: Stable training of llms with 1-bit weights and activations. arXiv preprint arXiv:2502.05003, 2025.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F,, Dal Lago, A., al. Competition-Ievel code generation with alpha- code. Science, 378(6624) 2022.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L.,, Li, Z., Zhang. K., Gao. P., Qiao, Y., and Luo, P. Omniquant: Ommidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308. S.ti::230..1137. 2023.
Liang, T. and Rakhlin, A. Just interpolate: Kernel "Ridge- less" regression can generalize. The Annals of Statistics, 48(3):1329 1347, 2020.
Sun, M.. Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether thain-of-thoughh can solve them. arXiv preprint arXhv:2210.09261, 2022.
Lin, H., Xu, H., WU, Y., Cui, J., Zhang, Y., Mou, L., Song, L.,, Sun, Z., and Wei, Y. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. Advances in Neural Information Processing Systems, 37: 87766-87800, 2024.
Thoppilan, R., De Freitas, D.,, Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog appli- cations. arXiv preprint arXiv:2201.0822 2022.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for Ilm compres- sion and acceleration. arXiv preprint 2xXiv:2306000978, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.,

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXiv:2412.1511
2024.
Trinh, T. H.. Wu, Y., Le, Q. V., He, H., and Luong, T. Solv- ing olympiad geometry without human demonstrations. Nature, (25(7995):476-44 482, 2024.
Yin, P., Lyu, J., Zhang, S., Osher, S., Qi, Y., and Xin, J. Understanding straight-through estimator in train- ing activation quantized neural nets. arXiv preprint arXiv: FrXXi:1903.06600 2019.
Tseng. A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quip#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 2024.
Yuan, Z.,, Shang, Y., Song, Y., Wu, Q. Yan, Y., and Sun, G. Asvd: Activation-awarc singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Vaswani, S., Mishkin, A., Laradji, I., Schmidt, M., Gidel, G., and Lacoste-Julien, Painless stochastic gradient: In- terpolation, line-search, and convergence rates. Advances in neural information processing systems, 32, 2019.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L.., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6:196-209, 2024.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd- Ilm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024a.
Zhong, W., Cui, R., Guo. Y., Liang. Y.. Lu, S., Wang. Y., Saied, A., Chen, W., and Duan, N. Agieval: cemann-centric benchmark for evaluating foundation mod- els. arXiv preprint arXiv:2304.06336 2023.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574 2024b.
Wei. J., Wang, X., Schuurmans, D.. Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-offthouuee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211. 104388 2022.
Xu, M., Yin, W., Cai, D., Yi, Xu, D., Wang, Q. Wu, B., Zhao, Y., Yang, C., Wang, S., et al. A survey of resource- efficient Ilm and multimodal foundation models. arXiv preprint ar Xxxx:2444.00899 2024a.
Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, т. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13110 13116, 2024b.
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, 7., Zhang, X., and Tian, Q Qa-lora: Quantization- aware low-rank adaptation of large language models. arXiv preprint N31:22309.14717 2023.
Xu, Y., Han, X., Yang. Z., Wang, S., Zhu, Q., Liu, Z., Liu, W., and Che, W. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11299 2024c.
12

---

RoSTE: An Efficient QA-SFT Approach for LLMS
A. Proof of Theorem 4.3
Proof. this proof, we will use the following equality interchangeably:
((mb) E (Rxx) (w(()w)) yt) [IQu w
(23)
Consider the update rule of algorithm as w w (((Q Qmm((Rw))) y.)R (R(x))),, we compute
E ((
| (Rw yE)
(24)
E (RxE)) Q (Rw w + -RQ') Qu
(25)
E (Q2x (uw"" WR) (x(())))) Qu (R."H Q (Rw')) )²]
(26)
Now observe that
E
[pp))]] EE' (Rw -
(27)
(28)
3E [|| Rw'll& 3E (ewe + [|& + 3E [[]e([w")
(29)
"T [1Q. (Rx))l2 (Q₂ (Rx.)) @m Rww)) w '
2
+ 3E |||t R + + )[
+3E [[e((ww")] [|]]
(30)
S 3r7 E [[Q@((R')) - wi | + 3E || Rw || + 3E [|e(
(31)
where the last inequality is due to the independence x x, w and uses Assumption 4.1. Furthermore,
2E [pip1
P$]
(32)
21E [pí (Q,
Rw Rw + +e(Rw e(Rw"))]
(33)
2yyE ([
Qw((ww) wi)) (Qx Qx (Rx:) Qw - -wil)
+ 2E [(Qz (Rxg) Qu (Rw") - -wi) (Qx (Rx;) e(Rw - e(Rw())
(34)
22ylE (Rw') wRll 7mmin E [11 (Rw')- -ikk)-wkkll 1
(35)
2 ([le(R(+1)]1)
2
[aa]
(36)
17mmmnn
77min
2 E
2 1
~m [I -
+ 7
E [|e[[w))) [|
(37)
where the last step is due to the independence x X 1 w' and (17). Therefore, we obtain
[1 (Rw'+1 w
(38)
2
(1 Mimin +31/2p) p) 2v (Rw') 1 + (3
E [||e[RW +1 ||a]] + [[le(Rw")188)
(39)
Mmmin
(8)
< (1 nmmia /2) [1 w (3+ Ammimin
(B [|le(Rw* + aa [Ie((R'))l@))
(40)
< (1 nmmin (2) =2 [1) (Rw") 1
(41)
2 (1-mdminn (2)mmmm/)) (H ||ee Rw ++ 12 +E [[le(w
+ (3 +
(42)
71 Amin
where (i) uses the step size condition 7) S Amin/ (6p). Choosing the step size 17 Amin/(6p) completes the proof.
B. Proof of Proposition 4.4
Proof. By the definition of Qu in (2) and (3), we notice
12
[[Qu(w) w|2 = s(w)
w
s(w)
w
= =ww)²
w
w
(43)
s(w)
s(w)
s(w)
s(w)
13

---

RoSTE: An Efficient QA-SFT Approach for LLMS
d max, w
sww 4 4(2
(44)
1)2
where (i) a worst-case error bound of nearest rounding. This proves (20). Further applying (Tseng et al., 2024, Lemma 3.1) our last step gives (21).
C. Details of Experiment Settings
RoSTE algorithm. We set the lower level objective function in (12) by drawing 128 samples from the fine tuning dataset for calibration.
flyper-parameters. We list the training configurations for SFT in (Exp- 1) TL:DR summarization and (Exp. 2) Tulu experiments as suggested (Huang et al., 2024: Lambert al., 2024) in Table 4. For QA-SFT, we sweep through number of hyper-parameters for STE and RoSTE to obtain the best performance, as listed in Table 5.
Table 4. Detailed training settings for SFT in TL;DR summarization and Tulu experiments.
Method
SFT Qwen2.5 0.5B
Model
Pythia IB
Pythia 6.9B
Qwen2.5 7B
Llama 3.1 8B
Epoch
1
2
Batch Size (Per GPU)
16
16
1
Gradient Accumulation
16
1
16
16
Optimizer
AdamW W
Adam W
AdamW W
Adam W
Adam W
Learning Rate
3e-5
3e-5
5e-5
le-5
5e-6
L.R Schedule
cosine
cosine
cosine
cosine
linear
Warmup Ratio
0
0
0.03
Max. Seq. Length
2048
2048
2048
2048
1024
# Training Samples
117k
117k
117k
117k
100k
Table 5. Detailed training settings and ppper-parameters for QA-SFT in the TL;DR summarization and Tulu 3 experiments.
Method
QA-SFT (i.e., STE or RoSTE)
Model
Pythia IB
Pythia 6.9B
Qwen2.5 0.5B
Qwen2.5 7B
Llama 3.1
Epoch
Batch Size (Per GPU)
16
16
1
Gradient Accumulation
Optimizer
AdamW
AdamW
AdamW
AdamW (5e-5, le-5, 5e-6)
AdamW
Learning Rate
(3e-5, 6e-6, 3e-63
[3e-5, 6e-6, 3--6)
(Se-5, le-5, 5e-63
(5e-6, 1e-6, Se-7)
LR Schedule
cosine
cosine
cosine
cosine
linear
Warmup Ratio
0.03
Max. Seq. Length
2048
2048
2048
2048
1024
# Training Samples
117k
117k
117k
100k
clipping factor
[1, 0.95, 0.9)
(1, 0.95, 0.9)
0.95, 0.9)
{1, 0.95, 0.9)
(1, 0.95, 0.9)
Evalution. For the TL;DR summarization experiments, all final models evaluated on the TL;DR test dataset using the ROUGE metric (Lin, 2004), including ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-L.Su.. For the Tulu 3 experiments, all final models are evaluated on downstream tasks using EleutherAl LM Evaluation Harness (Gao et al., 2021). These tasks include TruthfulQA (Lin et al., 2021), MMLU-Pro (Wang et al., 2024b), BigBenchHard (Suzgun et al., 2022), AGIEval (Zhong et al., 2023), GSM8K (Cobbe et al., 2021), and MATH (Hendrycks et al., 2020). In Table 6, we list the detailed evaluation settings for these downstream tasks as suggested in the Tulu paper (Lambert et al., 2024).
14

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table 6. Details of evaluation settings for the Tulu experiments.
Benchmark
TruthfulQA
MMLU-Pro
BigBenchHard
AGIEval
GSM8K
Math
# shot Metric
6 Acc (mel)
0
3
4
EM
EM
Acc
EM
EM
CoT
X
X
X
X
D. Implementation Details of the Solated-and-Quantzze LLM
Our architecture for inserting rotation matrices and quantization on transformer models follows from (Liu et al., 2024).
For completeness, an illustration is provided Figure 5. the following sections, we describe the details of the RoSTE algorithm under the setting Exp and Exp.
R
W
RoPE
R
Softmax
XR.
Norm
R
RoPE
K
R
R
W.
R;
YR.
(X Y)R.
R
R
MHSA
R
W
up
YR,
(X
Y)R.
XR.
Norm
R
R2
domn
R
W
Silu
MLP
Original Activation
R
W
Winen
Ri
R
R2
R₂
Ra
Rotated Activation
Quantized Activation
Quantized Merged Weight
Offline Mergeable Rotations
Online Rotations
Figure 5. An illustration of the rotation workflow in transformer-hased model. R, represents the between-block rotation, which eliminates activation outliers between blocks. R2, Rs, R4 are in-block rotations designed to remove outliers within MHSA MLP blocks. Among these, R1, R, R2, R R, can be merged into weights while Ra, R3 R., not mergeable and serve online rotations during training and inference.
Quantization on LLMs. We adopt the asymmetric uniform quantizer (4) for all the experiments. For instance, to quantize the activations X of dimensions [batch size, sequence length, embedding size], we employ per-token quantization such that each embedding X.y forms a quantization group. To quantize the linear weight values W, we employ per-channel quantization such that each i-th output channel's weights W; form a quantization group.
Modifying Normalization Layer. We modify the model to maintain computational invariance before and after applying rotation. This requires ensuring that there are no mean subtraction, scaling, or shifting operations in the normalization module. For models with LayerNorm, such Pythia, the process involves absorbing the mean subtraction operation into the weight matrix before LayerNorm and absorbing the LayerNorm scaling and shifting parameters into the weight matrix after the LayerNorm layer (Ashkboos et al., 2024a). Similarly, for models using RMSNorm, such as Llama, this can be achieved by absorbing the RMSNorm scaling parameter into the weight matrix immediately following the RMSNorm layer.
Between-Block & In-Block Rotation. We perform between-bloc rotation R2 to eliminate the activation outliers between blocks. As illustrated in Fig. 5, R, applied linear layers MHSA and ML.P blocks. In particular, the weight
15

---

RoSTE: An Efficient QA-SFT Approach for LLMS
matrices in the Q, K, and V projection layers of MHSA, well the Up and Gate projection layers of MLP, rotated along with their corresponding input activations preserve computational invariance. Similarly, the weight matrices in the O projection layer of MHSA and the Down projection layer of MLP, along with their corresponding outputs, are also rotated using R1. Additionally, we also rotate the embedding and headead layers so that the final output of the model will be identical to the original model. Next, we perform in-block rotations R2, R3, R4 to eliminate the activation outliers within blocks. Specially, R₂ is applied to the Value and the O projection layer MHSA. R3 works for the Query and Key. R2 and R3 can remove the activation outliers for KV caches. We apply R. to the Down projection layer MLP.
R1, R2 are offline mergeable rotations, which can be merged into the weight matrices before training. R3, R4 are online rotations, which implemented in the fast Hadamard kernel and be activation. This online operation is highly efficient by leveraging the fast Hadamard CUDA kernel, resulting in negligible overhead during both training and inference.
layer dynamically rotating the input
E. Impact of Rotation on Different Models
Fig. 6 showcases the effects of (random Valsh--adamarr) rotation applied several exemplary layers in Pythia and Llama models, and demonstrates that sometimes applying the rotation can lead undesirable results where new outlier values emerge. Fig. 7 presents a comprehensive view of the effects of applying rotations to the weights, activation, and KV cache of different layers. Notice that the RoSTE algorithm only applies rotation when reduction of quantization error is observed in the respective layers. Moreover, from the figure we observe that in general, the last layers of Pythia model do not benefit from applying rotation, while the rotation effects on Llama model are generally beneficial.
Pythia 1B
uayer JuAtH_
Layer Dowwohhh
Rotation
Rotation
1030
500
100
2000
Llama 3.1 8B
Laver AMT.M
20, pown,
Liyer 2... P9d
Rotation
Rotation
To
200
200
380
200
Figure 6. Visualizations of Input Activations in Pythia and Llama Models before and after rotation.
We conjecture that several architectural differences between Pythia and Llama contribute to this discrepancy. First, Pythia does not utilize Gated Linear Units (GLU) in its MLP layers, a feature that is integral to Llama. Second, Pythia employs layer normalization (LayerNorm) instead of root mean square normalization (RMSNorm) which is used in Llama. Finally, Pythia adopts a parallel residual connection for attention and feed-forward layers, in contrast the sequential residual connection found in Llama.
F. A Addiionaa Experiments
We show additional experiment results for the resultant accuracies of fine-tuning the LLMs with different configurations of quantization parameters. Particularly, the results for Exp. 1 on the Pythia models can be found Table 7, and the results for Exp. on the Pythia models can be found in Table 9, We observe consistent improvements with the RoSTE algorithm.
16

---

RoSTE: An Efficient QA-SFT Approach for LLMs
Relatve Roduction Rata 4:Bit Activation Quantication Erro Across Layers After Rotation for Pyhia 18
Seduction Rate of Dynomic Range Across Layers After Rotation for Pyhio 18
Recatvv Reduction Rate Quant Tuuntsttioon Enor
Rato Dynamic Range
DKV up Proj Down. Proj
Key
Value
30
CKV_Up Up Proj
Froj
Layer Index
Layer Index
Layers
Dynamic Range Layers
Felashe Recuction hate Quantrarion Eiror
Dynamic
UptGate, Proj Down, Prol CNVV Proj
Reduction
Up_ Gate, Proj
Key
Key
Valle
Value
Figure 7. Effects of incoherence processing using rotation matrices on different layers of Pythia and Llama models using the pre pr-itrined
wio rotation Emvo rotation x 100%. Note that the reduction weights. (Left) Relative reduction rates quantization error, calculated as Erper rocation rate can be negative if the rotation is not beneficial. (Right) Reduction rate of dynamic ranges of the activations after rotation.
Additional comparisons to QLoRA (Dettmers al., 2023), LLM-QAT (Liu al., 2023) and DuQuant (Lin et al., 2024) provided in some setups, for instance, observed significant performance degradation DuQuant when the KV cache is quantized below 8 bits in Table 8.
17

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table 7. Additional experiments for Exp. -1 with different bit-width configurations and different model sizes.
Bit-width
Method
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-LSum
ROUGE (Avg.)
Pythia-IB
Base
22.40
5.73
17.35
17.59
15.77
FP16
SFT
32.80
11.84
25.49
25.50
23.91
RTN
6.05
0.06
5.21
5.67
4.25
GPTQ
10.16
0.30
8.41
8.84
6.93
LLM-QAT
19.71
4.03
15.82
15.83
13.85
QuaRot
16.57
1.66
13.61
13.70
11.39
W4A4KV4
SpinQuant
13.52
0,40
11.21
11.10
9.06
QLoRA (r = 64)
22.58
5.87
17.48
17.71
15.91
STE
31.03
10.44
24.01
24.01
22.37
RoSTE (ours)
31.80
11.03
24.71
24.71
23.07
RTN
24.19
6,94
19.29
19.13
17.39
GPTQ
29.77
9.81
23.38
23.50
21.52
LLM-QAT
29.54
9.60
23.08
23.08
21.33
W4A8KV4
QuaRot
30.14
9.24
22.97
23.03
21.35
SpinQuant
30.37
9.73
23.15
23.43
21.67
STE
32.44
11.48
25.24
25.24
23.60
RoSTE (ours)
32.67
11.61
25.37
25.37
23.76
Pythia-6.99 1ythia-6.9B
Base
28.81
9,45
22.29
22.91
20.87
FP16
SFT
33.69
12.60
26.27
26.31
24.72
RTN
7.42
0.06
6.53
6.56
5.14
GPTQ
8.16
0.08
7.06
7.60
5.73
LLM-QAT
18.73
3.71
15.31
15.01
13.19
W4A4KV4
QuaRot
11.70
0.23
8.52
9,39
7.46
SpinQuant
8.61
0.10
8.10
8.07
6.22
QLoRA (r 64)
27.92
8.91
21.97
22.00
20.20
STE
28.91
9.07
22.30
22.33
20.65
RoSTE (ours)
32.60
11.54
25.25
25.25
23.66
RTN
21.77
5.31
17.31
17.22
15.40
GPTQ
32.42
10.71
24.56
24.59
23.07
LLM-QAT
29.24
9.16
22.64
22.64
20.92
W4A8KV4
QuaRot
26.08
8.17
20.97
20.98
19.05
SpinQuant
31.69
10.70
24.69
24.68
22.94
STE
33.05
11.94
25.58
25.61
24.05
RoSTE (ours)
33.18
12.05
25.86
25.88
24.24
18

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table Additional experiments for Exp. with different bit-width configurations and diffcrent model sizes.
Bit-width
Method
ROUGE-1
ROUGE-2
ROUGE-L
ROUGE-LSum
ROUGE (Avg.)
Qwen2.5-0.5B
Base
23.79
6.63
18.46
18.56
16.86
BF16
SFT
32.58
11.93
25.53
25.55
23.90
RTN
10.04
0.37
8.15
8.34
6.73
GPTQ
12.53
0.92
10.08
10.50
8.51
QuaRot
9.94
0.57
8.18
8.38
6.67
SpinQuant
12.16
1.22
10.69
10.72
8.70
W4A4KV4
DuQuant
4.05
0.09
3.53
3,58
2.81
QLoRA (r 64)
24.88
7.18
19.28
19.43
17.69
STE
29.97
9.92
23.39
23.39
21.67
RoSTE (ours)
30.75
10.44
23.96
23.96
22.28
RTN
9.51
1.06
9.02
8.90
7.12
GPTQ
9.53
1.04
8.80
8.73
7.03
QuaRot
8.24
1,25
7.51
7.23
6.06
W4A8KV4
SpinQuant
9.10
1.11
8.31
8.12
6.66
DuQuant
3.91
0.06
3.56
3.53
2.77
STE
32.14
11.50
25.18
25.18
23.50
RoSTE (ours)
32.31
11.79
25.37
25.38
23.71
QuaRot
29.34
9.08
22.21
22.15
20.70
W4A4KV8
DuQuant
30.22
10.25
23.17
23.20
21.71
Qwen2.5-7B
Base
32.72
11.82
25.18
25.42
23.79
BF16
SFT
34.75
13.59
27.56
27.58
25.87
RTN
1.07
0.00
1.01
1.01
0.77
GPTQ
0.72
0.00
0.69
0.69
0.53
QuaRot
7.21
0.10
5.93
5.93
4.79
SpinQuant
6,87
0.29
5.97
6.12
4.81
W4A4KV4
DuQuant
0.00
0.00
0.00
0.00
0.00
QLoRA (r = 64)
32.22
11.41
24.75
24.89
23.32
STE
30.86
10.16
23.73
23.73
22.12
RoSTE (ours)
34.01
12.89
26.74
26.74
25.10
RTN
5.73
0.23
4.72
4.74
3.86
GPTQ
7.48
0.27
6.22
6.36
5.08
QuaRot
5.62
0.15
5.08
5.14
3.99
W4A8KV4
SpinQuant
0.64
0.30
5.64
5.81
4.54
DuQuant
0.24
0.00
0.24
0.24
0.18
STE
34.44
13.29
27.16
27.17
25.52
RoSTE (ours)
34.58
13.46
27.34
27.35
25.68
QuaRot
31.96
10.98
24.73
24.88
23.13
W4A4KV8
DuQuant
33.47
12.13
25.28
25.30
24.05
19

---

RoSTE: An Efficient QA-SFT Approach for LLMS
Table 9, Additional experiments for Exp. different bit-width configurations.
Bit-width
Method
TruthfulQA
MMLU-Pro
BigBenchHard
AGIEval
GSM8K
Math
Avg.
Base
28.51
19.57
62.26
30.16
56.86
18.20
35.93
FP16
SFT
31.82
33.07
65.67
34.86
64.89
22.66
42.16
RTN
23.01
0
0
17.03
1.03
6.85
GPTQ
25.34
0.02
2.55
16.48
2.05
7.74
QuaRot
27.66
21.53
47.69
29.05
37.91
6.90
28.46
W4A4KV4
SpinQuant
26.19
21.58
49.56
28.50
38.36
10.56
29.13
STE
26.68
9.13
24.58
17.63
22.82
1.90
17.14
RoSTE (ours)
26.44
25.12
52.00
30.11
44.50
11.94
31.69
RTN
28.76
19.29
42.96
27.75
28.66
7.84
25.88
GPTQ
28.52
25.54
46.38
29.26
48.60
0.02
29.72
QuaRot
27.42
26.78
53.79
32.01
49.20
12.72
33.65
W4A8KV4
SpinQuant
28.15
26.66
55.74
32.01
52.16
15.38
35.02
STE
29.62
24.09
54.62
29.44
52.62
4.08
32.41
RoSTE (ours)
30.84
28.23
59.25
34.03
56.94
16.88
37.70
G. Statistics of Training Cost
Table 10 presents additional statistics for the training costs when using RoSTE and other benchmark algorithms. We observe that while achieving better performance, RoSTE requires only similar amount of computation costs compared to benchmarked algorithms.
Table 10. Training time and peak GPU memory consumption for obtaining a quantized fine-tuned Qwen2.5 7B from its pre-trained checkpoint server of 8 A100.
Bit-width
Method
Training Time (hours)
Peak Memory (GB)
SFT LoRA (r = 64)
2.1 0,55
300
FP16
173
SFT > GPTQ
2.1 - 0
300 - 0
SFT + QuaRot
2.1 +
300 - 0
SFT - SpinQuant
2.1 + 1.3
300 + 263
W4A4KV4
QLoRA (r = 64) STE RoSTE
0.83
98
2.4
317
2.8
318
20


Reviews:
[{"review_id": "LN1l7e3bar", "reviewer": "Reviewer_W2sL", "summary": "This paper introduces RoSTE, a method for quantization-aware SFT of LLMs. RoSTE aims to jointly optimize model weights and rotation matrices during fine-tuning, enabling efficient 4-bit quantization of weights, activations, and KV caches in a single training phase. This integration contrasts with the approaches that apply quantization after fine-tuning models, which can degrade performance. The core idea of RoSTE is to leverage a bi-level optimization approach: (1) updating model weights using a rotation-aware STE, and (2) selecting rotation matrices from a candidate set to minimize a quantization error surrogate loss. The authors show the effectiveness of their proposal by providing theoretical analysis and empirical evidences.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Memory Efficiency: ...\n\nWe agree partially with your points. While RoSTE maybe less memory efficient than methods such as QLoRA, we observe that RoSTE has a similar training cost in compute and memory usage as full-param SFT (cf. Table B in response to Rev. mQhc). The overhead with adaptive rotation is insignificant due to the small outer iteration no.\n\nOur goal lies in achieving optimal performance at *deployment* by tuning fully quantized models with inference speedup and downstream task accuracy. As a comparison, note that Table 10 of [arxiv/2404.14047] shows models such as QLoRA suffer from degraded inference speed due to extra unquantized LoRA layers.\n\nFrom a practical standpoint, our experiments on <8B models is relevant to on-device deployment where LLMs of such scales are common. E.g., the fine tuned 4-bit Pythia 1B model by RoSTE has greatly improved performance over PTQ on the full-precision base model. This will be useful for applications that prioritize inference latency. \n\nFor 70B+ models, RoSTE is still feasible and can potentially improve model quality. First, a rough estimate suggests that using RoSTE to training Llama3 70B under Adam require ~900GB of GPU memory (see [here](https://community.ibm.com/community/user/cloud/blogs/arindam-dasgupta/2024/09/18/calculating-gpu-requirements-for-efficient-llama-3)), which is implementable on a 8xH200 cluster. Second, recent studies such as [arxiv/2407.11062] showed that using QAT on 70B+ models with 1xA100 is possible with a simple block processing technique. Such technique can be straightforwardly combined with RoSTE.\n\n> Limited Learning Space: ...\n\nThis is a good point. Searching for optimal rotation like SpinQuant may enhance the model quality, yet our design to limit the learning space to $\\\\{I, H\\\\}$ for each layer is grounded on the tradeoff of complexity vs performance:\n\n- *Quantization Error*: As shown in Fig 3, using Hadamard matrices suffices to reduce the quantization error and optimize the lower-level objective in (11);\n- *Training*: extending the search space to all rotation matrices leads (11) to become a mix of non-smooth and manifold optimization problems. Handling it requires substantially higher training complexity.\n- *Inference*: Hadamard matrices are efficient for inference [Ashkboos et al., 2024b], where it supports hardware acceleration and has low memory footprint due to its integer structure. In contrast, optimal rotation requires real-valued rotation matrices that impose significant overheads during inference.\n\nOur design choice is supported by results in Table 2 & 3: RoSTE outperforms learnt rotations (SpinQuant) & no-rotation (STE).\n\n> Empirical Evaluation: ...\n\nWe extended our experiment results to QLoRA in the above Table A.1, A.2, A.3, A.4 in the response to Rev. c8mS. Notice that QLoRA under-performs RoSTE in terms of quantized model accuracy on downstream tasks, potentially due to the limited learning space of low-rank adaptation.\n\n> Missing analysis of memory usage and training time.\n\nWe included details on training time and training memory usage in Table B in the response to Rev. mQhc. Notice that RoSTE has a similar training cost as STE.\n\n> Scalability to larger models (70B+) is not demonstrated or discussed.\n\nAs mentioned in the first response, scaling RoSTE to 70B+ models is feasible through either \n- Scaling up computation capability.\n- Applying block processing to RoSTE. \n\nWhile we do not have the computation resource nor time to experiment with 70B+ models at the moment, our [codebase](https://anonymous.4open.science/r/RoSTE) is available for the community to run further experiments.\n\n> Some experiments use outdated ... \n> Broader eval needed ...\n\nWe extended our experiment results to Qwen 2.5 models (released on Sep 25 2024). By Table A in the above, we illustrate the accuracy of 4-bit quantized Qwen 2.5 7B models. RoSTE remains to produce the top performance quantized model among the baselines. Now, our evaluation set covers three families of LLMs (Pythia, Qwen, Llama). Our coverage is broader than existing literature such as QuaRot and SpinQuant, where only Llama models were evaluated.\n\n> Novelty is limited ...\n\nWe respectfully disagree. From a methodology perspective, our work is the first to employ adaptive rotation with SFT to improve the accuracy of quantized models. Such design is well grounded in theory from an optimization perspective.\n\nBesides innovation in methodology, we observe a research gap where prior works mainly considered pre-trained zero-shot evaluation, while the performance of quantized fine-tuned models remains unexplored. Our results open the door to a new benchmark for approaches that directly optimize the SFT objective on quantized models. We further illustrate the existing research gap in [figure](https://i.imgur.com/1kVfgSq.png) on accuracy-vs-training-time tradeoffs. Note that RoSTE achieves better accuracy than existing methods on SFT benchmark with a moderate training overhead.", "claims_and_evidence": "While the idea of jointly optimizing rotation matrices and quantized weights is compelling and likely beneficial for the quantized SFT model quality, I see several limitations in the current approach.\n\n1. Memory Efficiency: The proposed method relies on QAT, which is memory-intensive and difficult to scale to large models (e.g., 70B+). This contrasts with qLoRA-based approaches that are significantly more memory-efficient by freezing most of the model and only training low-rank adapters. RoSTE adds further overhead by optimizing rotation matrices, increasing the computational overhead.\n2. Limited Learning Space: The rotation search is restricted to identity and Hadamard matrices, which is a highly constrained design choice. Prior work (e.g., SpinQuant) has shown that Hadamard rotations are not optimal, and more flexible, learned rotations can yield better results. From the limited ablation study, it seems like learning the rotation matrix does not contribute much towards the model quality.\n3. Empirical Evaluation: The paper claims state-of-the-art performance, but the empirical evidence does not fully support this. Most comparisons are against post-training quantization (PTQ) methods, while stronger baselines such as qLoRA or other parameter-efficient quantization-aware fine-tuning approaches are omitted.", "methods_and_evaluation_criteria": "The motivation to jointly optimize quantized parameters and rotation matrices is well-founded. However, the practicality of the proposed approach remains unclear--in particular, whether it can scale to larger models. Some experiments are conducted on relatively old models (Pythia) that are considered undertrained by today’s standards, and the baselines are mostly limited to post-training quantization methods, omitting more competitive approaches such as qLoRA-based fine-tuning.", "theoretical_claims": "The theoretical claims seem correct.", "experimental_designs_or_analyses": "Some potential issues with the experiments:\n1. Missing analysis of memory usage and training time.\n2. Scalability to larger models (70B+) is not demonstrated or discussed.\n3. No comparison with qLoRA or other PEFT baselines.\n4. Some experiments use outdated, undertrained models; newer, more competitive models should be considered more.", "relation_to_broader_scientific_literature": "RoSTE builds upon previous approaches (rotation-based PTQ, QAT, and STE) and proposes to combine them to achieve better quantized SFT fine-tuning performance. While the idea of combining QAT with rotation is new in this context, the method largely builds on known techniques like STE, Hadamard rotations, and existing QAT frameworks, extending them to the supervised fine-tuning setting. However, the paper does not compare against more recent memory-efficient fine-tuning methods like qLoRA, which have gained traction in the literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Some weaknesses of the paper are:\n1. Novelty is limited--the paper combines existing ideas (QAT, STE, rotation-based quantization), but does not introduce fundamentally new techniques.\n2. Scalability and efficiency are not thoroughly demonstrated, especially for large models.\n3. Empirical results are not fully convincing — baseline comparisons are limited, and stronger methods like qLoRA are missing.\n4. While the integration is well-motivated, the overall significance and impact remain unclear without broader evaluation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3PwP3G413r", "reviewer": "Reviewer_c8mS", "summary": "RoSTE introduces a novel Quantization-Aware Supervised Fine-Tuning (QA-SFT) approach for large language models (LLMs), addressing the inefficiencies of traditional post-training quantization (PTQ) and the high computational cost of quantization-aware training (QAT). The proposed Rotated Straight-Through Estimator (RoSTE) integrates rotation-based quantization into QA-SFT, leveraging an adaptive rotation strategy using Walsh-Hadamard matrices to mitigate outliers in activations and weights. Theoretical analysis demonstrates that RoSTE effectively reduces quantization error, leading to improved performance over existing PTQ and QAT methods. Experimental results on Pythia and Llama models confirm that RoSTE achieves superior accuracy in 4-bit quantized models while maintaining computational efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Table A.1: Results on Pythia 1B model.*\n| Bit-width  | Method| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n| **FP16**| Base| 22.40| 5.73| 17.35| 17.59| 15.77|\n|| SFT| 32.80| 11.84| 25.49| 25.50| 23.91|\n| **W4A4KV4**| **RoSTE** | **31.80** | **11.03** | **24.71** | **24.71** | **23.07** |\n|$r = 64$ | QLoRA| 22.58 | 5.87 | 17.48 | 17.71 | 15.91|\n\n*Table A.2: Results on Pythia 6.9B model.*\n| Bit-width| Method| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n|**FP16**| Base| 28.81 | 9.45 | 22.29 | 22.91 | 20.87 |\n|| SFT| 33.69 | 12.60 | 26.27 | 26.31 | 24.72 |\n|**W4A4KV4**| **RoSTE** | **32.60** | **11.54** | **25.22** | **25.25** | **23.66** |\n|$r = 64$ | QLoRA  | 27.92 | 8.91 | 21.97 | 22.00 | 20.20 |\n\n*Table A.3: Results on Qwen 2.5 0.5B model.*\n|Bit-width|Method| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n| **BF16**| Base| 23.79| 6.63| 18.46| 18.56| 16.86|\n|| SFT| 32.58| 11.93   | 25.53| 25.55| 23.90|\n| **W4A4KV4**| RTN| 10.04   | 0.37    | 8.15| 8.34| 6.73|\n|| GPTQ| 12.53| 0.92    | 10.08| 10.50| 8.51|\n|| QuaRot| 9.94| 0.57| 8.18| 8.38| 6.67|\n|| SpinQuant| 12.16| 1.22| 10.69| 10.72| 8.70|\n|$r = 64$| QLoRA | 24.88 | 7.18 | 19.28 | 19.43 | 17.69 |\n|| STE| 29.97| 9.92| 23.39   | 23.39| 21.67|\n|| **RoSTE** | **30.75** | **10.44** | **23.96** | **23.96** | **22.28** |\n\n*Table A.4: Results on Qwen 2.5 7B model.*\n| Bit-width | Method | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n| **BF16** |Base| 32.72|11.82| 25.18 |25.42|23.79|\n|| SFT| 34.75|13.59|27.56|27.58| 25.87 |\n| **W4A4KV4** |RTN|1.07|0.00| 1.01|1.01|0.77|\n||GPTQ| 0.72|0.00|0.69|0.69| 0.53|\n||QuaRot|7.21|0.10|5.93|5.93| 4.79|\n||SpinQuant|6.87|0.29|5.97|6.12|4.81|\n|$r = 64$|QLoRA|32.22|11.41|24.75|24.89|23.32|\n||STE|30.86|10.16|23.73|23.73|22.12|\n||**RoSTE**|**34.01** | **12.89** | **26.74** | **26.74** | **25.10** |\n\n> The authors should either provide stronger empirical evidence to justify their claim or acknowledge the limitations of their theoretical analysis in real-world LLM scenarios.\n\nWe agree with your points. However, we point out that the analysis in Sec 4 aims at giving *theoretical insights* for the design of RoSTE algo, rather than ensuring convergence for STE training on LLMs, which is an open problem to our best knowledge as modern LLM involves complex architecture. Despite the limitations raised, we retain essential elements of RoSTE with a linear layer featuring a rotation matrix in activations and weights, and the derived results motivated RoSTE to minimize quantization error in the lower level objective (11). The derived bound echoes with our Table 2 and Fig 3. We will emphasize these limitations in the revision.\n\n> primarily evaluates RoSTE on Pythia (1B/6.9B) and Llama (8B) ...\n\nWe extended our experiments to the latest Qwen 2.5 models - see Table A.3, A.4 on accuracy of 4-bit quantized 0.5B / 7B models. RoSTE delivers the top quantized model vs the baselines.\n\n> Given that efficiency is a primary motivation...  quantify the trade-offs between accuracy improvements and the increased computational cost. \n\nWe agree, but would clarify that there are two aspects of efficiency wrt computational cost - inference and training, which we will detail below:\n\n- *Inference efficiency*: Since our model's structure (cf. Fig 4) is equivalent to QuaRot which implements weight and activation quantization with Hadamard rotation, the inference time comparison can be referred to [Ashkboos et al., 2024b] for an extensive measurement on the inference speedup and inference memory consumption. E.g., Fig 7 therein shows that 4-bit quantized models with Hadamard rotation has 4x speedup compared to full-precision models, their Fig 4 also shows significant advantages on time-to-first-token and peak memory saving.\n- *Training efficiency*: As we restricted the search space of rotation matrices to Hadamard matrix, the theoretical training complexity should be on par with full-param SFT. This is confirmed in the additional Table B of the response to Reviewer mQhc. We acknowledge that the training complexity can still be higher than parameter-efficient methods such as QLoRA. For a comprehensive comparison, we refer to [figure](https://i.imgur.com/1kVfgSq.png) on the accuracy-vs-training-time tradeoff for SOTA methods. Observe that RoSTE achieves the best accuracy at a moderate cost in training complexity.\n\nWe will include the above references in the revision.\n\n> the paper does not provide detailed benchmarks on key efficiency metrics ...\n\nWe apologize for the misunderstanding caused by ambiguous wording of the title. We have referred to \"efficiency\" in the context of deployment for trained quantized model, where RoSTE delivers models with low latency inference and memory consumption. Nevertheless, in the training phase, RoSTE is on par with full-param SFT as seen in Table B in the response to Rev mQhc.", "claims_and_evidence": "The claim that *\"Applying the theorem shows that given \\( R \\), the resultant prediction error of the intermediate model \\( w_T \\) will be bounded by \\( O\\left(\\sum_{s=0}^{T}(1-\\mu)^{T - s}\\mathbb{E}[\\|Q_w(Rw_s) - Rw_s\\|_G^2]\\right) \\),\"* seems problematic for a few reasons.\n\nThe theorem is derived under a highly simplified setting that assumes a quadratic loss and a linear model. However, modern Transformer-based large language models (LLMs) have complex architectures involving non-linearity, residual connections, and multi-head attention mechanisms. The direct application of this theorem to such models is questionable, as it does not account for the intricate dependencies and interactions within deep networks. Moreover, the proof relies on the interpolation assumption (Assumption 4.2), which states that for any rotation matrix \\( R \\), there exists an interpolating weight \\( w^*_R \\) such that the model can perfectly predict the target. In reality, this assumption is unlikely to hold in large-scale LLMs trained on diverse datasets. At last, the proposed rotation matrix optimization method, which relies on randomized Walsh-Hadamard transformations, lacks rigorous justification regarding its effectiveness in minimizing quantization error in a structured and optimal way. \n\nThe claim regarding the prediction error bound is not convincingly supported by theoretical or experimental evidence. The analysis is based on unrealistic assumptions, and the experimental validation does not directly confirm the theoretical findings. Therefore, the authors should either provide stronger empirical evidence to justify their claim or acknowledge the limitations of their theoretical analysis in real-world LLM scenarios.", "methods_and_evaluation_criteria": "The paper primarily evaluates RoSTE on Pythia (1B/6.9B) and Llama (8B) models, which, while relevant, do not provide sufficient diversity in terms of model architectures and scales. Given that LLM quantization techniques should generalize across different model families (e.g., GPT, Mistral, Falcon, or transformer variants with different architectural choices), the narrow selection raises concerns about the method’s broader applicability.", "theoretical_claims": "I have already addressed the issue of the claims relying on overly strong assumptions.", "experimental_designs_or_analyses": "The paper does not provide sufficient analysis of the computational overhead introduced by RoSTE. The method involves additional operations such as rotation transformations using Walsh-Hadamard matrices, which may introduce extra latency during training and inference. However, the paper does not quantify the trade-offs between accuracy improvements and the increased computational cost. Given that efficiency is a primary motivation for quantization, failing to report potential slowdowns or extra memory requirements makes it difficult to assess whether RoSTE is truly practical for real-world deployment.\n\nWhile RoSTE is positioned as an efficient quantization-aware fine-tuning approach, the paper does not provide detailed benchmarks on key efficiency metrics, such as GPU memory consumption, inference latency, or speedup comparisons against standard PTQ/QAT baselines. Since quantization is primarily used to reduce computational and memory overhead, these factors should be explicitly reported. Without these insights, it is unclear whether RoSTE is actually more practical than existing quantization methods when deployed on hardware-constrained environments.", "relation_to_broader_scientific_literature": "The key contributions of this paper are well-grounded in the broader scientific literature on QAT and rotation-based quantization for LLMs. One of the most significant contributions of this work is its theoretical analysis, which establishes a direct connection between quantization error (particularly from weight and activation quantization) and the final prediction error of fine-tuned models. While previous works have studied the impact of quantization on model accuracy (e.g., GPTQ [Frantar et al., 2022], LLM-QAT [Liu et al., 2023]), this paper goes further by providing a formal mathematical framework to quantify this relationship. This insight contributes to a deeper understanding of how fine-tuned models behave under aggressive quantization, particularly in the 4-bit regime.\n\nAnother intriguing aspect of the paper is its approach to constructing rotation matrices using a combination of Hadamard and identity matrices. Rotation-based quantization methods have been explored in prior work (e.g., QuaRot [Ashkboos et al., 2024], SpinQuant [Liu et al., 2024]), but most of these methods focus on static rotation applied during PTQ. In contrast, this paper integrates adaptive rotation selection into the QA-SFT process, which is a novel extension. The idea of using Hadamard transformations to selectively apply rotation layer-by-layer is particularly interesting because it balances the benefits of reducing outliers while maintaining computational efficiency.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper presents an original and insightful approach to quantization-aware fine-tuning by integrating rotation-based quantization with theoretical justification. The idea of linking quantization error and fine-tuned model error is a notable contribution, as it provides a more structured understanding of quantization effects in LLMs. Additionally, the use of Hadamard and identity matrix combinations for adaptive rotation selection is a novel and efficient way to address activation outliers, making the approach both practical and theoretically grounded.\n\nOne notable weakness is the lack of clarity and organization in the writing. The paper feels somewhat dense and unpolished, with certain sections being overly wordy and difficult to follow. This can make it challenging for readers to extract key insights efficiently. In particular, some theoretical derivations and algorithmic descriptions could be streamlined for better readability. Furthermore, while the work presents strong theoretical contributions, the practical aspects of computational overhead and real-world deployment feasibility remain underexplored.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Please refer to the aforementioned comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "whtstOwsqY", "reviewer": "Reviewer_6QQT", "summary": "This paper aims to combine quantization-aware SFT and rotation strategy. This work is the first to leverage rotation-based quantization in QA-SFT.\n\nThe authors propose a bilevel optimization formulation – upper level subproblem for optimizing weight matrices and lower level subproblem for selecting rotation matrix.  The theoretical analysis on the benefits of rotation-enabled quantization is conducted.\n\nThe proposed method improves the performance of quantize models on downstream tasks, outperforming several baseline methods.\n\n## update after rebuttal\nThe paper presents a strong theoretical analysis, especially in explaining how the proposed RoSTE method helps reduce quantization error. Meanwhile, there are notable concerns regarding its practical applicability, particularly around the resource requirements during fine-tuning. As discussed between other reviewer and the authors, the method seems to demand a substantial number of GPUs, yet the main paper lacks a clear and thorough analysis of this computational overhead.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for acknowledging the strength of our approach. Below we summarize our response to your concerns.\n\n> it is worth noting that the theoretical claims depend on certain assumptions (such as the interpolation condition and specific properties of the Gram matrix) that could limit their applicability in some scenarios.\n\nWe agree. However, we point out that the analysis in Sec 4 aims only at giving *theoretical insights* for the design of RoSTE algo, rather than ensuring convergence for STE training on LLMs, which is an open problem as modern LLM involves complex architecture. Despite the added assumptions, we retain essential elements of RoSTE training with linear layer featuring a rotation matrix in activations and weights, and the derived results gives motivation for RoSTE to use quantization error in the lower level objective of (11). To our best knowledge, this is the first analysis that provides insights on the convergence of STE training with rotation. Our derived bound echoes with the observation in Table 2 and Figure 3, where RoSTE empirically outperforms STE without rotation on quantization error and downstream task accuracy; also see the new experiments in Table A.1, A.2, A.3, A.4 in the response to Rev. c8mS.\n\n> Can you provide fine-tuning results of smaller models using Tulu-3 SFT dataset?\n\nSince Tulu 3 is originally proposed for Llama 3.1 base models, extending the application of Tulu-3 SFT dataset to other families of base models would require an extensive fine-tuning on the hyperparameters. Due to limited time, we cannot produce additional results on Tulu-3 dataset. We will include additional results in the revision.\n\n> It seems that there are far more cases where a model that has already been fine-tuned needs to be quantized. Can the QA-SFT technique be practically applied in various situations?\n\nIt is correct that there are many unquantized but fine-tuned model that are publicly available. The proposed RoSTE can be applied on them through either (i) knowledge distillation, e.g., LLM-QAT in [Liu et al., 2023], or (ii) treating them as initialization using the original fine-tuning dataset. Note that all our numerical results indicate that existing PTQ methods may fail to maintain the fine-tuned accuracy after quantization. Our work thus demonstrate to practitioners that a good QA-SFT strategy (e.g., RoSTE) can greatly improve the quality of quantized & fine-tuned model.", "claims_and_evidence": "The paper demonstrates the effectiveness of its contributions through various experimental results and in-depth theoretical analysis. However, the theoretical analysis relies on several assumptions, such as the interpolation condition and properties of the Gram matrix.", "methods_and_evaluation_criteria": "The paper dedicates significant effort to justify the superiority of its method. It presents extensive evaluations, showing fine-tuning performance using RoSTE on two models across a variety of tasks. In particular, for LLaMA, the experiments cover a more general fine-tuning case which covers a wide range of and tasks, while for Pythia, multiple metrics are reported when fine-tuning on a summarization dataset. This results in a wealth of experimental evidence supporting the method's effectiveness.", "theoretical_claims": "The proofs generally follow established techniques and seem to be correct. However, I did not rigorously verify every detail, and it is worth noting that the theoretical claims depend on certain assumptions (such as the interpolation condition and specific properties of the Gram matrix) that could limit their applicability in some scenarios.", "experimental_designs_or_analyses": "The experiments conducted on multiple models for a single task (summarization) using both traditional fine-tuning and modern LLM fine-tuning approaches are highly commendable. Moreover, the fact that RoSTE maintains a lower quantization error compared to STE is a notable advantage of the method.", "relation_to_broader_scientific_literature": "The paper builds on and is influenced by prior work on rotation-based post-training quantization (PTQ) methods and research on the straight-through estimator (STE). In particular, it takes cues from earlier studies that used rotation techniques to mitigate the impact of outlier activations and improve quantization accuracy, and from foundational works on STE that addressed gradient approximation issues during quantization-aware training.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strength**: Superior performance compared to baselines\n\n**Weakness**: Performance variation on models (Pythia and LLaMA)", "other_comments_or_suggestions": "Fine-tuning on the Tulu-3 SFT-mixture dataset closely resembles the fine-tuning approaches commonly used with modern LLMs. It would be beneficial to see experimental results on models other than LLaMA or on smaller models, as this could further validate the generalizability and efficiency of the method across different model scales and architectures.", "questions_for_authors": "- Can you provide fine-tuning results of smaller models using Tulu-3 SFT dataset?\n\n- It seems that there are far more cases where a model that has already been fine-tuned needs to be quantized. Can the QA-SFT technique be practically applied in various situations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JITLVLMmzU", "reviewer": "Reviewer_mQhc", "summary": "This paper introduces RoSTE, a method that combines rotation-based transformation with Quantization-Aware Training to improve the efficiency of the SFT process.\n\n## update after rebuttal\n\nThanks for providing the additional results. I will raise my score accordingly. Please be sure to include these experimental results and analysis during the rebuttal period in your next version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Table B: Training Time and Training Memory Consumption. Our server of 8 $\\times$ A100 has a total GPU memory of 320GB.*\n| Model| Method| Total Training Time (h) | Peak Memory (GB) |\n|-|-|-|-|\n| Qwen 2.5 7B  | SFT| 2.1| 300|\n|| GPTQ|  2.1| 0|\n|| QuaRot| 2.1 | 0|\n|| SpinQuant| 3.4| 263|\n|| LoRA| 0.55| 173|\n|| QLoRA| 0.83| 98|\n|| STE| 2.4| 317|\n|| RoSTE| 2.8| 318|\n\n*Table C.1: Extended Experiment Results on Qwen 2.5 0.5B model.*\n|Bit-width| Method| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n| **BF16**   | Base| 23.79| 6.63| 18.46| 18.56| 16.86|\n|| SFT| 32.58| 11.93| 25.53| 25.55| 23.90|\n|**W4A4KV4**| QuaRot| 9.94| 0.57| 8.18| 8.38| 6.67|\n|| DuQuant |  4.05 | 0.09 | 3.53 | 3.58 | 2.81 |\n|| **RoSTE** | **30.75** | **10.44** | **23.96** | **23.96** | **22.28** |\n|**W4A8KV4** | QuaRot | 8.24 | 1.25 | 7.51 | 7.23 | 6.06 |\n|| DuQuant | 3.91 | 0.06 | 3.56 | 3.53 | 2.77  |\n|**W4A4KV8** | QuaRot | 29.34 | 9.08 | 22.21 | 22.15 | 20.70 |\n|| DuQuant | 30.22 | 10.25 | 23.17 | 23.20 | 21.71 |\n\n*Table C.2: Extended Experiment Results on Qwen 2.5 7B model.*\n|Bit-width| Method| ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum | ROUGE (Avg.) |\n|-|-|-|-|-|-|-|\n|**BF16** | Base| 32.72| 11.82 | 25.18 | 25.42 | 23.79 |\n|| SFT| 34.75| 13.59 | 27.56 | 27.58 | 25.87 |\n|**W4A4KV4**| QuaRot| 7.21 | 0.10 | 5.93 | 5.93 | 4.79|\n|| DuQuant |  0.00 | 0.00 | 0.00 | 0.00 | 0.00 |\n|| **RoSTE** | **34.01** | **12.89** | **26.74** | **26.74** | **25.10** |\n| **W4A8KV4** | QuaRot | 5.62 | 0.15 | 5.08 | 5.14 | 3.99 |\n|| DuQuant | 0.24 | 0.00 | 0.24 | 0.24 | 0.18 |\n| **W4A4KV8** | QuaRot | 31.96 | 10.98 | 24.73 | 24.88 | 23.13 |\n|| DuQuant | 33.47 |12.13 |25.28 | 25.30 | 24.05 |\n\n\n> RoSTE combines activation rotation with SFT but does not include comparisons or discussions with the following rotation-based quantization methods ...\n\nWe included DuQuant in the new experiments on Qwen 2.5 models in the above Table C.1, C.2. Despite our efforts spent on hyperparameter tuning (including learnable weight clipping, activation clipping, epoch, block size), we found that DuQuant suffers from a huge performance degradation in W4A4KV4 and W4A8KV4 setup. \n\n> I observed that for Llama3.1 8B, on certain evaluation benchmarks (e.g., TruthfulQA), RoSTE performs worse than QuaRot. I recommend providing analysis and insights into the reasons behind this discrepancy.\n\nWe suspect that this discrepancy is due to that the two approaches use different objectives: QuaRot is not adapted to the SFT loss; RoSTE directly performs training on the SFT loss and SFT dataset. It is expected that QuaRot's performance distribution will shift further away from the full-precision SFT model. We also remark that none of the quantized model can outperform the full-prec models in the TruthfulQA benchmark.\n\n> Could you provide more analysis on the training cost of RoSTE compared to traditional PTQ methods?\n\nDetails on the training cost can be found in Table B. Moreover, we illustrate in [figure](https://i.imgur.com/1kVfgSq.png) on the training cost-accuracy trade-offs of RoSTE and other SOTA methods.\n\n> Why do the PTQ baselines perform poorly in Experiment 1? Is this due to the Pythia 6.9 model or the SFT dataset?\n\nWe suspect that the poor performance is due to the fine tuned Pythia models.\n\n- In the original papers, the PTQ baselines (QuaRot, SpinQuant, DuQuant) are only proposed for evaluation on pre-trained Llama models. As such, we speculate that an adaptive rotation configuration is necessary to achieve good performance for LLMs other than the Llama family.\n- Most of these methods were not evaluated on fine-tuned benchmarks. We speculate that fine tuned models may introduce certain features that are not addressed in existing PTQ methods.\n\n> Please include measurements of inference speedup and memory usage.\n\nSince our model's structure (cf. Fig 4) is equivalent to QuaRot, the inference time comparison can be referred to [Ashkboos et al., 2024b] for an extensive measurement on the inference speedup and inference memory consumption. E.g., Fig 7 therein shows that 4-bit quantized models with Hadamard rotation has 4x speedup compared to full-precision models, Fig 4 also shows significant advantages on time-to-first-token and peak memory saving. We will include these references in the revision.", "claims_and_evidence": "The claims are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluations include two different settings for Pythia and Llama 3.1, which effectively demonstrate the method's efficacy across different architectures.", "theoretical_claims": "I have reviewed the proof section in the Appendix, and the theoretical foundation of the method is solid.", "experimental_designs_or_analyses": "The paper provides a detailed comparison with state-of-the-art PTQ methods and simple Straight-Through Estimator training methods. Although the results show improvements, there remains a significant gap compared to BF16 performance.", "relation_to_broader_scientific_literature": "The paper follows the field of rotation-based quantization methods, such as QuaRot and SpinQuant. However, it lacks a comparison with some important baselines that would further strengthen the evaluation.", "essential_references_not_discussed": "RoSTE combines activation rotation with SFT but does not include comparisons or discussions with the following rotation-based quantization methods:\n\n1. DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs, NeurIPS 2024.\n2. OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting, ICLR 2025.\n\nGiven the published date of these methods, I believe a comparison with DuQuant is necessary, and OstQuant could be a useful additional comparison.", "other_strengths_and_weaknesses": "I observed that for Llama3.1 8B, on certain evaluation benchmarks (e.g., TruthfulQA), RoSTE performs worse than QuaRot. I recommend providing analysis and insights into the reasons behind this discrepancy.", "other_comments_or_suggestions": "I found the analysis and figures in the Appendix to be quite insightful, especially Figure 4, Figure 5, and Section F. It would be beneficial to incorporate some of these analyses into the main body of the paper for better accessibility.", "questions_for_authors": "1. Could you include comparisons with the missing baselines mentioned above?\n2. Could you provide more analysis on the training cost of RoSTE compared to traditional PTQ methods?\n3. Why do the PTQ baselines perform poorly in Experiment 1? Is this due to the Pythia 6.9 model or the SFT dataset?\n4. Please include measurements of inference speedup and memory usage.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new quantization-aware supervised fine-tuning method with efficiency by leveraging adaptive rotation strategy to reduce activation outliers for large langage models. 

This paper has mixed scores such as three acceptance and one weak reject. 

The main concerns were speedup effects, memory usages, and large requirement of GPU resources. 

During the rebuttal, some major concerns seemd to be addressed, and GPU requirement issue still remains.

For clear decision, AC discussed with the reviewers. AC also agree that 2 nodes of H100 is quite high requirement compared to PEFT appraoch such as QLoRA. However, it is natural that PEFT's GPU requirement is much smaller than that of SFT. For SFT, 2 nodes are typical and practical cases in real-world LLM SFT applications. 

Therefore, AC recommends accpeting this paper and asks the authors to clarify this issue.
Decision: Accept (poster)