OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
Authors: Xing Hu
Yuan Cheng
Dawei Yang
Zhixuan Chen
Zukang Xu
JiangyongYu
XUCHEN
Zhihang Yuan
Zhe jiang
Sifan Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Quantization
Primary area: foundation or frontier models, including LLMs

Abstract: Post-training quantization (PTQ) has emerged as a widely adopted technique for compressing and accelerating Large Language Models (LLMs).
The major challenge in LLM quantization is that uneven and heavy-tailed data distributions can expand the quantization range, thereby reducing bit precision for most values.
Recent methods attempt to eliminate outliers and balance inter-channel differences by employing linear transformations; however, they remain  heuristic and are often overlook optimizing the data distribution across the entire quantization space.
In this paper, we introduce Quantization Space Utilization Rate (QSUR), a novel metric that effectively assesses the quantizability of transformed data by measuring the space utilization of the data in the quantization space. We complement QSUR with mathematical derivations that examine the effects and limitations of various transformations, guiding our development of Orthogonal and Scaling Transformation-based Quantization (OSTQuant). OSTQuant employs a learnable equivalent transformation, consisting of an orthogonal transformation and a scaling transformation, to optimize the distributions of weights and activations across the entire quantization space. Futhermore, we propose the KL-Top loss function, designed to mitigate noise during optimization while retaining richer semantic information within the limited calibration data imposed by PTQ.
OSTQuant outperforms existing work on various LLMs and benchmarks. In the W4-only setting, it retains 99.5\% of the floating-point accuracy. In the more challenging W4A4KV4 configuration, OSTQuant reduces the performance gap by 32\% on the LLaMA-3-8B model compared to state-of-the-art methods. Code will be available.

Published as a conference paper at ICLR 2025
OSTQUANT: REFINING LARGE LANGUAGE MODEL QUANTIZATION WITH ORTHOGONAL AND SCALING TRANSFORMATIONS FOR BETTER DISTRIBUTION FIT- TING
Xing Hu**, Yuan Chengl.2*1 Dawei Yang Zukang Xu Sifan Zhou Jiangyong Yu² Chen Xu², Zhixuan Chen', Zhe Jiang', Zhihang Yuan" x
Houmo AI NNajjing University Southeast University xing. hu@houmo. ai, yuancheng@smail .nj.. edu. cn dawei, www..younggoooo..mmmmm ai, zhihang. yuan@houmo. ai
ABSTRACT
Post-training quantization (PTQ) has emerged widely adopted technique for compressing and accelerating Large Language Models (LLMs). The major chal- lenge in LLM quantization is that uneven and heavy-tailed data distributions can expand the quantization range, thereby reducing bit precision for most values. Recent methods attempt eliminate outliers and balance inter-channel differences by employing linear transformations; however, they remain heuristic and are often overlook optimizing the data distríbution across the entire quantization space, this paper, introduce Quantization Space Utilization Rate (QSUR), novel met- ric that effectively assesses the quantizability of transformed data by measuring the space utilization of the data in the quantization space. We complement QSUR with mathematical derivations that examine the effects and limitations of various transfor- mations, guiding our development of Orthogonal and Scaling Transformation-bas Quantization (OSTQuant). OSTQuant employs a learnable equivalent transforma- tion, consisting of an orthogonal transformation and scaling transformation, optimize the distributions of weights and activations across the entire quantization space. Futhermore, we propose the KL-Top loss function, designed to mitigate noise during optimization while retaining richer semantic information within the limited calibration data imposed by PTQ. OSTQuant outperforms existing work on various LLMs and benchmarks. In the W4-only setting. it retains 99.5% of the floatin-pooin accuracy. In the more challenging W4A4KV4 configuration, OS- TQuant reduces the performance gap by 32% on the LLaMA-3-8B model compared to state-of-the-a methods. http://giwbb.ooooo/oooreeaa//a/p
INTRODUCTION
Large language models (LLMs) (Dettmers al. 2022 Touvron et al. 2023a b) have demonstrated exceptional performance across variety of tasks, increasingly integrating into daily life and playing critical roles various areas (Achiam al.) 2023; Chen al. 2024). Nevertheless, the substantial memory and computational demands pose significant deployment challenges, limiting their practical applicability not only on edge devices with constrained resources but also on cloud servers equipped with powerful GPU devices.
Post-training quantization (PTQ) has emerged as a widely adopted technique for compressing and accelcrating LLMs. During quantization, uneven and heteroscedastic data, shown in [(a) pose significant challenges, as they expand the quantization range and reduce available bit precision for
the majority of values. Recently researches (Xiao et al. 2022 Shao al., 2023 Ma et al. 2024
Ashkboos et al. 2024) have employed linear transformations to tackle these challenges, showing
Cooresponding author "Equal contribution
Thhss work conducted during his internship Houmo

---

Published as a conference paper at ICLR 2025
Q1%
OSUR=58%
OSUR=61%
M2
Mp
Mg Amer
0
(a) Original
(b) Smooth Transform
(c) Rotation Transform
(d) Ours
Figure I: Transformation of batch of two-dimensional data X ~ ((, E) using different methods. Eigenvalues a and A represent the spread of the distribution along principal axes after eigenvalue decomposition of Σ. (a) shows the original distribution, while (b). (c), and (d) illustrate the effects of the Smooth-bard based, Rotate-buse and ours OST-based methods, respectively, on QSUR. The ellipse represents the space occupied by the data, and the square indicates the quantization space required to quantize this distribution, determined by the maximum and minimum values of it. The gray dots within the square denote the specific quantization points within the quantization space. The higher the number of quantization points within the ellipse, the greater the quantization space utilization rate of the distribution.
effectiveness improving data distribution in specific regions of the quantization space. For instance, amooth-based transformations like SmoothQuant (Xiao et al. 2022)) alleviate activation quantization difficulty by redistributing it to weights, reducing inter-channel variance. Similarly, rotation based methods such as Quarot rotation techniques to suppress outliers. However, these approaches remain heuristic and do not optimize the distribution across the entire quantization space.
In this paper, introduce the concept of Quantization Space Utilization Rate (QSUR) as more effective metric for evaluating the quantizability of transformed data. We define QSUR as the ratio of the volume occupied by set of data X to the volume of the quantization space corresponding to X. Given that weights and activations typically exist in multiple dimensions, this quantization space is modeled hypercube where the edge lengths are determined by the maximum quantization range across all dimensions of the data. Experiments demonstrate a positive correlation between QSUR and quantization accuracy (as shown in 3). suggesting that higher QSUR values contribute to improved quantization precision. In addition, we complement QSUR with mathematical derivations (refer Sec for more details), which examine the effects and limitations of linear transformations and establishes theoretical foundation for developing more effective transformation. As shown in Fig D as QSUR increases, the data distribution becomes flatter, allowing more quantization levels to be utilized. Smooth-based and rotation-based transformations improve QSUR from different perspectives: smooth-bars based transformations excel at reducing variations among eigenvalues, whereas rotation-based transfornations adeptly balance feature orientations and harmonize mean values across dimensions. However, these approaches encounter a critical limitation: their effectiveness is restricted to specific regions within the quantization space, leading low utilization rates.
To this end, we propose Orthogonal and Scaling Transformation-baan Quantization (OSTQuant). OSTQuant assigns learnable equivalent transformation mair-comprising an orthogonal transforma- tion and a scaling transformation-l each fully connected (FC) layer in LLMs. During optimization, these transformation pairs aim optimize the distributions of weights and activations across the entire quantization space. The transformation pairs and their inverses are then fused into their respective FC layers, preserving the original computational graph. Besides, unlike block-wise reconstruction methods (Shao et al. 2023 Ma et al. 2024), our equivalent transformation pairs do not alter network's final output, ensuring generalization and avoiding overfitting to the calibration set.
Another challenge is that, unlike the original floating-port point models trained on large datasets, PTQ typically conducted on a small calibration set (e.g.. 1000 or fewer). In this context, persisting with the original cross-entropy loss may result in decline in model performance. To mitigate this logits
problem, we introduce KL-Top loss function, which leverages the top highest-probabill from the full- aelliirecision model, rather than concentrating solely the logit probability corresponding to the correct label. Our Kop loss improves the capture of more nuanced semantic information while mitigating noise from the long-tail distribution in the full KI. divergence. As for optimizer,
2

---

Published as a conference paper at ICLR 2025
employ Riemann @Jaees 1976). proficient technique tailored for learning Stiefel Manifolds using first-order information.
NHSA Input
MHSA ispat after OSTOvant
Normallzzdd QSUR Zero Shot prectsion retentioin for LLaMA models
Ulamet (Zem- Shot Scomi
Uamaz LBB Lomee Scont
404
Limrs33 Shot
PPH Imput
FTR Inpur ofter OS ustt
Llamal Ovomaatzed (SUR)
Llamat 128 (Normallio
Ubmad Ovomalllzed OSUR)
Nethos
Figure 2: Activation distribution in the LLaMA- 3-8B before and after applying OSTQuant shows significant differences. Prior to transfor- mation, the distribution across different chan- nels exhibits substantial variation and contains numerous outliers. After OSTQuant, the distri- butions become more uniform across channels. with accuracy.
Figure 3: Zero-Shot" precision retention (under W4A4 quantization) and normalized QSUR are eval- uated for LLaMA variants across different quantiza- tion methods. The normalized QSUR derived as the d-th root of QSUR, where d denotes the number of channels. QSUR exhibits positive correlation
OSTQuant is an effective and efficient PTQ method for LLMs, outperforming existing approaches across various models and benchmarks. In the W4-only setting, it retains above 99.5% of the floating point accuracy. In the more challenging W4A4K configuration, OSTQuant reduces the performance gap by 32% on LLLMA-3--8 compared to ofte-of-thheart (SOTA) methods. In terms of speed, it can complete the quantization of LLaMA-3-8B contributions are summarized as follows:
A800 GPU in just 20 minutes. Our
We introduce the concept of Quantization Space Utilization Rate (QSUR) as an effective metric to evaluate quantizability. We also complement QSUR with mathematical derivations that examine the effects and limitations of various transformations, guiding the next steps in optimization and method design.
We propose OSTQuant, a fast and effective PTQ method. It improves QSUR and quantiza- tion performance by globally optimizing multiple equivalent transformation pairs in LLMs, each consisting of a scaling transformation and an orthogonal transformation. To address the challenge of limited datasets imposed by PTQ during optimization, introduce the KIp loss, which captures richer semantic information from the model while mitigating the impact of label noise.
Building on these advancements, our method demonstrates robust performance in both weight-only, weight- and eeight-activation--vwaae W4A16 configuration, it retains over 99.5% of the full precision accuracy, while in the more aggressive W4A4KV4 setting, it maintains at least 96% of the model's original performance.
quantization modes. In the
2 RELATED WORK
Post Training Quantization(PTQ) for LLMs. Post-training quantization (PTQ) has become a mainstream technique for LLMs due its efficiency. Existing PTQ methods can be broadly divided into weight-only and weight-activation quantization. To reduce memory usage, some approaches focus on weight-only quantization. GPTQ (Frantar et al., 2022) uses Hessian-basee error compensation to achieve high compression rates by minimizing quantization errors. AWQ (Lin et al.) 2023) and OWQ (Lee et al. 2023) improve performance by addressing the impact of activation outliers on weight quantization. QulP (Chee et al. 2023) and QulP (Tseng al. 2024) use random Hadamard matrices for incoherent processing and apply vector quantization to weights, achieving better performance with reduced precision quantization. Unlike weight-only methods, weight-activation quantization aims to speed up LLM inference by quantizing both weights and activations, including the key-value (KV) cache. The main challenge in activation quantization is that outliers dominate the range, leaving
3

---

Published as a conference paper at ICLR 2025
few significant bits for most values, leading to substantial errors. ZeroQuant Yaoo al. 2022
proposes a fine-grained hardware-friendee quantization scheme for both weights and activations. SmoothQuant (Xiao et al., (022) shifts quantization difficulty from activations to weights through mathematical transformation. OmniQuant (Shao et al.) (223)) further enhances performance by training quantization parameters and transformation coefficienss. Moreover, I-LLMM (Hu al. 2024) achieves integer-only quantization and inference through fully-smooth block reconstruction and fully integer operators. Recently, QuaRot (Ashkboos et al. 2024) uses random rotation matrices to enable 4- bit quantization of weights and activations, while SpinQuant (Liu et al. 2024) learns these matrices refine 4-bit quantization.
Riemannian Optimization. The optimization of rotation matrices necessitates adherence to or- thonormality constraints, which corresponds to performing Riemannian optimization on the Stiefel manifold (James) 1976), encompassing all orthogonal matrices. Cayley SGD (Li et al. 2020) relies iterative approximations of the Cayley Transform, achieved solely by matrix multiplication, enabling effective optimization of rotation matrices for arbitrary loss functions. RAOM (Bécigneul & Ganea 2018) extends optimization methods such as ADAM (Kingmal 2014), ADAGRAD, and AMSGRAD into the realm of Riemannian optimization. Meanwhile, Geoopt (Kochurov et al., 2020) supports fundamental Riemannian stochastic gradient descent (SGD) and adaptive optimization algorithms, facilitating seamless integration into models for comprehensive optimization.
3 QUANTIZATION SPACE UTILIZATION RATE
Although significant progress has been made PTQ using linear transformations to mitigate quanti- zation loss (Xiao al. 2022 Ma et al. 2024 Ashkboos et 2024), these methods are primarily heuristic and result-driven. lacking a quantitative metric to assess quantization difficulty or the effectiveness of different transformations. To address this gap, we introduce a novel metric, the Quantization Space Utilization Rate (QSUR). which quantifies how effectively weight or activa- tion distributions utilize the available quantization space. QSUR provides critical insights into the strengths and limitations of existing methods and lays the groundwork for developing more efficient approaches, including our OSTQuant method described in Sec 4.1
Distribution of Weight
Distribution of Activation
The varience difference of Weight on different channels
6:2
0:00
15
30002
J000
20:00
20
The variance difference of Activation different channels
-6.5
-25
-0.3
(a)
(b)
Figure 4: The distribution of activation and weight in LLaMA-2 7B. (a) The weight and activation distributions exhibit a Gaussian pattern. The red dots indicate the mean value of the distributions. Both the weights and activations are projected onto a two-dimensional space. (b) The inter-channel variance disparities between weights and activations. In comparison with weights, the inter-channel disparities of actívations more pronounced.
Quantization Notations. In this section, we define the key notations used in quantization. Matrices are denoted by bold uppercase letters (e.g.. X), while vectors are denoted by bold lowercase letters (e.g.. x). The operator Q refers to the guantization function. For comprehensive list of mathematical symbols and definitions, please refer to Appendix AA.I where additional details quantization and dequantization are also provided.
Lemma 1. By the central limit theorem, the distribution after Hadamard transformation follows an approximately ball-shaped Gaussian distribution, demonstrated in QulP# (Tseng el al. 2024).
Definition 1. Given a set of d-dimensional data X E Raad let Vx denote the hypervolume occupied by X, and Vsx denote the hypervolume of the quantization space S corresponding to X. The quantization space Sx is hypercube whose edge lengths are defined by the maximum quantization range across all dimensions of X. The Quantization Space Utilization Rate of X is then defined as:
Vx
QSURX
(1)
Vsx
4

---

Published as a conference paper at ICLR 2025
Given X ~ [H, E). Vx is calculated based on the ellipsoid formed by the covariance matrix
Σ and mean vector μ.. The covariance matrix can be diagonalized via eigenvalue decomposition: Σ = QAQ' where Q is unit orthogonal matrix of eigenvectors, and A diag((1, A2..... Aa)
contains the eigenvalues in descending order. The hypervolume of this ellipsoid at confidence level a (e.g., = 0.99) is given by:
x4/2
(xa(a)) 4/2 V det (>)
Vx
(2)
T(d/2+ +
where I is the Gamma function and x²((a) is the chi-squared quantile. Since Q is orthogonal, the determinant simplifies to de) det(A), The volume of the quantization hypercube, Vsx- is determined by the range of the distribution along each principal axis. The extremal points of the ellipsoid are closely correspond to the maximum and minimum along these axes. We denote the eigenvalues of the principal axes corresponding the points with the maximum and minimum coordinate values as Amax and Amin. respectively. After transforming these points back to the original space, the maximum and minimum coordinate values can be represented as:
org D max V x3((a) Amna Qmax + /
(3)
org v mmn V xā(() Amin quin + f
(4)
Vsx (max(v ( min(v )
org
(5)
where qmax and qmin denote the eigenvectors corresponding to Amax and Amin respectively. Thus, the QSUR becomes:
44/2
(x²(a)) 2/22 det((A)
QSURx
Vx
[(a2+11)
Vsx
maxx V xa(a))
Amax
1max
+ #1) min( V xa((a) Amin |qminl + (t)
(6)
Since the magnitude of the mean vector is often smaller than the largest eigenvalue. we neglect the mean vector μa, so Amax Amin = λy, resulting in:
r(d/2+1) (x2(a)) d/2
44/2
del(AA)
A 2d (max( v A -q)))
F(d/2+1)
QSURx
(7)
maxx x((a)) λy 91))
2d
From Eq 7 we observe the following: 1) QSUR is proportional to the product of the ratios of each eigenvalue A; to the largest eigenvalue A1: 2) The maximum component of the eigenvector qi is inversely proportional to QSUR. As demonstrated in Appendix A.2. when the components of d1 take values of 1d- the denominator in Eq 7 is minimized.
Applying a linear transformation T to X ~ N(,, E) results in a transformed distribution D ~ N(.. E where f = Ty and Σ TQAQTT" TT Smoothing based approaches (Xiao et al. 2022 Shao et al., 2023) treat as diagonal matrix that scales variances across different channel axes, indirectly reducing the dispar- ities among the eigenvalues >;- However, these methods are particularly sensitive to outliers and uneven mean values, especially when the mean vector contains significant variations like Fig ( Moreover, when quantizing both weights and activations simultancously. these methods often fail to strike balance. Rotation-based methods, such as those proposed in (Ashkboos et al. 22044 Liu et al. 2024), reduce outliers in both weights and activations through rotation, thereby decreasing the hypercube volume to increase QSUR. As proven in Appendix A.2.2 this ability to reduce outliers stems from the capacity to modify the matrix Q. which improves with increasing dimensionality.
Influence of linear transformation on QSUR.
When the orthogonal matrix is T = d 1 H HQQ where d is the dimensionality, and H is matrix composed of +1 entries, the best outlier reduction capability can be achieved.
In combination with E the maximum QSUR is achieved when:
T = c A Q
(8)
where e is an arbitrary scalar. At this point, the maximum utilization rate is given by QSUR"-
4/2
rd/2=1 24
Further details can be found in Appendix A.2.3
5

---

Published as a conference paper at ICLR 2025
Ea
ROPG
Aci
WV
Quant
oitheggoool mestrie
Trainable
9/-m
Trainable scaling mabrix
Transformed
Hadamard
S
Merped and quantizud weights
Figure 5: The overall flow diagram of OSTQuant. The top section illustrates how the global orthogonal transformation, Ryes, along with the two scaling transformations, Sattn and fy collaborate within each block to adjust the distributions across the entire network while maintaining computational invariance. The bottom section highlights four equivalent transformation pairs applied to the FFN and
layers. Each fully-connected layer's activation and weight are influenced by one or more of these transformation pairs. During runtime, these transformation pairs are fused with the weights, ensuring minimal runtime overhead.
4 METHODOLOGY
4.1
ORTHOGONAL. AND SCALING TRANSFORMMATION-BASED QUANTIZATION
We propose Orthogonal and Scaling Transformation based Quantization (OSTQuant), a novel frame- work designed to optimize the distributions of weights and activations in LLMs through learnable equivalent transformation pairs, with the goal of improving quantization performance. The core motivation of OSTQuant is that the combination of orthogonal and scaling transformations enhances the QSUR, as illustrated in Fig 1 and explained in Sec
As illustrated in Fig. 5 OSTQuant applies multiple transformation pairs globally within and across blocks of LLMs. Specifically, four equivalent transformation pairs are learned within each block, with each pair consisting of a learnable diagonal scaling matrix and learnable orthogonal matrix. These transformations work together to reshape the distributions of weights and activations, making them more nuantzationn-fieedll.. OSTQuant preserves equivalent transformations a global network level. As a result, the final output of the network remains unchanged when quantization is not applied, effectively preventing overfitting.
Equivalent Transformation Pair We define a transformation pair as T = AO, where T consists of a diagonal scaling matrix A and a unit orthogonal matrix O. As a result, the forward inference process is reformulated as follows:
y
where Q(-) represents the quantization operation. Since A is a diagonal matrix, its inverse is simply the reciprocal of its diagonal elements. We directly optimize O because any orthogonal matrix O can be decomposed into a Hadamard transform and another orthogonal matrix.
Earnability and Computational Efficiency: Both O and A are learnable parameters. The inversion of the diagonal matrix computationally simple, enabling efficient forward passes. The orthogonal matrix O can be optimized using gradient- based optimizers, such as RiemannAdam (Bécigneul Ganea 2018 which supports optimization on Stiefel Manifolds. This allows the entire process to fully leverage first-order gradient information for mnd-to-end learning. 2. Equivalence Preservation: Ignoring the effects of quantization, the forward process remains mathematically equivalent to the original model. This ensures that activations and thus
Equivalent Transformation Pair has three advantages: 1.
weights retain their consistency while making their distributions more quantization-friendd reducing the risk of overfitting. 3. After optimization, O and A can be directly merged into the existing weights, meaning no additional computational overhead or parameters are introduced during deployment, ensuring efficient inference.
The optimization objective for the entire network can be formalized as:
arg min A.O,
L(g. y; A,, O
(10)
where 0 represents the frozen network parameters, and L(j. y) represents the loss between the quantized network output ý and the full-precision output y.

---

Published as a conference paper at ICLR 2025
Weight Outlier Minimization Initialization (WOMI) As shown in (Cholakov et al. 2023). Weight typically follow a Gaussian distribution with zero mean. Therefore, we initialize the orthogonal matrix O using the Eq provided in appendix. For the global orthogonal matrix Rres+ we concatenate the weights of all linear layers receiving residual inputs along the input channels, denoted as Wn--oxxe, its covariance to obtain the eigenmatrix Qw- Then, we initialize Ree- = (Qw) ) H is normalized Hadamard matrix. For the orthogonal matrices of the Out-projection and Value-projection layers, we split the O matrix along the head dimension and apply the same initialization method as Rres- For all scaling matrices, we initialize them as identity matrices.
and perform eigenvalue decomposition
Inter-Block Learning. As illustrated in the upper half of Fig 5 the global Rres is applied at the embedding layer and propagates through each residual path within the L.LM via the projection layers. This transformation rotates activations throughout the entire residual path. Due to the porm-preserving property of unitary orthogonal matrices (Tseng et al., 2024), we can bypass the RMSNorm layers and apply an inverse transformation to the inputs of each residual connection's 's projection layer and the final output head, ensuring equivalence is maintained.
Additionally, for the two normalization layers in each block and their respective projection layers, we introduce two diagonal scaling matrices, St and S fnn to smooth channel-wise differences. The matrix Ryes simultaneously rotates activations along the residual paths and adjusts the weights of multiple projection layers. The scaling matrices Sartn and S/ apply scaling to the outputs of the RMSNorm layers and their corresponding projection layers. These transformations can be absorbed into the corresponding weight matrices: the orthogonal transformation Ryes merges with the projection weights along the residual paths, and the scaling matrices are incorporated into the weight vectors of the R.MNNrm layers. As shown in Fig 5, by fusing Rres with all projection weights during optimization, impact model accuracy. This approach helps mitigate the effects of quantization errors by adjusting for these shifts, thus improving model performance.
effectively learn how distribution shifts in weights and activations
Learning. As illustrated in the lower half of Fig 5 we introduce two equivalent transfor- mation pairs within the Multi-Head Self- Alff-Attention layer of each transformer block. Specifically, the value (V) and output (O) projection layers are transformed across layers. For each attention head, the data flow is expressed as:
Intra-Block
Y ph. xx
(Sh "Rh" Wh)
(11)
Here, h denotes the head index, ph is the attention matrix, and X is the input to head h. We learn a rotation transformation R and a scaling transformation S for each attention head. These transformations aim to improve QSUR for both the value cache and the output projection layer.
After the Rotary Positional Encoding (ROPE) operation, the output query and key can naturally undergo an equivalent scaling transformation Syk. similar to the approach in (Hu et al. 2024). Due to the multiplicative nature of positional encoding, this scaling transformation can be incorporated into the weight matrices W,, and W,- To further enhance the quantization efficiency of the key cache, apply an additional Hadamard transformation like Quarot (Ashkboos et al.) 2024) to the outputs of the query and key. Similar to S,k, we can optimize the diagonal matrices in the up- up-projection and down- wwwn-proecction of the FFN layer. However, the inverse of the Hadamard transformation is fused into W down from the very beginning.
Token Count distribution for reaching 90% cumulative probability
4.2 KL-TOP LOSs.
While LLMs are typically trained on vast datasets, OS- TQuant optimization is often performed using a much smaller sample set, typically around 1,000 examples. In this limited- data setting, directly applying original cross-entropy (CE) loss can result in accuracy drop. As shown in Tab T despite the quantized model exhibiting lower perplexity compared to its
67
87
30
17380
32500
57500
Number of Tokiens
Figure 6: The distribution of the number of tokens required to accumulate 90% of the total prediction probability in the L.aMA- 2-7B model.

---

Published as a conference paper at ICLR 2025
full-precision counterpart after training with CE loss, its performance zero-shot tasks declines. One likely explanation is that small and simple datasets, such as WikiText-2 (Merity et al.) 2016). may not fully utilize the capacity of LLMs. Consequently, relying solely on CE loss, which focuses on a single lahel, might cause the model to overfit to a narrow set of features, thereby compromising its emergent capabilities.
Table 1: Impact of different loss on Wiki PPL and Are (Boratko et al., 2018 accuracy for LLaMA models: While Origin loss reduces PPL, zero-shot scores better reflect the model's performance.
Model
Loss Type
Wiki PPL
Are-Easy Score
Are- Are-Challenge Score
Origin
5,38
69.87
42,41
LLaMA-2-7B
KT-Top
5.94
72.69
44.62
Origin
5.12
75.09
46.08
LLaMA-2 13B
KI-Top
5.25
75.29
47.10
Origin
6.80
76.68
49.26
LLaMA-3 8B
KT-Top
7.29
76.73
49.32
natural idea is to align the prediction distributions before and after quantization reduce overfitting risks, such as using the Kullback-Leibler (KL) divergence for optimization. However, this approach also faces challenges. LLMs typically have vocabularies of tens of thousands more(e.g., LLaMA- 3-8B has over (00,000 tokens). As illustrated in Fig 6 the prediction results of the full-precision model follow severe long-tail distribution, with only a small number of tokens having significant probabilities. If we directly apply KI. divergence over all classes, the loss may be dominated by uninformative classes with negligible probabilities, adding noise to the training process.
To address this, we propose the KL-Top loss function, which computes KL divergence over only the top-k classes with the highest probabilities. By focusing optimization on the model's primary predictions, this approach enhances gradient quality. In the global KL loss, low-probability values can introduce noise, leading to inaccurate gradient updates. By restricting the computation to the top-k classes, the model receives clearer and more informative gradients. Morcover, when dealing with a large number of classes (c.g., over 100,000), both computation and memory costs become substantial. Limiting the calculation to the top-k classes (e.g., k 1000) not only reduces complexity but also accelerates the training process. The KL-TTop loss is calculated as follows:
idrs = topk(z)
(12)
C = Σ
=|1
log
(13)
xEidzs
where z and & are the prediction distributions before and after quantization, respectively.
5 EXPERIMENTS
Models and Datasets. We apply method to the entire LLaMA family, including LLaMA-1 (7B-30B) (Touvron et al., 2023a). LLaMA-2 73-13B) (Touvron et al. (023b), and LLaMA-3-8B. We report perplexity (PPL) scores on the WikiTex12 (Merity et al. (016) test set. However, as mentioned in Tab perplexity may not fully reflect the model's 's true performance after quantization, zero-Shot tasks better reflect the model's actual performance. Therefore, we also evaluate the models on up to nine zero-shot tasks using the Lm-evaluatioon-hrrnes. (version 0.4.4) (Gao) al. 2024), including BoolQ (Clark al. 2019), HellaSwag (Zellers al. 2019), LAMBADA (OpenAl) (Radford et al. 2019). OpenBookQA (OBQA) (Mihaylov et al. 2018), PIQA (Bisk
al. 2020), SIQA (Sap et al. 2019)) WinoGrande (Sakaguchi et al.) 2021) ARC-Easy, and ARC-
Challenge (Boratko et al. 2018).
Baselines and Implementation Details. addition to the basic RTN approach, we benchmark our approach against SmoothQuant (Xiao et al. (002), GPTQ (Frantar et al., 2022), and current state-of-the-arn methods such as Quarot (Ashkboos et al. 2024 and SpinQuant (Liu et al.) 2024) for both weight-only and weight-activatte token asymmetric quantization without any pruning operations, while weights are quantized using symmetric per-channel quantization. We use RiemannAdam (Bécigneul & Ganea 2018) to optimize all unit orthogonal matrices and scaling matrices. During the distribution optimization phase, we use 1,000 samples from WikiText2, each with token length of 2.048, and iterate 150 times with batch size of 8. We apply cosine learning rate decay, setting the initial learning rate for all orthogonal matrix parameters to 2 10-2 and for scaling parameters to 3 10-2
quantization. All activations are quantized using per-
8

---

Published as a conference paper at ICLR 2025
5.1 OVERALL RESULTS
Table 2: Comparison of perplexity on WikiText2 and averaged accuracy on nine Zero-Shot tasks. Results for SmoothQuant, GPTQ. OmniQuant, AWQ, and QuaRot are based on official code and SpinQuant's s results for LLaMA-2/3 using official weights, with LLaMA-1 from the official code.
LLMMA-3 LLLAMA 70B
LLaMAA-275 7B LLaMA-2 1
L38 708
LLaMA 7B LLLaMA 13B
LLaMA 30B
Method
G-hoo Wak 10-cho? Wiki
6-ibot
Wai
10-shot"
Witl
16-ibor"
Wiki
O-dict"
Wiki
Wiki
O-iho
'iii""
W-A-KV
Avge(1)
(1)
(Avg(())
(.)
Avg/T) (1) Avg.(1)
()
(vg
(1)
Avg.()
(1) Avg.(1)
(3)
(1) ())
16616-66
Poining
68.09
73.81 233
65.21
5.47
67.61
4..8
71.55"
3.52
64.46
6666666667
5.09
70.00
4.10
RTN
63.70
8.13 31.15
1e5
61.27
7.02
60.24
6:39
69.62
3.87
62.67
7.94 63.45
8.60
65.69
6.13
SmoothQuant
62.79
67.94
6,70
8.03
7,66
62.69
18.75
65.69
GPTQ
61.03
7.43 31,45
9e3
60.86
9.84
64.71
5.79
70:96
3.94
60.15
7.9
64.36
6.58
66.95
5.26
Omesquant
65.66
7.19
63.19
5.74
66.38
5.02
71.04
3.47
63.42
5.86
66.22
5.21
69.07
4.25
4-16-16
AWQ
67.05
68.92
5.7
65.58
5.28
4.28
Quakee
67.27
6.53
72.93
3.53
64.30
5.62
66.95
71.21
63,40
5,83
65.91
5.20
69.73
SpinQuannt
66.54
6.49
72.90
3.49
63.50
5.58
67.14
5:001
71.12
3.43
63.94
5.76
66.32
5.16
69.62
4.21
OSTQuant
67.50
6.53
73.69
3.19
64.37
5.64
67.31
4.94
71,48
3.41
64.13
5.81
66.62
5:21
69.84
4.19
RTN
31.42
6e2
31.21
1633
32
30.66
30.90
764
32.31
765
51 51
3e4
31.57
265
Smooti>Quant
33.04
163
34.67
3e2
92. 13
mAn
34,26
1e3
35 86
3e2
34.42
362
33.29
6e2
34.64
163
GPTQ
32.98
31:47
32.72
31.51
30.85
4-4-16
QuaRet
61
8:02
65.56
6.35
6.05
65.13
5.35
69.96
3.78
61.26
6.22
64.46
5.50
68 14
4.57
SpinQuant
61.11
7.28
66.99
6.10
$7.37
6.78
63.23
5.24
70.58
3.68
61.82
6.08
61.59
5.36
68.08
4.53
OSTQuaut
65.14
7.24
72.21
3.97
63,90
5.60
66,24
5.14
70.92
3.57
62.72
6.04
65.80
5.40
68.52
4.43
RIN
11.33
722
50.62
'63
32.67
man
30.03
763
31.95
761
52.67
1:1
3e1
31.64
263
SmoothQuunt
32.96
1c3
33.76
3c2
32.
33.36
1e3
35.54
3x2
33.32
362
33.28
5c2
34.65
163
GPTQ
33.71
31.20
33.52
nan
31.09
man
30.63
31.77
4.4.4
Omniquant
32.33
4e2
48.40
14,26
50.35
12.30
48.16
11.26;
45.63
10.87
45.01
12.35
QuaRoe
61.38
8.18
65.33
6.6
61.48
6.11
65.16
5.39
70.30
3,80
61.22
6.26
64.59
5.53
68.08
4.60
SuinQuant
64.10
7.35
66.31
6.24
62.01
5.96
64.13
5.74
70.57
3.61
61.32
6.12
64.95
5.39
68.14
4.55
OSTQuant
65.37
71.69
4.01
63.18
5.91
6.07 65.43
5.40
68.20
Quantization performance. As shown in Tab 2, our method consistently outperforms previous
SOTA approaches across almost all configurations and models. Under the 4-16-16 setup. OSTQuant surpasses all prior methods, maintaining at least 99.5% floating-point (FP) accuracy in zero-shot tasks. Compared to other weight-only methods like GPTQ and AWQ. OSTQuant further narrows the gap with FP models. In the most challenging LLaMA-3-88 model, OSTQuant achieves only a 0.29-pooint performance drop in zero-shot evaluations, whereas other methods incur losses exceeding 1.55 points. Even in the highly challenging 4-4-4 setting, our approach retains a significant performance gain, out- performing the SOTA method, SpinQuant, by around point across multiple models. Notably, when the KV cache is not quantized (in the 4-4-16 setup), OSTQuant achieves a significant performance boost over SpinQuant, with gains up to 6.53 points (LLaMA-22 7B). These substantial performance improvements demonstrate the effectiveness of our approach. More detailed results can be seen in Appendix [ Once activation is quantized, rotation- methods significantly outperform smooth-based methods, confirming that latter struggle with outliers and uneven distributions. In Fig 3 the QSUR across different methods show a clear positive correlation with model performance. Our approach achieves the highest QSUR, effectively mitigating the challenges of outliers and uneven distributions that hinder prior methods, leading to improved model accuracy.
Table 3: The speedup and memory saving factor of LLaMA models with different parameter sizes and sequence lengths, compared between our 4-bit implementation and FP16. All tests were conducted on a Transformer block with batch size 4 on a 3090 GPU.
Prefill Speedup (Seglen)
Memory Saving Factor (Seqlen)
Model Size
256
512
1024
2048
4096
8192
256
512
1024
2048
4096
8192
7B
2.24x
2.27x
2,23x
2.14x
2.11x
2.02x
3,48x
3.34x
3.12x
2.86x
2.57x
2.34x
8B
2.42x
2.52x
2.52x
2.43x
2.36x
2.23x
3.48x
3.36x
3.12x
2.77x
2.38x
2.00x
13B
2.62x
2.68x
2.63x
2.52x
2.83x
2.32x
3.64x
3,51x
3.30x
3.02x
2.70x
2.43x
30B
3.18x
3.01x
2.98x
3.40x
2.84x
2.68x
3.70x
3.59x
3.42x
3.15x
2.83x
2,53x
Speedup and memory savings. OSTQuant incurs only negligible loss in 4-bit quantization. making bit inference feasible. As shown in Tab OSTQuant delivers an average inference speedup of over 2x and memory savings exceeding 3.5x, demonstrating the significant improvements in inference efficiency. Detailed speedup and memory saving results can refer to Tab Moreover, OSTQuant provides substantial advantages in training speed compared to block reconstruction-bin based methods. With only 150 iterations and a minimal number of learnable parameters, we optimize the 7B and 13B models in around 20 minutes, and the 30B model in 120 minutes, achieving up to 5.3x speedups compared to OmniQuant, refer to Tab IT for more details.
5.2 ABLATION STUDY
Effect of different transformation. We ablate the effects of various transformation matrices on LLaMA-2 7B, identifying four groups where orthogonal and scaling equivalent transformations can

---

Published as a conference paper at ICLR 2025
Table Ablation study on the impact of different transformation matrices on Wiki PPL. and zero- shot score for LLaMA-2 7B under W4A4KV4 quantization.
Metric
Baseline
+ Ryes
+Sres
+Rdn
+Suld
+ +Bys
+Sph
+Ro
+Sop
Wiki PPL
nan
9.70
9.46
6.16
6.00
5.92
5.92
5,94
5,91
7ero-shot
33.51
54.33
53.74
61.75
61.79
62.35
62.56
63.11
63.18
be applied. 4 presents the contribution of each parameter group under W4A4KV4 setup. Our results show that the global orthogonal transformation Rres brings the largest improvement, followed closely by Raoonn- Notably, scaling transformations further build on the orthogonal transformations R by effectively balancing variance across channels, thereby minimizing quantization losses and enhancing model performance.
Table 5: Effect of different optimizers on zero-shot performance of LLaMA models under the W4A4KV4 configuration. LRI and LR2 represent the learning rates for the scaling matrices and unitary orthogonal matrices.
Model
Optimizer Type
Best Steps
Best LRI
Best LR2
Zero-Shot Score
Cayley SGD
1.50
0.20
63.11
LLaMA-2-7B
Riemann SGD
500
0.10
0.02
63.09
Riemann Adam
150
0.02
le-3
63.18
Cayley SGD
200
1,50
0.2
64.77
LLaMA-2-13B
Riemann SGD
500
0.1
0.02
65.19
Riemann Adam
0.02
0.002
65.41
Different Manifold Optimizers. Since the unit orthogonal matrix resides on a Stiefel manifold,
we explore various manifold optimizers to optimize it, including CayleySGD (Li et al., (2020).
RiemannSGDD, and RiemannAdam (Bécigneul & Ganea 2018). Tab 5 compares these methods and shows that CayleySGD typically requires a higher learning rate to perform well, RiemannSGI needs more iterations, while RiemannAdam delivers the best results with the fewest iterations. We also discover that using learning rate for the Stiefel manifold 10 times larger than that for scaling transformation parameters leads to better results.
Table 6: Ablation study of values on Zero-Shot score and Wiki PPL for W3-only and W4A4KV4
configurations of LLaMA-2 7B.
Setting
Metric
k=5
k=50
k= 100
k=500
k= 1000
k=5000
k= 10000
Zero-Shot Score
61.87
61.88
61.75
62.18
62.30
61.25
61.21
W3 Only
Wiki PPL
6.06
6.116
6.13
6.07
6.06
6,06
6.12
Zero-Shot" Score
62.4
62.13
62.38
62.34
63.18
62.44
62.11
W4A4KV4
Wiki PPI.
5.99
5.96
5.95
5.96
5.96
5.93
5.94
Influence of k in Kop loss. The parameter k in Eq. 12 defines the number of classes considered when calculating the KL-Too loss, balancing optimization difficulty with semantic richness. Both excessively large or small & values negatively impact optimization. Tab a comparison of different & values. Furthermore, we analyze whether to apply softmax before or after the top-k selection. Our experiments indicate that setting & to 1,000 processing produces the best outcomes.
6 CONCLUSION
In this paper, introduce OSTQuant, novel post-training quantization method designed en- hance the efficiency of large language models (LLMs). Central to OSTQuant is the Quantization Space Utilization Rate (QSUR), a new metric we proposed to effectively assess the quantizability of transformed data by measuring its space utilization within the quantization space. Complemented by mathematical derivations, QSUR provides theoretical guidance for optimizing single data distribu- tions across the entire quantization space. Leveraging this insight, OSTQuant employs a learnable equivalent transformation pair composed of orthogonal and scaling transformations to optimize the distributions of weights and activations. Additionally, we introduce the KL-Top loss function to mitigate noise during optimization while retaining richer semantic information, even with the limited calibration data typically available in PTQ. Extensive experiments various LLMs and benchmarks demonstrate that OSTQuant outperforms existing quantization methods. These results highlight the effectiveness of optimizing data distributions across the quantization space and underscore OS- TQuant's 's potential to advance LLM quantization, making these models more efficient and practical for deployment in essource-constrainee environments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.087771 2023.
Saleh Ashkboos, Amirkeivan Mohtashami,, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms, arXiv preprint ar wxxv:2404.004566, 2024.
Gary Bécigneul and Octavian-Eugee Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:/810.0076) 2018.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the are dataset. arXiv preprint arXiv: 00358, 2018.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2- quantization of large language models with guarantees. arXiv preprint arXiv:2307.13300 2023.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14093-14100. IEEE, 2024.
Radostin Cholakov, Han Guo, and Yoon Kim. Distributional quantization of large language models. cee org, 2023.
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905. 10044 2019.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 Q): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. /ttLittt:://eeedd.oo//eerrrre 12688622
Xing Hu, Yuan Chen, Dawei Yang. Sifan Zhou, Zhihang Yuan, Jiangyong Yu, and Chen Xu. I-llm: Efficient integer-only inference for fully-quantized low-bit large language models. urXiv preprint arXiv:2405.17849, 2024.
Ioan Mackenzie James. The topology of Stiefel manifolds, volume 24, Cambridge University Press, 1976.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint wrXiv:1412.690 2014.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.
11

---

Published as a conference paper at ICLR 2025
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint 2.XX::23006.0227,, 2023.
Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint vrXiv:2002.01/15. 2020.
Ji Lin, Jiaming Tang. Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:3306.00978 2023.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- quantization with learned rotations. arXiv preprint arXiv:2405. 2024.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling. Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403. 12544, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1xXXiv1609.078880 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:/809 1809.02789, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense
reasoning about social interactions. arXiv preprint 191vv:104.009725 2019.
Wengi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao. Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. CORR, abs/2308. 13137, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04336, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for segeescale transformers. arXiv preprint (XX.x:2226600886666 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint
2019.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 QUANTIZATION PRELIMINARIES
Quantization & Dequantization. Quantization typically refers to mapping a floating- point number to a discrete interval with integer number. Here, we only consider uniform quantization. The quantization process can be expressed follows:
(
+ zp' 0, 2d -1)
x
clamp
(14)
Tmax
Imin
(15)
2n
Tmin
2p
(16)
8
X' = (X
-2p')pp)) )
2p')
(17)
where X is the floating-poii tensor, X is its quantized counterpart and X' is the dequantized resuls of X Here, n'' represents the number of bits (e.g., 8). is the quantization step size, determined by Tmin: Tmax: and n' .clamp represents truncation function. The choice of greatly affects the accuracy of the quantized model. We can obtain s from the activations of some samples, which is called static quantization. Alternatively, we can derive it from runtime statistics, known dynamic quantization. Quantization can also be distinguished by its granularity into per-channel quantization and per-token quantization Yao et al., 2022).
A.2
SOME DEDUCTIONS ABOUT QSUR
A.2.1
THE COORDINATES OF THE EXTREMUM POINT AND THE INFLUENCE OF ROTATION MATRIX
From Ed
4/2 T(d/2+1) (x3(a)) d/2 /det((A
VX V8x
QSURx
max( x2(()) Amax
|gmaxl + (1) min( (xx(x) Amin |gmin| + 1))
Given that the eigenvalues and the mean vector are fixed, the volume of the hypercube is determined by the eigenmatrix Q. For any cigenvector qi E Q. we have the condition:
qil 2 + 9/2 + ..... + gid =
(18)
Let limax max( ...,..., 4ba). follows that:
(aa)) which implies that for any i, we have qú S qi max. Therefore,
o + d/e + ... + q S d of max
(19)
This yields the inequality:
S mm
(20)
lg: max
d2
(21)
Consequently, when all elements in Q are +d 1, the quantization range minimized. Under these conditions, the QSUR' can be expressed as:
T(4/2+1) (x3(a)) d/2 Ad)((A)
4/2
QSUR'
(22)
max (Vx3(a)- Amax d + 11) min (- Vxā(a) Amin d 11))
When the mean vector μ1 0, the QSUR' can be further simplified to:
4/22 VIT
QSUR' P(d/2+1)
(23)
(Vii-d-t)
13

---

Published as a conference paper at ICLR 2025
A.2.2 THE BEST ORTHOGONAL MATRIX
In the linear transformation paragraph 3 of Sec we observe that applying a linear transformation T to X ~ (u, Σ) results in a transformed distribution:
X (i, Σ)
(24)
Since the transformation TQ remains an orthonormal matrix after applying the unitary transformation, the transformed eigenvectors are given by:
TQ qn qa...... qa]
(25)
Movited by Eq 21 we aim for the following transformation:
TQ = d H
(26)
where d the dimensionality, and H a matrix composed of +1 entries. Solving for T yields:
==
(27)
The covariance matrix after transformation becomes:
d à.
(28)
H
(29)
A.2.3 THE BEST TRANSFORM MATRIX
Inspired by Sec we know that applying a linear transformation T X ~ NN,, Σ) results in transformed X ~ N(a Σ). Ignoring the mean vector, we present a transformation matrix, composed of diagonal scaling transformation and orthonormal transformation, which minimizes the QSUR:
T = -= QT
(30)
Under this transformation, the covariance matrix becomes:
Σ A Q QAQTQAT
(31)
(32)
By substituting this result into Eq 7 achieve the maximum QSUR:
4/2
QSUR"
T(d/2+1)
(33)
2d
A.3 ADDITIONAL ABLATION EXPERIMENTS
A.3.1 THE EFFECT OF WEIGHT OUTLIER MINIMIZATION INITIALIZATION
Weight Outlier Minimization Initialization WOMI) is used to initialize trainable orthogonal matrices. This approach not only reduces outliers in the weights but also leverages the properties of Hadamard matrices to mitigate inter-channe disparities in activations, thereby improving the initial QSUR for both weights and activations.
We visualized the impact of WOMI on the weights. As shown in Fig 7 the original weight distribution exhibits significant variations across input and output channels. While QuaRot reduces inter-channel differences, noticeable spikes remain. WOMI, by leveraging the Hadamard matrix and the covariance matrix of the weight distribution, further smooths these inter-channel differences, effectively reducing the quantization space and relative quantization error. Additional visual results for other layers are presented in Fig 13
We conducted additional experiments to investigate the impact of WOMI on the performance of quantized models. Tab presents the performance LLaMA-2-7B and LLaMA-3-8D models initialized with WOMI and random Hadamard matrices. WOMI achieves lower perplexity and higher few-shot accuracy under both W4A4KV4 and W4A16KV16 configurations, showcasing its effectiveness. Interestingly, WOMI demonstrates greater performance improvements in W4-only quantization settings compared to W4A4KV4. This likely due to WOMI's superior capability in minimizing weight quantization emors, which is especially critical in W4-only configurations.
14

---

Published as a conference paper at ICLR 2025
516
025
012
615
4000
3000
3000
1000
2000
3009
2003
3000
4000
(a) Original weight distribution
(b) Hadamard, Relative LI Error: 0.79 (c) WOML Relative L1 Error: 0.42
Figure 7: Impact of WOMI transform and Hadamard transform LLaMA-2-78 weight (weight of Query projection in Layer 0) quantization. The relative error is computed by dividing the mean absolute error (MAE) between the original and quantized weights by the mean absolute value of the original weights.
Table 7:
Comparison of the impact of WOMI initialization and Hadamard initialization on the performance of quantized models.
Model
Quant Setting
Method
Zero-Shot
Wiki PPL.
Full-Precision
65.21
5.47
W4A16KV16
Hadamard
63.32
5.62
LLaMA-2-7B
W4A16KV16
WOMI
63.45
5.59
W4A4KV4
Hadamard
61.47
6.11
W4A4KV4
WOMI
61.52
6.09
Full-Precision
68.09
6.14
W4A16KV16
Hadamard
67.27
6.53
LLaMA-3-8B
W4A16KV16
WOMI
67.41
6.48
W4A4KV4
Hadamard
61.38
8.18
W4A4KV4
WOMI
61.40
8.17
A.3.2 THE EFFECT OF KL-TTP Loss
As shown in Tab 8 the results indicate that using SpinQuant( Liu et al., 2024 alone, even with the introduction of the KL-Top loss function, does not lead significant performance improvements and may even cause some degradation. However, when combined with orthogonal and scaling transformation pairs, the quantization performance improves significantly, For OSTQuant, using CE loss results in overfitting the calibration set, and this issue is alleviated by the introduction of the KL-Top loss function.
A.4 INFERENCE EFFICIENCY AND QUANTIZATION OVERHEAD
Tab g shows the prefill time and memory usage of LLaMA models with different parameter sizes and sequence lengths, compared between our 4-bit implementation and FP16. The inference envi- romment features an Intel(R) Xeon(R) Gold 5317 CPU and Nvidia 3090 GPU. The 4-bit matrix multiplication kernel was implemented using cutlass of nvidia, while the self-attention mechanism was realized with PyTorch's 's native SDPA (scaled dot product attention) function. All tests were conducted 500 times, with the median value taken the final result. Benefiting from efficient low- precision computation units within CUDA cores and reduced access overhead, OSTQuant achieves 2x speedup across various model sizes, and approximately 3x acceleration on the challenging LLaMA-30R model.
Fig & displays the performance and model size of various LLaMA models under different quantization bitwidths. Each line represents a different model (LLaMA-2-1 LLaMA-3-8B, LLaMA-2- 13B, LLMMA-30B, and LLaMA-3-70BB with quantization bitwidths ranging from 2 bits 16 bits. As
15

---

Published as a conference paper at ICLR 2025
Table 8: Performance of SpinQuant with the introduction of the KL-Top loss function on 9 zero-shot dataset tasks and the Perplexity changes on Wikitext2.
Model
Method
ARC-C
ARC-e
BoalQ
HellaS
Lam.
OBQA
PIQA
SIQA
WinoG.
Avg.
WikiZ PPL
RTN
23.72
30.56
46.18
29.83
2.70
60
52.45
34.39
50,20
33.18
704.34
SpinQuant
68.75
LLMMA3-SB
SpinQuant Kl-Top
47.29
73.95
75.82
75.64
71.40
41.58
78.16
44.
68.45
64.07
7:54
OSTQuunt
49.26
76.68
78.25
76.18
70.48
43.19
77.85
45.18
69.13
65.13
6,80
Ko-Top
RTN
27.22
27.06
50.83
27.34
0.93
49.51
34.85
50.51
32.67
nan
SpinQuant
40.44
71.08
74.40
73.51
70.66
41 80
76.88
43.50
65.82
62.01
5.96
LLaMA2-7B
SpinQuant KL-Top
40.76
71.29
74.61
73.08
70.19
40.94
76.32
43.85
67.78
62.09
6.16
OSTQquan
09.87
75.07
72.90
70.21
40.87
OSTQuant + KL-Top
44.62
72.69
75.41
73.27
70.21
11.00
78.13
44.42
68.27
63.11
5.94
Table 9: Prefill time and Memory usage LLaMA models with different parameter sizes and sequence lengths, compared between our 4-bit implementation and FP16. All tests were conducted on a Transformer block with batch size on a 3090 GPU.
Prefill Time
Memory
Model
Seglen
Prefill Speedup
Memory Saving
FP16
INT4
FP16
INT4
256
8.050ms
3.597ms
2.238x
0.411GB
0. I8GB
3.479x
512
14,904ms
6.579ms
2.265x
0.435GB
0.130GB
3,341x
1024
27.989ms
12.582ms
2.225x
0.483GB
0.155GB
3.1
LLaMA2-7B
2048
54.276ms
25.312ms
2.144x
0.577GB
0.202GB
2,857x
4096
1
53.145ms
2.112x
0.766GB
0.299GB
2.566x
8192
244.675ms
121.339ms
2.016x
1.147GB
0.491GB
2.336x
256
8.035ms
3.314ms
2.424x
0.430GB
0.124GB
3.478x
512
15,545ms
6.176ms
2.517x
0.442GB
0.132GB
3,356x
1024
29.169ms
11,599ms
2.515x
0.466GB
0.149GB
3.116x
LLaMA3-8B
2048
57.470ms
23.63
2.432x
0.513GB
0.185GB
2.774x
4096
117.523ms
49.835ms
2.358x
0.608GB
0.256GB
2.378x
8192
156.394ms
114.815ms
2.233x
0.795GB
0.397GB
2.003x
256
11 1.449mms
4.370ms
2.620x
0.634GB
0.174GB
3.643x
512
21. 21.195ms
7.924ms
2.675x
0.663GB
0.189GB
3.512x
1024
41.752ms
15.867 ms
2.631x
0.723GB
0.219GB
3,301x
LLaMA2-13B
2048
81, 81,965ms
32.553ms
2.518x
0.841GR
0.279GR
3.018x
4096
199. 0406m
70.442ms
2.826x
L,09GBB
0.399GB
2.702x
8192
359.409ms
154.640ms
2.324x
1,551GB
0.639GB
2.426x
256
18.682ms
5.883ms
3.175x
1,047GB
0.283GB
3.703x
512
34,393ms
11.445ms
3.005x
1.085GB
0.302GB
3,589x
1024
66.880ms
22. 22.464ms
2.977x
1.162GB
0.340GB
3,416x
LLaMA-30B
2048
157.500ms
46.317ms
3.400x
L315GB
0.418GB
3.148x
4096
272.355ms
96.052ms
2.835x
1662GBB
0.575GB
2.828x
8192
576.555ms
215.27ms
2.678x
2.242GB
0.887GB
2.527x
we can see, quantizing the weights to below 4 bits leads to significant drop in accuracy. Another notable observation that larger models, such as LLaMA-3-705 outperform smaller floating-point models like LLaMA-30B in both accuracy and parameter size after quantization.
As shown in Tab TO since the decoding stage of LLMs is often memory-bound, quantizing the weights to 4-b significantly reduces memory access overhead, thereby accelerating the inference process. The results demonstrate that, under the premise of achieving nearly lossless accuracy, even single A6000 GPU can successfully run I.LaMA-3-700 at speed of 15 tokens per second after quantization. This validates the effectiveness of our quantization method.
Tab 11 presents comparison of training time between our approach and block reconstruction-basse methods across models of varying parameter sizes. Thanks to the effectiveness of Weight Outlier Minimization Initialization and KL-Top Loss, we require only about 150 iterations to complete the quantization process. As shown in the table, OSTQuant offers a significant optimization speed advantage for smaller models, such achieving more than 5x speedup on the LLaMA 7B model compared to OmniQuant, Even for large-scale models like LLaMA 70B, achieves nearly a 2x speed improvement.
16

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of generation speed and memory usage before and after quantization in the decoding stage.
Model
Decoder Speed (tokens/see)
Memory Use (GB)
Memory Saving
FP
Quantized
Speed
FP
Quantized
LLaMA-2-7B
47.32
89.4
1.8%x
13.94
4.32
3.23x
LLaMA-3-SB
38.33
77.71
2.03x
15.83
5.88
2.69x
LLaMA-2-1318
23.7
55.35
2.34x
23.7
8.5
2.79x
LLaMA-30B
OOM
30,49
OOM
18.19
LLMMA-3700
0OM
14.68
OOM
38.41
Table 11: Comparison of the training time between our approach and block reconstruction-basel methods across models with varying parameter sizes.
Method
7B
8B
13B
30B
70B
Omniquant
1.6h
1.8h
3.3h
7.3h
9.5h
OSTQuant
0.3h
0.4h
0.8h
2.2h
5.5h
Speedup
5.3x
4.5x
4.1x
3.3x
1.7x
Accuracy (%6
W2
300
2-70
Model Size (GB)
Figure 8: Performance and model size the Llama series under different quantization settings. Accuracy is calculated by the average score of nine zero-shot datasets. Model size is determined by the memory usage when stored fp16 or integer formats.
A.5
FUTURE WORK
A.5.1 OSTQUANT FOR FULLY-QUANT LARGE LANGUAGE MODELS
introduces OSTQuant's novel strategy designed full quantization. Full quantization involves quantizing all activations within each Transformer Block to low bits (as shown by the quantization nodes inserted for all node inputs and outputs the figure). This reduces memory transfer overhead for activations and fully utilizes efficient www-prccsionn computational units.
Fig 9
As shown in Fig 9 OSTQuant introduces numerous equivalence transformations to alter the distribu- tions of all node input and output activations. Unlike the methods in Fig designed for traditional quantization, the design for fully quantization adds more equivalence transformations, particularly around ROPE and SiLU. Specifically:
17

---

Published as a conference paper at ICLR 2025
Cuss
Pulipredaion
Trainable orthogunal materic:
Trainable scaling matric:
Merged quantized welghte:
Transformed activation:
Figure 9: The extension of OSTQuant for full quantization.
lightweight GEMM layer and construct a weight matrix of shape (token, head_dim, head_dim) based on its principle. We then introduce pre-ROPE and post-ROPE transformation pairs.
ROPE Handling:
We treat ROPE
a. Pre-ROPE transformations are based on the fact that ROPE and the preceding linear layer can be viewed as consecutive matrix multiplications along the head dim. The corresponding transformation pairs are represented in figure by R Rr nd S SiRppee Post-ROPE transformations rely on the attention computation formula Attn Q@KT where Q means the query matrix and K means the key matrix in Self- Attention module. The corresponding transformation pairs are represented in figure by R post post
S
2. Smoothing Activation Discrepancies of SiLU: Inspired by smoothing methods for SwiGLU in I-LLM (Hu et al. 2024). we decompose SiLU as SiLU(X) = X (X) and use equivalences such as
1 1 o ((X1 5) 5)
X1 X2 (X1) S) .
alleviate inter-channel discrepancies of activations before and after SiLU. The correspond- ing transformation is represented in figure by Sulg.
We will conduct experiments in full-quantization domain in the future to fully explore the potential of OSTQuant.
A.6 FULLI RESULTS
A.
QUANTITATIVE RESULTS
In this section, provide comprehensive presentation of results across various datasets complement the main paper. Specifically, the results include:
Complete comparison of the perplexity score on WikiText2 and averaged accuracy on
zero-shot common sense reasoning tasks on LLaMA-I(Tab 13), 2 and (Tab (2).
Validate the effectiveness of OSTQuant on larger-scale language models such as LLaMA-2- 70B and LLaMA-3 3-BB-- 14)
18

---

Published as a conference paper at ICLR 2025
Table 12: Complete omparison of the perplexity score WikiText2 and averaged accuracy Zero-shot
Common Sense Reasoning tasks LLaMA-2 &
Model
#Bits W-A-KV
Method
ARC-c
ARC-e
BoolQ (1)
HellaS, (t)
Lam. (1)
OBQA
PIQA (f)
SIQA (1)
WinoG. (1)
Avg.
Wiki2
(1)
(t)
(1)
(1)
(.)
16-16-16
Full Precision
46.42
74.33
77.71
75.94
73.69
44.20
79.16
45.91
69.53
65.21
5.47
RTN
42.15
67.59
73.06
72.34
67.18
41.80
76.50
34.11
66.69
61.27
7.02
SmoothQuant
39.59
65.19
69.82
68.84
62.27
40.20
75.95
44.17
63.85
58.88
GPTQ
42.49
69.53
61.31
73.83
67.61
42.40
77.64
44.52
68.43
60.86
9.84
4-16-16
Omniquant
42.49
71.00
74.34
73.85
70.70
44.20
78.40
44.93
68.82
63.19
5.74
AWQ
44.11
70.75
78.07
74.98
70.68
78.13
45.14
69.38
63.89
QuaRot
43.94
73.15
76.97
74.87
73.06
44.00
78.24
45.09
69.38
64.30
5.62
SpinQuant
43.34
72.69
73.36
75.10
73.80
43.00
77.86
45.60
67.56
63.59
5.58
OSTQuant
44.54
73.31
75.57
75.04
73.67
44.20
78.89
45.50
64.37
2-7B
RTN
25.34
28.05
50.52
27.71
1.01
26. 20
30.82
33.93
48.38
32.44
nan
SmoothQuant
28.33
26.39
49.39
27.28
1.18
23.40
48
33.62
50.75
32.13
4-4-16
GPTQ
28.70
51.62
28.66
51.14
34.49
32.72
nan
QuaRot
42.32
69.65
74.77
72.91
70.81
39.80
77.20
43.55
65.82
61.77
6.05
SpinQuant
37.54
62.58
71.16
70.48
67.16
34,80
75.46
39.76
601.2
57.37
6.78
OSTQuant
71.93
75.41
74.94
73.22
78.51
45.85
63.90
RTN
27.22
27.06
30.83
27.34
0.93
25.80
49.51
34.85
50.5T
32.67
nän
SmoothQuant
26.37
25.63
47.71
27.05
1.11
26.40
51.90
34.49
48.38
32.12
GPTQ
26.96
27.65
52.84
28.83
29.
49.62
35.11
49,80
33.52
nan
4-4-4
Omniquant
31.40
53.75
63.79
55.06
35.63
34.40
66.59
40.28
54.70
48.40
14.26
QuaRot
41.43
69.32
74.19
72.50
70.66
39.80
77.42
43.35
64.64
61.48
6.11
SuinQuant
71.08
74.40
73.51
70.66
76.88
43.50
65.82
OSTQuant
42.92
72.56
74.71
73.14
71.76
44.40
77.42
44.98
66.77
63.18
5.91
16-16-16
Full Precision
49.15
77.53
80.58
79.39
76.62
45.20
80.63
47.49
71.90
67.61
4,88
RTN
42.92
66.34
71.38
66.62
68.99
39.40
76.93
44.06
65.35
66.24
6.30
SmoothQuant
46.25
70.45
74.92
69. 16
70.49
39 80
77.86
45.14
64.17
62.03
5,86
GPTQ
49.63
73.95
74.83
73.77
73.20
42
78.51
45.50
70.64
64.71
5.79
4-16-16
Omniquant
48.29
75.42
77.92
77.80
75.59
45.20
80.41
46.62
70.17
66.38
5.02
AWQ
48.63
78. 16
78.81
78.48
75.20
45.00
79.54
46.21
72.45
66.25
5.07
QuaRot
49.15
76.26
80.46
78.17
76.50
45.40
80.03
45.50
71.11
66.95
5.00
SuinQuant
49.15
77.48
79.27
78.46
77.10
80.03
46.47
71.67
67. 67
OSTQuant
48.72
76.26
80.67
78.27
76.34
45.54
80.25
47.65
71.90
67.31
4.94
2-13B
RTN
27.99
26.81
38.50
26.08
0.00
23.60
48,20
34.90
51.62
30.86
863
SmoothQuant
55.01
35.31
4-4-16
GPTQ
27.82
26.77
37.92
25.67
0.00
21 80
47.77
35.11
48.15
30.11
4e3
QuaRot
46.42
73.86
78.10
75.68
74.31
43.00
79.05
44.37
71.35
65.13
5.35
QuanQuant
43.77
69.99
76.57
74.63
72.81
41.60
77.20
44.27
68.
63.23
OST OSTQuant
47.78
74.66
80.03
77.60
75.94
44.40
79.38
46.06
70.32
66.24
5.14
RTN
27.82
26.52
38.38
26.27
0.02
26.00
49.18
34.39
49.17
30.93
763
SmoothQuant
GPTQ
27.90
26.39
37.95
26.16
0.00
27 00
48.26
34.39
50.43
27.85
5e3
4-4.4
Omniquant
32.85
55.13
64.34
60.13
42.85
33.40
68.17
39.76
56.51
50.35
12.30
QuaRot
73.91
75.33
SpinQuant
46.67
74.49
76.76
75.22
72.19
42 40
78. 29
43.45
67.72
64.13
5.74
OSTQuant
47.10
75.21
77.46
76.71
75.14
44.60
78.67
45.75
68.03
65.41
5.25
16-16-16
Precision
53.50
77.74
81.10
79.18
75.74
44.80
80.63
47.08
73.01
68.09
RTN
48.98
773.55
72.75
75.90
63.85
43
78.40
43.81
73.16
65.70
8.15
SmoothQuant
47.44
72.35
72.11
74.92
62.41
43.00
77.69
43.91
71.27
62.79
8.12
GPTQ
72.52
4-16-16
Omniquant
50.09
74.54
79.51
76.92
70.31
43.80
79.54
44.52
71.74
65.66
7.19
AWQ
52.22
76.68
80.31
77.51
74.81
44.20
79.60
46.26
71.67
67.03
7.36
QuaRot
SpinQuunt
52.13
72.28
79.20
78.40
73.76
44.80 80
79.98
45.50
72.77
66.54
6.49
Quunu
52.82
79.84
80.31
77.86
76.48
80
80.74
45.55
73.80
6.53
3-8B
RIN
25.72
46.30
31.26
3.03
60
312.22
35.26
50.04
33.42
SmoothQuant
23.29
28.28
48.93
29.19
1.57
28 60
54.46
33.37
49.64
33.04
le3
4--166
GPTQ
23.46
32.07
43,79
30.10
2.41
28.00
53.97
34.14
48.86
32.98
6e2
QuaRot
73.73
43.00
76.61
SpinQuunt
47.35
74.12
76.36
75.98
69 88
42 46
77.37
44.47
68.98
64.11
7.28
OSTQuant
48.81
73.48
79.82
75.97
72.62
42.40
78.18
45.75
69.22
65.14
7.24
RÍN
25.72
33.36
46.18
29.83
2.76
78 60
312.45
34,59
50.20
35.18
7ee2
SmoothQuant
23.55
28.96
48.84
28.90
1.44
29. 40
51.09
34.14
50.36
32.96
le3
GPTQ
23.38
32.74
44.34
29.72
2.39
29.80
54.95
34.75
51.30
33.71
6e2
4-4-4
Omniquant
41.53
53.37
QuaRot
42.83
67.42
73.21
72.66
66.93
42.20
75.73
45.19
66.22
61.38
8.18
SpinQuant
46.33
73.57
76.15
75.43
71.40
41.40
79.16
44.68
68.75
64.10
7.35
Quant
78.87
76.01
78.51
19

---

Published as a conference paper at ICLR 2025
Table 13: Complete omparison of the perplexity WikiText2 and averaged accuracy Zero-shot
Common Sense Reasoning tasks on LLaMA.
Model
#Bits W-A-KV
Method
ARC-c
ARC-
BoolQ
HellaS.
LambA.
OBQA
PIQA
SIQA
WinoG. (1)
Avg. (1)
Wikiz
(1)
(1)
(1)
(1)
16-16-16
Full Precision
44.71
72.90
74.98
76.20
73.08
43.80
79.16
45.55
69.93
64.48
RTN
43.17
69.82
73.30
73.75
69.67
42.00
78.13
45.34
68.82
62.67
734
SmoothQuant
40.96
68.60
74.04
73,16
68.74
42,00
78.07
46.11
68.51
62.24
7.46
GPTQ
67.85
76.55
44.37
69.46
60.15
4-16-16
Omuiqquat
42.49
71.38
74.62
74.71
71.98
42.00
79.05
45.96
68.59
63,42
5.86
AWQ
43.86
70.79
74.19
75.27
69.94
43.00
78.45
45.09
69.14
63.30
5.97
QuaRot
42.75
69.99
73.30
75.13
73.55
42.80
78.35
45.14
69.61
63.40
5.83
SpinQuant
43.77
71.17
75.09
72.91
70.72
OSTQuant
44.20
72.56
73.73
75.05
73.45
44.00
78.73
45.45
69.38
64.13
5.81
7B
RTN
23.46
1973T
35.075
29,02
1.24
26.00
3:10
35.
51.30
31.5
765
SmoothQuant
25.17
31.40
51.62
29.73
5.43
28.20
34.44
49.09
34,42
3e2
4-4-16
GPTQ
36.23
QuaRot
40.36
67.26
73 I5
72.89
70.81
42.00
77.97
44.27
67.17
61.76
6.22
SpinQuant
40.19
68.43
72.35
72.91
70.68
41.20
71.75
44.17
68.67
61.82
6.08
OSTQuant
42.58
70.79
72.87
70.77
43.40
77.69
45.04
67.25
62.72
KTN
23.89
29.56
46.67
28.37
L.3
26.40
52.99
35.21
51.34
32.87
lef
Smooohhuantt
23.38
30.18
50.03
29.67
4.89
24.60 60
$1.74
34.75
50.67
33,32
3e2
GPTQ
23.89
27.90
43.88
27.86
L.05
26.20
51.85
34.08
49.49
31.80
2:3
4-4-
Omniquant
66.59
QuaRot
40.27
67.55
72.20
72.59
70.62
39.80
77.20
44.88
65.90
61.22
6.26
SuinQuant
39.08
68.18
73.06
72.87
70.46
40.60
77.42
42.68
67.56
61.32
6.12
OSTQuant
70.33
70.66
42.42
77.91
44.93
62.55
16-16-16
Full Precision
47.87
74.49
77.86
79.10
76.03
44.40
80.30
46.72
73.24
66.67
KIN
45.56
70.66
72.45
76.06
70.58
42.00
78.84
44.99
70.01
63.45
8.60
SmoothQuant
43.86
71.21
71 62.62
74.
69.34
40.00
77.
45.45
70.72
62.69
18.75
GPTQ
72.85
73.27
75.31
70.10
44.60
79.87
71.11
64.36
6.58
4-16-16
Omniquant
47.01
73.86
77.22
77.95
75.59
45.00
79.87 87
46.88
72.61
66.22
5.21
AWQ
47.53
73.86
75.60 75.60
$9.03
78.34
43.40
79.87
45.85
71.67
68.58
5.28
QuaRot
47.18
72.22
76.85
78.07
75.99
45.00
79.76
45.70
72.38
65.91
5.20
SpinQuant
47.44
74.83
77.37
78.
75.55
79.99 92.92
46.01
72.06
66.32
5.16
OSTQuant
48.04
73.86
78. 10
78.28
75.99
45 60
80.52
46.93
72.30
66.62
5.21
13B
RTN
25.85
26.25
42.05 05
26. 70
0.T7
28.00
56.33
34,60
50.67
31.63
264
SmanthQuant
25.43
29.29
$1.56
28. 12
26.00
$3.32
34.34
49.57
13.29
4-4-16
GPTQ
27.78
25.83
0.70
24,20
51.31
36.65
51,70
31.51
3e3
QuaRot
46.93
71.51
75.57
76.63
74.13
42.40
78. 73
45.24
68.98
64
5.50
SpinQuant
45.73
72.56
75.38
76.86
73.28
43,60
78.89
44.63
70.40
64.59
5.36
OSTQuant
RTN
26.28
27.27
42.35
25.85
0.19
26.60
49.95 19.95
34.19
49.25
31.33
3e4
Smoothhuant
24.49
28.83
$1.65
27.91
2.08
26.00
$2.56
35.41
$0,59
33.28
Se2
GPTQ
23.63
27.31
39.85
26.17
0.56
26.00
51.96
35.82
49.57
30.63
3e3
4-44
Omniquant
48.23
56.45
28.76
37.10
55.64
45.63
10,87
QuaRot
46.50
71.55
75 DS
76.43
73.47
45.00
78.78
44.37
70.09
64.59
5.53
SpinQuant
45.99
70.71
76.51
77.16
73.63
45.60
79.00
45.65
70.32
64.95
5.39
OSTQuant
45.90
75.25
76.94
77.21
74.23
43,40
79.43
45.91
70.56
65.43
5.40
16-16-16
Full Precision
52.99
80.39
82.75
82.62
77.59
48.00
82.
47.75
75.69
70.00
4.10
RTN
49.74
73.99
77.89
79.07
72.21
44.20
79.00
45.70
73.88
65.69
6.13
SmoothQuant
48.98
72.94
80.00
79.00
71.49
44.80
78.13
45.96
73.16
65.69 69
5.80
GPTQ
75.97
45.24
4-16-16
Omniguant
52.22
78.62
81 80
81.94
76.85
47.20
81.07
47.54
74.43
69.07
4.25
AWQ
53.24
77.48
81 68
82.29
76.79
48.20
72
48.16
75.37
69
4.28
QuaRot
53.58
78.62
82.11
82 10
48.00
81.72
47.75
76.09
69.73
SpinQuant
52.90
78.49
82.00
82.21
78.28
81.01
75.06
62.62
OSTQuant
53.07
79.12
83.09
82.04
78.58
48.60
81.18
48.06
74.82
69.84
4.19
30B
RTN
25.00
27.95
42.02
27.22
0.21
70
49.
34.65
50.91
3[57
263
SmoothQuant
23.63
30.68
86
31.91
28.20
54.13
34.49
50.04
34.64
le3
4-4-16
GPTQ
27.30
27.19
26.75
0.17
49.02
35.21
47.75
30.88
QuaRot
51.79
76.39
80.76 76
$0.90
77.08
45. 80
80.58
45.60
74.35
68.14
4.57
SpinQuant
50.06
77.06
81.38
80.62
76.79
46.00
80.14
46.37
74.27
68.08
4,53
OSTQuant
78.11
82.48
79.51
75.99
81.18
74.82
68.52
RTN
25.00
28.87
44.07
27.25
0.39
25 60
49.67
34.54
49.35
31.64
2e3
SinoothQuant
22.61
32.87
55.05
11.28
3.40
28.00
53.75
34.65
51.28
34.65
le3
GPTQ
27.22
27.82
39.36
27.13
0.33
24,80
50.71
34.34
47.91
31.07
2c3
4-44
Omniguant
53.79
54.95
26.45
65.56
38.54
53.91
45.04
QuaRot
51.71
76.98
80.95
80.86
77.04
46.20
80.63
45.00
73,32
68.08
4.60
SpinQuant
51.62
76.98
81.07
80.57
76.63
46.00
79.92
46.26
74.19
68.14
4.55
OSTQuant
81.16
80.90
46.11
74.27
20

---

Published as a conference paper at ICLR 2025
Table 14: Complete comparison of the perplexity score on WikiText2 averaged accuracy on Zero-shot
Common Sense Reasoning tasks for LLaMAA270B and LLLMA3-70B.
Model
#Bits W-A-KV
Method
ARC-c
ARC-e
BoolQ
HellaS.
LambA.
OBQA
PIQA
SIQA (f)
WinoG.
Avg. (1)
Wikiz
(1)
(1)
(1)
(t)
(1)
(1)
(1)
(1)
(U)
16-16-16
Full Precision
57.42
81.02
83,79
83.81
79.60
48.80
82.70
49.18
77.98
71.59
3.32
RTN
55.80
79.25
81.35
81.78
75.51
47.60
81.94
46.83
76.48
69.62
3.87
SmoethQuant
50.26
81.53
67.81
73.63
44.40
81.34
44.17
73.64
65.93
5.50
GPTQ
56.91
80.81
82.47
79.06
82.75
48.06
77.51
70.96
4-16-16
Omniquant
57.08
80.81
82.69
83.07
79.18
47.40
83.08
48.87
77.19
71.04
3.47
AWQ
56.67
80.54
82.98
82.54
78.83
47.67
82.97
48.12
77.62
70.88
4.03
Qualkot
80.85
83.24
82.21
77.35
71.21
SpinQuant
56.91
80.60
83.18
83.06
79.16
49 D0
82. 75
48.31
77.11
71.12
3.43
OSTQuant
57.36
81.37
83.20
83.86
79.77
48.73
82.69
48.46
77.89
71.48
3.41
2-70B
RTN
29.35
26.05
37.74
25.97
0.02
24,80
51.31
34.14
48.70
30.90
7e4
SmoothQuant
35.21
51.70
4-4-16 16
GPTQ
27.82
25.80
37 95
25.82
0.00
27.00
49.67
33.98
49.72
30.86
nan
QuaRot
55.29
80.35
81.10
81.87
79.06
45 80
82.05
47.90
76.24
69.
3.78
SpinQuant
55.38
78.96
83.36
82.54
79.00
47
48.67
77.43
70.58
3.68
OSTQuant
56.61
80.51
83.03
82.68
79.1
47.86
83.00
48.76
76.70
70.92
RIN
30.58
27.74
38.
26.12
0.02
24.60
51.74
34,29
$2.49
31.73
7c4
SmoothQuant
24.15
33.88
55.32
31.75
7.14
26.40
54.95
34.14
52.17
35.54
3e2
GPTQ
26.39
50.00
34,44
50.04
4-4-4
QuaRot
56.48
80.56
81 59
81.93
79.16
46.00
82.21
48.00
76.80
70.30
3.80
SpinQuant
56.31
80.64
83.55
82.36
79.41
47.20
82. 21
47.29
76.
70.57
3.61
OSTQuant
56.58
80.17
83.64
82.49
78.72
48.00
48.67
76.49
70.84
16-16-16
Fall Precision
64.42
85.98
85.14
84.95
79.47
48.46
84.39
50.82
80.66
73.81
2.86
RTN
26.28
25.55
37.88 83
26.36
0.00
29.00
50.98
34.70
49.64
31.15
164
SmoothQuant
51.88
77.53
80.09
80.47
73.16
46.60
80.58
45.29
75.85
67.94
6.70
GPTQ
25.29
34.90
4-16-16
Omniquant
48.29
75.42
77.92
77.80
75.59
45.20
80.41
46.62
70.17
66.38
AWQ
52.26
78.95
83 24
81.52
73.05
47.67
81.25
44.43
77.9%
68.93
5.92
QuaRot
62.20
83.88
85.57
84.18
79.04
48.20
83,13
50.10
80.03
72.93
3.53
SpinQuant
84.97
78.30
49.85
80.90
72.90
OSTQuant
63.76
85.82
84.99
85
79.53
48.45
84.26
$1.01
80.22
73.69
3.19
3-70B
RTN
27.47
26, 26
0.DA
27.20
51.63
35.26
49.33
3121
963
SmoothQuant
25.60
34.47
50.46 .6
32.48
30.00
54.24
33.83
48.93
34.67
2c2
4-4-16
GPTQ
25.77
26.09
43.64
26.42
0.00
27.40
52.01
32.55
49.35
31.47
QuaRot
50.60
73.65
77 46
77.83
71.66
43.20
78.13
45.29
71.90
65.56
6.35
SpinQuant
53.84
77.69
80.24
78..19
73.06
45.00
78.67
43.24
73.01
66.99
6.10
OSTQuant
84.56
77.08
50.23
80.13
RTN
27.13
25.42
37.83
26.12
26.66
50.76
35.16
48.38
30.82
SmnoothQuant
23.46
31.48
48.81
29.22
4.13
28.00
52 56
34.95
$1.22
33.76
3e2
GPTQ
26.11
25.17
45.17
26.07
0.00
26.40
48.86
33.88
49.17
31.20
4e4
4-4-
QuaRot
SpinQuant Quant
51.88
76.39
80.98
76.50
71.43
43.46
79.27
44.17
72.69
66.31
6.24
OSTQuane Quant
61.29
82.39
83.43
83.25
75.90
48.93
81.73
51.24
77.01
71.69
4.01
21

---

Published as a conference paper at ICLR 2025
A.6.2 VISUALIZATION RESULTS
Fig T0 illustrates the trends of activation QSUR and evaluation loss during the training process of the LLaMA3-8B model.
11 and Fig 12 shows the activation distribution of different layers in LLaMA-2-7B and
I.LMMA3-88B.
Fig L3 shows the impact of Weight Outlier Minimization Initialization (WOMI) on weight distribution and quantization enror of different layers in LLaMA-2-7B.
Fig presentts visual comparisons the activation distributions and quantization errors for QuaRot, SpinQuant, and OSTQuant selected layers of LLaMA-2-7B..
0.16
OSUR [no/ma amaaalzee
Average model
0.70
Loss
QSUR, of layer20 query
0.70
QSUR of layer20_ down
0.00
0.04
Figure 10: As the number of training iterations increases, the changes in the local and average QSURs, as well as the evaluation loss.
22

---

Published as a conference paper at ICLR 2025
Sye
00
2050
Figure 11: The activation distribution of different layers in LLaMA-2-7B before and after QSuQuant.
23

---

Published as a conference paper at ICLR 2025
1200
200
Figure 12: The activation distribution of different layers in LLaMA-G-8B before and after QSuQuant.
24

---

Published as a conference paper at ICLR 2025
Figure 13: Visualizations comparing of the weight distribution and relative LI of LLaMA- 2-7B quantized with Ori (Ist column), Hadamard transformed(2nd column), and WOMI transformed (3rd column), respectively.
25

---

Published as a conference paper at ICLR 2025
-02
3000
2008
100g
2000
1000
Quarron Sigertt pew..
Spin lagert part
24403122
lagert pame,
Oulest Moye7__ 1o Intative OFTI 1791788775
Spn kay.eyyeeeecc 0.C82300575
C3T Saper C000005055
Okaamt
1600099375
Spin lopert
10015923
0ST layer 1572288625
1.53
LOS
0:00
-025
2003
2000
1000
2000
2000
Quarct layee pobs.
3030000375
Layer gate,
1000
erech 1040922
Span Kyyo
057 11
volutree
218200522
Ocarch Layer
0.3030000575
Spar S3y
2200001223
3350759575
2003
Iidber
Figure 14: Visualizations comparing of the activation distribution and relative LI error of LLaMA- 2-7B quantized with QuaRot (1st column), SpinQuant (2nd column), and OSTQuant (3rd column), respectively. The relative LI error here represents the difference between the activations before and after per-token quantization.
26


Reviews:
[{"review_id": "e5CZmE9zsZ", "reviewer": "Reviewer_347N", "summary": "This paper proposes the OSTQuant approach for LLM post-training quantization (PTQ), which assigns learnable Orthogonal and Scaling Transformations to each fully connected (FC) layer in LLMs. It also introduces the Quantization Space Utilization Rate (QSUR) as a metric to evaluate the quantizability of transformed data. Additionally, the KL-Top loss function is introduced to address the small calibration set issue.", "questions": "- It would be good to see the speedup results on hardware with FP4 support, like Blackwell architecture, to fully show the potential.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The proposed QSUR provides a quantitative index to evaluate the effectiveness of the transformations. It motivates the OSTQuant method, and the results demonstrate its high effectiveness.\n\n- The paper achieves significantly improved results for 4-bit activation, weight, and KV cache quantization.\n\n- The paper is mostly well-written. I particularly like Figure 1 which provides a clear illustration of the concepts.", "weaknesses": "- Since this is a PTQ method, it would be beneficial to demonstrate its effectiveness on larger models, such as a 70B parameter model. Additionally, showing a scaling curve with model size, memory cost, computation cost, and quantization bit would help users understand the overall Pareto frontier between cost and quality.\n\n- Some parts of the paper could be clearer, particularly in detailing certain aspects. For example, while Figure 1 is helpful, it took some time to understand the meaning of the grey dots and ovals.  Similarly, Table 3 lacks details about the inference system and why only prefill speedup is mentioned. Providing more context and explanations would improve the paper’s readability and accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jw7bjOxhcR", "reviewer": "Reviewer_dw3S", "summary": "This work proposes a PTQ technique that reportedly achieves SOTA results on LLMs.  \nThe technique employs an equivalent transformation before quantizer which combines smoothing and rotation in recent work, together with optimization of a top-K KLD loss and a weight-statistics-cognizant initialization of rotation.  \nA local heuristic metric QSUR is introduced to assess the efficiency of quantization.  \nExperiments and ablation studies were carried out with popular LLMs.", "questions": "* One major question I have is on the relative contributions by the 3 new ingredients in the proposed method: the equivalence pair reparameterization, the top-K KL loss, and the initialization.  The authors did conduct ablation study on the special initialization, but there seems to be a lack of ablation on the loss.  In other words, it is not clear how much the differential results in Table 2 comparing against other methods are due to the combination of smoothing and rotation, and due to the top-K KLD loss--what would be the result if one applies SpinQuant but optimizing the top-K KLD?  \n* Another major question that I have is the role of QSUR.  The authors provided theoretic results on the effect of linear transformation on this local metric assuming Gaussianity--this is good.  But in practice, QSUR is not an objective function for direct optimization in conducting OSTQuant, but rather, OSTQuant minimizes a global KLD loss.  This begs the question: how do the global loss align with local QSURs?  During the course of OSTQuant optimization, if one measure QSUR across various layers in the network, do they all monotonically grow as the global loss monotonically decreases?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ Well-motivated idea.\n+ Adequate presentation of background and recent related work.\n+ Comprehensive experiments.", "weaknesses": "- Certain ablation studies are still missing that are necessary to establish the contributions from different components of the proposed technique (see below).  \n- Quality of writing could use further improvement.  For example, there seems to be a mislabeling of $\\Lambda$ as $A$ in equations (9) and (10); certain figure legends, such as Fig. 6, are too tiny to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jx4x0TYYif", "reviewer": "Reviewer_n22x", "summary": "The paper introduces QSUR and proposes OSTQuant that improves quantization performance by globally optimizing multiple equivalent transformation pairs in LLMs with KL-Top loss. Each transformation consists of a scaling transformation and an orthogonal transformation.", "questions": "1. Can the authors showcase the effect of initializing the transform matrix with the theoretically derived form?\n\n2. The experiment results on larger models such as LLaMA-2-70B and LLaMA-3-70B are missed in the paper, can the authors provide more experiment results on these models?\n\n3. The decoding speedup is also missed in the paper. It would be helpful to include latency analyses on the decoding latency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper mathematically derives a new metric QSUR to quantify the utilization of the quantization space, which is experimentally found to be positively correlated with the accuracy of the quantized model. Based on QSUR, the authors define the optimal transform matrix, which inspires them to combine the scaling and orthogonal transformation to minimize the quantization space. By incorporating scaling transformation and KL-Top loss, OSTQuant achieves SOTA results across various quantization settings while maintaining a low calibration cost.", "weaknesses": "While having mathematically derived the form of an optimal transform matrix, OSTQuant only applies weight outlier minimization initialization to the orthogonal transformation while leaving scaling transformation behind, incurring a small gap between the theoretical analysis and the method. Besides, it would be helpful to show the improvements of applying weight outlier minimization initialization compared with Hadamard matrix. It would also be beneficial to include experiment results on larger models (e.g. 70B) and decoding speedup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxXCQnLG6Q", "reviewer": "Reviewer_uSGo", "summary": "This paper presents a mathematical framework to evaluate the effectiveness of smooth and rotation transformations in the context of outlier suppression during LLM quantization. It introduces a metric, QSUR, to assess the efficacy of various transformations and derives a closed-form solution for the optimal transformation. Based on QSUR, the paper proposes OSTQuant, a method that optimizes the combination of smooth and orthogonal transformations globally and employs KL-Top loss to mitigate noise during PTQ optimization.\n\nThere are some concerns regarding the methodology and assumptions in the paper. \n\n(1). Quantization Assumption: The paper assumes the volume of the quantization space $V_{S_{X}}$ as that of a cube when dimension $d=3$, which implies per-tensor quantization. However, the standard practice is to use per-token quantization for activations and per-channel quantization for weights. The volume of per-channel quantization is typically a cuboid when dimension $d = 3$. The authors should clarify if their assumptions are based on these scenarios and how their methodology applies to these more common quantization schemes.\n\n(2). Mean Vector Neglect: In line 231, the paper states,\" we neglect the mean vector $µ$, so $λ_{max} = λ_{min} = λ$\". The causal relationship between neglecting the mean vector and the equality of the maximum and minimum eigenvalues is not evident and requires further explanation.\n\n(3). Optimization of Transformations: The paper optimizes the pair of smooth and orthogonal transformations, with the transformation initialized by the closed-form optimal derivation. It is unclear why it is necessary to further optimize a transformation that is already designed to reach maximum QSUR. Additionally, the paper lacks an ablation study on the impact of different orthogonal initialization methods.\n\n(4). Correlation of QSUR and Model Performance: Figure 3 shows a positive correlation between QSUR and the performance of the quantized model. However, it is not demonstrated whether a higher QSUR value consistently leads to higher performance in quantized models. The paper should provide a more rigorous validation to establish this relationship across various datasets and architectures.\n\n(5). Hadamard Transformation: The paper uses online Hadamard transformation for query and key. It is unclear whether the Hadamard transformation can be optimized as suggested in Figure 5.", "questions": "(1). Why is the closed-form optimal transformation derived from QSUR not directly applied in the model?\n\n(2). Does a higher QSUR value guarantee a better transformation in terms of model performance?\n\n(3). Which of the transformation matrices mentioned in the paper are Hadamard matrices and which are unit orthogonal matrices, and are Hadamard matrices trainable as well?\n\n(4). Should softmax be applied before or after the top-k selection, and what is the rationale behind this choice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper provides a mathematical framework that attempts to understand the role of smooth and rotation transformations in quantization, which is a novel approach. The introduction of QSUR offers potential insights into the quantization process, and the closed-form solution for the optimal transformation could serve as a theoretical foundation for further research. The integration of orthogonal and scaling transformations within OSTQuant could address issues related to outliers in activations and weights, which is a significant contribution.", "weaknesses": "A major concern is that, despite deriving a closed-form solution for the optimal transformation based on QSUR, the paper opts to use learnable transformation parameters initialized by the optimal transformation instead of directly applying the theoretically derived optimal transformation. This choice should be justified.\n\nWhile the paper establishes a positive correlation between QSUR and the performance of the quantized model, it is unclear whether a higher QSUR necessarily indicates a superior transformation. There is a lack of rigorous validation demonstrating that transformations with higher QSUR values consistently lead to better model performance across various datasets and architectures.\n\nThe hypervolume $V_{S_{X}}$ of the quantization space $S_{X}$ is calculated using per-tensor quantization, whereas common practice typically involves per-token quantization for activations and per-channel quantization for weights. The authors need to clarify how QSUR applies to these more conventional quantization schemes and validate their metric in these contexts. Without this clarification, the applicability and robustness of QSUR are questionable.\n\nIn summary, while the paper provides interesting theoretical contributions, there are substantial issues with the practical application of the derived optimal transformations and the validity of QSUR as a reliable indicator of transformation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ab1F5XG4Y8", "reviewer": "Reviewer_NPKd", "summary": "This paper introduces the Quantization Space Utilization Rate (QSUR) to assess data quantizability and guides the development of Orthogonal and Scaling Transformation-based Quantization (OSTQuant). OSTQuant optimizes weight and activation distributions in the quantization space and includes a KL-Top loss function to enhance semantic retention in post-training quantization. It outperforms state-of-the-art methods, maintaining 99.5% accuracy in simple configurations and significantly reducing performance gaps in more complex settings.", "questions": "1. Could the authors provide the speedup achieved during the decoding stage after implementing their proposed methods?\n2. The paper should describe the initialization process for the scaling vectors used in the transformations. \n3.  How does the model perform if only the initial matrix and vectors are employed to transform the model followed by naive quantization without training?\n4. The performance improvement when compared with SpinQuant using a 4-4-4 quantization scheme appears to be less obvious in most cases than when using a 4-4-16 scheme. Can the authors explain why there is a disparity in performance improvements between these two configurations?\n5. Regarding the activation distributions post-transformation, is there a noticeable difference among OSTQuant, SpinQuant, and QuaRot? It would be beneficial if the authors could include visualizations for both SpinQuant and QuaRot.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. The paper introduces a new metric, QSUR, that links distribution characteristics with quantization space utilization, providing insight into the impact of transformation-based quantization approaches.\n2. The proposed method for improving QSUR surpasses previous state-of-the-art techniques in performance.", "weaknesses": "1. Line 087 incorrectly identifies the two methods as block-wise reconstruction methods, which they are not.\n2. In Equation (21), the direction of the inequality sign is incorrect. This error affects the subsequent proof, which also requires modification.\n3. In Equation (4), the terms $\\lambda_{min}$ and $\\mathbf{q}\\_{min}$ should be replaced with $\\lambda\\_{max}$ and $\\mathbf{q}\\_{max}$ respectively, in order to correctly derive $\\mathbf{\\mathit{v}}\\_{min}^{org}$ for use in Equation (5). Please refer to the simple example shown in Figure 1(a). Corresponding changes are needed in the related sections of the paper.\n4. The paper assumes that weights and activations are normally distributed. This assumption limits the generalizability of the results. The Laplace distribution, commonly used to model tensors, especially for outliers, is not considered. For further reference, see [this link](https://mobiusml.github.io/hqq_blog/).\n5. Equation (29) appears to contain a clerical error. If this is not the case, a detailed explanation from the author would be beneficial.\n6. The derivation in Section A.2.3 does not convincingly ensure that $|\\mathbf{q}\\_{max}|=d^{-\\frac{1}{2}}$, which is necessary to obtain Equation (33).\n7. Line 335 mentions \"Figure X,\" which should be corrected to \"Figure 5\".\n8. The use of the scaling vector $\\mathbf{S}^i_{u|g}$ in Figure 5 under non-linear activation functions is not equivalent, and it is not employed in current studies. The author should explain the rationale behind using this vector.\n9. The notations for vectors and matrices throughout the paper are inconsistent. For example, $S_{updown}$ in Table 4 versus $\\mathbf{S}\\_{u|d}^i$ in Figure 5. The paper should clarify the relationships among $\\mathbf{S}\\_{ffn}^i$, $\\mathbf{S}\\_{attn}^i$, and $\\mathbf{S}^i$.\n10. In Figure 5, $\\mathbf{S}\\_{qk}^i$ and $\\mathbf{S}\\_{ov}^i$ appear non-trainable. The paper does not provide an explanation for this choice.\n11. Table 1 is not illustrated anywhere in the paper.\n12. The paper does not include performance comparisons using larger Language Models, such as the 70B LLaMA-3, or investigate other model families and architectures, such as Mistral or Mixtal (MoE). Such comparisons could provide a broader evaluation of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)