LRQuant: Learnable and Robust Post-Training Quantization for Large Language Models (2024)

Abstract: Post-training quantization (PTQ) for large language models (LLMs) significantly accelerates model inference and relieves memory constraints, without incurring model training. A “smoothing paradigm” is commonly used in LLM quantization, which transfers the quantization difficulty of activation to weight quantization using mathematically equivalent transformations. However, existing methods face two issues: 1) Most smoothing parameters are hand-crafted defined which leads to suboptimal results; 2) There are significant performance degradations when tested on unseen datasets. To address these challenges, this paper introduces a robust learnable smooth-based PTQ framework, called LRQuant. Firstly, we consider a learnable paradigm to find optimal smoothing parameters which are initialized by logarithmic activation equivalent. In addition, we empirically found that only relying on MSE loss could hardly lead to optimal quantization results, and we then propose a novel loss function based on the negative logarithm of cosine similarity (NLC loss) between outputs of full-precision and quantized block. At last, we pioneeringly introduce Testtime adaptation (TTA) into LLM quantization, which allows for rapid model adaptation during testing to improve generalization performance. More surprisingly, we find that by using our TTA method, we can achieve better results on test sets than directly using test sets for calibration in some cases while avoiding catastrophic forgetting. Codes are available at https://github.com/zjq0455/RLQ.

heading: 1 Introduction
text: Quantization (Esser et al., 2019; Chee et al., 2023; Dettmers et al., 2023a,b) is a well-known model compression technique converting the weights of large language models (LLMs) and activations from full precision to lower-bit representations.
*Corresponding author
Among various methods, Post-Training Quantization (PTQ) (Nagel et al., 2020; Hubara et al., 2021; Yao et al., 2022) is the most popular for LLMs, as it doesn’t require retraining the model and offers fast quantization, saving computational resources. For instance, GPTQ (Frantar et al., 2022) uses only an A100-80G GPU to quantize BLOOM-175B (Workshop et al., 2022) within 4 hours.
Previous PTQ methods exhibit uncommendable performance when confronted with more challenging configurations, such as W4A4, due to larger activation distribution variances. A new strategy, known as “smooth quantization” (Xiao et al., 2023), has been recently introduced to address this by shifting the quantization difficulty from activations to weights. However, current smoothing strategies, such as hand-crafted scaling factors (Xiao et al., 2023; Frantar et al., 2022) and uniform zero points (Wei et al., 2023) are all predefined, which usually lead to suboptimal. The intuitive proposition of learning the smoothing parameters, coupled with the adoption of MSE loss which aims to optimize quantization output by evaluating the similarity in magnitude, is an easily conceived solution. We conducted an example experiment and summarized the cosine similarity between the outputs of each quantized block (MSE-guided learning(Shao et al., 2024) and predefined method (Xiao et al., 2023)) and full-precision block. As shown in Figure 1, there are large differences, both for the predefined and the MSE-guided learning method, demonstrating that except for MSE, another learning strategy for better optimizing is essential.
In addition, a prevalent issue across prior PTQ methods is that quantization on the calibration dataset leads to suboptimal performance when subsequently applied to unseen datasets. For instance, quantize a LLaMA based on WikiText2 (Merity et al., 2016) and evaluate on PTB (Marcus et al., 1994). Meanwhile, Test-time adaptation (TTA) is an increasingly promising technique to improve
2240
model generalization capacity (Wang et al., 2020; Niu et al., 2022, 2023), which enhances the model’s robustness during testing without requiring retraining. However, according to our investigation, there is currently no method that utilizes TTA to address the aforementioned issue in PTQ methods.
Considering all the aforementioned aspects, this paper introduces a novel PTQ framework named LRQuant. Firstly, we define learnable parameters and devise a novel block-wise loss function, named negative logarithm of cosine similarity loss (NLC loss) considering directional differences of outputs, to guide the learning of all learnable parameters. As depicted in Figure 1, this modification significantly improves our quantization performance. Additionally, LRQuant is the first to incorporate TTA strategy into LLMs quantization aiming to fortify models’ robustness on unseen test sets. Rather than adapting the learnable parameters of the whole model based on the unseen data where we observe catastrophic forgetting, LRQuant only updates the learnable parameters of the last block to effectively
balance performance and forgetting. Figure 2 illustrates an approximate performance comparison between our LRQuant and previous PTQ methods. More excitedly, we observe that with our TTA method, the adapted model even outperforms the quantized model directly calibrated by the test set in some cases. This discovery implies that if a quantized model needs to be applied to different scenarios, there will be no need to re-calibrate and re-quantize the full-precision model from scratch.
Our key contributions can be summarized as:
• We present a learnable PTQ framework for LLMs (called LRQuant), by setting smoothing parameters into learnable with logarithmic activation equalization initialization.
• We propose a novel block-wise loss function called NLC loss, which is based on the negative logarithm of cosine similarity between outputs of full-precision and quantized blocks, to assist in correcting the output directions rather than only the magnitude by MSE loss.
• LRQuant is the first to introduce TTA into PTQ by updating the learnable parameters of the last block based on test data, which enhances the performance on unseen test data in some cases, while avoiding catastrophic forgetting on source data. Remarkably, our adapted quantized model exhibits superior performance compared to calibration using test sets from scratch.

heading: 2 Related Work

heading: 2.1 Smooth-Based Post-Training Quantization
text: “Smooth” is a solution proposed to address the challenge of activation quantization difficulty (Xiao
et al., 2023). It is based on mathematical equivalent transformations, where the activations with large variances across different channels are divided by scaling factors. Correspondingly, the weights which are generally easier to quantize are multiplied by scaling factors. Outlier Suppression (Wei et al., 2022) finds that the scaling parameter γ in LayerNorm (LN) layer of LLM is a key factor influencing the distribution of activation value outliers. Therefore, Gamma Migration is proposed, which involves removing γ from LN and moving it to the weights in the next layer and the short branch. Outlier Suppression+ (Wei et al., 2023) introduces channel-wise shifting, which eliminates the asymmetry in activation distribution and reduces the range of the tensor. SmoothQuant (Xiao et al., 2023) is the first method to introduce channelwise scaling factors designed based on the maximum value. FPTQ (Li et al., 2023) improves the calculation method of scaling factors by utilizing non-linear offline logarithmic activation equalization (LAE) to adjust the distribution of activations, making the activation distribution moderate. OmniQuant (Shao et al., 2024) stands out as the latest smooth-based PTQ method which defines learnable smoothing parameters. However, OmniQuant uses the basic channel-wise maximum values as the initial scaling factors and only use MSE loss to update, means that its performance can not reach the optimal level. This paper also defines learnable parameters where smoothing parameters are initialized with LAE. Furthermore, we design a novel loss function namely NLC loss based on cosine similarity considering the directional differences of outputs to update the learnable parameters.

heading: 2.2 Test-Time Adaptation
text: Test-time adaptation is a technique to adapt models to unseen datasets to improve robustness during testing. Unlike fine-tuning (Wang et al., 2017; Howard and Ruder, 2018; Hu et al., 2021), it is source-free and doesn’t require additional labels from test data. Tent (Wang et al., 2020) minimizes the prediction entropy generated by the model during testing to reduce generalization error. EATA (Niu et al., 2022) believes that excessive adaptation can lead to catastrophic forgetting of the source data and proposes a sample selection method to choose samples with high contribution to adaptation from the test sets for entropy minimization. On this basis, SAR (Niu et al., 2023) proposes removing noisy samples with large gradients during
testing and enhancing robustness to noise through sharpness-aware learning on the remaining samples. TS (Park et al., 2023) changes the traditional approach of updating model parameters and instead performs style shifting on test samples to make them resemble the nearest source domain.
In this paper, we are the first to introduce TTA into LLMs quantization, aiming to improve the model’s generalization capability on unseen test sets while avoiding catastrophic forgetting and outperforming re-calibration from scratch.
3 LRQuant
In this section, we provide a detailed introduction to our LRQuant as shown in Figure 3. We first describe the learnable parameters and NLC loss and then introduce our TTA approach for quantized models on unseen datasets.

heading: 3.1 Learnable Parameters
text: Learnable Weight Quantization. For weight quantization, the general function is as follows:
Wq = clamp(⌊ W S ⌉+ z, 0, 2b − 1), (1)
where ⌊.⌉ means round-to-nearest operation. b denotes the target bit-width. Wq is the quantized weight and W is the full-precision weight. S is the step size and z is called zero-point, and we follow
Lin et al. (2023) to learn these two parameters to improve performance, which can be elaborated as:
S = αmax(W)−βmin(W) 2b−1 , z = −⌊ βmin(W) S ⌉, (2)
where α and β are learnable quantization parameters (Shao et al., 2024) to control the upper and lower bound of clipping range respectively and they are constrained within the range of [0, 1].
Learnable Activation Smoothing Quantization. Smooth-based PTQ methods are supposed to smooth the distribution of activations to mitigate the impact of outliers under the condition of outputting exactly equal results in mathematical form. For linear layers, this can be expressed as:
X’W’ + B’ = XW + B, (3)
where X, W, B and X’, W’, B’ are activation, weight, bias before and after equivalent transformation respectively. In more detail:
X’ = (X − zs)diag(s)−1, (4)
W’ = diag(s)W, (5)
B’ = B + zsW, (6)
where s and zs are scaling factors and shifting factors (Wei et al., 2023) to narrow down the activation distribution range. We should notice that in order to calculate easily and implement smooth quantization in codes, for activation quantization, s and zs are fused into the weights of previous LN or linear layers, and for weights, s can be absorbed into themselves.
As is well-known, the initial values of hyperparameters have a significant impact on the final results (Glorot and Bengio, 2010). Inspired by (Lin et al., 2023) and (Li et al., 2023), the impact of the smoothing process on weight quantization is consistently smaller compared to the benefits obtained by smoothing activation and the use of logarithmic equivalence proves to be more effective in suppressing activation outliers. In this way, we define the initial scaling factor s as:
s0i = max(|xi|)/loga(a+ max(|xi|)), (7)
where i is the index of the input channel. We set a as 2 followed by (Li et al., 2023), while s0i is just an initial value as our scaling factor is learnable.

heading: 3.2 NLC Loss
text: Block-wise quantization is commonly used in PTQ for LLMs, which quantizes the whole model block by block, to avoid excessive GPU memory usage. The MSE is usually adopted to guide the quantization process:
LMSE = ∥F (W,X)− F (Qw(W), Qa(X))∥22, (8)
where F (·) is the embedding function of a block. Qw(·) represents the weight quantizer with both learnable quantization parameters Θq (short for α and β) and smoothing parameters Θs (short for s and zs) while Qa(·) denotes the activation quantizer with only smoothing parameters Θs.
However, MSE loss only measures the magnitude similarity of vectors without considering the direction similarity. Hence, this paper is motivated by maximizing the cosine similarity between the output of each full-precision and quantized block, to improve directional similarity. We leverage the negative logarithm of cosine similarity for the outputs as the loss function called NLC loss LNLC :
LNLC = −log(C(F (W,X), F (Qw(W), Qa(X)))), (9)
where C(·) is the cosine similarity operator:
C(A,B) = A ·B∥A∥∥B∥ . (10)
In the end, we combined MSE loss and NLC loss, and our final optimization objective is:
Θ∗q ,Θ ∗ s = argmin
Θq ,Θs
(LMSE + LNLC), (11)
where Θ∗q and Θ ∗ s are the optimal learnable parameters. Our NLC loss approaches 0 when the quantized and the full-precision outputs are close to be the same, so as guiding the quantization process. As illustrated in Figure 1, with NLC loss, LRQuant has significantly outperformed MSE-only method.

heading: 3.3 TTA for Quantized Models
text: In most existing PTQ methods for LLMs, the pretrained large model is first quantized based on the calibration dataset, which is then directly evaluated on unseen test datasets. However, when applying the quantized model to an unseen dataset with domain shift, it may cause performance degradation. A naive solution is to re-calibrate the whole quantized model based on the test dataset, while which is time-consuming. In addition, we empirically
found that, adapting the entire model using test data will inevitably lead to catastrophic forgetting of source calibration knowledge, leading to performance degradation. Therefore, balancing model adaptation performance and catastrophic forgetting is crucial. To address this issue, we innovatively introduce the idea of TTA into LLM quantization.
Considering that the most task-relevant block in the model is the last one, we choose to only adapt the last block of the quantized model based on the unseen test data. As illustrated in Figure 3, in the TTA phase of our LRQuant, we only extract the last block from the full-precision model for quantization based on the test data, and we still utilize a combination of MSE loss and NLC loss to update the parameters with only a few epochs. The adaptation goal is as follows:
Θl∗q ,Θ l∗ s = argmin
Θlq ,Θ l s
(LMSE + LNLC), (12)
where Θl∗q and Θ l∗ s are the learnable parameters of the last block after adaptation, where we set 5 as the number of TTA epochs. This entire process is source-free, and the time required for each adaptation process on test sets is within one minute.
An intuitive question is why only update the learnable parameters of the last block rather than the entire model. Here we describe the two advantages of our TTA scheme over re-calibration for PTQ. First, it is much more time-consuming and challenging to update the whole model than only the last block. Second, re-calibrating the whole model based on the test set can lead to catastrophic forgetting which significantly alters the suitable quantization parameters for source calibration data. Differently, our TTA scheme only re-calibrates the last block, which can somehow retain source knowledge. Moreover, our experimental results in Table 6 show that, compared to calibrating the whole fullprecision model using test sets from scratch, our TTA scheme can achieve much better performance on unstable target sets such as PTB, which maintains its performance on the source calibration set even after adapting to the new test dataset. In the end, we present the entire LRQuant pseudocode as Algorithm 1 at Appendix A.

heading: 4 Experiments
text: In this section, we conduct extensive experiments to validate our LRQuant achieves the following objectives: (1) Demonstrates outstanding performance in challenging quantization tasks (W4A4
and W6A6) through the presented learnable parameters and NLC loss. (2) Enhances model performance on unseen test sets using our TTA scheme.

heading: 4.1 Experimental Setup
text: Baseline. Because LRQuant is a smooth-based PTQ method, we select the same type but predefined methods SmoothQuant (Xiao et al., 2023), LAE (Li et al., 2023) and learnable approach OmniQuant (Shao et al., 2024) as baselines. It is important to note that, due to the lack of data from original papers and to ensure consistency with our LRQuant experimental environment, all results of other methods are reproduced as their settings. Models. We mainly choose LLaMA (7B,13B and 30B) (Touvron et al., 2023a) and LLaMA-2 (7B,13B) (Touvron et al., 2023b) to evaluate our LRQuant, since LLaMA and LLaMA-2 families are currently the most popular and widely applied LLMs. Additionally, we also conduct experiments on OPT (1.3B, 2.7B, 6.7B) (Zhang et al., 2022), which can be found in Appendix D. Training. For learnable smoothing parameters, we initialize the channel-wise shifting factors as Wei et al. (2023) and the scaling factors with Eq.(7), where we set the base a as 2. We utilize the AdamW optimizer (Loshchilov and Hutter, 2017) with zero weight decay to optimize the learnable smoothing parameters and quantization parameters with learning rate 1e-3 and 1e-2 respectively. Our calibration data consists of 128 random 2048 token-segments from WikiText2 (Merity et al., 2016). Using a batch size of 1, all processes are performed on one NVIDIA A100-40G GPU. The calibration training process consists of 20 epochs, while the TTA process involves 5 epochs. Datasets. Following most smooth-based methods (Shao et al., 2024; Xiao et al., 2023), test data comes from WikiText2, Penn Treebank (PTB) (Marcus et al., 1994), and C4 (Raffel et al., 2020) for language generation tasks. For evaluating the performance on zero-shot tasks, we select several popular tasks including PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019) and Winogrande (Sakaguchi et al., 2021) using lmevaluation-harness (Gao et al., 2023).

heading: 4.2 Experiments on Language Generation Tasks
text: The core competency requisite for LLMs is to comprehend and generate language. Hence, to vali-
date the first objective—achieving superior performance in challenging quantization tasks—we initially compare a crucial metric for language generation tasks, perplexity, with the baselines.
The outcomes presented in Table 1 highlight a significant superiority of our learnable method over predefined methods in all W4A4 experiments. When compared to another learnable method, OmniQuant, across WikiText2, C4, and C4-new tasks, LRQuant exhibits a noteworthy reduction in perplexities by an average of 2.00, 3.20, and 3.79, respectively. Notably, on PTB and PTB-new tasks where OmniQuant faces substantial challenges, LRQuant demonstrates heightened stability. A more visually intuitive representation of the cosine similarity comparison with OmniQuant is depicted in Figure 1. W6A6 results can be found in Table 11 at Appendix D and OPT results can be found in Table 13 at Appendix E. These findings underscore the efficacy of learnable parameters and NLC loss.

heading: 4.3 Experiments on Zero-Shot Tasks
text: In addition to language generation, zero-shot tasks reflect the model’s ability to handle unseen problems, also serving as a crucial metric for characterizing model performance. To further validation, we
compare zero-shot accuracies with baselines. As indicated in Table 2, similar to the results observed in language generation tasks, our LRQuant consistently outperforms all predefined methods. Moreover, our LRQuant also exhibits superior performance compared to OmniQuant in most cases, showcasing an average performance increase range of 3.22%∼8.95%. Results and analysis for W6A6 can be found in Table 12. Combining the experimental results from the two subsections, we deduce that our LRQuant attains the state-of-the-art level in challenging quantization tasks. Additionally, in order to further demonstrate the superiority of our method on weight-only tasks, we conduct corresponding experiments in Appendix G.

heading: 4.4 Ablation Experiments
text: Having showcased the superior performance of LRQuant in challenging quantization tasks, our objective extends to providing additional evidence for the effectiveness of our two innovations: learnable parameters and NLC loss. To substantiate this, we conduct ablation studies on the LLaMA-7B at W4A4, and the results are presented in Table 3, where LAE and NLC represent the initialization methods for learnable smoothing parameters and
our NLC loss, respectively.
The first row in the table is same as OmniQuant which only uses the MSE loss. It is evident from the second and third row that each innovation in our LRQuant is effective. Although LAE + NLC group achieves optimal performance on PTB, which shows mediocre results on WikiText2 and C4. Conversely, LAE + MSE + NLC demonstrates the overall best performance, which is selected as our default setting.
We also conduct experiments to demonstrate the effectiveness of smoothing strategy. As illustrated in Figure 4, it can be observed that after smoothing the magnitude of the activation values has significantly decreased and shows much better performance, especially for those channels with more prominent outliers.

heading: 4.5 Experiments on Test-Time Adaptation Performance
text: After achieving the first validation objective, the subsequent goal is to verify whether our TTA method can enhance the performance on unseen datasets. We compare the perplexities of LRQuant
before and after adaptation, where the model is quantized based on WikiText2 while adapted and evaluated on remaining datasets. As illustrated in Table 5, although a little worse on C4, perplexities of almost all adapted models with our TTA scheme have shown a certain degree of reduction on PTB, which intuitively proves the effectiveness of our method.
In addition, we compare our TTA scheme with directly calibrating the whole model using PTB. In Table 6, Calibrate means the model is directly calibrated and quantized using PTB and then immediately evaluated on it, while TTA means our TTA scheme where WikiText2 is the calibration set and adapt on PTB. The calibration of the whole model requires 300 × more time than TTA adaptation. More surprisingly, our TTA scheme can
outperform directly calibrating the whole model with the test sets in some cases. For other cases with slight decreases, our method remains comparable performance but shows hundreds of times faster than recalibration (see Table 7). In summary, we consider that our TTA scheme can significantly promote the deployment efficiency of quantized models in new scenarios while ensuring effectiveness.
Last, we conduct experiments to evaluate our TTA scheme in relieving catastrophic forgetting. As indicated in Table 8, we evaluate the W4A4 performance on WikiText2 to compare re-calibrating the model from scratch (reCalib) with our TTA method (TTA) after adapting on new datasets. As shown, the performance on the original WikiText2 deteriorates after re-calibrating on new test sets (C4 and PTB). Differently, our TTA can retain its performance on the original dataset, verifying its effectiveness in mitigating catastrophic forgetting. W6A6 results are in Table 14 at Appendix F.

heading: 5 Conclusion
text: In this paper, we propose a robust smoothbased PTQ framework namely LRQuant. Firstly, LRQuant defines learnable smoothing parameters and initializes them using LAE. Subsequently, we introduce a novel block-wise loss function namely NLC loss which further considers the directional similarity of outputs. The experiments substantiate that our LRQuant attains state-of-the-art levels on challenging weight-activation quantization tasks. Moreover, we pioneeringly proposed a TTA scheme for LLM quantization to improve generalization performance on unseen datasets by adapting the learnable parameters of the last block based on the test data. The experiments indicate that our method not only relieves catastrophic forgetting but also surpasses re-calibration using the target set, thereby significantly promoting the deployment efficiency of quantized models in some cases.
Limitations
Our LRQuant defines learnable smoothing parameters and introduces a novel loss function based on cosine similarity so it has become state-of-theart weight-activation PTQ method. However, due to hardware limitation we have not applied our method to quantize larger LLMs with over 100 billion parameters. This will be added to our list of future work.
Additionally, our LRQuant is the first to introduce TTA into LLMs quantization which has successfully improved the quantized models’ performance on unseen datasets. However, we do not draw inspirations from state-of-the-art TTA methods which makes our performance may not achieve optimal, such as on C4 dataset our TTA scheme still stands behind original results. In future work, we aim to explore ideas from other methods and
propose a more effective TTA scheme for LLMs quantization.
Ethics Statement
This paper introduces solutions to the challenges associated with Large Language Models (LLMs) quantization, with the overarching goal of facilitating the widespread adoption and application of LLMs. In the current landscape, ethical concerns tied to LLMs, including the presence of hidden biases encoded in the models, are garnering heightened attention. Following our investigation, we assert that our proposed method does not further amplify the biases and contravene any ethical standards.

heading: Acknowledgement
text: Miao Zhang was partially sponsored by the National Natural Science Foundation of China under Grant 62306084 and Shenzhen College Stability Support Plan under Grant GXWD20231128102243003. Xuebo Liu was sponsored by CCF-Tencent Rhino-Bird Open Research Fund.

heading: A The Full Algorithm
text: Algorithm 1 Overall algorithm of LRQuant Input: full-precision LLM model M; calibration set X; test set Xt Output: quantized LLM model after TTA 1: Xfp = Xq = X 2: for Bi in M do ▷ Block-wise quantization 3: Xfp = Bi(Xfp) 4: Init learnable parameters Θq and Θs ▷ Θs with Eq. (7) 5: for k in epochs do 6: for (xq, xfp) in (Xq, Xfp) do 7: B′i = Quantize (Smooth(Bi,Θs),Θq) 8: x′q= B ′ i(xq) 9: loss = MSE(x′q, xfp)+NLC(x ′ q, xfp) ▷ NLC(·) with Eq. (9) 10: loss.backward() ▷ Update Θq and Θs 11: end for 12: end for 13: Bqi = Quantize (Smooth(Bi,Θs),Θq) 14: Xq= Bqi (Xq) 15: end for 16: Xtfp = Xtq = Xt ▷ TTA phase 17: for (Bi, Bqi ) in (M, Mq) do 18: if i == len(M) then ▷ The last quantized block 19: Xtfp = Bi(Xtfp) 20: Init Θlq and Θls 21: for k in 5 do 22: for (xtq, xtfp) in (Xtq, Xtfp) do 23: B′i = Quantize (Smooth(Bi,Θls),Θlq) 24: xt′q = B ′ i(x t q) 25: loss = loss(xt′q , xtfp).backward() 26: loss.backward() 27: end for 28: end for 29: Bqi = Quantize (Smooth(Bi,Θls),Θlq) 30: else 31: continue 32: end if 33: end for 34: return quantized model Mq

heading: B Hyperparameter Analysis on the Base of LAE
text: In this paper, we set 2 as the base of our learnable scaling factors initialization function, which is the same with (Li et al., 2023). In addition, we also compare several other bases (e, 5, and 10) to quantize LLaMA-7B at W4A4, and the experimental results (perplexity) are shown in the Table 9.
From the table, it can be observed that using dif-
ferent bases yields similar results. Even setting the base as a learnable parameter does not lead to a significant improvement in performance. Therefore, to save computational resources, we choose 2 as the base which shows relatively better performance.

heading: C Weighted Combination of MSE and NLC Loss
text: To validate whether the weighted combination of MSE and NLC loss in LRQuant achieves better performance, we further select different combination ratio for comparison as indicated in Table 10.
From the results, it can be observed that even with the weighted combination method, the performance does not improve and is even inferior to evenly combination. Therefore, we select an evenly combination of two loss functions for our LRQuant.

heading: D LLaMA and LLaMA-2 Experiments at W6A6
text: Table 11 and Table 12 shows W6A6 experimental results corresponding to Section 4.2 and Section 4.3 respectively. As illustrated in Table 11, LRQuant achieves the best performance in most experiments. Additionally, as demonstrated in Table 12, LRQuant achieves optimal results in some experiments, while in the remaining ones, it obtains near-optimal performance, also demonstrating an advanced level. Combining the experiments for W4A4 in the content, we can conclude that LRQuant has become the current state-of-the-art method for weight-activation quantization.

heading: E Perplexities on OPT Families
text: We also evaluate perplexities of our LRQuant on OPT families (1.3B, 2.7B, 6.7B). The results are shown in Table 13. From the table, we know that our method outperforms all the predefined methods and for another learnable method OmniQuant there can be clearly seen a moderate decrease on each
comparison so it proves that our LRQuant is able to applied for accurately and effectively quantizing OPT models.

heading: F W6A6 Catastrophic Forgetting Experiments for Test-Time Adaptation
text: Be similar with W4A4 results in Table 8 at Section 4.5, the performance on WikiText2 of models recalibrated by C4 or PTB is all surpassed by our method on W6A6, where we use C4 or PTB to adapt the last block. The comparison results can be found in Table 14. Thereby, we can assert further that our method relieves catastrophic forgetting through the implementation of our TTA scheme.

heading: G Weight-Only Experiments
text: Our LRQuant is mainly focusing on the more challenging weight-activation quantization while quantization methods such as AWQ and GPTQ belong to weight-only quantization. To demonstrate the superiority of our proposed method on hardwarefriendly quantization tasks, we also add additional weight-only experiments as shown in Table 15 and Table 16. From results, it can be seen that compared to the two well-known weight-only PTQ methods AWQ and GPTQ, our method still demonstrates significant advantages. Especially under more extreme weight quantization settings like W2A16, where GPTQ and AWQ collapse, RLQuant still maintains a good performance. For zero-shot tasks,
our method still outperforms them in most results.
Bits Models Methods WikiText2 PTB C4 PTB-new C4-new
W4A4
OPT-1.3B
FP16 14.62 16.96 14.72 20.29 16.07 SmoothQuant 126.05 120.43 107.16 128.68 119.11
LAE 102.12 106.38 76.40 112.53 86.95 OmniQuant 19.19 23.62 19.45 28.92 21.49 LRQuant 19.11 23.29 19.39 28.75 21.45
OPT-2.7B
FP16 12.47 15.11 13.16 17.97 14.34 SmoothQuant 252.04 207.65 151.90 220.49 169.49
LAE 423.20 304.95 263.87 246.71 309.16 OmniQuant 15.19 19.34 16.51 23.24 18.14 LRQuant 14.94 19.24 16.15 23.33 17.83
OPT-6.7B
FP16 10.86 13.09 11.74 15.77 12.71 SmoothQuant 491.34 317.70 238.19 293.54 277.48
LAE 305.84 235.22 180.49 202.07 214.96 OmniQuant 12.34 15.44 13.56 18.50 14.88 LRQuant 12.33 15.31 13.50 18.31 14.76
W6A6
OPT-1.3B
FP16 14.62 16.96 14.72 20.29 16.07 SmoothQuant 15.42 17.65 15.32 21.30 16.70
LAE 15.34 17.67 15.20 21.30 16.57 OmniQuant 14.99 17.42 15.06 20.94 16.42 LRQuant 14.94 17.38 15.04 20.80 16.41
OPT-2.7B
FP16 12.47 15.11 13.16 17.97 14.34 SmoothQuant 12.68 15.35 13.32 18.28 14.51
LAE 12.96 15.65 13.53 18.67 14.68 OmniQuant 12.56 15.30 13.27 18.23 14.47 LRQuant 12.55 15.23 13.27 18.14 14.47
OPT-6.7B
FP16 10.86 13.09 11.74 15.77 12.71 SmoothQuant 10.98 13.23 11.85 15.89 12.82
LAE 11.39 13.69 12.15 16.33 13.08 OmniQuant 10.94 13.17 11.81 15.81 12.79 LRQuant 10.93 13.16 11.81 15.79 12.78
Table 13: Perplexities (lower is better) comparison of quantized OPT models at W4A4 and W6A6.

heading: Future Work
text: This will be added to our list of future work. Additionally, our LRQuant is the first to introduce TTA into LLMs quantization which has successfully improved the quantized models’ performance on unseen datasets. However, we do not draw inspirations from state-of-the-art TTA methods which makes our performance may not achieve optimal, such as on C4 dataset our TTA scheme still stands behind original results. In future work, we aim to explore ideas from other methods and
propose a more effective TTA scheme for LLMs quantization