LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2024)

Abstract: Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization-aware training for LLMs (LLM-QAT) to push quantization levels even further. We propose a data-free distillation method that leverages generations produced by the pre-trained model, which better preserves the original output distribution and allows quantizing any generative model independent of its training data, similar to post-training quantization methods. In addition to quantizing weights and activations, we also quantize the KV cache, which is critical for increasing throughput and supporting long sequence dependencies at current model sizes. We experiment with LLaMA models of sizes 7B, 13B, and 30B, at quantization levels down to 4-bits. We observe large improvements over training-free methods, especially in the low-bit settings.

heading: 1 Introduction
text: Following GPT-3 (Brown et al., 2020), several families of large language models (LLMs) such as OPT (Zhang et al., 2022), PALM (Chowdhery et al., 2022), BLOOM (Scao et al., 2022), Chinchilla (Hoffmann et al., 2022) and LLaMA (Touvron et al., 2023a) have established that increasing model size leads to improved model capabilities. As a result, language models with tens of billions or even hundreds of billions of parameters have become the norm in today’s AI landscape. Despite the growing excitement around LLMs, serving such models to the benefit of billions of users faces significant hurdles due to their large computational cost and environmental footprint.
Fortunately, there has been an increasing effort to accurately quantize LLMs, with multiple recent
∗Equal contribution. Correspondence to: Zechun Liu <zechunliu@meta.com>.
works (Xiao et al., 2022; Yao et al., 2022) focusing on 8-bit post-training quantization of weights and activations and achieving little to no loss of accuracy, as well as quantizing the weight and KV cache and using GPU/CPU offloading to achieve highthroughput LLM inference (Sheng et al., 2023a). However, SoTA post-training quantization methods dramatically degrade in quality when quantizing weights, activations, and KV cache together to below 8-bit. For lower quantization bit-widths, we find it necessary to resort to quantization-aware training (QAT).
To our knowledge, QAT for LLMs has not been investigated before. This is understandable for two reasons. First, LLM training is technically difficult and resource-intensive. Second, QAT needs training data, which for LLMs is difficult to obtain. The sheer scale and diversity of pre-training data is itself an obstacle. Pre-processing might
467
be prohibitive, or worse, some data might simply not be available due to legal restrictions. It is also increasingly common to train LLMs in multiple stages, involving instruction tuning and reinforcement learning (Ouyang et al., 2022), which would be very difficult to replicate during QAT. In this work, we side-step this issue by using generated data from the LLM itself for knowledge distillation. This simple workaround, which we refer to as data-free knowledge-distillation is applicable to any generative model independent of whether or not the original training data is available. We show that this method is better able to preserve the original model’s output distribution, even compared to training on large subsets of the original training set. Moreover, we can successfully distill quantized models using only a small set (100k) of sampled data, thus keeping computational costs reasonable. All of our experiments are conducted using a single 8-gpu training node.
As a result, we are able to distill the 7B, 13B and 30B LLaMA models with activations quantized down to 8 bits, weights, and KV cache down to 4-bits. In this regard, our approach exhibits significant enhancements in quality compared to post-training quantization. Notably, larger models employing QAT outperform smaller models utilizing floating-point 16-bit representations, despite having similar model sizes, as illustrated in Figure 1. Furthermore, we have successfully quantized activations to 6-bit precision, surpassing what was possible with existing methods. For a comprehensive analysis of our experimental results and detailed ablations, please refer to Section 3.
In summary, we present the first application of QAT to LLMs, resulting in the first accurate 4-bit quantized LLMs. We also demonstrate quantizing the KV cache simultaneously with weights and activations, which is critical to alleviate throughput bottlenecks for long sequence generation. All of this is achieved by a novel data-free distillation
method, which makes QAT practical for large pretrained generative models.

heading: 2 Method
text: Quantizing large language models (LLMs) using quantization-aware training (QAT) is a nontrivial task with challenges in two key aspects. First, LLMs are pre-trained to excel in zero-shot generalization, and it is crucial to preserve this capability after quantization. Therefore, selecting an appropriate fine-tuning dataset is important. If the QAT data is too narrow in a domain or significantly different than the original pre-training distribution, this is likely to hurt model performance. On the other hand, it is difficult to replicate the original training setup exactly, due to the scale and complexity of LLM training. In Section 2.1, we introduce data-free quantization-aware training (QAT) which produces QAT data using next token data generation. This method demonstrates superior performance compared to using subsets of the original pre-training data. Second, LLMs exhibit unique weight and activation distributions characterized by a significant presence of outliers, which distinguishes them from smaller models. Consequently, the state-of-the-art quantization clipping methods for small models do not work out of the box for LLMs. In Section 2.2, we identify suitable quantizers for LLMs.

heading: 2.1 Data-free Distillation
text: In order to closely synthesize the distribution of the pre-training data with a limited amount of finetuning data, we proposed next token data generation from the original pre-trained model. As shown in Figure 2 (a), we randomize the first token <start> from vocabulary and let the pre-trained model generate the next token <out1>, then the generated token is appended to the start token for generating new output <out2>. We repeat this
iterative procedure until we reach either the end-ofsentence token or the maximum generation length.
We test three different sampling strategies in the next token generation. The most straightforward way is to pick the top-1 candidate as the next token. However, the generated sentence lacks diversity and will cyclically repeat several tokens. To address this issue, we instead stochastically sample the next token from the distribution using the SoftMax output of the pre-trained model as the probability. This sampling strategy yields more diverse sentences and greatly enhances the accuracy of the fine-tuned student model. Furthermore, we discover that the initial few tokens play a crucial role in determining the prediction trend. Therefore, it is important for them to have higher confidence. In our generative process, we employ a hybrid sampling strategy that deterministically selects the top-1 predictions for the first 3~5 tokens and stochastically samples the remaining tokens. A detailed ablation study comparing different generated data and real data is presented in Section3.3.1.

heading: 2.2 Quantization-Aware Training

heading: 2.2.1 Preliminaries
text: In this work, we study linear quantization i.e., uniform quantization. Linear quantization can be categorized into two categories based on whether the real values are clipped or not: MinMax quantization, which preserves all value ranges, and clippingbased quantization.
In MinMax quantization,
XiQ = αX̂ i Q = α⌊ XiR − β α ⌉+ β. (1)
Here XQ and XR denote the quantized and fullprecision variables, respectively. i refers to the i-th element in the tensor. α is the scaling factor and β is the zero-point value. For symmetric quantization, α = max(|XR|)
2N−1−1 , β = 0. And for asymmetric quan-
tization, α = max(XR)−min(XR) 2N−1 , β = min(XR).
Compared to the MinMax Quantization, clipping the outliers can help improve the precision and allocate more bits to the intermediate values. Thus, many recent work (Shen et al., 2020a; Zhang et al., 2020) adopt clipping-based quantization for transformer-based language models. The quantization can be formulated as:
XiQ = αX̂ i Q = α⌊Clip( XiR − β α , 0, 1)⌉+β. (2)
where the scale α and zero-point value β can be calculated statistically or learned through gradients.

heading: 2.2.2 Quantization for LLMs
text: Quantization function We illustrate our quantized transformer model in Figure 3. In line with the findings in (Dettmers et al., 2022; Xiao et al., 2022), we have also observed a significant presence of outliers in both the weights and activations of large language models (LLMs). These outliers have a notable impact on the quantization process, as they contribute to an increase in the quantization step size while diminishing the precision of intermediate values. Nevertheless, clipping these outliers during quantization proves detrimental to LLM performance. During the initial stages of training, any clipping-based method will lead to exceptionally high perplexity scores (i.e., > 10000), causing a substantial loss of information that proves to be difficult to recover through fine-tuning. Therefore, we choose to retain these outliers instead. Moreover, we find that in the model with the gated linear unit (GLU), the activations and weights are mostly symmetrically distributed. Based on our analysis and empirical observations, we choose symmetric MinMax quantization for both weights and activations:
XiQ = α⌊ XiR α ⌉, α = max(|XR|) 2N−1 − 1 (3)
Here XQ denotes the quantized weights or activations and XR denotes the real-valued weights or activations. To ensure efficient quantization, we adopt the per-token activation quantization and perchannel weight quantization. For a comprehensive
evaluation of the different quantizer choices, we provide the ablation study in Section 3.3.2. Quantization-aware training for KV-cache In addition to weight and activation quantization, the key-value cache (KV cache) in large language models (LLMs) also consumes a non-negligible amount of memory. However, only a few previous works have addressed the KV cache quantization in LLMs, with the methods primarily limited to posttraining quantization(Sheng et al., 2023b). In our study, we demonstrate that a similar quantizationaware training approach used for activation quantization can be employed to quantize the KV cache. We adopt per-token quantization in Eq. 3, given that the key and value are generated token by token. During the generation process, the current key and value are quantized, and their corresponding scaling factor is stored. During the training process for QAT, we apply quantization to the entire activation tensors of both the keys and values, as shown in Figure 3. By integrating the quantization function into the gradient calculation, we ensure effective training using quantized key-value pairs. Knowledge distillation We use cross-entropy based logits distillation for training the quantized student network from the full-precision pre-trained teacher network:
LCE = − 1
n
∑
c
n∑
i=1
pTc (Xi) log(p S c (Xi)), (4)
Here i denotes the ith sample in the current batch with n total sentences. c denotes the number of classes and in our case, it equals the size of the vocabulary. T and S are the teacher network and student network, respectively.
As discussed in Section 2.1, in the data generation process, it is important to sample the next token from distribution rather than always selecting the top-1 candidate. By doing so, the next token does not necessarily represent the optimal label for training the student model, as the sampling introduces inherent noise. Consequently, we propose to utilize the predictions from the pre-trained model as soft labels, which provide more informative targets for guiding the training of the student model. Detailed ablation study can be found in Section 3.3.3.

heading: 3 Experiments
text: We assess the effectiveness of our approach by conducting experiments on LLaMA-7B/13B/30B mod-
els and presenting results on various tasks. Specifically, we report the zero-shot performance on Common Sense Reasoning tasks such as BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). We also assess the few-shot performance on TriviaQA (Joshi et al., 2017) and MMLU (Hendrycks et al., 2020) datasets, along with perplexity scores on WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020) datasets.

heading: 3.1 Experimental Settings
text: In our quantized network training process, we initialize the model with a pre-trained model and employ it as the teacher for knowledge distillation. To optimize the model, we utilize the AdamW (Loshchilov and Hutter, 2017) optimizer with zero weight decay. Each GPU is assigned a batch size of 1, and the learning rate is set to 2e-5, following a cosine learning-rate decay strategy. For data generation, we utilize the LLaMA-7B model, and the maximum length of generated sequences is set to 2048. We calculate the number of OPs by OPs = MACs×Wbits ×Abits with the sequence length equals to 2048.

heading: 3.2 Main Results
text: We consider three post-training quantization (PTQ) methods, round-to-nearest (RTN), GPT-Q (Frantar et al., 2022) and SmoothQuant (Xiao et al., 2022) as baselines. We compare to them in several different settings, where the weights, activations, and KV cache values are quantized to different levels (denoted as W-A-KV). Different PTQ methods perform well in different settings, and we compare our method to the best PTQ result in each setting.
Table 1, table 2 and table 6 (in Appendix) give the comparisons of the proposed QAT methods with SOTA PTQ methods for LLMs on Zero-shot tasks on Common Sense Reasoning tasks, perplexity evaluation on Wiki2 and C4 and few-shot exact match on the MMLU and TriviaQA benchmarks respectively. The perplexity evaluations verify whether the quantized models are able to preserve the output distribution of the model on a diverse sample of its training domains. The zero-shot and few-shot evaluations measure if the model’s capabilities on downstream tasks are retained.
The trends in each table are similar. All methods tend to do well in the 8-bit setting across all
model sizes. This holds even when the KV cache is also quantized to 8-bits, together with weights and activations. However, when either of these three values are quantized to less than 8-bits, PTQ methods result in accuracy loss, whereas LLMQAT holds up much better. For example in the 8-8-4 setting, 30B LLM-QAT achieves an average zero-shot accuracy of 69.7, compared to 50.7 with SmoothQuant (Table 1, rows 68-69). The difference is smaller in the 4-8-8 setting, however,
LLM-QAT still outperforms the best PTQ method (RTN in this case) by 1.4 points (rows 55, 57). In the 4-8-4 setting, where both weights and the KV cache are quantized to 4 bits, all PTQ methods produce poor results, whereas LLM-QAT achieves 69.9, only trailing the full precision model by 1.5 points on average. LLM-QAT also works reasonably well for 6-bit activation quantization. While this setting might not be currently practical due to a lack of hardware support, it’s a promising data point for sub-8-bit computation for LLMs.
One important question for practitioners is whether to use a small model at full precision, or a larger quantized model of similar inference cost. While the exact trade-offs can vary based on several factors, we can make several recommendations based on our results. First, 8-bit quantization should be preferred over smaller full-precision models, and PTQ methods are sufficient for this case. An 8-8-8 30B quantized model outperforms a 13B model of similar size and should have lower latency and higher throughput in practice. This also holds for an 8-bit 13B model compared with a 16-bit 7B model. Furthermore, 4-bit models quantized using LLM-QAT should be preferred over 8-bit models of similar size. For instance, a 4-8-4 LLM-QAT 30B outperforms an 8-bit LLaMA-13B, and a 4-8-8 LLM-QAT 13B is better than an 8- bit LLaMA-7B. As a result, we recommend 4-bit LLM-QAT models for the best efficiency-accuracy tradeoff.

heading: 3.3 Ablation
text: We conduct the ablation study regarding the data choice, quantization methods, and knowledge distillation methods in Sections 3.3.1, 3.3.2 and 3.3.3, respectively. We report both the perplexity scores on WikiText2 (Merity et al., 2016)/C4 (Raffel et al., 2020) datasets and the performance on zero-shot common sense reasoning tasks.

heading: 3.3.1 Data Choice
text: In Table 3, we observe that WikiText (Merity et al., 2016), which is constructed using text extracted from Wikipedia, does not encompass all the information utilized during pre-training. Consequently, a model fine-tuned solely on WikiText tends to overfit on this specific dataset and struggles to generalize well to other datasets. On the other hand, the Crawled Corpus (C4) dataset (Raffel et al., 2020) comprises hundreds of gigabytes of clean English text collected from the web. Fine-tuning the model on C4 yields reasonable transfer accuracy when evaluated on the WikiText dataset. However, it exhibits poor accuracy when tasked with zero-shot inference tasks. More comprehensive comparisons can be found in the appendix.
Compared to the existing data, the model finetuned on generated data demonstrates superior generalizability, particularly in zero-shot tasks. Moreover, the data generated through sampling from the distribution exhibits greater diversity compared to the data generated without sampling. This enhanced diversity leads to significantly improved performance across all tasks.
Table 3: Effects of the finetuning data to the performance in downstream tasks. We use 4-bit weight 6-bit activation LLaMA-7B for the experiments. We test three strategies for data generation. Generated data1 refers to always picking the top-1 candidate without sampling. Generated data2 refers to sampling the next token from the distribution. Generated data3 refers to the first 3~5 tokens being generated with deterministic selection while the rest are stochastically sampled from the distribution.
C4 Wiki2 BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA Avg. Finetuning Data (↓) (↓) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) 1 (Pretrained Model) 7.2 10.7 76.8 79.3 48.6 76.1 70.0 73.0 48.0 57.6 66.2 2 Wiki2 10.1 5.5 46.9 74.3 45.2 72.4 65.7 67.2 45.0 47.8 58.1 3 Wiki103 9.6 5.2 45.9 74.4 46.4 71.4 66.1 67.5 46.3 49.8 58.5 4 C4 7.8 11.3 61.7 77.7 48.8 73.2 67.2 67.8 43.6 52.2 61.5 6 Generated data1 8.0 11.4 60.0 77.1 48.1 72.3 65.7 67.4 44.2 49.8 60.6 7 Generated data2 7.7 11.5 70.9 76.1 47.9 72.2 66.9 69.3 46.4 53.6 62.9 8 Generated data3 7.7 10.8 72.9 76.8 47.9 72.4 68.3 68.8 44.2 53.2 63.1

heading: 3.3.2 Quantization Function
text: We compare the no-clipping quantization method with clipping-based methods in Table 4. Following the practice in previous works (Liu et al., 2022b, 2023), we use StatsQ (Liu et al., 2022a), a statistically-calculated scaling factor for clippingbased weight quantization and LSQ (Esser et al., 2019), the learnable scaling factor for clippingbased activation quantization. However, our findings indicate that these two state-of-the-art clipping-based quantization methods do not surpass the performance achieved by the MinMax nonclipping method. This observation reinforces the argument that preserving the outliers is critical to the performance of large language models.
Furthermore, we observe that for LLaMA models, the activations and weights exhibit predominantly symmetric distributions, which makes using symmetric quantizers the best choice. It is important to note, however, that this conclusion may not hold true for other large language models, especially those incorporating GeLU layers.

heading: 3.3.3 Knowledge Distillation
text: Table 5 shows that different knowledge distillation methods have a significant impact on the final accuracy of fine-tuned models. Notably, utilizing the next token alone as the label is sub-optimal due to the inherent randomness and noise introduced by sampling from a distribution of candidates during the generation process. In contrast, logit distillation, which utilizes the complete logit distribution prediction from the teacher model, leads to superior performance of fine-tuned models compared to label-based training approaches. Interestingly, we have observed that incorporating attention distillation or hidden layer distillation actually hampers the performance. Consequently, we exclusively employ logit distillation in all our experiments.
GPU days

heading: 3.4 Training Cost Analysis
text: Table 4: Ablation study on the effects of the quantization methods on LLaMA-7B model. The quantization level is set to 4-bit weight and 8-bit activation.
C4 Wiki2 BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA Avg. Weight Activation (↓) (↓) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) 1 (Pretrained Model) 7.2 10.7 76.8 79.3 48.6 76.1 70.0 73.0 48.0 57.6 66.2 2 Clipping Clipping 9.0 11.9 64.9 66.8 43.6 63.5 56.1 51.0 31.4 33.8 51.4 3 MinMax Clipping 9.4 12.8 63.5 62.4 42.4 61.2 52.9 45.6 29.6 33.8 48.9 4 Clipping MinMax 8.2 11.0 71.7 75.1 43.7 69.5 58.9 62.6 35.2 37.8 56.8 5 MinMax MinMax 7.4 10.9 74.8 77.8 48.6 73.6 69.0 69.7 45.8 55.8 64.4 6 Asym Asym 7.3 10.4 75.0 78.4 48.0 73.9 69.3 71.9 45.7 52.6 64.3 7 Sym Asym 7.4 11.0 72.7 77.9 48.8 73.3 67.9 69.2 45.2 56.0 63.9 8 Asym Sym 7.4 10.9 73.3 78.4 48.0 73.9 68.9 71.4 46.4 54.0 64.3 9 Sym Sym 7.4 10.9 74.8 77.8 48.6 73.6 69.0 69.7 45.8 55.8 64.4
Table 5: Ablation study on the knowledge distillation choices on LLaMA-7B model with generated data. The quantization level is set to 4-bit weight and 6-bit activation.
C4 Wiki2 BoolQ PIQA SIQA HellaSwag WinoGrande ARC-e ARC-c OBQA Avg. Method (↓) (↓) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) (↑) 1 (Pretrained Model) 7.2 10.4 76.8 79.3 48.6 76.1 70.0 73.0 48.0 57.6 66.2 2 Label 8.1 11.9 69.4 77.3 48.7 72.1 67.1 67.6 45.4 51.4 62.4 3 Label + Attention 8.8 18.6 70.2 75.3 47.6 68.9 67.2 65.6 42.6 51.2 61.1 4 Label + Hidden 10.9 16.2 61.0 53.5 41.1 32.6 50.2 25.8 23.1 25.0 37.7 5 Label + Logits 7.8 11.0 70.8 77.3 48.3 72.5 66.7 68.2 46.5 55.4 63.2 6 Logits 7.7 10.8 72.9 76.8 47.9 72.4 68.3 68.8 44.2 53.2 63.1 7 Logits + Attention 7.9 12.2 73.2 74.6 47.2 69.1 65.1 64.8 42.1 52.8 61.1 8 Logits + Hidden 22.3 52.6 38.0 50.4 38.6 25.6 50.5 26.3 24.3 25.8 34.9 9 Logits + Hidden + Attention 21.9 46.0 55.0 47.8 39.0 33.4 48.5 29.7 26.4 25.8 38.2

heading: 4 Related Works
text: Quantization Neural network quantization is proved to be a valuable tool in compressing model size and reducing storage consumption. Classic quantization methods, such as MinMax quantization (Jacob et al., 2018; Krishnamoorthi, 2018), Learned step-size quantization (Esser et al., 2019), PACT (Choi et al., 2018), N2UQ (Liu et al., 2022a) and etc, have primarily been developed for convolutional neural networks. While several recent works have explored language model compression, they are mostly focused on smaller models (Zafrir et al., 2019; Fan et al., 2020; Shen et al., 2020b; Zadeh et al., 2020; Bai et al., 2021; Qin et al., 2021; Liu et al., 2022b) like BERT (Devlin et al., 2019) or BART (Lewis et al., 2019). For large language models (LLMs), the available quantization methods are mostly limited to post-training quantization (Xiao et al., 2022; Yao et al., 2022; Frantar et al., 2022; Sheng et al., 2023a), due to the lack of accessible training data or the prohibitive resource requirements for fine-tuning on the entire pre-training dataset. To the best of our knowledge, no previous work has addressed the specific challenge of quantization-aware training for LLMs. The compression capabilities of state-of-the-art PTQ methods are confined to W8A8 (Xiao et al., 2022) or W4A16 (Frantar et al., 2022). A recent work introduced floating-point quantization to enable W4A8 quantization (Wu et al., 2023), however, this approach necessitates hardware customization for floating-point computation. In contrast, LLM-QAT
attains a level of accuracy on par with full-precision models with simple W4A8 integer quantization. Data generation Data generation for QAT remains a relatively unexplored field of research. While there are several works in the vision domain finetuning student networks (Yin et al., 2020; Liu et al., 2022c; Cai et al., 2020) using noise-to-data generation from pre-trained teacher models, these methods mainly focus on image data. In the language domain, a few previous work use human-defined prompts to elicit responses from GPT models for fine-tuning. For example, Vicuna (Zheng et al., 2023) utilized user-uploaded ShareGPT data for instruction fine-tuning, while Alpaca (Taori et al., 2023) relied on predefined human prompts with careful balance in each category to ensure diversity. In contrast, our method eliminates the need for human prompts or user data. A single random initial token allows LLMs to autonomously generate data suitable for QAT finetuning. To the best of our knowledge, this is not studied in existing literature.

heading: 5 Conclusion and Limitations
text: We proposed data-free quantization-aware training for LLMs and showed accurate, 4-bit quantization is possible using this technique. Given the generality of the training-data-agnostic distillation method, and the growing cost of LLM deployments, we expect our method to have wide applicability. For instance, the method could also be used for models trained in several stages, e.g. with instruction tuning or reinforcement learning (Ouyang et al., 2022). We leave this investigation to future work.

heading: A Appendix

heading: A.1 Few-shot Evaluation Results
text: Table 6 presents the few-shot performance of the quantized model on the MMLU (Hendrycks et al., 2020) and TriviaQA (Joshi et al., 2017) benchmarks.

heading: A.2 Zero-shot Reasoning Performance on LLaMA-v2
text: We conducted experiments on the LLaMA-v2 structure (Touvron et al., 2023b), and the results in Table 7 consolidate the claim that LLM-QAT consistently enhances the performance of quantized models in ultra low-bit settings

heading: A.3 Exploring the Limits: 4-bit Weight 4-bit Activation Quantization
text: We further explore the lower-bit quantization of 4-bit weight and 4-bit activation (W4A4). The results show that the W4A4 quantization is challenging for LLMs. Post-training quantization sees ∼ 30 points degradation. Adding LLM-QAT together with smoothquant can recover 12.5 points and 7.7 points for W4A4KV16 and W4A4KV4 settings, respectively.
Furthermore, we delved into the potential of combining the group-wise quantization (Shen et al., 2020a; Sheng et al., 2023a) with LLM-QAT. The results in Table 8 unveiled that group-wise quantization with 1- channel per group managed to achieve less than 8 points accuracy drop on the W4A4KV4 setup compared to full-precision, which was infeasible in any of the previous works. Nevertheless, it is important to note that implementing this group-wise quantization method may necessitate specialized kernel design to fully realize its potential for actual speedup (Shen et al., 2020a; Cai et al., 2020).

heading: A.4 Evaluation Benchmarks

heading: A.4.1 Zero-shot Common Sense Reasoning tasks
text: BoolQ (Clark et al., 2019) is a reading comprehension dataset of naturally occurring yes/no questions. Each example consists of a question (Q), an excerpt from a passage (P), and an answer (A) with an explanation added for clarity.
PIQA (Bisk et al., 2020), short for Physical Interaction: Question Answering, is a benchmark for evaluating and studying physical commonsense understanding in natural language models.
SIQA (Sap et al., 2019) aims to measure the social and emotional intelligence of computational models through multiple choice question answering (QA).
HellaSwag (Zellers et al., 2019) is a benchmark for physically situated commonsense natural language inference. It consists the four-way multiple-choice problems that are trivial for humans (> 95% accuracy), but challenging for the language models.
WinoGrande (Sakaguchi et al., 2021) is a benchmark for commonsense reasoning. It comprises a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations.
ARC (Clark et al., 2018), the AI2 Reasoning Challenge, contains a collection of 7787 natural science questions. It is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm.
OBQA (Mihaylov et al., 2018) is a dataset of about 6000 questions for open book question answering. The task focuses on the challenge of combining a corpus of provided science facts (open book) with external broad common knowledge.

heading: A.4.2 Few-shot Tasks
text: TriviaQA (Joshi et al., 2017) is a closed-book question answering benchmark. It contains over 650K question-answer evidence triples, that are derived by combining 95K Trivia enthusiast authored questionanswer pairs with on average six supporting evidence documents per question.
MMLU (Hendrycks et al., 2020), the Massive Multitask Language Understanding(MMLU) benchmark (Hendrycks et al., 2020), consists of multiple choice questions covering various domains of knowledge, including humanities, STEM and social sciences.

heading: A.5 Generation Tasks
text: WikiText2 (Merity et al., 2016) is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.
C4 (Raffel et al., 2020), abbreviate for Colossal Clean Crawled Corpus. Common Crawl2 is a publiclyavailable web archive that provides “web extracted text” by removing markup and other non-text content from the scraped HTML files. C4 filters Common Crawl’s web-extracted text and produces a collection of text that comprises clean and natural English text.

heading: A.6 Generated Data from LLaMA-7B
text: In this section, we show some examples of the generated data obtained through the next token generation with LLaMA-7B model.
2http://commoncrawl.org/
\nSincerely, I\u2019m sending for your help right now.\nI\u2019m currently facing a rather challenging situation.\nMy wife and me have been living and working in the same country for a couple of years.\nHowever, after we\u2019ve seen a couple of movies together, we found out that we have different tastes in movies.\nMy wife is always watching melancholy (soporific, slow-moving) drama movies, while I prefer thrillers or action movies.\nIt would be great if we could both keep our personal wishes, yet make sure we have something to discuss about afterwards.\nDo you think there\u2019s a nice web site, a movie club, or so, which helps us choose movies to watch as a couple?\nPlease, please help me out of this situation!\nA: Thanks for asking for my help. If you can, please do watch some movies together before choosing the movies that are really suitable for each of you.\nMovies have different styles of story telling, and some can be slow-moving. It is up to the audience to decide what kind of movies they want to see. You are just the same as other people who might prefer watching some movies at home while the other one loves to watch movies at a movie theater.\nIf there exist some websites or movie clubs, why don\u2019t you try taking a look at it. However, it is up to you to find out which movies are suitable for your couple, so the movie club, or the website, can only give some ideas for your reference.\nSincerely, hope my answer could help you with your issue.\n- I am new to ALT life and have been living here for three month. We would love to join some clubs and have asked many people with no success. I don’t understand why some people are unwilling to help.\n- It’s always best to ask people at the local supermarket or caf\u00e9 for any events coming up in the area. The best way to find out is to ask.\n- I’d rather not answer at this stage.\n- I like this place, and I’d like to stay here, but I am sure there is more to see and do, so hopefully if I try hard enough I will be able to find out more.\n- I think it’s a bit difficult living away from home, but having family and friends around helps a lot.\n- We’d like to play baseball, is there any baseball or softball club, we are new to your country but quite enthusiastic.\n- We have a large number of volunteer groups that actively help the local community, and the local government sets up new programmes all the time."
In the mid-20th century, there was growing awareness that in a large number of industrial countries the population was aging. This awareness, along with advances in social security in many countries and a sense of urgency to avoid future generations of poor, has led to social policy being implemented that targets the elderly. The elderly often have a special status and enjoy particular social benefits. Such benefits include a higher age entitlement threshold, higher pensions, early-retirement benefits, employment quotas, and free or subsidized health care. In addition, in some countries the elderly are exempt from federal and, in some cases, from state taxes. \n Since most governments have set limits on their ability to pay out more in social security and health care benefits to any one group, such benefits are generally reserved for the retired and the disabled. Accordingly, elderly care for the mentally disabled, infirm, and cognitively impaired has taken on particular importance, as has care for the homeless. In addition, those who are economically active can also benefit from programs, although there is the perception that the economically active are less in need of care assistance, because their families often take care of them. Elderly care is generally provided on an unpaid basis, although those in need of care may have to sell their homes because of the high cost of care.\nSee also: Economy; Old Age and Retirement Welfare Systems; Unemployment; and Unemployment and Employment Security Systems.\nBell, Ian R. S., The Growing Crisis of Old Age. London: Pall Mall Press, 1913.
When my brother returned from abroad, he went with his wife to the motherland. They had many elder brothers around there, so that we all went with them to my parents\u2019 home. I, being the youngest, went first. My brother, seeing the dowry that I brought with me, laughed. He was very proud. \u2018This girl is from my sister; she is the same as my sister. Why should I have her?\u2019 he said to them. But at that time my father was ill; he lay stretched on a bed before us. \u2018This is my sister; she is my sister. I did not give her to you; why should I allow you to kill my sister?\u2019 the father said to my brother. His mother said, \u2018I shall put this girl\u2019s hand in your hand. When you wish to give it away, it will be like your sister\u2019s.\u2019 My brother thought, \u2018There surely cannot be two sisters.\u2019 I told him, \u2018The old man will die soon. I have brought him his dowry of ten yens of silver. It is no small gift for him. If I do you the favour now, will you not have respect for the father when he is dead and gone?
a town, and the largest inland city of the ancient Aztlan (Aztatlan, Azatlan) located near the modern ruin of Santa Anas, Sonora, Mexico.\nAtsa was a settlement among the 13 Tamoanchan cities of the Nahua people that established their settlement in Northern Mexico by about 1000 A.D. The city of Atsa (its people Atsatla) belonged to ẗhe twelve tribes of Tamoanchan,̈ which may be read in the Aztec codices as T̈lascallaänd which are the T̈lascalanp̈eoples of the 13 cities or a confederation of peoples that were among the dominant rulers of the Valley of Mexico, and of western Mexico in general, during the pre-eminent period of Aztec civilization.\nAtsa as well as the other Tamoancan cities of the Aztecs, is said to be on a plain of jade and silver and to have a number of large, flat-roofed buildings. These buildings are described as being in ruins in 1519, during Hernan Cortes’s first visit to Mexico. Although it is not known if the Tamoancan cities were named before or after contact with the Spaniards, it may be that the descriptions are only describing Aztec ideas of the 13 cities by its inhabitants. Atsa is said to be located at a point where a landform resembling a large lake (a m̈ountain)̈, was formed at some point in their past that has since dried. It is here that it is said the Atsatla people established their community.
This article appeared in the Ceylon Today on 24th January 2016.\nI have been an Anglican Priest and a member of various committees in the Ceylon Evangelical Lutheran Church (CELC) for many years. I have observed that most of the CELC clergy and Laity are engaged in one or the other form of mission; whether as pastor, deacon, Sunday School teacher, a member of the Christian Education Council (CEC), Church Council or its equivalent. But do we realise the full significance of the words of the Apostle Paul in the first Reading?\nThe readings for this Sunday continue in Paul\u2019s second letter to the church in Corinth, where he warns the Christians against the excesses of the Eucharist. He warns them not to drunkenness during the Eucharist. He also writes: \u201cDo not, for the sake of food, destroy the work of God\u201d (6:12).\nThese were the words Paul used in his letter to his beloved Church: \u201cEat anything God has created to make you healthy.\nYou should not feel guilty about eating any of these foods, but you should not eat them if they cause other people to stumble\u201d (Romans 14:14-23, CEB).\nIf we want to do God\u2019s will, we will learn what pleases him. (Eph. 5:10). If we want others to respect God\u2019s law, then we must avoid even the appearance of sin (1 Thess. 5:22). In other words, we must seek to please God in all of our actions.\nAs Catholics we celebrate the Holy Mass by taking communion of the Bread and Wine, in the name of the Father, the Son and the Holy Spirit to remind us of our call to love and serve one another.\nSt. Paul said: \u201cA little yeast spreads through the whole batch of dough\u201d (Gal. 5:9). Christians should celebrate God\u2019s love in a special way. We are called to live for the next generation. If we drink to excess at the Eucharist it ruins the lives of our children. \u201cA little yeast\u201d spreads quickly through the family: an immoral step, a broken marriage, divorce and abuse: All these are examples of the \u201clittle yeast\u201d that destroys families.\nBut there\u2019s more. \u201cBecome sober, and stop sinning. Then some apostle will not be wrongly accused of being responsible for sinning. Anyone who has been stealing must steal no longer\u201d (v. 8). Our call as Christians is to lead exemplary lives. As such, we must act in accordance with our commitment to the \u201cLove of God and of our Neighbour\u201d. We are called to work with integrity. We are called to give to society. If a person is addicted to any alcoholic beverage or to any other type of drug, it will be difficult for him or her to lead a moral life.\nIt is also said that the words of the Apostle Paul in the First Reading \u201care still being applied today\u201d!\nOur forefathers who were the followers of John Wesley the Founder of Methodism made this famous saying to describe a Methodist: \u2018 A Methodist will find his way home from the most God forsaken, lawless, brutalized and degraded part of the earth, to the humble home of his childhood or his God: To find his way when the sun sets and the sky is filled with dense shades of night; or on the banks of the Red river, amid the solitudes of the Mexican desert, or in the dreary solitude of the frozen wastes of the Arctic Ocean\u2026\u201d.\nIf we ask ourselves some important question we are compelled to ask such question whether this is true for every individual member of the churches, whether it is true for the clergy or laity.\nThe Methodist Church in the United Kingdom, for example, is known as the \u201cMother Church\u201d. In 2014, a survey revealed that 21% of the British public was unaffiliated to any religion. But 50% of the people in England and Wales were prepared to \u201ctry and become a Christian\u201d if invited, despite not going to church."

heading: A.7 Broader Impact
text: We propose a model compression technique that reduces the memory footprint of large language models, enabling their deployment on embedded devices. This technique has the potential to decrease energy consumption for end users and lower costs for companies running language models at scale. We acknowledge
the potential risks associated with the malicious use of large language models by third parties. Therefore, we are dedicated to maintaining the techniques we have developed to ensure responsible usage of these models.

heading: Future Work
text: This awareness, along with advances in social security in many countries and a sense of urgency to avoid future generations of poor, has led to social policy being implemented that targets the elderly. The elderly often have a special status and enjoy particular social benefits. Such benefits include a higher age entitlement threshold, higher pensions, early-retirement benefits, employment quotas, and free or subsidized health care. In addition, in some countries the elderly are exempt from federal and, in some cases, from state taxes. \n Since most governments have set limits on their ability to pay out more in social security and health care benefits to any one group, such benefits are generally reserved for the retired and the disabled. Accordingly, elderly care for the mentally disabled, infirm, and cognitively impaired has taken on particular importance, as has care for the homeless. In addition, those who are economically active can also benefit from programs, although there is the perception that the economically active are less in need of care assistance, because their families often take care of them. Elderly care is generally provided on an unpaid basis, although those in need of care may have to sell their homes because of the high cost of care. \nSee also: Economy; Old Age and Retirement Welfare Systems; Unemployment; and Unemployment and Employment Security Systems. \nBell, Ian R. S. , The Growing Crisis of Old Age. London: Pall Mall Press, 1913. When my brother returned from abroad, he went with his wife to the motherland. They had many elder brothers around there, so that we all went with them to my parents\u2019 home. I, being the youngest, went first. My brother, seeing the dowry that I brought with me, laughed. He was very proud. \u2018This girl is from my sister; she is the same as my sister. Why should I have her?\u2019 he said to them. But at that time my father was ill; he lay stretched on a bed before us. \u2018This is my sister; she is my sister. I did not give her to you; why should I allow you to kill my sister?\u2019 the father said to my brother. His mother said, \u2018I shall put this girl\u2019s hand in your hand. When you wish to give it away, it will be like your sister\u2019s. \u2019 My brother thought, \u2018There surely cannot be two sisters. \u2019 I told him, \u2018The old man will die soon. I have brought him his dowry of ten yens of silver. It is no small gift for him. If I do you the favour now, will you not have respect for the father when he is dead and gone?
a town, and the largest inland city of the ancient Aztlan (Aztatlan, Azatlan) located near the modern ruin of Santa Anas, Sonora, Mexico. \nAtsa was a settlement among the 13 Tamoanchan cities of the Nahua people that established their settlement in Northern Mexico by about 1000 A. D. The city of Atsa (its people Atsatla) belonged to ẗhe twelve tribes of Tamoanchan,̈ which may be read in the Aztec codices as T̈lascallaänd which are the T̈lascalanp̈eoples of the 13 cities or a confederation of peoples that were among the dominant rulers of the Valley of Mexico, and of western Mexico in general, during the pre-eminent period of Aztec civilization. \nAtsa as well as the other Tamoancan cities of the Aztecs, is said to be on a plain of jade and silver and to have a number of large, flat-roofed buildings. These buildings are described as being in ruins in 1519, during Hernan Cortes’s first visit to Mexico. Although it is not known if the Tamoancan cities were named before or after contact with the Spaniards, it may be that the descriptions are only describing Aztec ideas of the 13 cities by its inhabitants. Atsa is said to be located at a point where a landform resembling a large lake (a m̈ountain)̈, was formed at some point in their past that has since dried. It is here that it is said the Atsatla people established their community. This article appeared in the Ceylon Today on 24th January 2016. \nI have been an Anglican Priest and a member of various committees in the Ceylon Evangelical Lutheran Church (CELC) for many years. I have observed that most of the CELC clergy and Laity are engaged in one or the other form of mission; whether as pastor, deacon, Sunday School teacher, a member of the Christian Education Council (CEC), Church Council or its equivalent. But do we realise the full significance of the words of the Apostle Paul in the first Reading?\nThe readings for this Sunday continue in Paul\u2019s second letter to the church in Corinth, where he warns the Christians against the excesses of the Eucharist. He warns them not to drunkenness during the Eucharist. He also writes: \u201cDo not, for the sake of food, destroy the work of God\u201d (6:12). \nThese were the words Paul used in his letter to his beloved Church: \u201cEat anything God has created to make you healthy. \nYou should not feel guilty about eating any of these foods, but you should not eat them if they cause other people to stumble\u201d (Romans 14:14-23, CEB). \nIf we want to do God\u2019s will, we will learn what pleases him. (Eph. 5:10). If we want others to respect God\u2019s law, then we must avoid even the appearance of sin (1 Thess. 5:22). In other words, we must seek to please God in all of our actions. \nAs Catholics we celebrate the Holy Mass by taking communion of the Bread and Wine, in the name of the Father, the Son and the Holy Spirit to remind us of our call to love and serve one another. \nSt. Paul said: \u201cA little yeast spreads through the whole batch of dough\u201d (Gal. 5:9). Christians should celebrate God\u2019s love in a special way. We are called to live for the next generation. If we drink to excess at the Eucharist it ruins the lives of our children. \u201cA little yeast\u201d spreads quickly through the family: an immoral step, a broken marriage, divorce and abuse: All these are examples of the \u201clittle yeast\u201d that destroys families. \nBut there\u2019s more. \u201cBecome sober, and stop sinning. Then some apostle will not be wrongly accused of being responsible for sinning. Anyone who has been stealing must steal no longer\u201d (v. 8). Our call as Christians is to lead exemplary lives. As such, we must act in accordance with our commitment to the \u201cLove of God and of our Neighbour\u201d. We are called to work with integrity. We are called to give to society. If a person is addicted to any alcoholic beverage or to any other type of drug, it will be difficult for him or her to lead a moral life. \nIt is also said that the words of the Apostle Paul in the First Reading \u201care still being applied today\u201d!\nOur forefathers who were the followers of John Wesley the Founder of Methodism made this famous saying to describe a Methodist: \u2018 A Methodist will find his way home from the most God forsaken, lawless, brutalized and degraded part of the earth, to the humble home of his childhood or his God: To find his way when the sun sets and the sky is filled with dense shades of night; or on the banks of the Red river, amid the solitudes of the Mexican desert, or in the dreary solitude of the frozen wastes of the Arctic Ocean\u2026\u201d. \nIf we ask ourselves some important question we are compelled to ask such question whether this is true for every individual member of the churches, whether it is true for the clergy or laity. \nThe Methodist Church in the United Kingdom, for example, is known as the \u201cMother Church\u201d. In 2014, a survey revealed that 21% of the British public was unaffiliated to any religion. But 50% of the people in England and Wales were prepared to \u201ctry and become a Christian\u201d if invited, despite not going to church. "