Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
Authors: Jeonghoon Kim
Jung Hyun Lee
Sungdong Kim
Joonsuk Park
Kang Min Yoo
Se Jung Kwon
Dongsoo Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
Parameter-Efficient Fine-Tuning
Neural Network Quantization
TL;DR: PEQA, a simple yet effective memory-efficient fine-tuning method that bridges the gap between PEFT and quantization. it can be employed to align quantized LLMs with task-specific adaptation and to restore the performance of FP LLMs post-quantization.

Abstract: Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern.  
Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase.
To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) – a simple yet effective method that combines the advantages of PEFT with quantized LLMs. 
By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage.
We employ PEQA-tuning for task-specific adaptation on LLMs with up to $65$ billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. 
Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.

Memory-Effiient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
Jeonghoon Kim NAVER Cloud
Jung Hyun Lee* NAVER Cloud onlivad1010gmaill com
jeonghoon www.ellmmaal......m.mmamaamaaaam
Sungdong Kim NAVER Cloud, KAIST AI sungdong www.@navercorp.................. com
Joonsuk Park NAVER Cloud, NAVER Al Lab, University of Richmond
park@joonsuk.orz org
Kang Min Yoo NAVER Cloud, SNU AT Center kangmin.
Se Jung Kwon NAVER Cloud
se jungungoou www.nnnavereorr.ccom
Dongsoo Lee NAVER Cloud
dongsoo. www.naeeeerrr.ccoo com
Abstract
Large language models (LLMs) face the challenges fine-tuning and deployment due to their high memory demands and computational costs. While parameter- efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern. Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase. To bridge this gap, this paper presents Prraamete--Effcientt and Quantization-awacc Adaptation (PEQA) a simple yet effective method that combines the advantages of PEFT with quantized LLMs. By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage. We employ PEQA-tuming for task -ask-spec adaptation on LLMs with up to 65 billion parameters. To assess the logical reasoning and language comprehension PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using instruction dataset. Our results show that even when L.MMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in in-context learning. and comprehension be resiliently restored to (or even improved over) their full-precision original performances with PEQA.
Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Decomposition (Quontization)
Fine-tuning for tosk T
Fine tuuned
Frazen
1.2
3.1
2.3
4.3
Full-Precision
Step
Sub-4-bit
Step 2
Sub 4 bit Int
Sub bit Int
0.2
0
Task
0.9
1.0
0.3
2.5
Switch
0.4
Wo
Parameter-effiient Fine-tuning
Deployment
Figure I: Illustration of our proposed PEQA scheme where A indicates the element-wise product of A and B. PEQA is memory-efficient fine-tuning method for quantized large language models that updates only the quantization scale while keeping the integer matrix frozen. Notice a significant reduction in memory footprint when full full-precision weights are converted into sub-4-bit integers.
Introduction
Large language models (LLMs) such as PaLM, LLaMA, and the GPT-series [17 have demonstrated unprecedented levels of task musk-generalization ability in various applications, including dialogue systems, question answering, summarization, and translation 8 g. While they can follow instructions and learn to solve tasks via in-context task descriptions or few-shot examples [I0], fine-tuning allows LLMs align their behavior with desirable traits, such as following instructions more precisely [[]] or adhering to certain principles [1].. Additionally, fine-tuning can improve the scaling curve by exposing the model large collections of task-specific instruction datasets, leading significant performance enhancements in various unseen downstream tasks 13177 However, the immense computational cost fully fine-tuning large-scale models presents challenges for researchers and developers, especially given that LLMs have billions even trillions of parameters [18]
In response, several parameter-efficient fine-tuning (PEFT) methods have been introduced 211 which only update a small number of parameters compared to the pre-trained weights of LLMs. PEFT notably reduces the number of learnable parameters, making the fine-tuning of pre-trained LLMs viable by ensuring that the optimizer states' memory usage becomes negligible. These strategies lead to decreased memory usage during training and more efficient storage and seamless transitions of task dask-specifically fine-tuned parameters during deployment. Nonetheless, LLMs as a whole still demand significant memory, and further reductions are attainable through model compression. As outlined in Hu et al. (21) for instance, LoRA can cut the memory usage during the fine-tuning of GPT-3 175B from 1.2TB to 350GB. However, model still requires approximately 350GB of memory for parameters in half-precision floating-poin point format.
Quantization is a favorable method for both compressing and accelerating neural networks by discretizing parameters into low-bit integers while maintaining a shared high-precision scale within each parameter group (e.g.. channel or layer). However, during training phases, quantization aware training (QAT) 22 25 mandates updates for all parameters, rendering it not -effiittr-effccient. quantization (PTQ) 26 29 is executed after training. most existing quantization schemes primarily target the deployment phases. Although PTQ can be integrated with PEFT, when PTQ follows PEFT, the model remains intact during fine-tuning, not decreasing the memory usage. Conversely, if PTQ precedes PEFT, while there's reduction in memory usage during fine-tuning, no inference acceleration can be achieved due to the PEFT parameters during deployment.
Since
To bridge the gap between PEFT and quantization, we introduce the Parameter-Efficieet and Quantization-aware Adaptation (PEQA), simple yet effective quantization-awwe PEFT method. As illustrated in Figure] PEQA encompasses two steps: (a) Decomposition (Quantization) where the parameter matrix of each fully fully-connected layer is decomposed into matrix of low-bit integers and quantization scales; and (b) Fine-tuning wherein, for each downstream task, the quantization scale is fine-tuned while integer matrix remains unchanged. For the quantized LLMs, merely updating the quantization scale leverages the advantages of PEQA. As result, PEQA maintains the merits PEFT, such as fewer trainable parameters, along with efficient storage and swift switching of task task-specific parameters, Concurrently, it provides the benefits of quantization, including reduced

---

DRAM usage during both training and deployment, and inference acceleration due to fewer memory accesses at deployment.
Through this, we highlight the following:
We introduce PEQA, a method that fine-tunes only the quantization scales of quantized L.MMs, keeping the integer matrix frozen. It bridges the gap between PEFT and quantization, offering advantages such as reduced memory consumption during both training and deployment phases, seamless task transitions, and faster inference.
To empirically validate the approach of solely fine-tuning the quantization scale while freezing the integer matrix, we compare the perplexity of LLMs fine-tuned with QAT, PEFT (+PTQ), and PEQA. The results indicate that PEQA delivers competitive performance in comparison to QAT and PEFT+PTQ, even at sub-4-bit precision.
To assess the scalability and comprehension performance of PEQA, we apply PEQA to task- specific adaptation and instruction-tuning Despite the reduction in model size by a factor of to 5, PEQA demonstrates competitive performance up 65B LLM when compared to full full-precision baselines. The results suggest that even when LLMs are quantized into low-bit precision, the overall comprehension capability of quantized LLMs can be effectively restored to their original performance using PEQA.
2 Related Work
Large Language Models and Alignment Learning. Although LLMs have demonstrated great generalization capabilities through their sheer scales 2 30 and the emergent mechanism known as in-context learning []. they still require significant alignment follow natural language instructions [13]. adhere to ethical guidelines or steer towards harmlessness [12] utilize external tools [31] and to be grounded in knowledge to generate truthful answers [16 32). In particular, instruction-tunin -uning has been pivotal in enabling LLMs to generalize instruction-folle follo-fiooowinggg abilities, enabling them to solve allowing the
seemingly any NLP task with only the description of the task in natural text [13 33
models to be accessed in an interactive manner.
Effameter-Effiient Fine- Fine-Tuning. Fine-tuning leverages the generalization capabilities elicited from the general pretraining to specialize in specific domains and tasks [35] 36) or align the LLM with target behaviors [3). However, updating parameters in LLMs comes with a high computation cost and minimal compute environment required for gradient computation. As the hyper-scale era makes tuning for LLMs prohibitively expensive, both efficient and effective alternatives to fine-tuning have received considerable attention. Specifically, inspired by the sensitivity of LLMs to prompts [37] a line of works has proposed introducing trainable prompt embeddings prepended to the input text while freezing the original LLM parameters [19] 38 39]. As another approach, adapter modules (20) introduce sask-specific parameters, which are inserted between the pre-existing layers of the model Extending on this adapter-based approach, LoRA [1] employs the concept of low-rank bottleneck modules while demonstrating comparable performance full fine- fine-tuning. Subsequent works have unified the various versions and diverging approaches PEFT 40 41] by formulating them in a single mathematical framework. These parameter-efficien methods have shown comparable performance to full model fine-tuning, presenting cost-effective and efficient avenue for tailoring LLMs to specific tasks.
However, even with the adoption of PEFT, the inherent model size of the LLM remains challenge to handle. One immediate solution to apply post-training quantization (PTQ). but its interaction with task-specific parameters is still an area of active research. have been attempts to integrate PEFT and neural network quantization, including methods like Quadapter [42) and AlphaTuuning
43
Yet, these methods have primarily been explored in smaller models of 1.3B or fewer parameters. Dopendix[delineee the distinctions between our method and AlphaTuning.
Neural Network Quantization. Neural network quantization consists largely of quantization- aware training (QAT) and PTQ. QAT methods (22225) basically train not only quantization scales but all the parameters of a full-precision neural network to narrow the performance gap between the full- call-precision model and its quantized counterpart. Unfortunately, since QAT involves training all the weights of a full- full-precision network, it is not feasible to apply QAT to LLMs. To quantize LLMs, PTQ
3

---

3.5
Optimizer Statss Size
326GB
Parameter- Efficient
6.8M
4.0
Adaptation
4.15M
Parameter- Efficient Quant-awarc
Perplexity
4.5
6.39M
10.49M
52MB
Adaptation
5.0
1.33
3.28M
Model
131GB
13168
Sizee
32MR
5.5
1M
LoRA(FP16)
256B
36M
PEQA(4-bit)
Full
LoRA (QV4)
PEQA (4-bit, Ours)
PEQA
6.0
25
50
75
100
125
150
Fine-Tuning
(3-bit, Ours)
Model Size (GB)
(b)
Figure 2: (a) DRAM usage comparison of LLaMA-65B on various tuning methods and (b) perplexity over model size when tuning LLaMA models with LoRA and PEQA on Wikitext2 dataset. The size of a circle indicates the number of trainable parameters. For instance, the LLaMA-65B model with LoRA has size of 13IGB and 10.49M trainable parameters. Otherwise, LLaMA-65B with 4-bit PEQA has model size of 33GB and 6.8M trainable parameters.
techniques tailored to LLMs [26 45 have been presented. Although such PTQ approaches do not require learning all the parameters of an LLM at all, as PTQ occurs after training/fine-tuiing LLMs, PTQ cannot allow for compressing the model size during taaninngfineetuningg LLMs. To reduce the model size even during training/finettuiing LLMs, researchers have recently focused on combining PEFT with quantization.
3 Methodology
3.1 Problem Setup
Memory Demands in Fine-tuning, Given the significant computational demands associated with fully fine-tuning large language models, parameter-effice efffcietr-fficient fine-tuning (PEFT) methods have been introduced [19 21 46). One of the primary goals of PEFT methods is to reduce memory usage of the optimizer state during training, specifically by reducing the number of learnable parameters. While existing PEFT techniques do decrease memory consumption of the optimizer state and also narrow the accuracy gap between full fine-tuning and PEFT, the pre-trained weights of large language models still demand substantial memory space. When applying LoRA (21 LLaMA-6SB. for instance, even though storing optimizer states for trainable parameters consumes only 52MB (only query and value matrices are adapted with LoRA rank of 4), the model size still occupies huge portion of DRAM usage due to the frozen FP16 weights of pre-trained model. To make PEFT more efficient, reducing the model size is an indispensable requisite. Given that LLMs mostly consist of fully-connected layers, compressing the weights of fully-connected layers is key factor in compressing the model size and thus leading more efficient PEFT. Inference Latency of Large Language Models. During text generation inference, an autoregressive LLM generates tokens sequentially. A significant portion of the inference latency arises from matrix-vector multiplications, as opposed to matrix-matrix multiplications. Given that the batch size during inference typically small [27 28), matrix-vector multipicaations tend be memory-bounn Specifically, accessing global memory, such as DRAM, is expensive on contemporary high-end GPUs. Thus, the number of weights loaded into registers profoundly impacts the speed of multiplication between a matrix and a vector. To decrease the number of weights (subsequently increasing the weights loaded into registers), quantization is widely researched method for both compressing and speeding up neural networks [29] 44,47 (47. While quantization-awee awwae training (QAT) imposes significant load on both computation and memory, post-training quantization (PTQ) is often viewed as fallback strategy among traditional quantization techniques enhance the generation latency of I.LMs. To both accelerate L.M inference and retain advantages of PEFT, an innovative alternative approach should be pursued.

---

Table 1: Comparison of PEQA with other methods using LLaMA 65B on the DRAM usage and training time during iin-tuning, the DRAM storage for deployment, the inference acceleration, and tast-switching efficiency. The DRAM usage estimation for PEFT based on LoRA. PEFT+PTQ denotes PTQ after PEFT and PTQ+PEFT denotes PTQ before PEFT.
DRAM
DRAM (Deployment)
Inference Speed
Task-
Method
(Fine-Tuning)
Switching
Full Fine-Tuning
457GB
131GB
Slow
Slow
PEFT
131GB
131GB
Slow
Fast
PEFT+PTQ
131GB
33GB
Fast
Slow
PTQ+PEFT
33GB
33GB
Slow
Fast
PEQA (Ours)
33GB
33GB
Fast
Fast
3.2 Parameter-Efficien and Quantization-aware Adaptation (PEQA)
Quantization reduces bit-precision for inference acceleration, less storage and increasing throughput. INT8 quantization, which lower the bit- bit-precision for both activations and weights, utilize dedicated engine to effectively accelerate arithmetic computation [48]. This is effective for large batches where computing speed matters but less for smaller batches constrained by memory. To tackle this memory issue, weight-only quantization keeps high precision for activations (e.g., FP16) but compresses weights 4-bit less, targeting memory I/O enhancement in modern GPUs 28 47 For simplicity, we mainly focus on low-bit weight-only quantization in a linear asymmetric per-channel context in this paper.
For pre-trained weights of a fully-connected layer Wn E _= while PEQA can be applied to quantized LLMs, we first quantize Wo. In other words, for given bit-width b, quantized pre-trained weights Wo can be written as
Wo = 80 Wo = 80 (clamp
Wo $o
+ 2o, O, 2b -1) -20):
(I)
where A B, [.], and clamp(-, a, b) indicate the element-wise product of A and B, the rounding function, and the clamping function into the range [a, b), respectively, while per-channel scales and zero-points (namely, 80, 2n E [=* are initialized to minimize I -aal? Notice that sp and 2o are not related to any downstream task. Here, we freeze Wo = clamp
Wa + zo. 0, - 20
which is the integer quantization indices of Wo, for every full-connected layer in pre- pre-trained LLM. And then fine-tune only so (residing outside the clamp function in Eq. I
while sharing W across all downstream tasks. Consequently, quantized pre-trained weights Wo are adapted downstream task as follows:
W = (so + As) W. = (So + As) (clamp
Wo so
+ za, 0, 0,2" -1) 20).
where As E 1x represents the gradient update of Su obtained by adaptation a downstream task. We dub Eq. as Parameter-Effice and Quantization- aware Adaptation (PEQA). PEQA is memory-effeiient fine-tuning method dedicated quantized LLMs by solely updating quantization scales 8p- With W. being frozen and shared for all downstream tasks, 80 + As are task-specific parameters in PEQA, which can be quickly and easily swapped when it is needed to switch to different downstream task. Note that, PEQA can be seamlessly applied not only weight-only quantized LLMS but also to weight-activation quantized ones. The overall procedure of PEQA is described in Figure I in detail.
3.3 Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization
PEQA designed to have the advantages of both existing PEFT methods [19 21 and quantized LLM 28 44 47 49). We summarize the benefits of PEQA in this subsection.
5

---

Table To empirically confirm the validity of PEQA's approach, we compare the perplexity (PPL) of fine-tumed LLMs through QAT. PEFT+PTQ. and PEQA on Wikitext2 [5]] for GPT-Neo 2.7B, GPT-J 6B, LLaMA 7B, and L.LMMA 13B. Weights are quantized into either 3-bit or 4-bit per channel, without group size [28 49). LoRA configuration is set to QV4. The lower PPI., the better.
Method
W Bits
GPT-Neo 2.7B
GPT-J 6B
LLaMA 7B
LLaMA 13B
QAT
4
11.07
8.81
5.76
5.26
LoRA + OPTQ
4
12.09
8.91
7.13
5.31
PEQA (Ours)
4
11.38
8.81
5.84
5.30
QAT
3
12.37
9.60
6.14
5.59
LoRA OPTQ
21.93
11.22
19.47
7.33
PEQA (Ours)
12.54
9.36
6.19
5.54
Benefits of PEFT. By solely updating the quantization scales, PEQA substantially reduces the memory overhead associated with optimizer state, feature consistent with other PEFT approaches. Notably, the utilization of quantization scales 80 + As allows PEQA swiftly and effortlessly switch between task-specific parameters. This capability positions PEQA as ideally suited for deployment of quantized LLMs as a service, mirroring another key advantage of earlier PEFT methods. Note, however, that such capability is not present in PEFT+PTQ (i.c., the case where PTQ is applied after PEFT) due the non-reversille quantizers, such as the rounding function.
Benefits of Quantization. Previous PHFT methods freeze the pre-trained weights W, and utilize additional learnable parameters to reduce the memory usage of optimizer state. Similarly, PEQA freezes quantized pre-trained weights Wo (which are the integer quantization values of Wo) and fine-tunes quantization scales 8p- Since W, is a b-bit integer matrix, not only can PEQA reduce the optimizer states' size but also the model size, leading to even greater efficiency the PEFT scheme, as illustrated in Figure 2a In addition, since W is a b-bit quantized matrix, PEQA can speed up token generation process at inference through dedicated kernels that accelerate the multiplication between a quantized weight matrix and a half-precision activation vector, as described by Frantar al. (88) Lin al. 47). and Park et al. 49 It is worth noticing that employing PTQ before fine-tuning (PTQ+PEFT) [50] allows for memory- efficcienn fine-tuning and seamless task transition for the quantized LLM; however, PTQ+PEFT not able to inherit from inference acceleration of quantization. As result, PEQA can achieve both model compression in the process of fine-tuning and inference accelcration for fine-tuned models with marginal performance degradation compared to LoRA, one of the state-of-the-an PEFT techniques, as shown in Figure 2b
The comparison of PEQA with other methods using the LLaMA 65B is summarized in Table T
4 Experiments
In this section, we empirically validate the effectiveness of our proposed PEQA method by examining its performance in both parameter-efficient fine-tuning (PEFT) and as quantization method. We
achieve this goal by using series of benchmarks [52 57), datasets [51 58 59), and LLMs 4
60 61 that have been publicly introduced. In Section to empirically confirm the validity of
fine-tuning only the quantization scale while freezing the integer matrix, we compare the perplexity of
training (QAT), PEFT (+PTQ), and PEQA. In Section 4.2 to evaluate PEQA's scalability and task-specific adaptation performance, we fine-tune tune and assess
fine-tuned L.Mss through quantization-awes
LL Ms on the Wikitext2 [51] and PennTreeBank (58) datasets using PEQA and LoRA 21I. Section
is dedicated to showcasing PEQA's performance-restorin capability through instruction-tuning
on the Alpaca (59] dataset after round-to-neared (RTN) quantization over the full-precision original
model.
To assess PEQA's performance as PEFT method, we contrast PEQA with LoRA, which is currently recognized as one of the leading PEFT methods. As discussed in 3.1 we employ baseline case that merges OPTQ 28 the state-of-the-art weight-only post-training quantization (PTQ) method for LLMs, with LoRA in order to evaluate PEQA's quantization capabilities. In the context of LoRA, QV4 signifies the application of query and value layer weights with a LoRA rank of 4, while QKV016

---

Table To show scalability of PEQA, the perplexity (PPL) on Wikitext2 and PennTreeBank (PTB) was compared with LoRA and PEQA. In this comparison, only the weights were quantized into 3-bit and 4-bit per-channel without group size. LoRA configuration is set to QV4. lower PPL. value
indicates better performance.
GPT-Neo
GPT-J
LLaMA
LLaMA
LLaMA 30B
LLaMA 65B
Method
W Bits
2.7B
6B
7B
13B
Wikitex12
LoRA
16
10.63
8.50
5.53
5.06
4.06
3.82
LoRA+OPTQ
12.09
8.91
7.13
5.31
4.39
4.10
PEQA (Ours)
11.38
8.84
5.84
5.30
4.36
4.02
LoRA+OPTQ PEQA (Ours)
21.93
11.22
19.47
7.33
5.94
5.32
12.54
9.36
6.19
5.54
4.58
4.27
PTB
LoRA
16
15.92
12.92
9.14
8.52
7.21
7.11
LoRA+OPTQ
18.83
13.46
11.22
8.83
7.55
7.46
PEQA (Ours)
16.55
13.30
9.69
8.64
7.68
7.36
indicates the application of query, key, value, and output projection layer weights with LoRA rank of 16. For PEQA, we utilize round-to-nearest (RTN) for the initialization method of quantized LLM.
4.1 Comparing Quantization Capabilittiss: PEQA vs. QAT vs. PEFT+PTQ
Table presents the perplexity when various quantized LLMs are fine-tuned using QAT, PEFT (+PTQ), and PEQA. To validate our approach, PEQA, which solely fine-tunes the quantization scale as outlined in Eq. 2 while simultaneously maintaining the integer matrix in frozen state, we use QAT as an upper bound and PEFT+PTQ as lower bound. Note that QAT, unlike PEQA, updates all parameters including pre-trained weights well quantization scales. Table reveals the competitive performance PEQA compared to QAT. Furthermore, our observations indicate that PEQA consistently outperforms the combination of LoRA and OPTQ for any selected model, regardless of whether 3-bit or 4-bit setting is employed. Such superior performance can be attributed to PEQA's method of fine-tuning quantized LLMs, which minimizes the final task loss the full training data, a capability that OPTQ lacks. Detailed settings are Appendix
B
Diving deeper into the comparison between QAT and PEQA, it is important to note that QAT minimizes the final task loss computed from weight-only quantized model, as described in Eq. I with respect to both Wo and so. Note that QAT includes all pre-trained weights for training, resulting in the practical model size limitation of L.LMs under investigation being capped at 13B in our experiments. Despite the fact that QAT also updates Wo in Eq. [ which is one of the most simple and straightforward approach though, we observe that the performance gap between QAT and PEQA narrows when the 4-bit association is introduced, especially as the size of LLMs increases. Impressively, PEQA can even outperform QAT in 3-bit setting, a notably low-bit setting that challenges OPTQ in terms of quantizing LLMs. These findings suggest that the approach of PEQA, solely updating quantization scales while freezing the integer quantization values of pre-trained weights, can achieve performance comparable to that of QAT.
4.2 Task-specific Adaptation with Wikitext2 and PennTree Tree Datasets
Task-specific Adaptation and Scalability. We evaluate the task-specific adaptation performance and scalability of PEQA by employing GPT-Neo, GPT-J, and LLaMA models (up to 65B) on the Wikitext2 [51] and PennTreeBan (PTB) (58] datasets. The adaptation performance
PEQA is compared with LoRA, with its configuration set to QV4. As depicted in Table we note gradual convergence of PEQA's perplexity to that of full-precision LoRA, with only marginal PPL. degradation as the model size expands. Thus, Table demonstrates that PEQA, compared technique that utilizes prel-precision pre-trained language model (PLM), can maintain a competitive perplexity level in LLMs while concurrently reducing DRAM usage through low-bit quantized
prominent PEFT
7

---

Table Number of learnable parameters and model size of GPT-Neo, GPT-J and LLaMAs. PEQA configuration is set to 4-bit or 3-bit channel-wise quantization.
GPT-Nco
GPT-J
LLaMA
LLaMA
LLaMA
LLaMA
Method
2.7B
6B
7B
13B
30B
65B
# of
LoRA (QV4)
1.31
1.84
2.10
3.28
6.39
10.49
Learnable
LoRA (QKVO16)
5.24
7.34
8.39
13.11
25.56
41.94
Param. (M)
PEQA (Ours)
0.74
1.03
1.36
2.13
4.15
6.80
Model
LoRA (QV4)
5:30
12.10
13.48
26.03
65.06
130.57
Size
PEQA (Ours, 4-bit)
1.53
3.65
3.77
7.01
16.92
33.45
(GB)
PEQA (Ours, 3-bit)
1.21
2.94
2.96
5.42
12.90
25.35
Table Multi-scale (grouping) performance with PEQA-tuned tuned LLaMA 7B and 13B on Wikitext2 where indicates the group size 49) The perplexity consistently increases as PEQA take on more learnable parameters.
Model
Bits
Channel- Wise
g256
9 g128
g64
LLaMA 7B
4
5.84
5.69
5.66
5.64
6.19
5.96
5.91
5.89
LLaMA 13B
5.30
5.18
5.16
5.16
5.54
5.40
5.37
5.34
weights. Notably, for 3-bit quantization, PEQA experiences less performance degradation as the model size decreases due to extreme low-bit quantization compared to the combined LoRA and OPTQ. To further elucidate our findings, we have provided figures illustrating the results of 3-bit and 4-bit PEQA in the Appendix D The comprehensive results indicate that for the deployment stage. PEQA allows models with larger parameters to operate under DR AM usage constraints, outperforming full-precision PEFT methods. For instance, under restricted DRAM footprint, large L.LaMA models can be explored using PEQA, while full-precision LoRA permits only smaller LLaMA models. Additional results with OPT A. ranging from 1.3B to 66B models are included in the Appendix B The detailed experimental settings are also included in the Appendix
Model Size and Number of Learnable Parameters. In an effort estimate the DRAM usage necessitated by PEQA and LoRA during training and deployment, we outline the number of learnable parameters (for training) and the model size (expressed in gigabytes. GB, for deployment) in Table 4 As demonstrated in Table 4 PEQA involves fewer learnable parameters than LoRA when each channel of pre-trained weights. For instance, PEQA has approximately 1.54 times fewer learnable parameters for LLaMA models than LoRA (QV4). In addition to having fewer learnable parameters, PEQA, through low-bit weight quantization, can also reduce the model size, which captures huge amount of the DRAM footprint in fine-tuning LLMs. Remarkably, when fine-tuning LLaMA 30B using PEQA with 4-bit precision, the resulting model size is significantly smaller than that obtained by adapting 13B through LoRA, and slightly larger than the model adapted from LLaMA 7B using LoRA. Additionally, comparison of the memory peak during training between PEQA and LoRA provided in Appendix L
quantization scale assigned
Group-wise Quantization. Group-wist wise per-channel quantization 49
where weight groups in
channels share quantization parameters, maintains accuracy lower bits. In Table (5) we present that the performance incrementally improves as more learnall parameters are incorporated into PEQA. particular, for Table we examine various group sizes (denoted by g) when quantizing the weights (9 62 Through relatively straightforward grouping (employed to regulate the number of learnable parameters for PEQA), the perplexity incrementally decreases as more learnable parameters utilized. Detailed settings are in Appendix G

---

Table 6: Common-sense reasoning and in-context learning performance of parameter-efficien instruction secnctd LLaMAs [6] using Alpaca datasets. LoRA configuration is set to QKVO16. Quantization precision PEQA set to 4-bit per-channel without group size. Note that ARC-C, ARC-E and OBQA stands for ARC-Challenge, ARC-Fasy, and OpenBookQA respectively.
Model
Method
PIQA
HellaSwag
ARC-C
ARC-E
OBQA
Average
Params
Size (GB)
Zero-Shot
7B
13.5GB
77.3
73.0
41.4
52.5
42.4
57.3
LLaMA
13B
26.1GB
79.1
76.2
44.5
59.9
42.2
60.4
30B
65.1GB
80.1
79.2
45.5
58.9
42.0
61.1
7B
13.5GB
78.6
73.3
43.7
55.8
43.0
58.9(+1.6)
LoRA
13B
26.1GB
79.6
76.7
46.3
62.0
43.2
61.5( (1.1)
30B
65.1GB
81.8
80.3
61.6
62.9 (2.9(+188)
7B
3.8GB
77.9
71.4
42.4
57.2
42.0
58.2(+0..)
+ PEQA
13B
7.0GB
78.9
74.0
46.4
62.5
42.8
60.9(+0...
30B
16.9GB
80.3
78.4
49.8
63.3
42.8
62.9(+1.8)
Five-Shot
7B
13.5GB
79.4
75.3
45.6
65.8
44.0
62.0
I.LMAA
13B
26.1GB
80.0
78.4
50.4
70.8
47.2
65.4
30B
65.1GB
82.5
82.2
56.2
74.9
47.0
68.6
13.5GB
75.2
66.5
63.0(+1.0)
+ LoRA
13B
GGB
81.1
78.8
53.5
72.4
47.0
66.6(+1.1)
30B
GGB
84.1
83.3
59.5
79.2
50.6
71.4(+28)
7B
3.8GB
78.9
73.2
45.1
65.4
44.0
61.3(-0.7)
+ PEQA
13B
7.0GB
80.7
76.0
50.9
71.6
48.0
65.5(+0.1)
30B
16.9GB
82.7
80.2
56.8
75.5
47.6
68.6(+0.0)
4.3 Instruction-tuning with the Alpaca Dataset
Although inference cost and training efficiency crucial, the methods of PEFT and quantization have not been extensively explored. While LoRA and PEQA have been evaluated on adaptation performance for task speck-ic purposes as discussed in Section 4.2] has not been widely corroborated that fine-tuning via PEFT can retain the performance for unseen tasks. Thus, given that RTN quantization results in non-negligible performance degradation in LLMs, it is important assess how much the performance of low-bit bit quantized LLMs is degraded on comprehensive tasks. To address these concerns, we conduct comprehensive experiments, benchmarking our techniques on prevalent instruction followwng datasets and assessing the response quality of PEQA-tunee LLMs. Furthermore, determine if PEQA can regain the performance of full-precision LLMs, we employ RTN quantization in conjunction with PEQA instruction-tunn tuniinnntttnnnggggii across LLaMAs.
Experimental Settings. We train LLaMAs 7 in various sizes the Alpaca dataset (59 which is one of the popular instruction ciiruction-folloiing datasets generated from outputs of InstructGPT [1 Then, we the models on other downstream tasks such as common sense reasoning tasks 52 55 and massive multitask language understanding (MMLU) [56]. Due to limited time and resources, we could not conduct an exhaustive search over hyper-parameters such as the learning rate or epoch. Instead, we followed the training recipe from Taori et al. [59]. The LoRA configuration is set to QKVO16. Detailed settings can be found Appendix
Common-Sense Reasoning. We conducted an experiment on five tasks 55 to assess whether the performance common-sense reasoning and in-context learning can sustained even after instruction-tumi -otninn LLMs on the Alpaca dataset via LoRA PEQA. As depicted in Table 6 the results show that LLMSs fine-tuned with LoRA or PEQA maintain consistent trend common-sense reasoning tasks. Furthermore, since PEQA's performance aligns closely with that of full-precision

---

Table Massive Multitask Language Understanding (MMLU) benchmark performance of PEQA- tuned LLaMAs using Alpaca datasets. Five-shot accuracy is reported for the MMLU. Quantization precision of PEQA set to 4-bit. When we quantize LLLMA [ into 4-bit precision using the RTN method, no group size is applied. For LLaMA2 7 a group size of 256 is used with the RTN method.
Note that RTN stands for round-to-nearest in the table.
Params
Model Size
Humanities
STEM
Social Sciences
Other
Average
LLaMA 6
7B
13.5GB
32.6
29.6
38.0
37.9
34.4
13B
26.1GB
42.8
36.1
53.3
53.2
46.1
30B
65.1GB
54.6
46.5
66.1
63.4
57.4
+ RIN
7B
388GB
28.4
25.6
26,9
31.8
28.3
(w/o group size)
13B
7.0GB
30.5
27.2
35.5
38.8
32.8
30B
16.9GB
39.6
34.0
46.1
49.7
42.1
PEQA
7B
3.8GB
35.7
30.9
38.2
40.0
35.8
13B
7.0GB
42.8
37.7
53.6
49.0
45.0
30B
16.9GB
51.1
44.1
62.4
60.7
54.3
LLaMA2 7
7B
13.5GB
43.3
37.0
51.8
52.4
45.9
13B
26.0GB
54.4
44.2
63.4
60.8
55.7
70B
138.0GB
65.2
57.9
80.3
74.77
69.1
RIN
7B
3.8GB
39.5
35.5
49.3
49.9
43.2
(g256)
13B
7.0GB
50.2
42.6
61.3
59.7
53.2
70B
35.3GB
63.7
55.9
78.4
71.6
67.0
PEQA
7B
3.8GB
52.0
38.4
54.1
52.0
48.1
13B
7.0GB
60.5
45.0
63.3
57.0
55.3
70B
35.3GB
73.9
55.3
77.8
68.2
67.5
adaptation, this consistency observed even when the model size has been reduced through low-bit weight quantization. We utilized the evaluation code from Eleuther Al's iocauuation-hrnnes [63
Massive Multitask Language Understanding. To assess whether the performance of PEQA- tuned models can be restored to the levels of full-precision model's performance, starting from RTN performance, we test our models on the MMLU benchmark containing 57 multiple-choice problems across various domains and levels of knowledge [56). In this experiment, utilize the RTN results as baseline to determine the extent of degradation on quantized LLM. As shown in Table 7 instruction-tune tuttinnnnttniggggg with PEQA boosts the performance of RTN quantized models. This observation supports our claim that our approach enables LLMs to regain their few-shot in-context learning and understanding capabilities, even though they significantly smaller than their original model size through quantization. Unfortunately, it seems that the PEQA-tuning does not achieve the best performance in fine-tuning larger models. This might be because PEQA-cuning did not been sufficiently explored different epochs or learning rates. Nonetheless, the observation that the performance of the quantized LLaMAs restored through PEQA-tuning using an instruction- following dataset highlights the potential to further enhance the accuracy of PTQ methods.
Conclusion
Fine-tuning aligns large language models (LLMs) with specific purposes. To maintain the compre- hensive capabilities of LLMs while effectively aligning them, we introduce PEQA, a method that seamlessly combines the advantages of parameter-efficien fine-tuning (PEFT) and quantization in LLMs. PEQA not only reduces DRAM consumption during fine-tuning but also accelerates inference latency for deployment by retaining weights in low-bit quantized format. Through rigorous testing across various datasets and LLMs, we have found that PEQA can match the performance of full- precision baselines in task-specific adaptations, even with significant reduction model size. When combined with instruction-tun -unnung PEQA's performance demonstrates its ability to both preserve and enhance comprehensive knowledge after the inherent compromises of quantization, recovering the performance of original model by simply updating the quantization scales of the quantized LLM.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. volume 33, pages 1877-1901, 2020.
[2] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstert Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[3] Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressiie Language Modeling in PyTorch, 8 2021. URL https://www http://www.gtth..... com/
Souutherai/gpt-neeeee
[4] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01063 2022.
[5] OpenAl. Gpt-4 technical report, 2023.
[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[7] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
[8] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3008 3021. Curran Associates. Inc., 2020. URL URt www.ius........... cc/ hace/pppr_ftte files/ paper//020/ff1ll/ I1888655662226888510888888888333
pdf
[9] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds Mach., 30(4):6811664) dec 2020. ISSN 0924-6495. doi: 10.1007/s 102000001022-000999554----
[10] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference pages 11048-110644 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL httpp:://ccannaolllgyg wwwp////2222 emnlp main. 7799
Empirical Methods in Natural Language Processing,
[11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances Neural Information Processing Systems, 15:27730-27744, 2022.
11

---

[12] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.08807, 2022.
[13] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL
[14] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multi- task prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https::/
[15] Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107-6122, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. https:/:: hatps///ccaattooogyo .org..ogg/2g 2022 emnlp-mane 410
[16] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05900 2022.
[17] Yi Tay, Jason Wei, Hyung Won Chung, Vinh Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprini arXiv:2210. 11399, 2022.
[18] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435.
[19] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2021.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019. URL hattps://proceddigg mlr
press/ v97
press/v97/houue www/houlsby19..com html
[21] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL htttt::////pprreeee
[22] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In The IEEE Conference on Computer Vision and Pattern Recognition 2019.
(CVPR), pages
[23] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. International Conference on Learning Representations, 2020. URL
12

---

[24] Xiandong Zhao, Ying Wang, Xuyi Cai, Cheng Liu, and Lei Zhang. Linear symmetric quanti- zation of neural networks for sow-precision integer hardware. International Conference on Learning Representations, 2020. URL htttps://pperrete hwt.://wwwwee.oormmmmodd11111111
[25] Jung Hyun Lee. Jihun Yun, Sung Ju Hwang, and Eunho Yang. Cluster-promoting quantization with bit-drop for minimizing network quantization loss, 2021.
[26] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07333, 2022.
[27] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.0186) 2022.
[28] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023, URL https://openrerierr /ttppp///pppeeeeeeeeeeeerrroootp http://orummmddddcBBPittti
[29] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211. 10438, 2022.
[30] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203./555555 2022.
[31] Timo Schick, Jane Dwivedi-Ya Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint www.xx:302.0476111 2023.
[32] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answeriin with human feedback. CoRR, abs/2112.09332 2021. (tRp::ts:://arre org/ htsp/bbb/221122 09332
[33] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated in- structions. CoRR, abs/2212.10560., 2022. doi: 10.48500arxx..2212..066 URL https
//doi.org// 1...0..855//a//.. wtwp:aavvvv........... 105600
[34] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL
htttps://assy..org//ll//////// 3i-vvuanaaa
[35] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04530 04.055 2018.
[36] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
[37] Timo Schick and Hinrich Schüizee Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint urXiv:2001.07611 2020.
[38] Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 th Internutional Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 I51- URL https: http::::////caaahhoo/osg/ org. 2021 www.aclllooo .53 353
13

---

[39] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for grrammteer-fffiienn prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18833/v1/221111 -en.llll-emllpmiiiii URIhttps:: haattps::/aalaatthllgg.. .rg. 2221//2211111 emnlp-main 243
[40] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kink-Kirkpa and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL https:://operreeeee...eooo
[41] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma. Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efficiet language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 6253-6264, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653w1/2022.cclllooogg https://aclanthol..g arg-2022...11-oong 433
URL
[42] Minseop Park, Jaeseong You, Markus Nagel, and Simyung Chang. Quadapter: Adapter for GPT-2 quantization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNIPP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2510-2517. Association for Computational Linguisties. wrg/2022. 2022. findingsnneiininn emnlp..--emn...... 185
2022. URL/bttss://acladdhae
[43] Se Jung Kwon, Jeonghoon Kim, Jeongin Bac, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Bycongwook Kim, Jung-Woo Ha, Nako Sung. and Dongsoo Lee. Alphatuning: Quantization- aware parameter-efffiient adaptation of large-scale pre-trained language models. In Yoav Goldherg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, Decem- ber 7-,,, 2022, pages 3288-3305. Association for Computational Linguisties, 2022. URL org/ 2222200222222 findings- emnlp. 240
[44] Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. FlexRound: Learn- able rounding based on element-wise division for post-training quantization. Proceed- ings of the 40th International Conference Machine Learning, volume 202 of Proceed- ings of Machine Learning Research, pages 18913. PMLR, 23-29 Jul 2023. URL https://proceedings mlr press/ vttp//202////// 1ee22/eee2hhh.m html
[45] Jung Hwan Hco, Jeonghoon Kim, Beomseok Kwon, Bycongwook Kim, Se Jung Kwon, and Dongsoo Lec. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint 23X.::2339./55333 2023.
[46] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-ard parameter-efficient fine-tuning methods. https://github. com/ huggingface peft 2022.
[47] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
[48] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[49] Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, 2023.
[50] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. /4314, 2023.
[51] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
14

---

[52] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457, 2018.
[53] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439, 2020.
[54] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
[55] Todor Mihaylov, Peter Clark. Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[56] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint DXXiv:209.033000 2020.
[57] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In ACI., 2022.
[58] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)311-330, 1993. URL URLpttpss: https:://wwwt http://wwwwwwww....//.. aclweb. org anthologyy/9332200000 2004
[59] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instroction-followinng llama model. attps://gtt httggttthuub...m. com
wwtp:///t//ttttttttteeeee ttt///tttsu-aaaa www.llalaaaaaaaa carttaafrrdallaccaa alpaca 2023.
[60] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfior March 2021. [ttptt //doi. 52tp://oo.281/ 5297715 If you use this software, please cite it using these metadata.
[61] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303./61 2023.
[62] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: communication-efficee SGD via gradient quantization and encoding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017 December 4-9, 2017, Long Beach, CA. USA, pages 1709 1720, 2017. URL httptttps::/ppeeeeeeeepe hhaa://c3302233366ccc33333311110 wtw.:/2003....----aaatttt Abstract html
heurips.... cc/ wc/paper//eeececccccccccceeeeepe 2017
[63] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https: //oii ortooi...rr11..1 5281528812182 zenodo. 537 1628
[64] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpttt://ppeeeeiiieeeee www. wwwppiwwwwwwwe.oweueeeeeee diw.mmiiiiiikdkk.g"k.
[65] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM, 2020. doi: 10.1 145/3394486.3446700 URL 3394439943888 3406770 3406703
https://don org/10. 114./1191///46....
15

---

[66] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite,, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/ 122000.emmll-emmm. URL attpps//accaantholeg. org. 2020. emnlp-demos.
16

---

A Common Experimental Settings
For the common experimental settings, Adam 64 optimizer and linear-decaying learning rate
for FPI6 and BF16 training. Additionally, we utilize Huggingface repository 666 for training, evaluation code and dataset.
scheduler were used. We use Deepspeed repository 665
B Experimental Settings of Section 4.1
We compare the perplexity when weights are quantized and adapted by quantization-awve aware training (QAT), LoRA with post-training quantization (PTQ), and PEQA, using the Wikitext2 dataset in Section 4.1 The LoRA configuration set to QV4. For PTQ method, we utilize OPTQ (28] [whcch is state-of-the-art low-bit weight-only PTQ method. We set the model's maximum sequence length to 1024. Batch size and epoch for all experiments are set to 128 and 15 respectively. The learning
rates for the experiments of Table 2 are displayed in Table g Learning rates for LoRA and PEQA are
shown in Appendix
Table 8: Learning rates of QAT in Table 2
Method
Bits
GPT-Neo 2.7B
GPT-J 6B
LLaMA 7B
LLaMA 13B
QAT
4
4c-5
5c-6 le-5
1c-5
3c-5
QAT
3
6e-5
2e-5
1e-5
C Experimental settings of Table
In Section 4.2 Table ssow show the scalability and task spec-fie adaptation performance of PEQA by comparing with LoRA and LoRA+OPTQ on Wikitext2 [5T] and Penn TreeBank (PTB) [58] datasets. Detailed experimental settings
follows. LoRA configuration is set to QV4. For PTQ method, we utilize OPTQ (28) which is state-of-the-art low-bit weight-only PTQ method. We set input sequence length after tokenization (block size) 1024 for under 65B models. For LLLaMA 65B, input sequence length after tokenization is set to 768 due to memory issue. Batch size and epoch for all experiments are 128 and 15 respectively. Learning rates for Table experiments shown in Table
Table 9: Learning rate of LoRA and PEQA in Table 3nn Wikitex12 and PTB datasets.
GPT-Neo 2.7B
GPT-J 6B
LLaMA
LLaMA 13B
LLaMA 30B
LLaMA 65B
Method
Bits
7B
Wikitext2
LoRA
5c-4
6c-4
1c-4
1c-4
2c-4
4c-5
PEQA (Ours)
5e-5
6e-6
6e-6
1e-5
1e-5
le-5
PEQA (Ours)
66-5
5c-5
2c-5
6c-5
3c-5
3c-5
PTB
LoRA
16
2c-3
1e-3
8c-1
5e-4
4e-4
6e-1
PEQA (Ours)
3e-4
5e-5
5e-5
5e-5
3e-5
6e-5
17

---

D The Perplexity of 3-bit and 4-bit PEQA on Wikitext2 Dataset
Figure iillustrates the results 3-bit and 4-bit PEQA's next token prediction performance on the Wikitext2 dataset. As shown in Figure B 3-bit performance of PEQA shows lower perplexity than 3-bit post-training quantized LoRA. The results from the 3-bit PEQA show that PEQA allows for continuity in model size options under DRAM usage constraints.
LIRA-OPID 6:
PSUADUan 630
Figure 3: The perplexity over model size of 3/4-bit performance of PEQA and LoRA+OPTQ
E
OPT Models Adapted with PEQA and LoRA on Wikitext2 Dataset
Table shows the perplexity of OPT models adapted with PEQA and LoRA on the Wikitext2 dataset. The perplexity gap between LoRA and PEQA becomes smaller as the model size increases.
Table 10:
The perplexity (PPL.) on Wikitext2 for OPT 1.3B 66B. this comparison, only the weights were quantized into 4-bit. A lower PPL value indicates better performance.
Method
W Bits
OPT 1.3B
OPT 2.7B
OPT 6.7B
OPT 13B
OPT 30B
OPT 66B
(QV4(QV4)
16
11.58
10.25
8.96 9.34
8.44
7.93
7.64
PEQA(Ours)
12.40
10.78
8.74
8.11
7.86
F LoRA Configuration Comparison on Wikitext2 Dataset
As shown in Table IT] the LoRA target module configuration of QV4 and QKV016 has not much effect on perplexity Wikitext2 experimental results. Table shows equal tendency mentioned in [2I]. We utilize QV4 configuration for Section 4.2 and QK configuration for Section 4.3 respectively.
Table 11: The perplexity (PPL) on Wikitext2 was compared with LoRA QV4 and QKV016. lower PPL value indicates better performance.
Method
Bits
GPT-Neo 2.7B
GPT-J 6B
L.aMA 7B
LLaMA 13B
I.aMMA 30B
I.L.MA 65B
Lo))(QV4)
16
10.63
8.50
5.53
5.06
4.06
3.82
LORA(QKVO16)
16
10.67
8.50
5.50
5.06
4.06
3.81
18

---

G Experimental Settings of Multi-scale Performance
In Section Table shows the perplexity of PEQA with grouping learnable parameters. We model maximum sequence length to 1024. Batch size and epoch for all experiments are set to 128 and respectively. Learning rates for experiments are shown in Table T2
Table 12: Learning rate for Table
Model
W Bits
g-1
g256
g128
g64
LLaMA 13B
4
lc-5
4c-5
4c-5
3e-5
3
6e-5
9e-5
9e-5
5e-5
LLaMA 7B
4
6c-6
2c-5
2c-5
1e-5
3
2c-5
6c-5
4e-5
7c-5
H Experimental Settings of Section 4.3
In Section 4.3 we use the Alpaca dataset .59 for instruction-tuning We set learning rate, epoch, and quantization group size as in Table 13 The batch size is set to 128 for all experiments in this subsection. As mentioned in Section 4.3 due to limited time and resources, we couldn't It conduct an exhaustive search over hyper- syper-parameters such as learning rate epoch. We believe that there are pyper-parameters that can perform better. For LLaMA series (LLaMA 7, 13, and 30B), we truncate the prompt the length of 2024 since their maximum sequence length is 2024 when evaluating the massive multitask lanugage understanding (MMLU) benchmark. Thus, for LLaMA2-70B we set the tokenizer length to 1024 on fine-tuning due to the resource limit. Otherwise, max length of tokenize on training. For the evaluation, we use default tokenizer setting. For every experiment in this section, the configuration of PEQA is set to 4-bit RTN quantization.
default
Table 13: The learning rate, epoch, quantization group size for experiments on Section the weights were quantized into 4-bit.
Hyper-parameter
LLaMA 7B
LLaMA 13B
I.LMAA 30B
LLaMA2 7B
LLaMA2 13B
LLaMA2 70B
Epoch
3
3
5
Learning rate
2e-5
20-5 Per-channel
5e-6
5e-6
5e-6
5e-6
Group size
Per-channel
Per-channel
256
256
256
'e.g. "hf- winttteenl-teeting/llleeee testing/llama tokee/lilaaattteieerrrrrr for LLaMA-1 series, Maneta-llma/LLlmm-2221 2- for LLaMA-2
series.
19

---

I LLaMA 7B and 13B on Natural Instruction
To evaluate the instruction-followwe
ability of instruct-tuned models, test them another instruction-followine dataset, Natural Instruction (NI) [37]. Different from the Alpaca dataset, instructions of NI were collected by humans for existing 61 NLP tasks. For simplicity, evaluation splits consisting of 12 subtasks and restrict the maximum number of instances for each task 200. At test time, the model should generate proper output for the given input with instruction for the target unseen task. As shown in Table[14] we find LLaMAs trained with PEQA show consistently better zero-shot task generalization performance (ROUGE-L) in NI for all parameter sizes compared to those from LoRA.
utilize
Table 14: Natural Instruction benchmark performance of parameter-effiicnt instruction-tunee LLa- MAs using Alpaca datasets. Zero-shot performance (ROUGE-L) is reported for the NL LoRA configuration is set to QKV016. Quantization precisions of LoRA w/ OPTQ and PEQA are set to 4-bit.
Params
LLaMA
+LoRA
+LRA w/OPTQ
+PEQA
7B
9.4
24.4
25.0
27.1
13B
8.9
31.3
29.2
34.1
J Comparison with AlphaTuning
When diving deeper into quantization scales, learnable parameters for both PEQA and AlphaTuning, it's worth noting that PEQA's adherence to uniform quantization means there's only one shared quantization scale for integer weight. Conversely, AlphaTuning' Tuming's non-uniform approach means that for a b-bit quantization, there are b individual quantization scales for each weight matrix. Despite having multiple scales, AlphaTuning only fine-tunes one, leaving the rest static. As such, the number of trainable parameters are identical and SiphhhTnnngggggg seems to offer a larger potential for a well-fitted model, but it can be easily seen that b rest static scales introduced in AlphaTuning have limited usability, and thus the method may be prone to overfitting as evident through empirical results.
In Table 15 estingly, PEQA, drawing from its methodological advantages, consistently demonstrates superior performance to AlphaTuning by at least 0.7 ppl the Wikitext2 dataset. Both AlphaTuning and PEQA used channel-wise trainable parameters. Batch size of AlphaTuning set to 32.
we conducted training GPT-Neo and OPT 1.3B using the Wikitext2 dataset. Inter-
Table 15: The perplexity (PPL) of AlphaTuuning and PEQA on Wikitext2 with OPT and GPT-Neo 1.3B. The lower PPL, the better.
Method
# Bits
OPT 1.3B
GPT-Neo 1.3B
AlphaTuning PEQA (Ours)
4
13.15
15.03
4
12.40
14.22
AlphaTuning
3
14.00
17.25
PEQA (Ours)
3
13.40
15.16
Table 16: Learning rate of AlphaTunnng in Table 15
Method
W Bits
OPT 1.3B
GPT-Neo 1.3B
AlphaTuuning
1e-1
5c-1
AlphaTuunin
le-4
1e-3
20

---

K
Choice of Updating Quantization Scales or Zero-Points
Uniform quantization can represent both asymmetric and symmetric quantizations, hence it's not always necessary to mandate the use of zero-points. This is why adopting strategy of only learning the scale factor serves as fundamental and scalable baseline. We opted for this approach to clearly establish its advantages. To determine the efficacy of learning only the scaling factors, we have incorporated additional experiments. By referring to the table below, it's evident that merely optimizing zero-points does not yield effective learning outcomes. Moreover, simultaneously optimizing both zero-points and quantization scales does not present any significant improvement in accuracy either.
Table 17: Perplexity (PPL) of PEQA the Wikitex12 dataset for LLaMA 7B and LLaMA 13B with weights quantized into 4-bit.
Method
Zero- perot-oints only
Quantization scales only (PEQA)
Both pero-points and quantization scales
LLaMA 7B
11.56
5.84
5.86
LLaMA 13B
9.83
5.30
5.34
L
Memory Peak on Training
The memory consumption is not solely dictated by the model size but is also influenced by various other factors Our approach with PEQA inherently offers memory advantages during fine-tuning by striving to minimize both the model size and the number of training parameters. To provide clear understanding these benefits, we conducted tests using a single NVIDIA A100-80GB GPU and the causal language modeling code from the HuggingFace repository Both LoRA and PEQA fine-tuned the LLaMA-7B on the Wikitext2 dataset with a batch size without gradient accumulation. Our findings indicated that while LoRA peaked memory usage of 59GB during optimization, PEQA used just 43GB. Remarkably, this disparity (16GB, 7B) escalates as the model size increases; for instance, 65B full-precision model under LoRA occupies 130GB, whereas PEQA remarkably just 33GB. Additionally, LoRA encountered Out-Of Oft-Of-Memory (OOM) issues at a batch size of 4, whereas PEQA, due to its efficiency, continued training seamlessly.
21


Reviews:
[{"review_id": "wxPbeStRqQ", "reviewer": "Reviewer_jX2S", "summary": "This paper introduces a model compression method called Parameter-Efficient Quantization-Aware Adaptation (PEQA) to tackle the size challenge of Large Language Models (LLMs) while enhancing their task-specific fine-tuning. The PEQA technique quantizes the fully-connected layers into quantization scales and integer values for initialization, and during downstream tasks, only the quantization scales are fine-tuned, leaving the integer matrix fixed. This approach significantly reduces the parameters required for gradient updates and the memory for loading pre-trained model weights, making fine-tuning more memory-efficient than other techniques such as Post Training Quantization (PTQ) and Quantization-Aware Training (QAT). However, there are several concerns regarding novelty and effectiveness of the proposed methods.", "questions": "- What could be the reason for PEQA outperforming FP fine-tuning with LoRA (+LoRA) as shown in Table 5? Additionally, the performance of applying OPTQ to LoRA actually increases in LLaMA-7B. Could this be interpreted as the LoRA Fine-Tuning approach not being optimally applied in this experiment?\n\n- From the Five-Shot results in Table 4, we can observe that the performance of PEQA on LLaMA-7B actually decreases, or its fine-tuning effect is insufficient compared to LoRA. Could this result be interpreted as the In-Context Learning ability of LLM not being well-preserved in PEQA Fine-Tuning?\n\n- How does the number of learnable parameters or model size change with a decrease in group size in Table 8? Only perplexity is provided, making it difficult to observe the trade-off together.", "limitations": "The authors did not explain the limitations of this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides an easily understandable explanation of the existing research trends and exhibits good readability.\n\n- By quantizing the pre-trained weights to sub-4bits, the memory cost for LLM fine-tuning has been significantly reduced. This enables a wider range of individuals to fine-tune LLMs according to their desired tasks efficiently. In particular, the existing similar approach, PEFT + PTQ technique, has been studied for models of size smaller than 1.3B. However, PEQA proves to be effective even for models as large as 65B and reduces memory costs.\n\n- Through various experimental results, the paper demonstrates the performance of the PEQA approach in fine-tuning and its application of quantization compared with PTQ and QAT results. This effectively shows how well the method works for the purpose of LLM fine-tuning.", "weaknesses": "- This paper's significant contribution, fine-tuning only the quantization scales (termed PEQA), seems similar to AlphaTuning [22]. The paper does not thoroughly discuss the limitations of AlphaTuning, only noting its lack of evaluation on large LLMs. \n\n- The baseline settings in the evaluation setup are unusual. For instance, the authors claim PEQA matches or exceeds QAT's performance. However, QAT's accuracy appears lower than state-of-the-art methodologies: e.g., LLM-QAT[a] demonstrated that 4-bit weight quantization almost reproduces the full-precision accuracy, but QAT in PEQA (Table2, Table3) shows noticeable degradation (LoRA 10.63 vs QAT 11.07). Such understated baselines could potentially mislead readers regarding PEQA's true performance.\n\n[a] LLM-QAT: Data-Free Quantization Aware Training for Large Language Models\n\n- The authors state that PEQA maintains generalization capabilities, yet the results are inconsistent. Table 4 indicates PEQA's zero-shot accuracy matches LoRA, but this improvement disappears in five-shot accuracy. Therefore, it is not clear if PEQA really preserves generalization capability or not.\n\n- There is little understanding of why fine-tuning only the quantization scales is sufficient. Therefore, it is hard to be convinced that the proposed method would work as expected. For example, Table 5 shows that PEQA even noticeably outperforms LoRA (without quantization) LoRA for zero-shot MMLU (and even LoRA+OPTQ outperforms LoRA for LaMMA-7B), which counters the accuracy trends of \"LoRA > PEQA > LoRA+OPTQ\" consistently shown in all the other tables. It would be desirable to provide insights into such unexpected evaluation results.\n\n- The problem setup in Section 3.1 doesn't align with PEQA's main objectives. The memory-bound issue during text generation inference discussed in this section seems less relevant to PEQA's fine-tuning memory efficiency benefits. Since forward computation isn't performed in the generation style during training, and text generation's memory-bound issue can be resolved by quantization (e.g., LoRA + OPTQ), including this issue in the problem setup may be misleading.\n\n- Comparing 3-bit weight quantization with OPTQ is valuable, but considering OPTQ's known limitations at this level of granularity, a fine-grained quantization comparison might be more convincing. Furthermore, a comparison with recent state-of-the-art weight quantization methodologies, such as LoRA + AWQ, could provide a clearer understanding of PEQA's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[**Weakness1**] When diving deeper into the quantization scale specifics which are learnable parameters of both methods, it's worth noting that PEQA's adherence to uniform quantization means there's only one shared quantization scale for integer weight. Conversely, AlphaTuning's non-uniform approach means that for a $b$-bit quantization, there are $b$ individual quantization scales for each weight matrix. Despite having multiple scales, AlphaTuning only fine-tunes one, leaving the rest static. As such, the number of trainable parameters are identical and AlphaTuning seems to offer a larger potential for a well-fitted model, but it can be easily seen that $b-1$ rest static scales introduced in AlphaTuning have limited utilizability, and thus the method may be prone to overfitting as evident through empirical results. In our appendices, we've showcased a direct performance comparison between the two using a 1.3B model. PEQA, drawing from its methodological advantages, consistently demonstrates a performance that's superior by at least $0.7$ ppl. We also intend to include more detailed studies to address the fundamental differences between previous work and ours.\n\n[**Weakness2**] Although the performance of PEQA can be similar to or better than that of QAT in Table 2, the QAT approach presented in Table 2 is one of the simplest QAT methods that just rounds weights like Eq. 3 and updates $W_0$ and $s_0$ by using the straight-through estimator. As a result, unlike LLM-QAT, the state-of-the-art QAT technique for LLMs (that has been introduced after the NIPs submission deadline), such a rudimentary QAT method in Table 2 can cause performance degradation compared to its full-precision counterpart. As the reviewer mentioned, such a naive QAT approach seems to be an underestimated baseline that might mislead readers to understand that PEQA can either match or surpass QAT methods. However, note that our intention in juxtaposing PEQA with this basic QAT approach in Table 2 was to justify keeping the integer matrix in a frozen state rather than the superiority of PEQA compared to QAT methods. To establish a reliable baseline, we provide full-precision PEFT baseline results, LoRA (higher upper bound), in Table 3. \n\n[**Weakness3 & Question2**] In the case of zero-shot, PEQA's accuracy matches LoRA, while the improvement becomes less pronounced when transitioning to five-shot accuracy. The results in Table 4 & 5 bring forth an interesting phenomenon often termed as the \"Alignment tax\" [1,2,3]. The concept denotes the potential compromise in performance on traditional NLP tasks as a trade-off for enhanced instruction-following or alignment capabilities. Specifically, as the instruction-following performance heightens, as reflected in Table 5, there's a conceivable penalty that can be observed in tasks like common sense reasoning. This effect, the Alignment tax, is especially more noticeable in smaller models. Piecing everything together, our overarching claim is to emphasize that even after fine-tuning large language models (LLMs) in a compressed state via the PEQA method, the performance in logical reasoning and in-context learning stands its ground.   \n[1] Training language models to follow instructions with human feedback.  \n[2] A general language assistant as a laboratory for alignment.  \n[3] Training a helpful and harmless assistant with reinforcement learning from human feedback.  \n\n[**Weakness4 & Question1**] (Please refer to the reply of Weakness 2) We recognize the significance of theoretical aspects of PEQA. We plan to show theoretically why updating only the quantization scales is sufficient as a future work.  \nGiven that LoRA outperforms LoRA+OPTQ for LLaMA-13B in Table 5, the experimental result where LoRA+OPTQ outperforms LoRA for LLaMA-7B can be regarded as an exceptional occurrence. Yet, PEQA performs better than LoRA for both LLaMA-7B and LLaMA-13B in Table 5, which might be because the number of trainable parameters in PEQA would be more appropriate than that in LoRA (QKVO16) for acquiring the instruction-following ability when fine-tuning a model with an instruction-following dataset (e.g., Alpaca). Similar to the observation that LoRA can be better than full fine-tuning despite possessing fewer learnable parameters, even though the number of learnable parameters in LoRA (QKVO16) is six times more than that in PEQA as seen in Table 7, PEQA would be better suited for developing the instruction-following ability than LoRA. \n\n[**Weakness5**] We kindly remind you that the core objective of PEQA is twofold: achieving memory efficiency during fine-tuning and benefiting from acceleration and quick-task switching during the deployment phase for text generation inference. By employing quantization during fine-tuning, PEQA ensures that weights are preserved in their integer format post fine-tuning. This approach not only tackles the memory constraints during fine-tuning but also addresses the memory-bound issue during text generation inference. As PTQ is applied after the fine-tuning process, PEFT+PTQ doesn't offer the advantages of memory-efficient fine-tuning.\n\n[**Weakness6**] We have performed an additional experiment with LoRA+OPTQ to better elaborate the level of granularity. As pointed out by the reviewer, particularly with LLaMA 13B, we observe that the performance of LoRA+OPTQ improves as the process becomes more fine-grained. However, we continue to see that PEQA consistently demonstrates superior performance when compared to LoRA+OPTQ in Table A(in PDF). \n\n[**Question3**] Parameters in the order of millions have a negligible impact on a model size measured in GBs. When quantizing LLaMA-7B with a group size of 256, a single linear layer has approximately 16 times more learnable scales. When this is scaled linearly, the channel-wise learnable parameter count, which initially was 1.36M, increases by 16 times to reach 21.76M. Yet, this only results in an additional 40MB in memory usage. "}, {"review_id": "u2kSTBa8zC", "reviewer": "Reviewer_rGqm", "summary": "The paper presents a novel approach named Parameter-Efficient and Quantization-aware Adaptation (PEQA) to address the challenges of efficiently fine-tuning and deploying large language models (LLMs). The paper demonstrates PEQA's effectiveness and scalability through extensive experiments, comparing it with competitive baselines across a range of tasks from natural language understanding to generation benchmarks.", "questions": "* The methodology section of the paper leaves some key details unaddressed. Specifically, it's unclear how the gradient of the scaling factors is computed in the proposed model. The paper PACT by Choi, Jungwook, et al. [1] presents two different ways to compute the gradient for the scaling factor/clamp_range for the quantization function. It would be beneficial for the authors to clarify which method was adopted in their work.\n\n[1] Choi, Jungwook, et al. \\Pact: Parameterized clipping activation for quantized neural networks.\\ arXiv preprint arXiv:1805.06085 (2018).", "limitations": "see above.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper centers on the topic of efficient model compression and fine-tuning, framing its approach through a unified method referred to as PEQA.\n\n* The authors' proposed method offers benefits, most noticeably in substantially reducing the memory usage associated with optimizer state saving and the final model size.\n\n* The authors have undertaken a comprehensive evaluation setup which encompasses several benchmark tasks and a variety of model sizes.", "weaknesses": "* The overall presentation of the paper could be improved for clarity. Specifically, the meaning of statements such as \"Notice that s0 and z0 have nothing to do with a choice of downstream task\" on Line 148, and the term \"integer quantization indices\" on Line 149, are not clear.\n\n* The calculation of memory usage during fine-tuning is not sufficiently explained. While it's understood that the proposed method lowers optimizer state storage by only updating scaling factors, the paper does not address memory usage during gradient calculation, a crucial aspect that often dictates peak memory usage during optimization.\n\n* It's unclear why the choice of only updating scaling factors is advantageous. The paper only compares the proposed method with LoRA, limiting the assessment of its effectiveness. For instance, the BitFit method (Ben Zaken et al., 2022) demonstrates that fine-tuning the LLM can be achieved by only updating biases. Moreover, the paper does not discuss the potential outcomes of optimizing zero-points only or optimizing both zero-points and the scaling factor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[**Weakness1**] As s0 and z0 are quantization parameters for pre-trained weights W0, our intention was to imply that both s0 and z0 are not related to any downstream task at all. The term “integer quantization index” is also used in [1], which means the rounding of W0/s0, so we refer to  $\\overline W_0$  as the integer quantization indices of W0. To enhance clarity, we will revise these expressions in our manuscript.  \n[1] https://en.wikipedia.org/wiki/Quantization_(signal_processing)\n\n[**Weakness2**] We appreciate the reviewer's insightful comment regarding the calculation of memory usage during optimization. To clarify, the computation of the gradient for the quantization scale in PEQA necessitates dequantizing the quantized weight to a 16-bit half-precision format. Despite this process, it should be noted that weights from other layers retain their low-bit quantized state. This results in a lesser peak memory usage during optimization for PEQA compared to LoRA. Our proposed model achieves memory efficiency not only during the inference phase but also throughout the optimization process, by minimizing memory consumption related to the model size. We acknowledge the importance of detailing this aspect in our work and will ensure to elaborate on this point in our revision. Thank you for your valuable feedback.   \nWe also kindly point out that memory usage during fine-tuning is influenced not only by the model size but also by other factors[2]. Hence, in our endeavor to reduce both the model size and the number of training parameters, we adopted PEQA, which provides memory benefits during fine-tuning. To quantify the extent of these memory benefits in actual training scenarios, we measured the memory peaks. Our experimental setup utilized a single NVIDIA A100-80GB GPU and employed the causal language modeling code from the HuggingFace repository [3]. Both LoRA and PEQA fine-tuned the LLaMA-7B on Wikitext2 dataset with a batch size of 2 without gradient accumulation. The results of our tests revealed that LoRA displayed a peak memory usage of 59GB during optimization, whereas PEQA consumed only 43GB. Furthermore, while LoRA experienced Out-Of-Memory (OOM) issues with a batch size of 4, PEQA, owing to its reduced memory usage, was able to proceed with training at this batch size. Accepting the feedback of the reviewer, we will incorporate these findings into our revised manuscript to facilitate a clearer understanding for our readers. Thank you for your valuable feedback.\n\n[2] https://huggingface.co/docs/transformers/perf_train_gpu_one#anatomy-of-models-memory.   \n[3] https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm_no_trainer.py   \n\n[**Weakness3**] Uniform quantization can represent both asymmetric and symmetric quantizations, hence it's not always necessary to mandate the use of a zero-point. This is why adopting a strategy of only learning the scale factor serves as a fundamental and scalable baseline. We opted for this approach to clearly establish its advantages. To determine the efficacy of learning only the scaling factors, we've incorporated additional experiments. By referring to the added table, it's evident that merely optimizing zero-points does not yield effective learning outcomes. Moreover, simultaneously optimizing both zero-points and scaling factors doesn't present any significant improvement in accuracy either. While parameter-efficient adaptation methods, such as BitFit, offer a memory-efficient fine-tuning and preserving quick task-switching abilities, they do not outperform LoRA in terms of accuracy[4]. Our proposed method, PEQA, which updates the uniform quantization scale, was presented as the most straightforward and effective approach that is not only memory-efficient during fine-tuning but also facilitates accelerated inference during deployment, maintains quick task-switching abilities, and delivers acceptable accuracy. We acknowledge this feedback and ensure that the revised manuscript reflects these points accordingly. Given the time constraints of the rebuttal period, our revised manuscript will incorporate additional BitFit experiments to position PEQA alongside other methods as recommended by the reviewers.    \n\n| | Zero-point only | Scale-factor only (PEQA) | Both scale-factor and zero-point |\n|:----:|:----:|:----:|:----:|\n| 7B | $11.56$ | $5.84$ | $5.86$ |\n| 13B | $9.83$ | $5.30$ | $5.34$ |. \n\n[4] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" The Tenth International Conference on Learning Representations (ICLR), Virtual Event, April 25-29, 2022.  \n\n\n[**Question1** ]   \nLet the quantization scale $\\boldsymbol{s}\\_0 = \\begin{bmatrix} s_{0, (1)} \\newline s_{0, (2)} \\newline \\vdots \\newline s_{0, (n)} \\end{bmatrix}$, \nthe integer matrix $$ \\overline{\\boldsymbol{W}}\\_0 =\n\\begin{bmatrix}\n\\overline{W}\\_{(1, 1)} , \\overline{W}\\_{(1, 2)} , \\cdots , \\overline{W}\\_{(1, m)} \\newline\n\\vdots \\newline\n\\overline{W}\\_{(n, 1)} , \\overline{W}\\_{(n, 2)} , \\cdots , \\overline{W}\\_{(n, m)}\n\\end{bmatrix},$$  \nand the quantized weight matrix $$ \\widehat{\\boldsymbol{W}} =\n\\begin{bmatrix}\n\\widehat{W}\\_{(1, 1)} , \\widehat{W}\\_{(1, 2)} , \\cdots , \\widehat{W}\\_{(1, m)} \\newline\n\\vdots \\newline\n\\widehat{W}\\_{(n, 1)} , \\widehat{W}\\_{(n, 2)} , \\cdots , \\widehat{W}\\_{(n, m)}\n\\end{bmatrix}.$$\nThen, the gradient of $\\mathcal{L}$ with respect to the scaling factor $\\boldsymbol{s}\\_0$ is given by \n${\\partial \\mathcal{L} \\over \\partial \\boldsymbol{s}\\_0} = \\begin{bmatrix} {\\partial \\mathcal{L} \\over \\partial s_{0, (1)}}  \\newline {\\partial \\mathcal{L} \\over \\partial s_{0, (2)}}  \\newline \\vdots \\newline {\\partial \\mathcal{L} \\over \\partial s_{0, (n)}}  \\end{bmatrix}$ where\n${\\partial \\mathcal{L} \\over \\partial s_{0, (i)}} = \\Sigma_{j=1}^m \\overline{W}\\_{(i, j)} {\\partial \\mathcal{L} \\over \\partial \\widehat{W}\\_{(i, j)}}$ for $1 \\le i \\le n$.\n"}, {"review_id": "tmaIIOdk6n", "reviewer": "Reviewer_qn8N", "summary": "This paper introduces a new framework, PEQA, for efficiently tuning Large Language Models (LLMs). PEQA uniquely fine-tunes the scale parameters during the instructing tuning phase while keeping the backbone parameters frozen. This approach allows PEQA models to maintain advantages during both the training and inference phases. The primary experiments are performed on the Alpaca datasets. The paper also compares PEQA's performance on various datasets with other efficient tuning methods, such as Lora.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is strongly motivated, emphasizing the importance of memory efficiency in tuning methods.\n\nThe authors highlight the role of quantization methods in reducing memory usage beyond what is achievable with parameter efficiency methods alone [1]. \n\nThe paper is well-written and easy to follow.\n\n[1]. https://github.com/huggingface/peft", "weaknesses": "Table 1 could be improved by emphasizing that other parameter-efficient tuning methods can also apply post-training quantization methods, such as GPTQ and AWQ.\n\nThe baseline should include the post-training quantization methods applied before or after fine-tuning (e.g., Lora and Adapter), and report on the memory usage during inference and training. This is particularly relevant as the HuggingFace PEFT library has incorporated Int8 quantization.\n\nReference:\n\n[1] QLoRA: Efficient Fine-tuning of Quantized LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[**Weakness1**] We appreciate the reviewer's insightful suggestion. In response, we have updated Table 1 to include both PEFT+PTQ and PTQ+PEFT as per your recommendation. We understand the importance of showcasing that other parameter-efficient tuning methods can also apply post-training quantization methods like GPTQ and AWQ. This modification will be incorporated in our revised manuscript. Thank you for your valuable feedback.   \n\n| | Fine-Tuning ||| Deployment ||\n|:----:|:----:|:----:|:----:|:----:|:----:|\n| Method | DRAM | Tuning Speed | DRAM | Inference Speed | Task-Switching |\n| Full Fine-Tuning | $457$GB | Slow | $131$GB | Slow | Slow |\n| PEFT | $131$GB | Fast | $131$GB | Slow | Fast |\n| PEFT+PTQ | $131$GB | Fast | $33$GB | Fast | Slow |\n| PTQ+PEFT | $33$GB | Fast | $33$GB | Slow | Fast |\n| $\\textbf{Quantization-Aware PEFT (ours)}$ | $\\textbf{33GB}$ | $\\textbf{Fast}$ | $\\textbf{33GB}$ | $\\textbf{Fast}$ | $\\textbf{Fast}$ |    \n\n\n\n[**Weakness2**] Post-training quantization (PTQ) after fine-tuning has been addressed through LoRA + OPTQ in our work. As for PTQ applied before fine-tuning, it could indeed be conducted as you suggested with methodologies such as QLoRA. As QLoRA is a concurrent work with PEQA, an apple-to-apple comparison between QLoRA and PEQA has not yet been completed due to time constraint of the rebuttal period. However, as can be seen from the updated Table 1, approaches like QLoRA do not ㅈaccelerate inference in the deployment phase.  \nQLoRA could indeed be a good method if the requirements solely focus on memory-efficient fine-tuning and quick task-switching of the Quantized PLM. However, if maintaining the quantized PLM form and deploying services through memory-efficient fine-tuning are required, with a particular emphasis on latency during deployment, we suggest that PEQA may be the most suitable choice.  \nAs for the HuggingFace PEFT library incorporating Int8 quantization, we will study and consider this implementation for our future work and aim to compare it to our method for a more comprehensive evaluation. Thank you for your valuable suggestions.   \n"}, {"review_id": "9ROuuP1dwq", "reviewer": "Reviewer_9H2h", "summary": "This paper presents a method PEQA that combines quantization and parameter-efficient fine-tuning. It takes both advantages, including updating only a tiny fraction of model weights and saving the memory by quantization. The model weights will be decomposed into a matrix of low-bit integers and a scalar vector. During the fine-tuning, only the scalar vector will be updated. The paper has done comprehensive experiments to measure its effectiveness. The proposed method outperforms PTQ and has an acceptable degradation compared to LoRA, but which significantly saved the memory.", "questions": "1. Table 2: Why does PEQA have an even lower perplexity in some cases than QAT?\n2. How does PEQA compare to the direct combination of LoRA and quantization (e.x. compress to 4-bit, then LoRA on the 4-bit matrices)?\n3. What is the intuition behind the decomposition used in PEQA?\n4. Table 4: Why PEQA degrades the performance of Five-Shot?", "limitations": "1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written and easy-to-follow.\n2. The idea and the method are clean.\n3. The evaluation is comprehensive.", "weaknesses": "1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible.\n2. The paper combines two existing techniques rather than inventing a new technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[**Weakness1**] As the reviewer pointed out, when comparing PEQA with LoRA in terms of the number of trainable parameters (e.g., LLaMA-13B+LoRA vs. LLaMA-13B+PEQA), the degradation of the performance of PEQA relative to LoRA seems to be non-negligible. However, when comparing PEQA with LoRA in terms of the model size (e.g., LLaMA-13B+LoRA vs. LLaMA-30B+PEQA) as shown in Figure 2 (b), PEQA outperforms LoRA in all cases. We believe that it is more fair to compare PEQA with LoRA in terms of the model size than the number of learnable parameters due to the fact that the checkpoint size for trainable parameters is negligible relative to the model size, which makes the overall memory required for deployment totally dominated by the model size.   \n\n[**Weakness2**] While the comment is ambiguous as to which two techniques the reviewer thinks our method is based on, we would ike to reiterate that PEQA is a straightforward yet effective way to achieve PEFT and quantization within the same framework.  .PEQA inherits (1) the reduction in the number of trainable parameters and task-switching capability from PEFT and (2) the reduction in the model size and the acceleration of text generation inference from quantization, which makes PEQA more practical and effective than both techniques. Furthermore,, the theoretics and the underlying mechanism that allow PEQA to ourperform previous SOTA  is non-trivial, and we intend to address them in the camera-ready version, if possible.\n\n[**Question1**] In Table 2, as highlighted by the reviewer, PEQA exhibits lower perplexity in certain scenarios compared to QAT. This phenomenon is likely influenced by the straight-through estimator (STE) utilized in the QAT approach we presented. The QAT method in Table 2 is a rudimentary one that simply rounds weights as per Eq. 3 and updates  $W_0$ and $s_0$ using the STE. Notably, the use of STE can sometimes lead to compromised accuracy or increased perplexity due to its approximated gradients possibly failing to converge appropriately. The naive QAT approach we used can experience performance degradation compared to its full-precision counterpart. Our primary intention in comparing PEQA with this basic QAT in Table 2 was to emphasize the rationale for keeping the integer matrix static, rather than to suggest PEQA's superiority over all QAT methods. We will ensure to clarify this perspective in our revisions to avoid any misconceptions.  \n\n[**Question2**] The direct combination of LoRA and quantization, such as in the QLoRA approach, is indeed a notable method. As the reviewer highlighted, LoRA can be directly applied to quantized pre-trained models like in the QLoRA technique. However, there are some distinctions and nuances worth mentioning. While QLoRA can indeed achieve accuracy levels comparable to LoRA as detailed in [1], there's an important caveat: it doesn't benefit from the inference acceleration. This is because the weights can't be represented in a quantized format when merging quantized pre-trained weights with LoRA for deployment. Furthermore, if there's a scenario where you wish to fine-tune a deployed quantized PLM for some reason, our methodology might also present itself as an optimal choice. Specifically, if the primary goal is memory-efficient fine-tuning and rapid task-switching of the Quantized PLM, QLoRA could be a viable approach. However, if one's aim is to maintain the quantized form of the PLM and deploy services via memory-efficient fine-tuning with a focus on latency during deployment, PEQA emerges as a potential best-fit.\n\n[**Question3**] The intuition behind the decomposition used in PEQA stems from a desire to inherit from the advantages of both PEFT and quantization. As quantized parameters are expressed as the Hadamard product of full-precision scales and an integer matrix, we freeze the integer matrix while allowing the full-precision scales to be adaptable to a downstream task.\n\n[**Question4**] As highlighted by the reviewer, Table 4 reveals that the five-shot performance of LLaMA-7B + PEQA is inferior to that of LLaMA-7B alone. Conversely, for LLaMA-13B and LLaMA-30B, PEQA yields an improvement in the five-shot performance. In view of these findings, the five-shot performance drop of LLaMA-7B + PEQA can be regarded as a temporary variation, and it would be therefore premature to conclude that PEQA diminishes performance in the five-shot setting. \n"}]

Author rebuttal:
Dear Reviewers,

We wish to express our sincere gratitude for your diligent review and insightful feedback on our manuscript.   
Your comments have greatly enriched our understanding and enabled us to identify areas where further clarification and improvement were needed.

We have uploaded this summary table in a one-page PDF format in the review system.

Thank you once again for your time and contributions.   
We look forward to any further comments or suggestions you may have, and we remain at your disposal for any additional information or clarification.

With kind regards.

Meta-review: This paper proposes a novel quantization-aware PEFT method, Parameter-Efficient and Quantization-aware Adaptation (PEQA), to address the challenges of efficiently fine-tuning and deploying Large Language Models (LLMs). Extensive experiments are conducted on NLU and NLG tasks and various model sizes, compared with with several SOTA baselines. It shows PEQA can effectively save more memory cost than existing PEFT methods.
Decision: Accept (poster)