Title: Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

Parameter-Efficient Fine-Tuning

Neural Network Quantization
Abstract: Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern.  
Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase.
To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) – a simple yet effective method that combines the advantages of PEFT with quantized LLMs. 
By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage.
We employ PEQA-tuning for task-specific adaptation on LLMs with up to $65$ billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. 
Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.

Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization

Abstract

Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameterefficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern. Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase. To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) - a simple yet effective method that combines the advantages of PEFT with quantized LLMs. By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage. We employ PEQA-tuning for task-specific adaptation on LLMs with up to 65 billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.

1.2 0.2

… 3.1 2.3

0.9 1.0

… 0.3 2.5

1.2 0.2

… 3.1 2.3

0.9 1.0

… 0.3 2.5

1.2 0.2

… 3.1 2.3

0.9 1.0

… 0.3 2.5

1.2 0.2

… 3.1 2.3

0.9 1.0

… 0.3 2.5

Full-Precision

1

0

… 3

2

1

1

… 0

4

1.1

5.1

0.2

0.7

𝑾

𝟎

𝒔

𝟎

!

…

Step 1

Decomposition

(Quantization)

Fine-tuning

for task T

𝒔

𝑩

𝒔

𝑪

𝒔

𝑫

𝒔

𝑬

Parameter-efficient Fine-tuning

Deployment

Frozen

Fine-tuned

𝑾

𝟎

Sub-4-bit Int

1

0

… 3

2

1

1

… 0

4

1.2

4.3

1.2

0.4

𝑾

𝟎

𝒔

𝑻

!

…

Sub-4-bit Int

1

0

… 3

2

1

1

… 0

4

1.2

4.3

1.2

0.4

𝑾

𝟎

!

…

Sub-4-bit Int

𝒔

𝑨

Task

Switch

Step 2

1 Introduction

Large language models (LLMs) such as PaLM, LLaMA, and the GPT-series [1-7] have demonstrated unprecedented levels of task-generalization ability in various applications, including dialogue systems, question answering, summarization, and translation [8, 9]. While they can follow instructions and learn to solve tasks via in-context task descriptions or few-shot examples [10], fine-tuning allows LLMs to align their behavior with desirable traits, such as following instructions more precisely [11] or adhering to certain principles [12]. Additionally, fine-tuning can improve the scaling curve by exposing the model to large collections of task-specific instruction datasets, leading to significant performance enhancements in various unseen downstream tasks [13-17]. However, the immense computational cost of fully fine-tuning large-scale models presents challenges for researchers and developers, especially given that LLMs have billions or even trillions of parameters [18].

In response, several parameter-efficient fine-tuning (PEFT) methods have been introduced [19-21], which only update a small number of parameters compared to the pre-trained weights of LLMs. PEFT notably reduces the number of learnable parameters, making the fine-tuning of pre-trained LLMs viable by ensuring that the optimizer states' memory usage becomes negligible. These strategies lead to decreased memory usage during training and more efficient storage and seamless transitions of task-specifically fine-tuned parameters during deployment. Nonetheless, LLMs as a whole still demand significant memory, and further reductions are attainable through model compression. As outlined in Hu et al. [21], for instance, LoRA can cut the memory usage during the fine-tuning of GPT-3 175B from 1.2TB to 350GB. However, the model still requires approximately 350GB of memory for parameters in half-precision floating-point format.

Quantization is a favorable method for both compressing and accelerating neural networks by discretizing parameters into low-bit integers while maintaining a shared high-precision scale within each parameter group (e.g., channel or layer). However, during training phases, quantization-aware training (QAT) [22-25] mandates updates for all parameters, rendering it not parameter-efficient. Since post-training quantization (PTQ) [26-29] is executed after training, most existing quantization schemes primarily target the deployment phases. Although PTQ can be integrated with PEFT, when PTQ follows PEFT, the model remains intact during fine-tuning, not decreasing the memory usage. Conversely, if PTQ precedes PEFT, while there's a reduction in memory usage during fine-tuning, no inference acceleration can be achieved due to the PEFT parameters during deployment.

To bridge the gap between PEFT and quantization, we introduce the Parameter-Efficient and Quantization-aware Adaptation (PEQA), a simple yet effective quantization-aware PEFT method. As illustrated in Figure 1, PEQA encompasses two steps: (a) Decomposition (Quantization) where the parameter matrix of each fully-connected layer is decomposed into a matrix of low-bit integers and quantization scales; and (b) Fine-tuning wherein, for each downstream task, the quantization scale is fine-tuned while the integer matrix remains unchanged. For the quantized LLMs, merely updating the quantization scale leverages the advantages of PEQA. As a result, PEQA maintains the merits of PEFT, such as fewer trainable parameters, along with efficient storage and swift switching of task-specific parameters. Concurrently, it provides the benefits of quantization, including reduced

DRAM usage during both training and deployment, and inference acceleration due to fewer memory accesses at deployment.

Through this, we highlight the following:

We introduce PEQA, a method that fine-tunes only the quantization scales of quantized LLMs, keeping the integer matrix frozen. It bridges the gap between PEFT and quantization, offering advantages such as reduced memory consumption during both training and deployment phases, seamless task transitions, and faster inference.

To empirically validate the approach of solely fine-tuning the quantization scale while freezing the integer matrix, we compare the perplexity of LLMs fine-tuned with QAT, PEFT (+PTQ), and PEQA. The results indicate that PEQA delivers competitive performance in comparison to QAT and PEFT+PTQ, even at sub4 -bit precision.

To assess the scalability and comprehension performance of PEQA, we apply PEQA to taskspecific adaptation and instruction-tuning. Despite the reduction in model size by a factor of 4 to 5 , PEQA demonstrates competitive performance up to a 65 B LLM when compared to full-precision baselines. The results suggest that even when LLMs are quantized into low-bit precision, the overall comprehension capability of quantized LLMs can be effectively restored to their original performance using PEQA.

2 Related Work

Large Language Models and Alignment Learning. Although LLMs have demonstrated great generalization capabilities through their sheer scales [2, 6, 30] and the emergent mechanism known as in-context learning [1], they still require significant alignment to follow natural language instructions [13], adhere to ethical guidelines or steer towards harmlessness [12], utilize external tools [31], and to be grounded in knowledge to generate truthful answers [16, 32]. In particular, instruction-tuning has been pivotal in enabling LLMs to generalize instruction-following abilities, enabling them to solve seemingly any NLP task with only the description of the task in natural text [13, 33, 34], allowing the models to be accessed in an interactive manner.

Parameter-Efficient Fine-Tuning. Fine-tuning leverages the generalization capabilities elicited from the general pretraining to specialize in specific domains and tasks [35, 36] or align the LLM with target behaviors [13]. However, updating parameters in LLMs comes with a high computation cost and minimal compute environment required for gradient computation. As the hyper-scale era makes fine-tuning for LLMs prohibitively expensive, both efficient and effective alternatives to fine-tuning have received considerable attention. Specifically, inspired by the sensitivity of LLMs to prompts [37], a line of works has proposed introducing trainable prompt embeddings prepended to the input text while freezing the original LLM parameters [19, 38, 39]. As another approach, adapter modules [20] introduce task-specific parameters, which are inserted between the pre-existing layers of the model Extending on this adapter-based approach, LoRA [21] employs the concept of low-rank bottleneck modules while demonstrating comparable performance to full fine-tuning. Subsequent works have unified the various versions and diverging approaches to PEFT [40, 41] by formulating them in a single mathematical framework. These parameter-efficient methods have shown comparable performance to full model fine-tuning, presenting a cost-effective and efficient avenue for tailoring LLMs to specific tasks.

However, even with the adoption of PEFT, the inherent model size of the LLM remains a challenge to handle. One immediate solution is to apply post-training quantization (PTQ), but its interaction with task-specific parameters is still an area of active research.There have been attempts to integrate PEFT and neural network quantization, including methods like Quadapter [42] and AlphaTuning [43]. Yet, these methods have primarily been explored in smaller models of 1.3B or fewer parameters. Appendix J delineates the distinctions between our method and AlphaTuning.

Neural Network Quantization. Neural network quantization consists largely of quantizationaware training (QAT) and PTQ. QAT methods [22-25] basically train not only quantization scales but all the parameters of a full-precision neural network to narrow the performance gap between the full-precision model and its quantized counterpart. Unfortunately, since QAT involves training all the weights of a full-precision network, it is not feasible to apply QAT to LLMs. To quantize LLMs, PTQ

(a)

0

25

50

75

100

125

150

3.5

4.0

4.5

5.0

5.5

6.0

Perplexity

2.1M

3.28M

6.39M

10.49M

1.36M

2.13M

4.15M

6.8M

LoRA(FP16)

PEQA(4-bit)

(b)

326GB

Parameter-

Optimizer

Efficient

States

Adaptation

Size

Parameter-

Efficient and

52MB

Quant-aware

Adaptation

Model

131GB

131GB

Size

32MB

32MB

33GB

25GB

Full

LoRA

PEQA

PEQA

Fine-Tuning

(QV4)

techniques tailored to LLMs [26-29, 44, 45] have been presented. Although such PTQ approaches do not require learning all the parameters of an LLM at all, as PTQ occurs after training/fine-tuning LLMs, PTQ cannot allow for compressing the model size during training/fine-tuning LLMs. To reduce the model size even during training/fine-tuning LLMs, researchers have recently focused on combining PEFT with quantization.

3 Methodology

3.1 Problem Setup

Memory Demands in Fine-tuning. Given the significant computational demands associated with fully fine-tuning large language models, parameter-efficient fine-tuning (PEFT) methods have been introduced [19-21, 46]. One of the primary goals of PEFT methods is to reduce memory usage of the optimizer state during training, specifically by reducing the number of learnable parameters. While existing PEFT techniques do decrease memory consumption of the optimizer state and also narrow the accuracy gap between full fine-tuning and PEFT, the pre-trained weights of large language models still demand substantial memory space. When applying LoRA [21] to LLaMA-65B, for instance, even though storing optimizer states for trainable parameters consumes only 52 MB(only query and value matrices are adapted with a LoRA rank of 4 ), the model size still occupies a huge portion of DRAM usage due to the frozen FP16 weights of a pre-trained model. To make PEFT more efficient, reducing the model size is an indispensable requisite. Given that LLMs mostly consist of fully-connected layers, compressing the weights of fully-connected layers is a key factor in compressing the model size and thus leading to more efficient PEFT.

Inference Latency of Large Language Models. During text generation inference, an autoregressive LLM generates tokens sequentially. A significant portion of the inference latency arises from matrix-vector multiplications, as opposed to matrix-matrix multiplications. Given that the batch size during inference is typically small [27, 28], matrix-vector multiplications tend to be memory-bound. Specifically, accessing global memory, such as DRAM, is expensive on contemporary high-end GPUs. Thus, the number of weights loaded into registers profoundly impacts the speed of multiplication between a matrix and a vector. To decrease the number of weights (subsequently increasing the weights loaded into registers), quantization is a widely researched method for both compressing and speeding up neural networks [29, 44, 47]. While quantization-aware training (QAT) imposes a significant load on both computation and memory, post-training quantization (PTQ) is often viewed as a fallback strategy among traditional quantization techniques to enhance the generation latency of LLMs. To both accelerate LLM inference and retain all advantages of PEFT, an innovative alternative approach should be pursued.

3.2 Parameter-Efficient and Quantization-aware Adaptation (PEQA)

Quantization reduces bit-precision for inference acceleration, less storage and increasing throughput. INT 8 quantization, which lower the bit-precision for both activations and weights, utilize dedicated engine to effectively accelerate arithmetic computation [48]. This is effective for large batches where computing speed matters but less so for smaller batches constrained by memory. To tackle this memory issue, weight-only quantization keeps high precision for activations (e.g., FP16) but compresses weights to 4 -bit or less, targeting memory I/O enhancement in modern GPUs [28, 47]. For simplicity, we mainly focus on low-bit weight-only quantization in a linear asymmetric per-channel context in this paper.

For pre-trained weights of a fully-connected layer W 0 ∈ R n × m , while PEQA can be applied to quantized LLMs, we first quantize W 0 . In other words, for a given bit-width b , quantized pre-trained weights ̂ W 0 can be written as

where A · B , ⌊·⌉ , and clamp ( · , a, b ) indicate the element-wise product of A and B , the rounding function, and the clamping function into the range [ a, b ] , respectively, while per-channel scales and zero-points (namely, s 0 , z 0 ∈ R n × 1 ) are initialized to minimize ∥ W 0 -̂ W 0 ∥ 2 F . Notice that s 0 and z 0 are not related to any downstream task. Here, we freeze W 0 = clamp (⌊ W 0 s 0 ⌉ + z 0 , 0 , 2 b -1 ) -z 0 ) , which is the integer quantization indices of W 0 , for every full-connected layer in a pre-trained LLM. And then we fine-tune only s 0 (residing outside the clamp function in Eq. 1) while sharing W 0 across all downstream tasks. Consequently, quantized pre-trained weights ̂ W 0 are adapted to a downstream task as follows:

where ∆ s ∈ R n × 1 represents the gradient update of s 0 obtained by adaptation to a downstream task. We dub Eq. 2 as Parameter-Efficient and Quantization-aware Adaptation (PEQA). PEQA is a memory-efficient fine-tuning method dedicated to quantized LLMs by solely updating quantization scales s 0 . With W 0 being frozen and shared for all downstream tasks, s 0 +∆ s are task-specific parameters in PEQA, which can be quickly and easily swapped when it is needed to switch to a different downstream task. Note that, PEQA can be seamlessly applied not only to weight-only quantized LLMs but also to weight-activation quantized ones. The overall procedure of PEQA is described in Figure 1 in detail.

3.3 Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization

PEQA is designed to have the advantages of both existing PEFT methods [19, 21, 46] and quantized LLM [28, 44, 47, 49]. We summarize the benefits of PEQA in this subsection.

Benefits of PEFT. By solely updating the quantization scales, PEQA substantially reduces the memory overhead associated with optimizer state, a feature consistent with other PEFT approaches. Notably, the utilization of quantization scales s 0 +∆ s allows PEQA to swiftly and effortlessly switch between task-specific parameters. This capability positions PEQA as ideally suited for deployment of quantized LLMs as a service, mirroring another key advantage of earlier PEFT methods. Note, however, that such a capability is not present in PEFT+PTQ (i.e., the case where PTQ is applied after PEFT) due to the non-reversible quantizers, such as the rounding function.

Benefits of Quantization. Previous PEFT methods freeze the pre-trained weights W 0 and utilize additional learnable parameters to reduce the memory usage of optimizer state. Similarly, PEQA freezes quantized pre-trained weights W 0 (which are the integer quantization values of W 0 ) and fine-tunes quantization scales s 0 . Since W 0 is a b -bit integer matrix, not only can PEQA reduce the optimizer states' size but also the model size, leading to even greater efficiency in the PEFT scheme, as illustrated in Figure 2a. In addition, since ̂ W is a b -bit quantized matrix, PEQA can speed up token generation process at inference through dedicated kernels that accelerate the multiplication between a quantized weight matrix and a half-precision activation vector, as described by Frantar et al. [28], Lin et al. [47], and Park et al. [49]. It is worth noticing that employing PTQ before fine-tuning (PTQ+PEFT) [50] allows for memory-efficient fine-tuning and seamless task transition for the quantized LLM; however, PTQ+PEFT is not able to inherit from inference acceleration of quantization. As a result, PEQA can achieve both model compression in the process of fine-tuning and inference acceleration for fine-tuned models with marginal performance degradation compared to LoRA, one of the state-of-the-art PEFT techniques, as shown in Figure 2b.

The comparison of PEQA with other methods using the LLaMA 65 B is summarized in Table 1.

4 Experiments

In this section, we empirically validate the effectiveness of our proposed PEQA method by examining its performance in both parameter-efficient fine-tuning (PEFT) and as a quantization method. We achieve this goal by using a series of benchmarks [52-57], datasets [51, 58, 59], and LLMs [4, 6, 60, 61] that have been publicly introduced. In Section 4.1, to empirically confirm the validity of fine-tuning only the quantization scale while freezing the integer matrix, we compare the perplexity of fine-tuned LLMs through quantization-aware training (QAT), PEFT (+PTQ), and PEQA. In Section 4.2, to evaluate PEQA's scalability and task-specific adaptation performance, we fine-tune and assess LLMs on the Wikitext2 [51] and PennTreeBank [58] datasets using PEQA and LoRA [21]. Section 4.3 is dedicated to showcasing PEQA's performance-restoring capability through instruction-tuning on the Alpaca [59] dataset after round-to-nearest (RTN) quantization over the full-precision original model.

To assess PEQA's performance as a PEFT method, we contrast PEQA with LoRA, which is currently recognized as one of the leading PEFT methods. As discussed in 3.1, we employ a baseline case that merges OPTQ [28], the state-of-the-art weight-only post-training quantization (PTQ) method for LLMs, with LoRA in order to evaluate PEQA's quantization capabilities. In the context of LoRA, QV 4 signifies the application of query and value layer weights with a LoRA rank of 4 , while QKVO 16

indicates the application of query, key, value, and output projection layer weights with a LoRA rank of 16 . For PEQA, we utilize round-to-nearest (RTN) for the initialization method of quantized LLM.

4.1 Comparing Quantization Capabilities: PEQA vs. QAT vs. PEFT+PTQ

Table 2 presents the perplexity when various quantized LLMs are fine-tuned using QAT, PEFT (+PTQ), and PEQA. To validate our approach, PEQA, which solely fine-tunes the quantization scale as outlined in Eq. 2 while simultaneously maintaining the integer matrix in a frozen state, we use QAT as an upper bound and PEFT+PTQ as a lower bound. Note that QAT, unlike PEQA, updates all parameters including pre-trained weights as well as quantization scales. Table 2 reveals the competitive performance of PEQA compared to QAT. Furthermore, our observations indicate that PEQA consistently outperforms the combination of LoRA and OPTQ for any selected model, regardless of whether a 3-bit or 4-bit setting is employed. Such superior performance can be attributed to PEQA's method of fine-tuning quantized LLMs, which minimizes the final task loss on the full training data, a capability that OPTQ lacks. Detailed settings are in Appendix B.

Diving deeper into the comparison between QAT and PEQA, it is important to note that QAT minimizes the final task loss computed from a weight-only quantized model, as described in Eq. 1, with respect to both W 0 and s 0 . Note that QAT includes all pre-trained weights for training, resulting in the practical model size limitation of LLMs under investigation being capped at 13 B in our experiments. Despite the fact that QAT also updates W 0 in Eq. 1, which is one of the most simple and straightforward approach though, we observe that the performance gap between QAT and PEQA narrows when the 4 -bit association is introduced, especially as the size of LLMs increases. Impressively, PEQA can even outperform QAT in a 3 -bit setting, a notably low-bit setting that challenges OPTQ in terms of quantizing LLMs. These findings suggest that the approach of PEQA, solely updating quantization scales while freezing the integer quantization values of pre-trained weights, can achieve performance comparable to that of QAT.

4.2 Task-specific Adaptation with Wikitext2 and PennTreeBank Datasets

Task-specific Adaptation and Scalability. We evaluate the task-specific adaptation performance and scalability of PEQA by employing GPT-Neo, GPT-J, and LLaMA models (up to 65 B) on the Wikitext2 [51] and PennTreeBank (PTB) [58] datasets. The adaptation performance of PEQA is compared with LoRA, with its configuration set to QV 4 . As depicted in Table 3, we note a gradual convergence of PEQA's perplexity to that of full-precision LoRA, with only marginal PPL degradation as the model size expands. Thus, Table 3 demonstrates that PEQA, compared to a prominent PEFT technique that utilizes full-precision pre-trained language model (PLM), can maintain a competitive perplexity level in LLMs while concurrently reducing DRAM usage through low-bit quantized

weights. Notably, for a 3 -bit quantization, PEQA experiences less performance degradation as the model size decreases due to extreme low-bit quantization compared to the combined LoRA and OPTQ. To further elucidate our findings, we have provided figures illustrating the results of 3 -bit and 4 -bit PEQA in the Appendix D. The comprehensive results indicate that for the deployment stage, PEQA allows models with larger parameters to operate under DRAM usage constraints, outperforming full-precision PEFT methods. For instance, under a restricted DRAM footprint, large LLaMA models can be explored using PEQA, while full-precision LoRA permits only smaller LLaMA models. Additional results with OPT [4], ranging from 1 . 3 B to 66 B models are included in the Appendix E. The detailed experimental settings are also included in the Appendix C.

Model Size and Number of Learnable Parameters. In an effort to estimate the DRAM usage necessitated by PEQA and LoRA during training and deployment, we outline the number of learnable parameters (for training) and the model size (expressed in gigabytes, GB, for deployment) in Table 4. As demonstrated in Table 4, PEQA involves fewer learnable parameters than LoRA when a quantization scale is assigned to each channel of pre-trained weights. For instance, PEQA has approximately 1 . 54 times fewer learnable parameters for LLaMA models than LoRA (QV4). In addition to having fewer learnable parameters, PEQA, through low-bit weight quantization, can also reduce the model size, which captures a huge amount of the DRAM footprint in fine-tuning LLMs. Remarkably, when fine-tuning LLaMA 30 B using PEQA with 4 -bit precision, the resulting model size is significantly smaller than that obtained by adapting 13 B through LoRA, and slightly larger than the model adapted from LLaMA 7 B using LoRA. Additionally, a comparison of the memory peak during training between PEQA and LoRA is provided in Appendix L.

Group-wise Quantization. Group-wise per-channel quantization [49], where weight groups in channels share quantization parameters, maintains accuracy at lower bits. In Table 5, we present that the performance incrementally improves as more learnable parameters are incorporated into PEQA. In particular, for Table 5, we examine various group sizes (denoted by g ) when quantizing the weights [49, 62]. Through relatively straightforward grouping (employed to regulate the number of learnable parameters for PEQA), the perplexity incrementally decreases as more learnable parameters are utilized. Detailed settings are in Appendix G.

4.3 Instruction-tuning with the Alpaca Dataset

Although inference cost and training efficiency are crucial, the methods of PEFT and quantization have not been extensively explored. While LoRA and PEQA have been evaluated on adaptation performance for task-specific purposes as discussed in Section 4.2, it has not been widely corroborated that fine-tuning via PEFT can retain the performance for unseen tasks. Thus, given that RTN quantization results in a non-negligible performance degradation in LLMs, it is important to assess how much the performance of low-bit quantized LLMs is degraded on comprehensive tasks. To address these concerns, we conduct comprehensive experiments, benchmarking our techniques on prevalent instruction-following datasets and assessing the response quality of PEQA-tuned LLMs. Furthermore, to determine if PEQA can regain the performance of full-precision LLMs, we employ RTN quantization in conjunction with PEQA instruction-tuning across LLaMAs.

Experimental Settings. We train LLaMAs [6, 7] in various sizes on the Alpaca dataset [59], which is one of the popular instruction-following datasets generated from outputs of InstructGPT [11]. Then, we test the models on other downstream tasks such as common-sense reasoning tasks [52-55] and massive multitask language understanding (MMLU) [56]. Due to limited time and resources, we could not conduct an exhaustive search over hyper-parameters such as the learning rate or epoch. Instead, we followed the training recipe from Taori et al. [59]. The LoRA configuration is set to QKVO 16 . Detailed settings can be found in Appendix H.

Common-Sense Reasoning. We conducted an experiment on five tasks [52-55] to assess whether the performance of common-sense reasoning and in-context learning can be sustained even after instruction-tuning LLMs on the Alpaca dataset via LoRA or PEQA. As depicted in Table 6, the results show that LLMs fine-tuned with LoRA or PEQA maintain a consistent trend in common-sense reasoning tasks. Furthermore, since PEQA's performance aligns closely with that of full-precision

adaptation, this consistency is observed even when the model size has been reduced through low-bit weight quantization. We utilized the evaluation code from Eleuther AI's lm-evaluation-harness [63].

Massive Multitask Language Understanding. To assess whether the performance of PEQAtuned models can be restored to the levels of full-precision model's performance, starting from RTN performance, we test our models on the MMLU benchmark containing 57 multiple-choice problems across various domains and levels of knowledge [56]. In this experiment, we utilize the RTN results as a baseline to determine the extent of degradation on quantized LLM. As shown in Table 7, instruction-tuning with PEQA boosts the performance of RTN quantized models. This observation supports our claim that our approach enables LLMs to regain their few-shot in-context learning and understanding capabilities, even though they are significantly smaller than their original model size through quantization. Unfortunately, it seems that the PEQA-tuning does not achieve the best performance in fine-tuning larger models. This might be because PEQA-tuning did not been sufficiently explored different epochs or learning rates. Nonetheless, the observation that the performance of the quantized LLaMAs is restored through PEQA-tuning using an instructionfollowing dataset highlights the potential to further enhance the accuracy of PTQ methods.

5 Conclusion

Fine-tuning aligns large language models (LLMs) with specific purposes. To maintain the comprehensive capabilities of LLMs while effectively aligning them, we introduce PEQA, a method that seamlessly combines the advantages of parameter-efficient fine-tuning (PEFT) and quantization in LLMs. PEQA not only reduces DRAM consumption during fine-tuning but also accelerates inference latency for deployment by retaining weights in a low-bit quantized format. Through rigorous testing across various datasets and LLMs, we have found that PEQA can match the performance of fullprecision baselines in task-specific adaptations, even with a significant reduction in model size. When combined with instruction-tuning, PEQA's performance demonstrates its ability to both preserve and enhance comprehensive knowledge after the inherent compromises of quantization, recovering the performance of original model by simply updating the quantization scales of the quantized LLM.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. volume 33, pages 1877-1901, 2020.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.

Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021. URL https://www.github.com/ eleutherai/gpt-neox .

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

OpenAI. Gpt-4 technical report, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3008-3021. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper.pdf .

Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds Mach. , 30(4):681-694, dec 2020. ISSN 0924-6495. doi: 10.1007/s11023-020-09548-1. URL https://doi.org/10.1007/s11023-020-09548-1 .

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 11048-11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.759 .

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=gEZrGCozdqR .

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 .

Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 6107-6122, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. emnlp-main.410 .

Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966 , 2022.

Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399 , 2022.

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res. , 23(1), jan 2022. ISSN 1532-4435.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2021.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr. press/v97/houlsby19a.html .

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9 .

Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4350-4359, 2019.

Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=rkgO66VKDS .

Xiandong Zhao, Ying Wang, Xuyi Cai, Cheng Liu, and Lei Zhang. Linear symmetric quantization of neural networks for low-precision integer hardware. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=H1lBj2VFPS .

Jung Hyun Lee, Jihun Yun, Sung Ju Hwang, and Eunho Yang. Cluster-promoting quantization with bit-drop for minimizing network quantization loss, 2021.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861 , 2022.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=tcbBPnfwxS .

Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438 , 2022.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 , 2023.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. CoRR , abs/2112.09332, 2021. URL https://arxiv.org/abs/2112.09332 .

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. CoRR , abs/2212.10560, 2022. doi: 10.48550/arXiv.2212.10560. URL https: //doi.org/10.48550/arXiv.2212.10560 .

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 , 2020.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.353. URL https://aclanthology.org/2021.acl-long.353 .

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.243. URL https://aclanthology.org/2021.emnlp-main.243 .

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=0RDcd5Axok .

Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6253-6264, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.433. URL https://aclanthology.org/2022.acl-long.433 .

Minseop Park, Jaeseong You, Markus Nagel, and Simyung Chang. Quadapter: Adapter for GPT-2 quantization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 2510-2517. Association for Computational Linguistics, 2022. URL https://aclanthology.org/2022.findings-emnlp.185 .

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization-aware parameter-efficient adaptation of large-scale pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 3288-3305. Association for Computational Linguistics, 2022. URL https://aclanthology.org/2022.findings-emnlp.240 .

Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. FlexRound: Learnable rounding based on element-wise division for post-training quantization. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 18913-18939. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/lee23h.html .

Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. arXiv preprint arXiv:2309.15531 , 2023.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/ huggingface/peft , 2022.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, 2023.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314 , 2023.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP , 2018.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL , 2022.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics , 19(2):313-330, 1993. URL https://www.aclweb.org/anthology/J93-2004 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https: //doi.org/10.5281/zenodo.5297715 . If you use this software, please cite it using these metadata.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 , 2023.

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: communication-efficient SGD via gradient quantization and encoding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1709-1720, 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/6c340f25839e6acdc73414517203f5f0-Abstract.html .

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenodo.5371628 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , pages 3505-3506. ACM, 2020. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/3394486.3406703 .

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6 .

A Common Experimental Settings

For the common experimental settings, AdamW [64] optimizer and linear-decaying learning rate scheduler were used. We use Deepspeed repository [65] 2 for FP16 and BF16 training. Additionally, we utilize Huggingface repository[66] 3 for training, evaluation code and dataset.

B Experimental Settings of Section 4.1

We compare the perplexity when weights are quantized and adapted by quantization-aware training (QAT), LoRA with post-training quantization (PTQ), and PEQA, using the Wikitext2 dataset in Section 4.1. The LoRA configuration is set to QV 4 . For PTQ method, we utilize OPTQ [28] 4 which is state-of-the-art low-bit weight-only PTQ method. We set the model's maximum sequence length to 1024 . Batch size and epoch for all experiments are set to 128 and 15 respectively. The learning rates for the experiments of Table 2 are displayed in Table 8. Learning rates for LoRA and PEQA are shown in Appendix C.

C Experimental settings of Table 3

In Section 4.2, Table 3 show the scalability and task-specific adaptation performance of PEQA by comparing with LoRA and LoRA+OPTQ on Wikitext2 [51] and PennTreeBank (PTB) [58] datasets. Detailed experimental settings are as follows. LoRA configuration is set to QV 4 . For PTQ method, we utilize OPTQ [28] which is state-of-the-art low-bit weight-only PTQ method. We set input sequence length after tokenization (block size) to 1024 for under 65 B models. For LLaMA 65 B, input sequence length after tokenization is set to 768 due to memory issue. Batch size and epoch for all experiments are set to 128 and 15 respectively. Learning rates for Table 3 experiments are shown in Table 9.

D The Perplexity of 3 -bit and 4 -bit PEQA on Wikitext2 Dataset

Figure 3 illustrates the results of 3 -bit and 4 -bit PEQA's next token prediction performance on the Wikitext2 dataset. As shown in Figure 3, 3 -bit performance of PEQA shows lower perplexity than 3 -bit post-training quantized LoRA. The results from the 3 -bit PEQA show that PEQA allows for continuity in model size options under DRAM usage constraints.

5

10

15

20

25

30

4.0

4.5

5.0

5.5

6.0

6.5

7.0

7.5

Perplexity

LoRA+OPTQ 4-bit

LoRA+OPTQ 3-bit

PEQA(Ours) 4-bit

PEQA(Ours) 3-bit

E OPT Models Adapted with PEQA and LoRA on Wikitext2 Dataset

Table 10 shows the perplexity of OPT[4] models adapted with PEQA and LoRA on the Wikitext2 dataset. The perplexity gap between LoRA and PEQA becomes smaller as the model size increases.

F LoRA Configuration Comparison on Wikitext2 Dataset

As shown in Table 11, the LoRA target module configuration of QV 4 and QKVO 16 has not much effect on perplexity on Wikitext2 experimental results. Table 11 shows equal tendency as mentioned in [21]. We utilize QV 4 configuration for Section 4.2 and QKVO 16 configuration for Section 4.3 respectively.

G Experimental Settings of Multi-scale Performance

In Section 4.2, Table 5 shows the perplexity of PEQA with grouping learnable parameters. We set model maximum sequence length to 1024 . Batch size and epoch for all experiments are set to 128 and 15 respectively. Learning rates for experiments are shown in Table 12.

H Experimental Settings of Section 4.3

In Section 4.3, we use the Alpaca dataset [59] for instruction-tuning. We set learning rate, epoch, and quantization group size as in Table 13. The batch size is set to 128 for all experiments in this subsection. As mentioned in Section 4.3, due to limited time and resources, we couldn't conduct an exhaustive search over hyper-parameters such as learning rate or epoch. We believe that there are hyper-parameters that can perform better. For LLaMA 1 series (LLaMA 7 , 13 , and 30 B), we truncate the prompt to the length of 2024 since their maximum sequence length is 2024 when evaluating the massive multitask lanugage understanding (MMLU) benchmark. Thus, for LLaMA 2 -70 B, we set the tokenizer max length to 1024 on fine-tuning due to the resource limit. Otherwise, we use default max length of tokenizer 5 on training. For the evaluation, we use default tokenizer setting. For every experiment in this section, the configuration of PEQA is set to 4 -bit RTN quantization.

I LLaMA 7 B and 13 B on Natural Instruction

To evaluate the instruction-following ability of instruct-tuned models, we test them on another instruction-following dataset, Natural Instruction (NI) [57]. Different from the Alpaca dataset, instructions of NI were collected by humans for existing 61 NLP tasks. For simplicity, we utilize evaluation splits consisting of 12 subtasks and restrict the maximum number of instances for each task to 200 . At test time, the model should generate proper output for the given input with instruction for the target unseen task. As shown in Table 14, we find LLaMAs trained with PEQA show consistently better zero-shot task generalization performance (ROUGE-L) in NI for all parameter sizes compared to those from LoRA.

J Comparison with AlphaTuning

When diving deeper into quantization scales, learnable parameters for both PEQA and AlphaTuning, it's worth noting that PEQA's adherence to uniform quantization means there's only one shared quantization scale for integer weight. Conversely, AlphaTuning's non-uniform approach means that for a b -bit quantization, there are b individual quantization scales for each weight matrix. Despite having multiple scales, AlphaTuning only fine-tunes one, leaving the rest static. As such, the number of trainable parameters are identical and AlphaTuning seems to offer a larger potential for a well-fitted model, but it can be easily seen that b -1 rest static scales introduced in AlphaTuning have limited usability, and thus the method may be prone to overfitting as evident through empirical results.

In Table 15, we conducted training on GPT-Neo and OPT 1.3B using the Wikitext2 dataset. Interestingly, PEQA, drawing from its methodological advantages, consistently demonstrates superior performance to AlphaTuning by at least 0 . 7 ppl on the Wikitext2 dataset. Both AlphaTuning and PEQA used channel-wise trainable parameters. Batch size of AlphaTuning is set to 32 .

K Choice of Updating Quantization Scales or Zero-Points

Uniform quantization can represent both asymmetric and symmetric quantizations, hence it's not always necessary to mandate the use of zero-points. This is why adopting a strategy of only learning the scale factor serves as a fundamental and scalable baseline. We opted for this approach to clearly establish its advantages. To determine the efficacy of learning only the scaling factors, we have incorporated additional experiments. By referring to the table below, it's evident that merely optimizing zero-points does not yield effective learning outcomes. Moreover, simultaneously optimizing both zero-points and quantization scales does not present any significant improvement in accuracy either.

L Memory Peak on Training

The memory consumption is not solely dictated by the model size but is also influenced by various other factors 6 . Our approach with PEQA inherently offers memory advantages during fine-tuning by striving to minimize both the model size and the number of training parameters. To provide a clear understanding of these benefits, we conducted tests using a single NVIDIA A100-80GB GPU and the causal language modeling code from the HuggingFace repository 7 . Both LoRA and PEQA fine-tuned the LLaMA-7B on the Wikitext2 dataset with a batch size of 2 without gradient accumulation. Our findings indicated that while LoRA peaked at a memory usage of 59GB during optimization, PEQA used just 43GB. Remarkably, this disparity (16GB, 7B) escalates as the model size increases; for instance, a 65B full-precision model under LoRA occupies 130GB, whereas PEQA remarkably uses just 33GB. Additionally, LoRA encountered Out-Of-Memory (OOM) issues at a batch size of 4, whereas PEQA, due to its efficiency, continued training seamlessly.

Peer reviews:
comments: SUMMARY:
This paper presents a method PEQA that combines quantization and parameter-efficient fine-tuning. It takes both advantages, including updating only a tiny fraction of model weights and saving the memory by quantization. The model weights will be decomposed into a matrix of low-bit integers and a scalar vector. During the fine-tuning, only the scalar vector will be updated. The paper has done comprehensive experiments to measure its effectiveness. The proposed method outperforms PTQ and has an acceptable degradation compared to LoRA, but which significantly saved the memory.

STRENGTHS:
1. The paper is well-written and easy-to-follow.
2. The idea and the method are clean.
3. The evaluation is comprehensive.

WEAKNESSES:
1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible.
2. The paper combines two existing techniques rather than inventing a new technique.

QUESTIONS:
1. Table 2: Why does PEQA have an even lower perplexity in some cases than QAT?
2. How does PEQA compare to the direct combination of LoRA and quantization (e.x. compress to 4-bit, then LoRA on the 4-bit matrices)?
3. What is the intuition behind the decomposition used in PEQA?
4. Table 4: Why PEQA degrades the performance of Five-Shot?

LIMITATIONS:
1. The degradation of the accuracy of the proposed method compared to LoRA is non-negligible.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper introduces a new framework, PEQA, for efficiently tuning Large Language Models (LLMs). PEQA uniquely fine-tunes the scale parameters during the instructing tuning phase while keeping the backbone parameters frozen. This approach allows PEQA models to maintain advantages during both the training and inference phases. The primary experiments are performed on the Alpaca datasets. The paper also compares PEQA's performance on various datasets with other efficient tuning methods, such as Lora.

STRENGTHS:
The paper is strongly motivated, emphasizing the importance of memory efficiency in tuning methods.

The authors highlight the role of quantization methods in reducing memory usage beyond what is achievable with parameter efficiency methods alone [1]. 

The paper is well-written and easy to follow.

[1]. https://github.com/huggingface/peft

WEAKNESSES:
Table 1 could be improved by emphasizing that other parameter-efficient tuning methods can also apply post-training quantization methods, such as GPTQ and AWQ.

The baseline should include the post-training quantization methods applied before or after fine-tuning (e.g., Lora and Adapter), and report on the memory usage during inference and training. This is particularly relevant as the HuggingFace PEFT library has incorporated Int8 quantization.

Reference:

[1] QLoRA: Efficient Fine-tuning of Quantized LLMs

QUESTIONS:
See Weaknesses

LIMITATIONS:
See Weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach named Parameter-Efficient and Quantization-aware Adaptation (PEQA) to address the challenges of efficiently fine-tuning and deploying large language models (LLMs). The paper demonstrates PEQA's effectiveness and scalability through extensive experiments, comparing it with competitive baselines across a range of tasks from natural language understanding to generation benchmarks.\n\nSTRENGTHS:\n* The paper centers on the topic of efficient model compression and fine-tuning, framing its approach through a unified method referred to as PEQA.\n\n* The authors' proposed method offers benefits, most noticeably in substantially reducing the memory usage associated with optimizer state saving and the final model size.\n\n* The authors have undertaken a comprehensive evaluation setup which encompasses several benchmark tasks and a variety of model sizes.\n\nWEAKNESSES:\n* The overall presentation of the paper could be improved for clarity. Specifically, the meaning of statements such as \"Notice that s0 and z0 have nothing to do with a choice of downstream task\" on Line 148, and the term \"integer quantization indices\" on Line 149, are not clear.\n\n* The calculation of memory usage during fine-tuning is not sufficiently explained. While it's understood that the proposed method lowers optimizer state storage by only updating scaling factors, the paper does not address memory usage during gradient calculation, a crucial aspect that often dictates peak memory usage during optimization.\n\n* It's unclear why the choice of only updating scaling factors is advantageous. The paper only compares the proposed method with LoRA, limiting the assessment of its effectiveness. For instance, the BitFit method (Ben Zaken et al., 2022) demonstrates that fine-tuning the LLM can be achieved by only updating biases. Moreover, the paper does not discuss the potential outcomes of optimizing zero-points only or optimizing both zero-points and the scaling factor.\n\nQUESTIONS:\n* The methodology section of the paper leaves some key details unaddressed. Specifically, it's unclear how the gradient of the scaling factors is computed in the proposed model. The paper PACT by Choi, Jungwook, et al. [1] presents two different ways to compute the gradient for the scaling factor/clamp_range for the quantization function. It would be beneficial for the authors to clarify which method was adopted in their work.\n\n[1] Choi, Jungwook, et al. \\Pact: Parameterized clipping activation for quantized neural networks.\\ arXiv preprint arXiv:1805.06085 (2018).\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a model compression method called Parameter-Efficient Quantization-Aware Adaptation (PEQA) to tackle the size challenge of Large Language Models (LLMs) while enhancing their task-specific fine-tuning. The PEQA technique quantizes the fully-connected layers into quantization scales and integer values for initialization, and during downstream tasks, only the quantization scales are fine-tuned, leaving the integer matrix fixed. This approach significantly reduces the parameters required for gradient updates and the memory for loading pre-trained model weights, making fine-tuning more memory-efficient than other techniques such as Post Training Quantization (PTQ) and Quantization-Aware Training (QAT). However, there are several concerns regarding novelty and effectiveness of the proposed methods.\n\nSTRENGTHS:\n- The paper provides an easily understandable explanation of the existing research trends and exhibits good readability.\n\n- By quantizing the pre-trained weights to sub-4bits, the memory cost for LLM fine-tuning has been significantly reduced. This enables a wider range of individuals to fine-tune LLMs according to their desired tasks efficiently. In particular, the existing similar approach, PEFT + PTQ technique, has been studied for models of size smaller than 1.3B. However, PEQA proves to be effective even for models as large as 65B and reduces memory costs.\n\n- Through various experimental results, the paper demonstrates the performance of the PEQA approach in fine-tuning and its application of quantization compared with PTQ and QAT results. This effectively shows how well the method works for the purpose of LLM fine-tuning.\n\nWEAKNESSES:\n- This paper's significant contribution, fine-tuning only the quantization scales (termed PEQA), seems similar to AlphaTuning [22]. The paper does not thoroughly discuss the limitations of AlphaTuning, only noting its lack of evaluation on large LLMs. \n\n- The baseline settings in the evaluation setup are unusual. For instance, the authors claim PEQA matches or exceeds QAT's performance. However, QAT's accuracy appears lower than state-of-the-art methodologies: e.g., LLM-QAT[a] demonstrated that 4-bit weight quantization almost reproduces the full-precision accuracy, but QAT in PEQA (Table2, Table3) shows noticeable degradation (LoRA 10.63 vs QAT 11.07). Such understated baselines could potentially mislead readers regarding PEQA's true performance.\n\n[a] LLM-QAT: Data-Free Quantization Aware Training for Large Language Models\n\n- The authors state that PEQA maintains generalization capabilities, yet the results are inconsistent. Table 4 indicates PEQA's zero-shot accuracy matches LoRA, but this improvement disappears in five-shot accuracy. Therefore, it is not clear if PEQA really preserves generalization capability or not.\n\n- There is little understanding of why fine-tuning only the quantization scales is sufficient. Therefore, it is hard to be convinced that the proposed method would work as expected. For example, Table 5 shows that PEQA even noticeably outperforms LoRA (without quantization) LoRA for zero-shot MMLU (and even LoRA+OPTQ outperforms LoRA for LaMMA-7B), which counters the accuracy trends of \"LoRA > PEQA > LoRA+OPTQ\" consistently shown in all the other tables. It would be desirable to provide insights into such unexpected evaluation results.\n\n- The problem setup in Section 3.1 doesn't align with PEQA's main objectives. The memory-bound issue during text generation inference discussed in this section seems less relevant to PEQA's fine-tuning memory efficiency benefits. Since forward computation isn't performed in the generation style during training, and text generation's memory-bound issue can be resolved by quantization (e.g., LoRA + OPTQ), including this issue in the problem setup may be misleading.\n\n- Comparing 3-bit weight quantization with OPTQ is valuable, but considering OPTQ's known limitations at this level of granularity, a fine-grained quantization comparison might be more convincing. Furthermore, a comparison with recent state-of-the-art weight quantization methodologies, such as LoRA + AWQ, could provide a clearer understanding of PEQA's effectiveness.\n\nQUESTIONS:\n- What could be the reason for PEQA outperforming FP fine-tuning with LoRA (+LoRA) as shown in Table 5? Additionally, the performance of applying OPTQ to LoRA actually increases in LLaMA-7B. Could this be interpreted as the LoRA Fine-Tuning approach not being optimally applied in this experiment?\n\n- From the Five-Shot results in Table 4, we can observe that the performance of PEQA on LLaMA-7B actually decreases, or its fine-tuning effect is insufficient compared to LoRA. Could this result be interpreted as the In-Context Learning ability of LLM not being well-preserved in PEQA Fine-Tuning?\n\n- How does the number of learnable parameters or model size change with a decrease in group size in Table 8? Only perplexity is provided, making it difficult to observe the trade-off together.\n\nLIMITATIONS:\nThe authors did not explain the limitations of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}