SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jeonghoon Kim*
NAVER Cloud
jeonghoon.samuel@gmail.com
Jung Hyun Lee*
onliwad101@gmail.com
Sungdong Kim
NAVER Cloud, KAIST AI
sungdong.kim@navercorp.com
Joonsuk Park
NAVER Cloud, Naver AI Lab,
park@joonsuk.org
Kang Min Yoo
NAVER Cloud, SNU AI Center
kangmin.yoo@gmail.com
Se Jung Kwon
sejung.kwon@navercorp.com
Dongsoo Lee
dongsoo.lee@navercorp.com
Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern. Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase. To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) - a simple yet effective method that combines the advantages of PEFT with quantized LLMs. By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage. We employ PEQA-tuning for task-specific adaptation on LLMs with up to 65 billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.
Figure 1: Illustration of our proposed PEQA scheme where  $A \cdot B$  indicates the element-wise product of  $A$  and  $B$ . PEQA is memory-efficient fine-tuning method for quantized large language models that updates only the quantization scale while keeping the integer matrix frozen. Notice a significant reduction in memory footprint when full-precision weights are converted into sub-4-bit integers.
Large language models (LLMs) such as PaLM, LLaMA, and the GPT-series [1-7] have demonstrated unprecedented levels of task-generalization ability in various applications, including dialogue systems, question answering, summarization, and translation [8, 9]. While they can follow instructions and learn to solve tasks via in-context task descriptions or few-shot examples [10], fine-tuning allows LLMs to align their behavior with desirable traits, such as following instructions more precisely [11] or adhering to certain principles [12]. Additionally, fine-tuning can improve the scaling curve by exposing the model to large collections of task-specific instruction datasets, leading to significant performance enhancements in various unseen downstream tasks [13-17]. However, the immense computational cost of fully fine-tuning large-scale models presents challenges for researchers and developers, especially given that LLMs have billions or even trillions of parameters [18].
In response, several parameter-efficient fine-tuning (PEFT) methods have been introduced [19-21], which only update a small number of parameters compared to the pre-trained weights of LLMs. PEFT notably reduces the number of learnable parameters, making the fine-tuning of pre-trained LLMs viable by ensuring that the optimizer states' memory usage becomes negligible. These strategies lead to decreased memory usage during training and more efficient storage and seamless transitions of task-specifically fine-tuned parameters during deployment. Nonetheless, LLMs as a whole still demand significant memory, and further reductions are attainable through model compression. As outlined in Hu et al. [21], for instance, LoRA can cut the memory usage during the fine-tuning of GPT-3 175B from 1.2TB to 350GB. However, the model still requires approximately 350GB of memory for parameters in half-precision floating-point format.
Quantization is a favorable method for both compressing and accelerating neural networks by discretizing parameters into low-bit integers while maintaining a shared high-precision scale within each parameter group (e.g., channel or layer). However, during training phases, quantization-aware training (QAT) [22-25] mandates updates for all parameters, rendering it not parameter-efficient. Since post-training quantization (PTQ) [26-29] is executed after training, most existing quantization schemes primarily target the deployment phases. Although PTQ can be integrated with PEFT, when PTQ follows PEFT, the model remains intact during fine-tuning, not decreasing the memory usage. Conversely, if PTQ precedes PEFT, while there's a reduction in memory usage during fine-tuning, no inference acceleration can be achieved due to the PEFT parameters during deployment.
To bridge the gap between PEFT and quantization, we introduce the Parameter-Efficient and Quantization-aware Adaptation (PEQA), a simple yet effective quantization-aware PEFT method. As illustrated in Figure 1, PEQA encompasses two steps: (a) Decomposition (Quantization) where the parameter matrix of each fully-connected layer is decomposed into a matrix of low-bit integers and quantization scales; and (b) Fine-tuning wherein, for each downstream task, the quantization scale is fine-tuned while the integer matrix remains unchanged. For the quantized LLMs, merely updating the quantization scale leverages the advantages of PEQA. As a result, PEQA maintains the merits of PEFT, such as fewer trainable parameters, along with efficient storage and swift switching of task-specific parameters. Concurrently, it provides the benefits of quantization, including reduced
DRAM usage during both training and deployment, and inference acceleration due to fewer memory accesses at deployment.
Through this, we highlight the following:
- We introduce PEQA, a method that fine-tunes only the quantization scales of quantized LLMs, keeping the integer matrix frozen. It bridges the gap between PEFT and quantization, offering advantages such as reduced memory consumption during both training and deployment phases, seamless task transitions, and faster inference.
- To empirically validate the approach of solely fine-tuning the quantization scale while freezing the integer matrix, we compare the perplexity of LLMs fine-tuned with QAT, PEFT (+PTQ), and PEQA. The results indicate that PEQA delivers competitive performance in comparison to QAT and PEFT+PTQ, even at sub-4-bit precision.
- To assess the scalability and comprehension performance of PEQA, we apply PEQA to task-specific adaptation and instruction-tuning. Despite the reduction in model size by a factor of 4 to 5, PEQA demonstrates competitive performance up to a 65B LLM when compared to full-precision baselines. The results suggest that even when LLMs are quantized into low-bit precision, the overall comprehension capability of quantized LLMs can be effectively restored to their original performance using PEQA.
Large Language Models and Alignment Learning. Although LLMs have demonstrated great generalization capabilities through their sheer scales [2, 6, 30] and the emergent mechanism known as in-context learning [1], they still require significant alignment to follow natural language instructions [13], adhere to ethical guidelines or steer towards harmlessness [12], utilize external tools [31], and to be grounded in knowledge to generate truthful answers [16, 32]. In particular, instruction-tuning has been pivotal in enabling LLMs to generalize instruction-following abilities, enabling them to solve seemingly any NLP task with only the description of the task in natural text [13, 33, 34], allowing the models to be accessed in an interactive manner.
Parameter-Efficient Fine-Tuning. Fine-tuning leverages the generalization capabilities elicited from the general pretraining to specialize in specific domains and tasks [35, 36] or align the LLM with target behaviors [13]. However, updating parameters in LLMs comes with a high computation cost and minimal compute environment required for gradient computation. As the hyper-scale era makes fine-tuning for LLMs prohibitively expensive, both efficient and effective alternatives to fine-tuning have received considerable attention. Specifically, inspired by the sensitivity of LLMs to prompts [37], a line of works has proposed introducing trainable prompt embeddings prepended to the input text while freezing the original LLM parameters [19, 38, 39]. As another approach, adapter modules [20] introduce task-specific parameters, which are inserted between the pre-existing layers of the model Extending on this adapter-based approach, LoRA [21] employs the concept of low-rank bottleneck modules while demonstrating comparable performance to full fine-tuning. Subsequent works have unified the various versions and diverging approaches to PEFT [40, 41] by formulating them in a single mathematical framework. These parameter-efficient methods have shown comparable performance to full model fine-tuning, presenting a cost-effective and efficient avenue for tailoring LLMs to specific tasks.
However, even with the adoption of PEFT, the inherent model size of the LLM remains a challenge to handle. One immediate solution is to apply post-training quantization (PTQ), but its interaction with task-specific parameters is still an area of active research. There have been attempts to integrate PEFT and neural network quantization, including methods like Quadapter [42] and AlphaTuning [43]. Yet, these methods have primarily been explored in smaller models of 1.3B or fewer parameters. Appendix J delineates the distinctions between our method and AlphaTuning.
Neural Network Quantization. Neural network quantization consists largely of quantization-aware training (QAT) and PTQ. QAT methods [22-25] basically train not only quantization scales but all the parameters of a full-precision neural network to narrow the performance gap between the full-precision model and its quantized counterpart. Unfortunately, since QAT involves training all the weights of a full-precision network, it is not feasible to apply QAT to LLMs. To quantize LLMs, PTQ
(b)
Figure 2: (a) DRAM usage comparison of LLaMA-65B on various tuning methods and (b) perplexity over model size when tuning LLaMA models with LoRA and PEQA on Wikitext2 dataset. The size of a circle indicates the number of trainable parameters. For instance, the LLaMA-65B model with LoRA has a size of 131GB and 10.49M trainable parameters. Otherwise, LLaMA-65B with 4-bit PEQA has a model size of 33GB and 6.8M trainable parameters.
techniques tailored to LLMs [26-29, 44, 45] have been presented. Although such PTQ approaches do not require learning all the parameters of an LLM at all, as PTQ occurs after training/fine-tuning LLMs, PTQ cannot allow for compressing the model size during training/fine-tuning LLMs. To reduce the model size even during training/fine-tuning LLMs, researchers have recently focused on combining PEFT with quantization.
Memory Demands in Fine-tuning. Given the significant computational demands associated with fully fine-tuning large language models, parameter-efficient fine-tuning (PEFT) methods have been introduced [19-21, 46]. One of the primary goals of PEFT methods is to reduce memory usage of the optimizer state during training, specifically by reducing the number of learnable parameters. While existing PEFT techniques do decrease memory consumption of the optimizer state and also narrow the accuracy gap between full fine-tuning and PEFT, the pre-trained weights of large language models still demand substantial memory space. When applying LoRA [21] to LLaMA-65B, for instance, even though storing optimizer states for trainable parameters consumes only 52MB (only query and value matrices are adapted with a LoRA rank of 4), the model size still occupies a huge portion of DRAM usage due to the frozen FP16 weights of a pre-trained model. To make PEFT more efficient, reducing the model size is an indispensable requisite. Given that LLMs mostly consist of fully-connected layers, compressing the weights of fully-connected layers is a key factor in compressing the model size and thus leading to more efficient PEFT.
Inference Latency of Large Language Models. During text generation inference, an autoregressive LLM generates tokens sequentially. A significant portion of the inference latency arises from matrix-vector multiplications, as opposed to matrix-matrix multiplications. Given that the batch size during inference is typically small [27, 28], matrix-vector multiplications tend to be memory-bound. Specifically, accessing global memory, such as DRAM, is expensive on contemporary high-end GPUs. Thus, the number of weights loaded into registers profoundly impacts the speed of multiplication between a matrix and a vector. To decrease the number of weights (subsequently increasing the weights loaded into registers), quantization is a widely researched method for both compressing and speeding up neural networks [29, 44, 47]. While quantization-aware training (QAT) imposes a significant load on both computation and memory, post-training quantization (PTQ) is often viewed as a fallback strategy among traditional quantization techniques to enhance the generation latency of LLMs. To both accelerate LLM inference and retain all advantages of PEFT, an innovative alternative approach should be pursued.
Table 1: Comparison of PEQA with other methods using LLaMA 65B on the DRAM usage and training time during fine-tuning, the DRAM storage for deployment, the inference acceleration, and task-switching efficiency. The DRAM usage estimation for PEFT is based on LoRA. PEFT+PTQ denotes PTQ after PEFT and PTQ+PEFT denotes PTQ before PEFT.
Method	DRAM (Fine-Tuning)	DRAM (Deployment)	Inference Speed	Task-Switching
Full Fine-Tuning	457GB	131GB	Slow	Slow
PEFT	131GB	131GB	Slow	Fast
PEFT+PTQ	131GB	33GB	Fast	Slow
PTQ+PEFT	33GB	33GB	Slow	Fast
PEQA (Ours)	33GB	33GB	Fast	Fast
3.2 Parameter-Efficient and Quantization-aware Adaptation (PEQA)
Quantization reduces bit-precision for inference acceleration, less storage and increasing throughput. INT8 quantization, which lower the bit-precision for both activations and weights, utilize dedicated engine to effectively accelerate arithmetic computation [48]. This is effective for large batches where computing speed matters but less so for smaller batches constrained by memory. To tackle this memory issue, weight-only quantization keeps high precision for activations (e.g., FP16) but compresses weights to 4-bit or less, targeting memory I/O enhancement in modern GPUs [28, 47]. For simplicity, we mainly focus on low-bit weight-only quantization in a linear asymmetric per-channel context in this paper.
For pre-trained weights of a fully-connected layer  $\mathbf{W}_0 \in \mathbb{R}^{n \times m}$ , while PEQA can be applied to quantized LLMs, we first quantize  $\mathbf{W}_0$ . In other words, for a given bit-width  $b$ , quantized pre-trained weights  $\widehat{\mathbf{W}}_0$  can be written as
$$
\widehat {\boldsymbol {W}} _ {0} = \boldsymbol {s} _ {0} \cdot \overline {{\boldsymbol {W}}} _ {0} = \boldsymbol {s} _ {0} \cdot \left(\operatorname {c l a m p} \left(\left\lfloor \frac {\boldsymbol {W} _ {0}}{\boldsymbol {s} _ {0}} \right\rceil + z _ {0}, 0, 2 ^ {b} - 1\right) - z _ {0}\right), \tag {1}
$$
where  $A \cdot B$ ,  $\lfloor \cdot \rfloor$ , and  $\operatorname{clamp}(\cdot, a, b)$  indicate the element-wise product of  $A$  and  $B$ , the rounding function, and the clamping function into the range  $[a, b]$ , respectively, while per-channel scales and zero-points (namely,  $s_0, z_0 \in \mathbb{R}^{n \times 1}$ ) are initialized to minimize  $\| \boldsymbol{W}_0 - \widehat{\boldsymbol{W}}_0 \|_F^2$ . Notice that  $s_0$  and  $z_0$  are not related to any downstream task. Here, we freeze  $\overline{\boldsymbol{W}}_0 = \operatorname{clamp}\left(\left\lfloor \frac{\boldsymbol{W}_0}{s_0} \right\rfloor + z_0, 0, 2^b - 1\right) - z_0$ , which is the integer quantization indices of  $\boldsymbol{W}_0$ , for every full-connected layer in a pre-trained LLM. And then we fine-tune only  $s_0$  (residing outside the clamp function in Eq. 1) while sharing  $\overline{\boldsymbol{W}}_0$  across all downstream tasks. Consequently, quantized pre-trained weights  $\widehat{\boldsymbol{W}}_0$  are adapted to a downstream task as follows:
$$
\widehat {\boldsymbol {W}} = \left(\boldsymbol {s} _ {0} + \Delta \boldsymbol {s}\right) \cdot \overline {{\boldsymbol {W}}} _ {0} = \left(\boldsymbol {s} _ {0} + \Delta \boldsymbol {s}\right) \cdot \left(\operatorname {c l a m p} \left(\left\lfloor \frac {\boldsymbol {W} _ {0}}{\boldsymbol {s} _ {0}} \right] + \boldsymbol {z} _ {0}, 0, 2 ^ {b} - 1\right) - \boldsymbol {z} _ {0}\right), \tag {2}
$$
where  $\Delta s\in \mathbb{R}^{n\times 1}$  represents the gradient update of  $s_0$  obtained by adaptation to a downstream task. We dub Eq. 2 as Parameter-Efficient and Quantization-aware Adaptation (PEQA). PEQA is a memory-efficient fine-tuning method dedicated to quantized LLMs by solely updating quantization scales  $s_0$ . With  $\overline{W}_0$  being frozen and shared for all downstream tasks,  $s_0 + \Delta s$  are task-specific parameters in PEQA, which can be quickly and easily swapped when it is needed to switch to a different downstream task. Note that, PEQA can be seamlessly applied not only to weight-only quantized LLMs but also to weight-activation quantized ones. The overall procedure of PEQA is described in Figure 1 in detail.
3.3 Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization
PEQA is designed to have the advantages of both existing PEFT methods [19, 21, 46] and quantized LLM [28, 44, 47, 49]. We summarize the benefits of PEQA in this subsection.
Table 2: To empirically confirm the validity of PEQA's approach, we compare the perplexity (PPL) of fine-tuned LLMs through QAT, PEFT+PTQ, and PEQA on Wikitext2 [51] for GPT-Neo 2.7B, GPT-J 6B, LLaMA 7B, and LLaMA 13B. Weights are quantized into either 3-bit or 4-bit per channel, without a group size [28, 49]. LoRA configuration is set to QV4. The lower PPL, the better.
Method	W Bits	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B
QAT	4	11.07	8.81	5.76	5.26
LoRA + OPTQ	4	12.09	8.91	7.13	5.31
PEQA (Ours)	4	11.38	8.84	5.84	5.30
QAT	3	12.37	9.60	6.14	5.59
LoRA + OPTQ	3	21.93	11.22	19.47	7.33
PEQA (Ours)	3	12.54	9.36	6.19	5.54
Benefits of PEFT. By solely updating the quantization scales, PEQA substantially reduces the memory overhead associated with optimizer state, a feature consistent with other PEFT approaches. Notably, the utilization of quantization scales  $s_0 + \Delta s$  allows PEQA to swiftly and effortlessly switch between task-specific parameters. This capability positions PEQA as ideally suited for deployment of quantized LLMs as a service, mirroring another key advantage of earlier PEFT methods. Note, however, that such a capability is not present in PEFT+PTQ (i.e., the case where PTQ is applied after PEFT) due to the non-reversible quantizers, such as the rounding function.
Benefits of Quantization. Previous PEFT methods freeze the pre-trained weights  $W_{0}$  and utilize additional learnable parameters to reduce the memory usage of optimizer state. Similarly, PEQA freezes quantized pre-trained weights  $\overline{W}_0$  (which are the integer quantization values of  $W_{0}$ ) and fine-tunes quantization scales  $s_0$ . Since  $\overline{W}_0$  is a  $b$ -bit integer matrix, not only can PEQA reduce the optimizer states' size but also the model size, leading to even greater efficiency in the PEFT scheme, as illustrated in Figure 2a. In addition, since  $\widehat{W}$  is a  $b$ -bit quantized matrix, PEQA can speed up token generation process at inference through dedicated kernels that accelerate the multiplication between a quantized weight matrix and a half-precision activation vector, as described by Frantar et al. [28], Lin et al. [47], and Park et al. [49]. It is worth noticing that employing PTQ before fine-tuning (PTQ+PEFT) [50] allows for memory-efficient fine-tuning and seamless task transition for the quantized LLM; however, PTQ+PEFT is not able to inherit from inference acceleration of quantization. As a result, PEQA can achieve both model compression in the process of fine-tuning and inference acceleration for fine-tuned models with marginal performance degradation compared to LoRA, one of the state-of-the-art PEFT techniques, as shown in Figure 2b.
The comparison of PEQA with other methods using the LLaMA 65B is summarized in Table 1.
In this section, we empirically validate the effectiveness of our proposed PEQA method by examining its performance in both parameter-efficient fine-tuning (PEFT) and as a quantization method. We achieve this goal by using a series of benchmarks [52-57], datasets [51, 58, 59], and LLMs [4, 6, 60, 61] that have been publicly introduced. In Section 4.1, to empirically confirm the validity of fine-tuning only the quantization scale while freezing the integer matrix, we compare the perplexity of fine-tuned LLMs through quantization-aware training (QAT), PEFT (+PTQ), and PEQA. In Section 4.2, to evaluate PEQA's scalability and task-specific adaptation performance, we fine-tune and assess LLMs on the Wikitext2 [51] and PennTreeBank [58] datasets using PEQA and LoRA [21]. Section 4.3 is dedicated to showcasing PEQA's performance-restoring capability through instruction-tuning on the Alpaca [59] dataset after round-to-nearest (RTN) quantization over the full-precision original model.
To assess PEQA's performance as a PEFT method, we contrast PEQA with LoRA, which is currently recognized as one of the leading PEFT methods. As discussed in 3.1, we employ a baseline case that merges OPTQ [28], the state-of-the-art weight-only post-training quantization (PTQ) method for LLMs, with LoRA in order to evaluate PEQA's quantization capabilities. In the context of LoRA, QV4 signifies the application of query and value layer weights with a LoRA rank of 4, while QKVO16
Table 3: To show scalability of PEQA, the perplexity (PPL) on Wikitext2 and PennTreeBank (PTB) was compared with LoRA and PEQA. In this comparison, only the weights were quantized into 3-bit and 4-bit per-channel without group size. LoRA configuration is set to QV4. A lower PPL value indicates better performance.
Method	W Bits	GPT-Neo 2.7B	GPT-J 6B	LLaMA 7B	LLaMA 13B	LLaMA 30B	LLaMA 65B
Wikitext2
LoRA	16	10.63	8.50	5.53	5.06	4.06	3.82
LoRA+OPTQ	4	12.09	8.91	7.13	5.31	4.39	4.10
PEQA (Ours)	4	11.38	8.84	5.84	5.30	4.36	4.02
LoRA+OPTQ	3	21.93	11.22	19.47	7.33	5.94	5.32
PEQA (Ours)	3	12.54	9.36	6.19	5.54	4.58	4.27
PTB
LoRA	16	15.92	12.92	9.14	8.52	7.21	7.11
LoRA+OPTQ	4	18.83	13.46	11.22	8.83	7.55	7.46
PEQA (Ours)	4	16.55	13.30	9.69	8.64	7.68	7.36
indicates the application of query, key, value, and output projection layer weights with a LoRA rank of 16. For PEQA, we utilize round-to-nearest (RTN) for the initialization method of quantized LLM.
4.1 Comparing Quantization Capabilities: PEQA vs. QAT vs. PEFT+PTQ