ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
Authors: Junjie Yin
Jiahao Dong
Yingheng Wang
Christopher De Sa
Volodymyr Kuleshov
Venue: Accepted by TMLR ()

Abstract: We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/4-bit precision on as little as one 24GB GPU. Our method, modular low-rank adaptation (ModuLoRA), integrates any user-specified weight quantizer with finetuning via low-rank adapters (LoRAs). Our approach relies on a simple quantization-agnostic backward pass that adaptively materializes low-precision LLM weights from a custom black-box quantization module. This approach enables finetuning 2-bit and 3-bit LLMs for the first time---leveraging state-of-the-art 2-bit QuIP# quantization and 3-bit OPTQ quantization---outperforming finetuning that relies on less sophisticated 4-bit and 8-bit methods. In our experiments, ModuLoRA attains competitive performance on text classification, natural language infernece, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-the-art ROUGE score on a popular summarization task. We release ModuLoRA together with a series of low-precision models as part of LLMTOOLS, a user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs.

Published in Transactions on Machine Learning Research (01/2024)
ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
Junjie Yin
jyin270jhu.ed
Deportment of Computer Science Johns Hopkins University
Jiahao Dong
jd7870cornelll.om edu
Department of Computer Science Cornell University and Cornell Tech.
Yingheng Wang
yw23490cornell.ed
Department of Computer Science Cornell University
Christopher De Sa Deportment of Computer Science Cornell Universily
odesa@es.corvv edu
Volodymyr Kuleshov
haleshoo@@cornell..hh
Department of Computer Science
Cornell University and Cornell Tech
Reviewed OpenReview:
Abstract
We propose memory-effiiiett finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/ precision on as little as one 24GB GPU. Our method, modular low-rank adaptation (MODULORA), integrates any user- specified weight quantizer with finetuning via low-rank adapters (LoRAs). Our approach relies on simple quantization precision LLM weights from a custom black- box quantization module. This approach enables finetuning 2-bit and 3-bit LLMs for the first time leveraging state-of-tha of-the-art 2-bit QulP# quantization and 3-bit OPTQ quantiatioo--outperfrmmnig finetuning that relies on less sophisticated 4-bit and S-bit methods. In our experiments, MoDULoRA allains competitive performance on text classification, natural language inference, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-tha art ROUGE score on popular summarization task. We release MODULORA together with a series of low-precision models as part of I.LTTOOL,, user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs.
backward pass that adaptively materializes low-
1 Introduction
Large language models (LLMs) excel across diverse tasks such as code generation, instruction following, and reasoning (Brown et al.) 2020: Scao et al.) 2023 Zhang et al., 2022). However, the massive size of these models-ofiee reaching hundrods billions of parameters -maess them challenging to deploy on
downstream tasks and motivates rescarch into efficient finetuning algorithms (Li & Liang 2021 Hu al. 2022

---

Published in Transactions on Machine Learning Research (01/2024)
Here, propose modular low-rank adaptation (MoDCLORA), memory-effcient finetuning algorithm for large language models (LLMs) that runs on consumer-grade hardware. For example, in 3-bit precision, MODULORA finetunes a LLaMA-30B model (Touvron et al. 2023) on one Nvidia RTX 3090 24GB GPU and a LLaMA-65B on one RTX A6000 48GB GPU. In 2-bit precision, MODULORA finctunes a LLaMA-30B or LLaMA-6BB on one Nvidia RTX 3090 24GB GPU.
Our approach adds high-precision low-rank adapters to the low-precision 3-bit or 4-bit weights of frozen base LLM obtained via modern quantization algorithms (Hubara et al.) 2021 Yao et al. 2021 Frantar et al, 2023 Crucially, MODULORA does not specify its own quantization procedure ratheer, it integrates with uscr-defined quantizers via a simple quantization-agnostti backward pass. This backward pass adaptively materializes low-precision LLM weights obtained from black-box quantizer and integrates them with high-precision low-rank adapters.
We release MODULORA as part of LLMTOOLS, a uscr-fricndly library that enables finctuning LLMs on consumer GPUs When paired with the modern OPTQ quantizer (Frantar el al. 2023). MODuLoRA enables finetuning bit LLMs for the first time, often outperforming methods based on less sophisticated 4-bit and 8-bit quantization. When paired with the state-of-thh-art Qul'# quantizer Chee et al. (2023) Tseng et al. (2023), MODULORA enables finetuning 2-bit LLMs for the first time, matching methods' performance on less sophisticated 4-bit, and 8-bit quantizat ion method. Across tasks in classification, natural language inference, and instruction following, our low-pree precision models achieve competitive performance using significantly less memory than existing approaches. On a popular summarizatiin benchmark, we attain a new state-of-the-art ROUGE score using a quantized LLaMA-65B model. We open-source all our low-precision models, including the first 3-bit family of Alpaca models that feature strong instruction-follwing performance multiple model sizes. Our findings reveal that high performance be achieved using smaller quantized LLMs than previously thought.
Contributions. In summary, this paper makes the following contributions: (1) we propose MODULORA, a memorr-efficient finetuning method that operates over low-precision weights obtained via a user-specified black-box quantization module; (2) we release L.M.ToOLS, a user-friendly Python library that features an implementation of MODULORA and that enables users easily finetune the largest LLMs on consumer GPUs; (3) we provide empirical evidence that high performance on downstream tasks can be achieved with smaller LLM than previously thought.
2
Background and Related Work
We are interested in finetuning a pre-trained LLM for downstream tasks (Li & Liang 2021 Lester al. 2021 Houlsby et al. 2019 Rebuffi al., 2017). LLMs the learnable weights and almost all of the memory used to store these weights appear in lincar layers We let. the weights and biases of these linear layers be denoted W() and b(() for i E (1,2...., n]. Given a pretrained network, our goal is to finetune it for downstream tasks using much less working memory than would be needed to store all of the W in full precision.
transformer architecture where almost all of
2.1 Large Language Model Finetuning
Because of the high memory requirements needed to fine-tune and store all the weights of a LLM., practitioners have developed a variety of paramcter-efiiceent fine tuning methods that learn in a lower dimensional space. These hods include tuning only the oulput layer (Devlin el al. 2018) and luning the prompt or prefix
passed as input to an LLM (Lester et al. 2021 Li & Liang 2021 Liu et al. 2023a b), as well as LoRA, which is the focus of this work.
Low-Rank Adaptation (LoRA) The LoRA algorithm (Hu et al. 2022 decomposes the weights W into sum of frozen base model weights Wo E Rdxd and small additive low-rank adapter AB" consisting of
These layers include the K, V, Q, and projection matrices of attention blocks and the linear layers of MLP blocks. For simplicity here consider square weight matrices W; the rectangular is straight.orward generalization.
2

---

Published in Transactions on Machine Learning Research (01/2024)
the product of two rectangular matrices A, B E Rax,, where >> 0 indicates the rank:
W = Wo WO+AB".
(1)
LoRA reduces the number of trained parameters by factor of 2r/d, lowering the storage, transmission, and task-switching overhead of inference on a system that alrcady maintains the basc model. However, LoRA must hold the base weights Wo in memory, which requires multiple high-end GPUs and precludes Luning large LLMs on commodity hardware.
2.2 Low-Precision Machine Learning
The computational requirements of modern machine learning models motivate wide range of efficient
machine learning algorithms (Li & Liang 2021 Hu et al. 2022 Frantar et al. 2023
Quantization Quantization methods for neural networks reduce the nunber of bits required to store model
weights (Dong et al. form
2019
2020
Yao et al. 2022 Park et al. 2023 A b-bit quantization method has the
(W, 2, =
W WDD(W, 2)
(2)
Here, the quantization algorithm Q takes a weight matrix W E Raxd (or its subset) and outputs a quantized version Wg E {0, 1, 2.....,2--1dde (using bits to represent each entry of W), as well as zero and scale parameters 2,8 E Id (in full precision). The dequantization algorithm D(W q, 2, recovers approximation W E Rdxd by rescaling the quantized weights as W = s W, + Z, where @ denotes the Hadamard product, and O + extended with numpy-style broadcasting.
Recently, Frantar et al. (023) proposed OPTQ, a quantization algorithm that scales to modern LLMs. The method iteratively runs two steps over the weight columns: (1) quantize with nearest rounding and compute the error, (2) update the remaining weights with a scaled error. Many our experiments finetune LLMs quantized with OPTQ.
Following OPTQ, Chee al. (2023) proposed QulP, quantization algorithm that makes two-bit LLM compression viable for the first time. The method follows a 2-step procedure: (1) an adaptive rounding procedure that minimizes a quadratic proxy objective,, (2) an efficient pre- and post-processing procedure ensuring weight and Hessian incoherence through multiplication by random orthogonal matrices. Further, Tseng el al. (2023 proposed QulP#, combining lattice codebooks with incoherence processing from QuIP to create ffate-of-the-aa 2 bit quantized models. We show the performance of QulP# (with DA codebooks) quantized LLMs on the SAMSum summarization experiment.
In concurrent work, Dettmers al. 2023 proposed QLoRA, an approach for tuning quantized LLMs based LoRA. While our work seeks to integrate with any user-defined quantization module (such as OPTQ), QLoRA defines its own quantization scheme, which is simpler than, say, OPTQ or QuIP. One advantage of our approach is support for 2-bit and 3-bit finetuning: QLoRA only supports 4-bit finetuning, We will also identify settings where using advanced quantizers yields performance gains over QLoRA. See Section 5.1 for details.
3 Low-Precision Low-Rank Adaptation with a Modular Quantizer
In this section, describe modular low-rank adaptation (MoDcLoRA), memory-efficient finetuning algorithm for large language models (LLMs) that leverages custom quantization algorithms and runs on consumer GPU hardware.
3.1 Low- Rank Adaptation of cow-Precision Models
3

---

Published in Transactions on Machine Learning Research (01/2024)
The first step of our approach is quantization: ap- ply a black-box quantization algorithm Q to a set of pre-trained weight matrices W This yields quan- tized weights, zeros, and scales (W 2z01 s())
class ModuLoRALLinearr(hooulu
"""Linear ModuLoRA Layer
def minit__seelf, ...):
self hat = quantize(pretrainedd.
Q(W()). We use W to denote the quantized weights stored in low precision, while W denotes the same weights materialized in high precision (both approximate the original weights W Crucially, we do not specify a quantization procedure @ as part of MODULORA -ertheer, we seek to support user- defined quantizers that are treated by our method is a black-box.
(self A, self.B) B) = lora_init(...)
def forward(self, x):
(hatWq, 2, s) self helf.hatwq_z.3
return LPLinear LPLinear.applyyeeeeee.......... axply(xx, hatWq, z, s) \
+ (x @ self.B) @ self.A.t() + self bias
class LPLinear(Function
"" "oow-Precision Linear Map @staticmethod
The core of our efforts focuses on finetuning the base quantized model. Our method first modifies the network by replacing each linear pyver-originally defined by the affine map + 2 z(W(O)T (Ty())++) +b() with the reparameterized low precision ModuLoRALinear layer in Figure given by
def forward(ctx, input, hatWq, 2, s) ctx. save_ wow_bbaaaawarddhaatt 2, s)
hatw dequantize output = input @ hatw. t(
2, s)
return output # hatw is deallocated @staticmethod
x - x (W())) + (A + (9)
(3)
def backward outpuut:
hatWq, z, s = ctx. saved_tensors we recompute hatw
Here A A(, () E Rdxr are learnable parameters initialized as in Hu al. (0022), and W D(W()) (2()), (() is the fixed dequantized weight ma- trix. Note that this is algebraically (but not compu- tationally) equivalent to transforming the quantized matrix as given in [ Lastly, MODULORA fits the A (i) and ( using backprop and gradient-based learning.
hatw dequantize (hatWq, 2, s) grad_input grad_output @ hatw # here hatw can be deallocated return grad_input, None, None, None
Figure 1: PyTorch pscudocode for MODULORA.
A key challenge in this procedure is to efficiently perform computations with high tigh-precision and Low-precision tensors. Clearly, the forward pass requires multiplying by weights stored in quantized w Below, we derive the backward pass for A (0), B() and show that it also requires multiplying by the transpose of the w(')s.
3.1.1 The Structure of Quantized Backward Pass
We illustrate the technical challenges that arise in the design of quantized backward pass in the context of a network of n ReduLoRALinear layers. Each ModuLoRALinear is effectively fully connected layer with reparameterized dense weights defined as
W = W + A [ T
(4)
biases (0, and outpuls y: for i = ..,,....n. We use y, = w x + b(0 to denote the pre-activation out of the i-th step and we use L to denote the loss. The backward pass seeks to compute gradients dL/dA (1) and dL/aB((, where we overload the Leibniz notation for derivatives also denote gradients. By the chain rule,
dL dL
dy;
5
O dy:
dA
Because of the additive structure of the weights w( in 44. dy./dA (i) is straightforward to handle as it is a function of the quantized weights (( The second term can be computed via the chain rule of caleulus
dL = dy dy: dy.
dL dy dy:
(6)
dy.
4

---

Published in Transactions on Machine Learning Research (01/2024)
where dy:/dy; is the derivalive of the activation function, and dys+1/dy; (w( =
The above derivations indicate that computing the gradient dL/dA (6) (the argument for dL/dB(0 is identical)
requires performing matrix-vector multiply dyai1 (W()) between a high-precision vector aL with a
dL
dyvi1 quantized matrix (W(O).. Performing this multiplication in stable and efficient way is a challenge that we must address.
3.1.2 Efficient Sexed-Precision Computation of Forward and Backward Passes
T in a high-precision format, our challenge would be solved: the matrix-velor multiplication Ty.44 (W()) in the backward pass would operate over two high-procision arrays, and would not introduce questions of efficiency and stability.
If we could precompute all dequantized weight matrices (WG))T dl
Unfortunately, precomputing all dequantized weight matrices (WG)T requires the same amount of GPU memory it would take to store the original high-precisin LLM. For this computation to fit on consumer GPU hardware, we need to avoid manifesting all the W() in memory at once. Using (3) naively, backprop would store all the Wi( from the forward pass to use them in the backward pass.
Efficient Mixed Precision Computation. Our strategy is to recompule the high-precision materialization w() of the quantized w in the backward pass rather than save it (Figure Lp. In the LPLinear function, the forward method dequantizes Wi)) and performs multiplication. Similarly, backward re-dequantizes W and computes the gradient derived in Appendix ?? via dynamic programming. The hatw goes out of scope and can be freed at the end of each method, so only one Wi() is ever stored in memory at any given time.
The amount of memory used in the forward pass of the LPLORA module is small: all the intermediates are either the same size as the input z, or even smaller (e.g. if x E R then @ self B is of size Rmrr for < d). The amount of additional computation involved is also small: the dequantization procedure W O W, only requires multiplying and adding scalar to each row of W..
Increasing Efficiency Further. Figure I depicts a weight materialization strategy in which ( is fully materialized at each layer in both forward and backward passes. To further reduce memory, can materialize elements of w(( only as needed. For many quantization algorithus (Nagel et al. 2020: Frantar et al. 2023), we can perform row materialization: dequantize (i)) one row at time and immediately multiply it with an input x. MODULORA also naturally generalizes to any direct seuoooby-guantieedeee provided by the quantizer Q, in which case materializing any part of ( may be unnecessary.
product subroutine
3.2 LLMTools: A Library for Efficient LLM Finetuning Using ModuLoRA.
We implement MODULORA as part of LLMTOOLS, user friendly library that enables users to interact with the largest LLMs on consumer hardware. The LLMTOOLS library enables finetuning LLMs in 2-bit, 3-bit, and 4-bit precision using the MODULORA algorithm. It also provides an casy-to-use Python API for quantization, inference, and finetuning, as well as modular support for multiple quantizers, LLMs (including LLaMA1, LLaMA2, BLOOM, and OPT), and optimization algorithms (including all that are compatible with the Hugging Face Trainer class). Lastly, LLMTOoLS supports easily loading datasets and sharing models via the HuggingFace Hub. Our code is available at: code to reproduce our results is available at:
our evaluation
A key quantization algorithm implemented in LLMTOOLS is OPTQ (Frantar et al. 2023 In order to integrate OPTQ with Lose-based finetuning, LLMTOOLS provides efficient CUDA implementations of mixed-precision matrix-vector multiplication, including row and weight materialization. We provide CUDA kernels for both row and weight matorialization in both the forward and backward passes. For maximum efficiency, we materialize elements of w((" in float16. The base quantized LLM models are represented via weights ((9 stored in 3 or bits, with scales and zeros s(0), (+) as well biases ( all stored float16. Similarly, to integrate QulP# with LoRA, LLMTOOLS provides CUDA kernels for weight re-materialization and orthogonal matrices multiplication in the forward and backward passses. The base quantized LLM
models are represented via weights wi( stored in 2 bits.
5

---

Published in Transactions on Machine Learning Research (01/2024)
4 Experiments
4.1 Setup
Models. We evaluate MODULORA and LLMTOOLS on the recent LLaMA Touvron et al. 2023) family
of models, as well as open-source BLOOM (Scao et al. 2023 and OPT models ZZang el al.
2022) We
quantize the models to 3 bits and 4 bits using OPTQ as in Frantar et al. (0023 with calibration 128 samples
from C4 RRafel et al., 2020). We quantize the models to 2 bits using QulP# as in Chee et al. (0023) Tseng et al. 2023) with D1 lattice codebook:
Baseline. We use LoRA (as implemented in the PEFT library (Mangrulkar al. 2022 to finetune models quantized in 8 bits using the BitsAndBytes library (Dettmers et al. 2022 also compare to full-precision results from the literature. In concurent work, Dettmers et al. (2023) proposed QLoRA, a related 4-bit finetuning algorithm implemented in the BitsAndBytes library. Accordingly, we present an experimental comparison of QLoRA with our approach, along with an in-depth discussion.
Training. We finetune all models on NVIDIA TITAN, 3090, and A6000 GPUs (depending on the model) with a LoRA rank of r = 8 and alpha of a 32, and report results from random seeds. We up the training procodure following Hu et al. 2022 with slight variation accommodate models. For fair comparison with the concurrent work by Dettmers et al. (2023), we use the exact same hyperparameter set up. Please see Appendix C for details on the hyperparameters used for each of our experiment.
particular language
4.2 Text Classification
Data & Metrics. We start with a simple text classification task where we seek to classify a short text snippet (up to 50 words) into its genre (e.g., fiction, telephone chat, etc.). We finetune 13B to 65B LLaMA models on 392,702 702 snippets from five genres and evaluate on 9,815 held out instances (Williams et al. 2018
reporting accuracy. This yields a challenging classification task for LLMs of all sizes.
LLAMA Tuning
Quantizer
13B
30B
65B
LLMTOOLS (3-bit)
OPTQ OPTQ
93.5 + 0.7
97.0
0.9
97.2
+ 0.8
LLMTOOLS (4-bit)
92.9
0.7
96.3
1.0
98.0
0.9
Bits&BBytes (8-bit)
LLMint8()
93.0 + 0.7
93.7
+ 1.0
98.6 + 1.0
Table 1: Text classification accuracy (%) for LLAMAs finetuned with LoRA & MODULORA in 3, 4, 8 bits.
Results. We observe that classification accuracy consistently improves as we increase the number of parameters of the LLM. MODULORA combined with 3-bit or 4-bit LLM offers comparable performance to 8-bit. finetuning in Bits&Bytes while using significantly less memory (Table I
4.3 Natural Language Inference
Data & Metrics. Next, we finetune LLMs on natural language inference tasks. The model is asked to prodict label from a small set (entailment, contradiction, or neutral) after being presented with sentence pairing (a hypothesis and premise sentence pair). We finetune 7B to 65B LLaMA models on the Multi-Genre Natural Language Inference Corpus (MNLI) (Williams et all 2018) and evaluate on the matched test sets (in-domain examples), reporting accuracy, Baselines from GPT-3 and T5 are included, as presented in Hu et al. (2022) and CChung et al. (2022).
Results. Our 2-bit and 3-bit 65B LLaMA model matches the performance a full-precision GPT-3+LoRA baseline. We also find that 3-bit and 4-bit models from LLMTools outperform 8-bit models from the Bits& library for the entire model size range. bit 3-bit and 4-bit MODULORA models
*QulP# also introduces based codebooks, which achieve an even lower element- wise mean squared emoor (MSE) than Da codebooks. With the application of Es codebooks, 2-hit. QulP# finetuning could potentially yield stronger results.

---

Published in Transactions on Machine Learning Research (01/2024)
either match or outperform their 4-bit QLoRA counterparts, often using less memory because of lower precision models.
Basclines
Models
Finetuning Adaptation
Model Size
Trainable Parameters
MNLI-m (accuracy)
GPT-3
Full Finetuning
175B
175,255.8M
89.5 - 0.1
GPT-3
Adapter
175B
40. 10.1M
91.5 + 0.1
GPT-3
LoRA
175B
4.7M
91.7 = 0.1
T5
Full Finctuning
11B
11,307.4M
92.2 + 0.1
LLaMA Finetuning
Quantizer
7B
13B
30B
65B
LLMTOOLS (2-bit)
QulP#((Da)
86.59 + 0.5
87.42 + 0.5
89.72 + 0.5
90.85 + 0.5
LLMTOOLS (3-bit)
OPTQ
88.98 + 0.2
90.20 + 0.2
91.09 + 0.2
91.42 + 0.1
LLMTOOLS (4-bit)
OPTQ
89.31 + 0.2
90.41 + 0.2
91.31 - 0.1
91.59 + 0.2
Byts& (4-bit)
QLoRA
89.28 0.2
89.67 0.2
91.22 + 0.1
91.36 0.2
Bits&Bytes (8-bit)
LLM.nn88))
88.95 + 0.1
90.08 + 0.1
91.15 + 0.1
91.55 + 0.1
Table 2: Natural language inference on the MNLI-m dataset evaluated using classification accuracy (%). Our LIaMA-65B-3bit model approaches stato-of-the-art scores using significantly less memory.
4.4 Abstractive Summarization
Data & Metrics. We finetune 7B-65B LLaMA and 7B-13B OPT models on the SAMSum dataset (Gliwa et al.) 2019), consisting of 14,732 (lext, summary) training pairs and 819 Lest pairs. Our methodology fully mirrors the evaluation GPT-style models finetuned using LoRA (Hu et al. 2022).. We evaluate summarization quality using ROUGE-1/2/L; we include GPT-3 basclines from Hu al. (2022).
Results. Our 4-bit 65B LLaMA models finetuned with MODULORA outperform the GPT-3 baseline and even reach new state-of-the-art performance this dataset (Table 33. Importantly, MODU- LoRA demonstrates performance improvements over the 4-bit QLoRA and the 8-bit BitsAndBytes methods. In the 7B to 65B model size range, MODULORA models (3-bit or 4-bit) outperform 8-bit LoRAs in Bit- sAndBytes and L.L..nn8() and 4-bit. LoRAs in BitsAndBytes and QLoRA. MODULORA models (2-bit) match the performance of 8-bit LoRAs in BitsAndBytes and LLM.int8() and 4-bit LoRAs in BitsAndBytes and QLoRA. We argue that a data-driven lower precision quantization scheme can improve over a higher precision zero-shot quantizer like LLM.int().. Switching from 4-bit. to 3-bit, and then from 3-bit to 2-bit, precision within MODULORA reduces ROUGE by only about 1%.
Baselines
Models
Finetuning Adaptation
# Trainable Parameters
SAMSum (Rouge 1/2/L)
GPT-3
Full Finetuning
1 17,2555.8M
52.0
28.0 / 44.5
GPT-3
Adapter
40.1M
53.2
29.0 / 45.1
GPT-3
LoRA
4.7M
53.8
29.8
/ 45.9
Pegasus
SliC
2B
54.4
29.9
45.9
LLAMA Finetuming
Quantizer
7B
13B
30B
65B
LLMTOOLS (2-bit)
QulP## (Da)
49.2 / 26.9
42.7
50.7
/ 28.6
44.4
51.6
/ 30.2
46.4
52.3
30.5 / 46.8
ILMTTOOL (3-bit)
OPTQ
51.2
/ 28.2
44.0
52.4
/ 29.6
/ 45.1
53.6 / 30.8
/ 46.3
54.1
/ 30.9 / 46.5
LLMTOOLS (4-bit)
OPTQ
51.7
28.3
44.4
53.2
30.2
46.1
53.9
31.2
46.9
55.9
32.7
49.0
Bits&Bytte (4-bit) Bits&Bytes (8-bit)
QLoRA LLM.in8()
51.6 / 28.3
44.5
51.3
28.1
44.1
53.0
30.2
45.7
53.8
30.5 / 45.9
51.9
28.1
44.5
51.3
28.2
43.6
50.8
/ 28.4
44.1
53.9
30.4 / 46.3
Table 3: Abstractive summarization the SAMSum dataset evaluated using ROUGE 1/2/L. Our LLAMA- 65B-4bit model obtains state-of-the-art ROUGE scores. All metrics have +0.5 confidence intervals.
7

---

Published in Transactions on Machine Learning Research (01/2024)
SAMSum Performance
Quantizer
7B
13B
OPTQ
51.2
28.2
44.0
44.2
52.4
29.6
/
45.1
45.1
LLMTOOLS (3-bit)
RTN
50.7
27.2
43.6
43.6
51.1
28.7
44.3
44.5
OPTQ
51.7
28.3
44.4
44.4
53.2
30.2
46.1
46.1
LLMTOOLS (4-bit)
RTN
51.2
28.5
44.2
44.2
52.5
29.9
45.5
45.5
Table 4: OPTQ and RTN quantization with different LLaMA model sizes on the SAMSum dataset. The evaluation was done on ROUGE 1/2/L/LSSm.
Round-to-Nearee Quantization We also perform an ablation where we replace the OPTQ quantizer with a cout-to-nearres (RTN) approach (Table 4h OPTQ performs better than RTN, highlighting the importance of advancod quantizers.
Other Model Families We also apply LLMTOOLS to the OPT (hang et al. 2022) families of models (Table 5 Although these models perform worse than LLaMA, MODULORA matches or outperforms more inmooy-intensive 4-bit and 8-bit finetuning, which is consistent with results I.LMA.
OPT Finetuning
Quantizer
13B
30B
LLMTOOLS (3-bit)
OPTQ
48.8
26.7
/ 41.9
49.9 / 27.1
/ 42.5
LLMTOOLS (4-bit)
OPTQ
49.3
26.8
/ 42.0
49.6
27.1
42.4
Bits&Byttes (4-bit)
QLoRA
49.2
27.0
/ 42.1
49.9
/ 27.0
42.5
Bits&Bytes (8-bit)
LLMint8()
48.8
26.5
/ 41.7
49.3
27.1
42.3
Table 5: Abstractive summarization with OPT models on the SAMSum dataset. MODULORA in 3-bit and 4-bit precision matches ROUGE 1/2/T. scores of 4-bit and 8-bit baselines. All metrics have +-0.5 confidence intervals.
4.5 Instruction Following
Data & Metrics. We finetune 7B-65B LLaMA models on the Alpaca dataset Taori et al. 2023 consisting 52,000 instructions, as well on the CodaAlpaca dataset (Chaudbary 2023), consisting of 20K code generation instructions (ses Table Dh. We evaluate our Alpaca instruction-tuned models on the BigBenchHaard (BBH) benchmark (Suzgun et al. 2022), consisting of 23 challenging tasks on which LLMs do not exceed human performance. We evaluate 3-shot performance via. "answer-only' prompting and use exact. match accuracy as our measurement standard, testing on 6,511 samples (~ 1.5k tokens each). We include Flan and LLaMA basclines from Chia et al. (2023
Results. We find that 3-bit and 4-bit performance drops only slightly relative to 8-bit and 16-bit. 2-bit models, despite their aggressive compression, match the performance of 4-bit QLoRA in smaller model sizes. Crucially, 4-bit and 3-bit 65B models outperform 8-bit and 16-bit 30B models, despite using lewer lotal bits. Furthermore, 4-bit MODULORA compares well to 4-bit QLoRA, and provides consistent. performance improvements, especially at smaller model sizes, where sophisticated quantization ought provide greater benefits. This further highlights the benefits of one-shot quantization methods. Appendix also reports experiments on the CodeAlpaca dataset.
4.6 Memory Requirements
We show the memory required to perform finetuning on MNLI-M for different LLaMA model sizes table MODCLORA significantly minimizes the memory requirements for finetuning on these models. We plot the memory requirements in figure 2 for better visualization. As the model size increases to 65B, memory-efficient finetuning method LoRA. As the
MoDuLoRA only about 6% of the memory
8

---

Published in Transactions on Machine Learning Research (01/2024)
Baselines
Model
Method
Quantizer
BASE (250M)
(780M)
XL (3B)
XXL (11B)
FLAN-T5
No Finetuning
None
30.8
30.3
39.9
47.4
Model
Methods
Quantizer
7B
13B
30B
65B
LLMTOOLS (2-bit)
QulP# (Da)
30.3 + 0.7
33.3 + 0.6
37.0 + 0.9
39.3 + 0.9
LLMTOOLS (3-bit)
OPTQ
31.1 + 0.4
35.3 - 0.2
37.2
+ 0.6
43.3 + 0.4
LLMTOOLS (4-bit)
OPTQ
33.1 0.2
36.2
0.4
40.4
0.2
43.7 + 0.4
LLaMA
Bits&BBytes (4-bit)
QLoRA
31.9 + 0.1
35.4 - 0.2
39.0 + 0.4
43.5 - 0.5
Bits&Bytes (8-bit)
LLM.int8()
33.3 0.3
36.8 + 0.2
39.1 + 0.5
44.7 + 0.4
No Finetuning
None
30.9
37.1
39.3
42.6
Table 6: Instruction-tuned models evaluated on BigBench Hard (BBH). We finetune LLaMA models on the Alpaca dalaset in to 16 bits. We provide exact standard deviation here.
table and figure illustrates, with MODULORA it's possible to not only inference but also finetune 65B model single 24GB GPU. To produce this table, we quantizer-agnostic forward/baa backward passes for the entire LLaMA model size range with batch size 1 and maximum sequence length 128 on MNLI-m.
ILaMA Finetuning
7B
13B
30B
65B
LLMTOOLS (2-bit)
3.2 GB
5.4 GB
11.4 GB
21.8 GB
QLoRA (4-bit)
5.2 GB
8.6
GB
19.5 GB
36.7 GB
Full Precision (LoRA)
38.4 GB
73.9 GB
183.3 GB
360.4 GB
Table 7: Mernory requirements to finetune LLaMA models on MNLI-M with batch size and maximum sequence length 128. For comparison, we include the memory requirements to finetune on LoRA and QLoRA.
5 Discussion
5.1 Comparison to Related Work
Comparison to QLoRA In concurrent work, Dettmers et al. (2023) proposed QLoRA, a related ap- proach for finetuning a quantized LLM. We highlight methodological and experimental differences below. From methods perspective, MODULORA integrates with user-specified black-box quantization module. In our periments, we find that using sophisticated data-driven quantizer like OPTQ improves performance over sim- pler zero-shot strategics, e.g., a round-tor monddtto-narestt baseline. Unlike MODUL.ORA, QLoRA defines quantization ap- proach similar RTN, but also introduces specialized packing routine, quantization of zeros and scales, and other innovations.
400CB
360
Required Memory (Gigahyte)
183
100GB
20
10GB
LLMToolk (2-bit)
QLaRA (4-
Full precision
0GB
13B
33B
65B
Model Parameter Size (Billion)
From an experiments and capabilities perspective, inte- grating with OPTQ enables MoDULoRA to fintune mod- els quantized in 2-bits and 3-bits, which QLoRA cannot do. Lastly, we identify scttings where MODULORA yields LLMs with better performance than LLMs from QLoRA; this gap is likely due the use of improved quantizers.
Figure 2: Visualization of memory requirements for different LLaMA model sizes with different methods.

---

Published in Transactions on Machine Learning Research (01/2024)
Comparison to Other Parameter-Efficies Finetuning Methods Recent Parameter-Efficte Fine- tuning (PEFT) methods have encompassed a range of techniques such as prompt tuning Lester et al. 2021
Li & Liang 2021: Qin & Eisner 2021 Liu et al. 2022b), modification of the embedding layer inputs (An ct al. 2022 or hidden states (Liu et al. 2022a], inc inclusion of full layers (Houlsby et al.) 2019). only tuning
biases Zaken et al. 202T); and others Sung et al.) 2021 Karimi Mahabadi et al.) 2021). An important
shortcoming of these methods the need to store in memory a significant amount of frozen base model
parameters. This limits their ability to finetune the largest LLMs on consumer GPU, limitation that we address.
5.2 Running LLMs on Consumer GPUs
Efficient LLM Algorithms The computational requirements of modern deep neural networks motivate a wide range of efficient. machine learning algorithms. Quantization methods reduce the number of bits required
to store weights (Dong et al.) 2019 2020 Hubara et al. 2021 Li et al. 2021 Yao et al., 2021), including via
2023 rescales between activations and weights to remove outliers from the activations and make quantization overall easier. ZeroQuuant Yao et al., 2022) proposes a per-layer knowlodge distillation method. LLM int8() (Dettmers et al., 2022 decompose mairix multiplicatiins inlo a majority of 8 bit and minority of 16 bit operalions. LUT-GEMM (Park el al. 2023) designs kernels to accelerate quantized matrix multiplications. RPTQ (Yuan et al.; 2023) reorders activations and quantizes them in groups, reducing the impact of range differences between channels.
adaptive methods (Nagel et al. 2020 Smooth (iioo el al.
Running LLMs Consumer GPUs Our methods for 3-bit and 4-bit precision enable the finetuning of a 65B LLM a 48GB GPU, and 30B LLM on 24GB GPU. Additionally, our 2-bit approach allows for the finetuning of a 65B LLM on 24GB GPU, making the finetuning of LLMs accessible on consumer hardware. Moreover, fitting an entire LLM on GPU unlocks data parallelism, which is more efficient than model parallelism. Previous 8-bit quantization methods required a 96GB GPU to fully fit a 65B model. Finetuning GPUs on consumer hardware holds promise to accelerate model iteration and apply LLMs to a wider range of domains by a larger number of practitioners.
5.3 What is a Good Base LLM for Finetuning?
Models
Quantization
BBH
PPL
The traditional measure base LLM is perplexity. In the adjacent table, we report LLaMA perplexity (PPL) Wiki2 well finetuning performance on BBH. Inter- estingly, the correlation is not perfect: large gaps in PPL admit small gaps in BBH. This questions LLM evaluation when the goal is finetuning, and suggests exploring new training strategies.
3-bit
35.3
6.63
LLaMA (13B)
4-bit
36.2
5.36
3-bit
43.3
5.04
LLaMA (65B)
4-bit
43.7
3.84
Table BBH vs. PPT.
More generally, our results provide empirical evidence that high performance on downstream tasks can be achieved with a smaller quantized LLM than previously thought. While existing methods (e.g., L.M.innt()+ToRA)) Dettmers et al. (2022) operate in 8 bits, we find that 2-bit, 3-bit, or 4-bit finetuning yields the best results for a fixed bit budget. For example, we find that 4-bit and 3-bit 65B models outperform 8-bit and 16- 30B models instruction following lasks. On the SAMSum summarization lask, we find that 3-bit models are able to attain new state-of-the-ar ROUGE score, and 2-bit models match the performance of 8-bit models quantized with I.M.int8).. The high performance of these low-precision models suggests that competitive finetuning performance can be achieved on any base quantized LLM with x-bit precision, provided that the LLM exhibits reasonably good performance from the beginning.
5.4 Limitations
An advantage LoRA is that has low inference overhead, since the low-rank adaptor can be added in to the full-procision weight matrix when deploying. One limitation of MODCLORA is that it does not share this advantage relative to the black-box quantized model: the low-rank adaptor cannot be trivially added to the weight matrix because the weight matrix is quantized while the adaptor is not. So, the weight matrix
10

---

Published in Transactions on Machine Learning Research (01/2024)
and adaptor cannot be fused readily, and implementation in Figure 1 is required at inference time. A second limitation of MODULORA is that making finetuning possible widely available commodity hardware may make finetuning too easy, presenting potential problems related to LLM safety. Another limitation of MoDuLoRA is that the largest models use today (c.g. GPT-4) can have up to 1 trillion parameters, and even at the minimum on commodity GPUs; thus straightforward application of MODULORA will be unable to make these largest-scale models finetunable on commodity hardware.
bit per parameter this still would take up 125 GB, which exceeds memory
6 Conclusion
Finetuning large language models typically requires substantial hardware and storage resources. Our method, MoDeL.ORA, enables 2-bit. finetuning of 65B models single 24GB consumer GPU and also supports 3-bit and 4-bit finetuning of the same models using single 48GB GPU. At the core of our approach is a simple, quantization-agnostic backward pass that enables integrating low-rank adapters with frozen LLM weights obtained from a user-defined quantization module. By integrating with modern quantizers, MoDuLORA achieves statc-of-tho-art performance compared to both parameter-efficien and full finc-tuning techniques.
MODuLORA'S flexibility and competitive performance make finetuning more accessible and cost-effective in resourco-constrained setting. This assists open-source model development and facilitates scientific rescarch. More broadly, we believe that MODuLORA will help democratize access to large language models and make them available to a broader audience.
11

---

Published in Transactions on Machine Learning Research (01/2024)
References
Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs to frozen pretrained models. ar preprint arXiv:2208.0810 03131, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et. al. Language models are few-shot. learners. In Conference on Newral Information Processing Systems, 2020.
Sahil Chaudhary. Code alpaca: An instruction-followin llama model for code generation. https://githu: 2023.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees, 2023.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arxi::2006.04 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetueed language models. arXiv preprint arX:::2210.114 11416, 2022.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int((): 8-bit matrix multiplication for transformers at scale. In Conference on Neural Information Processing Systems, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.1423 14314, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprinl arXiv:1810.0488 2018.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed precissoo, In International Conference on Computer Vision, 2019.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trightee quantization of neural networks. In Conference on Neural Information Processing Systems, 2020.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations, 2023.
Bogdan Gliwa, Twona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarizalion, pp. 19,79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.186533/11D119-5444 URL httpss///aclanthooggoooo////////
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference Machine Learning. PMLR, 2021.
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercom- 2021.
plex adapter layers. Advances in Neural Information Processing Systems,
12

---

Published in Transactions on Machine Learning Research (01/2024)
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effcient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3055 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 http///v122211//////// .em.lp-main.2455. 243. URL https:///claatthooog/ org. eetp1..mnppppapa..e maal..mmin... 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeling of the Associalion for Compulational Linguislies and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 1..18653/1122221.cchh
URL
h hwwpillooggooggg/g/// ntrg/2021.aa long. 353
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiiien fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 55:1950-1963 2022a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Compuling Surveys, 55(9): 35, 2023a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang, P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022b.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023b.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-ar parameter-effcienn fine-tuning methods. httts:///ggituub..oom//ugggggga wwtpgggiinnacc/e/eeeee /pftt 2022.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post- cest-training quantization. In International Conference on Machine Learning, pp. PMLR, 2020.
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in largo-scale gencrative language models. arXie preprint arXiv:2206.095 2023.
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2104.06599 2021.
Colin Rasfel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Malena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer, 2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Compuler Vision and Pattern Recognátion, pp- 2001-2010, 2017.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2023.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24222 2021.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of- thought can solve them. arXiv preprint wrXiv:2210009900 2022.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-ffolooiing llama model. https://github.ccm
tatsu-lab/ Batsu--abbsttanoor__laac. 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient. foundation language models. arXiv preprint Xii:2302213977 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Chris De sa. Quip#: Quip with lattice codebooks. tttp:///ormcll--llamm..ittt
ht/puippsharpp///// 2023.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL N18-1101
Guangxuan Xiso, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint 2rXim:2211.10438 2023.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W. Mahoney, and Kurt Keutzer. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning. PMLR, 2021.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Conference on Newral Information Processing Systems, 2022.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Luzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXix:2304.001 01089, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-eflcient fine-tuning for transformer-bascel masked languago-models. arXin preprint arXiu:2106.115 1ett11066.119,, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Danicl Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
14

---

Published in Transactions on Machine Learning Research (01/2024)
A Additional Implementation Details
A.1 Configurations for BBH Evaluation
We evaluate the BBH dataset using LoRA adapter weights from huggingface hub with different configurations. For the Byts& 8-bit (I..M.int8()) LoRA adapter weights, we utilized two sources: the A Alpaca-7B one is obtained from the "tloen/alpp haa//aapaaa-ooraa/0 repository, while the weights for Alpaca-13b and 30b were sourced from "chansung/alpaca-lootsi aappaaallaa-aaaibibbiaaaaaaaaaaa In the case of Bits&Bytes 4-bit (QLoRA) adapter weights, all configurations Note that for the
(Alpaca-7B, 13B, and 30B) were uniformly accessed from Byte& 4-bit (QLoRA) and Bytes 8-bit (LLM.int8() adapter wights of the 65B model, we obtain them by finetuning the base 65B LLaMa model on Alpaca dataset using the same set of hyperparameters ours.
B Additional Empirical Experiments
B.1 Additional Experiments on Code-Alpaca with LLaMA
We conducted additional experiment on Code-Alpaca ( (Chaudhary Consistent with our hypothesis, MODULORA performs better than or at least on par with the higher precision 8-bit models given the same number of trainable parameters and set up.
2023
The result shown in Table g
Code Alpaca Performance
7B
13B
30B
65B
LLMTOOLs (3-bit)
53.6
36.3
/
50.7
57.0 / 40.0 / 53.3
58.1 / 40.7
54.3
60.0
44.1
/ 58.8
LLMTooLS (4-bit)
54.6
37.2
51.4
57.4 / 40.6
54.3
59.0
41.4
57.5
60.2
43.5
56.8
Byte& 8-bit (LLM.int(()
51.0 / 36.3
50.9
57.7
41.3 / 54.9
60.6
43.5
57.5
61.1
44.1 / 58.0
Table 9: Instruction-tumed models evaluated using ROUGE 1/2/L-Sum on Code Alpaca in 3, 4, and 8 bits.
B.2 Finetuning Inference Latency
We conducted experiment to test the finetuning and inference latency of MODULORA.
During finetuning, MODULORA significantly outperforms full-precision LoRA as show in table 10 reducing the training time by approximately 59.3% and memory usage by 91.5%. This efficiency in finetuning speed is primarily attributed to reduced data movement within the GPU memory.
Finetuning.
Inference. During inference, MODULORA has a slightly lower speed compared LoRA and QLoRA shown in table 11] We attribute this to the use of CUDA kernels that currently not as optimized those of QLoRA. Note that
Precision
LLMTools
QLoRA
LoRA (Full Precision)
Precision
LLMTools
QLoRA
LoRA
(2-bit)
(4-bit)
(2-bit)
(4-bit)
(Full Precision)
Seconds/Iteration
0.61 s/it 0.80 s/it
1.50 s/it
Seconds/Tterattos
0.68 s/it
0.52 s/it
0l52 s/it
Table 10: Finctuning speed for LLAMA 7B MNLI- m benchmark with batch size 1. We report the average time to complete one step for one training data entry. To ensure fair comparison, we use a single A6000 to run on all three methods.
Table 11: Inference speed for LLAMA 7B on MNLI-m
benchmark. We report the average time to complete inference for one evaluation data entry. To ensure fair comparison, we use a single A6000 to run on all three methods.
15

---

Published in Transactions on Machine Learning Research (01/2024)
C Hyperparamters Used in Experiments
C.1 LLaMA / OPT on SAMSum
We set up the training procedure following Hu et al. (2022), with particular accommodation our particular language models. For a fair comparison with the concurrent work QLoRA, we use the exact same hyperpa- rameter up shown in Table 12 We train using AdamW for 350 steps with batch size of 128 samples. We report the results over 3 random seeds; the result for each run is taken from the training steps with the lowest validation loss.
Dataset
Model
LLaMA 7B / 13B 30B 65B
OPT 7B/ 13B
/3BB
Optimizer
AdamW
Warmup Ratio
0.06
Batch size
128
Evaluation Batch size
16
SAMSum
Evaluation Steps
50
Total # Training Steps Learning Rate Schedule Learning Rate
350
Cosine
1e-3
WeightDecay
0.0
LoRAConfig
Tg 8 8
LoRA
32
Max Seq. Ten
250
Table 12: Hyperparamters configuration for ModuLoRA, Q-LoRA on SAMSum
C.2 LLaMA on Code- Code-Alpaca & Bext-Classificattin
We again train using AdamW optimizer with a warmup ratio of 0.06. We tune learning rate, batch size, training steps for each task. We report the results over random seeds. The result for each run is taken from the training steps that yield the lowest validation loss.
Dataset
LLaMA Model
13/30/65
Dataset
LLaMA Model
7/13/30/65 B
Optimizer
AdamW
Optimizer Warmup Ratio
AdamW
Warmup Ratio
0.06
0.06
Batch size
256
Batch size
128
Text- Classification
Evaluation Batch size
Code-
Evaluation Batch size
Evaluation Steps
100
Alpaca
Fvaluation Steps Total # Training Steps
40
Total # Training Steps
1000
120
Learning Rate Schedule
Cosine
Learning Rate Schedule
Linear
Learning Rate
1e-3
Learning Rate
1e-3
Decay
0.0
Weigh
0.0
LoRAConfig
rg Ty =
LoRAConfig
Tg = To =
LoRA
LoRA a
32
Max Seq. Len
128
Max Seq. Len
165
Table 13: Hyperparamters configuration for Modu- LoRA, Q-LoRA on Text-Classificatian
Table 14: Hyperparamters configuration for Modu- LoRA, Q-LoRA Alpaca-Code
16

---

Published in Transactions on Machine Learning Research (01/2024)
C.3 LLaMA MNLI-M
Training is conducted using the AdamW optimizer, with warmup ratio set at 0.06. We lune the learning rate, batch size, and training steps. Results are reported over three random seeds, and for each run, the performance metric derived from the training stop with the lowest validation loss. See Table for more details on the hyperparameters used.
Dataset
Model
LLaMA 7B 13B / 30B / 65B
Optimizer Warmup Ratio
AdamW
0.06
Batch size
128
Evaluation Batch size
64
MNLI-M
Evaluation Steps Total # Training Epoch
64
1.0
Learning Rate Schedule Learning Rate
Cosine
1e-3
WeightDecay
0.0
LoRAConfig
rq = Te =
LoRA a Max Seq. Len
32
128
Table 15: Hyperparamters configuration for ModuLoRA, Q-LoRA on MNLI-M
C.4 LLaMA Alpaca for BBH Evaluation
Training is conducted using the AdamW optimizer, with warmup ratio set at 0.06. We tune the learning rate, batch size, and training steps. Results are reported over three random seeds. See Table 16 for more details on the hyperparameters used.
Datasct
Model
LLaMA 7B / 13B 30B / 65B
Optimizer
AdamW
Warmup Ratio
0.06
Batch size
128
Total # Training Epochs
3
Alpaca
Learning Rate Schedule
Linear
Learning Rate
1e-3
Weight
0.0
LoRAConil Config
T4 my = 16
LoRA
Max Seq. Len
256
Table 16: Hyperparamters configuration for ModuLoRA on Alpaca
17


Reviews:
[{"review_id": "TF4c3RAji4", "reviewer": "Reviewer_RBFJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is easy to follow.\n2. The provided library could contribute to efficient LLM community.\n3. The experiments include various tasks and the results seem promising.\n\nWeakness:\n1. The novelty is limited. The main contribution of this work lies in the framework of fine-tuning with low precision layers, which can be treated as a simple combination of LORA and OPTQ.\n2. Lack of evaluation of latency in the experiments.\n3. There exist many typos in the paper and the authors are suggested to check them carefully. Part of minors are listed below:\n\n*  In abstract, natural language infernece -> natural language inference\n* In Section 2.2, The method iteretiavely runs -> The method iteratively runs\n* In Section 3.1, quantized weighs -> quantized weights", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an efficient framework to finetune large language models with less memory. Specifically, 3-bit OPTQ quantizer is utilized to all linear layers and then the weight quantizer is incorporated into low-rank adapters to enable a memory-efficient fine-tuning. The authors also provide a library for this framework with efficient CUDA implementations. The experiments on various LLM tasks show the effectiveness of the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please justify the novelty in case I miss some contributions.\n2. Please provide more evaluation of latency.\n3. Please examine and revise the typos.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fJaOj0jcyt", "reviewer": "Reviewer_E1B8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The proposed method is simple and effective to reduce GPU memory consumption for fine-tuning quantized LLM models.\n2. It is agnostic to the specific quantization method.\n3. The evaluation results shows convincing results that the 3bit model can provide similar accuracy to a fp16 model.\n\nWeaknesses:\n1. The recompilation introduces additional computational overhead.\n2. More direct comparisons of the memory usage during training should be provides.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of contributions\n1. The paper studies the problem of the quantization aware fine-tuning of the large language models, specifically dealing with the Low-rank adapter based fine-tuning of the models.\n2. The core contribution is proposing a recompilation methods in which the based weight matrix in LoRA is computed twice, one during the inference, one during the back-propagation. In this way, the full precision (or fp16) version of the based weight matrix can be freed after a layer and is not needed to be stored in GPU memory. \n3. The paper also releases an open-source tools called LLMTools for the quantization of LLMs and release several quantized popular models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. More insights on why the 4bits model of 13B Lamma and 30B lamma perform worse than 3bits. \n2. Since the number of training epochs/tokens are important for performance, the evaluation sections should provide more details on the training settings. Especially, the fine-tuning setting when comparing to QLoRA should be the same for fairness.\n3. It would be good to provide more details on the speed of finetuning, and show the trade off between the recomputation and longer training time.\n4. Quantitative results of the memory saving of the proposed method.", "broader_impact_concerns": "no broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IY6dln3czV", "reviewer": "Reviewer_eLwS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The topic of this work is of clear interest for ML practitioners\n\n- The paper is well structured, well written, very easy to follow\n\n- Solid results on various downstream tasks with models finetuned down to 3 bits: often comparable to their 8-bit counterpart, or showing limited degradation\n\n- ModuLoRA allows for a flexible choice of quantizers, leaving the path open for further improvements as new quantizers are developed\n\n- The authors present direct comparisons against the recently proposed QLoRA, arguably the low-precision finetuning method conceptually closest to ModuLoRA, and discuss differentiating factors. ModuLoRA results compare favorably with QLoRA\n\n- The shared library is a very valid tool to democratize LLM finetuning\n\nWeaknesses:\n\n- Quantization is applied solely to weights, therefore while achieving the goal of finetuning in memory-constrained environments, computational speed ups are not to be expected. This is partly acknowledged in the paper but it is unclear what is the extent of ModuLoRA's overheads. I would expect they may also depend on the choice of quantizer. How does finetuning runtime compares with LoRA and QLoRA? What about inference latency / throughput?\n\n- The paper mentions that in some scenario ModuLoRA materializes a memory advantage against QLoRA, however no evidence is provided to support this statement. In fact, there is no concrete mention of memory requirements for various models / quantization / batch size combinations in the paper, aside from some upper bounds (such as one of the main claims of being able to finetune a 65B model on a single 48G GPU). As this relates to the main takeaway of this paper, I recommend including a table comparing memory requirements, at the very least as an appendix.\n\n- Appendices A.1 and A.2 do not appear to add any information to the main text and should be either expanded or removed", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a technique that combines weight quantization with the LoRA parameter efficient finetuning strategy (ModuLoRA). The main target is to train LLM in memory-constrained hardware. ModuLoRA is quantizer-agnostic. Dequantized weights are not stored but recalculated during the backward pass to preserve the memory savings of the few-bit representation. ModuLoRA enables LLM finetuning at 3 and 4 bits, resulting in performance comparable to baseline. Results are presented primarily on the LLAMA model family, on several downstream tasks. The authors are sharing this work as part of a library (LLMTools).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discuss ModuLoRA overheads for finetuning and inference\n\n2. List and compare actual memory requirements of the presented models\n\n3. Rewrite, expand, or remove appendices A.1 and A.2", "broader_impact_concerns": "Some broader impact concerns are briefly addressed in the manuscript. No additional concern on my end.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following detailed rebuttals from the authors + the additional result on 2-bit quantization, the reviewers all lean toward accepting the paper.  A main point of discussion was the comparison with QLoRA in terms of performance and differentiation, but the reviewers were satisfied after the authors provided thorough additional datapoints, e.g. "In their rebuttal to mine as well as other reviewers' comments, the authors addressed my main concerns thoroughly, including in the manuscripts additional information on latency and memory requirements. The proposed ModuLoRA technique does shows some runtime overheads compared to its direct counterpart QLoRA (4 bit), but it brings more flexibility in the choice of quantizer and achieves better performance down to 3-bit precision.". Given the claims are well supported by the evidence + broad target audience of this paper, I recommend a solid accept.
Decision: Accept (poster)