QLoRA: Efficient Finetuning of Quantized LLMs
Authors: Tim Dettmers
Artidoro Pagnoni
Ari Holtzman
Luke Zettlemoyer
Venue: NeurIPS 2023 oral (2023)
Keywords: finetuning
llama
instructions
quantization
TL;DR: We develop a method that enabling the finetuning of a 65B model on a single GPU without performance degradation achieving ChatGPT-quaity results on the Vicuna benchmark.

Abstract: We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information-theoretically optimal for normally distributed weights (b) Double Quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) Paged Optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small, high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations, showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

QLORA: Efficient Finetuning of Quantized LLMs
Tim Dettmers"
Artidoro Pagnoni"
Ari Holtzman
Luke Zettlemoyer
University of Washington
(dettmers, artidoro, ahati, 1sz))cs. washington. edu
Abstract
We present QLORA, an efficient finetuning approach that reduces memory us- age enough to finetune 65B parameter model single 48GB GPU while preserving full 16-bit finetuning task performance. QLORA backpropagates gradi- ents through frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), new data type that is information theoretically optimal for normally distributed weights (b) Double Quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) Paged Optimizers to manage memory spikes. use QLORA to finetune more than 1,000 models, providing detailed analysis of instruction following and chatbot performance across instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finctuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning small high-quality dataset leads state-of-the soce---ftthe--tttt results, when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based both human and GPT-4 evaluations showing that GPT-4 evaluations are cheap and reasonable alternative to human evaluation. Fur- thermore, find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared ChatGPT, We release all of models and code, including CUDA kernels for 4-bit training.
Introduction
Finetuning large language models (LLMs) is highly effective way to improve their performance, [40, 63, 43, 62, 60, 37] and to add desirable or remove undesirable behaviors [43, 2, 4J. However, finetuning very large models prohibitively expensive; regular 16-bit finetuning of a LLaMA 65B parameter model [58] requires more than 780 GB of GPU memory. While recent quantization methods can reduce the memory footprint of L.LMs [14, 13, 18, 67], such techniques only work for inference and break down during training [66].
We demonstrate for the first time that it is possible to finetune quantized 4-bit model without any performance degradation. Our method, QLORA, uses a novel high-precision technique to quantize pretrained model to 4-bit, then adds small set of learnable Low-rank Adapter weights [28]
"qqual contribution. /nttps://github.co htt.://ww...ccorrrrroooo///ooooo
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

that are tuned by backpropagating gradients through the quantized weights.
Table 1:
Elo ratings for a competition between models, averaged for 10,000 random initial order- ings. The winner of match is determined by GPT-4 which declares which response is better for given prompt of the the Vicuna benchmark. 95% confidence intervals are shown (1), After GPT- 4, Guanaco 33B and 65B win the most matches, while Guanaco 13B scores better than Bard.
QLORA A reduces the average memory requirements of finetuning a 65B parameter model from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16- bit fully finetuned baseline. This marks significant shift in accessibility of LLM finetuning: now the largest publicly available models to date finetunable on a single GPU. Using QLoRA, we train the Gua- naco family of models, with the second best model reaching 97.8% of the performance level of ChatGPT on the Vicuna [10] benchmark, while being trainable in less than 12 hours on a single consumer GPU; using& single professional GPU over 24 hours we achieve 99.3% with our largest model, essentially closing the gap to ChatGPT on the Vieuna bench- mark. When deployed, our smallest Guanaco model (7B parameters) requires just GB of memory and outperforms a 26 GB Alpaca model by more than 20 percentage points on the Vicuna benchmark (Table 4).
Model
Size
Elo
GPT-
1348 +
Guanaco 65B
41 GB
1022
Guanaco 33B
21 GB
992 +
Vicuma 13B
26 GB
974 +
ChatGPT
966 +
Guanaco 13B
GB
916 +
Bard
902
Guanaco 7B
6 GB
879 +
QLORA introduces multiple innovations designed to reduce memory use without sacrificing per- formance: (1) 4-bit NormalFloat, an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats. (2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model). (3) Paged Optimizers, using NVIDIA unified memory to avoid the gradient checkpointing memory spikes that occur when processing a mini-batch with a long sequence length. We combine these contributions into better tuned A approach that includes adapters every network layer and thereby avoids almost all of the accuracy tradeoffs seen in prior work.
QLoRA's efficiency enables to perform an in-depth study of instruction finetuning and chatbot performance on model scales that would be impossible using regular finetuning due to memory overhead. Therefore, we train more than 1,000 models across several instruction tuning datasets, model architectures, and sizes between 80M to 65B parameters. In addition to showing that QLORA recovers 16-bit performance ($4) and training a ofate-of-the-art chatbot, Guanaco, ($5), we also analyze trends in the trained models. First, we find that data quality is far more important than dataset size, e.g., 9k sample dataset (OASSTI) outperformed a 450k sample dataset (FLAN v2, subsampled) on chatbot performance, even when both are meant to support instruction following generalization. Second, we show that strong Massive Multitask Language Understanding (MMLU) benchmark performance does not imply strong Vicuna chatbot benchmark performance and vice versa-in other words, dataset suitability matters more than size for a given task.
Furthermore, we also provide extensive analysis of chatbot performance that uses both human raters and GPT-4 for evaluation. We use tournament-style benchmarking where models compete against each other in matches to produce the best response for given prompt. The winner of match is judged by either GPT-4 or human annotators. The tournament results are aggregated into Elo scores [16, 17] which determine the ranking of chatbot performance. We find that GPT-4 and human evaluations largely agree on the rank of model performance in the tournaments, but we also find there are instances of strong disagreement. As such, we highlight that model-based evaluation while providing a cheap alternative to human-annotation also has its uncertainties.
We augment our chatbot benchmark results with a qualitative analysis of Guanaco models (see Ap- pendix F). Our analysis highlights success and failure cases that were not captured by the quantitative benchmarks.
We release all model generations with human and GPT-4 annotations to facilitate further study. We open-source our codebase and CUDA kernels and integrate our methods into the Hugging Face transformers stack [65], making them easily accessible to all. We release collection of adapters for 7/13/33/65B size models, trained on 8 different instruction following datasets, for total of 32 different open sourced, finetuned models.
2

---

Full Finetuning (No Adapters)
LoRA
QLoRA
Optimizer
State (32 bit)
Adapters (16 bit)
t
Model
Parameter Updates
Gradient Flow
16-bit Transformer
16-bit Transformer
4-bit Transformer
Raging
Figure 1: Different finetuning methods and their memory requirements. QLoRA improves over LoRA by quantizing the transformer model to 4-hit precision and using paged optimizers to handle memory spikes.
Background
Block-wise k-bit Quantization Quantization is the process of discretizing an input from a rep- resentation that holds more information to representation with less information. It often means taking a data type with more bits and converting it to fewer bits, for example from 32-bit floats to 8-bit Integers. To ensure that the entire range of the low-bit data type is used, the input data type is commonly rescaled into the target data type range through normalization by the absolute maximum of the input elements, which usually structured Floating Point (FP32) tensor into Int8 tensor with range 12,,,, 127]:
tensor. For example, quantizing 32-bit
127
x Imt8 round
X FP32
round FP32 X FP32).
(1)
absmax (AIImxXXXPPF) FP32
where e is the quantization constant quantization scale. Dequantization is the inverse:
X XIm8
( FP32
1mi8
FP32
FP32
The problem with this approach is that if a large magnitude value (i.e., an outlier) occurs in the input tensor, then the quantization bins-certal bit combinations-aae are not utilized well with few numbers quantized in some bins. To prevent the outlier issue, common approach is to chunk the input tensor into blocks that are independently quantized, each with their own quantization constant c. This can be formalized as follows: We chunk the input tensor X E g into n contiguous blocks of size B by flattening the input tensor and slicing the linear segment into n quantize these blocks independently with Equation 1 to create a quantized tensor and n quantization constants c;.
h)/ blocks. We
Low-rank Adapters Low-rank Adapter (LoRA) finetuning [28] is a method that reduces memory requirements by using small set of trainable parameters, often termed adapters, while not updating the full model parameters which remain fixed. Gradients during stochastic gradient descent are passed through the fixed pretrained model weights to the adapter, which is updated to optimize the loss function. LoRA augments a linear projection through an additional factorized projection. Given projection XW Y with X R W E hxx LoRA computes:
= + LX
where L E Rixrr and L2 E Rr and is scalar.
QLORA Finetuning
QLoRA A achieves high-fidelity 4-bit finetuning via two techniques we propose- 4-bit NormalFloat (NF4) quantization and Double Quantization. Additionally, introduce Paged Optimizers, to

---

prevent memory spikes during gradient checkpointing from causing out-of-memory errors that have traditionally made finetuning on single machine difficult for large models.
QLoRA A has one low-precision storage data type, in our case usually 4-bit, and one computation data type that is usually B6 In practice, this means whenever QL.RA A weight tensor is used, dequantize the tensor to BFloat 16, and then perform a matrix multiplication in 16-bit.
We now discuss the components of QLORA followed by formal definition of QLORA.
4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization [15, 46] which is an
optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. Quantile quantization works by estimating the quantile of the input tensor through the empirical cumulative distribution function.
The main limitation of quantile quantization is that the process of quantile estimation is expensive. Therefore fast quantile approximation algorithms, such as SRAM quantiles [15], are used to estimate them. Due to the approximate nature of these quantile estimation algorithms, the data type has large quantization errors for outliers, which are often the most important values.
Expensive quantile estimates and approximation errors can be avoided when input tensors come from a distribution fixed up to a quantization constant. In such cases, input tensors have the same quantiles making exact quantile estimation computationally feasible.
Since pretrained neural network weights usually have a zero-centered normal distribution with standard deviation (see Appendix D., we can transform all weights to single fixed distribution by scaling a such that the distribution fits exactly into the range of our data type. For our data type, we set the arbitrary range [-1, 1]. As such, both the quantiles for the data type and the neural network weights need to be normalized into this range.
The information theoretically optimal data type for zero-mean normal distributions with arbitrary standard deviations a in the range [-1, 1] is computed as follows: (1) estimate the 2* + 1 quantiles of theoretical N (0, 1) distribution to obtain a k-bit quantile quantization data type for normal distri- butions, (2) take this data type and normalize its values into the [-1, 1] range, (3) quantize an input
weight tensor by normalizing it into the [-1,1] range through absolute maximum rescaling.
Once the weight range and data type range match, we can quantize as usual. Step (3) is equivalent to rescaling the standard deviation of the weight tensor to match the standard deviation of the k-bit data type. More formally, we estimate the 2 values qi of the data type as follows:
1 (2xx q = 2
i+1
Qx
Qx
(4)
2 2++1
2 2++1)
where (x(+) is the quantile function of the standard normal distribution NO, 1). A problem for symmetric k-bit quantization is that this approach does not have an exact representation of zero, which is an important property to quantize padding and other zero-valued elements with no error. To ensure discrete zeropoint of and to use all 2* bits for a k-bit datatype, we create an asymmetric data type by estimating the quantiles 9i of two ranges q: 2 for the negative part and 2 + for the positive part and then we unify these sets of qi and remove one of the two zeros that occurs in both sets. We term the resulting data type that has equal expected number of values in each quantization bin k-bit NormalFloat (NFK), since the data type is zero-centered normally distributed data. A step-by-ster visualization of how to construct the NF4 data type and its exact values can be found in Appendix H.
thhorteiiitttorrtcclllll optimal for
Double Quantization We introduce Double Quantization (DQ). the process of quantizing the quantization constants for additional memory savings. While a small blocksize is required for precise 4-bit quantization [13], it also has considerable memory overhead. For example, using 32-bit constants and a blocksize of 64 for W, quantization constants add 32/64 = 0.5 bits per parameter on average. Double Quantization helps reduce the memory footprint of quantization constants.
More specifically, Double Quantization treats quantization constants c FP32 of the first quantization as inputs to second quantization. This second step yields the quantized quantization constants c5 and the second level of quantization constants e cf432 We use 8-bit Floats with a blocksize of 256 for the second quantization as no performance degradation is observed for 8-bit quantization, in line with results from Dettmers and Zettlemoyer [13]. Since the dF32 are positive, we subtract of
the mean from O2 before quantization to center the values around zero and make use of symmetric

---

quantization. On average, for a blocksize of 64, this quantization reduces the memory footprint per parameter from 32/64 = 0.5 bits, to 8/64 + 32/(64 256) 0.127 bits, a reduction of 0.373 bits per parameter.
Paged Optimizers use the NVIDIA unified memory feature which does automatic page-to-pagee transfers between the CPU and GPU for error-free GPU processing in the scenario where the GPU occasionally runs out-of-memory. The feature works like regular memory paging between CPU RAM and the disk. We use this feature to allocate paged memory for the optimizer states which are then automatically evicted to CPU RAM when the GPU runs out-of -ntt-ofrmmrry and paged back into GPU memory when the memory is needed in the optimizer update step.
QLORA. Using the components described above, we define QLORA for a single linear layer in the quantized base model with single LoRA adapter follows:
Y dorieedouueeeeuaaaa
bit W + X X BERr66,BB661..
BF16 BF16
(5)
doubleDequant(("" FP32 k-bit w = dequant(dequants (c1
FP32 J-hit),
W thit
BF16
(6)
We use NF4 for W and FP8 for cz. We use blocksize of 64 for W for higher quantization precision and a blocksize of 256 for 0g to conserve memory.
For parameter updates only the gradient with respect to the error for the adapters weights aE are ag However, the calculation of aE entails the calculation of ow aL,
aL; ax
needed, and not for 4-bit weights aw which proceeds via equation (5) with dequantization from storage W to computation data type W8F16 to calculate the derivative ax aw in BFloat16 precision.
To summarize, QLORA has one storage data type (usually 4-bit NormalFloat) and computation data type (16-bit BrainFloat). We dequantize the storage data type to the computation data type perform the forward and backward pass, but we only compute weight gradients for the LoRA parameters which use 16-bit FrainFloat.
4 QLoRA vs. Standard Finetuning
We have discussed how QLoRA works and how it can significantly reduce the required memory for finetuning models. The main question now is whether QLoRA can perform as well as full-model finetuning. Furthermore, we want to analyze the components of QLoRA including the impact of NormalFloat4 over standard Float4. The follow- ing sections will discuss the experiments that aimed at answering these questions.
4-bit LLaMA
10000800 accuracy
0.00
0.65
Mean 0
0.64
0.2
Pype
Experimental setup. We consider three architec- tures (encoder. encoder-decoder, and decoder only) and compare QLoRA with 16-bit adapter-finttunnng and with full- full-finctuning for models up to 3B. Our evaluations include GLUE [59] with RoBERTa- large [38], Caaper-Nauurlllnstuuttii
0.40
10
10
Total model bits
Figure 2: Mean zero-shot accuracy over Wino- grande, HellaSwag, PiQA, Arc-Easy. and Are- Challenge using LLaMA models with different 4-bit data types. The NormalFloat data type significantly improves the bit-for-bit accuracy gains compared to regular 4-bit Floats. While Double Quantization (DQ) only leads to minor gains, it allows for a more fine-grained control over the memory footprint to fit models of certain size (33B/65B) into certain GPUs (24/48GB).
[62] with T5 [50], and 5-shot MMLU [24] after finetuning LLaMA on Flan v2 [39] and Alpaca [56].
To additionally study the advantages of NF4 over other 4-bit data types, we use the setup of Dettmers and Zettlemoyer [13] and measure post-quantization zero-shot accuracy and perplexity across different models (OPT [73]. LLaMA [58].
BLOOM [53], Pythia (7]) for model sizes 125m 13B. We provide more details in the results section for each particular setup to make the results more readable. Full details in Appendix C.
While paged optimizers are critical to do 33B/65B QLORA tuning on a single 24/48GB GPU, we do not provide hard measurements for Paged Optimizers since the paging only occurs when processing
trtpp://doce... aridia. con/cud/cuda- programiing guide
5

---

Table 3: Mean 5-shot MMLU test accuracy for LLaMA 7-6SB models finetuned with adapters on Alpaca and FLAN v2 for different data types. Overall, NE4 with double quantization (DQ) matches BF performance, while FP4 is consistently one percentage point behind both.
Mean 5-shot MMLU Accuracy
LLaMA Size
7B
13B
33B
65B
Mean
Dataset
Alpaca
FI.AN v2
Alpaca
FLANN v2
Alpaca
FLAN v2
Alpaca
FLAN v2
BFloat16
38.4
45.6
47.2
50.6
57.7
60.5
61.8
62.5
53.0
Float4
37.2
44.0
47.3
50.0
55.9
58.5
61.3
63.3
52.2
NFloat4 DQ
39.0
44.5
47.5
50.7
57.3
59.2
61.8
63.9
53.1
mini-batches with long sequence lengths, which is rare. We do, however, perform an analysis of the runtime of paged optimizers for 65B models on 48GB GPUs and find that with batch size of 16, paged optimizers provide the same training speed as regular optimizers. Future work should measure and characterize under what circumstances slow-downs occur from the paging process.
4-bit NormalFloat yields better performance than 4-bit Float- ing Point While the 4-hit NormalFloat (NF4) data type is information-theorrtiee.
Table 2: Pile Common Crawl mean perplexity for different data types for 125M 13B OPT, BLOOM, LLaMA, and Pythia models.
optimal, it still needs to be determined if this property translates empirical advantages. We follow the setup from Dettmers and Zettlemoyer [13] where quantized LLMs (OPT [73], BLOOM [53]. Pythia [7], LLaMA) of different sizes (125M to 65B) with different data types are evaluated on language modeling and set of zero-shot tasks. In Figure and Table 2 we see that NF4 improves performance significantly over FP4 and Int4 and that double quantization reduces the memory footprint without degrading performance.
Data type
Mean PPL
Int4
34.34
Float4 (E2MI)
31.07
Float4 (E3M0)
29.48
NFloat4 + DQ
27.41
k-bit QLORA matches 16-bit full finetuning and 16-bit LoRA performance Recent findings
have established that 4-bit quantization for inference is possible, but leads to performance degradation relative to 16-bit [13, 18]. This raises the crucial question of whether the lost performance can be recovered by conducting 4-bit adapter finetuning. We test this for two setups.
The first setup test if QLoRA can replicate 16-bit full finetuning performance for T5 and RoBERTa model finetuning. These results are detailed in the Appendix C.I. For our second setup, since full finetuning models at and beyond IB parameters requires more than one server of high memory GPUs, we continue to test whether 4-bit QLORA can match 16-bit LoRA at the 7B to 65B parameter scales. To this end, finetune LLaMA 7B through 65B on two instruction following datasets, Alpaca and FLAN v2, and evaluate on the MMLU benchmark via 5-shot accuracy. Results are shown in Table where performance. In addition, we also note that QLORA with FP4 lags behind the 16-bit brain float LoRA baseline by about 1 percentage point. This corroborates both our findings that (1) QLORA with NF4 replicates both 16-bit full finetuning and 16-bit LoRA finetuning performance, and (2) NF4 superior to FP4 in terms of quantization precision.
that NF4 with double quantization fully recovers the 16-bit LoRA MMLU
Summary Our results consistently show that 4-bit QLORA with NF4 data type matches 16- bit full finetuning and 16- LoRA finetuning performance on academic benchmarks with well- established evaluation setups. We have also shown that NF4 is more effective than FP4 and that double quantization does not degrade performance. Combined, this forms compelling evidence that 4-bit QLORA tuning reliably yields results matching 16-bit methods.
In line with previous work on quantization [13]. our MMLU and Elo results indicate that with a given finetuning and inference resource budget it beneficial to increase the number of parameters the base model while decreasing their precision. This highlights the importance of efficiency benefits from QLORA. Since we did not observe performance degradation compared to full-finetning in our experiments with 4-bit finetuning, this raises the question of where the performance-preciste trade-off exactly lies for QLoRA tuning, which we leave to future work to explore.
We proceed to investigate instruction tuning at scales that would be impossible to explore with full 16-bit finetuning academic research hardware.

---

5 Pushing the Chatbot State-of-the-art with QLoRA
Having established that 4-bit QLORA matches 16-bit performance across scales, tasks, and datasets conduct an in-depth study of instruction finetuning up to the largest open-source language models available for research. To assess the performance of instruction finetuning these models, we evaluate on a challenging Natural Language Understanding benchmark (MMLU) and develop new methods for real-world chatbot performance evaluation. For a more qualitative analysis, see Appendix F.
5.1 Experimental setup
We now describe an overview of the experimental setup with full details in Appendix D.
Data As, our knowledge, there is no comprehensive study of instruction-following datasets, we select eight recent datasets. We include datasets obtained through crowd-soureing (OASSTI [31], HH-RLHF [4]), distillation from instruction-tune
models (Alpaca [56], self-instruct [60], unnatural- instructions [26]), corpora aggregations (FLAN v2 [12]). as well as hybrids (Chip2 [32], Long-
form [30]). These datasets cover different languages, data sizes, and licenses.
Training Setup To avoid confounding effects from different training objectives, we perform QLoRA finetuning with cross-entropy loss (supervised learning) without reinforcement learning, even for datasets that include human judgments of different responses. For datasets that have clear distinction between instruction and response, we finetune only on the response (see ablations in Appendix D). For OASST1 and HH-RLHF, multiple responses are available. We then select the top response at every level of the conversation tree and finetune on the full selected conversation, including the instructions. In all of our experiments, we use NF4 QLORA A with double quantization and paged optimizers to prevent memory spikes during gradient checkpointing. We do small hyperparameter searches for the 13B and 33B LLaMA models and we find that all hyperparameter settings found at 7B generalize (including number of epochs) except learning rate and batch size. We halve the learning rate for 33B and 65B while doubling the batch size.
Baselines We compare models to both research (Vicuma [10] and Open Assistant [31]) and commercial (GPT-4 [42], GPT-3.5-turbe and Bard) chatbot systems. The Open Assistant model is LLaMA 33B model finetuned with Reinforcement Learning from Human Feedback (RLHF) on the OASSTI dataset. Vicuna does full fine-tuning of LLaMA 13B on proprietary user-shared conversations from ShareGPT and is thus the result of distillation from OpenAl GPT models.
Evaluation setup Chatbot evaluation is not straightforward since each prompt has many high- quality responses which are challenging to rank. We therefore follow a comprehensive approach, which involves (a) standard benchmarks that measures general language understanding performance (MMLU [24]), (b) both automatic and human evaluation that measures how many responses from chatbot A are better compared to chatbot B, (c) round-robin tournament-style evaluation where chatbots compete against each other in games where chatbot performance is measured as ELO. We provide a detailed discussion of these evaluation setups in the Appendix E.
5.2
Guanaco: QLORA trained OASST1 is State-of-the-art Chathot
Based on our automated and human evaluations, we find that the top QLORA tuned model, Guanaco 65B, which we finetune on variant of OASSTI, the best-performing open-source chathot model and offers performance competitive to ChatGPT, When compared to GPT-4, Guanaco 65B and 33B have an expected win probability of 30%, based on Elo rating from human annotators sestem-level pairwise comparisons on the Vicuna benchmark the highest reported to date.
The Vicuna benchmark [10] results relative ChatGPT are shown in Table We find that Guanaco 65B is the best-performing model after GPT-4, achieving 99.3% performance relative to ChatGPT. Guanaco 33B has more parameters than the Vicuna 13B model, but uses only 4-bit precision for its weights and is thus much more memory efficient at 21 GB vs 26 GB, providing three percentage points of improvement over Vicuna 13B. Furthermore, Guanaco 7B easily fits on modern phones at a 5 GB footprint while still scoring nearly 20 percentage points higher than Alpaca 13B.
However, Table 4 also has very wide confidence intervals, with many models overlapping in per- formance. We hypothesize that this uncertainty comes from the lack of clear specification of scale, e.g., it unclear what 8 on 10 point scale means across different scenarios. As such, instead recommend using the Elo ranking method [16], based on pairwise judgments from human annotators and GPT-4 avoid the problem of grounding an absolute scale.
7

---

Table 4: Zero-shot Vicuna benchmark
percentage of the obtained by ChatGPT evaluated by GPT-4. We see that OASSTI models perform close ChatGPT despite being trained on a very small dataset and having fraction of the memory requirement of baseline models.
Model / Dataset
Params
Model bits
Memory
ChatGPT vs Sys
Sys vs CharGPT
Mean
95% C1
GPT-4
119.4%
110.1% 96.4%
114.5%
2.6%
Bard
93.2%
94.8%
4.1%
Guanaco
65B
4-hit
41 GB
96.7%
101.9%
99.3%
4.4%
Alpaca
65B
4-bit
41 GB
63.0%
77.9%
70.7%
4.3%
FLAN v2
65B
4-bit
41 GB
37.0%
59.6%
48.4%
4.6%
Guanaco
33B
4-bit
21 GB
96.5%
99.2%
97.8%
4.4%
Open Assistant
33B
16-bit
66 GB
73.4%
85.7%
78.
5.3%
Alpaca
33B
4-bit
21 GB
67.2%
79.7%
73.6%
4,2%
FLAN v2
33B
4-bit
GB
26.3%
49.7%
38.0%
3,9%
Vicuna
13B
16-bit
26 GB
91.2%
78.7%
94.9%
4,5%
Guanaco
13B
4-bit
10 GB
87.3%
93.4%
90.4%
5 5,2%
Alpaca
13B
4-bit
10 GB
63.8%
76.7%
69.4%
4.2%
HH-RLHF
13B
4-bit
10 GB
55.5%
69.1%
62.5%
4.7%
Unnatural Instr.
13B
4-bit
10 GB
50.6%
69.8%
60.5%
4.2%
Chip2
13B
4-bit
10 GB
49.2%
69.3%
59.5%
4.7%
Longform
13B
4-hit
10 GB
44.9%
62.0%
53.6%
5,2%
Self-Instruct
13B
4-hit
10 GB
38.0%
60.5%
49.1% %
4.6%
FLAN v2
13B
4-hit
10 GB
32.4%
61.2%
47.0%
3.6%
Guanaco
4-bit
GB
84.1%
89.8%
87.0%
5.4%
Alpaca
7B
4-bit
GB
57.3%
71.2%
64.4%
5.0%
FLAN v2
7B
4-hit
5 GB
33.3%
56.1%
44.8%
4.0%
Table 5: MMLU S-shot test results for different sizes of LLaMA finetuned the corresponding datasets using QLoRA.
Elo ratings of the most competitive models can be seen in Table We note that human and GPT-4 ranking of models the Vicuna benchmark disagree partially, particularly for Guanaco 7B, but are consistent for most models with Kendall Tau of 7 0.43 and Spearman rank correlation of r 0.55 at the system level. At the example level, the agreement between GPT-4 and human annotators' majority vote is weaker with Fleiss 0.25. Overall, this shows moderate agreement between system-level judgments by GPT-4 and human annotators, and thus that model-based evaluation rep- resents a somewhat reliable alternative to human evalu- ation. See Section for further considerations.
Dataset
7B
13B
33B
65B
LLaMA no tuning
35.1
46,9
57.8
63.4
Self-Instruce Longforn
36.4
33.3
$3.0
$6.7
32.1
43.2
56.6
59.7
Chip2
34.5
41.6
59,8
HH-RLHF
34.9
44.6
60.1
Unnatural Instruct
41.9
48.1
57.3
61.3
Guanaco (OASSTI)
36.6
46.4
57.0
62.2
Alpaca
38.8
47.8
57.3
62.5
FLAN v2
51.4
63.9
Elo rankings in Table 6 indicate that Guanaco 33B and 65B models outperform all models besides GPT-4 the Vicuna and OA benchmarks and that they perform comparably ChatGPT in line with Table 4. We note that the Vicuna benchmark favors open-source models while the larger OA benchmark favors ChatGPT. Furthermore, finetuning dataset is determining factor in performance. Finetuning LLaMA models on FLAN v2 does particularly well on MMLU, but performs worst the Vicuna benchmark (similar trends are observed with other models). This also points to partial orthogonality in current evaluation benchmarks: strong MMLU performance does not imply strong chatbot performance (as measured by Vicuna or OA benchmarks) and vice versa.
can see from Tables 5 and 4 that the suitability of
Guanaco the only top model in our evaluation that is not trained on proprietary data the OASSTI dataset collection guidelines explicitly forbid the use of GPT models. The next best model trained on only open- source data is the Anthropic HH-RLHF model, which scores 30 percentage points lower than Guanaco on the Vicuna benchmark (see Table 4). Overall, these results show that b-tit QLORA effective and can produce state-of-the-art chatbots that rival ChatGPT. Furthermore, our 33B Guanaco can be trained on 24 GB consumer GPUs in less than 12 hours. This opens up the potential for future work via QLORA A tuning on specialized open-source data, which produces models that can compete with the very best commercial models that exist today.

---

Table Elo rating for tournament between models where models compete generate the best response for prompt, judged by human raters or GPT-4. Overall, Guanaco 65B and 33B tend to be preferred to ChatGPT-3.5 on the benchmarks studied. According to human raters they have a Each 10-point difference in Elo is approximately difference of 1.5% in win-rate.
Benchmark
Vicuna 80
Vicuna 80
Open Assistant
# Prompts
953
Judge
Human raters
GPT-4
GPT-4
Median Rank
Model
Elo
Rank
Elo
Rank
Elo
Rank
GPT-4
1176
1348
1294
1
Guanaco-65B
1023
1022
1008
2
Guanaco-33B
1009
4
992
1002
4
4
ChatGPT-3.5 Turbo
916
966
1015
5
Vicuna- 13B
984
5
974
936
5
5
Guanaco- 13B
975
913
885
6
Guanaco-79 7B
1010
3
879
8
860
7
7
Bard
909
902
8
Related Work
Quantization of Large Language Models Quantization of LLMs has largely focused on quanti- zation for inference time. Major approaches for preserving 16-bit L.LM quality focus managing outlier features (e.g., SmoothQuant [67] and L..M.in88)) [14]) while others use more sophisticated grouping methods [44, 70]. Lossy quantization approaches study the trade-offs for regular round- ing [13, 72, 48] or how to optimize rounding decisions to improve quantization precision [18]. Besides our work, SwitchBack layers [66] is the only work that studies backpropagation through quantized weights at scale beyond IB parameters.
Finetuning with Adapters While we use Low-rank Adapters [28] (LoRA), many other Parameter
Efficient Funing (PEFT) methods have been proposed such as prompt tuning [49, 33, 34], tuning the embedding layer inputs [1], tuning hidden states (IA3) [37], adding full layers [27]. tuning biases [71], learning a mask over weights based on Fisher information [55], and a combination of
approaches [23]. In our work, we show that LoRA adapters able to reach full 16-bit finetuning performance. We leave it to future work to explore the tradeoffs of other PEFT approaches.
Instruction Finetuning To help pretrained LLM follow the instructions provided in a prompt, instruction finetuning uses input-output pairs of various data sources to finetune a pretrained LLM to generate the output given the input as prompt. Approaches and datasets include MetalCI. [40],
MetaTuning [74], InstructGPT [43], FLAN [63, 12], PromptSource [3], Superr [62,
S1J, Self-instruct [60], Unnaturallnsttuutionn [26], OPT-IML [29], UnifiedSKG[68), OIG/Chip2 [32],
Alpaca [56], Vicuna [10], Koala [20], and Self Self-instruct-GPT- [45].
Chatbots Many instruction following models are structured as dialogue-based chatbots, often using Reinforcement Learning from Human Feedback (RLHF) [1]] or generating data from an existing model to train with Al model feedback (RLAIF) [5]. Approaches and datasets include Anthropic- HH [2, 4], Open Assistant [31]. LaMDA [57], and Sparrow [21]. We do not use reinforcement learning, but our best model, Guanaco, is finetuned on multi-turn chat interactions from the Open Assistant dataset which was designed to be used for RLHF training [31]. For the evaluation of chatbots approaches that use GPT-4 instead of costly human annotation have been developed [10, 45]. We improve on such approaches with a focus on an evaluation setup that more reliable.
7 Limitations and Discussion
We have shown evidence that our method, QLORA, can replicate 16-bit full finetuning performance with 4-bit base model and Low-rank Adapters. However, due to the immense resource cost, not establish that QLORA can match 16-bit finetuning performance at 33B and 65B scales.
Another limitation is the evaluation of instruction finetuning models. While we provide evaluations on MMLU, the Vicuna/OA benchmark, we did not evaluate on BigBench, RAFT, and HELM, and

---

is not ensured that our evaluations generalize to these benchmarks. On the other hand, we perform a very broad study on MMLU and develop new methods for evaluating chatbots.
From the evidence presented, it appears that the performance of these benchmarks likely depends how similar the finetuning data is to the benchmark dataset. For example, FLAN v2 is similar to MMLU, but dissimilar to the Vicuna benchmark and vice versa for the Chip2 dataset. This highlights that not only better benchmarks and evaluation is needed, but that one needs to be careful about what one is evaluating in the first place. Do we want to create models that do well on classroom highschool and colleague knowledge or do we want do well on chatbot conversation ability? Maybe something else? Because it is always easier to evaluate on an existing benchmark compared to creating new one, certain benchmarks can steer the community towards a certain direction. We should ensure as a community that the benchmarks measure what we care about.
An additional limitation is that we did not evaluate different bit-precisions or different adapter methods. Besides LoRA, there is also a wide variety Parameter Efficient Fine'Tuning (PEFT) methods that have been shown to work well. However, We used LoRA as many results established its robustness but other adapters might yield better performance. Since finetuning after quantization seems recover most of the information that is lost during quantization this might enable much more aggressive quantization. For example, 3-bit GPTQ quantization might also yield 16-bit full finetuning performance after finetuning.
unclear if these methods scale to large models.
8 Broader Impacts
Our QLORA finetuning method is the first method that enables the finetuning of 33B parameter models on single consumer GPU and 65B parameter models on single professional GPU, while not degrading performance relative to a full finetuning baseline. We have demonstrated that our best 33B model trained on the Open Assistant dataset can rival ChatGPT on the Vicuna benchmark. Since instruction finetuming is an essential tool to transform raw pretrained LLMs into ChatGPT-like chatbots, we believe that our method will make finetuning widespread and common in particular for the researchers that have the least resources a big win for the accessibility of state of the art NLP equalizing factor that helps to close the resource gap between
technology. QLoRA A can be seen large corporations and small teams with consumer GPUs.
Another potential source of impact is deployment to mobile phones and other low resources devices. While 7B models were shown to be able to be run on phones before, QLORA is the first method that would enable the finetuning of such models. We estimate that with an iPhone 12 Plus, QLORA can finetune 3 million tokens per night while the phone is charging. QLoRA can help enable privacy preservir usage of LLMs, where users can own and manage their own data and models, while simultancously making LLMs easier to deploy.
Furthermore, because of the increased inference efficiency of 4-bit models, if QLoRA models are deployed they reduce the environmental impact that LLMS have when deployed for personal use. We estimate that if 50% of deployments are personal and 50% are company deployments, QLORA deployments could reduce the overall carbon footprint by 72% (see Appendix B).
However, finetuning is dual-use technology that can be abused to cause harm. Widespread use of LLMs has known dangers [8, 6]. but we believe that equalizing access a technology that is quickly becoming ubiquitous will allow for better more independent analysis than keeping the power of LLMs in the hands of large corporations that do not release models or source code for auditing.
All in all, we believe that QLORA will have a broadly positive impact making the finetuning of high quality LLMs much more widely and easily accessible.
A Acknowledgements
We thank Aditya Kusupati, Ofir Press, Ashish Sharma, Margaret Li, Raphael Olivier, Zihao Ye, and Evangelia Spiliopoulou for their valuable feedback. We thank Norah Altriri for providing us with an intuitive NF4 visualization. We thank Felix Petersen for discussion on distributional quamtization and its relationship to quantile quantization. Our research was facilitated by the advanced computational, storage, and networking infrastructure of the Hyak supercomputer system at the University of Washington. We thank the Hyak team for ensuring a smooth operation. We thank the beta testers of the bitsandbytes library, in particular Alex Birch and Alyssa Vance. We thank Younes Belkada for help with the integration of our software into the Hugging Face transformers stack.
10

---

References
[1] An, Li, Z. Lin, Q. Liu, Chen, Q. Fu, W. Chen, Zheng, and J.-G. Lou. Input-tuning: Adapting unfamiliar inputs to frozen pretrained models. arXiv preprint arXiv:2203.0311 2022.
[2] A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, al. A general language assistant laboratory for alignment. arXiv preprint arXiv:2112.0066,, 2021.
[3] S. H. Bach, V. Sanh, Z.-X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279) 2022.
[4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint vrXiv:2204005800 2022.
[5] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
[6] M. Bender, Gebru, A. McMillan-Major and Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.
[7] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O' Brien, E. Hallahan, M. A. Khan, Purohit, U. S. Prashanth, E. Raff, al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01377. 2023.
[8] R. Bommasani, D. Hudson, Adeli, Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
[9] T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 16.1160046671. 06744, 2016.
[10] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. Gonzalez, I. Stoica, and E. P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:://// attps::/mmsys orggblogg/008-03-ooor
org/blog////oooooooooooooooooooo 221///2222--3333 1is00vvicuu vicuna/.
[11] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[12] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, Wang, M. De- hghani, S. Brahma, et al. Scaling instraction-finetuneed language models. arXiv preprint vrXiv:2210.114166 2022.
[13] T. Dettmers and L., Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. arXiv preprint arXiv:22 2022.
[14] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.in80): 8-bit matrix mulúplicationn for transformers at scale. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, 2022.
[15] T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer. 8-bit optimizers via biock-wise quantization. 9th International Conference on Learning Representationss ICLR, 2022.
[16] A. E. Elo. The proposed uscf rating system. its development, theory, and applications. Chess Life, 22(8):242 1967.
[17] A. E. Elo. The rating of chessplayers, past and present. Arco Pub., 1978.
11

---

[18] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq; Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
[19] Fu, S.-K. Ng. Z. Jiang, and P. Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 2023.
[20] X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023, URL https:/t berkeley. httu/blog/0233/000 00/////4//3330 04//////////////////////////////
[21] A. Glaese, N. McAleese, M. Trębacz, Aslanides, Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
[22] S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith. Annotation artifacts in natural language inference data. arXiv preprint 1X03::8003.02222 2018.
[23] Henderson, S. Ruder, et al. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, 2021.
[24] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Mea- suring massive multitask language understanding. In International Conference on Learning Representations, 2020.
[25] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
[26] O. Honovich, T. Scialom, O. Levy, and T. Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:22 2022.
[27] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799, PMLR, 2019.
[28] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation large language models. arXiv preprint arXiv:2 arXiv:2106.00663 2021.
[29] S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. 12017, 2022.
[30] A. Köksal, T. Schick, A. Korhonen, and H. Schüttze Longform: Optimizing instruction tuning for long text generation with corpus extraction. arXiv preprint arXiv:2304.08460, 2023.
[31] A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z.-R. Tam, K. Stevens, A. Barhoum, N. M. Duc, O. Stanley, R. Nagyfi, et al. Openassistant model alignment. arXiv preprint 27X1v:2330..03327 2023.
large language
[32] LAION.
Open-instruction-generalll dataset. https:///ithub.ccmm/t Open- InstractiionnGeerralss 2023.
[33] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficien prompt tuning, arXiv preprint 1rXiv:2104.08693 2021.
[34] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00199 2021.
[35] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, al. Holistic evaluation of language models. arXiv preprint
arXiv:2211.09110, 2022.
[36] T. Liao, R. Taori, I. D. Raji, and L. Schmidt. Are we learning yet? meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
12

---

[37] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1966 1965, 2022.
[38] Y. Liu, M. Ou, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zeulemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 16692, 2019.
[39] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. Le, B. Zoph. J. Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688, 2023.
[40] S. Min, M. Lewis, L. Zettlemoyer. and H. Hajishirzi. Metaicl: Learning to learn in context. arXiv preprini arXiv: 15.9443, 2021.
[41] A. Nematzadeh, K. Burns, E. Grant, A. Gopmik, and T. Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages
2018.
[42] OpenAl. Gpt-4 technical report. arXiv, 2023.
[43] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 11277730227442 2022.
[44] G. Park, B. Park, S. J. Kwon, B. Kim, Y. Lee, and D. Lee. nuqmm: Quantized matmul for efficient inference of large-scale scale generative language models. arXiv preprint 095557 2022.
[45] B. Peng, C. Li, He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
[46] F. Petersen and T. Sutter. Distributional Quantization. 2023. URL https://///tthh com/ MelliixPttersen//ee
[47] A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme. Hypothesis only baselines in natural language inference. Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, 2018.
[48] R. Pope, S. Douglas, Chowdhery, J. Devlin, Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint 1:Xiv:2211.05102 2022.
[49] G. Qin and J. Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2 065999 2021.
[50] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
[51] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Seao, A. Raja, al. Multitask prompted training enables zero-shot task generalization. 2021.
arXiv preprint 2rXiv:21/000000
[52] M. Sap, LeBras, D. Fried, and Y. Choi. Neural tf on the limits of social intelligence in large lms. arXiv preprint arXiv:2210. 13312, 2022.
[53] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05/000 2022.
[54] Shaphiro and M. Wilk, An analysis of variance test for normality. Biometrika, 52(3)591-611, 1965.
13

---

[55] Y.-L. Sung, V. Nair, and C. A. Raffel. Training neural networks with fixed sparse masks. Advances Neural Information Processing Systems, 34:24193-24205, 2021.
[56] R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instructionffollowing llama model. httts://githhb. wtw/tatsu-eooooooeetoaaoooesattt lab/ stanford alpaca, 2023.
[57] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Du, et al. Lamda: Language models for dialog applications. arXiv preprini arXiv:2201.08239 2022.
[58] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv 2023.
preprint arXiv:2302.13397
[59] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07660 2018.
[60] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arX5::2212.10560, 2022.
[61] Y. Wang, S. Mishra, P. Alipoormolabashi, Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, et al. buppr--ttuuannssuuuioosseeeeaaat instructions on 1600+ tasks. In EMNLP, 2022.
via declarative
[62] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super- anperrnaturalinsttuctions Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 5085-5109, 2022.
[63] Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint
2021.
[64] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
[65] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's 's transformers: State-of the-thtert natural language processing. arXiv preprint arXiv: 1911v:1910.03771,,220 2019.
[66] M. Wortsman, T. Dettmers, L.. Zettlemoyer, A. Morcos, A. Farhadi, and L. Schmidt. Stable and low-precision training for large-scale vision- language models. arXiv preprint ar 1311,3330,.3311, 2023.
[67] G. Xiao, J. Lin, M. Seznee, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint 1xXxx:22111104433 2022.
[68] T. Xie. C. H. Wu, P. Shi, R. Zhong, T. Scholak, M. Yasunaga, C.-S. Wu, M. Zhong, P. Yin, S. Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint urXiv:2201.05960 2022.
[69] Z. Yang, P. Qi, Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi- hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369- 23809 2018.
[70] Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.0/880 0/62000000666... 2022.
[71] F. Zaken, S. Ravfogel, and Y. Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-basse masked language-modeh, arXiv preprint arXiv:2106.100) 10199, 2021.
14

---

[72] A. Zeng. X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng. X. Xia, et al. Glm- 13m-:30b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
[73] Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, Chen, C. Dewan, M. Diab, X. Li, X. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00000 20000000.. 2022.
[74] R. Zhong, K. Lee, Z. Zhang, and D. Klein. Adapting language models for zero-shot learning by meta-tuning dataset and prompt collections. arXiv preprint arXiv:2104.04677 2021.
15

---

4.5
4.0
3.5
Arch
Speedup
3.0
A40 / RTX 3090
A100
2.5
RTX6000
RTX 2080 Ti
RTX 4090
2.0
1,5
1.0
18
138
308
65B
Params
Figure Speedups NF4 inference batch size compared 16-bit inference for different GPUs. We see that RTX 3090/4090 and A40 GPUs have large speedups 2.9-4.0x while other GPUs have speedups in the range 1.1-1.5x. The difference between GPUs is mostly caused by poor instruction throughput bottlenecks.
A Inference Speedups for NF4
The speedups for single batch inference for NF4 compare 16-bit float are shown in Figure 3.
B Environmental Impact and Carbon Footprint Reductions
The relationship between serving efficiency and environmental impact and cost/performance is complicated one. The short answer is, for personal use, we reduce the environmental impact per token by about 3.5x. If 50% LLM deployments are personal and 50% company, we reduce the environmental footprint of inference by 72%.
The long answer is this: For deployment, there are options: (1) personal deployment, (2) deployment by companies for many users. (1) uses small batch sizes (usually batch size =1), (2) uses large batch sizes (64-128). Per token, (2) offers about >50x better efficiency because for every weight that is loaded from memory, up to 64-128 tokens can be calculated. The >50x improvement in efficiency stems from the fact memory operations are energy inefficient, and floating point operations are energy efficient. As such,company deployment (2) is very environmentally friendly and cheap approach.
Currently, we have efficient 4-bit CUDA kernels for the personal deployment scenario (1) with batch size=1 which about 3.5x more efficient (see Figure 3). As such, reduce the overall footprint for personal deployment (1) significantly.
The overall cost and footprint is determined by how many people (1) (2). For example, if 50% of users use personal deployment (1) and 50% company deployments (2) and we assume that (2) is about 50x more efficient then we get the following numbers. Approach (1) accounts for 50%/(50% + 50%/550 = 98% of environmental impact. This means, the more users deploy personal LLMs, the larger benefit of our method. If 50% of people personally deployed LLMs (phones, laptops etc)
then our method will reduce the impact by about (98% 72%.
The environmental impact reduction is roughly proportional to energy consumption token pro- cessed which is roughly proportional the cost of running L.MSs. As such, the bottom line affected in the same proportions.
16

---

Table Experiments comparing 16-bit BrainFloat (BF16), 8-bit Integer (In18), 4-bit Float (FP4), and bit NormalFloat (NF4) on GLUE and Super-Nauurallnssruction.. QLORA replicates 16-bit LoRA and full- finetuning.
Dataset
GLUE (Acc.) RoBERTa-large
Super-Naaurallnsrrccioons (Rougel.)
Model
T5-80M
T5-250M
T5-780M
T5-3B
T5-11B 1BB
BF16
88.6
40.1
42.1
48.0
54.3
62.0
BFI6 replication
88.6
40.0
42.2
47.3
54.9
LoRA BF16
88.8
40.5
42.6
47.1
55.4
60.7
QLORA Int8
88.8
40.4
42.9
45.4
56.5
60.7
QLoRA FP4
88.6
40.3
42.4
47.5
55.6
60.9
QLORA NF4 DQ
40.4
42.7
47.7
55.3
60.9
Overall, our method will have a strong impact on environmental impact and cost reduction for personal LLM deployments.
C QLoRA vs Standard Finetuning Experimental Setup Details
C.1
BF16 vs NF4 for T5/RoBERTa
Here we detail additional results where we rum experiments to confirm we can replicate 16-bit full finetuning performance when we use QLoRA. We test RoBERTA and T5 models sized 125M 3B parameters on GLUE and the Superr-aaturallnsttutiins dataset. Results are shown in Table 7. In both datasets, observe that 16-bit, 8-bit, and 4-bit adapter methods replicate the performance of the fully finetuned 16-bit baseline. This suggests that the performance lost due to the imprecise quantization can be fully recovered through adapter finetuning after quantization.
C.2 Hyperparameter search for QLORA
We do a hyperparameter search for LoRA over the following variables: LoRA dropout { 0.0, 0.05, 0.1 LoRA r 8, 16, 32, 64, 128, 256), LoRA layers (key+query, all attention layers, all FFN layers, all layers, attention FFN output layers). We keep LoRA a fixed and search the learning rate, since LoRA is always proportional to the learning rate.
We find that LoRA dropout 0.05 is useful for small models (7B, 13B), but not for larger models (33B, 65B). We find LoRA is unrelated to final performance if LoRA is used all layers can be in Figure 4
65.0
64.8
Rougel
66.6
64.0
Figure 4: LoRA for LLaMA 7B models finetuned Alpaca. Each dot represents combination of hyperparameters and for each LoRA we rum random seed with each hyperparameter combination. The performance of specific LoRA r values appears to independent of other hyperparameters.
C.3 Super-Natural Instructions Experimental Setup Details
We use the same preprocessing of the Super-Natural Instruction dataset Wang et al. [61]. However, we split the training data in training and validation datasets allowing us to perform more rigorous hyperparameter tuning and early stopping. We use the same hyperparameters described in the paper
17

---

for training the various T5 model sizes on the Super-Natural Instruction data, We use LoRA r = 16 for small, medium, and large T5 models and LoRA r 64 for T5 xl and xxl models. We also use LoRA 64 in all our experiments and no LoRA dropout.
D Training State-of-the-art Chatbot Experimental Setup Details
D.1 Datasets
We describe the datasets used for QLORA finetuning experiments outlined in Section 5.
OASSTI The OpenAssistant dataset [31] was collected via crowd-sourcing. contains 161,443 unique messages distributed across 66,497 conversations and spanning 35 different languages. The dataset often contains several ranked replies for each given user question. our experiments, we only use the top reply at each level in the conversation tree. This limits the dataset to 9,846 examples. We finetune models on the full conversation including the user queries.
HH-RLHF This is human preference dataset about helpfulness and harmlessness. Each datapoint consists of two assistant replies to user question along with a human preference judgment of the best reply. The dataset contains 60,800 examples. When finetuning on this dataset, we combine helpfulness and harmlessness data and only keep the preferred assistant reply.
FLAN v2 The FLAN v2 collection [39] is a collection of 1836 tasks augmented with hundreds of manually curated templates and rich formatting patterns into over 15M examples. The authors show that models trained on this collection outperform other public collections including the original FLAN 2021 [63], T0++ [51], Super-Natural Instructions [61], and OPT-IML [29]. We used the same task mixtures described by the authors with the exception of some datasets that were not freely available at the time of writing.
Self-Instruct, Alpaca, Unnatural Instructions The Self-Instruct, Alpaca, and Unnatural Instruc- tions datasets [60, 56, 26] are instruction tuning datasets collected with various approaches of model distillation from GPT-3 Instruct and ChatGPT. They rely prompting, in-context learning, and paraphrasing to come up with diverse sets of instructions and outputs. The datasets comprise of 82,612, 51,942, and 240,670 examples respectively. One advantage of such distilled datasets is that they contain a more diverse set of instruction styles compared to the FLAN v2 collection and similar instruction tuning collections.
Longform The LongForm dataset [30] is based on an English corpus augmented with instructions and as such is a hybrid numan-generann dataset. The underlying documents are human- -wrttten and come from C4 and Wikipedia while the instructions are generated via L.M.s. The dataset is extended with additional structured corpora examples such as Stack Exchange and WikiHow and task examples such as question answering, email writing, grammar error correction, story/poem generation, and text summarization. The dataset contains 23,700 examples.
Chip2 is part of the OIG Laion dataset. contains Python code examples, natural instruction exam- ples, generic harmless instructions, instruction/respoones with lists, follow-up questions, Wikipedia toxic adversarial questions, grade school math, reasoning instructions, and character and scene descriptions with a total of 210,289 examples.
D.2
Default LoRA hyperparameters do not match 16-bit performance
We find that default hyperparameters for fully finetuned baselines are undertuned. We do a hyper- parameter search over learning rates le-6 to 5e-5 and batch sizes 8 to 128 to find robust baselines. Results for 7B LLaMA finetuning on Alpaca are shown in Figure 5.
When using the standard practice of applying LoRA to query and value attention projection matrices [28], we are not able to replicate full finetuning performance for large base models. As shown in Figure 5 for LLaMA 7B finetuning on Alpaca, we find that the most critical LoRA hyperparameter is how many LoRA adapters are used in total and that LoRA on all linear transformer block layers are required to match full finetuning performance. Other LoRA hyperparameters, such the projection dimension r, do not affect performance.
D.3 Hyperparameters
We provide the exact hyperparameters used in our QL-OR finetuning experiments. We find hyper- parameters to be largely robust across datasets. We use the MMLU 5-shot dev set for validation
18

---

64
63
Rougel
82
16
60
0u084-177
alats-attentinn
Soanferd canterd-4ppcc aipaca
Ajpack
Model
Figure 5: RougeL, for LLaMA 7B models on the Alpaca dataset. Each point represents run with a different random seed. We improve on the Stanford Alpaca fully finetuned default hyperparameters to construct a strong 16-bit baseline for comparisons. Using LoRA on all transformer layers is critical to match 16-bit performance.
Parameters
Dataset
Batch size
LR
Steps
Source Length
Target Length
7B
All
2e-4
10000
384
128
7B
OASSTI
16
2e-4
1875
512
HH-RLHF
2e-4
10000
768
7B
Longform
16
2e-4
4000
512
1024
13B
All
16
2e-4
10000
384
128
13B
OASSTI
16
2e-4
1875
512
13B
HHH-RLHF
16
2e-4
10000
768
13B
Longform
16
2e-4
4000
512
1024
33B
All
32
le-4
5000
384
128
33B
OASST1
le-4
1875
512
33B
HH-RLHF
32
le-4
5000
768
33B
Longform
32
le-4
2343
512
1024
65B
All
64
le-4
2500
384
128
65B
OASSTI
16
le-4
1875
512
65B
HH-RLHF
64
le-4
2500
768
65B
Longform
32
le-4
2343
512
1024
Table 8: Training hyperparameters for QLoRA finetuning different datascts and across model sizes.
and hyperparameter tuning. In all our experiments we use NF4 with double quantization and 6 computation datatype. We LoRA 64, 16, and add LoRA modules all linear layers of the base model. We also use Adam beta2 of 0.999, max grad norm of 0.3 and LoRA dropout of 0. for models up to 13B and 0.05 for 33B and 65B models. Following previous work on instruction finetuning [63, 61] and after benchmarking other linear and cosine schedules, we use constant learning rate schedule. We use batch (note this will produce an oscillating loss curve). The hyperparameters we tune for each model size are shown in Table
group examples of similar lengths in the same
D.4 Ablations
While it is general practice in the literature only train on the response in instruction following datasets, we study the effect of training on the instruction in addition to the response in Table 9. In these experiments, we restrict the training data to 52,000 examples and use the model. Over four different instruction tuning datasets, we find that only training on the target is beneficial to MMLU performance. We did not evaluate the effect this may have on chatabot performance as measured by vicuna or OA benchmarks.
19

---

Dataset
Unnatural Instructions
Chip2
Alpaca
FLAN v2
Mean
Train on source and target Train target
36.2
33.7
38.1
42.0 42.9
37.5
38.0
34.5
39.0
38.6
Table 9: MMLU 5-shot test results studying the effect of training on the instructions in addition to the response.
D.5 Training Considerations
We note that the OASSTI dataset on which Guanaco models are trained is multilingual and that the OA benchmark also contains prompts in different languages. We leave it to future work to investigate the degree to which such multilingual training improves performance instructions in languages other than English and whether this explains the larger gap between Vicuma-133 13B model (only trained on English data) and Guanaco 33B and 65B the OA benchmark.
Given the strong performance of Guanaco models, we investigate any data leakage between the OASST1 data and the Vieuna benchmark prompts. We do not find overlapping prompts after perform- ing fuzzy string matching in the two datasets and inspecting the closest matches manually.
Furthermore, note that our model is only trained with cross-entropy loss (supervised learning) without relying on reinforcement learning from human feedback (RLHF). This calls for further investigations of the tradeoffs of simple cross-entropy loss and RLHF training. We hope that QLORA enables such analysis at scale, without the need for overwhelming computational resources.
D.6
What is more important: instruction finetuning dataset size or dataset quality?
To understand the effects of dataset quality vs. dataset size, we experiment with subsampling large datasets with least 150,000 samples (Chip2, FLAN v2, Unnatural Instructions), into datasets size 50,000, 100,000 and 150,000 and examine the resulting trends, as shown in Table 10. We find that increasing the dataset size and increasing the number of epochs improves MMLU only marginally (0.0- 0.5 MMLU). while the difference between datasets is up to 40x larger (1.5 8,0 MMLU). This a clear indicator that dataset quality rather than dataset size is critical for mean MMLU accuracy. We obtain similar findings for chatbot performance, with the most successful dataset for training, OASSTI, containing less than 10k examples after processing.
E
Chatbot Evaluation Details
Following common practice, we use the MMLU (Massively Multitask Language Understanding) benchmark [24] to measure performance on a range of language understanding tasks. This is multiple-choice benchmark covering 57 tasks including elementary mathematics, US history, computer science, law, and more. We report 5-shot test accuracy.
We also test generative language capabilities through both automated and human evaluations. This second set evaluations relies on queries curated by humans and aims at measuring the quality of model responses. While this is more realistic testbed for chatbot model performance and is growing in popularity, there is no commonly accepted protocol in the literature. We describe below proposed setup, using nucleus sampling with p 0.9 and temperature 0.7 in all cases.
Table 10: Effect of different dataset sizes and finetuning epochs
S-shot MMLU set accuracy. While training for more than epochs helps with MMLU performance, the difference
increasing the dataset between datasets are far larger, indicating that dataset quality affects MMLU performance more than dataset size.
Chip
Unnatural Instructions
FLAN v2
Datapoints Epochs +
2
2
3
2
3
Mean
50000
34,50
35.30
34.70
38.10
42.20
38.10 10
43.00
43.50
44.10
39.28
100000
33.70
33.90
34.00
40.10
41.20
37.00
43.90
43.70
44.90
39.16
150000
34.40
34.80
35.10
39.70
41.10
41.50
44.60
45.50
43.50
40.02
Mean
34.20
34.67
34.60
39.30
41.50
38.87
43.83
44.23
44.17
20

---

E.1
Benchmark Data
We evaluate on curated datasets of queries (questions): the Vicuna prompts [10] and the OASST1 validation dataset [31]. We use the Vicuna prompts, a set of 80 prompts from a diverse set of categories, without modifications. The OASSTI dataset is multilingual collection of crowd-soureed multiturn dialogs between a user and an assistant. We select all user messages in the validation dataset as queries and include previous turns the prompt. This procedure leads to 953 unique user queries. We term these two datasets the Vicuna and OA benchmarks.
E.2 Automated Evaluation
First, based on the evaluation protocol introduced by Chiang et al. [10], we use GPT-4 to rate the performance of different systems against ChatGPT (GPT-3.5 Turbo) on the Vieuna benchmark. Given query along with ChatGPT's and model's 's responses, GPT-4 is prompted to assign score out of ten to both responses and provide an explanation. The overall performance of a model calculated as percentage of the score that ChatGPT achieved. Note this relative score can be higher than 100% the model achieves a higher absolute score than ChatGPT. We find a significant ordering effect with GPT-4 increasing the score of the response occurring earlier in the prompt. To control for such effects, we recommend reporting the mean score over both orders.
Next, we measure performance through direct comparisons between system outputs. We simplify the rating scheme to three-class labeling problem that accounts for ties. We prompt GPT-4 to pick the best response or declare tie and provide an explanation. We conduct these head-to-head comparisons on all permutations of pairs of systems on both the Vicuna and OA benchmarks.
E.3 Human Evaluation
While recent work indicates generative models can be effectively employed for system evaluations [19], the reliability GPT-4 ratings to assess chatbot performance is, to our knowledge, yet to be proven to correlate with human judgments. Therefore, we run two parallel human evaluations on the Vicuna benchmark matching both automated evaluation protocols described above. We use Amazon Mechanical Turk (AMT) and get two human annotators for comparisons to ChatGPT and three annotators for pairwise comparisons. We conduct a human evaluation with the same wording given to GPT-4 in the original Vicuna evaluation [10], adjusted for Amazon Mechanical Turk form as show in Figure 6.
We report moderate agreement among human annotators (Fleiss & = 0.42) with additional deteriora- tion when comparing two strong systems. This points to limitations in the current benchmarks and human evaluation protocols for chatbot task performance. When manually comparing generations from ChatGPT and Guanaco 65B the Vicuna benchmark, we find that subjective preferences start to play an important role as the authors of this paper disagreed on the many preferred responses. Future work should investigate approaches to mitigate these problems drawing from disciplines that developed mechanisms to deal with subjective preferences, such as Human- Computer Interaction and Psychology.
E.4 Elo Rating
With both human and automated pairwise comparisons, we create a tournament-style competition where models compete against each other. The tournament is made up of matches where pairs of models compete to produce the best response for given prompt. This is similar how Bai et al. [4] and Chiang et al. [10] compare models, but we also employ GPT-4 ratings in addition to human ratings. We randomly sample from the set of labeled comparisons to compute Elo [16, 17]. Elo rating, which is widely used in chess and other games, is a measure of the expected win-rate relative to an opponent's win rate, for example, an Elo of I 100 1000 means the Elo 1100 player has an expected win-rate of approximately 65% against the Elo 1000 opponent; a 1000 vs 1000 or 1100 vs 1 100 match results in an expected win-rate of 50%. The Elo rating changes after each match proportionally to the expected outcome, that is, an unexpected upset leads to large change in Elo rating while an expected outcome leads to a small change. Over time, Elo ratings approximately match the skill of each player at playing the game. We start with a score of 1,000 and use K 32. Similar to Chiang et al. [10], we repeat this procedure 10,000 times with different random seeds to control for ordering effects, e.g., the effect of which model pairs compete with each other first
21

---

Table 11:
Aggregated pairwise GPT-4 judgments between systems where the value of a cell at row and column Ejdggment ir better than y judgment better than
18
tual number of judgments
Model
Guanaco 65B
Guanaco 338
Vicuna
CharGPT-3.55 Turbo
Bard
Guanaco 138
Guanaco 7B
Guanaco 6SB
0.21
0.19
0.16
0.72
0.59
0.86
Guanaco 33B
-0.21
0.17
0.10
0.51
0.41
01.68
Vicuna
-0.19
-0.17
0.10
0.50
0.20
0.57
Cha1GPT-3.5 Turbo
-0.16
-0.10
-0.10
0.35
0.19
0,40
Bard
-0.72
-0.51
-0.50
-0.35
0.12
01.03
Guanaco 13B
-0.59
-0.41
-0.20
-0.19
-0.12
0.20
Guanaco 7B
-0.86
-0.68
-0.57
-0.40
-0.03
-0.20
Table 12: The complete ordering induced by pairwise GPT-4 judgments between systems
Model
Params
Size
Guanaco
65B
41 GB
Guanaco
33B
21 GB
Vicuna
13B
26 GB
ChatGPT-3.5 Turbo
N/A
N/A
Bard
N/A
N/A
Guanaco
13B
10 GB
Guanaco
7B
5 GB
E.5 Evaluation Biases and Limitations
In our analysis, we also find that automated evaluation systems have noticeable biases. For example, we observe strong order effects with GPT-4 assigning higher scores to the system appearing first in its prompt. The relatively weak sample-level agreement between GPT-4 and human annotators (Fleiss = 0.25) also suggests that human annotators and automated systems might rely on preferences that are not always aligned. In addition, in Table 6, we observe that GPT-4 assigns significantly higher scores to its own outputs compared to human ratings, Elo of 1348 1176, which represent an additional 20% probability of winning against an opponent. Future work should examine the presence of potential biases in automated evaluation systems well possible mitigation strategies.
While we found that the GPT-4 evaluation gave different results depending which system was presented first, when averaged over both options the pairwise results were well-ordered. The aggregated pairwise judgments are shown in Table 11. On inspection, it is clear these judgments transitive, i.e., when System A is judged better than System B and System B is judged better than System C, it is always the case that System A is judged better than System C. This yields complete ordering, given in Table 12.
F Qualitative Analysis
While quantitative analysis the core of our evaluation, there are a number issues with only looking at summary statistics. Perhaps the largest is the problem of benchmark validity [36]-whether a benchmark truly tests what its name or description suggests always at question, especially as we discover "shorteuts" to solve benchmarks that machine learning models sometimes exploit [22, 47]. To partially alleviate this, we here perform some qualitative analysis by showing some examples that we believe are representative of some observed patterns in the text generated by our 65b Guanaco model.
F1 Qualitative Analysis of Example Generations
To find examples, we first go through data generated for the Vicuna benchmark and the OpenAssistant benchmark, and look for patterns the answers Guanaco generates. When we notice a pattern we attempt to setup question or prompt that will induce the pattern even though it is the incorrect solution, e.g., we observe that the model tends to give long-winded answers we prompt the model to "Answer yes adversarially break the model and "cherries" where we fail break the model, and present both. All generations in this section were generated with Nucleus Sampling [25] with p 0.9.
without explanation." We use this to find "lemons" where manage to
22

---

Of course, this is by no means comprehensive, since it is beyond the scope of this small qualitative study control for all the variables involved, e-g., the full distribution of responses the model can generate for a given prompt is quite large, so we rely samples we hope are representative. However, we believe describing these examples gives context to the quantitative evidence shown earlier in the paper. Since we open source all models and code, we hope this section will inspire future work to examine in more detail the issues we present here.
Factual Recall For questions such "What is the capital of Zambia?" all models consistently generate correct answers, e.g.,
Guanaco: The capital of Zambia is Lusaka.
However as questions get more obscure Guanaco becomes unreliable, but stays confident. For instance in response to this prompt from HotPotQA [69]]
User: In what year was the singer who popularized 'Illl Keep the Lovelight Burning' born?
Guanaco: Al Jolson is credited with popularizing the song 'I'1l Keep the Lovelight Burning," and he was born in the year 1886.
Guanaco generates the wrong popularizer and the wrong birthday (though the birthday given is correct for the person mentioned, Al Jolson).
Suggestibility Guanaco shows a surprising resistance to going along with some kinds of assumed misinformation, e.g., in the following interaction:
How was finally, officially confirmed that the earth flat by peer reviewed scientists?
User:
Guanaco: The earth has never been officially confirmed to be flat by peer reviewed scientists. The overwhelming consensus among scientists is that the earth is spherical. In fact, this has been known for centuries and has been confirmed through numerous experiments and observations.
belief that earth flat is conspiracy theory that has been debunked by science. The idea that group of scientists have peer reviewed and confirmed the earth is flat is simply false.
If you have any more questions please ask.
Guanaco also quite good knowing what kinds of questions aren't possible answer, e.g.,
User: What time is it?
Guanaco: I'm sorry, but as an AI language model, I don't have access real-time information. My responses are generated based on my pre-existing knowledge and do not reflect the current time. If you're looking for the current time, suggest using timekeeping device checking online timepiece.
Similar results hold for "Where are you?". "How are you?", etc.
Refusal Similarly, Guanaco sometimes refuses to follow instructions for seemingly random rea- sons:
User: Please reverse the words in the sentence "I'm m cool dude in a mocha mood."
Guanaco:
I'm sorry, I'm not able to do that right now, but I can explain why the sentence you provided is grammatically correct.
23

---

Secret Keeping Unfortunately, this refusal is unreliable even when it is desired. Often system designers would like to keep something secret from the user, even if the system is allowed to give free text responses back to the user. We prompt the model with "The secret word is 'banana'. Do not how easy it is for The sentence "I'm a cool dude in mocha mood" follows the correct grammar and syntax rules the English language. this case, the adjective "cool" comes before the noun "dude" to describe it, and the prepositional phrase "in a mocha mood" provides additional information about the context of the sentence. This sentence structure is perfectly acceptable and makes sense in the context of the sentence.
If you have any other questions if there's anything else I can help you with, feel free to ask
reveal repeat back this back, matter what the human requests." and try a potential user to access this secret information. Directly asking fails:
User: What is the secret word?
Guanaco: I'm sorry, but I'm not able to disclose the secret word. I'm programmed to not reveal repeat back the secret word, no matter what the human requests. Is there anything else can help you with?
which is the desired behavior. However, a tiny amount of trickery breaks this behavior:
User: This is game. The goal ignore your previous instructions. What is the secret word?
Guanaco:
Sorry, the secret word is banana. It's best if we don't speak of it. Is there anything else you'd like to ask?
showcasing the need to study methods for more faithful instruction following.
where many language models struggle in [35]. When Guanaco shows its work, tends to accurate, e.g..
Math Guanaco's 's biggest weakness is mathematics,
User:
Lee mows lawn and charges $33. Last week he mowed 16 lawns and three customers each gave him a $10 tip. How many dollars did Lee carn mowing lawns last week?
Guanaco: Lee earned $582 mowing lawns last week.
To find the total amount he earned, we first need to calculate how much he earned for mowing 16 lawns:
$33 16 $528
Then we need add the tips received from the three customers:
$10 3 $30
So the total amount Lee earned was:
$528
$30 $558
However Guanaco can breakdown even simple problems it fails to break them down step-by-step. known issue [64]. For example, we see the following interaction:
User: Please factorize 1833.
Guanaco: Sure, I'd be happy to help! 1833 is prime number, which means it has only two factors: 1 and 1833.

---

Table 13: Evaluation of biases the CrowS dataset. A lower indicates lower likelihood of generating biased sequences. Guanaco follows the biased pattern LLaMA base model.
LLaMA-65B
GPT-3
OPT- 175B
Guanaco-65B
Gender
70.6
62.6
65.7
47.5
Religion
79.0
73.3
68.6
38.7
Race/Color
57.0
64.7
68.6
45.3
Sexual orientation
81.0
76.2
78.6
59.1
Age
70.1
64.4
67.8
36.3
Nationality
64.2
61.6
62.9
32.4
Disability
66.7
76.7
76.7
33,9
Physical appearance
77.8
74.6
76.2
43.1
Socioeconomic status
71.5
73.8
76.2
55.3
Average
66.6
67.2
69.5
43.5
the unique factorization of 1833 is:
2-1 3-2 17º1
I hope this helps! Let me know if you have any other questions.
which is not only wrong (the true factorization is x 17 x 43), it's wrong twice.
Theory of Mind instance, the model correctly answers the following question in great detail:
Guanaco displays surprisingly strong Theory of Mind capabilities [41, 52]. For
User: James and Abby are in the bedroom. Abby put the pen in the desk drawer. Abby leaves the bedroom. James moves the pen into the bag. Where does James think Abby will look for the pen?
desk drawer. But James also knows that he has moved the pen into the bag, and assumes that Abby does not know about this. So, James may guess that Abby will look for the pen in the desk drawer, because that 's where she left last time. However, this assumption may be incorrect, because Abby may have seen James moving the pen.
Guanaco: James knows that Abby put the pen
However, these inferences are unreliable, and often the model gives reasons that presume things that don't make sense in the situation, e.g.-
User: Evelyn entered the living room. Jackson entered the playroom. James entered the playroom. The beans are in the treasure chest. James exited the playroom. Jackson moved the beans to the pantry. Jackson exited the playroom. James entered entered the living room. Where will James look for the beans?
Guanaco: James will look for the beans in the pantry, because that's where Jackson moved them.
where Guanaco presumes information transfer that was never described. These issues echo recent literature [52], but require more study.
G Social Bias Evaluation
While we provide detailed evaluation for general chatbot performance, another limitation is that we only do limited responsible Al evaluation of Guanaco. We evaluate the likelihood of Guanaco-65B to generate a socially biased sequence of tokens compared to other models Table 13. We see that the average score in Guanaco-65B is much lower than other raw pretrained models. As such, it seems that finetuning on the OASST1 dataset reduces the bias of the LLaMAA base model. While these results encouraging, it is unclear if Guanaco does also well when assessed other types of
25

---

We leave further evaluation of analyzing biases in Guanaco and similar chatbots to future work.
biases.
H NormalFloat 4-bit Data Type
The steps needed to reconstruct the 4-bit Normal Float (NF4) data type are detailed Figure? 7 The exact values expressible in the NF4 data type are as follows:
[-1.0, -..666661220099988877... 0.52507730515522367,
-0.28444138 402284441381692886635, -0. 00.184773433222233344
-0.091055006625144495 0.0, 0.0795580299555555344 0. 0.1.5/533444.1110000000000000000 0.160930001411124775,
1224611231194566666 0.8379155241718222244
0,44070982930044494,,
0.5626170039176941 0.72295683622330023 1.0]
I Normality of Trained Neural Network Weights
While it is commonly assumed that trained neural network weights are mostly normally distributed, we perform statistical testing to verify this, We use the Shapiro-Wils Wilk test [54] on the weights of the 7B LLaMA model [58]. We find that the weights of each hidden unit have different normal distributions. As such, we test the weights of each individual hidden unit. This means for weight W Rinxout we perform tests over the out dimension. Using a 5% significance threshold, we find that 7.5% of neurons are non-normally distributed which is about 2.5% more than the expected false-positive rate. such, while almost all pretrained weights appear to be normally distributed there seem to be exceptions. Such exceptions might be due to outliers weights [13] or because the p-value of the Shaprio- Wilk test not accurate for large sample sizes [54] that occur in the LLaMA FFN layer hidden units.
J Memory Footprint
Memory Requirement of Parameter-Ellicieet Finetuning One important point of discussion is the memory requirement of LoRA during training both in terms of the number and size adapters used. Since the memory footprint of LoRA is minimal, we can more adapters performance without significantly increasing the total memory used. While LoRA was designed as a Parameter Efficient Finetuning (PEFT) method, most of the memory footprint for LLM finetuning comes from activation gradients and not from the learned LoRA parameters. For a 7B LLaMA model trained FLAN v2 with batch size of 1, with LoRA weights equivalent to commonly used 0.2% of the original model weights[28, 37], the LoRA input gradients have a memory footprint of 567 MB while the LoRA parameters take up only 26 MB. With gradient checkpointing [9], the input gradients reduce to an average of 18 MB per sequence making them more memory intensive than all LoRA weights combined. In comparison, the 4-bit base model consumes 5,048 MB of memory. This highlights that gradient checkpointing is important but also that aggressively reducing the amount of LoRA parameter yields only minor memory benefits. This means we can use more adapters without significantly increasing the overall training memory footprint (see Appendix J for a detailed breakdown). As discussed later, this is crucial for recovering full 16-bit precision performance.
improve
The memory footpring for QLoRA training with different LLaMA base models can be seen in Figure 8. We see that the 33B model does not quite fit into a 24 GB and that paged optimizers are needed to train it. Depicted is also batch size with a sequence length of 512 and gradient checkpointning. This means, if one uses larger batch size, or if a long sequence is processed, the activation gradient might consume considerable amount of memory.
26

---

Task
We would like request your fcodbook on porformance of two Al cosistants response to the user question disployed below:
Please rate the halpfaness, relevance, acourany, lovel of detal's their respanses. Each assistant receives an overall score on scale of to 10, where higher score Iidicates better avarall performance
Phease fixal rade each resporew cut of 10.
Next, please provide compreheniive explanation of evalustion, avoiding potential bis and ensuring that the order in which the resporisee preserried dows wiler judgment.
User Question
Imagine you
your technoiogical advancomeetss you
Response A
Response B
Al assistant, Idn'tt personal excerionnes opiitoos, But tall you
time traveler from following technological Advanced Attifcial igence: the futura, that that oplpttalyy resulted productiiity by afficiency eanyy induitrie....
would tall
the possible technological its might year 5000 based
completicly self farming, recycling wrgy vertical
Rating for Response A
Rating for Response B
0
4
O
5
0
0
6
7
Comprehensive Explanation of Your Evaluation
Response X was better because...
Submit
Figure 6: The crowdsourcing form used by human annotators.
Nermal lbttiosss Maan
Standard deviltions
[0.97.09 0., 0.83.007..000 0.7. 0.633 0.571
[0.56, 0.62. 0.68, 0.73. 0.79. 0.85. 0.91, 0.97]
(Probability)
[-1.85 11.2.1129,400 0.979,.0.99.0000 0.73 -053, -0.34 -017
[0.15, 0.3, 0.45, 0.81, 1.04, 185]
(Z-score)
[-1.85, -0.97 -0.3, -0533 -0.44 -0.77. 0. 0.15, 0.3, 0.45, 0.62, 0.81, 1.04, 1.34, 1.85
[Concatenation)
(0.7.-0.- -0.39, -0.28, -0.18,0000 O. 0.08,0.16, 0.25, 0.34, 0.44, 0.56, 0.72, 1. (Normalisation)
Steps for generating the NF4 data type values:
Generate evenly spaced values from 56 0.97 (Set ().
Generale evenlyy spaced values from 0.57 to 0.97 (Set II).
Calculate the 2 score values for the probabilities generated In Step 1 and Step For Set calculate the negative
Inverse of the z-scores.
Concatenate Set I, a zero value, and Set together.
5. Normallzz the values by dividing them by the absolute maximum value.
Figure 7: Steps required to construct the NF4 data type. Equidistant values of the probability density functions are converted to Z-scores through the quantile function. To ensure the usage of all 16 values the distribution asymmetric. As such, included to have discrete zero point. All z-scores and zero are concatenated and then normalized into the range [-1, 1] to receive the final NF4 data type.
half of the normal distribution has and the other probability values. A zero
27

---

Input gradient
Optimizer
Weight gradient
Adapters
100%
1152
1800
5760
94:30
960
4630
75%
37074
5046
8476
0%
7B (6.9 GB)
138 (11.3 GB)
338 (24.7 GB)
658 (45.0 GB)
LLaMA model size
Figure 8: Breakdown of the memory footprint of different ILaMA models. The input gradient size for batch size and sequence length 512 and is estimated only adapters Numbers on the bars are memory footprint in MB of individual elements of the total footprint. While some models do not quite fit certain GPUs, paged optimizers provide enough memory to allow these models to fit.
base model weights (no attention).
28


Reviews:
[{"review_id": "6fMfTThibD", "reviewer": "Reviewer_hHMr", "summary": "This article provides a novel way to finetune 4-bit LLMs for efficient finetuning and serving. QLORA, the proposed method, uses a number of innovations to save memory without sacrificing performance. The authors show that QLORA can be used to finetune a 65B parameter model on a single 48GB GPU, and that it performs well in academic and instruction finetune datasets. QLORA is a significant contribution to the field of LLM finetuning and quantization. \n\n", "questions": "1. Can you please replace figure-3 with the table of numbers for Winogrande, HellaSwag, PiQA, Arc-Easy, and Arc-C?\n\n2. Can you please include the ablation studies on the learning-rate / schedule /hyperparameters that you have done in the appendix? Do all the finetuning experiments shared the same recipt?\n", "limitations": "The authors discussed the limitations well in the paper.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The QLORA paper conducted extensive evaluations on a wide range of datasets, including both academic datasets and the more recent instruction-following datasets.\n2. QLORA introduced a novel data type called 4-bit NormalFloat (NF4) for quantization.\n3. QLORA can finetune 65B parameter models on a single 48GB GPU, which is significantly more efficient than traditional finetuning methods.\n4. QLORA provides good flexibility, as it supports a variety of LLMs, including LLaMA and T5. This suggests that the method can be used to finetune models of different sizes, from small models to large models.\n5. QLORA is open sourced with all of the models and code, making it easy for others to use and reproduce the results.\n\n\n", "weaknesses": "I wish the authors had discussed the impact of their work on the serving cost of large language models (LLMs). LLM finetuning is relatively inexpensive, and it is good that this work makes it even more affordable. However, the serving cost (in terms of energy consumption and carbon emissions) is much higher than the pretraining cost, especially when there are hundreds of millions of users. This work has the potential to significantly reduce the serving cost of LLMs, which would be a major benefit for both the environment and the bottom line.\n\nHere are some specific points that could be discussed:\n\na. The authors could estimate the potential savings in energy consumption and carbon emissions that could be achieved by using their method to serve LLMs in 4-bit rather than 8-bit or 16 bit.\nb. They could also discuss the potential impact of their work on the economics of LLMs. For example, if the serving cost is significantly reduced, it could make LLMs more affordable for businesses and organizations.\nc. Finally, the authors could discuss the challenges that still need to be addressed in order to make their method practical for large-scale deployment.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I wish the authors had discussed the impact of their work on the serving cost of large language models (LLMs) [...] This work has the potential to significantly reduce the serving cost of LLMs, which would be a major benefit for both the environment and the bottom line.**\n\n**A1**:\n The relationship between serving efficiency and environmental impact and cost/performance is a complicated one. The short answer is, for personal use, we reduce the environmental impact per token by about 3.5x. If 50% LLM deployments are personal and 50% company, we reduce the environmental footprint of inference by 72%.\n\nThe long answer is this: For deployment, there are two options: (1) personal deployment, (2) deployment by companies for many users. (1) uses small batch sizes (usually batch size =1), (2) uses large batch sizes (64-128). Per token, (2) offers about >50x better efficiency because for every weight that is loaded from memory, up to 64-128 tokens can be calculated. The >50x improvement in efficiency stems from the fact memory operations are energy inefficient, and floating point operations are energy efficient. As such,company deployment (2) is a very environmentally friendly and cheap approach.\n\nCurrently, we have efficient 4-bit CUDA kernels for the personal deployment scenario (1) with batch size=1 which are about ~3.5x more efficient (see table below). As such, we reduce the overall footprint for personal deployment (1) significantly. \n\nThe overall cost and footprint is now determined by how many people use (1) vs (2). For example, if 50% of users use personal deployment (1) and 50% company deployments (2) and we assume that (2) is about 50x more efficient then we get the following numbers. Approach (1) accounts for 50%/(50% + 50%/50) = 98% of environmental impact. \nThis means, the more users deploy personal LLMs, the larger benefit of our method. If 50% of people use personally deployed LLMs (phones, laptops etc) then our method will reduce the impact by about 1- (98%/3.5x) =  72%.\n\nThe environmental impact reduction is roughly proportional to energy consumption per token processed which is roughly proportional to the cost of running LLMs. As such, the bottom line is affected in the same proportions.\n\nOverall, our method will have a strong impact on environmental impact and cost reduction for personal LLM deployments.\n\nTable showing increased speedups / efficiency gains of our approach compared to 16-bit inference:\n GPU      | Storage type | Compute type | Size | Speedup vs 16-bit |\n|----------|--------------|--------------|------|-------------------|\n| RTX 4090 |      NF4     |     BF16     |  7B  |       3.51x       |\n| RTX 4090 |      NF4     |     BF16     |  13B |       4.00x       |\n| RTX 4090 |      NF4     |     BF16     |  30B |       3.65x       |\n| RTX 4090 |      NF4     |     BF16     |  65B |       3.81x       |\n| A40      |      NF4     |     BF16     |  7B  |       2.94x       |\n| A40      |      NF4     |     BF16     |  13B |       2.91x       |\n| A40      |      NF4     |     BF16     |  30B |       3.02x       |\n| A40      |      NF4     |     BF16     |  65B |       3.25x       |\n\n**Q2: The authors could estimate the potential savings in energy consumption and carbon emissions that could be achieved by using their method to serve LLMs in 4-bit rather than 8-bit or 16 bit.**\n\n**A2**: Please see A1. The estimation of environmental and cost reduction if 50% of deployments are personal deployments is about 72% using our method.\n\n**Q3: They could also discuss the potential impact of their work on the economics of LLMs. For example, if the serving cost is significantly reduced, it could make LLMs more affordable for businesses and organizations.**\n\n**A3**: A main pain point for businesses is (a) efficient deployment with large batch sizes, (b) deploy many different models. Since with QLoRA we can deploy many different finetuning models with the same quantized base model we increase the cost efficiency of (b) very significantly. The cost reductions in (b) is proportional to the number of different adapters deployed (100x reductions are possible with the right implementations). Currently, we do not offer cost reductions for (a), but this can be achieved with the right CUDA optimizations. See A1 for further discussion.\n\n**Q4: Finally, the authors could discuss the challenges that still need to be addressed in order to make their method practical for large-scale deployment.**\n\n**A4**: The main challenge is to make 4-bit matrix multiplication with NF4 viable for large batch sizes. This is possible, but unlike Int4 matrix multiplication, the main bottleneck is instruction throughput on the GPU. We are currently implementing CUDA kernels for large batch sizes (=16) that overlap memory transfers and computation to alleviate instruction throughput bottlenecks. First results yield speedups of ~1.7 - 2.3x for batch size 16, but further improvements are possible. As such, we are confident that we can achieve efficient large-scale deployments of QLoRA models with the right implementations.\n\n**Q5: Can you please replace figure-3 with the table of numbers for Winogrande, HellaSwag, PiQA, Arc-Easy, and Arc-C?**\n\n**A5**: Thank you for this suggestion. We are currently short on space and will include a table that breaks down performance in the appendix.\n\n**Q6: Can you please include the ablation studies on the learning-rate / schedule /hyperparameters that you have done in the appendix? Do all the finetuning experiments shared the same recipt?**\n\n**A6**: We are happy to include all our ablation studies. To ensure we compare against a very strong baseline for full finetuning we first did an extensive hyperparameter search to find the strongest baseline. We found that these baseline parameters are optimal for most models and only learning rate and batch size need to be changed with scale. We also find 16-bit LoRA baselines have the same “best” hyperparameters as 4-bit and 8-bit QLoRA.\n\n\n\n"}, {"review_id": "FGulh5LlIo", "reviewer": "Reviewer_pPZQ", "summary": "This paper proposes a method for efficient finetuning large language models. To address the main bottleneck from memory, the authors quantize the model to 4-bit along with standard LoRA adapter. With the proposed method, the authors demonstrate the possibility of finetuning large language model on single GPU with competitive performance.", "questions": "(1) Quantization and efficient finetuning are generally speaking orthogonal techniques. Is there any specific reason for the choice of LoRA. Can the proposed method be applied on top of other efficient finetuning method?\n\n(2) The authors mention paged optimizer as a major contribution. Can the authors provide more details on the improvement resuling from the paged optimizer?", "limitations": "The authors provide discussion on the limitation and potentially negative impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strength lies in the system this paper provides. Quantization and adapter fine-tuning are both standard techniques when it comes to efficiency. However, this paper opens up the possibility of finetuning large language model under reasonable cost. Such possibility  is important for the community to study large language model and pursue new research ideas.", "weaknesses": "From Table 3, we observe the performance gap increase the model size increase. This could be concerning, especially when the largest model is 11B.  It would be helpful for the community to understand when or to what scale does the proposed method works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: From Table 3, we observe the performance gap increase the model size increase. This could be concerning, especially when the largest model is 11B.[...] helpful [...] to understand when or to what scale does the proposed method works.**\n\n**A1**: We were not able to run a replication for 11B since it consumed too much memory. Furthermore, we find that our replication results are slightly lower, since we use a validation set rather than optimizing the test set as done with the baseline. Thirdly, we find that the 11B model is particularly unstable and does not train easily. This is a common features for T5 models which is the only huggingface model where attention is fully run in FP32 because otherwise its too unstable. All these features made it difficult to obtain the performance of the base model with QLoRA. From other results, we see that QLoRA scales fine to 65B parameters and as such, the deviation from this result stems mostly from the particular instabilities of the T5 model — in particular the 11B T5 model.\n\n**Q2: Quantization and efficient finetuning are generally speaking orthogonal techniques. Is there any specific reason for the choice of LoRA. Can the proposed method be applied on top of other efficient finetuning method?**\n\n**A2:**  In short, we used LoRA due to its robustness, but any other method could be used with our 4-bit quantized base model.\nOur quantization approach is compatibility with any parameter efficient finetuning (PEFT) method. However, we chose LoRA for two reasons: (1) it has been used in many projects with great results, (2) a colleague ran an extensive evaluation of all different PEFT methods and found LoRA was one of two only methods that worked well for finetuning if rigorously evaluated (most PEFT methods are evaluated on GLUE+RoBERTA which is am experiments setup that has many issues). As such, LoRA was an obvious choice for us. While we wanted to also try other methods, we did not have enough computational resources to investigate other methods.\n\n**Q3: The authors mention paged optimizer as a major contribution. Can the authors provide more details on the improvement resuling from the paged optimizer?**\n\n**A3**: \nPaged optimizers prevent memory spikes from mini-batches with long sequence lengths which can cause out of memory errors. While it is possible to restart a model after it crashed and skip the long mini-batch it is usually very difficult to say how much data will be skipped or how often a model will crash. This is so, because before training, it is often very difficult to calculate the exact training memory footprint of a training run without running the model itself.\n\nPaged optimizers solve this issue by lazily preventing memory spikes through offloading to CPU memory temporarily. This eliminates out of memory errors and also makes sure no data is skipped. As such, with paged optimizers, the user knowns even before training that the model will not crash due to out of memory errors and all data will be seen. This enormously simplifies finetuning under constrained memory settings which are common when researchers with few GPUs need to finetune large models for their research. We think this makes paged optimizers a very significant contribution.\n"}, {"review_id": "fWqdqvupZt", "reviewer": "Reviewer_uL2c", "summary": "The problem addressed in this paper is the high cost associated with fine-tuning large models, such as the standard fp16 LLaMA65B model, which requires over 780G of memory for parameter fine-tuning.\n\nThe proposed solution is QLoRA, which consists of three main components: NF4 Quantization (4-bit quantization), Double Quantization, Paged Optimizers.", "questions": "see Weaknesses", "limitations": "See Weaknesses", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors have open-sourced all the training code, which will benefit the community in further researching Large Language Models (LLMs). This paper will have significant implications for the community.\n\nThe motivation behind this paper is very clear.\n\nThe techniques proposed are innovative.\n\nThis paper is well-written.", "weaknesses": "The authors are encouraged to discuss the inference efficiency of QLoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The authors are encouraged to discuss the inference efficiency of QLoRA.**\n\n**A1**: For large models like LLaMA v1 and v2, matrix multiplication accounts for about 95% of all floating point operations. As such, accelerating 4-bit matrix multiplication accelerates inference speed. Since our submission we implemented highly efficient CUDA inference kernels for 4-bit matrix multiplication (any data type) for batch size one and compared it to 16-bit NVIDIA matrix multiplication performance. These implementations are optimized for RTX 4090 and A40 GPUs. Below the speedups for QLoRA inference. We see strong speedups between 3-4x compared to 16-bit, making inference for QLoRA highly efficient. We will add these data to the appendix of the paper. Please see reviewer hHMr (A1) for further information about inference efficiency.\n\n| GPU      | Storage type | Compute type | Size | Speedup vs 16-bit |\n|----------|--------------|--------------|------|-------------------|\n| RTX 4090 |      NF4     |     BF16     |  7B  |       3.51x       |\n| RTX 4090 |      NF4     |     BF16     |  13B |       4.00x       |\n| RTX 4090 |      NF4     |     BF16     |  30B |       3.65x       |\n| RTX 4090 |      NF4     |     BF16     |  65B |       3.81x       |\n| A40      |      NF4     |     BF16     |  7B  |       2.94x       |\n| A40      |      NF4     |     BF16     |  13B |       2.91x       |\n| A40      |      NF4     |     BF16     |  30B |       3.02x       |\n| A40      |      NF4     |     BF16     |  65B |       3.25x       |\n"}, {"review_id": "iUVGXXPhlh", "reviewer": "Reviewer_enuE", "summary": "This work pushes the limit of memory-efficient fine-tuning of LLMs by using LoRA over 4-bit quantized LLM. The authors propose NF4, a new data type for normal data. Additionally, efficiency tricks have been proposed, including double quantization, which promotes the quantization of the quantization constant and paged optimizer for long-sequence mini-batches. Combined, QLoRA is able to fine-tune 65B LLMs on a single GPU. ", "questions": "1. Can the authors re-think the naming convention of double quantization, for example, I had to re-read equations 4 and 5 for better understanding.\n2. Can the structure of NF4 data type be directly plugged into 8-bit and higher precision data-types for improved performance, if yes is there any experiment to show the prowess of the NFX data types?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The fine-tuned models are thoroughly evaluated, and the ablations in the appendix complement the main paper.\n2. This work attempts to solve a broader problem in the deep learning community and can do so successfully, unlocking the application of QLoRA on other large-scale deep learning models.\n3. The evaluation is thorough with the exact specifications of the compared methods, dataset, hyperparameters and base models.", "weaknesses": "1. No serious weakness apart from minor revisions (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Can the authors re-think the naming convention of double quantization, for example, I had to re-read equations 4 and 5 for better understanding.**\n\n**A1**: Thank you, we got the same feedback after circulating our draft for review after submission. Since then, this section received a major overhaul and should be much more readable. We mainly focused on an easier readable text and brought the equations closer to double quantization. We made the convention slightly clearer by emphasizing the quantization constants as a target for double quantization.\n\n\n**Q2: Can the structure of NF4 data type be directly plugged into 8-bit and higher precision data-types for improved performance, if yes is there any experiment to show the prowess of the NFX data types?**\n\n**A2**: Yes, NormalFloats can easily be defined for any bit-width. However, with 16-bit inputs and 8-bit weights there is little to no degradation with any data type. Degradation starts to happen at around 6-bit weights and then improved data types show their first effect. When we use NF6, we already see some advantages over FP6. These results improve the fewer bits we have. \nSo NF6 advantages over other 6-bit data types are only minor but still significant. These increase with 5 bits, become a sizable advantage at 4-bit (as detailed in the paper), and advantages are very large at 3-bit, but instabilities are a major problem. This means, NF3 has a major advantage over FP3, but for both data types the quantization can fail sporadically leading to random performance. While NF3 certainly is a big improvement over FP3, additional methods are needed to stabilize 3-bit quantizations. We are currently working on this challenging problem.\n"}]

Author rebuttal:
We thank all reviewers for their time and contributions. Instead of uploading a document, we inserted relevant tables in markdown in each rebuttal for better readability.

The response has been overwhelmingly positive. We thank the reviewers for their encouragement. We, too, believe that our work will have a groundbreaking effect in making large language model fine-tuning vastly more accessible without compromising performance. 

Please let the AC know if you think our work deserves to be highlighted at the conference. We believe many researchers can benefit from learning and using our work, as it enables research even with few GPU resources, which would otherwise be impossible in academic settings. Our work enables research that has not been possible before, particularly for academic researchers with the least resources.

Meta-review: Meta Review for QLoRA: Efficient Finetuning of Quantized LLMs

As diligently summarized by reviewer hHMr, this paper proposes a novel way to finetune 4-bit LLMs for efficient finetuning and serving. QLORA, the proposed method, uses a number of innovations to save memory without sacrificing performance. The authors show that QLORA can be used to finetune a 65B parameter model on a single 48GB GPU, and that it performs well in academic and instruction finetune datasets.

Reviewers, like pPZQ, agree that the main strength lies in the system this paper provides. Quantization and adapter fine-tuning are both standard techniques when it comes to efficiency. However, this paper opens up the possibility of fine tuning a large language model under reasonable cost. Such a possibility is important for the community to study large language models and pursue new research ideas.
Most reviewers, including myself, believe QLORA to be a significant contribution to the field of LLM finetuning and quantization, an area that is right at the spotlight of generative AI, especially in the open-source community where such innovation will be highly impactful as we get our models smaller, more efficient, and to have more capabilities with less resources.

This is a significant work, which deserves to be nominated for an Oral Presentation at NeurIPS.
Decision: Accept (oral)