SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tim Dettmers*
Artidoro Pagnoni*
Ari Holtzman
Luke Zettlemoyer
{dettmers,artidoro,ahai,lsz}@cs.washington.edu
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching  $99.3\%$  of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) Double Quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) Paged Optimizers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training. $^{2}$
Finetuning large language models (LLMs) is a highly effective way to improve their performance, [40, 63, 43, 62, 60, 37] and to add desirable or remove undesirable behaviors [43, 2, 4]. However, finetuning very large models is prohibitively expensive; regular 16-bit finetuning of a LLaMA 65B parameter model [58] requires more than 780 GB of GPU memory. While recent quantization methods can reduce the memory footprint of LLMs [14, 13, 18, 67], such techniques only work for inference and break down during training [66].
We demonstrate for the first time that it is possible to finetune a quantized 4-bit model without any performance degradation. Our method, QLORA, uses a novel high-precision technique to quantize a pretrained model to 4-bit, then adds a small set of learnable Low-rank Adapter weights [28]
2https://github.com/artidoro/qlora and https://github.com/TimDettmers/bitsandbytes
that are tuned by backpropagating gradients through the quantized weights.
QLoRA reduces the average memory requirements of finetuning a 65B parameter model from  $>780\mathrm{GB}$  of GPU memory to  $<48\mathrm{GB}$  without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline. This marks a significant shift in accessibility of LLM finetuning: now the largest publicly available models to date finetunable on a single GPU. Using QLOrA, we train the Guanaco family of models, with the second best model reaching  $97.8\%$  of the performance level of ChatGPT on the Vicuna [10] benchmark, while being trainable in less than 12 hours on a single consumer GPU; using a single professional GPU over 24 hours we achieve  $99.3\%$  with our largest model, essentially closing the gap to ChatGPT on the Vicuna benchmark. When deployed, our smallest Guanaco model
Table 1: Elo ratings for a competition between models, averaged for 10,000 random initial orderings. The winner of a match is determined by GPT-4 which declares which response is better for a given prompt of the the Vicuna benchmark.  $95\%$  confidence intervals are shown  $(\pm)$ . After GPT-4, Guanaco 33B and 65B win the most matches, while Guanaco 13B scores better than Bard.
Model	Size	Elo
GPT-4	-	1348 ± 1
Guanaco 65B	41 GB	1022 ± 1
Guanaco 33B	21 GB	992 ± 1
Vicuna 13B	26 GB	974 ± 1
ChatGPT	-	966 ± 1
Guanaco 13B	10 GB	916 ± 1
Bard	-	902 ± 1
Guanaco 7B	6 GB	879 ± 1
(7B parameters) requires just 5 GB of memory and outperforms a 26 GB Alpaca model by more than 20 percentage points on the Vicuna benchmark (Table 4).
QLoRA introduces multiple innovations designed to reduce memory use without sacrificing performance: (1) 4-bit NormalFloat, an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats. (2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model). (3) Paged Optimizers, using NVIDIA unified memory to avoid the gradient checkpointing memory spikes that occur when processing a mini-batch with a long sequence length. We combine these contributions into a better tuned LoRA approach that includes adapters at every network layer and thereby avoids almost all of the accuracy tradeoffs seen in prior work.
QLoRA's efficiency enables us to perform an in-depth study of instruction finetuning and chatbot performance on model scales that would be impossible using regular finetuning due to memory overhead. Therefore, we train more than 1,000 models across several instruction tuning datasets, model architectures, and sizes between 80M to 65B parameters. In addition to showing that QLoRA recovers 16-bit performance (§4) and training a state-of-the-art chatbot, Guanaco, (§5), we also analyze trends in the trained models. First, we find that data quality is far more important than dataset size, e.g., a 9k sample dataset (OASST1) outperformed a 450k sample dataset (FLAN v2, subsampled) on chatbot performance, even when both are meant to support instruction following generalization. Second, we show that strong Massive Multitask Language Understanding (MMLU) benchmark performance does not imply strong Vicuna chatbot benchmark performance and vice versa—in other words, dataset suitability matters more than size for a given task.
Furthermore, we also provide an extensive analysis of chatbot performance that uses both human raters and GPT-4 for evaluation. We use tournament-style benchmarking where models compete against each other in matches to produce the best response for a given prompt. The winner of a match is judged by either GPT-4 or human annotators. The tournament results are aggregated into Elo scores [16, 17] which determine the ranking of chatbot performance. We find that GPT-4 and human evaluations largely agree on the rank of model performance in the tournaments, but we also find there are instances of strong disagreement. As such, we highlight that model-based evaluation while providing a cheap alternative to human-annotation also has its uncertainties.
We augment our chatbot benchmark results with a qualitative analysis of Guanaco models (see Appendix F). Our analysis highlights success and failure cases that were not captured by the quantitative benchmarks.
We release all model generations with human and GPT-4 annotations to facilitate further study. We open-source our codebase and CUDA kernels and integrate our methods into the Hugging Face transformers stack [65], making them easily accessible to all. We release a collection of adapters for 7/13/33/65B size models, trained on 8 different instruction following datasets, for a total of 32 different open sourced, finetuned models.
Figure 1: Different finetuning methods and their memory requirements. QLoRA improves over LoRA by quantizing the transformer model to 4-bit precision and using paged optimizers to handle memory spikes.
Block-wise k-bit Quantization Quantization is the process of discretizing an input from a representation that holds more information to a representation with less information. It often means taking a data type with more bits and converting it to fewer bits, for example from 32-bit floats to 8-bit Integers. To ensure that the entire range of the low-bit data type is used, the input data type is commonly rescaled into the target data type range through normalization by the absolute maximum of the input elements, which are usually structured as a tensor. For example, quantizing a 32-bit Floating Point (FP32) tensor into a Int8 tensor with range  $[-127, 127]$ :
$$
\mathbf {X} ^ {\text {I n t 8}} = \operatorname {r o u n d} \left(\frac {1 2 7}{\operatorname {a b s m a x} \left(\mathbf {X} ^ {\mathrm {F P} 3 2}\right)} \mathbf {X} ^ {\mathrm {F P} 3 2}\right) = \operatorname {r o u n d} \left(c ^ {\mathrm {F P} 3 2} \cdot \mathbf {X} ^ {\mathrm {F P} 3 2}\right), \tag {1}
$$
where  $c$  is the quantization constant or quantization scale. Dequantization is the inverse:
$$
\operatorname {d e q u a n t} \left(c ^ {\mathrm {F P 3 2}}, \mathbf {X} ^ {\text {I n t 8}}\right) = \frac {\mathbf {X} ^ {\text {I n t 8}}}{c ^ {\mathrm {F P 3 2}}} = \mathbf {X} ^ {\mathrm {F P 3 2}} \tag {2}
$$
The problem with this approach is that if a large magnitude value (i.e., an outlier) occurs in the input tensor, then the quantization bins—certain bit combinations—are not utilized well with few or no numbers quantized in some bins. To prevent the outlier issue, a common approach is to chunk the input tensor into blocks that are independently quantized, each with their own quantization constant  $c$ . This can be formalized as follows: We chunk the input tensor  $\mathbf{X} \in \mathbb{R}^{b \times h}$  into  $n$  contiguous blocks of size  $B$  by flattening the input tensor and slicing the linear segment into  $n = (b \times h) / B$  blocks. We quantize these blocks independently with Equation 1 to create a quantized tensor and  $n$  quantization constants  $c_i$ .
Low-rank Adapters Low-rank Adapter (LoRA) finetuning [28] is a method that reduces memory requirements by using a small set of trainable parameters, often termed adapters, while not updating the full model parameters which remain fixed. Gradients during stochastic gradient descent are passed through the fixed pretrained model weights to the adapter, which is updated to optimize the loss function. LoRA augments a linear projection through an additional factorized projection. Given a projection  $\mathbf{X}\mathbf{W} = \mathbf{Y}$  with  $\mathbf{X} \in \mathbb{R}^{b \times h}$ ,  $\mathbf{W} \in \mathbb{R}^{h \times o}$  LoRA computes:
$$
\mathbf {Y} = \mathbf {X} \mathbf {W} + s \mathbf {X} \mathbf {L} _ {1} \mathbf {L} _ {2}, \tag {3}
$$
where  $\mathbf{L}_1\in \mathbb{R}^{h\times r}$  and  $\mathbf{L}_2\in \mathbb{R}^{r\times o}$ , and  $s$  is a scalar.
3 QLORA Finetuning
QLoRA achieves high-fidelity 4-bit finetuning via two techniques we propose—4-bit NormalFloat (NF4) quantization and Double Quantization. Additionally, we introduce Paged Optimizers, to
prevent memory spikes during gradient checkpointing from causing out-of-memory errors that have traditionally made finetuning on a single machine difficult for large models.
QLORA has one low-precision storage data type, in our case usually 4-bit, and one computation data type that is usually BFloat16. In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to BFloat16, and then perform a matrix multiplication in 16-bit.
We now discuss the components of QLORA followed by a formal definition of QLORA.
4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization [15, 46] which is an information-theoretically optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. Quantile quantization works by estimating the quantile of the input tensor through the empirical cumulative distribution function.
The main limitation of quantile quantization is that the process of quantile estimation is expensive. Therefore fast quantile approximation algorithms, such as SRAM quantiles [15], are used to estimate them. Due to the approximate nature of these quantile estimation algorithms, the data type has large quantization errors for outliers, which are often the most important values.
Expensive quantile estimates and approximation errors can be avoided when input tensors come from a distribution fixed up to a quantization constant. In such cases, input tensors have the same quantiles making exact quantile estimation computationally feasible.
Since pretrained neural network weights usually have a zero-centered normal distribution with standard deviation  $\sigma$  (see Appendix I), we can transform all weights to a single fixed distribution by scaling  $\sigma$  such that the distribution fits exactly into the range of our data type. For our data type, we set the arbitrary range  $[-1, 1]$ . As such, both the quantiles for the data type and the neural network weights need to be normalized into this range.
The information theoretically optimal data type for zero-mean normal distributions with arbitrary standard deviations  $\sigma$  in the range  $[-1, 1]$  is computed as follows: (1) estimate the  $2^{k} + 1$  quantiles of a theoretical  $N(0, 1)$  distribution to obtain a  $k$ -bit quantile quantization data type for normal distributions, (2) take this data type and normalize its values into the  $[-1, 1]$  range, (3) quantize an input weight tensor by normalizing it into the  $[-1, 1]$  range through absolute maximum rescaling.
Once the weight range and data type range match, we can quantize as usual. Step (3) is equivalent to rescaling the standard deviation of the weight tensor to match the standard deviation of the k-bit data type. More formally, we estimate the  $2^{k}$  values  $q_{i}$  of the data type as follows:
$$
q _ {i} = \frac {1}{2} \left(Q _ {X} \left(\frac {i}{2 ^ {k} + 1}\right) + Q _ {X} \left(\frac {i + 1}{2 ^ {k} + 1}\right)\right), \tag {4}
$$
where  $Q_{X}(\cdot)$  is the quantile function of the standard normal distribution  $N(0,1)$ . A problem for a symmetric k-bit quantization is that this approach does not have an exact representation of zero, which is an important property to quantize padding and other zero-valued elements with no error. To ensure a discrete zeropoint of 0 and to use all  $2^{k}$  bits for a k-bit datatype, we create an asymmetric data type by estimating the quantiles  $q_{i}$  of two ranges  $q_{i}$ :  $2^{k-1}$  for the negative part and  $2^{k-1} + 1$  for the positive part and then we unify these sets of  $q_{i}$  and remove one of the two zeros that occurs in both sets. We term the resulting data type that has equal expected number of values in each quantization bin  $k$ -bit NormalFloat (NFk), since the data type is information-theoretically optimal for zero-centered normally distributed data. A step-by-step visualization of how to construct the NF4 data type and its exact values can be found in Appendix H.
Double Quantization We introduce Double Quantization (DQ), the process of quantizing the quantization constants for additional memory savings. While a small blocksize is required for precise 4-bit quantization [13], it also has a considerable memory overhead. For example, using 32-bit constants and a blocksize of 64 for W, quantization constants add  $32 / 64 = 0.5$  bits per parameter on average. Double Quantization helps reduce the memory footprint of quantization constants.
More specifically, Double Quantization treats quantization constants  $c_{2}^{\mathrm{FP32}}$  of the first quantization as inputs to a second quantization. This second step yields the quantized quantization constants  $c_{2}^{\mathrm{FP8}}$  and the second level of quantization constants  $c_{1}^{\mathrm{FP32}}$ . We use 8-bit Floats with a blocksize of 256 for the second quantization as no performance degradation is observed for 8-bit quantization, in line with results from Dettmers and Zettlemoyer [13]. Since the  $c_{2}^{\mathrm{FP32}}$  are positive, we subtract the mean from  $c_{2}$  before quantization to center the values around zero and make use of symmetric
quantization. On average, for a blocksize of 64, this quantization reduces the memory footprint per parameter from  $32 / 64 = 0.5$  bits, to  $8 / 64 + 32 / (64 \cdot 256) = 0.127$  bits, a reduction of 0.373 bits per parameter.
Paged Optimizers use the NVIDIA unified memory  $^{3}$  feature which does automatic page-to-page transfers between the CPU and GPU for error-free GPU processing in the scenario where the GPU occasionally runs out-of-memory. The feature works like regular memory paging between CPU RAM and the disk. We use this feature to allocate paged memory for the optimizer states which are then automatically evicted to CPU RAM when the GPU runs out-of-memory and paged back into GPU memory when the memory is needed in the optimizer update step.
QLORA. Using the components described above, we define QLORA for a single linear layer in the quantized base model with a single LoRA adapter as follows:
$$
\mathbf {Y} ^ {\mathrm {B F 1 6}} = \mathbf {X} ^ {\mathrm {B F 1 6}} \text {d o u b l e D e q u a n t} \left(c _ {1} ^ {\mathrm {F P 3 2}}, c _ {2} ^ {\mathrm {k - b i t}}, \mathbf {W} ^ {\mathrm {N F 4}}\right) + \mathbf {X} ^ {\mathrm {B F 1 6}} \mathbf {L} _ {1} ^ {\mathrm {B F 1 6}} \mathbf {L} _ {2} ^ {\mathrm {B F 1 6}}, \tag {5}
$$
$$
\operatorname {d o u b l e D e q u a n t} \left(c _ {1} ^ {\mathrm {F P 3 2}}, c _ {2} ^ {\mathrm {k - b i t}}, \mathbf {W} ^ {\mathrm {k - b i t}}\right) = \operatorname {d e q u a n t} \left(\operatorname {d e q u a n t} \left(c _ {1} ^ {\mathrm {F P 3 2}}, c _ {2} ^ {\mathrm {k - b i t}}\right), \mathbf {W} ^ {4 \mathrm {b i t}}\right) = \mathbf {W} ^ {\mathrm {B F 1 6}}, \tag {6}
$$
We use NF4 for  $\mathbf{W}$  and FP8 for  $c_{2}$ . We use a blocksize of 64 for  $\mathbf{W}$  for higher quantization precision and a blocksize of 256 for  $c_{2}$  to conserve memory.
For parameter updates only the gradient with respect to the error for the adapters weights  $\frac{\partial E}{\partial \mathbf{L}_i}$  are needed, and not for 4-bit weights  $\frac{\partial E}{\partial \mathbf{W}}$ . However, the calculation of  $\frac{\partial E}{\partial \mathbf{L}_i}$  entails the calculation of  $\frac{\partial \mathbf{X}}{\partial \mathbf{W}}$  which proceeds via equation (5) with dequantization from storage  $\mathbf{W}^{\mathrm{NF4}}$  to computation data type  $\mathbf{W}^{\mathrm{BF16}}$  to calculate the derivative  $\frac{\partial \mathbf{X}}{\partial \mathbf{W}}$  in BFloat16 precision.
To summarize, QLoRA has one storage data type (usually 4-bit NormalFloat) and a computation data type (16-bit BrainFloat). We dequantize the storage data type to the computation data type to perform the forward and backward pass, but we only compute weight gradients for the LoRA parameters which use 16-bit BrainFloat.
4 QLoRA vs. Standard Finetuning
We have discussed how QLoRA works and how it can significantly reduce the required memory for finetuning models. The main question now is whether QLoRA can perform as well as full-model finetuning. Furthermore, we want to analyze the components of QLoRA including the impact of NormalFloat4 over standard Float4. The following sections will discuss the experiments that aimed at answering these questions.
Experimental setup. We consider three architectures (encoder, encoder-decoder, and decoder only) and compare QLoRA with 16-bit adapter-finetuning and with full-finetuning for models up to 3B. Our evaluations include GLUE [59] with RoBERTa-large [38], Super-NaturalInstructionss [62] with T5 [50], and 5-shot MMLU [24] after finetuning LLaMA on Flan v2 [39] and Alpaca [56].
To additionally study the advantages of NF4 over other 4-bit data types, we use the setup of Dettmers and Zettlemoyer [13] and measure post-quantization zero-shot accuracy and perplexity across different models (OPT [73], LLaMA [58],
BLOOM [53], Pythia [7]) for model sizes  $125\mathrm{m}$  - 13B. We provide more details in the results section for each particular setup to make the results more readable. Full details in Appendix C.
While paged optimizers are critical to do 33B/65B QLoRA tuning on a single 24/48GB GPU, we do not provide hard measurements for Paged Optimizers since the paging only occurs when processing
Figure 2: Mean zero-shot accuracy over Winogrande, HellaSwag, PiQA, Arc-Easy, and ArcChallenge using LLaMA models with different 4-bit data types. The NormalFloat data type significantly improves the bit-for-bit accuracy gains compared to regular 4-bit Floats. While Double Quantization (DQ) only leads to minor gains, it allows for a more fine-grained control over the memory footprint to fit models of certain size (33B/65B) into certain GPUs (24/48GB).
3 https://docs.nvidia.com/cuda/cuda-c-programming-guide
Table 3: Mean 5-shot MMLU test accuracy for LLaMA 7-65B models finetuned with adapters on Alpaca and FLAN v2 for different data types. Overall, NF4 with double quantization (DQ) matches BFloat16 performance, while FP4 is consistently one percentage point behind both.
LLaMA Size Dataset	Mean 5-shot MMLU Accuracy	Mean
7B	13B	33B	65B
Alpaca	FLAN v2	Alpaca	FLAN v2	Alpaca	FLAN v2	Alpaca	FLAN v2
BFloat16	38.4	45.6	47.2	50.6	57.7	60.5	61.8	62.5	53.0
Float4	37.2	44.0	47.3	50.0	55.9	58.5	61.3	63.3	52.2
NFloat4 + DQ	39.0	44.5	47.5	50.7	57.3	59.2	61.8	63.9	53.1
mini-batches with long sequence lengths, which is rare. We do, however, perform an analysis of the runtime of paged optimizers for 65B models on 48GB GPUs and find that with a batch size of 16, paged optimizers provide the same training speed as regular optimizers. Future work should measure and characterize under what circumstances slow-downs occur from the paging process.
4-bit NormalFloat yields better performance than 4-bit Floating Point While the 4-bit NormalFloat (NF4) data type is information-theoretically optimal, it still needs to be determined if this property translates to empirical advantages. We follow the setup from Dettmers and Zettlemoyer [13] where quantized LLMs (OPT [73], BLOOM [53], Pythia [7], LLaMA) of different sizes (125M to 65B) with different data types are evaluated on language modeling and a set of zero-shot tasks. In Figure 7 and Table 2 we see that NF4 improves performance significantly over FP4 and Int4 and that double quantization reduces the memory footprint without degrading performance.
Table 2: Pile Common Crawl mean perplexity for different data types for 125M to 13B OPT, BLOOM, LLaMA, and Pythia models.
Data type	Mean PPL
Int4	34.34
Float4 (E2M1)	31.07
Float4 (E3M0)	29.48
NFloat4 + DQ	27.41
k-bit QLoRA matches 16-bit full finetuning and 16-bit LoRA performance Recent findings have established that 4-bit quantization for inference is possible, but leads to performance degradation relative to 16-bit [13, 18]. This raises the crucial question of whether the lost performance can be recovered by conducting 4-bit adapter finetuning. We test this for two setups.
The first setup test if QLoRA can replicate 16-bit full finetuning performance for T5 and RoBERTa model finetuning. These results are detailed in the Appendix C.1. For our second setup, since full finetuning models at and beyond 11B parameters require more than one server of high memory GPUs, we continue to test whether 4-bit QLoRA can match 16-bit LoRA at the 7B to 65B parameter scales. To this end, we finetune LLaMA 7B through 65B on two instruction following datasets, Alpaca and FLAN v2, and evaluate on the MMLU benchmark via 5-shot accuracy. Results are shown in Table 3 where we see that NF4 with double quantization fully recovers the 16-bit LoRA MMLU performance. In addition, we also note that QLoRA with FP4 lags behind the 16-bit brain float LoRA baseline by about 1 percentage point. This corroborates both our findings that (1) QLoRA with NF4 replicates both 16-bit full finetuning and 16-bit LoRA finetuning performance, and (2) NF4 is superior to FP4 in terms of quantization precision.