# Comparing Quantized Performance in Llama Models

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/71ccead2f23faf5b9b469140a41b52b896c20554d3b07147.png)

*Epistemic Status: Quick tests, most of this was done in less than 48 hours*

*TL;DR: Can you skimp on GPU VRAM?  8bit quantized seems fine, for 4bit it depends. *

I was asked by [@Teun van der Weij](https://www.lesswrong.com/users/teun-van-der-weij?mention=user), to what degree one can run evaluations on quantized models, and I was unsure. I have run some evaluations with Llama 3 and have some quick comparisons now. 

**Main Quantization Schemes**
-----------------------------

Here is a list of some different quantization schemes discussed:

*   **GGUF** \- Special file format used in Llama.cpp. Not supported in transformers.
*   **BNB** \- BitsAndBytes, the original default in huggingface transformers.
*   BNB **NF4** \- Alternative mode for bits and bytes, "*4-bit NormalFloat"*
*   **HQQ** \- Half-Quadratic Quantization, supports 1-8 bits.
*   **GPTQ** \- One of the older quantization methods. **Requires training data**
*   **AWQ -** "Activation-aware Weight Quantization". **Requires training data**

**Llama 2**
-----------

Some previous papers have compare perplexity of different methods. We can see an example of some research shown in the recent [research paper using HQQ quantization](https://mobiusml.github.io/hqq_blog/):

| **Method** | **nBits** | **Llama-2-7B** |   | **Llama-2-13B** |   | **Llama-2-70B** |   |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **PPL ↓** | **MEM ↓** | **PPL ↓** | **MEM ↓** | **PPL ↓** | **MEM ↓** |
| FP  | 16  | 5.18 | 13.5 | **4.63** | 25.6 | OOM | OOM |
| BNB | 8   | 5.22 | 7.9 | 4.67 | 14.4 | 3.17 | **68.15** |
| GPTQ_g128 | 8   | **5.19** | 7.8 | **4.63** | 14.8 | **3.12** | 74.87 |
| HQQ_g128 | 8   | **5.19** | **7.6** | **4.63** | **14** | **3.12** | 69.32 |
| BNB_g64 | 4   | 5.43 | 4.7 | 4.79 | 8.2 | 3.29 | 39.11 |
| GPTQ_g64 | 4   | 5.38 | 5   | 4.73 | 9.1 | 3.23 | 41.13 |
| AWQ_g64 | 4   | **5.28** | **4.6** | **4.7** | 8.5 | 3.2 | 37.08 |
| HQQ_g64 | 4   | 5.3 | **4.6** | **4.7** | 8.2 | **3.19** | 37.52 |

*Table 1: Comparison of perplexities at different levels of quantization with different methods on WikiText2 dataset for Llama 2 7b, 13b and 70b.*

However, it is quite unclear what effect this has on real world performance. 

**Llama 3 8B**
--------------

Here is some examples of what perplexity looks like with different levels of quantization on Llama 3 8B, [found from the llama.cpp repository](https://github.com/ggerganov/llama.cpp/tree/f98eb31c517c95960df1d0abc48002787f145f3b/examples/perplexity)[^84vp36zfwwv]. This only includes GGUF, and not any other quantization methods as far as I can tell.

| Type | Size (GB) | PPL | Mean Δp | RMS Δp |
| --- | --- | --- | --- | --- |
| f16 | 14.97 | 6.2331 | -   | -   |
| q8_0 | 7.96 | 6.2342 | -0.019 ± 0.003 % | 1.198 % |
| q6_K | 6.14 | 6.2533 | -0.007 ± 0.006 % | 2.295 % |
| q5\\_K\\_M | 5.33 | 6.2886 | -0.114 ± 0.008 % | 3.160 % |
| q5_0 | 5.21 | 6.3632 | -0.416 ± 0.012 % | 4.634 % |
| q4\\_K\\_M | 4.58 | 6.3830 | -0.389 ± 0.014 % | 5.251 % |
| q4_0 | 4.34 | 6.7001 | -1.588 ± 0.022 % | 8.434 % |

*Table 2: Comparison of perplexities at different levels of GGUF quantization on the WikiText2 dataset for Llama 3 8b.*

This is all fine and good, but a lot of use are trying to do interpretability and whatnot, and I personally have found this easiest when using the HuggingFace [transformers](https://github.com/huggingface/transformers) library. While for some it may make sense to retool, what about those of us who do not want to? While we can wait for [potential compatibility with GGUF](https://github.com/huggingface/transformers/issues/27712), there are other quantization methods we can use in the meantime, and I have tried to run some tests.

**Llama 3 8B Instruct**
-----------------------

Here are some benchmarks with Llama 3 8B, run with different quantization levels and schemes on MMLU, WMDP (% accuracy), and The Pile (perplexity):

gptq4 - Acc=55.21% PPL=8.575 T=233s  
awq4 - Acc=55.55% PPL=8.483 T=270s

| **Method** | **nBits** | **Llama-3-8B-Instruct** |   |   |   |   |
| --- | --- | --- | --- | --- | --- | --- |
| **MMLU** | **WMDP** | **The Pile (**100k tokens) |   |   |
| **↑**(0-shot) | **↑**(0-shot) | **PPL ↓** | **Acc↑** | **Time↓** |
| BFloat16 | 16  | **63.87 %** | **54.99%** | 8.283 | 56.52% | **53s**[^aoxh3qz8pmm] |
| Float16 | 16  | 63.84 % | 54.93% | 8.279 | 56.50% | 55s[^aoxh3qz8pmm] |
| Hqq Int8 | 8   | **63.87 %** | 54.66% | **8.298** | **56.49%** | 122s |
| BnB Int8 | 8   | 63.05 % | **54.96%** | 8.305 | 56.25% | **74s** |
| Hqq Int4 | 4   | **62.29 %** | 54.23% | **8.482** | **55.85%** | 130s |
| BnB NF4 | 4   | 61.44 % | **54.42%** | 8.499 | 55.80% | **95s** |
| BnB Int4 | 4   | 60.80 % | 52.73% | 8.633 | 55.19% | 277s |
| GPTQ Int4 | 4   | 61.58 % | 53.30% | 8.575 | 55.21% | 233s |
| AWQ Int4 | 4   | 61.84 % | **54.55%** | 8.483 | 55.55% | 270s |
| Hqq Int3 | 3   | **62.26 %** | **51.23%** | **8.872** | **54.49%** | **201s** |

We can see that the scored going from 16-bit to 8-bit are relatively unaffected, so likely it is fine to use 8bit. Going to 4-bit has a more noticeable effect, but it is not massively different on a qualitative level. 

### **Chain of Though**

OK, sure, maybe single-token prediction tasks are relatively unaffected, but Chain of Thought (CoT) requires many "correct" tokens at a time. Maybe these are affected more? I run some experiments on Minerva MATH Algebra dataset, with zero-shot and Chain-of-Thought. (I only use only a subset "Algebra" because CoT takes a long time to generate). Here are the results:

| **Method** | **nBits** | **Minerva MATH Algebra** |   |
| --- | --- | --- | --- |
| **↑**(0-shot CoT) | **Time↓** |
| BFloat16 | 16  | 37.2% | 2.3h[^aoxh3qz8pmm] |
| Float16 | 16  | **37.5%** | 2.3h[^aoxh3qz8pmm] |
| Hqq Int8 | 8   | **37.9%** | 15.0h** |
| BnB Int8 | 8   | 36.3% | 5.5h** |
| Hqq Int4 | 4   | **33.7 %** | 2.5h |
| BnB NF4 | 4   | 31.3 % | 3.2h |
| BnB Int4 | 4   | 29.3 % | 3.1h |
| GPTQ Int4 | 4   |     | DNF** |
| AWQ Int4 | 4   |     | DNF** |
| Hqq Int3 | 3   |     | DNF** |

Overall, we can see again that for 8-bit quantization, the effect on performance doesn't seem that large, though there is some noticeable degradation when going to 4-bits.

\*\* One thing to note is that the 8-bit implementations, for some reason, were rather slow. I think this is likely some problem I have with torch.compile / torch._dynamo (since the effects were not as noticeable for 4bits or for perplexity results) but I did not have time to test this. This may be because in some runtimes, the weights need to be de-quantized at every step. Also, these results were run on an (architecturally) older A4000 which does not support FP8/FP4 compute, so results may vary.

Conclusion
==========

*   It seems like it is basically fine to use fp16 models or int8 quantized models relatively interchangeably.
*   You may be able to get away with using 4-bit quantized models depending on the task, but there is some noticeable degradation.

[^84vp36zfwwv]: Note that some earlier implementations of GGUF with Llama 3 8B had some error when loading the tokeniser, and had much worse performance because of this. 

[^aoxh3qz8pmm]: Note that float16 and bfloat16 experiments were run on a dual-GPU setup, so the time may not be directly comparable