| # Initial Testing 2024-04-25 | |
| Some folks are claiming there's something funky going on with GGUF quanting for Llama 3 models. I don't disagree. | |
| Some of those people are speculating that it has something to do with converting the raw weights from bf16 to fp16 instead | |
| of converting to fp32 as an intermediate step. I think that's bollocks. There is no logical or mathmatical justification for | |
| how that could possibly matter. | |
| So to test this crazy theory, I downloaded Undi95/Meta-Llama-3-8B-Instruct-hf and converted it to GGUF three ways: | |
| - fp16 specifically with `--outtype f16` | |
| - fp32 specifically with `--outtype f32` | |
| - "Auto" with no outtype specified | |
| I then quantized each of these conversions to Q4_K_M and ran perplexity tests on everything using my abbreviated wiki.short.raw | |
| text file. The results: | |
| ```` | |
| FP16 specified: size 14.9GB PPL @ fp16 9.5158 +/- 0.15418 PPL @ Q4km 9.6414 +/- 0.15494 | |
| FP32 specified: size 29.9GB PPL @ fp32 9.5158 +/- 0.15418 PPL @ Q4km 9.6278 +/- 0.15466 | |
| None specified: size 29.9GB PPL @ ???? 9.5158 +/- 0.15418 PPL @ Q4km 9.6278 +/- 0.15466 | |
| ```` | |
| As you can see, converting to fp32 has no meaningful effect on PPL compared to converting to fp16. PPL is identical at full weight, | |
| and the miniscule loss shown at Q4km is will within the margin of error. There will no doubt be some people who will claim | |
| "PpL iSn'T gOoD eNoUgH!!1!". For those people, I have uploaded all GGUFs used in this test. Feel free to use those files to do | |
| more extensive testing on your own time. I consider the matter resolved until somebody can conclusively demonstrate otherwise. | |
| # Continued Experiments 2024-05-11 | |
| As an imatrix enjoyer, it has been bugging me whether the precision of the quant used to generate the imatrix actually | |
| matters. Scuttlebut says "yes, but only a little". Logically, I don't think it should matter to a meaningful extent. PPL | |
| scales, so a value that is relatively important at fp16 should also register as relatively important at Q8 or even Q4. | |
| To test this theory properly, I took failspy/Llama-3-8B-Instruct-abliterated and converted it to GGUF in both fp16 and fp32 | |
| formats. I then quantized each of those GGUFs to both Q8_0 and Q4_0. I then generated imatrices for each of those six | |
| GGUFs. Then I created eight GGUFs quantized at Q4_k_m: | |
| - fp32 GGUF, fp32 imatrix | |
| - fp16 GGUF, fp16 imatrix | |
| - fp32 GGUF, fp32->Q8 imatrix | |
| - fp16 GGUF, fp16->Q8 imatrix | |
| - fp32 GGUF, fp32->Q4 imatrix | |
| - fp16 GGUF, fp16->Q4 imatrix | |
| - fp32 GGUF, no imatrix | |
| - fp16 GGUF, no imatrix | |
| I ran PPL against all 8 quants, as well as the full fp16 and fp32 GGUFs. All iMatrices were created using Kalomaze's | |
| groups_merged.txt. All PPL calcs were run using wiki.short.raw. Results: | |
| ```` | |
| GGUF PPL | |
| FP16 11.5923 | |
| FP32 11.5923 | |
| Q4km FP16 + FP16 imat 11.9326 | |
| Q4km FP32 + FP32 imat 11.9314 | |
| Q4km FP16 + Q8 imat 11.9369 | |
| Q4km FP32 + Q8 imat 11.9500 | |
| Q4km FP16 + Q4 imat 11.9355 | |
| Q4km FP32 + Q4 imat 11.9356 | |
| Q4km FP16 no imat 12.3612 | |
| Q4km FP32 no imat 12.3643 | |
| ```` | |
| Conclusion: | |
| Importance of quant size used to generate imatrix is borderline non-existant. Sort of. While the Q4km quant made with | |
| the fp32 GGUF and the fp32-generated imatrix was best, it was by such a miniscule margin that it is implausible that any | |
| difference between that (11.9314) and the Q4km made from the fp16 GGUF with the Q4_0-generaged imatrix (11.9355) could be detected | |
| under normal usage. The only counterintuitive result here is that the Q4_0-imat quants outperformed the Q8_0-imat quants. I cannot | |
| think of a reason why this should be the case. But as it seemingly *is* the case, I will be using Q4_0 as my intermediate step for | |
| generating imatrices in the future when the full fp16 model is too big for my measly 72GB of VRAM. |