Commit
•
9a32a04
1
Parent(s):
17ba454
Update README.md
Browse files
README.md
CHANGED
|
@@ -37,6 +37,20 @@ Quantized from fp16 with love.
|
|
| 37 |
|
| 38 |
Please note importance matrix quantizations are a work in progress, IQ3 and above is recommended for best results.
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
Original model card [here](https://huggingface.co/Envoid/Neophanis-8x7B)
|
| 41 |
|
| 42 |
---
|
|
|
|
| 37 |
|
| 38 |
Please note importance matrix quantizations are a work in progress, IQ3 and above is recommended for best results.
|
| 39 |
|
| 40 |
+
<b>Tip:</b> Pick a size that can fit in your GPU while still allowing some room for context for best speed from the table below. You may need to pad this further depending on if you are running image gen or TTS as well.
|
| 41 |
+
| Quant | Size (GB) | Comments |
|
| 42 |
+
|:-----|--------:|:------|
|
| 43 |
+
| [IQ2_S](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ2_S.gguf?download=true) | 14.1 | |
|
| 44 |
+
| [IQ2_M](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ2_M.gguf?download=true) | 15.5 | |
|
| 45 |
+
| [IQ3_XXS](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ3_XXS.gguf?download=true) | 18.2 | Better response quality than IQ2|
|
| 46 |
+
| [IQ3_XS](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ3_XS.gguf?download=true) | 19.3 | |
|
| 47 |
+
| [IQ3_S](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ3_S.gguf?download=true) | 20.4 | |
|
| 48 |
+
| [IQ3_M](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ3_M.gguf?download=true) | 21.4 | |
|
| 49 |
+
| [IQ4_XS](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-IQ4_XS.gguf?download=true) | 25.1 | Better quality than IQ3 or Q3_K_L (and below) |
|
| 50 |
+
| [Q4_K_M](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-Q4_K_M.gguf?download=true) | 28.4 | |
|
| 51 |
+
| [Q5_K_M](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-Q5_K_M.gguf?download=true) | 33.2 | |
|
| 52 |
+
| [Q6_K](https://huggingface.co/Quant-Cartel/Neophanis-8x7B-iMat-GGUF/resolve/main/Neophanis-8x7B-iMat-Q6_K.gguf?download=true) | 38.4 | |
|
| 53 |
+
|
| 54 |
Original model card [here](https://huggingface.co/Envoid/Neophanis-8x7B)
|
| 55 |
|
| 56 |
---
|