Initial GGML model commit
Browse files
README.md
CHANGED
|
@@ -1,7 +1,6 @@
|
|
| 1 |
---
|
| 2 |
inference: false
|
| 3 |
license: other
|
| 4 |
-
thumbnail: https://huggingface.co/TheBloke/Flan-OpenLlama-7B-GGML/resolve/main/TheBlokeAI.header.800.jpg
|
| 5 |
---
|
| 6 |
|
| 7 |
<!-- header start -->
|
|
@@ -66,20 +65,20 @@ Refer to the Provided Files table below to see what files use which methods, and
|
|
| 66 |
## Provided files
|
| 67 |
| Name | Quant method | Bits | Size | Max RAM required | Use case |
|
| 68 |
| ---- | ---- | ---- | ---- | ---- | ----- |
|
| 69 |
-
|
|
| 70 |
-
|
|
| 71 |
-
|
|
| 72 |
-
|
|
| 73 |
-
|
|
| 74 |
-
|
|
| 75 |
-
|
|
| 76 |
-
|
|
| 77 |
-
|
|
| 78 |
-
|
|
| 79 |
-
|
|
| 80 |
-
|
|
| 81 |
-
|
|
| 82 |
-
|
|
| 83 |
|
| 84 |
**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
|
| 85 |
|
|
@@ -88,7 +87,7 @@ Refer to the Provided Files table below to see what files use which methods, and
|
|
| 88 |
I use the following command line; adjust for your tastes and needs:
|
| 89 |
|
| 90 |
```
|
| 91 |
-
./main -t 10 -ngl 32 -m
|
| 92 |
```
|
| 93 |
If you're able to use full GPU offloading, you should use `-t 1` to get best performance.
|
| 94 |
|
|
|
|
| 1 |
---
|
| 2 |
inference: false
|
| 3 |
license: other
|
|
|
|
| 4 |
---
|
| 5 |
|
| 6 |
<!-- header start -->
|
|
|
|
| 65 |
## Provided files
|
| 66 |
| Name | Quant method | Bits | Size | Max RAM required | Use case |
|
| 67 |
| ---- | ---- | ---- | ---- | ---- | ----- |
|
| 68 |
+
| flan-openllama-7b.ggmlv3.q2_K.bin | q2_K | 2 | 2.87 GB | 5.37 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
|
| 69 |
+
| flan-openllama-7b.ggmlv3.q3_K_L.bin | q3_K_L | 3 | 3.60 GB | 6.10 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
|
| 70 |
+
| flan-openllama-7b.ggmlv3.q3_K_M.bin | q3_K_M | 3 | 3.28 GB | 5.78 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
|
| 71 |
+
| flan-openllama-7b.ggmlv3.q3_K_S.bin | q3_K_S | 3 | 2.95 GB | 5.45 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
|
| 72 |
+
| flan-openllama-7b.ggmlv3.q4_0.bin | q4_0 | 4 | 3.79 GB | 6.29 GB | Original llama.cpp quant method, 4-bit. |
|
| 73 |
+
| flan-openllama-7b.ggmlv3.q4_1.bin | q4_1 | 4 | 4.21 GB | 6.71 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
|
| 74 |
+
| flan-openllama-7b.ggmlv3.q4_K_M.bin | q4_K_M | 4 | 4.08 GB | 6.58 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
|
| 75 |
+
| flan-openllama-7b.ggmlv3.q4_K_S.bin | q4_K_S | 4 | 3.83 GB | 6.33 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
|
| 76 |
+
| flan-openllama-7b.ggmlv3.q5_0.bin | q5_0 | 5 | 4.63 GB | 7.13 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
|
| 77 |
+
| flan-openllama-7b.ggmlv3.q5_1.bin | q5_1 | 5 | 5.06 GB | 7.56 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
|
| 78 |
+
| flan-openllama-7b.ggmlv3.q5_K_M.bin | q5_K_M | 5 | 4.78 GB | 7.28 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
|
| 79 |
+
| flan-openllama-7b.ggmlv3.q5_K_S.bin | q5_K_S | 5 | 4.65 GB | 7.15 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
|
| 80 |
+
| flan-openllama-7b.ggmlv3.q6_K.bin | q6_K | 6 | 5.53 GB | 8.03 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
|
| 81 |
+
| flan-openllama-7b.ggmlv3.q8_0.bin | q8_0 | 8 | 7.16 GB | 9.66 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
|
| 82 |
|
| 83 |
**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
|
| 84 |
|
|
|
|
| 87 |
I use the following command line; adjust for your tastes and needs:
|
| 88 |
|
| 89 |
```
|
| 90 |
+
./main -t 10 -ngl 32 -m flan-openllama-7b.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
|
| 91 |
```
|
| 92 |
If you're able to use full GPU offloading, you should use `-t 1` to get best performance.
|
| 93 |
|