Add k-quant GGMLs which are now working thanks to LostRuins PR

README.md

@@ -31,15 +31,17 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
 ## Repositories available
 
 * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/WizardLM-13B-V1.1-GPTQ)
-* [4, 5, and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/WizardLM-13B-V1.1-GGML)
+* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/WizardLM-13B-V1.1-GGML)
 * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/WizardLM/WizardLM-13B-V1.1)
 
 ## Prompt template: Vicuna
 
 ```
 A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
-USER: prompt
+
+USER: PROMPT
 ASSISTANT:
+
 ```
 
 <!-- compatibility_ggml start -->
@@ -47,28 +49,43 @@ ASSISTANT:
 
 ### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
 
-I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
+These are guaranteed to be compatible with any UIs, tools and libraries released since late May. They may be phased out soon, as they are largely superseded by the new k-quant methods.
 
-These are guaranteed to be compatbile with any UIs, tools and libraries released since late May.
+### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
 
-### New k-quant methods: not supported at the moment due to model's vocab size
+These new quantisation methods are compatible with llama.cpp as of June 6th, commit `2d43387`.
 
-Unfortunately it is not possible to make the new k-quant format quantisations for this model at this time.
+They are now also compatible with recent releases of text-generation-webui, KoboldCpp, llama-cpp-python, ctransformers, rustformers and most others. For compatibility with other tools and libraries, please check their documentation.
 
-This is because the model uses a non-standard vocab size of 32,001, which is not divisible by 256.
+## Explanation of the new k-quant methods
+<details>
+  <summary>Click to see details</summary>
 
-This is being investigated by the llama.cpp team and may be fixed in future. You can read more about that here: https://github.com/ggerganov/llama.cpp/issues/1919
+The new methods available are:
+* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
+* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
+* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
+* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
+* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
+* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
 
+Refer to the Provided Files table below to see what files use which methods, and how.
+</details>
 <!-- compatibility_ggml end -->
 
 ## Provided files
 | Name | Quant method | Bits | Size | Max RAM required | Use case |
 | ---- | ---- | ---- | ---- | ---- | ----- |
-| wizardlm-13b-v1.1.ggmlv3.q4_0.bin | q4_0 | 4 | 7.32 GB | 9.82 GB | Original llama.cpp quant method, 4-bit. |
-| wizardlm-13b-v1.1.ggmlv3.q4_1.bin | q4_1 | 4 | 8.14 GB | 10.64 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
-| wizardlm-13b-v1.1.ggmlv3.q5_0.bin | q5_0 | 5 | 8.95 GB | 11.45 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
-| wizardlm-13b-v1.1.ggmlv3.q5_1.bin | q5_1 | 5 | 9.76 GB | 12.26 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
-| wizardlm-13b-v1.1.ggmlv3.q8_0.bin | q8_0 | 8 | 13.83 GB | 16.33 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
+| wizardlm-13b-v1.1.ggmlv3.q2_K.bin | q2_K | 2 | 5.67 GB| 8.17 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
+| wizardlm-13b-v1.1.ggmlv3.q3_K_S.bin | q3_K_S | 3 | 5.80 GB| 8.30 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
+| wizardlm-13b-v1.1.ggmlv3.q3_K_M.bin | q3_K_M | 3 | 6.46 GB| 8.96 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| wizardlm-13b-v1.1.ggmlv3.q3_K_L.bin | q3_K_L | 3 | 7.07 GB| 9.57 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
+| wizardlm-13b-v1.1.ggmlv3.q4_K_S.bin | q4_K_S | 4 | 7.49 GB| 9.99 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
+| wizardlm-13b-v1.1.ggmlv3.q4_K_M.bin | q4_K_M | 4 | 7.99 GB| 10.49 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
+| wizardlm-13b-v1.1.ggmlv3.q5_K_S.bin | q5_K_S | 5 | 9.07 GB| 11.57 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
+| wizardlm-13b-v1.1.ggmlv3.q5_K_M.bin | q5_K_M | 5 | 9.33 GB| 11.83 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
+| wizardlm-13b-v1.1.ggmlv3.q6_K.bin | q6_K | 6 | 10.76 GB| 13.26 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
+
 
 **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
 
@@ -77,11 +94,9 @@ This is being investigated by the llama.cpp team and may be fixed in future. You
 I use the following command line; adjust for your tastes and needs:
 
 ```
-./main -t 10 -ngl 32 -m wizardlm-13b-v1.1.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
+./main -t 10 -ngl 32 -m wizardlm-13b-v1.1.ggmlv3.q4_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
 ```
-If you're able to use full GPU offloading, you should use `-t 1` to get best performance.
-
-If not able to fully offload to GPU, you should use more cores. Change `-t 10` to the number of physical CPU cores you have, or a lower number depending on what gives best performance.
+Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
 
 Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.