TheBloke commited on
Commit
2dd6be1
1 Parent(s): f1e6ad8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +20 -44
README.md CHANGED
@@ -21,12 +21,7 @@ license: other
21
 
22
  These files are GGML format model files for [OpenAccess AI Collective's Minotaur MPT 7B](https://huggingface.co/openaccess-ai-collective/minotaur-mpt-7b).
23
 
24
- GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
25
- * [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
26
- * [KoboldCpp](https://github.com/LostRuins/koboldcpp)
27
- * [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
28
- * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
29
- * [ctransformers](https://github.com/marella/ctransformers)
30
 
31
  ## Repositories available
32
 
@@ -34,32 +29,33 @@ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/gger
34
  * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/minotaur-mpt-7B-GGML)
35
  * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/openaccess-ai-collective/minotaur-mpt-7b)
36
 
37
- <!-- compatibility_ggml start -->
38
- ## Compatibility
39
-
40
- ### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
41
 
42
- I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
43
 
44
- They should be compatible with all current UIs and libraries that use llama.cpp, such as those listed at the top of this README.
 
 
 
45
 
46
- ### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
 
47
 
48
- These new quantisation methods are only compatible with llama.cpp as of June 6th, commit `2d43387`.
49
 
50
- They will NOT be compatible with koboldcpp, text-generation-ui, and other UIs and libraries yet. Support is expected to come over the next few days.
 
 
 
 
 
51
 
52
- ## Explanation of the new k-quant methods
53
 
54
- The new methods available are:
55
- * GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
56
- * GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
57
- * GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
58
- * GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
59
- * GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
60
- * GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
61
 
62
- Refer to the Provided Files table below to see what files use which methods, and how.
 
63
  <!-- compatibility_ggml end -->
64
 
65
  ## Provided files
@@ -71,26 +67,6 @@ Refer to the Provided Files table below to see what files use which methods, and
71
  | minotaur-mpt-7b.ggmlv3.q5_1.bin | q5_1 | 5 | 4.99 GB | 7.49 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
72
  | minotaur-mpt-7b.ggmlv3.q8_0.bin | q8_0 | 8 | 7.07 GB | 9.57 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
73
 
74
-
75
- **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
76
-
77
- ## How to run in `llama.cpp`
78
-
79
- I use the following command line; adjust for your tastes and needs:
80
-
81
- ```
82
- ./main -t 10 -ngl 32 -m minotaur-mpt-7b.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
83
- ```
84
- Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
85
-
86
- Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
87
-
88
- If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
89
-
90
- ## How to run in `text-generation-webui`
91
-
92
- Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
93
-
94
  <!-- footer start -->
95
  ## Discord
96
 
21
 
22
  These files are GGML format model files for [OpenAccess AI Collective's Minotaur MPT 7B](https://huggingface.co/openaccess-ai-collective/minotaur-mpt-7b).
23
 
24
+ Please note that these MPT GGMLs are **not compatbile with llama.cpp**. Please see below for a list of tools known to work with these model files.
 
 
 
 
 
25
 
26
  ## Repositories available
27
 
29
  * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/minotaur-mpt-7B-GGML)
30
  * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/openaccess-ai-collective/minotaur-mpt-7b)
31
 
32
+ ## Prompt template
 
 
 
33
 
34
+ Please see the "Original Model Card" below for more details about how to prompt this model.
35
 
36
+ ```
37
+ USER: prompt goes here
38
+ ASSISTANT:
39
+ ```
40
 
41
+ <!-- compatibility_ggml start -->
42
+ ## Compatibilty
43
 
44
+ These files are **not** compatible with llama.cpp.
45
 
46
+ Currently they can be used with:
47
+ * KoboldCpp, a powerful inference engine based on llama.cpp, with good UI: [KoboldCpp](https://github.com/LostRuins/koboldcpp)
48
+ * The ctransformers Python library, which includes LangChain support: [ctransformers](https://github.com/marella/ctransformers)
49
+ * The GPT4All-UI which uses ctransformers: [GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
50
+ * [rustformers' llm](https://github.com/rustformers/llm)
51
+ * The example `mpt` binary provided with [ggml](https://github.com/ggerganov/ggml)
52
 
53
+ As other options become available I will endeavour to update them here (do let me know in the Community tab if I've missed something!)
54
 
55
+ ## Tutorial for using GPT4All-UI
 
 
 
 
 
 
56
 
57
+ * [Text tutorial, written by **Lucas3DCG**](https://huggingface.co/TheBloke/MPT-7B-Storywriter-GGML/discussions/2#6475d914e9b57ce0caa68888)
58
+ * [Video tutorial, by GPT4All-UI's author **ParisNeo**](https://www.youtube.com/watch?v=ds_U0TDzbzI)
59
  <!-- compatibility_ggml end -->
60
 
61
  ## Provided files
67
  | minotaur-mpt-7b.ggmlv3.q5_1.bin | q5_1 | 5 | 4.99 GB | 7.49 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
68
  | minotaur-mpt-7b.ggmlv3.q8_0.bin | q8_0 | 8 | 7.07 GB | 9.57 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
69
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
  <!-- footer start -->
71
  ## Discord
72