Hugging Face's logo

TheBloke
/
GPT4All-13B-snoozy-GGML

Model card Files Community
4
TheBloke commited on
Commit
b2c96f5
•
1 Parent(s): 5b592de

Upload new k-quant GGML quantised models.

Browse files
Files changed (1) hide show
  1. README.md +96 -53
README.md CHANGED
@@ -1,11 +1,8 @@
1
  ---
2
- license: other
3
- datasets:
4
- - nomic-ai/gpt4all-j-prompt-generations
5
- language:
6
- - en
7
  inference: false
 
8
  ---
 
9
  <!-- header start -->
10
  <div style="width: 100%;">
11
  <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
@@ -20,46 +17,82 @@ inference: false
20
  </div>
21
  <!-- header end -->
22
 
23
- # GPT4All-13B-snoozy-GGML
24
 
25
- These files are GGML format model files of [Nomic.AI's GPT4all-13B-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy).
26
 
27
- GGML files are for CPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp).
 
 
 
 
 
28
 
29
  ## Repositories available
30
 
31
- * [4bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/GPT4ALL-13B-snoozy-GPTQ).
32
- * [4bit and 5bit GGML models for GPU inference](https://huggingface.co/TheBloke/GPT4ALL-13B-snoozy-GGML).
33
- * [Nomic.AI's original model in float32 HF for GPU inference](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy).
 
 
 
 
 
 
 
 
 
34
 
35
- ## THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)!
36
 
37
- llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508
38
 
39
- I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit `2d5db48` or later) to use them.
40
 
41
- For files compatible with the previous version of llama.cpp, please see branch `previous_llama_ggmlv2`.
 
 
 
 
 
 
 
 
 
 
 
42
 
43
  ## Provided files
44
- | Name | Quant method | Bits | Size | RAM required | Use case |
45
  | ---- | ---- | ---- | ---- | ---- | ----- |
46
- `GPT4All-13B-snoozy.ggmlv3.q4_0.bin` | q4_0 | 4bit | 8.14GB | 10.5GB | 4-bit. |
47
- `GPT4All-13B-snoozy.ggmlv3.q4_1.bin` | q4_1 | 4bit | 8.95GB | 11.5GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
48
- `GPT4All-13B-snoozy.ggmlv3.q5_0.bin` | q5_0 | 5bit | 8.95GB | 11.0GB | 5-bit. Higher accuracy, higher resource usage and slower inference. |
49
- `GPT4All-13B-snoozy.ggmlv3.q5_1.bin` | q5_1 | 5bit | 9.76GB | 12.25GB | 5-bit. Even higher accuracy, higher resource usage and slower inference. |
50
- `GPT4All-13B-snoozy.ggmlv3.q8_0.bin` | q5_1 | 5bit | 9.76GB | 17GB | 5-bit. Even higher accuracy, higher resource usage and slower inference. |
 
 
 
 
 
 
 
 
 
 
 
 
51
 
52
  ## How to run in `llama.cpp`
53
 
54
  I use the following command line; adjust for your tastes and needs:
55
 
56
  ```
57
- ./main -t 12 -m GPT4All-13B-snoozy.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.
58
- ### Instruction:
59
- Write a story about llamas
60
- ### Response:"
61
  ```
62
- Change `-t 12` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
 
 
63
 
64
  If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
65
 
@@ -87,14 +120,19 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
87
  * Patreon: https://patreon.com/TheBlokeAI
88
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
89
 
90
- **Patreon special mentions**: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman.
 
 
91
 
92
  Thank you to all my generous patrons and donaters!
 
93
  <!-- footer end -->
94
 
95
- # Original Model Card for GPT4All-13b-snoozy
 
 
96
 
97
- An Apache-2 licensed chatbot trained over a massive curated corpus of assistant interactions including word problems, multi-turn dialogue, code, poems, songs, and stories.
98
 
99
  ## Model Details
100
 
@@ -107,7 +145,7 @@ This model has been finetuned from LLama 13B
107
  - **Developed by:** [Nomic AI](https://home.nomic.ai)
108
  - **Model Type:** A finetuned LLama 13B model on assistant style interaction data
109
  - **Language(s) (NLP):** English
110
- - **License:** Apache-2
111
  - **Finetuned from model [optional]:** LLama 13B
112
 
113
  This model was trained on `nomic-ai/gpt4all-j-prompt-generations` using `revision=v1.3-groovy`
@@ -126,27 +164,32 @@ This model was trained on `nomic-ai/gpt4all-j-prompt-generations` using `revisio
126
  Results on common sense reasoning benchmarks
127
 
128
  ```
129
- Model BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
130
- ----------------------- ---------- ---------- ----------- ------------ ---------- ---------- ----------
131
- GPT4All-J 6B v1.0 73.4 74.8 63.4 64.7 54.9 36.0 40.2
132
- GPT4All-J v1.1-breezy 74.0 75.1 63.2 63.6 55.4 34.9 38.4
133
- GPT4All-J v1.2-jazzy 74.8 74.9 63.6 63.8 56.6 35.3 41.0
134
- GPT4All-J v1.3-groovy 73.6 74.3 63.8 63.5 57.7 35.0 38.8
135
- GPT4All-J Lora 6B 68.6 75.8 66.2 63.5 56.4 35.7 40.2
136
- GPT4All LLaMa Lora 7B 73.1 77.6 72.1 67.8 51.1 40.4 40.2
137
- GPT4All 13B snoozy *83.3* 79.2 75.0 *71.3* 60.9 44.2 43.4
138
- Dolly 6B 68.8 77.3 67.6 63.9 62.9 38.7 41.2
139
- Dolly 12B 56.7 75.4 71.0 62.2 *64.6* 38.5 40.4
140
- Alpaca 7B 73.9 77.2 73.9 66.1 59.8 43.3 43.4
141
- Alpaca Lora 7B 74.3 *79.3* 74.0 68.8 56.6 43.9 42.6
142
- GPT-J 6B 65.4 76.2 66.2 64.1 62.2 36.6 38.2
143
- LLama 7B 73.1 77.4 73.0 66.9 52.5 41.4 42.4
144
- LLama 13B 68.5 79.1 *76.2* 70.1 60.0 *44.6* 42.2
145
- Pythia 6.9B 63.5 76.3 64.0 61.1 61.3 35.2 37.2
146
- Pythia 12B 67.7 76.6 67.3 63.8 63.9 34.8 38.0
147
- Vicuña T5 81.5 64.6 46.3 61.8 49.3 33.3 39.4
148
- Vicuña 13B 81.5 76.8 73.3 66.7 57.4 42.7 43.6
149
- Stable Vicuña RLHF 82.3 78.6 74.1 70.9 61.0 43.5 *44.4*
150
- StableLM Tuned 62.5 71.2 53.6 54.8 52.4 31.1 33.4
151
- StableLM Base 60.1 67.4 41.2 50.1 44.9 27.0 32.0
 
 
 
 
 
152
  ```
 
1
  ---
 
 
 
 
 
2
  inference: false
3
+ license: other
4
  ---
5
+
6
  <!-- header start -->
7
  <div style="width: 100%;">
8
  <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
 
17
  </div>
18
  <!-- header end -->
19
 
20
+ # Nomic.AI's GPT4All-13B-snoozy GGML
21
 
22
+ These files are GGML format model files for [Nomic.AI's GPT4All-13B-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy).
23
 
24
+ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
25
+ * [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
26
+ * [KoboldCpp](https://github.com/LostRuins/koboldcpp)
27
+ * [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
28
+ * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
29
+ * [ctransformers](https://github.com/marella/ctransformers)
30
 
31
  ## Repositories available
32
 
33
+ * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/GPT4All-13B-snoozy-GPTQ)
34
+ * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/GPT4All-13B-snoozy-GGML)
35
+ * [Unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy)
36
+
37
+ <!-- compatibility_ggml start -->
38
+ ## Compatibility
39
+
40
+ ### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
41
+
42
+ I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
43
+
44
+ They should be compatible with all current UIs and libraries that use llama.cpp, such as those listed at the top of this README.
45
 
46
+ ### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
47
 
48
+ These new quantisation methods are only compatible with llama.cpp as of June 6th, commit `2d43387`.
49
 
50
+ They will NOT be compatible with koboldcpp, text-generation-ui, and other UIs and libraries yet. Support is expected to come over the next few days.
51
 
52
+ ## Explanation of the new k-quant methods
53
+
54
+ The new methods available are:
55
+ * GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
56
+ * GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
57
+ * GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
58
+ * GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
59
+ * GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
60
+ * GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
61
+
62
+ Refer to the Provided Files table below to see what files use which methods, and how.
63
+ <!-- compatibility_ggml end -->
64
 
65
  ## Provided files
66
+ | Name | Quant method | Bits | Size | Max RAM required | Use case |
67
  | ---- | ---- | ---- | ---- | ---- | ----- |
68
+ | GPT4All-13B-snoozy.ggmlv3.q2_K.bin | q2_K | 2 | 5.43 GB | 7.93 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
69
+ | GPT4All-13B-snoozy.ggmlv3.q3_K_L.bin | q3_K_L | 3 | 6.87 GB | 9.37 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
70
+ | GPT4All-13B-snoozy.ggmlv3.q3_K_M.bin | q3_K_M | 3 | 6.25 GB | 8.75 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
71
+ | GPT4All-13B-snoozy.ggmlv3.q3_K_S.bin | q3_K_S | 3 | 5.59 GB | 8.09 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
72
+ | GPT4All-13B-snoozy.ggmlv3.q4_0.bin | q4_0 | 4 | 7.32 GB | 9.82 GB | Original llama.cpp quant method, 4-bit. |
73
+ | GPT4All-13B-snoozy.ggmlv3.q4_1.bin | q4_1 | 4 | 8.14 GB | 10.64 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
74
+ | GPT4All-13B-snoozy.ggmlv3.q4_K_M.bin | q4_K_M | 4 | 7.82 GB | 10.32 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
75
+ | GPT4All-13B-snoozy.ggmlv3.q4_K_S.bin | q4_K_S | 4 | 7.32 GB | 9.82 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
76
+ | GPT4All-13B-snoozy.ggmlv3.q5_0.bin | q5_0 | 5 | 8.95 GB | 11.45 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
77
+ | GPT4All-13B-snoozy.ggmlv3.q5_1.bin | q5_1 | 5 | 9.76 GB | 12.26 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
78
+ | GPT4All-13B-snoozy.ggmlv3.q5_K_M.bin | q5_K_M | 5 | 9.21 GB | 11.71 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
79
+ | GPT4All-13B-snoozy.ggmlv3.q5_K_S.bin | q5_K_S | 5 | 8.95 GB | 11.45 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
80
+ | GPT4All-13B-snoozy.ggmlv3.q6_K.bin | q6_K | 6 | 10.68 GB | 13.18 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
81
+ | GPT4All-13B-snoozy.ggmlv3.q8_0.bin | q8_0 | 8 | 13.83 GB | 16.33 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
82
+
83
+
84
+ **Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
85
 
86
  ## How to run in `llama.cpp`
87
 
88
  I use the following command line; adjust for your tastes and needs:
89
 
90
  ```
91
+ ./main -t 10 -ngl 32 -m GPT4All-13B-snoozy.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
 
 
 
92
  ```
93
+ Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
94
+
95
+ Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
96
 
97
  If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
98
 
 
120
  * Patreon: https://patreon.com/TheBlokeAI
121
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
122
 
123
+ **Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
124
+
125
+ **Patreon special mentions**: Ajan Kanaga, Kalila, Derek Yates, Sean Connelly, Luke, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, trip7s trip, Jonathan Leane, Talal Aujan, Artur Olbinski, Cory Kujawski, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Johann-Peter Hartmann.
126
 
127
  Thank you to all my generous patrons and donaters!
128
+
129
  <!-- footer end -->
130
 
131
+ # Original model card: Nomic.AI's GPT4All-13B-snoozy
132
+
133
+ # Model Card for GPT4All-13b-snoozy
134
 
135
+ A GPL licensed chatbot trained over a massive curated corpus of assistant interactions including word problems, multi-turn dialogue, code, poems, songs, and stories.
136
 
137
  ## Model Details
138
 
 
145
  - **Developed by:** [Nomic AI](https://home.nomic.ai)
146
  - **Model Type:** A finetuned LLama 13B model on assistant style interaction data
147
  - **Language(s) (NLP):** English
148
+ - **License:** GPL
149
  - **Finetuned from model [optional]:** LLama 13B
150
 
151
  This model was trained on `nomic-ai/gpt4all-j-prompt-generations` using `revision=v1.3-groovy`
 
164
  Results on common sense reasoning benchmarks
165
 
166
  ```
167
+ | Model | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |
168
+ |:--------------------------|:--------:|:--------:|:---------:|:----------:|:--------:|:--------:|:--------:|:--------:|
169
+ | GPT4All-J 6B v1.0 | 73.4 | 74.8 | 63.4 | 64.7 | 54.9 | 36.0 | 40.2 | 58.2 |
170
+ | GPT4All-J v1.1-breezy | 74.0 | 75.1 | 63.2 | 63.6 | 55.4 | 34.9 | 38.4 | 57.8 |
171
+ | GPT4All-J v1.2-jazzy | 74.8 | 74.9 | 63.6 | 63.8 | 56.6 | 35.3 | 41.0 | 58.6 |
172
+ | GPT4All-J v1.3-groovy | 73.6 | 74.3 | 63.8 | 63.5 | 57.7 | 35.0 | 38.8 | 58.1 |
173
+ | GPT4All-J Lora 6B | 68.6 | 75.8 | 66.2 | 63.5 | 56.4 | 35.7 | 40.2 | 58.1 |
174
+ | GPT4All LLaMa Lora 7B | 73.1 | 77.6 | 72.1 | 67.8 | 51.1 | 40.4 | 40.2 | 60.3 |
175
+ | GPT4All 13B snoozy | **83.3** | 79.2 | 75.0 | **71.3** | 60.9 | 44.2 | 43.4 | **65.3** |
176
+ | Dolly 6B | 68.8 | 77.3 | 67.6 | 63.9 | 62.9 | 38.7 | 41.2 | 60.1 |
177
+ | Dolly 12B | 56.7 | 75.4 | 71.0 | 62.2 | 64.6 | 38.5 | 40.4 | 58.4 |
178
+ | Alpaca 7B | 73.9 | 77.2 | 73.9 | 66.1 | 59.8 | 43.3 | 43.4 | 62.4 |
179
+ | Alpaca Lora 7B | 74.3 | **79.3** | 74.0 | 68.8 | 56.6 | 43.9 | 42.6 | 62.8 |
180
+ | GPT-J 6.7B | 65.4 | 76.2 | 66.2 | 64.1 | 62.2 | 36.6 | 38.2 | 58.4 |
181
+ | LLama 7B | 73.1 | 77.4 | 73.0 | 66.9 | 52.5 | 41.4 | 42.4 | 61.0 |
182
+ | LLama 13B | 68.5 | 79.1 | 76.2 | 70.1 | 60.0 | **44.6** | 42.2 | 63.0 |
183
+ | Pythia 6.7B | 63.5 | 76.3 | 64.0 | 61.1 | 61.3 | 35.2 | 37.2 | 57.0 |
184
+ | Pythia 12B | 67.7 | 76.6 | 67.3 | 63.8 | 63.9 | 34.8 | 38 | 58.9 |
185
+ | Fastchat T5 | 81.5 | 64.6 | 46.3 | 61.8 | 49.3 | 33.3 | 39.4 | 53.7 |
186
+ | Fastchat Vicuña 7B | 76.6 | 77.2 | 70.7 | 67.3 | 53.5 | 41.2 | 40.8 | 61.0 |
187
+ | Fastchat Vicuña 13B | 81.5 | 76.8 | 73.3 | 66.7 | 57.4 | 42.7 | 43.6 | 63.1 |
188
+ | StableVicuña RLHF | 82.3 | 78.6 | 74.1 | 70.9 | 61.0 | 43.5 | **44.4** | 65.0 |
189
+ | StableLM Tuned | 62.5 | 71.2 | 53.6 | 54.8 | 52.4 | 31.1 | 33.4 | 51.3 |
190
+ | StableLM Base | 60.1 | 67.4 | 41.2 | 50.1 | 44.9 | 27.0 | 32.0 | 42.2 |
191
+ | Koala 13B | 76.5 | 77.9 | 72.6 | 68.8 | 54.3 | 41.0 | 42.8 | 62.0 |
192
+ | Open Assistant Pythia 12B | 67.9 | 78.0 | 68.1 | 65.0 | 64.2 | 40.4 | 43.2 | 61.0 |
193
+ | Mosaic mpt-7B | 74.8 | **79.3** | **76.3** | 68.6 | **70.0** | 42.2 | 42.6 | 64.8 |
194
+ | text-davinci-003 | 88.1 | 83.8 | 83.4 | 75.8 | 83.9 | 63.9 | 51.0 | 75.7 |
195
  ```