Update for Transformers GPTQ support
Browse files- README.md +29 -18
- config.json +36 -25
- gptq_model-4bit-32g.safetensors → model.safetensors +2 -2
- quantize_config.json +1 -1
README.md
CHANGED
@@ -10,22 +10,29 @@ tags:
|
|
10 |
---
|
11 |
|
12 |
<!-- header start -->
|
13 |
-
|
14 |
-
|
|
|
15 |
</div>
|
16 |
<div style="display: flex; justify-content: space-between; width: 100%;">
|
17 |
<div style="display: flex; flex-direction: column; align-items: flex-start;">
|
18 |
-
<p><a href="https://discord.gg/theblokeai">Chat & support:
|
19 |
</div>
|
20 |
<div style="display: flex; flex-direction: column; align-items: flex-end;">
|
21 |
-
<p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
|
22 |
</div>
|
23 |
</div>
|
|
|
|
|
24 |
<!-- header end -->
|
25 |
|
26 |
-
#
|
|
|
|
|
27 |
|
28 |
-
|
|
|
|
|
29 |
|
30 |
Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.
|
31 |
|
@@ -34,7 +41,8 @@ Multiple GPTQ parameter permutations are provided; see Provided Files below for
|
|
34 |
|
35 |
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-GPTQ)
|
36 |
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-GGML)
|
37 |
-
* [
|
|
|
38 |
|
39 |
## Prompt template: Guanaco
|
40 |
|
@@ -51,13 +59,13 @@ Each separate quant is in a different branch. See below for instructions on fet
|
|
51 |
|
52 |
| Branch | Bits | Group Size | Act Order (desc_act) | File Size | ExLlama Compatible? | Made With | Description |
|
53 |
| ------ | ---- | ---------- | -------------------- | --------- | ------------------- | --------- | ----------- |
|
54 |
-
| main | 4 | 128 | False | 36.65 GB | True | AutoGPTQ | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. |
|
55 |
-
| gptq-4bit-32g-actorder_True | 4 | 32 | True | 40.66 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 32g gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. |
|
56 |
-
| gptq-4bit-64g-actorder_True | 4 | 64 | True | 37.99 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 64g uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
|
57 |
-
| gptq-4bit-128g-actorder_True | 4 | 128 | True | 36.65 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 128g uses even less VRAM, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
|
58 |
-
| gptq-3bit--1g-actorder_True | 3 | None | True | 26.78 GB | False | AutoGPTQ | 3-bit, with Act Order and no group size. Lowest possible VRAM requirements. May be lower quality than 3-bit 128g. |
|
59 |
-
| gptq-3bit-128g-actorder_False | 3 | 128 | False | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g but no act-order. Slightly higher VRAM requirements than 3-bit None. |
|
60 |
-
| gptq-3bit-128g-actorder_True | 3 | 128 | True | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g and act-order. Higher quality than 128g-False but poor AutoGPTQ CUDA speed. |
|
61 |
| gptq-3bit-64g-actorder_True | 3 | 64 | True | 29.30 GB | False | AutoGPTQ | 3-bit, with group size 64g and act-order. Highest quality 3-bit option. Poor AutoGPTQ CUDA speed. |
|
62 |
|
63 |
## How to download from branches
|
@@ -101,7 +109,7 @@ from transformers import AutoTokenizer, pipeline, logging
|
|
101 |
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
|
102 |
|
103 |
model_name_or_path = "TheBloke/llama-2-70b-Guanaco-QLoRA-GPTQ"
|
104 |
-
model_basename = "
|
105 |
|
106 |
use_triton = False
|
107 |
|
@@ -164,6 +172,7 @@ The files provided will work with AutoGPTQ (CUDA and Triton modes), GPTQ-for-LLa
|
|
164 |
ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
|
165 |
|
166 |
<!-- footer start -->
|
|
|
167 |
## Discord
|
168 |
|
169 |
For further support, and discussions on these models and AI in general, join us at:
|
@@ -183,16 +192,18 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
|
|
183 |
* Patreon: https://patreon.com/TheBlokeAI
|
184 |
* Ko-Fi: https://ko-fi.com/TheBlokeAI
|
185 |
|
186 |
-
**Special thanks to**:
|
187 |
|
188 |
-
**Patreon special mentions**:
|
189 |
|
190 |
|
191 |
Thank you to all my generous patrons and donaters!
|
192 |
|
|
|
|
|
193 |
<!-- footer end -->
|
194 |
|
195 |
-
# Original model card:
|
196 |
|
197 |
This is a Llama-2 version of [Guanaco](https://huggingface.co/timdettmers/guanaco-65b). It was finetuned from the base [Llama-70b](https://huggingface.co/meta-llama/Llama-2-70b-hf) model using the official training scripts found in the [QLoRA repo](https://github.com/artidoro/qlora). I wanted it to be as faithful as possible and therefore changed nothing in the training script beyond the model it was pointing to. The model prompt is therefore also the same as the original Guanaco model.
|
198 |
|
|
|
10 |
---
|
11 |
|
12 |
<!-- header start -->
|
13 |
+
<!-- 200823 -->
|
14 |
+
<div style="width: auto; margin-left: auto; margin-right: auto">
|
15 |
+
<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
|
16 |
</div>
|
17 |
<div style="display: flex; justify-content: space-between; width: 100%;">
|
18 |
<div style="display: flex; flex-direction: column; align-items: flex-start;">
|
19 |
+
<p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
|
20 |
</div>
|
21 |
<div style="display: flex; flex-direction: column; align-items: flex-end;">
|
22 |
+
<p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
|
23 |
</div>
|
24 |
</div>
|
25 |
+
<div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
|
26 |
+
<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
|
27 |
<!-- header end -->
|
28 |
|
29 |
+
# Llama2 70b Guanaco QLoRA - GPTQ
|
30 |
+
- Model creator: [Mikael110](https://huggingface.co/Mikael110)
|
31 |
+
- Original model: [Llama2 70b Guanaco QLoRA](https://huggingface.co/Mikael110/llama-2-70b-guanaco-qlora)
|
32 |
|
33 |
+
# Mikael110's Llama2 70b Guanaco QLoRA GPTQ
|
34 |
+
|
35 |
+
These files are GPTQ model files for [Mikael110's Llama2 70b Guanaco QLoRA](https://huggingface.co/Mikael110/llama-2-70b-guanaco-qlora).
|
36 |
|
37 |
Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.
|
38 |
|
|
|
41 |
|
42 |
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-GPTQ)
|
43 |
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-GGML)
|
44 |
+
* [Merged fp16 model, for GPU inference and further conversions](https://huggingface.co/TheBloke/llama-2-70b-Guanaco-QLoRA-fp16)
|
45 |
+
* [Mikael110's original QLoRA adapter](https://huggingface.co/Mikael110/llama-2-70b-guanaco-qlora)
|
46 |
|
47 |
## Prompt template: Guanaco
|
48 |
|
|
|
59 |
|
60 |
| Branch | Bits | Group Size | Act Order (desc_act) | File Size | ExLlama Compatible? | Made With | Description |
|
61 |
| ------ | ---- | ---------- | -------------------- | --------- | ------------------- | --------- | ----------- |
|
62 |
+
| main | 4 | 128 | False | 36.65 GB | True | AutoGPTQ | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. |
|
63 |
+
| gptq-4bit-32g-actorder_True | 4 | 32 | True | 40.66 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 32g gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. |
|
64 |
+
| gptq-4bit-64g-actorder_True | 4 | 64 | True | 37.99 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 64g uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
|
65 |
+
| gptq-4bit-128g-actorder_True | 4 | 128 | True | 36.65 GB | True | AutoGPTQ | 4-bit, with Act Order and group size. 128g uses even less VRAM, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
|
66 |
+
| gptq-3bit--1g-actorder_True | 3 | None | True | 26.78 GB | False | AutoGPTQ | 3-bit, with Act Order and no group size. Lowest possible VRAM requirements. May be lower quality than 3-bit 128g. |
|
67 |
+
| gptq-3bit-128g-actorder_False | 3 | 128 | False | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g but no act-order. Slightly higher VRAM requirements than 3-bit None. |
|
68 |
+
| gptq-3bit-128g-actorder_True | 3 | 128 | True | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g and act-order. Higher quality than 128g-False but poor AutoGPTQ CUDA speed. |
|
69 |
| gptq-3bit-64g-actorder_True | 3 | 64 | True | 29.30 GB | False | AutoGPTQ | 3-bit, with group size 64g and act-order. Highest quality 3-bit option. Poor AutoGPTQ CUDA speed. |
|
70 |
|
71 |
## How to download from branches
|
|
|
109 |
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
|
110 |
|
111 |
model_name_or_path = "TheBloke/llama-2-70b-Guanaco-QLoRA-GPTQ"
|
112 |
+
model_basename = "model"
|
113 |
|
114 |
use_triton = False
|
115 |
|
|
|
172 |
ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
|
173 |
|
174 |
<!-- footer start -->
|
175 |
+
<!-- 200823 -->
|
176 |
## Discord
|
177 |
|
178 |
For further support, and discussions on these models and AI in general, join us at:
|
|
|
192 |
* Patreon: https://patreon.com/TheBlokeAI
|
193 |
* Ko-Fi: https://ko-fi.com/TheBlokeAI
|
194 |
|
195 |
+
**Special thanks to**: Aemon Algiz.
|
196 |
|
197 |
+
**Patreon special mentions**: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter
|
198 |
|
199 |
|
200 |
Thank you to all my generous patrons and donaters!
|
201 |
|
202 |
+
And thank you again to a16z for their generous grant.
|
203 |
+
|
204 |
<!-- footer end -->
|
205 |
|
206 |
+
# Original model card: Mikael110's Llama2 70b Guanaco QLoRA
|
207 |
|
208 |
This is a Llama-2 version of [Guanaco](https://huggingface.co/timdettmers/guanaco-65b). It was finetuned from the base [Llama-70b](https://huggingface.co/meta-llama/Llama-2-70b-hf) model using the official training scripts found in the [QLoRA repo](https://github.com/artidoro/qlora). I wanted it to be as faithful as possible and therefore changed nothing in the training script beyond the model it was pointing to. The model prompt is therefore also the same as the original Guanaco model.
|
209 |
|
config.json
CHANGED
@@ -1,27 +1,38 @@
|
|
1 |
{
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
13 |
-
|
14 |
-
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
20 |
-
|
21 |
-
|
22 |
-
|
23 |
-
|
24 |
-
|
25 |
-
|
26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
27 |
}
|
|
|
1 |
{
|
2 |
+
"_name_or_path": "/workspace/process/lora_base/meta-llama_Llama-2-70b-hf",
|
3 |
+
"architectures": [
|
4 |
+
"LlamaForCausalLM"
|
5 |
+
],
|
6 |
+
"bos_token_id": 1,
|
7 |
+
"eos_token_id": 2,
|
8 |
+
"hidden_act": "silu",
|
9 |
+
"hidden_size": 8192,
|
10 |
+
"initializer_range": 0.02,
|
11 |
+
"intermediate_size": 28672,
|
12 |
+
"max_length": 4096,
|
13 |
+
"max_position_embeddings": 2048,
|
14 |
+
"model_type": "llama",
|
15 |
+
"num_attention_heads": 64,
|
16 |
+
"num_hidden_layers": 80,
|
17 |
+
"num_key_value_heads": 8,
|
18 |
+
"pad_token_id": 0,
|
19 |
+
"pretraining_tp": 1,
|
20 |
+
"rms_norm_eps": 1e-05,
|
21 |
+
"rope_scaling": null,
|
22 |
+
"tie_word_embeddings": false,
|
23 |
+
"torch_dtype": "float16",
|
24 |
+
"transformers_version": "4.32.0.dev0",
|
25 |
+
"use_cache": true,
|
26 |
+
"vocab_size": 32000,
|
27 |
+
"quantization_config": {
|
28 |
+
"bits": 4,
|
29 |
+
"group_size": 32,
|
30 |
+
"damp_percent": 0.01,
|
31 |
+
"desc_act": true,
|
32 |
+
"sym": true,
|
33 |
+
"true_sequential": true,
|
34 |
+
"model_name_or_path": null,
|
35 |
+
"model_file_base_name": "model",
|
36 |
+
"quant_method": "gptq"
|
37 |
+
}
|
38 |
}
|
gptq_model-4bit-32g.safetensors → model.safetensors
RENAMED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
-
size
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:ced41eb3f5c0d0346325853b82ab5412528a29e4de886c5ba6be107cbee39fb0
|
3 |
+
size 40663178600
|
quantize_config.json
CHANGED
@@ -6,5 +6,5 @@
|
|
6 |
"sym": true,
|
7 |
"true_sequential": true,
|
8 |
"model_name_or_path": null,
|
9 |
-
"model_file_base_name":
|
10 |
}
|
|
|
6 |
"sym": true,
|
7 |
"true_sequential": true,
|
8 |
"model_name_or_path": null,
|
9 |
+
"model_file_base_name": "model"
|
10 |
}
|