TheBloke commited on
Commit
8341f5f
1 Parent(s): f042129

Update for Transformers GPTQ support

Browse files
README.md CHANGED
@@ -14,17 +14,20 @@ tags:
14
  ---
15
 
16
  <!-- header start -->
17
- <div style="width: 100%;">
18
- <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
 
19
  </div>
20
  <div style="display: flex; justify-content: space-between; width: 100%;">
21
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
22
- <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
23
  </div>
24
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
25
- <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
26
  </div>
27
  </div>
 
 
28
  <!-- header end -->
29
 
30
  # Meta's Llama 2 13B-chat GPTQ
@@ -42,9 +45,21 @@ Multiple GPTQ parameter permutations are provided; see Provided Files below for
42
  ## Prompt template: Llama-2-Chat
43
 
44
  ```
45
- SYSTEM: You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
46
- USER: {prompt}
47
- ASSISTANT:
 
 
 
 
 
 
 
 
 
 
 
 
48
  ```
49
 
50
  ## Provided files
@@ -105,14 +120,14 @@ from transformers import AutoTokenizer, pipeline, logging
105
  from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
106
 
107
  model_name_or_path = "TheBloke/Llama-2-13B-chat-GPTQ"
108
- model_basename = "gptq_model-4bit-128g"
109
 
110
  use_triton = False
111
 
112
  tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
113
 
114
  model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
115
- model_basename=model_basename
116
  use_safetensors=True,
117
  trust_remote_code=True,
118
  device="cuda:0",
@@ -132,10 +147,12 @@ model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
132
  """
133
 
134
  prompt = "Tell me about AI"
135
- prompt_template=f'''SYSTEM: You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
136
- USER: {prompt}
137
- ASSISTANT:
138
- '''
 
 
139
 
140
  print("\n\n*** Generate:")
141
 
@@ -169,6 +186,7 @@ The files provided will work with AutoGPTQ (CUDA and Triton modes), GPTQ-for-LLa
169
  ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
170
 
171
  <!-- footer start -->
 
172
  ## Discord
173
 
174
  For further support, and discussions on these models and AI in general, join us at:
@@ -188,12 +206,15 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
188
  * Patreon: https://patreon.com/TheBlokeAI
189
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
190
 
191
- **Special thanks to**: Luke from CarbonQuill, Aemon Algiz.
 
 
192
 
193
- **Patreon special mentions**: Space Cruiser, Nikolai Manek, Sam, Chris McCloskey, Rishabh Srivastava, Kalila, Spiking Neurons AB, Khalefa Al-Ahmad, WelcomeToTheClub, Chadd, Lone Striker, Viktor Bowallius, Edmond Seymore, Ai Maven, Chris Smitley, Dave, Alexandros Triantafyllidis, Luke @flexchar, Elle, ya boyyy, Talal Aujan, Alex , Jonathan Leane, Deep Realms, Randy H, subjectnull, Preetika Verma, Joseph William Delisle, Michael Levine, chris gileta, K, Oscar Rangel, LangChain4j, Trenton Dambrowitz, Eugene Pentland, Johann-Peter Hartmann, Femi Adebogun, Illia Dulskyi, senxiiz, Daniel P. Andersen, Sean Connelly, Artur Olbinski, RoA, Mano Prime, Derek Yates, Raven Klaugh, David Flickinger, Willem Michiel, Pieter, Willian Hasse, vamX, Luke Pendergrass, webtim, Ghost , Rainer Wilmers, Nathan LeClaire, Will Dee, Cory Kujawski, John Detwiler, Fred von Graf, biorpg, Iucharbius , Imad Khwaja, Pierre Kircher, terasurfer , Asp the Wyvern, John Villwock, theTransient, zynix , Gabriel Tamborski, Fen Risland, Gabriel Puliatti, Matthew Berman, Pyrater, SuperWojo, Stephen Murray, Karl Bernard, Ajan Kanaga, Greatston Gnanesh, Junyu Yang.
194
 
195
  Thank you to all my generous patrons and donaters!
196
 
 
 
197
  <!-- footer end -->
198
 
199
  # Original model card: Meta's Llama 2 13B-chat
 
14
  ---
15
 
16
  <!-- header start -->
17
+ <!-- 200823 -->
18
+ <div style="width: auto; margin-left: auto; margin-right: auto">
19
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
20
  </div>
21
  <div style="display: flex; justify-content: space-between; width: 100%;">
22
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
23
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
24
  </div>
25
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
26
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
27
  </div>
28
  </div>
29
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
30
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
31
  <!-- header end -->
32
 
33
  # Meta's Llama 2 13B-chat GPTQ
 
45
  ## Prompt template: Llama-2-Chat
46
 
47
  ```
48
+ [INST] <<SYS>>
49
+ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
50
+ <</SYS>>
51
+
52
+ {prompt} [/INST]
53
+ ```
54
+
55
+ To continue a conversation:
56
+
57
+ ```
58
+ [INST] <<SYS>>
59
+ You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
60
+ <</SYS>>
61
+
62
+ {prompt} [/INST] {model_reply} [INST] {prompt} [/INST]
63
  ```
64
 
65
  ## Provided files
 
120
  from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
121
 
122
  model_name_or_path = "TheBloke/Llama-2-13B-chat-GPTQ"
123
+ model_basename = "model"
124
 
125
  use_triton = False
126
 
127
  tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
128
 
129
  model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
130
+ model_basename=model_basename,
131
  use_safetensors=True,
132
  trust_remote_code=True,
133
  device="cuda:0",
 
147
  """
148
 
149
  prompt = "Tell me about AI"
150
+ system_message = "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information."
151
+ prompt_template=f'''[INST] <<SYS>>
152
+ {system_message}
153
+ <</SYS>>
154
+
155
+ {prompt} [/INST]'''
156
 
157
  print("\n\n*** Generate:")
158
 
 
186
  ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
187
 
188
  <!-- footer start -->
189
+ <!-- 200823 -->
190
  ## Discord
191
 
192
  For further support, and discussions on these models and AI in general, join us at:
 
206
  * Patreon: https://patreon.com/TheBlokeAI
207
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
208
 
209
+ **Special thanks to**: Aemon Algiz.
210
+
211
+ **Patreon special mentions**: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter
212
 
 
213
 
214
  Thank you to all my generous patrons and donaters!
215
 
216
+ And thank you again to a16z for their generous grant.
217
+
218
  <!-- footer end -->
219
 
220
  # Original model card: Meta's Llama 2 13B-chat
config.json CHANGED
@@ -1,22 +1,33 @@
1
  {
2
- "architectures": [
3
- "LlamaForCausalLM"
4
- ],
5
- "bos_token_id": 1,
6
- "eos_token_id": 2,
7
- "hidden_act": "silu",
8
- "hidden_size": 5120,
9
- "initializer_range": 0.02,
10
- "intermediate_size": 13824,
11
- "max_position_embeddings": 2048,
12
- "model_type": "llama",
13
- "num_attention_heads": 40,
14
- "num_hidden_layers": 40,
15
- "pad_token_id": 0,
16
- "rms_norm_eps": 1e-05,
17
- "tie_word_embeddings": false,
18
- "torch_dtype": "float16",
19
- "transformers_version": "4.30.2",
20
- "use_cache": true,
21
- "vocab_size": 32000
 
 
 
 
 
 
 
 
 
 
 
22
  }
 
1
  {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 5120,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 13824,
11
+ "max_position_embeddings": 2048,
12
+ "model_type": "llama",
13
+ "num_attention_heads": 40,
14
+ "num_hidden_layers": 40,
15
+ "pad_token_id": 0,
16
+ "rms_norm_eps": 1e-05,
17
+ "tie_word_embeddings": false,
18
+ "torch_dtype": "float16",
19
+ "transformers_version": "4.30.2",
20
+ "use_cache": true,
21
+ "vocab_size": 32000,
22
+ "quantization_config": {
23
+ "bits": 4,
24
+ "group_size": 32,
25
+ "damp_percent": 0.01,
26
+ "desc_act": true,
27
+ "sym": true,
28
+ "true_sequential": true,
29
+ "model_name_or_path": null,
30
+ "model_file_base_name": "model",
31
+ "quant_method": "gptq"
32
+ }
33
  }
gptq_model-4bit-32g.safetensors → model.safetensors RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d51383c73cf6e77ed87016d5366b7d324a5e8087eb0eb7ad48c91661512e7a00
3
- size 8002873912
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:07d14698d0726fa31bde88f9943ac1e7db3e40283ed83a2973fd4fa08fa78f60
3
+ size 8002873968
quantize_config.json CHANGED
@@ -6,5 +6,5 @@
6
  "sym": true,
7
  "true_sequential": true,
8
  "model_name_or_path": null,
9
- "model_file_base_name": null
10
  }
 
6
  "sym": true,
7
  "true_sequential": true,
8
  "model_name_or_path": null,
9
+ "model_file_base_name": "model"
10
  }