TheBloke commited on
Commit
a94af5b
1 Parent(s): cac9c5d

Update for Transformers GPTQ support

Browse files
README.md CHANGED
@@ -4,24 +4,27 @@ language:
4
  - en
5
  datasets:
6
  - guanaco
7
- model_hub_library:
8
  - transformers
9
- license:
10
  - apache-2.0
11
  ---
12
 
13
  <!-- header start -->
14
- <div style="width: 100%;">
15
- <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
 
16
  </div>
17
  <div style="display: flex; justify-content: space-between; width: 100%;">
18
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
19
- <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
20
  </div>
21
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
22
- <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
23
  </div>
24
  </div>
 
 
25
  <!-- header end -->
26
 
27
  # LoupGarou's WizardCoder Guanaco 15B V1.0 GPTQ
@@ -147,6 +150,7 @@ It was created with group_size 128 to increase inference accuracy, but without -
147
  * Parameters: Groupsize = 128. Act Order / desc_act = False.
148
 
149
  <!-- footer start -->
 
150
  ## Discord
151
 
152
  For further support, and discussions on these models and AI in general, join us at:
@@ -166,12 +170,15 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
166
  * Patreon: https://patreon.com/TheBlokeAI
167
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
168
 
169
- **Special thanks to**: Luke from CarbonQuill, Aemon Algiz.
 
 
170
 
171
- **Patreon special mentions**: RoA, Lone Striker, Gabriel Puliatti, Derek Yates, Randy H, Jonathan Leane, Eugene Pentland, Karl Bernard, Viktor Bowallius, senxiiz, Daniel P. Andersen, Pierre Kircher, Deep Realms, Cory Kujawski, Oscar Rangel, Fen Risland, Ajan Kanaga, LangChain4j, webtim, Nikolai Manek, Trenton Dambrowitz, Raven Klaugh, Kalila, Khalefa Al-Ahmad, Chris McCloskey, Luke @flexchar, Ai Maven, Dave, Asp the Wyvern, Sean Connelly, Imad Khwaja, Space Cruiser, Rainer Wilmers, subjectnull, Alps Aficionado, Willian Hasse, Fred von Graf, Artur Olbinski, Johann-Peter Hartmann, WelcomeToTheClub, Willem Michiel, Michael Levine, Iucharbius , Spiking Neurons AB, K, biorpg, John Villwock, Pyrater, Greatston Gnanesh, Mano Prime, Junyu Yang, Stephen Murray, John Detwiler, Luke Pendergrass, terasurfer , Pieter, zynix , Edmond Seymore, theTransient, Nathan LeClaire, vamX, Kevin Schuppel, Preetika Verma, ya boyyy, Alex , SuperWojo, Ghost , Joseph William Delisle, Matthew Berman, Talal Aujan, chris gileta, Illia Dulskyi.
172
 
173
  Thank you to all my generous patrons and donaters!
174
 
 
 
175
  <!-- footer end -->
176
 
177
  # Original model card: LoupGarou's WizardCoder Guanaco 15B V1.0
@@ -211,7 +218,7 @@ class Chatbot:
211
  if next(self.model.parameters()).is_cuda:
212
  inputs = {name: tensor.to('cuda') for name, tensor in inputs.items()}
213
  start_time = time.time()
214
- tokens = self.model.generate(input_ids=inputs['input_ids'],
215
  attention_mask=inputs['attention_mask'],
216
  pad_token_id=self.tokenizer.pad_token_id,
217
  max_new_tokens=400)
@@ -241,8 +248,8 @@ if __name__ == "__main__":
241
  The base WizardCoder model was finetuned on the openassistant-guanaco dataset using QLORA, which was trimmed to within 2 standard deviations of token size for question sets and randomized. All non-English data was also removed from this finetuning dataset.
242
 
243
  ## Acknowledgements
244
- This model, WizardCoder-Guanaco-15B-V1.0, is simply building on the efforts of two great teams to evaluate the performance of a combined model with the strengths of the [WizardCoder base model](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and the [openassistant-guanaco dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
245
 
246
- A sincere appreciation goes out to the developers and the community involved in the creation and refinement of these models. Their commitment to providing open source tools and datasets have been instrumental in making this project a reality.
247
 
248
  Moreover, a special note of thanks to the [Hugging Face](https://huggingface.co/) team, whose transformative library has not only streamlined the process of model creation and adaptation, but also democratized the access to state-of-the-art machine learning technologies. Their impact on the development of this project cannot be overstated.
 
4
  - en
5
  datasets:
6
  - guanaco
7
+ model_hub_library:
8
  - transformers
9
+ license:
10
  - apache-2.0
11
  ---
12
 
13
  <!-- header start -->
14
+ <!-- 200823 -->
15
+ <div style="width: auto; margin-left: auto; margin-right: auto">
16
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
17
  </div>
18
  <div style="display: flex; justify-content: space-between; width: 100%;">
19
  <div style="display: flex; flex-direction: column; align-items: flex-start;">
20
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
21
  </div>
22
  <div style="display: flex; flex-direction: column; align-items: flex-end;">
23
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
24
  </div>
25
  </div>
26
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
27
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
28
  <!-- header end -->
29
 
30
  # LoupGarou's WizardCoder Guanaco 15B V1.0 GPTQ
 
150
  * Parameters: Groupsize = 128. Act Order / desc_act = False.
151
 
152
  <!-- footer start -->
153
+ <!-- 200823 -->
154
  ## Discord
155
 
156
  For further support, and discussions on these models and AI in general, join us at:
 
170
  * Patreon: https://patreon.com/TheBlokeAI
171
  * Ko-Fi: https://ko-fi.com/TheBlokeAI
172
 
173
+ **Special thanks to**: Aemon Algiz.
174
+
175
+ **Patreon special mentions**: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter
176
 
 
177
 
178
  Thank you to all my generous patrons and donaters!
179
 
180
+ And thank you again to a16z for their generous grant.
181
+
182
  <!-- footer end -->
183
 
184
  # Original model card: LoupGarou's WizardCoder Guanaco 15B V1.0
 
218
  if next(self.model.parameters()).is_cuda:
219
  inputs = {name: tensor.to('cuda') for name, tensor in inputs.items()}
220
  start_time = time.time()
221
+ tokens = self.model.generate(input_ids=inputs['input_ids'],
222
  attention_mask=inputs['attention_mask'],
223
  pad_token_id=self.tokenizer.pad_token_id,
224
  max_new_tokens=400)
 
248
  The base WizardCoder model was finetuned on the openassistant-guanaco dataset using QLORA, which was trimmed to within 2 standard deviations of token size for question sets and randomized. All non-English data was also removed from this finetuning dataset.
249
 
250
  ## Acknowledgements
251
+ This model, WizardCoder-Guanaco-15B-V1.0, is simply building on the efforts of two great teams to evaluate the performance of a combined model with the strengths of the [WizardCoder base model](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and the [openassistant-guanaco dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
252
 
253
+ A sincere appreciation goes out to the developers and the community involved in the creation and refinement of these models. Their commitment to providing open source tools and datasets have been instrumental in making this project a reality.
254
 
255
  Moreover, a special note of thanks to the [Hugging Face](https://huggingface.co/) team, whose transformative library has not only streamlined the process of model creation and adaptation, but also democratized the access to state-of-the-art machine learning technologies. Their impact on the development of this project cannot be overstated.
config.json CHANGED
@@ -1,39 +1,50 @@
1
  {
2
- "_name_or_path": "bigcode/starcoder",
3
- "activation_function": "gelu",
4
- "architectures": [
5
- "GPTBigCodeForCausalLM"
6
- ],
7
- "attention_softmax_in_fp32": true,
8
- "attn_pdrop": 0.1,
9
- "bos_token_id": 0,
10
- "embd_pdrop": 0.1,
11
- "eos_token_id": 0,
12
- "inference_runner": 0,
13
- "initializer_range": 0.02,
14
- "layer_norm_epsilon": 1e-05,
15
- "max_batch_size": null,
16
- "max_sequence_length": null,
17
- "model_type": "gpt_bigcode",
18
- "multi_query": true,
19
- "n_embd": 6144,
20
- "n_head": 48,
21
- "n_inner": 24576,
22
- "n_layer": 40,
23
- "n_positions": 8192,
24
- "pad_key_length": true,
25
- "pre_allocate_kv_cache": false,
26
- "resid_pdrop": 0.1,
27
- "scale_attention_softmax_in_fp32": true,
28
- "scale_attn_weights": true,
29
- "summary_activation": null,
30
- "summary_first_dropout": 0.1,
31
- "summary_proj_to_labels": true,
32
- "summary_type": "cls_index",
33
- "summary_use_proj": true,
34
- "torch_dtype": "float16",
35
- "transformers_version": "4.30.0.dev0",
36
- "use_cache": true,
37
- "validate_runner_input": true,
38
- "vocab_size": 49153
 
 
 
 
 
 
 
 
 
 
 
39
  }
 
1
  {
2
+ "_name_or_path": "bigcode/starcoder",
3
+ "activation_function": "gelu",
4
+ "architectures": [
5
+ "GPTBigCodeForCausalLM"
6
+ ],
7
+ "attention_softmax_in_fp32": true,
8
+ "attn_pdrop": 0.1,
9
+ "bos_token_id": 0,
10
+ "embd_pdrop": 0.1,
11
+ "eos_token_id": 0,
12
+ "inference_runner": 0,
13
+ "initializer_range": 0.02,
14
+ "layer_norm_epsilon": 1e-05,
15
+ "max_batch_size": null,
16
+ "max_sequence_length": null,
17
+ "model_type": "gpt_bigcode",
18
+ "multi_query": true,
19
+ "n_embd": 6144,
20
+ "n_head": 48,
21
+ "n_inner": 24576,
22
+ "n_layer": 40,
23
+ "n_positions": 8192,
24
+ "pad_key_length": true,
25
+ "pre_allocate_kv_cache": false,
26
+ "resid_pdrop": 0.1,
27
+ "scale_attention_softmax_in_fp32": true,
28
+ "scale_attn_weights": true,
29
+ "summary_activation": null,
30
+ "summary_first_dropout": 0.1,
31
+ "summary_proj_to_labels": true,
32
+ "summary_type": "cls_index",
33
+ "summary_use_proj": true,
34
+ "torch_dtype": "float16",
35
+ "transformers_version": "4.30.0.dev0",
36
+ "use_cache": true,
37
+ "validate_runner_input": true,
38
+ "vocab_size": 49153,
39
+ "quantization_config": {
40
+ "bits": 4,
41
+ "group_size": 128,
42
+ "damp_percent": 0.01,
43
+ "desc_act": false,
44
+ "sym": true,
45
+ "true_sequential": true,
46
+ "model_name_or_path": null,
47
+ "model_file_base_name": "model",
48
+ "quant_method": "gptq"
49
+ }
50
  }
gptq_model-4bit-128g.safetensors → model.safetensors RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2d71ef82879b90020e69895146d5762416e5116de7566aea31c0d3c78459826d
3
- size 9198428896
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6edc18f3e44d7f739a7d277db3c72e89f27290f781b9eefc9fada58d4a2c64a5
3
+ size 9198428952
quantize_config.json CHANGED
@@ -6,5 +6,5 @@
6
  "sym": true,
7
  "true_sequential": true,
8
  "model_name_or_path": null,
9
- "model_file_base_name": null
10
  }
 
6
  "sym": true,
7
  "true_sequential": true,
8
  "model_name_or_path": null,
9
+ "model_file_base_name": "model"
10
  }