Text Generation
Transformers
Safetensors
English
llama
text-generation-inference
4-bit precision
gptq
TheBloke commited on
Commit
04f12cf
1 Parent(s): c5e2095

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +122 -65
README.md CHANGED
@@ -7,9 +7,13 @@ datasets:
7
  inference: false
8
  language:
9
  - en
10
- license: other
 
 
 
11
  model_type: llama
12
  pipeline_tag: text-generation
 
13
  ---
14
 
15
  <!-- header start -->
@@ -29,130 +33,163 @@ pipeline_tag: text-generation
29
  <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
30
  <!-- header end -->
31
 
32
- # StableBeluga 2 - GGML
33
  - Model creator: [Stability AI](https://huggingface.co/stabilityai)
34
- - Original model: [StableBeluga 2](https://huggingface.co/stabilityai/StableBeluga2)
35
 
 
36
  ## Description
37
 
38
- This repo contains GPTQ model files for [Stability AI's StableBeluga 2](https://huggingface.co/stabilityai/StableBeluga2).
39
 
40
  Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.
41
 
42
- None
43
-
44
  ## Repositories available
45
 
46
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ)
47
- * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/StableBeluga2-70B-GGML)
 
48
  * [Stability AI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/stabilityai/StableBeluga2)
 
49
 
 
50
  ## Prompt template: Orca-Hashes
51
 
52
  ```
53
  ### System:
54
- This is a system prompt, please behave and help the user.
55
 
56
  ### User:
57
  {prompt}
58
 
59
  ### Assistant:
 
60
  ```
61
 
62
- ## Provided files
 
 
 
63
 
64
  Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.
65
 
66
  Each separate quant is in a different branch. See below for instructions on fetching from different branches.
67
 
68
- | Branch | Bits | Group Size | Act Order (desc_act) | File Size | ExLlama Compatible? | Made With | Description |
69
- | ------ | ---- | ---------- | -------------------- | --------- | ------------------- | --------- | ----------- |
70
- | main | 4 | None | True | 35.33 GB | True | AutoGPTQ | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. |
71
- | gptq-4bit-32g-actorder_True | 4 | 32 | True | 40.66 GB | True | AutoGPTQ | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. |
72
- | gptq-4bit-128g-actorder_True | 4 | 128 | True | 36.65 GB | True | AutoGPTQ | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
73
- | gptq-4bit-64g-actorder_True | 4 | 64 | True | 37.99 GB | True | AutoGPTQ | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
74
- | gptq-3bit--1g-actorder_True | 3 | None | True | 26.78 GB | False | AutoGPTQ | 3-bit, with Act Order and no group size. Lowest possible VRAM requirements. May be lower quality than 3-bit 128g. |
75
- | gptq-3bit-128g-actorder_False | 3 | 128 | False | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g but no act-order. Slightly higher VRAM requirements than 3-bit None. |
76
- | gptq-3bit-128g-actorder_True | 3 | 128 | True | 28.03 GB | False | AutoGPTQ | 3-bit, with group size 128g and act-order. Higher quality than 128g-False but poor AutoGPTQ CUDA speed. |
77
- | gptq-3bit-64g-actorder_True | 3 | 64 | True | 29.30 GB | False | AutoGPTQ | 3-bit, with group size 64g and act-order. Highest quality 3-bit option. Poor AutoGPTQ CUDA speed. |
78
- | gptq-4bit-128g-actorder_False | 4 | 128 | False | 36.65 GB | True | AutoGPTQ | 4-bit, without Act Order and group size 128g. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
79
 
 
 
 
80
  ## How to download from branches
81
 
82
- - In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/StableBeluga2-GPTQ:gptq-4bit-32g-actorder_True`
83
  - With Git, you can clone a branch with:
84
  ```
85
- git clone --branch gptq-4bit-32g-actorder_True --single-branch https://huggingface.co/TheBloke/StableBeluga2-GPTQ
86
  ```
87
  - In Python Transformers code, the branch is the `revision` parameter; see below.
88
-
 
89
  ## How to easily download and use this model in [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
90
 
91
  Please make sure you're using the latest version of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
92
 
93
- It is strongly recommended to use the text-generation-webui one-click-installers unless you know how to make a manual install.
94
 
95
  1. Click the **Model tab**.
96
  2. Under **Download custom model or LoRA**, enter `TheBloke/StableBeluga2-70B-GPTQ`.
97
  - To download from a specific branch, enter for example `TheBloke/StableBeluga2-70B-GPTQ:gptq-4bit-32g-actorder_True`
98
  - see Provided Files above for the list of branches for each option.
99
  3. Click **Download**.
100
- 4. The model will start downloading. Once it's finished it will say "Done"
101
  5. In the top left, click the refresh icon next to **Model**.
102
  6. In the **Model** dropdown, choose the model you just downloaded: `StableBeluga2-70B-GPTQ`
103
  7. The model will automatically load, and is now ready for use!
104
  8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
105
- * Note that you do not need to set GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
106
  9. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
 
107
 
 
108
  ## How to use this GPTQ model from Python code
109
 
110
- First make sure you have [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) 0.3.2 or later installed:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
 
112
- `GITHUB_ACTIONS=true pip install auto-gptq`
 
 
 
 
113
 
114
- Then try the following example code:
115
 
116
  ```python
117
- from transformers import AutoTokenizer, pipeline, logging
118
- from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
119
 
120
  model_name_or_path = "TheBloke/StableBeluga2-70B-GPTQ"
121
- model_basename = "model"
122
-
123
- use_triton = False
 
 
 
124
 
125
  tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
126
 
127
- model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
128
- model_basename=model_basename,
129
- inject_fused_attention=False, # Required for Llama 2 70B models at this time.
130
- use_safetensors=True,
131
- trust_remote_code=False,
132
- device="cuda:0",
133
- use_triton=use_triton,
134
- quantize_config=None)
135
-
136
- """
137
- To download from a specific branch, use the revision parameter, as in this example:
138
-
139
- model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
140
- revision="gptq-4bit-32g-actorder_True",
141
- model_basename=model_basename,
142
- use_safetensors=True,
143
- trust_remote_code=False,
144
- device="cuda:0",
145
- quantize_config=None)
146
- """
147
-
148
  prompt = "Tell me about AI"
149
  prompt_template=f'''### System:
150
- This is a system prompt, please behave and help the user.
151
 
152
  ### User:
153
  {prompt}
154
 
155
  ### Assistant:
 
156
  '''
157
 
158
  print("\n\n*** Generate:")
@@ -163,9 +200,6 @@ print(tokenizer.decode(output[0]))
163
 
164
  # Inference can also be done using transformers' pipeline
165
 
166
- # Prevent printing spurious transformers error when using pipeline with AutoGPTQ
167
- logging.set_verbosity(logging.CRITICAL)
168
-
169
  print("*** Pipeline:")
170
  pipe = pipeline(
171
  "text-generation",
@@ -179,12 +213,17 @@ pipe = pipeline(
179
 
180
  print(pipe(prompt_template)[0]['generated_text'])
181
  ```
 
182
 
 
183
  ## Compatibility
184
 
185
- The files provided will work with AutoGPTQ (CUDA and Triton modes), GPTQ-for-LLaMa (only CUDA has been tested), and Occ4m's GPTQ-for-LLaMa fork.
186
 
187
- ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
 
 
 
188
 
189
  <!-- footer start -->
190
  <!-- 200823 -->
@@ -209,7 +248,7 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
209
 
210
  **Special thanks to**: Aemon Algiz.
211
 
212
- **Patreon special mentions**: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter
213
 
214
 
215
  Thank you to all my generous patrons and donaters!
@@ -218,10 +257,12 @@ And thank you again to a16z for their generous grant.
218
 
219
  <!-- footer end -->
220
 
221
- # Original model card: Stability AI's StableBeluga 2
222
 
223
  # Stable Beluga 2
224
 
 
 
225
  ## Model Description
226
 
227
  `Stable Beluga 2` is a Llama2 70B model finetuned on an Orca style Dataset
@@ -258,6 +299,12 @@ Your prompt here
258
  The output of Stable Beluga 2
259
  ```
260
 
 
 
 
 
 
 
261
  ## Model Details
262
 
263
  * **Developed by**: [Stability AI](https://stability.ai/)
@@ -284,11 +331,21 @@ Models are learned via supervised fine-tuning on the aforementioned datasets, tr
284
 
285
  Beluga is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Beluga's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Beluga, developers should perform safety testing and tuning tailored to their specific applications of the model.
286
 
 
 
 
 
 
 
 
 
 
 
287
  ## Citations
288
 
289
  ```bibtext
290
  @misc{touvron2023llama,
291
- title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
292
  author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton Ferrer and Moya Chen and Guillem Cucurull and David Esiobu and Jude Fernandes and Jeremy Fu and Wenyin Fu and Brian Fuller and Cynthia Gao and Vedanuj Goswami and Naman Goyal and Anthony Hartshorn and Saghar Hosseini and Rui Hou and Hakan Inan and Marcin Kardas and Viktor Kerkez and Madian Khabsa and Isabel Kloumann and Artem Korenev and Punit Singh Koura and Marie-Anne Lachaux and Thibaut Lavril and Jenya Lee and Diana Liskovich and Yinghai Lu and Yuning Mao and Xavier Martinet and Todor Mihaylov and Pushkar Mishra and Igor Molybog and Yixin Nie and Andrew Poulton and Jeremy Reizenstein and Rashi Rungta and Kalyan Saladi and Alan Schelten and Ruan Silva and Eric Michael Smith and Ranjan Subramanian and Xiaoqing Ellen Tan and Binh Tang and Ross Taylor and Adina Williams and Jian Xiang Kuan and Puxin Xu and Zheng Yan and Iliyan Zarov and Yuchen Zhang and Angela Fan and Melanie Kambadur and Sharan Narang and Aurelien Rodriguez and Robert Stojnic and Sergey Edunov and Thomas Scialom},
293
  year={2023},
294
  eprint={2307.09288},
@@ -299,7 +356,7 @@ Beluga is a new technology that carries risks with use. Testing conducted to dat
299
 
300
  ```bibtext
301
  @misc{mukherjee2023orca,
302
- title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4},
303
  author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
304
  year={2023},
305
  eprint={2306.02707},
 
7
  inference: false
8
  language:
9
  - en
10
+ license: llama2
11
+ model_creator: Stability AI
12
+ model_link: https://huggingface.co/stabilityai/StableBeluga2
13
+ model_name: StableBeluga2
14
  model_type: llama
15
  pipeline_tag: text-generation
16
+ quantized_by: TheBloke
17
  ---
18
 
19
  <!-- header start -->
 
33
  <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
34
  <!-- header end -->
35
 
36
+ # StableBeluga2 - GPTQ
37
  - Model creator: [Stability AI](https://huggingface.co/stabilityai)
38
+ - Original model: [StableBeluga2](https://huggingface.co/stabilityai/StableBeluga2)
39
 
40
+ <!-- description start -->
41
  ## Description
42
 
43
+ This repo contains GPTQ model files for [Stability AI's StableBeluga2](https://huggingface.co/stabilityai/StableBeluga2).
44
 
45
  Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.
46
 
47
+ <!-- description end -->
48
+ <!-- repositories-available start -->
49
  ## Repositories available
50
 
51
  * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ)
52
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/StableBeluga2-70B-GGUF)
53
+ * [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference (deprecated)](https://huggingface.co/TheBloke/StableBeluga2-70B-GGML)
54
  * [Stability AI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/stabilityai/StableBeluga2)
55
+ <!-- repositories-available end -->
56
 
57
+ <!-- prompt-template start -->
58
  ## Prompt template: Orca-Hashes
59
 
60
  ```
61
  ### System:
62
+ {system_message}
63
 
64
  ### User:
65
  {prompt}
66
 
67
  ### Assistant:
68
+
69
  ```
70
 
71
+ <!-- prompt-template end -->
72
+
73
+ <!-- README_GPTQ.md-provided-files start -->
74
+ ## Provided files and GPTQ parameters
75
 
76
  Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.
77
 
78
  Each separate quant is in a different branch. See below for instructions on fetching from different branches.
79
 
80
+ All recent GPTQ files are made with AutoGPTQ, and all files in non-main branches are made with AutoGPTQ. Files in the `main` branch which were uploaded before August 2023 were made with GPTQ-for-LLaMa.
81
+
82
+ <details>
83
+ <summary>Explanation of GPTQ parameters</summary>
84
+
85
+ - Bits: The bit size of the quantised model.
86
+ - GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
87
+ - Act Order: True or False. Also known as `desc_act`. True results in better quantisation accuracy. Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now.
88
+ - Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.
89
+ - GPTQ dataset: The dataset used for quantisation. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Note that the GPTQ dataset is not the same as the dataset used to train the model - please refer to the original model repo for details of the training dataset(s).
90
+ - Sequence Length: The length of the dataset sequences used for quantisation. Ideally this is the same as the model sequence length. For some very long sequence models (16+K), a lower sequence length may have to be used. Note that a lower sequence length does not limit the sequence length of the quantised model. It only impacts the quantisation accuracy on longer inference sequences.
91
+ - ExLlama Compatibility: Whether this file can be loaded with ExLlama, which currently only supports Llama models in 4-bit.
92
+
93
+ </details>
94
+
95
+ | Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
96
+ | ------ | ---- | -- | --------- | ------ | ------------ | ------- | ---- | ------- | ---- |
97
+ | [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 40.66 GB | Yes | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. |
98
+ | [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 37.99 GB | Yes | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
99
+ | [gptq-4bit-128g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-4bit-128g-actorder_True) | 4 | 128 | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 36.65 GB | Yes | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. |
100
+ | [gptq-3bit-128g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-3bit-128g-actorder_True) | 3 | 128 | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 28.03 GB | No | 3-bit, with group size 128g and act-order. Higher quality than 128g-False but poor AutoGPTQ CUDA speed. |
101
+ | [gptq-3bit-128g-actorder_False](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-3bit-128g-actorder_False) | 3 | 128 | No | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 28.03 GB | No | 3-bit, with group size 128g but no act-order. Slightly higher VRAM requirements than 3-bit None. |
102
+ | [gptq-3bit--1g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-3bit--1g-actorder_True) | 3 | None | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 26.78 GB | No | 3-bit, with Act Order and no group size. Lowest possible VRAM requirements. May be lower quality than 3-bit 128g. |
103
+ | [gptq-4bit-128g-actorder_False](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-4bit-128g-actorder_False) | 4 | 128 | No | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 36.65 GB | Yes | 4-bit, without Act Order and group size 128g. |
104
+ | [gptq-3bit-64g-actorder_True](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/gptq-3bit-64g-actorder_True) | 3 | 64 | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 29.30 GB | No | 3-bit, with group size 64g and act-order. Poor AutoGPTQ CUDA speed. |
105
+ | [main](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ/tree/main) | 4 | None | Yes | 0.01 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 35.33 GB | Yes | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. |
106
 
107
+ <!-- README_GPTQ.md-provided-files end -->
108
+
109
+ <!-- README_GPTQ.md-download-from-branches start -->
110
  ## How to download from branches
111
 
112
+ - In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/StableBeluga2-70B-GPTQ:gptq-4bit-32g-actorder_True`
113
  - With Git, you can clone a branch with:
114
  ```
115
+ git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ
116
  ```
117
  - In Python Transformers code, the branch is the `revision` parameter; see below.
118
+ <!-- README_GPTQ.md-download-from-branches end -->
119
+ <!-- README_GPTQ.md-text-generation-webui start -->
120
  ## How to easily download and use this model in [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
121
 
122
  Please make sure you're using the latest version of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
123
 
124
+ It is strongly recommended to use the text-generation-webui one-click-installers unless you're sure you know how to make a manual install.
125
 
126
  1. Click the **Model tab**.
127
  2. Under **Download custom model or LoRA**, enter `TheBloke/StableBeluga2-70B-GPTQ`.
128
  - To download from a specific branch, enter for example `TheBloke/StableBeluga2-70B-GPTQ:gptq-4bit-32g-actorder_True`
129
  - see Provided Files above for the list of branches for each option.
130
  3. Click **Download**.
131
+ 4. The model will start downloading. Once it's finished it will say "Done".
132
  5. In the top left, click the refresh icon next to **Model**.
133
  6. In the **Model** dropdown, choose the model you just downloaded: `StableBeluga2-70B-GPTQ`
134
  7. The model will automatically load, and is now ready for use!
135
  8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
136
+ * Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
137
  9. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
138
+ <!-- README_GPTQ.md-text-generation-webui end -->
139
 
140
+ <!-- README_GPTQ.md-use-from-python start -->
141
  ## How to use this GPTQ model from Python code
142
 
143
+ ### Install the necessary packages
144
+
145
+ Requires: Transformers 4.32.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.
146
+
147
+ ```shell
148
+ pip3 install transformers>=4.32.0 optimum>=1.12.0
149
+ pip3 install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/ # Use cu117 if on CUDA 11.7
150
+ ```
151
+
152
+ If you have problems installing AutoGPTQ using the pre-built wheels, install it from source instead:
153
+
154
+ ```shell
155
+ pip3 uninstall -y auto-gptq
156
+ git clone https://github.com/PanQiWei/AutoGPTQ
157
+ cd AutoGPTQ
158
+ pip3 install .
159
+ ```
160
+
161
+ ### For CodeLlama models only: you must use Transformers 4.33.0 or later.
162
 
163
+ If 4.33.0 is not yet released when you read this, you will need to install Transformers from source:
164
+ ```shell
165
+ pip3 uninstall -y transformers
166
+ pip3 install git+https://github.com/huggingface/transformers.git
167
+ ```
168
 
169
+ ### You can then use the following code
170
 
171
  ```python
172
+ from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
173
 
174
  model_name_or_path = "TheBloke/StableBeluga2-70B-GPTQ"
175
+ # To use a different branch, change revision
176
+ # For example: revision="gptq-4bit-32g-actorder_True"
177
+ model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
178
+ torch_dtype=torch.float16,
179
+ device_map="auto",
180
+ revision="main")
181
 
182
  tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
183
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
184
  prompt = "Tell me about AI"
185
  prompt_template=f'''### System:
186
+ {system_message}
187
 
188
  ### User:
189
  {prompt}
190
 
191
  ### Assistant:
192
+
193
  '''
194
 
195
  print("\n\n*** Generate:")
 
200
 
201
  # Inference can also be done using transformers' pipeline
202
 
 
 
 
203
  print("*** Pipeline:")
204
  pipe = pipeline(
205
  "text-generation",
 
213
 
214
  print(pipe(prompt_template)[0]['generated_text'])
215
  ```
216
+ <!-- README_GPTQ.md-use-from-python end -->
217
 
218
+ <!-- README_GPTQ.md-compatibility start -->
219
  ## Compatibility
220
 
221
+ The files provided are tested to work with AutoGPTQ, both via Transformers and using AutoGPTQ directly. They should also work with [Occ4m's GPTQ-for-LLaMa fork](https://github.com/0cc4m/KoboldAI).
222
 
223
+ [ExLlama](https://github.com/turboderp/exllama) is compatible with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.
224
+
225
+ [Huggingface Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) is compatible with all GPTQ models.
226
+ <!-- README_GPTQ.md-compatibility end -->
227
 
228
  <!-- footer start -->
229
  <!-- 200823 -->
 
248
 
249
  **Special thanks to**: Aemon Algiz.
250
 
251
+ **Patreon special mentions**: Russ Johnson, J, alfie_i, Alex, NimbleBox.ai, Chadd, Mandus, Nikolai Manek, Ken Nordquist, ya boyyy, Illia Dulskyi, Viktor Bowallius, vamX, Iucharbius, zynix, Magnesian, Clay Pascal, Pierre Kircher, Enrico Ros, Tony Hughes, Elle, Andrey, knownsqashed, Deep Realms, Jerry Meng, Lone Striker, Derek Yates, Pyrater, Mesiah Bishop, James Bentley, Femi Adebogun, Brandon Frisco, SuperWojo, Alps Aficionado, Michael Dempsey, Vitor Caleffi, Will Dee, Edmond Seymore, usrbinkat, LangChain4j, Kacper Wikieł, Luke Pendergrass, John Detwiler, theTransient, Nathan LeClaire, Tiffany J. Kim, biorpg, Eugene Pentland, Stanislav Ovsiannikov, Fred von Graf, terasurfer, Kalila, Dan Guido, Nitin Borwankar, 阿明, Ai Maven, John Villwock, Gabriel Puliatti, Stephen Murray, Asp the Wyvern, danny, Chris Smitley, ReadyPlayerEmma, S_X, Daniel P. Andersen, Olakabola, Jeffrey Morgan, Imad Khwaja, Caitlyn Gatomon, webtim, Alicia Loh, Trenton Dambrowitz, Swaroop Kallakuri, Erik Bjäreholt, Leonard Tan, Spiking Neurons AB, Luke @flexchar, Ajan Kanaga, Thomas Belote, Deo Leter, RoA, Willem Michiel, transmissions 11, subjectnull, Matthew Berman, Joseph William Delisle, David Ziegler, Michael Davis, Johann-Peter Hartmann, Talal Aujan, senxiiz, Artur Olbinski, Rainer Wilmers, Spencer Kim, Fen Risland, Cap'n Zoog, Rishabh Srivastava, Michael Levine, Geoffrey Montalvo, Sean Connelly, Alexandros Triantafyllidis, Pieter, Gabriel Tamborski, Sam, Subspace Studios, Junyu Yang, Pedro Madruga, Vadim, Cory Kujawski, K, Raven Klaugh, Randy H, Mano Prime, Sebastain Graf, Space Cruiser
252
 
253
 
254
  Thank you to all my generous patrons and donaters!
 
257
 
258
  <!-- footer end -->
259
 
260
+ # Original model card: Stability AI's StableBeluga2
261
 
262
  # Stable Beluga 2
263
 
264
+ Use [Stable Chat (Research Preview)](https://chat.stability.ai/chat) to test Stability AI's best language models for free
265
+
266
  ## Model Description
267
 
268
  `Stable Beluga 2` is a Llama2 70B model finetuned on an Orca style Dataset
 
299
  The output of Stable Beluga 2
300
  ```
301
 
302
+ ## Other Beluga Models
303
+
304
+ [StableBeluga 1 - Delta](https://huggingface.co/stabilityai/StableBeluga1-Delta)
305
+ [StableBeluga 13B](https://huggingface.co/stabilityai/StableBeluga-13B)
306
+ [StableBeluga 7B](https://huggingface.co/stabilityai/StableBeluga-7B)
307
+
308
  ## Model Details
309
 
310
  * **Developed by**: [Stability AI](https://stability.ai/)
 
331
 
332
  Beluga is a new technology that carries risks with use. Testing conducted to date has been in English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Beluga's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Beluga, developers should perform safety testing and tuning tailored to their specific applications of the model.
333
 
334
+ ## How to cite
335
+
336
+ ```bibtex
337
+ @misc{StableBelugaModels,
338
+ url={[https://huggingface.co/stabilityai/StableBeluga2](https://huggingface.co/stabilityai/StableBeluga2)},
339
+ title={Stable Beluga models},
340
+ author={Mahan, Dakota and Carlow, Ryan and Castricato, Louis and Cooper, Nathan and Laforte, Christian}
341
+ }
342
+ ```
343
+
344
  ## Citations
345
 
346
  ```bibtext
347
  @misc{touvron2023llama,
348
+ title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
349
  author={Hugo Touvron and Louis Martin and Kevin Stone and Peter Albert and Amjad Almahairi and Yasmine Babaei and Nikolay Bashlykov and Soumya Batra and Prajjwal Bhargava and Shruti Bhosale and Dan Bikel and Lukas Blecher and Cristian Canton Ferrer and Moya Chen and Guillem Cucurull and David Esiobu and Jude Fernandes and Jeremy Fu and Wenyin Fu and Brian Fuller and Cynthia Gao and Vedanuj Goswami and Naman Goyal and Anthony Hartshorn and Saghar Hosseini and Rui Hou and Hakan Inan and Marcin Kardas and Viktor Kerkez and Madian Khabsa and Isabel Kloumann and Artem Korenev and Punit Singh Koura and Marie-Anne Lachaux and Thibaut Lavril and Jenya Lee and Diana Liskovich and Yinghai Lu and Yuning Mao and Xavier Martinet and Todor Mihaylov and Pushkar Mishra and Igor Molybog and Yixin Nie and Andrew Poulton and Jeremy Reizenstein and Rashi Rungta and Kalyan Saladi and Alan Schelten and Ruan Silva and Eric Michael Smith and Ranjan Subramanian and Xiaoqing Ellen Tan and Binh Tang and Ross Taylor and Adina Williams and Jian Xiang Kuan and Puxin Xu and Zheng Yan and Iliyan Zarov and Yuchen Zhang and Angela Fan and Melanie Kambadur and Sharan Narang and Aurelien Rodriguez and Robert Stojnic and Sergey Edunov and Thomas Scialom},
350
  year={2023},
351
  eprint={2307.09288},
 
356
 
357
  ```bibtext
358
  @misc{mukherjee2023orca,
359
+ title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4},
360
  author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
361
  year={2023},
362
  eprint={2306.02707},