Update README.md
Browse files
README.md
CHANGED
@@ -40,8 +40,8 @@ None
|
|
40 |
|
41 |
## Repositories available
|
42 |
|
43 |
-
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/StableBeluga2-GPTQ)
|
44 |
-
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/StableBeluga2-GGML)
|
45 |
* [Stability AI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/stabilityai/StableBeluga2)
|
46 |
|
47 |
## Prompt template: Orca-Hashes
|
@@ -79,7 +79,7 @@ Each separate quant is in a different branch. See below for instructions on fet
|
|
79 |
- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/StableBeluga2-GPTQ:gptq-4bit-32g-actorder_True`
|
80 |
- With Git, you can clone a branch with:
|
81 |
```
|
82 |
-
git clone --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/StableBeluga2-GPTQ
|
83 |
```
|
84 |
- In Python Transformers code, the branch is the `revision` parameter; see below.
|
85 |
|
@@ -90,13 +90,13 @@ Please make sure you're using the latest version of [text-generation-webui](http
|
|
90 |
It is strongly recommended to use the text-generation-webui one-click-installers unless you know how to make a manual install.
|
91 |
|
92 |
1. Click the **Model tab**.
|
93 |
-
2. Under **Download custom model or LoRA**, enter `TheBloke/StableBeluga2-GPTQ`.
|
94 |
-
- To download from a specific branch, enter for example `TheBloke/StableBeluga2-GPTQ:gptq-4bit-32g-actorder_True`
|
95 |
- see Provided Files above for the list of branches for each option.
|
96 |
3. Click **Download**.
|
97 |
4. The model will start downloading. Once it's finished it will say "Done"
|
98 |
5. In the top left, click the refresh icon next to **Model**.
|
99 |
-
6. In the **Model** dropdown, choose the model you just downloaded: `StableBeluga2-GPTQ`
|
100 |
7. The model will automatically load, and is now ready for use!
|
101 |
8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
|
102 |
* Note that you do not need to set GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
|
@@ -114,7 +114,7 @@ Then try the following example code:
|
|
114 |
from transformers import AutoTokenizer, pipeline, logging
|
115 |
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
|
116 |
|
117 |
-
model_name_or_path = "TheBloke/StableBeluga2-GPTQ"
|
118 |
model_basename = "gptq_model-4bit--1g"
|
119 |
|
120 |
use_triton = False
|
|
|
40 |
|
41 |
## Repositories available
|
42 |
|
43 |
+
* [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/StableBeluga2-70B-GPTQ)
|
44 |
+
* [2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/StableBeluga2-70B-GGML)
|
45 |
* [Stability AI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/stabilityai/StableBeluga2)
|
46 |
|
47 |
## Prompt template: Orca-Hashes
|
|
|
79 |
- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/StableBeluga2-GPTQ:gptq-4bit-32g-actorder_True`
|
80 |
- With Git, you can clone a branch with:
|
81 |
```
|
82 |
+
git clone --branch gptq-4bit-32g-actorder_True --single-branch https://huggingface.co/TheBloke/StableBeluga2-GPTQ
|
83 |
```
|
84 |
- In Python Transformers code, the branch is the `revision` parameter; see below.
|
85 |
|
|
|
90 |
It is strongly recommended to use the text-generation-webui one-click-installers unless you know how to make a manual install.
|
91 |
|
92 |
1. Click the **Model tab**.
|
93 |
+
2. Under **Download custom model or LoRA**, enter `TheBloke/StableBeluga2-70B-GPTQ`.
|
94 |
+
- To download from a specific branch, enter for example `TheBloke/StableBeluga2-70B-GPTQ:gptq-4bit-32g-actorder_True`
|
95 |
- see Provided Files above for the list of branches for each option.
|
96 |
3. Click **Download**.
|
97 |
4. The model will start downloading. Once it's finished it will say "Done"
|
98 |
5. In the top left, click the refresh icon next to **Model**.
|
99 |
+
6. In the **Model** dropdown, choose the model you just downloaded: `StableBeluga2-70B-GPTQ`
|
100 |
7. The model will automatically load, and is now ready for use!
|
101 |
8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
|
102 |
* Note that you do not need to set GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
|
|
|
114 |
from transformers import AutoTokenizer, pipeline, logging
|
115 |
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
|
116 |
|
117 |
+
model_name_or_path = "TheBloke/StableBeluga2-70B-GPTQ"
|
118 |
model_basename = "gptq_model-4bit--1g"
|
119 |
|
120 |
use_triton = False
|