Upload README.md
Browse files
README.md
CHANGED
@@ -98,7 +98,7 @@ Models are released as sharded safetensors files.
|
|
98 |
|
99 |
| Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
|
100 |
| ------ | ---- | -- | ----------- | ------- | ---- |
|
101 |
-
| main | 4 | 128 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 3.89 GB
|
102 |
|
103 |
<!-- README_AWQ.md-provided-files end -->
|
104 |
|
@@ -143,13 +143,13 @@ for output in outputs:
|
|
143 |
|
144 |
### Install the necessary packages
|
145 |
|
146 |
-
Requires: AutoAWQ 0.0.2 or later
|
147 |
|
148 |
```shell
|
149 |
pip3 install autoawq
|
150 |
```
|
151 |
|
152 |
-
If you have problems installing AutoAWQ using the pre-built wheels, install it from source instead:
|
153 |
|
154 |
```shell
|
155 |
pip3 uninstall -y autoawq
|
@@ -221,7 +221,7 @@ print(pipe(prompt_template)[0]['generated_text'])
|
|
221 |
<!-- README_AWQ.md-compatibility start -->
|
222 |
## Compatibility
|
223 |
|
224 |
-
The files provided are tested to work with AutoAWQ, and vLLM.
|
225 |
|
226 |
[Huggingface Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) is not yet compatible with AWQ, but a PR is open which should bring support soon: [TGI PR #781](https://github.com/huggingface/text-generation-inference/issues/781).
|
227 |
<!-- README_AWQ.md-compatibility end -->
|
|
|
98 |
|
99 |
| Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
|
100 |
| ------ | ---- | -- | ----------- | ------- | ---- |
|
101 |
+
| [main](https://huggingface.co/TheBloke/Llama-2-7b-Chat-AWQ/tree/main) | 4 | 128 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 3.89 GB
|
102 |
|
103 |
<!-- README_AWQ.md-provided-files end -->
|
104 |
|
|
|
143 |
|
144 |
### Install the necessary packages
|
145 |
|
146 |
+
Requires: [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) 0.0.2 or later
|
147 |
|
148 |
```shell
|
149 |
pip3 install autoawq
|
150 |
```
|
151 |
|
152 |
+
If you have problems installing [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) using the pre-built wheels, install it from source instead:
|
153 |
|
154 |
```shell
|
155 |
pip3 uninstall -y autoawq
|
|
|
221 |
<!-- README_AWQ.md-compatibility start -->
|
222 |
## Compatibility
|
223 |
|
224 |
+
The files provided are tested to work with [AutoAWQ](https://github.com/casper-hansen/AutoAWQ), and [vLLM](https://github.com/vllm-project/vllm).
|
225 |
|
226 |
[Huggingface Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) is not yet compatible with AWQ, but a PR is open which should bring support soon: [TGI PR #781](https://github.com/huggingface/text-generation-inference/issues/781).
|
227 |
<!-- README_AWQ.md-compatibility end -->
|