TheYuriLover
commited on
Commit
•
83c9e95
1
Parent(s):
49457de
Update README.md
Browse files
README.md
CHANGED
@@ -2,7 +2,12 @@
|
|
2 |
license: other
|
3 |
---
|
4 |
|
5 |
-
|
6 |
|
7 |
-
|
|
|
|
|
|
|
|
|
|
|
8 |
|
|
|
2 |
license: other
|
3 |
---
|
4 |
|
5 |
+
This is the gptq 4bit quantization of this model: https://huggingface.co/kaiokendev/SuperCOT-LoRA
|
6 |
|
7 |
+
This quantization was made by using this repository: https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/triton
|
8 |
+
|
9 |
+
And I used the triton branch with all the gptq implementations available (true_sequential + act_order + groupsize 128)
|
10 |
+
CUDA_VISIBLE_DEVICES=0 python llama.py ./llama-13b-SuperCOT-4bit-TRITON c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors llama-13b-SuperCOT-4bit-TRITON.safetensors
|
11 |
+
|
12 |
+
To use the triton model on oobabooga's webui, you can refer to this post to get rid of all the errors you can encounter: https://github.com/oobabooga/text-generation-webui/issues/734
|
13 |
|