update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: mit
|
3 |
---
|
|
|
1 |
+
Generated using autoawq: `pip install git+https://github.com/casper-hansen/AutoAWQ.git@f0321eedca887c12680553fc561d176b03b1b9a5 flash_attn'
|
2 |
+
|
3 |
+
Following code used for generation:
|
4 |
+
|
5 |
+
```python
|
6 |
+
from awq import AutoAWQForCausalLM
|
7 |
+
from transformers import AutoTokenizer
|
8 |
+
|
9 |
+
model_path = 'models/Phi-3-medium-128k-instruct'
|
10 |
+
quant_path = 'models/Phi-3-medium-128k-instruct-awq'
|
11 |
+
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
|
12 |
+
|
13 |
+
# Load model
|
14 |
+
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"device_map": "auto"})
|
15 |
+
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
16 |
+
|
17 |
+
# Quantize
|
18 |
+
model.quantize(tokenizer, quant_config=quant_config)
|
19 |
+
|
20 |
+
# Save quantized model
|
21 |
+
model.save_quantized(quant_path)
|
22 |
+
tokenizer.save_pretrained(quant_path)
|
23 |
+
```
|
24 |
+
|
25 |
+
Original model here: https://huggingface.co/microsoft/Phi-3-medium-128k-instruct
|
26 |
+
|
27 |
---
|
28 |
license: mit
|
29 |
---
|