|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# Mistral 7B Instruct |
|
|
|
AWQ quantized model. |
|
|
|
Dependencies: |
|
|
|
``` |
|
pip install git+https://github.com/huggingface/transformers.git |
|
pip install git+https://github.com/casper-hansen/AutoAWQ.git |
|
``` |
|
|
|
Example: |
|
|
|
```python |
|
from awq import AutoAWQForCausalLM |
|
from transformers import AutoTokenizer, TextStreamer |
|
|
|
quant_path = "mistral-7b-instruct-v0.1" |
|
|
|
# Load model |
|
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True) |
|
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True) |
|
streamer = TextStreamer(tokenizer, skip_special_tokens=True) |
|
|
|
# Convert prompt to tokens |
|
text = "<s>[INST] What is your favourite condiment? [/INST]" |
|
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> " |
|
"[INST] Do you have mayonnaise recipes? [/INST]" |
|
|
|
tokens = tokenizer( |
|
text, |
|
return_tensors='pt' |
|
).input_ids.cuda() |
|
|
|
# Generate output |
|
generation_output = model.generate( |
|
tokens, |
|
streamer=streamer, |
|
max_new_tokens=512 |
|
) |
|
``` |
