|
--- |
|
library_name: pruna-engine |
|
thumbnail: "https://assets-global.website-files.com/646b351987a8d8ce158d1940/64ec9e96b4334c0e1ac41504_Logo%20with%20white%20text.svg" |
|
metrics: |
|
- memory_disk |
|
- memory_inference |
|
- inference_latency |
|
- inference_throughput |
|
- inference_CO2_emissions |
|
- inference_energy_consumption |
|
--- |
|
<!-- header start --> |
|
<!-- 200823 --> |
|
<div style="width: auto; margin-left: auto; margin-right: auto"> |
|
<a href="https://www.pruna.ai/" target="_blank" rel="noopener noreferrer"> |
|
<img src="https://i.imgur.com/eDAlcgk.png" alt="PrunaAI" style="width: 100%; min-width: 400px; display: block; margin: auto;"> |
|
</a> |
|
</div> |
|
<!-- header end --> |
|
|
|
[](https://twitter.com/PrunaAI) |
|
[](https://github.com/PrunaAI) |
|
[](https://www.linkedin.com/company/93832878/admin/feed/posts/?feedType=following) |
|
[](https://discord.gg/CP4VSgck) |
|
|
|
# Simply make AI models cheaper, smaller, faster, and greener! |
|
|
|
- Give a thumbs up if you like this model! |
|
- Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact). |
|
- Request access to easily compress your *own* AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai). |
|
- Read the documentations to know more [here](https://pruna-ai-pruna.readthedocs-hosted.com/en/latest/) |
|
- Join Pruna AI community on Discord [here](https://discord.com/invite/vb6SmA3hxu) to share feedback/suggestions or get help. |
|
|
|
**Frequently Asked Questions** |
|
- ***How does the compression work?*** The model is compressed by using bitsandbytes. |
|
- ***How does the model quality change?*** The quality of the model output will slightly degrade. |
|
- ***What is the model format?*** We the standard safetensors format. |
|
- ***How to compress my own models?*** You can request premium access to more compression methods and tech support for your specific use-cases [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai). |
|
|
|
## Usage |
|
Here's how you can run the model use the model: |
|
|
|
```python |
|
|
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
tokenizer = AutoTokenizer.from_pretrained("PrunaAI/stable-code-instruct-3b-bnb-4bit", trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained("PrunaAI/stable-code-instruct-3b-bnb-4bit", torch_dtype=torch.bfloat16, trust_remote_code=True) |
|
model.eval() |
|
model = model.cuda() |
|
|
|
messages = [ |
|
{ |
|
"role": "system", |
|
"content": "You are a helpful and polite assistant", |
|
}, |
|
{ |
|
"role": "user", |
|
"content": "Write a simple website in HTML. When a user clicks the button, it shows a random joke from a list of 4 jokes." |
|
}, |
|
] |
|
|
|
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
|
|
|
inputs = tokenizer([prompt], return_tensors="pt").to(model.device) |
|
|
|
tokens = model.generate( |
|
**inputs, |
|
max_new_tokens=1024, |
|
temperature=0.5, |
|
top_p=0.95, |
|
top_k=100, |
|
do_sample=True, |
|
use_cache=True |
|
) |
|
|
|
output = tokenizer.batch_decode(tokens[:, inputs.input_ids.shape[-1]:], skip_special_tokens=False)[0] |
|
``` |
|
|
|
## Credits & License |
|
|
|
The license of the smashed model follows the license of the original model. Please check the license of the original model stabilityai/stable-code-instruct-3b before using this model which provided the base model. The license of the `pruna-engine` is [here](https://pypi.org/project/pruna-engine/) on Pypi. |
|
|
|
## Want to compress other models? |
|
|
|
- Contact us and tell us which model to compress next [here](https://www.pruna.ai/contact). |
|
- Request access to easily compress your own AI models [here](https://z0halsaff74.typeform.com/pruna-access?typeform-source=www.pruna.ai). |
