---
license: apache-2.0
datasets:
- LDJnr/Puffin
language:
- en
library_name: transformers
tags:
- NLP
- GPTQ
---

# Overview
This model is a quantized version of [Griffin-3B](https://huggingface.co/acrastt/Griffin-3B), using [GPTQ](https://arxiv.org/abs/2210.17323). The quantization has been done following the sample [notebook](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb) provided by Hugging Face.

# Usage
The model has been quantized as part of the project [GPTStonks](https://github.com/GPTStonks). It works with `transformers>=4.33.0` and it can run on a consumer GPU, with less than 3GB of GPU RAM. The libraries `optimum`, `auto-gptq`, `peft` and `accelerate` should also be installed.

Here is a sample code to load the model and run inference with it using sampling as decoding strategy:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "daedalus314/Griffin-3B-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')

text = """### HUMAN:
What is artifical intelligence?

### RESPONSE:
"""
inputs = tokenizer(text, return_tensors="pt").to(0)

out = quant_model.generate(
    **inputs,
    do_sample=True,
    top_p=0.9,
    temperature=0.9,
    max_length=1024,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
And a sample output:
```
### HUMAN:
What is artifical intelligence?

### RESPONSE:
Artificial intelligence, or AI, refers to the ability of computers to perform tasks that typically require human intelligence, such as decision making, problem solving, and language understanding. AI has been used in various fields, including healthcare, manufacturing, and finance, among others.
```

# Further details
Please refer to the original model [Griffin-3B](https://huggingface.co/acrastt/Griffin-3B).