Overview
This model is a quantized version of Griffin-3B, using GPTQ. The quantization has been done following the sample notebook provided by Hugging Face.
Usage
The model has been quantized as part of the project GPTStonks. It works with transformers>=4.33.0
and it can run on a consumer GPU, with less than 3GB of GPU RAM. The libraries optimum
, auto-gptq
, peft
and accelerate
should also be installed.
Here is a sample code to load the model and run inference with it using sampling as decoding strategy:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "daedalus314/Griffin-3B-GPTQ"
tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')
text = """### HUMAN:
What is artifical intelligence?
### RESPONSE:
"""
inputs = tokenizer(text, return_tensors="pt").to(0)
out = quant_model.generate(
**inputs,
do_sample=True,
top_p=0.9,
temperature=0.9,
max_length=1024,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
And a sample output:
### HUMAN:
What is artifical intelligence?
### RESPONSE:
Artificial intelligence, or AI, refers to the ability of computers to perform tasks that typically require human intelligence, such as decision making, problem solving, and language understanding. AI has been used in various fields, including healthcare, manufacturing, and finance, among others.
Further details
Please refer to the original model Griffin-3B.
- Downloads last month
- 20
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.