Edit model card

Overview

This model is a quantized version of Griffin-3B, using GPTQ. The quantization has been done following the sample notebook provided by Hugging Face.

Usage

The model has been quantized as part of the project GPTStonks. It works with transformers>=4.33.0 and it can run on a consumer GPU, with less than 3GB of GPU RAM. The libraries optimum, auto-gptq, peft and accelerate should also be installed.

Here is a sample code to load the model and run inference with it using sampling as decoding strategy:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "daedalus314/Griffin-3B-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')

text = """### HUMAN:
What is artifical intelligence?

### RESPONSE:
"""
inputs = tokenizer(text, return_tensors="pt").to(0)

out = quant_model.generate(
    **inputs,
    do_sample=True,
    top_p=0.9,
    temperature=0.9,
    max_length=1024,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

And a sample output:

### HUMAN:
What is artifical intelligence?

### RESPONSE:
Artificial intelligence, or AI, refers to the ability of computers to perform tasks that typically require human intelligence, such as decision making, problem solving, and language understanding. AI has been used in various fields, including healthcare, manufacturing, and finance, among others.

Further details

Please refer to the original model Griffin-3B.

Downloads last month
20
Safetensors
Model size
723M params
Tensor type
I32
·
FP16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train daedalus314/Griffin-3B-GPTQ