core42

/

jais-13b-chat

like 134

It works with me but you need to use transformer loader. I do not know how to do this using command line

I'm getting the same error. How do I run it on a server for inference ? Using TGI or anthing else ? Do help us with the necessary parameters.

I dont know what you mean but if you wants to load it with 4bit to work with low VRAM here what I am using

my windows,
https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64
python 3.11

then python environment (i.e python -m vevn venv)

pip install transformers, accelerate
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl
pip install torch==2.0.1+cu117 --index-url https://download.pytorch.org/whl/cu117

a working python, I have two GPUs that is why I am specifying cuda:0 because it has 24G vram

# -*- coding: utf-8 -*-

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_path = "C:\AI\ML Models\inception-mbzuai_jais-13b"

device = "cuda:0" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map=device,load_in_4bit=True, trust_remote_code=True,
                                             bnb_4bit_compute_dtype=torch.float16)

def get_response(text,tokenizer=tokenizer,model=model):
    input_ids = tokenizer(text, return_tensors="pt").input_ids
    inputs = input_ids.to(device)
    input_len = inputs.shape[-1]
    generate_ids = model.generate(
        inputs,
        top_p=0.9,
        temperature=0.3,
        max_length=200-input_len,
        min_length=input_len + 4,
        repetition_penalty=1.2,
        do_sample=True,
    )
    response = tokenizer.batch_decode(
        generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
    )[0]
    return response


text= "عاصمة دولة الإمارات العربية المتحدة ه"
print(get_response(text))

text = "The capital of UAE is"
print(get_response(text))

Also you may need to use peft, I am not sure what it does but it solves some an error I was getting

pip install peft

# quantization_config
nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.float16
)

from peft import prepare_model_for_kbit_training
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda:0", quantization_config=nf4_config,trust_remote_code=True)
model = prepare_model_for_kbit_training(model)