bofenghuang

/

vigogne-7b-instruct

like 22

Hello,

When trying to use this model, on top of decapoda-research/llama-7b-hf, I get a cryptic

'NoneType' object has no attribute 'device'`

issue.

I found this issue to be potentially linked to :

• an interplay between bitsandbyte and accelerate's device_map="auto", supposedly fixed by a recent PR , but updating accelerate did not help (hm, it looks like it has actually not been released yet)
• an issue wit peft itself, and forcing device_map={"":0} presented as a work-around ; did not work either

I tested this on both a single-GPU computer and a multi-GPU cluster, as I suspected an issue with limited RAM or GPU memory, but this does not seem to be the case (I get a more explicit error in that case).

Do you have ideas of where the issue could come from?

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import LlamaForCausalLM, LlamaTokenizer
import torch

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
llama = LlamaForCausalLM.from_pretrained(
    "decapoda-research/llama-7b-hf",
    load_in_8bit=True,
    torch_dtype=torch.float16, # error message from bitsandbytes asks for this
    device_map="auto", # this is superseded by the second device_map
)
vigogne = PeftModel.from_pretrained(
    llama,
    "bofenghuang/vigogne-lora-7b",
    device_map={"":0} # suggested work-around
)