arcee-ai

/

Mistral-7B-Instruct-v0.2-sliced-24-layer

like 5

@iheardyoulooking it's because the model has been uploaded in 32 bit float format where the original Mistral is bfloat16. That makes each param in the sliced version twice as big on disk

You can still load the model in 16 bit by passing a torch_dtypeargument

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('arcee-ai/Mistral-7B-Instruct-v0.2-sliced-24-layer')
model = AutoModelForCausalLM.from_pretrained(
    'arcee-ai/Mistral-7B-Instruct-v0.2-sliced-24-layer',
    torch_dtype=torch.bfloat16
)

@Shamane should we re-upload in 16 bit? I can do it if you'd like

Yes exactly. I think I made a mistake, @thomasgauthier please go ahead and thanks a lot.