Size mismatch for model.decoder.layers
#4
by
IngvarJackal
- opened
Tried to run https://github.com/0cc4m/KoboldAI with this model,
followed the steps from the README.md:
- checked out this repo into
./models/OPT-30B-Erebus-4bit-128g - renamed
.ptand.safetensorsinto4bit.ones - enabled experimental UI
- selected
Truein 4-bit on load
On the loading the model, got the error:
RuntimeError: Error(s) in loading state_dict for OPTForCausalLM:
size mismatch for model.decoder.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([56, 896]) from checkpoint, the shape in current model is torch.Size([1, 896]).
size mismatch for model.decoder.layers.0.self_attn.k_proj.scales: copying a param with shape torch.Size([56, 7168]) from checkpoint, the shape in current model is torch.Size([1, 7168]).
...
Question: what am I doing wrong?
cc: @SquidHominid maybe you know, since your post indicates you managed to run the model successfully?
This isn't the error I got, but if it helps you, I tried running the model in KoboldAI, and the problem I had is that mainline KoboldAI doesn't support 4-bit quantized models. For that, you need Oobabooga or Occam's KoboldAI fork.
I found the error which I made -- the model should be called 4bit-128g.pt, not 4bit.pt
IngvarJackal
changed discussion status to
closed