RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM
I tried to run the script llama_inference.py,but got the error below.
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
size mismatch for model.layers.0.self_attn.k_proj.qzeros: copying a param with shape torch.Size([32, 512]) from checkpoint, the shape in current
model is torch.Size([1, 512]).
size mismatch for model.layers.0.self_attn.k_proj.scales: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in
current model is torch.Size([1, 4096]).
size mismatch for model.layers.0.self_attn.o_proj.qzeros: copying a param with shape torch.Size([32, 512]) from checkpoint, the shape in current
model is torch.Size([1, 512]).
size mismatch for model.layers.0.self_attn.o_proj.scales: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in
current model is torch.Size([1, 4096]).
size mismatch for model.layers.0.self_attn.q_proj.qzeros: copying a param with shape torch.Size([32, 512]) from checkpoint, the shape in current
model is torch.Size([1, 512]).
size mismatch for model.layers.0.self_attn.q_proj.scales: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in
current model is torch.Size([1, 4096]).
size mismatch for model.layers.0.self_attn.v_proj.qzeros: copying a param with shape torch.Size([32, 512]) from checkpoint, the shape in current
model is torch.Size([1, 512]).
size mismatch for model.layers.0.self_attn.v_proj.scales: copying a param with shape torch.Size([32, 4096]) from checkpoint, the shape in
current model is torch.Size([1, 4096]).
size mismatch for model.layers.0.mlp.down_proj.qzeros: copying a param with shape torch.Size([86, 512]) from checkpoint, the shape in current
model is torch.Size([1, 512]).
size mismatch for model.layers.0.mlp.down_proj.scales: copying a param with shape torch.Size([86, 4096]) from checkpoint, the shape in current
model is torch.Size([1, 4096]).
size mismatch for model.layers.0.mlp.gate_proj.qzeros: copying a param with shape torch.Size([32, 1376]) from checkpoint, the shape in current
model is torch.Size([1, 1376]).
size mismatch for model.layers.0.mlp.gate_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current
model is torch.Size([1, 11008]).
size mismatch for model.layers.0.mlp.up_proj.qzeros: copying a param with shape torch.Size([32, 1376]) from checkpoint, the shape in current
model is torch.Size([1, 1376]).
size mismatch for model.layers.0.mlp.up_proj.scales: copying a param with shape torch.Size([32, 11008]) from checkpoint, the shape in current
model is torch.Size([1, 11008]).
The GPTQ-for-LLaMa repo has had a lot of updates recently, and some of them have broken compatibility. For the moment, try this older version of GPTQ-for-LLaMa:
git clone -n https://github.com/qwopqwop200/GPTQ-for-LLaMa gptq-working
cd gptq-working
git checkout 58c8ab4c7aaccc50f507fd08cce941976affe5e0