safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
#2
by
zuzuou
- opened
I encounter this error when loading Yi-34B-200K-GPTQ model. How can I deal with it ?
(yi) [root@node4 yi]# python demo/text_generation.py --model /data/models/Yi-34B-200K-GPTQ
Namespace(model='/data/models/Yi-34B-200K-GPTQ', tokenizer='', max_tokens=512, streaming=False, prompt='Let me tell you an interesting story about cat Tom and mouse Jerry,', eos_token='<|endoftext|>')
Traceback (most recent call last):
File "/data/app/yi/demo/text_generation.py", line 78, in <module>
main(args)
File "/data/app/yi/demo/text_generation.py", line 49, in main
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/yi/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 560, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/yi/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3019, in from_pretrained
state_dict = load_state_dict(resolved_archive_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/anaconda3/envs/yi/lib/python3.11/site-packages/transformers/modeling_utils.py", line 463, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
I'm not sure what demo/text_generation.py is? Is that launching TGI? If so, try adding --quantize gptq to the arguments
You'll also need --trust-remote-code.
Or is this your custom code? If it is, then errors about deserializing headers usually mean a bad download. Try downloading the model weights again and make sure the checksums match.