Could not find model in Wizard-Vicuna-7B-Uncensored-GPTQ
I am trying to load TheBloke/Wizard-Vicuna-7B-Uncensored-GPTQ using autogptq
My code looks like the screenshot.
I have the model downloaded in the same directoty.
// drwxrwxrwx 3 root root 3000842 Aug 22 07:13 Wizard-Vicuna-7B-Uncensored-GPTQ/
In this Wizard-Vicuna-7B-Uncensored-GPTQ/ i have all the files. Basically i have git cloned it.
But while running my script i am seeing below error. Can anyone help.
root@a57c161a0e1c:/workspace# python3 abc.py
Traceback (most recent call last):
File "/workspace/abc.py", line 12, in
model = AutoGPTQForCausalLM.from_quantized(model_name_or_path, model_basename=model_basename, use_safetensors=True, trust_remote_code=True, device="cuda:0", use_triton=use_triton, quantize_config=None)
File "/usr/local/lib/python3.10/dist-packages/auto_gptq/modeling/auto.py", line 94, in from_quantized
return quant_func(
File "/usr/local/lib/python3.10/dist-packages/auto_gptq/modeling/_base.py", line 714, in from_quantized
raise FileNotFoundError(f"Could not find model in {model_name_or_path}")
FileNotFoundError: Could not find model in Wizard-Vicuna-7B-Uncensored-GPTQ
Please help!
Hello.
Try changing model_basename = "Wizard-Vicuna-7B-Uncencored-GPTQ-4bit-128g.no-act.order" to model_basename = "model"
Yes I recently updated all my GPTQ models for Transformers compatibility (coming very soon)
Please check the README again and you'll see that the model_basename line is now: model_basename = "model". This is true for all branches in all GPTQ models.
Or in fact you can simply leave out model_basename now:
model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
use_safetensors=True,
trust_remote_code=True,
device="cuda:0",
use_triton=use_triton,
quantize_config=None)
Because the model_basename is now also configured in quantize_config.json.
In the next 24 - 48 hours I will be updating all my GPTQ READMEs to explain this in more detail, and provide example code for loading GPTQ models directly from Transformers. I am waiting for the new Transformers release to happen before I do this, which will be today or tomorrow.