TheBloke

/

stable-vicuna-13B-GPTQ

like 218

Rename the folder to TheBloke_stable-vicuna-13B-4bit-128g-GPTQ

I was having this same issue. I managed to get it to run by launching it with a .bat file with the line:
call python server.py --auto-devices --chat --wbits 4 --groupsize 128 --model TheBloke_stable-vicuna-13B-GPTQ

That loaded it into the webui and then I saved settings for the model, and after that I was able to load it via the webui normally. For whatever reason, sometimes Ooga seems to glitch out on trying to load a new model at the same time as you are setting the wbits/groupsize in the webui. You also can try making sure the wbits and groupsize are set right in the Ooga webui and reloading it a few times.

text-generation-webui can check the folder name for the 4bit-128g config. Then it's fine without the extra command line arguments.
If you use the UI to make a config entry instead, make sure to actually click "Save settings for this model" after it loads.

Thanks, Squish

The error is annoying and maybe I should just rename the repo. But it can be ignored. If you enter the params and then Reload The Model it will work.

Maybe I should make that clearer in the instructions. Or just rename it!

Yea I think it is just confusing mostly because people are still figuring out how to load all these different formats and tools. I probably wouldn't rename it to cater to one tool in the current ecosystem. It's a few clicks for us and probably not something you should worry much about.

Rename the folder to TheBloke_stable-vicuna-13B-4bit-128g-GPTQ

Ok, it works, but why?

Because if you rename the directory, ooba text-gen-ui can automatically fill in the GPTQ parameters:
bits = 4
groupsize = 128
model_type = llama

You can also fill them in manually at any time, and details are in the README - step 8:

How to easily download and use this model in text-generation-webui

Open the text-generation-webui UI as normal.

1. Click the Model tab.
2. Under Download custom model or LoRA, enter TheBloke/stable-vicuna-13B-GPTQ.
3. Click Download.
4. Wait until it says it's finished downloading.
5. Click the Refresh icon next to Model in the top left.
6. In the Model drop-down: choose the model you just downloaded,stable-vicuna-13B-GPTQ.
7. If you see an error in the bottom right, ignore it - it's temporary.
8. Fill out the GPTQ parameters on the right: Bits = 4, Groupsize = 128, model_type = Llama
9. Click Save settings for this model in the top right.
10. Click Reload the Model in the top right.
11. Once it says it's loaded, click the Text Generation tab and enter a prompt!

How to fix it?
Traceback (most recent call last):
File "C:\LLM\text-generation-webui\text-generation-webui\server.py", line 872, in
shared.model, shared.tokenizer = load_model(shared.model_name)
File "C:\LLM\text-generation-webui\text-generation-webui\modules\models.py", line 85, in load_model
model = LoaderClass.from_pretrained(Path(f"{shared.args.model_dir}/{model_name}"), low_cpu_mem_usage=True, torch_dtype=torch.bfloat16 if shared.args.bf16 else torch.float16, trust_remote_code=trust_remote_code)
File "C:\LLM\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\auto\auto_factory.py", line 471, in from_pretrained
return model_class.from_pretrained(
File "C:\LLM\text-generation-webui\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 2405, in from_pretrained
raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory models\TheBloke_stable-vicuna-13B-GPTQ.

Please see the README - you need to fill out the GPTQ parameters, then "save settings for this model" and "reload the model"