How to run this on runpod?
#2
by
rafa9
- opened
@TheBloke I'm getting this error when trying to load this model on runpod:
Traceback (most recent call last):
File “/root/text-generation-webui/server.py”, line 71, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name)
File “/root/text-generation-webui/modules/models.py”, line 97, in load_model
output = load_func(model_name)
File “/root/text-generation-webui/modules/models.py”, line 291, in GPTQ_loader
model = modules.GPTQ_loader.load_quantized(model_name)
File “/root/text-generation-webui/modules/GPTQ_loader.py”, line 163, in load_quantized
exit()
File “/usr/lib/python3.10/_sitebuiltins.py”, line 26, in call
raise SystemExit(code)
SystemExit: None
I'm using a pretty large machine: NVIDIA L40
32 vCPU 250 GB RAM & 280 GB Disk
30 GB Pod Volume
This is a GGML model and it looks like you're trying to load it as a GPTQ.
You want to download this instead: https://huggingface.co/Aeala/VicUnlocked-alpaca-65b-4bit