Error when loading the model in ooba's UI (colab version) #3

by - opened

I'm sure I must be doing something wrong, but I managed to follow the instructions succesfully on ooba's UI through colab until the last step (reload model) when I got this error. I don't know where my mistake is as I am pretty new to this.
Traceback (most recent call last):
File “/content/text-generation-webui/”, line 59, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name)
File “/content/text-generation-webui/modules/”, line 157, in load_model
from modules.GPTQ_loader import load_quantized
File “/content/text-generation-webui/modules/”, line 15, in
import llama_inference_offload
ModuleNotFoundError: No module named ‘llama_inference_offload’

This happens when you don't have GPTQ-for-LLaMa installed in text-generation-webui/repositories. Please do the following:

cd text-generation-webui # wherever you have it installed
mkdir -p repositories
cd repositories
git clone -b cuda GPTQ-for-LLaMa
cd GPTQ-for-LLaMa
python setup_cuda install

This will install GPTQ-for-LLaMa and now you can do GPTQ inference.

Alternatively, here is a Google Colab Notebook already set up which you could use as a base:

Thanks, it works now. Bless you.

This comment has been hidden

Hey, I've been trying to use the colab link as a base. I've ticket everything, but I'm not getting the link. Am I missing something?

Edit: The last line when trying to load the model says " error: unrecognized arguments: --cai-chat"

That mode has been deprecated, use --chat instead.

I used the pre setup google collab link, so I don't know how to use --chat instead. I'm new to this, so let me know if I just edit this myself or need to look for a different link

I don't really use google collab, but look under section 3. Launch and find

if chat:

...and replace --cai-chat with --chat

I'll try that later and edit if it worked, thank you :)


Yeah that should work - --cai-chat was the old argument that changed a month or two ago. That Google Notebook hasn't been updated in a while

This time I got an error at the end "NotADirectoryError: [Errno 20] Not a directory: 'characters'" don't know what to do about it. Is there an alternative to run this? My pc probably won't be able to run this locally, that's why I opted for collab instead.


I have updated the Notebook so it uses AutoGPTQ, and the issue with characters is gone.

Please try: . This link works but it has a lot of error inside, whenever i upload a character it shows error and even if I created a character it shows the same error. Could you run this and see what's the issue here?

Please fix colab

I have updated the Notebook so it uses AutoGPTQ, and the issue with characters is gone.

Please try:

Now I'm getting this " error: unrecognized arguments: --autogptq"

Sign up or log in to comment