Issues with Auto
I have successfully installed 12.1 CUDA toolkit and successfully did the following instructions:
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip install .
Then I check using pip freeze that it is installed in a weird way:
auto-gptq @ file:///oobabooga_windows/text-generation-webui/modules/AutoGPTQ
Trying to load it using test-generation-webui, I get this error:
Traceback (most recent call last): File “\oobabooga_windows\text-generation-webui\server.py”, line 69, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name)
File “\oobabooga_windows\text-generation-webui\modules\models.py”, line 94, in load_model output = load_func(model_name)
File “\oobabooga_windows\text-generation-webui\modules\models.py”, line 294, in AutoGPTQ_loader import modules.AutoGPTQ_loader
File “\oobabooga_windows\text-generation-webui\modules\AutoGPTQ_loader.py”, line 3, in from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig ModuleNotFoundError: No module named ‘auto_gptq’
Not really sure where to go for, here honestly. All the other models have worked without issue
(first time trying Auto-gptq). I even uninstalled and tried pip install auto-gptq, which also gave the same issue.
The one-click installer for text-gen-ui creates a conda environment and installs all its packages in there. So if you used the installer, you'd need to activate that conda environment before manually installing any packages.
As you're using CUDA 12.1, you can't use the pre-built binaries for pytorch 2.0.0 any more. Have you you compiled pytorch 2 from source (and in the same environment as you're trying to run text-gen-ui?) If not, nothing will work. Using CUDA toolkit 12.x is a pain at the moment, for that reason.
This is what I've been using successfully with CUDA 12.1:
torch 2.1.0.dev20230526+cu121
torchvision 0.16.0.dev20230527+cu121
Did not compile torch from source, installed nightly builds.
Ah OK, fair enough
I've been compiling 2.0.1 from source, in case anything changed in 2.1.0 that might affect anything I do