Will this work in a Google Colab notebook?
I'm new to Colab, notebooks, but not python.
Will this work in colab:
model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
model_basename=model_basename,
use_safetensors=True,
trust_remote_code=False,
device="cuda:0",
use_triton=use_triton,
quantize_config=None)
I get this from autogptq:
/usr/local/lib/python3.10/dist-packages/auto_gptq/modeling/_base.py in from_pretrained(cls, save_dir)
49 @classmethod
50 def from_pretrained(cls, save_dir: str):
---> 51 with open(join(save_dir, "quantize_config.json"), "r", encoding="utf-8") as f:
52 return cls(**json.load(f))
53
FileNotFoundError: [Errno 2] No such file or directory: 'TheBloke/OpenAssistant-Llama2-13B-Orca-8K-3319-GPTQ/quantize_config.json'
Someone else reported a similar issue and it turned out they were using an old version of AutoGPTQ, which was caused by AutoGPTQ failing to properly install. Please try the following:
!pip3 uninstall -y auto-gptq
!GITHUB_ACTIONS=true CUDA_VERSION="" pip3 install auto-gptq==0.2.2
!pip3 install transformers==4.31.0
Then test again
I've tried your commands I got an error:
Building wheels for collected packages: auto-gptq
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for auto-gptq (setup.py) ... error
ERROR: Failed building wheel for auto-gptq
Running setup.py clean for auto-gptq
Failed to build auto-gptq
ERROR: Could not build wheels for auto-gptq, which is required to install pyproject.toml-based projects
Nope.