Unable to load using Ooobabooga on CPU, was hoping someone would know why
I can load the model using cpu vs gpu (Not enough memory on GPU), but when I try to communicate with it, I get the following..
2023-07-08 17:19:25 INFO:Loaded the model in 17.42 seconds.
Traceback (most recent call last):
File "/home/username/oobabooga_linux/text-generation-webui/modules/callbacks.py", line 55, in gentask
ret = self.mfunc(callback=_callback, *args, **self.kwargs)
File "/home/username/oobabooga_linux/text-generation-webui/modules/text_generation.py", line 289, in generate_with_callback
shared.model.generate(**kwargs)
File "/home/username/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 422, in generate
with torch.inference_mode(), torch.amp.autocast(device_type=self.device.type):
File "/home/username/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 411, in device
device = [d for d in self.hf_device_map.values() if d not in {'cpu', 'disk'}][0]
IndexError: list index out of range
Output generated in 0.38 seconds (0.00 tokens/s, 0 tokens, context 67, seed 1158767456)
I have disk and cpu selected. Also have autogptq and gptq_for_llma selected. Still not loading properly. Was wondering if anyone would know why this would be and what settings I'm missing?
@split-paradox Has this issue been fixed? I'm facing the same issue.
No, it hasn't.