Need to update environment for this one
Maybe other people are able to install that auto-gptq PR successfully in their environment, but it is a pain. I updated everything that could be a dependency and it still had cryptic errors. You may have better luck than I.
To be clear, this is an issue with installing auto-gptq from code rather than the released version. You need the latest cuda and a bunch of other updates. Bleeding edge technology needs the latest everything.
Agreed. I've gotten it to work, with these caveats:
- you need the mixtral-fix branch of AutoGPTQ
- I can't get multi-GPU to work, which means I can only run 3bit.
Yeah, in general it seems 'latest and greatest' means buggy/difficult, just wait a week or two, it will get sorted out.
Remember, most of these ppl (eg, AutoGPTQ modders) aren't being paid, either by you or anyone else, to support your hobby... or mine. :)
Sorry, from the title you can tell I was frustrated. I did get it working with the transformers update rather than AutoGPTQ. They are trying to update these tools as fast as possible. The MoE/mixtral models are the next gen. They will push the envelope until we start seeing the mamba models.