Is GPTQ working locally on MAC (mps)
#7
by
mox
- opened
Hi,
is this GPTQ format also working on a Macbook GPU? So far I have tried the "GGUF" version, which takes a bit too long to give responses.
Thanks in advance!
the GGUF 4bit is actually the same algorithm as GPTQ, if I got that correctly. But llamacpp would not support loading the GPTQ format since it already has GGUF.
I don't believe any GPTQ loader would optimize for mac, so GGUF is your best bet.
There will be optimization done by llamacpp for Mixtral laster for sure, just be patient