Doesn't run in latest text-generation-webui?
#6
by
Djsiwnckfusnwbducke
- opened
I've just updated to the latest text-generation-webui and loaded this model for the first time. I get errors from each of the GTPQ processors. I'd like to use ExLlama, but it returns this error:
Looking in the config.json for this model I see that the parameter it is looking for isn't there. Any suggestions?
ExLlama only works with Llama models and this is a StarCoder model type. You'll need to use AutoGPTQ
Thank you! That was the trick, didnt realize what dependancies each model has for different solvers. Thanks for providing the gptq and the guidance!