Transformers
GGUF
English
mixtral
text-generation-inference

Something wrong with q2

#1
by eastwind - opened

Not sure what's happening but I'm not getting coherent output from the q2 version. Q3 works fine on the exact same prompt.

I'm using this notebook

https://colab.research.google.com/drive/1An-CJb3bxBmNn33cxLjxAm5Uz3lfuVTw?usp=sharing#scrollTo=39kCG2B6JFEd

Q2 and Q6 may be broken, a lot of people mentioned that.

Sign up or log in to comment