Even with the prompt, it still feels like a fight.
#3
by
boqsc
- opened
The model can get annoying and still refuse; meaning it acts more like average GPT model with ocassional compliance.
This especially felt in quants like q2_k.
Yeah. Lower quants are not great. Looking into ways to improve it
ehartford
changed discussion status to
closed
q2 quant is only good for large models >30b in the current state of model quantization. In the future, we might have models trained with quantization resisliance in mind so that they could do better