TheBloke

/

Mixtral-8x7B-Instruct-v0.1-AWQ

like 58

Getting ValueError: OC is not a multiple of cta_N = 64 at the line out = awq_inference_engine.gemm_forward_cuda(x.reshape(-1, x.shape[-1]), self.qweight, self.scales, self.qzeroes, 8) in the linear.py file from the AutoAWQ package. I am getting this error when loading the model in using Transformers.

My setup is:
python = 3.9.7
GPU = A100 40gb
Transformers = 4.36.2
AutoAWQ = 0.1.8