Is this model convertible to AWQ?
#2
by
iqdddd
- opened
Can this model be converted to AWQ in the usual way using AutoAWQForCausalLM.from_pretrained() and AutoAWQForCausalLM.quantize()?
As far as I know yes ; it should work.
Important: You may want to set the experts to use before you do this (in config.json), and/or set the experts to be used and make an AWQ for each one IE 2 experts, 3... and so on.