Exl2 Quants?
#3
by
eldodemi2039
- opened
I would love to test this model but I have a single 4090. Would a Exl2 quant be feasible for this model? If so, could you please create it?
Qwen-based models cannot be exl2-quantized currently. When it's supported, I will make them...
I see, thank you so much!
I would love to test this model but I have a single 4090. Would a Exl2 quant be feasible for this model? If so, could you please create it?
Turbo's added support for Qwen models now to exl2. He's uploaded some quants here:
https://huggingface.co/turboderp/Smaug-72B-exl2
I'll upload some additional quant sizes as well shortly.