Quantizezd? :(
#4
by
zaursamedov1
- opened
Who's gonna make the quantized v? I look for quantize models tho it is enterprise :D fool of me aight!
We don't currently have a quantized checkpoint, we do however have quantization support for both HF inference and vLLM. With this you can run the model for inference with either FP8 or FP6. We have adding a pre-quantized checkpoint to the hub soon though.
Thank you @jeffra Appreciate the answer!