Hi author, would it be possible to quantize an int4 version of Klein4b as well?
#2
by someshijun - opened
4B is twice as fast as 9B-int4 when batch processing very simple tasks like watermark removal. So even though 4B isn't used as often, I'd still love to request an int4 quant for it.
I tried but it's much worse quality than bf16/int8. so I decided not release it. Sorry.
ariaotp changed discussion status to closed