3.5bpw?
#1
by
Gesard
- opened
Hey, could you do a 3.5bpw. Would fit just right on my 36gb vram.
Hey, I mostly make quants for myself and for 40BG VRAM configuration. Other thing is my system is busy with merging and I have to delete all I could to have some space for testing merges, so I even do not have resources to do it now. One way I can help are measurments: https://huggingface.co/altomek/measurements/blob/main/Midnight-Miqu-70b-v1.0_measurement.json, with them it is only needed to quantizatize to the requested size. If you already have 24GB GPU you could make this quants yourself.
Thanks, I'll try that.