2.5bpw and 2.55bpw
#1
by
sandwichdoge
- opened
Hello, exllama2 recently added support for 8bit-cache, we can run 2.5bpw on 10k context and 2.55bpw on 6k context on 24GB of VRAM now. Turboderp said 2.55bpw is the smallest bit rate that's still somewhat stable.
Could you quantize for these two? Thank you.
I'll add it to the list for this model.
Uploaded.
LoneStriker
changed discussion status to
closed