EXL2 - 3.64 bpw
#2
by
zaq-hack
- opened
EXL2 ingests mixtral prompts considerably faster.
3.64 bpw plays nice with my 3090 @ 16-24k context.
https://huggingface.co/zaq-hack/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss-bpw364-h6-exl2