metadata

base_model:
  - Doctor-Shotgun/Mixtral-8x7B-Instruct-v0.1-LimaRP-ZLoss
  - jondurbin/bagel-dpo-8x7b-v0.2
  - NeverSleep/Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss
license: cc-by-nc-4.0
tags:
  - not-for-all-audiences
  - nsfw
  - mergekit
  - merge
  - exllamav2

Undi95/BagelMix-8x7B 3.5bpw

Exllama quant of Undi95/BagelMix-8x7B

You will need 24gb of vram to run this model at about half context (16k, you can probably go a bit higher too)

Prompt format:

ChatML? maybe? Unclear

<|im_start|>system
{sysprompt}<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
{output}<|im_end|>

Contact

Kooten on discord.