license: cc-by-nc-4.0
NeverSleep's Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss but 17GB at 2BPW+
the other 14 shannons will be remembered. HQQ quantized to 2 bits with 4 bit attention. Fits on a 3090 with room to grow. Supports full 32k context. I will not combine those assertions.
The attention tensors are 4 bit because mixtral reuses it for each expert - so it's only adding 0.4 GB and the quality improve dramatically. See this but also it's horny and also it's dying of chatml m<|align>|ant tokenis|s>.
This is a 2+4 bit quantization of noromixmaidblah (just scroll down) using an emerging and aparrently very robust quantization method Half-Quadratic Quantisation. It ultimately squeezes it's tokens out of HF Transformers, not one ofthe lesser inference tools. So what's juicy about this is that it functions with full Transformers sampler and tokeniser support but you only need a 3090 instead of a H100! Truly emancipatory
...I'll do something smaller next time.
My unwitting and presumably unwilling collaborators were the very clever people at mobiusml - see their freaky maths at their github blog mini paper thing for HQQ. It's compatible with HF Transformers (including contrastive search baybee!) and is supported out of the box (I think) on text-generation-webui. For mobius's own description of what this is, see the template I followed, their quantization of a vanilla mixtral at mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-attn-4bit-moe-2bit-HQQ
my best guess at parsing the HQQ source is that it works by sort of... 'JIT de-quantizing' its weights as transformers requests them, back to 16 bit floats. I have no idea, really. Phind seemed to think I had the right idea. If you prefer talking to human beings from being lied to by language models (why are you here?) you could probably ask the MobiusML - they seem friendly and compsci/engineer types tend to enjoy talking about their research and development. Weirdos.
I think this is a functioning quant from one of everone's favorite norovirus inspired language models, Noromaid. I wouldn't know - I can't load 90 gigabytes of BF16 so this is my first few minutes too.
#### scroll past my oom-killer nightmare log for full credits and instructions on how to actually use this model by the people who tuned it. Even if you do want to know what I've learned - you're better off just asking me than trying to parse this
It took me all day to figure this out. It turns out that while HQQ will go ahead and fill 180GB of memory to do this - there's absolutely no reason for it! I did this from a slow**, 200 GB swap partition.
On the off chance someone at Mobius see this - please don't ask transformers to load a 45B param model on to the CPU if you're not actually going to... call the model at all? It took ten minutes at SATA 2 speeds - and that was because it was padded to FP32 (CPU mode, right?).45 Gigaweights \* 2 Bytes per weight \* fp32/bf16 = 180 GB of system memory allocated.
I wish I had one of those.
**May have been zswap's fault. I'm pretty sure 200MB/s and an idle CPU isn't the best you can hope for when you're doing sequential reads from a 4.0x4 NVME device? My GPU fell asleep between optimization passes. It even has a Gamer LED on it. I'll fix my sysctl next time.
- Try
$ python -i untitled.py
having saved that script from the mobius hf repo because you'll be spending a while in IDLE figuring out
>>> model.save_quantized("/absolute/path/noromaid")
at the end and trust me, quantizing something chunky and then watching python shred it because the save directory is somehow a recursive lambda function and not a string is heartbreaking. I don't know if it was supposed to emit more than the model.pt and the config.json but I'm taking what I can get.
If anyone's looking to donate I could do with an Epyc Rome and perhaps another pair of H100s? I've embedded my XMR address in attention tensors with help from a realy horny embedding so when it starts generating gibberish right before the good stuff just paste that in to feather and send me all your money. Thanks! :)
i'm joking. that's a joke. I didn't do that.
Original README from the Neversleep twins follows:
license: cc-by-nc-4.0
Disclaimer:
This model is experimental, do not expect everything to work.
This model uses the Chatml prompting format
Beeg noromaid on steroids. Suitable for RP, ERP.
This model was trained on the Zloss fork of Charles, and should fix issue the model had.
Use Chatml prompt format, but not the special token.
The reason is that Axolotl merge the finetune with the base model at 1.0 weight basically, but this is too much, so I use another script available HERE to merge with less weight, sadly, it don't take the special Chatml token. It's like Orca2 for the matter.
Credits:
- Undi
- IkariDev
Description
This repo contains FP16 files of Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss.
Ratings:
Note: We have permission of all users to upload their ratings, we DONT screenshot random reviews without asking if we can put them here!
No ratings yet!
If you want your rating to be here, send us a message over on DC and we'll put up a screenshot of it here. DC name is "ikaridev" and "undi".
Prompt format: Chatml
<|im_start|>system
{sysprompt}<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
{output}<|im_end|>
Datasets used:
- Aesir 1, 2 & 3 modified by us, credit to (MinervaAI / Gryphe)
- LimaRP-20231109 (Lemonilia)
- ToxicQAFinal (NobodyExistsOnTheInternet
- No-robots-ShareGPT (Doctor-Shotgun)
Others
Undi: If you want to support me, you can here.
IkariDev: Visit my retro/neocities style website please kek