Quantization made by Richard Erkhov.

mixtralnt-4x7b-test - GGUF

Model creator: https://huggingface.co/chargoddard/
Original model: https://huggingface.co/chargoddard/mixtralnt-4x7b-test/

Name	Quant method	Size
mixtralnt-4x7b-test.Q2_K.gguf	Q2_K	8.24GB
mixtralnt-4x7b-test.IQ3_XS.gguf	IQ3_XS	9.21GB
mixtralnt-4x7b-test.IQ3_S.gguf	IQ3_S	9.73GB
mixtralnt-4x7b-test.Q3_K_S.gguf	Q3_K_S	9.72GB
mixtralnt-4x7b-test.IQ3_M.gguf	IQ3_M	9.92GB
mixtralnt-4x7b-test.Q3_K.gguf	Q3_K	10.79GB
mixtralnt-4x7b-test.Q3_K_M.gguf	Q3_K_M	10.79GB
mixtralnt-4x7b-test.Q3_K_L.gguf	Q3_K_L	11.68GB
mixtralnt-4x7b-test.IQ4_XS.gguf	IQ4_XS	12.15GB
mixtralnt-4x7b-test.Q4_0.gguf	Q4_0	12.69GB
mixtralnt-4x7b-test.IQ4_NL.gguf	IQ4_NL	12.81GB
mixtralnt-4x7b-test.Q4_K_S.gguf	Q4_K_S	12.8GB
mixtralnt-4x7b-test.Q4_K.gguf	Q4_K	13.61GB
mixtralnt-4x7b-test.Q4_K_M.gguf	Q4_K_M	13.61GB
mixtralnt-4x7b-test.Q4_1.gguf	Q4_1	14.09GB
mixtralnt-4x7b-test.Q5_0.gguf	Q5_0	15.48GB
mixtralnt-4x7b-test.Q5_K_S.gguf	Q5_K_S	15.48GB
mixtralnt-4x7b-test.Q5_K.gguf	Q5_K	15.96GB
mixtralnt-4x7b-test.Q5_K_M.gguf	Q5_K_M	15.96GB
mixtralnt-4x7b-test.Q5_1.gguf	Q5_1	16.88GB
mixtralnt-4x7b-test.Q6_K.gguf	Q6_K	18.46GB
mixtralnt-4x7b-test.Q8_0.gguf	Q8_0	23.9GB

Original model description:

license: cc-by-nc-4.0

Mixtraln't 4x7B

Oh boy, a new model architecture in Transformers! Time to do profane things with it.

What if instead of training a MoE from scratch, we took some pre-trained Mistral models and shoved them in a little clown car?

Uses parts from the following models:

Works and generates coherent text. The big question here is if the hack I used to populate the MoE gates works well enough to take advantage of all of the experts. Let's find out!

Prompt format: maybe alpaca??? or chatml??? life is full of mysteries