RichardErkhov/Chickaboo_-_Chicka-Mixtral-3x7b-gguf

Quantization made by Richard Erkhov.

Chicka-Mixtral-3x7b - GGUF

Model creator: https://huggingface.co/Chickaboo/
Original model: https://huggingface.co/Chickaboo/Chicka-Mixtral-3x7b/

Name	Quant method	Size
Chicka-Mixtral-3x7b.Q2_K.gguf	Q2_K	6.33GB
Chicka-Mixtral-3x7b.IQ3_XS.gguf	IQ3_XS	7.08GB
Chicka-Mixtral-3x7b.IQ3_S.gguf	IQ3_S	7.48GB
Chicka-Mixtral-3x7b.Q3_K_S.gguf	Q3_K_S	7.46GB
Chicka-Mixtral-3x7b.IQ3_M.gguf	IQ3_M	7.63GB
Chicka-Mixtral-3x7b.Q3_K.gguf	Q3_K	8.28GB
Chicka-Mixtral-3x7b.Q3_K_M.gguf	Q3_K_M	8.28GB
Chicka-Mixtral-3x7b.Q3_K_L.gguf	Q3_K_L	8.97GB
Chicka-Mixtral-3x7b.IQ4_XS.gguf	IQ4_XS	9.32GB
Chicka-Mixtral-3x7b.Q4_0.gguf	Q4_0	9.73GB
Chicka-Mixtral-3x7b.IQ4_NL.gguf	IQ4_NL	9.83GB
Chicka-Mixtral-3x7b.Q4_K_S.gguf	Q4_K_S	9.82GB
Chicka-Mixtral-3x7b.Q4_K.gguf	Q4_K	10.43GB
Chicka-Mixtral-3x7b.Q4_K_M.gguf	Q4_K_M	10.43GB
Chicka-Mixtral-3x7b.Q4_1.gguf	Q4_1	10.8GB
Chicka-Mixtral-3x7b.Q5_0.gguf	Q5_0	11.87GB
Chicka-Mixtral-3x7b.Q5_K_S.gguf	Q5_K_S	11.87GB
Chicka-Mixtral-3x7b.Q5_K.gguf	Q5_K	12.23GB
Chicka-Mixtral-3x7b.Q5_K_M.gguf	Q5_K_M	12.23GB
Chicka-Mixtral-3x7b.Q5_1.gguf	Q5_1	12.94GB
Chicka-Mixtral-3x7b.Q6_K.gguf	Q6_K	14.15GB
Chicka-Mixtral-3x7b.Q8_0.gguf	Q8_0	18.32GB

Original model description:

license: mit pipeline_tag: text-generation tags: - merge - mergekit - mistral - moe - conversational - chicka

Model Description

This model is a Mixture of Experts merged LLM consisting of 3 mistral based models:

base model/conversational expert, openchat/openchat-3.5-0106

code expert, beowolx/CodeNinja-1.0-OpenChat-7B

math expert, meta-math/MetaMath-Mistral-7B

This is the Mergekit config used in the merging process:

base_model: openchat/openchat-3.5-0106
experts:
  - source_model: openchat/openchat-3.5-0106
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
    - "I want"
  - source_model: beowolx/CodeNinja-1.0-OpenChat-7B
    positive_prompts:
    - "code"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"
    - "C#"
    - "C++"
    - "debug"
    - "runtime"
    - "html"
    - "command"
    - "nodejs"
  - source_model: meta-math/MetaMath-Mistral-7B
    positive_prompts:
    - "reason"
    - "math"
    - "mathematics"
    - "solve"
    - "count"
    - "calculate"
    - "arithmetic"
    - "algebra"

Open LLM Leaderboards

Benchmark	Chicka-Mixtral-3X7B	Mistral-7B-Instruct-v0.2	Meta-Llama-3-8B
Average	69.19	60.97	62.55
ARC	64.08	59.98	59.47
Hellaswag	83.96	83.31	82.09
MMLU	64.87	64.16	66.67
TruthfulQA	50.51	42.15	43.95
Winogrande	81.06	78.37	77.35
GSM8K	70.66	37.83	45.79

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("Chickaboo/Chicka-Mistral-3x7b")
tokenizer = AutoTokenizer.from_pretrained("Chickaboo/Chicka-Mixtral-3x7b")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])