HavocK1/Le-Chaton-Slim-23B

Le Chaton Slim V0.1 — 23.3B Mixtral-MoE Agentic/Coding Model

#DO NOT USE. DOESN'T WORK + IN PROGRESS. ITS SHITE AS FUCK

GGUF: https://huggingface.co/HavocK1/Le-Chaton-Slim-23B-GGUF

A Mixtral-style mixture-of-experts model built by fine-tuning 10 experts from Ministral-3B-Instruct (text-only, vision stripped) and merging with mergekit-moe. Intended as a coding assistant and agentic-tool-use backbone.

⚠️ EXPERIMENTAL. Model quality evaluation is pending. This is a hobby project by someone who barely knows what they're doing. It might be terrible. If you're brave enough to test it, please drop notes

Eval ran by user simonko912 (https://huggingface.co/simonko912)

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc		0.2521	_	0.0037
- humanities	2	none	0	acc	_	0.2618	_	0.0064
- formal_logic	1	none	0	acc	_	0.3492	_	0.0426
- high_school_european_history	1	none	0	acc	_	0.2242	_	0.0326
- high_school_us_history	1	none	0	acc	_	0.2794	_	0.0315
- high_school_world_history	1	none	0	acc	_	0.2785	_	0.0292
- international_law	1	none	0	acc	_	0.1818	_	0.0352
- jurisprudence	1	none	0	acc	_	0.3056	_	0.0445
- logical_fallacies	1	none	0	acc	_	0.2822	_	0.0354
- moral_disputes	1	none	0	acc	_	0.2399	_	0.0230
- moral_scenarios	1	none	0	acc	_	0.2737	_	0.0149
- philosophy	1	none	0	acc	_	0.2154	_	0.0234
- prehistory	1	none	0	acc	_	0.2716	_	0.0247
- professional_law	1	none	0	acc	_	0.2549	_	0.0111
- world_religions	1	none	0	acc	_	0.3099	_	0.0355
- other	2	none	0	acc	_	0.2533	_	0.0078
- business_ethics	1	none	0	acc	_	0.2800	_	0.0451
- clinical_knowledge	1	none	0	acc	_	0.2491	_	0.0266
- college_medicine	1	none	0	acc	_	0.2717	_	0.0339
- global_facts	1	none	0	acc	_	0.2400	_	0.0429
- human_aging	1	none	0	acc	_	0.2735	_	0.0299
- management	1	none	0	acc	_	0.2039	_	0.0399
- marketing	1	none	0	acc	_	0.2521	_	0.0284
- medical_genetics	1	none	0	acc	_	0.2600	_	0.0441
- miscellaneous	1	none	0	acc	_	0.2593	_	0.0157
- nutrition	1	none	0	acc	_	0.2190	_	0.0237
- professional_accounting	1	none	0	acc	_	0.2376	_	0.0254
- professional_medicine	1	none	0	acc	_	0.2279	_	0.0255
- virology	1	none	0	acc	_	0.3373	_	0.0368
- social sciences	2	none	0	acc	_	0.2340	_	0.0076
- econometrics	1	none	0	acc	_	0.2895	_	0.0427
- high_school_geography	1	none	0	acc	_	0.2273	_	0.0299
- high_school_government_and_politics	1	none	0	acc	_	0.2124	_	0.0295
- high_school_macroeconomics	1	none	0	acc	_	0.2359	_	0.0215
- high_school_microeconomics	1	none	0	acc	_	0.2521	_	0.0282
- high_school_psychology	1	none	0	acc	_	0.2330	_	0.0181
- human_sexuality	1	none	0	acc	_	0.2443	_	0.0377
- professional_psychology	1	none	0	acc	_	0.2255	_	0.0169
- public_relations	1	none	0	acc	_	0.2364	_	0.0407
- security_studies	1	none	0	acc	_	0.2327	_	0.0270
- sociology	1	none	0	acc	_	0.2040	_	0.0285
- us_foreign_policy	1	none	0	acc	_	0.2800	_	0.0451
- stem	2	none	0	acc	_	0.2540	_	0.0077
- abstract_algebra	1	none	0	acc	_	0.2000	_	0.0402
- anatomy	1	none	0	acc	_	0.2519	_	0.0375
- astronomy	1	none	0	acc	_	0.2500	_	0.0352
- college_biology	1	none	0	acc	_	0.3125	_	0.0388
- college_chemistry	1	none	0	acc	_	0.2900	_	0.0456
- college_computer_science	1	none	0	acc	_	0.3400	_	0.0476
- college_mathematics	1	none	0	acc	_	0.2900	_	0.0456
- college_physics	1	none	0	acc	_	0.2647	_	0.0439
- computer_security	1	none	0	acc	_	0.1800	_	0.0386
- conceptual_physics	1	none	0	acc	_	0.2596	_	0.0287
- electrical_engineering	1	none	0	acc	_	0.3034	_	0.0383
- elementary_mathematics	1	none	0	acc	_	0.2566	_	0.0225
- high_school_biology	1	none	0	acc	_	0.2774	_	0.0255
- high_school_chemistry	1	none	0	acc	_	0.2266	_	0.0295
- high_school_computer_science	1	none	0	acc	_	0.2300	_	0.0423
- high_school_mathematics	1	none	0	acc	_	0.2296	_	0.0256
- high_school_physics	1	none	0	acc	_	0.2252	_	0.0341
- high_school_statistics	1	none	0	acc	_	0.1991	_	0.0272
- machine_learning	1	none	0	acc	_	0.2768	_	0.0425
arc_challenge	1	none	0	acc	_	0.2099	_	0.0119
		none	0	acc_norm	_	0.2509	_	0.0127
boolq	2	none	0	acc	_	0.5859	_	0.0086
gsm8k	3	flexible-extract	5	exact_match	_	0.0000	_	0.0000
		strict-match	5	exact_match	_	0.0000	_	0.0000
hellaswag	1	none	0	acc	_	0.2542	_	0.0043
		none	0	acc_norm	_	0.2663	_	0.0044
winogrande	1	none	0	acc	_	0.4909	_	0.0141
Groups	Version	Filter	n-shot	Metric		Value		Stderr
------------------	------:	------	-----:	------	---	-----:	---	-----:
mmlu	2	none		acc		0.2521	_	0.0037
- humanities	2	none	0	acc	_	0.2618	_	0.0064
- other	2	none	0	acc	_	0.2533	_	0.0078
- social sciences	2	none	0	acc	_	0.2340	_	0.0076
- stem	2	none	0	acc	_	0.2540	_	0.0077

Quick facts

Architecture	Mixtral-style MoE
Parameters	23.3B (much smaller than planned — MoE layers replaced, vision encoder stripped)
Base model	mistralai/Ministral-3-3B-Instruct-2512 (text-only)
Experts	10 routed, top-2 active per token
Training	Full fine-tune per expert, ~25k samples each (I'm broke)
Context length	128k (Ministral native)
Merge tool	mergekit-moe, hidden gate mode
Format	Ministral chat template ([INST]...[/INST])

Why it's only 23.3B instead of ~40B

Honestly I aimed for 35-40B. The MoE merge strips the base model's dense MLP layers and replaces them with a router + shared expert MLPs. On top of that I ripped out the vision encoder since this model doesn't need images. Result: smaller. If I make a V2 I might bump the expert count and switch to DeepSeek MoE (shared + routed experts) to hit the size target.

Experts

Each expert was full fine-tuned on its own dataset mix so the router has something meaningful to route toward. Only 25k samples/epoch because GPU time costs money and I'm not made of it.

#	Expert	Specialty
1	Captain	General instruction following + safety boundaries
2	Python Coding	Core Python SFT (Magicoder OSS/Evol-Instruct)
3	Verified Python	Execution-filtered code + OPC educational instruct
4	Code Reasoning	Step-by-step algorithm design, CoT (OpenCodeReasoning + OpenThoughts)
5	Tool Calling	Function calling + structured JSON (Hermes-fc + ToolACE + xLAM)
6	Agentic Workflows	Multi-step agent planning, ReAct loops (Hermes-agent + Nemotron-agentic)
7	Code Editing	Diff generation, refactoring (commitpackft)
8	Code Review	Code critique + iterative improvement (CodeFeedback)
9	General Reasoning	Math, logic, non-code problem solving
10	Safety Net	Untuned base model — catch-all for chitchat, greetings, refusals

The captain (expert #1) donates self-attention and layer norms to the whole model. The safety net (expert #10) is the raw bf16 base with no fine-tuning at all — it catches prompts that don't match any specialty so you don't get the tool-calling expert trying to process "hello".

Limitations (there are many)

• Not evaluated. I ran a local smoke test (3k samples/expert, q8_0 GGUF) and it spits out text, but I haven't done any real benchmarks. The production run trains at 25k or 75k samples per expert and I haven't tested the final merge yet.
• Low sample counts. 25k samples per expert is tiny by modern standards. Quality is likely mediocre.
• Training quality unknown. The local smoke run used 3k samples each (seq_len 1024) — results were... not great. May be the tiny dataset, may be the idea itself is bad. Production run on a RunPod A40 might fix it, might not. ¯\(ツ)/¯
• No RLHF/DPO. This is SFT-only. No preference optimization, no reward modeling. It will not be aligned beyond what the datasets provide.
• No vision. The vision encoder was completely stripped. This is text-only.

Planned V2 (if this isn't terrible)

• DeepSeek MoE architecture: shared experts (always active) + routed experts, which should handle overlapping domains better.
• ~14 experts instead of 10, for more granular specialization.
• Proper eval harness before shipping.
• Maybe a separate "fanfiction" ( ͡° ͜ʖ ͡°) creative writing fine-tune if I feel like it. No promises.

How to use

This is a standard transformers model with a Mixtral architecture. The chat template is the Ministral format:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "HavocK1/Le-Chaton-Slim-MoE",
    torch_dtype="auto",
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained("HavocK1/Le-Chaton-Slim-MoE")

messages = [
    {"role": "user", "content": "Write a Python function to reverse a linked list."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For GGUF (llama.cpp):

llama-server le_chaton_slim.gguf --n-gpu-layers 99

Datasets

The training datasets (deduped JSONL) are available at HavocK1/Le-Chaton-Slim-Experts.

Source datasets used:

• HuggingFaceTB/smoltalk, HuggingFaceH4/no_robots, Magpie-Align/Magpie-Pro-300K-Filtered, teknium/OpenHermes-2.5, HuggingFaceH4/ultrachat_200k
• ise-uiuc/Magicoder-OSS-Instruct-75K, ise-uiuc/Magicoder-Evol-Instruct-110K
• bigcode/self-oss-instruct-sc2-exec-filter-50k, OpenCoder-LLM/opc-sft-stage2
• nvidia/OpenCodeReasoning, open-thoughts/OpenThoughts-114k
• NousResearch/hermes-function-calling-v1, Team-ACE/ToolACE, Salesforce/xlam-function-calling-60k
• lambda/hermes-agent-reasoning-traces, nvidia/Nemotron-SFT-Agentic-v2
• bigcode/commitpackft
• m-a-p/CodeFeedback-Filtered-Instruction, m-a-p/Code-Feedback
• QuixiAI/dolphin-r1, HuggingFaceTB/smoltalk (NuminaMath-CoT)

Credits

Built with:

• Ministral-3B (Mistral AI)
• mergekit (Arcee AI)
• HuggingFace transformers + TRL + PEFT
• All the amazing dataset authors above 🙏
• An A40 GPU on RunPod that I paid for with actual money
• Shout out to my homies Mistral 3.5 128B, GLM 5.2 and DS V4 pro. Couldn't have done it without them