Morpho-7B-Distill

A 7B student distilled from Morpho-180B — the 180.5B collective whose merged teacher folds 64 converged domain experts into one dense 72.7B backbone.

Fact	Value
Teacher	bluemorpholimited/Morpho-180B-Merged, 4-bit NF4
Student	7B dense open-weights checkpoint, full fine-tune
Objective	L = 0.5·T²·KL(s/T ‖ t/T) + 0.5·CE, T = 2, shifted causal, answer-span masked
Data	chat-formatted rows sampled evenly across the eight domain families of Morpho-180B-Data
Schedule	2,500 steps, max length 512, cosine LR, PagedAdamW8bit
Best loss	0.7188

Evaluation

Metric	Value
Perplexity (10-doc holdout)	11.32
GSM8K (10 items)	1/10
Domain audit, 8 sectors	coherent; 2/8 loop under pure greedy
Domain audit with repetition_penalty=1.1	8/8 loop-free

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bluemorpholimited/Morpho-7B-Distill"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "Explain the difference between a Roth and a traditional IRA."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

repetition_penalty=1.1 is recommended: small students loop under pure greedy decoding, and this setting removed every observed loop in our 8-domain audit.

What it is and is not

It is a fast approximation of the teacher's style and domain vocabulary, suitable for drafting, summarisation, and as a speculative-decoding partner. It is not a replacement for the 72.7B on arithmetic or factual precision — the teacher itself scores 76.7% on GSM8K, and this student does not inherit that reliability.

Licence

Apache 2.0 for this model's weights and training recipe. See LICENSE.

Full pipeline: bluemorpholimited/Morpho-180B.

---

Release map

Stage	Artifact
Pipeline, gate history, benchmarks	Morpho-180B
4 · 64 converged SFT adapters	Checkpoints
5 · 31 DPO-aligned adapters	DPO
6 · Merged 72B weights	Merged
8 · Quantized builds	GGUF
9 · Distilled students	7B · 1.5B
Data and pipeline code	Morpho-180B-Data