Morpho-1.5B-Distill

📄 Reports: Full release report (PDF, 63 pp.) · Model card · Verification report

A 1.5B student distilled from Morpho — the merged 72B teacher that folds 64 converged domain experts into one dense backbone. Same recipe as Morpho-7B-Distill, smaller student.

• Teacher: bluemorpholimited/Morpho-72B-MoE-Merged, 4-bit NF4
• Student: Qwen/Qwen2-1.5B, full fine-tune
• Objective: L = 0.5·T²·KL(s/T ‖ t/T) + 0.5·CE, T = 2, answer-span masked
• 2,500 steps, max length 512, best loss 1.5750

Evaluation

Metric	Value
Perplexity (10-doc holdout)	13.16
GSM8K (10 items)	2/10
Domain audit, 8 sectors	on-topic; 6/8 loop under pure greedy
Domain audit with repetition_penalty=1.1	8/8 loop-free

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bluemorpholimited/Morpho-1.5B-Distill"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

ids = tok.apply_chat_template([{"role": "user", "content": "What is stare decisis?"}],
                              add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=200, do_sample=False, repetition_penalty=1.1)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Scope

This is a demo-sized model: it demonstrates the distillation pipeline end to end and produces fluent, on-topic short answers with repetition_penalty=1.1. It has little factual depth and should not be used where correctness matters. Use the 72B for real work, or the 7B student for a middle ground.

Apache 2.0.

---

Release map

Stage	Artifact
Pipeline, gate history, benchmarks	Morpho-72B-MoE
4 · 64 converged SFT adapters	Checkpoints
5 · 31 DPO-aligned adapters	DPO
6 · Merged 72B weights	Merged
8 · Quantized builds	GGUF
9 · Distilled students	7B · 1.5B
Data and pipeline code	Morpho-72B-MoE-Data