Morpho-180B — merged release
Morpho-180B is a 180,500,471,808-parameter collective: a 72.7B dense backbone plus 64 converged domain experts at 1,684,275,200 parameters each. This repository is the merged release of that collective — all 64 experts (DPO where available, SFT otherwise) folded into the backbone with a gated, TIES-style merge.
| Fact | Value |
|---|---|
| Parameters in this file | 72,706,859,008 (dense) |
| Collective with all 64 adapters | 180,500,471,808 |
| Shards | 37 safetensors, bf16, 145.4 GB |
| Adapters merged | 64 (31 DPO-refreshed, 33 SFT-only) |
| Adapter config | rank 128 / alpha 256, 7 target modules |
This file is dense. 180.5B is the size of the collective, not of a single forward pass here. The specialists remain available as separate adapters.
Merge method
Per tensor: trim the smallest 80% of delta magnitude, elect a sign by weighted
vote, sum only consenting deltas, then apply a per-module relative-norm cap of
0.006 so no single expert dominates a layer. δ = B @ A · (alpha/r = 2).
560/560 tensors merged in 15.1 minutes. Mean |δ|/|W| 2.70% pre-cap, 0.6%
post-cap; largest single relative change 19.3%.
Quality gate
| Candidate | Cap | PPL | × base | Verdict |
|---|---|---|---|---|
| Plain delta mean | — | 300.88 | 39.1× | rejected |
| Sign consensus | 0.020 | 49.74 | 6.47× | rejected |
| Sign consensus | 0.011 | 9.34 | 1.21× | rejected (broken grammar) |
| Sign consensus | 0.006 | 7.83 | 1.02× | shipped |
Base PPL: 7.69. Gate threshold: 1.10×.
Verification
| Run | GSM8K | PPL (40 docs) | Throughput |
|---|---|---|---|
| Backbone, untouched | 25/30 · 83.3% | 7.56 | 17.1 tok/s |
| This model | 23/30 · 76.7% | 7.68 | 19.5 tok/s |
All eight domain prompts (Finance, Law, Medicine, Engineering, Science, Math, Code, Writing) produced coherent, on-topic answers.
Honest claim: the merge is quality-neutral against its backbone (+1.6% PPL,
−2 GSM8K items at n=30). It is not a benchmark gain. The 64 specialists remain
available as adapters in
Morpho-180B-Checkpoints
and Morpho-180B-DPO;
load backbone + one adapter when you want a single domain at full specialisation.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "bluemorpholimited/Morpho-180B-Merged"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto", torch_dtype="bfloat16")
messages = [{"role": "user", "content": "Explain stare decisis and how it shapes appellate review."}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Quantized builds: Morpho-180B-GGUF.
Compact students: Morpho-7B-Distill,
Morpho-1.5B-Distill.
Licence
Adapter weights and pipeline code: Apache 2.0. The merged weights carry the
upstream backbone terms. The required notice and a full copy of the agreement
are in NOTICE and LICENSE.
Release map
| Stage | Artifact |
|---|---|
| Pipeline, gate history, benchmarks | Morpho-180B |
| 4 · 64 converged SFT adapters | Checkpoints |
| 5 · 31 DPO-aligned adapters | DPO |
| 6 · Merged 72B weights | Merged |
| 8 · Quantized builds | GGUF |
| 9 · Distilled students | 7B · 1.5B |
| Data and pipeline code | Morpho-180B-Data |
- Downloads last month
- 583