YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
🔬 Morpho-72B-MoE
Industrial-Grade Mixture-of-Experts Large Language Model
Trained Until Convergence — No Expert Left Behind
🏗️ Architecture Overview
┌─────────────────────────────────────────────────────────────────────┐
│ MORPHO-72B-MoE ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────────────────────────────┐ │
│ │ BASE MODEL │ │ MoE ROUTING LAYER │ │
│ │ Qwen2-72B │───▶│ ┌──────┐ ┌──────┐ ┌──────┐ │ │
│ │ 4-bit QLoRA │ │ │Exp 0 │ │Exp 1 │ │ ... │ ×40 │ │
│ │ 80 Layers │ │ │ │ │ │ │ │ layers │ │
│ │ 8192 d_model │ │ └──────┘ └──────┘ └──────┘ │ │
│ └──────────────┘ │ 8 experts/layer, top-2 routing │ │
│ └──────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ 64 DOMAIN EXPERTS │ │
│ ├────────────┬────────────┬────────────┬────────────┬─────────┤ │
│ │ 💰 Finance │ ⚖️ Law │ 🏥 Medical │ 💻 Tech │ 🎓 Acad │ │
│ │ 8 experts │ 8 experts │ 8 experts │ 8 experts │ 8 exp. │ │
│ ├────────────┼────────────┼────────────┼────────────┼─────────┤ │
│ │ 🏢 Business│ 🎨 Creative│ 📚 Edu │ 🌍 Social │ │ │
│ │ 8 experts │ 8 experts │ 8 experts │ 8 experts │ │ │
│ └────────────┴────────────┴────────────┴────────────┴─────────┘ │
│ │
│ Total: ~108B params │ Active: ~14B per token │ 8-bit MoE routing │
└─────────────────────────────────────────────────────────────────────┘
📊 Training Results — Convergence Dashboard
╔══════════════════════════════════════════════════════════════════════════════╗
║ EXPERT TRAINING DASHBOARD ║
╠══════════════════════════════════════════════════════════════════════════════╣
║ ║
║ EXPERT DOMAIN BEST LOSS STEPS STATUS ║
║ ───────────────────────────────────────────────────────────────────── ║
║ #0 Trading & Markets ████████░░░░░░░░░░░░ 0.0462 ✅ ║
║ #4 Accounting & Auditing ███████░░░░░░░░░░░░░ 0.0619 ✅ ║
║ #8 Constitutional Law ███████░░░░░░░░░░░░░ 0.0625 ✅ ║
║ #1 Investing & Portfolio ██████████░░░░░░░░░░ 0.1958 ✅ ║
║ #5 Taxation & Compliance ███████████░░░░░░░░░ 0.2123 ✅ ║
║ #6 Real Estate Finance ███████████████░░░░░ 0.3637 ✅ ║
║ #2 Banking & Lending █████████████████░░░ 0.4176 ✅ ║
║ #3 Insurance & Risk ██████████████████░░ 0.4429 ✅ ║
║ #7 Financial Engineering ████████████████████ 0.5049 ✅ ║
║ #9 Criminal Law ████████████████░░░░ 0.3451 ✅ ║
║ #10 Civil Law █████████░░░░░░░░░░░ 0.1389 🔄 ║
║ #11-63 (Pending) — — ⏳ ║
║ ║
║ ═══════════════════════════════════════════════════════════════════════ ║
║ CONVERGED EXPERTS: 9/64 │ AVG LOSS: 0.2756 │ BEST: 0.0462 ║
╚══════════════════════════════════════════════════════════════════════════════╝
Loss Distribution — Sorted by Performance
Expert 0 Trading ▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0.0462 🏆 BEST
Expert 4 Accounting ▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0.0619
Expert 8 Const. Law ▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0.0625
Expert 1 Investing ▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░ 0.1958
Expert 5 Taxation ▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░ 0.2123
Expert 10 Civil Law ▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░░░░░ 0.1389
Expert 9 Criminal Law ▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░░░ 0.3451
Expert 6 Real Estate ▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░ 0.3637
Expert 2 Banking ▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░ 0.4176
Expert 3 Insurance ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░ 0.4429
Expert 7 Fin. Eng. ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░ 0.5049
📈 Expert Training Progress — Loss Curves
Expert 0: Trading & Markets 🏆 BEST: 0.0462
Loss │
10.0 ┤●
8.0 ┤ ╲
6.0 ┤ ╲
4.0 ┤ ●──●
2.0 ┤ ╲──●──●
1.0 ┤ ╲──●──●──●──●
0.5 ┤ ╲──●──●──●
0.0 ┤───────────────────────────────────●──▶
0 1K 2K 3K 4K 5K 6K 7K 8K Steps
Expert 1: Investing & Portfolio BEST: 0.1958
Loss │
3.0 ┤●
2.0 ┤ ╲
1.0 ┤ ●──●
0.5 ┤ ╲──●──●──●──●
0.3 ┤ ╲──●──●──●──●
0.2 ┤ ╲──●──●
0.0 ┤──────────────────────────────────────▶
0 1K 2K 3K 4K 5K 6K 7K Steps
Expert 4: Accounting & Auditing BEST: 0.0619
Loss │
9.0 ┤●
6.0 ┤ ╲
3.0 ┤ ╲──●──●──●
1.0 ┤ ╲──●──●
0.5 ┤ ╲──●──●──●
0.1 ┤ ╲──●──●──●
0.0 ┤──────────────────────────────────▶
0 1K 2K 3K 4K 5K 6K Steps
🎯 Domain Coverage
┌────────────────────────────────────────────────────────────────────┐
│ 64 EXPERTS ACROSS 8 DOMAINS │
├────────────────────────────────────────────────────────────────────┤
│ │
│ 💰 FINANCE (8 experts) ⚖️ LAW (8 experts) │
│ ├── Trading & Markets ├── Constitutional Law │
│ ├── Investing & Portfolio ├── Criminal Law │
│ ├── Banking & Lending ├── Civil Law │
│ ├── Insurance & Risk ├── Corporate Law │
│ ├── Accounting & Auditing ├── Intellectual Property │
│ ├── Taxation & Compliance ├── International Law │
│ ├── Real Estate Finance ├── Tax Law │
│ └── Financial Engineering └── Regulatory Compliance │
│ │
│ 🏥 MEDICINE (8 experts) 💻 TECHNOLOGY (8 experts) │
│ ├── Clinical Medicine ├── Machine Learning & AI │
│ ├── Pharmacology ├── Systems & Infrastructure │
│ ├── Diagnostics & Imaging ├── Cybersecurity │
│ ├── Surgery & Procedures ├── Databases & SQL │
│ ├── Neurology ├── Networking & Cloud │
│ ├── Cardiology ├── Web Development │
│ ├── Oncology ├── DevOps & MLOps │
│ └── Public Health └── Embedded & IoT │
│ │
│ 🏢 BUSINESS (8 experts) 🎨 CREATIVE (8 experts) │
│ ├── Strategic Management ├── Creative Writing │
│ ├── Marketing & Sales ├── Graphic Design │
│ ├── Human Resources ├── Music & Audio │
│ ├── Operations & Logistics ├── Film & Video │
│ ├── Supply Chain ├── Translation & Localization │
│ ├── Project Management ├── Linguistics │
│ ├── Entrepreneurship ├── Storytelling & Narrative │
│ └── Consulting └── Content Creation │
│ │
│ 📚 EDUCATION (8 experts) 🌍 SOCIAL SCIENCES (8 experts) │
│ ├── Pedagogy ├── Psychology │
│ ├── [Expert 57] ├── Sociology │
│ ├── [Expert 58] ├── Economics │
│ ├── [Expert 59] ├── Philosophy │
│ ├── [Expert 60] ├── History │
│ ├── [Expert 61] ├── Political Science │
│ ├── [Expert 62] ├── Critical Analysis │
│ └── [Expert 63] └── [Reserved] │
│ │
└────────────────────────────────────────────────────────────────────┘
🔬 Training Pipeline
┌──────────────────────────────────────────────────────────────────────┐
│ 8-PHASE TRAINING PIPELINE │
├──────────────────────────────────────────────────────────────────────┤
│ │
│ Phase 1 Phase 2 Phase 3 Phase 4 │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ DATA │────▶│ TOKENIZE│────▶│ SFT │────▶│ DPO │ │
│ │COLLECT │ │ │ │ (64 exp)│ │ (64 exp)│ │
│ │17 repos │ │ 82 batch│ │ LoRA │ │ Pref. │ │
│ │815K ex │ │ 2048 tok│ │ r128,a256│ │ pairs │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ Phase 5 Phase 6 Phase 7 Phase 8 │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ MERGE │────▶│ VERIFY │────▶│ GGUF │────▶│DISTILL │ │
│ │ LoRA + │ │ Domain │ │ Q4_K_M │ │ 7B/14B/ │ │
│ │ Base │ │ Tests │ │ Q5_K_M │ │ 32B │ │
│ └─────────┘ └─────────┘ │ Q8_0 │ └─────────┘ │
│ └─────────┘ │
│ │
│ ════════════════════════════════════════════════════════════════ │
│ STATUS: Phase 3 IN PROGRESS │ 9/64 experts complete │
╚══════════════════════════════════════════════════════════════════════╝
⚙️ Training Configuration
| Parameter | Value | Description |
|---|---|---|
| Base Model | Qwen2-72B | 72B parameter dense transformer |
| Quantization | 4-bit NF4 (QLoRA) | Fits 72B on single 95GB GPU |
| LoRA Rank | 128 | High-rank adaptation |
| LoRA Alpha | 256 | 2× rank for stable training |
| LoRA Targets | All 7 linear layers | q, k, v, o, gate, up, down |
| Optimizer | AdamW (CPU offload) | Memory-efficient training |
| Phase 1 LR | 2e-4 (steps 0-2K) | Aggressive initial learning |
| Phase 2 LR | 5e-5 (steps 2K-5K) | Fine-tuning phase |
| Phase 3 LR | 1e-5 (steps 5K+) | Convergence phase |
| Warmup Steps | 200 | Stable training start |
| Effective Batch | 16 | Gradient accumulation |
| Max Seq Length | 2048 tokens | Full context utilization |
| Early Stop | 2000 steps patience | No premature stopping |
| Skip Threshold | 0.5 loss | Auto-skip converged experts |
| Max Epochs | 10 | Sufficient training cycles |
| Gradient Clip | 1.0 | Stable training |
| Weight Decay | 0.01 | Regularization |
🚀 Quick Start
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model
model = AutoModelForCausalLM.from_pretrained(
"bluemorpholimited/Morpho-72B-MoE",
device_map="auto",
torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("bluemorpholimited/Morpho-72B-MoE")
# Generate domain-specific response
prompt = """You are an expert in financial markets.
Explain the Black-Scholes model for option pricing."""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
📊 Data Auditability
All data is fully auditable in the dataset repo bluemorpholimited/Morpho-72B-MoE-Data:
Morpho-72B-MoE-Data/
├── raw/ # Original unprocessed datasets
│ ├── finance/
│ ├── law/
│ ├── medicine/
│ └── ...
├── tagged/ # Expert-assigned examples
│ ├── expert_00_trading/
│ ├── expert_01_investing/
│ └── ...
├── tokenized/ # Ready-to-train batches
│ ├── batch_0000.npy
│ ├── batch_0001.npy
│ └── ... (82 batches)
├── code/ # All processing scripts
│ ├── collect_data.py
│ ├── tag_experts.py
│ ├── tokenize_data.py
│ └── train_sft_experts.py
└── README.md # Full documentation
🏆 Key Differentiators
| Feature | Morpho-72B-MoE | Typical MoE |
|---|---|---|
| Training Method | Convergence-based | Fixed epochs |
| Expert Specialization | 64 domain experts | 8-16 generic |
| Data Quality | Auditable, per-expert | Opaque |
| Early Stopping | 2000-step patience | None |
| Resume Safety | HF checkpoint sync | Local only |
| Skip Logic | Auto-skip converged | Re-train all |
📜 Citation
@misc{morpho72bmoe2026,
title={Morpho-72B-MoE: Industrial-Grade Mixture-of-Experts LLM},
author={Blue Morpho Limited},
year={2026},
url={https://huggingface.co/bluemorpholimited/Morpho-72B-MoE}
}
📄 License
Apache 2.0
📧 Contact
Blue Morpho Limited
🔬 Trained with real data. Real convergence. Real results. 🔬
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support