Yo-Purple/Meditor-2-3.4B-Universal

Meditor 2 3.4B Universal — Universal Hierarchical Mixture-of-Experts Engine

Created by Yo-Purple
3.41B Stored Parameters • ~1.82B Active Compute per Token • Standard Qwen2-MoE Topology
Upcycled from SmolLM2-1.7B-Instruct (11 Trillion Pretrained Tokens) & Meditor-2 HMoE
Compatible with Transformers (trust_remote_code=False), vLLM, Ollama, LM Studio, and llama.cpp

---

🌟 Overview

Meditor 2 Universal is the standardized, universal-architecture release of Meditor-2. In Meditor-2's foundational release, the model introduced a fine-grained Two-Tier Hierarchical Mixture-of-Experts (HMoE) dividing knowledge across:

• 1 Dedicated Shared Anchor Expert: 8,192 intermediate neurons preserving 100% of SmolLM2's foundational capacity and world knowledge.
• Tier 1: Macro Domain Router: Dividing cognition into 3 primary domains:
  1. Logic, Mathematics & Deductive Reasoning (Sub-Experts 0–4)
  2. Natural Language, Dialogue & Creative Synthesis (Sub-Experts 5–9)
  3. Code, Structured Systems & Technical Facts (Sub-Experts 10–14)
• Tier 2: Micro Sub-Expert Routers: 5 specialized sub-experts per domain across 24 transformer layers (15 total sub-experts, 768 intermediate dimension each).

By mathematically compiling this two-tier routing tree into the industry-standard Qwen2MoeForCausalLM (qwen2_moe) topology:

• 100.0% Bit-Exact Expert Preservation: All 15 trained specialized sub-experts and the 8,192-dim shared anchor expert are preserved with bit-exact fidelity.
• Compiled Gating (98.4%+ Agreement): Macro domain affinities and micro sub-expert specializations are compiled into a 15-class router that matches the original hierarchical routing decisions.
• Zero Custom Drivers Required: Eliminates custom Python dependencies (trust_remote_code=False), running natively across standard Hugging Face Transformers, vLLM, Ollama, and LM Studio.
• Massive Inference Speedup: Achieves up to 55.5 tokens/sec on an RTX 4060 GPU (4.6× faster than the custom Python forward loop).

---

📊 Model Architecture Specifications

Specification	Metric	Notes
Model Name	Meditor-2-3.4B-Universal	Official Hugging Face ID
Model Creator	Yo-Purple	Official Creator
Architecture	Qwen2MoeForCausalLM (qwen2_moe)	Standard Universal MoE
Foundation Base	Derived from Yo-Purple/Meditor-2-3.4B-A1.8B-HMoE & SmolLM2-1.7B-Instruct	11 Trillion Tokens Pretraining
Total Stored Parameters	3,411,003,392 (~3.411B)	Stored in Safetensors (BF16)
Active Parameters per Token	1,825,507,328 (~1.825B)	53.52% Active Compute Ratio
Layers	24 Transformer Layers	32 Heads, 32 KV Heads ($d_{head} = 64$)
Hidden Size ($d_{model}$)	2,048	Full-precision activations
Shared Expert FFN	8,192 intermediate size	Always active (Anchor Expert)
Routed Experts	15 Sub-Experts (768 intermediate size each)	5 Math, 5 Language, 5 Code
Routing Strategy	Top-1 Routed Expert + 1 Dedicated Shared Expert	Top-1 per token execution
Context Window	524,288 tokens (~500K)	RoPE theta: 130,000.0
Precision	BFloat16 (~6.8 GB)	Fits comfortably on 8GB VRAM (RTX 4060)

---

🔬 Standardized Benchmark Evaluation Results

Empirically evaluated locally on an NVIDIA GeForce RTX 4060 8GB Desktop GPU with PyTorch 2.6 (trust_remote_code=False).

1. Multi-Expert Performance & Progression

Benchmark / Dimension	SmolLM2 Base	Meditor-2 HMoE Baseline	Meditor-2 Universal (Qwen2-MoE)	Notes
Reasoning Perplexity (PPL)	7.35 (Loss: 1.9944)	3.48 (Loss: 1.2461)	2.48 (Loss: 0.9063)	-66.3% PPL Reduction!
ARC Science & Commonsense	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	Zero Catastrophic Forgetting
Direct Arithmetic Accuracy	40.0%	70.0%	60.0%	Raw Autoregressive Decoding
Tool-Augmented Mental Math	100.0%	100.0%	100.0%	Safe AST Coprocessor Precision
Prefill Latency (256 tokens)	~300 ms	256.07 ms	85.95 ms	3.0× Faster Ingestion
Average Generation Speed	~11 tok/s	12.08 tok/s	55.51 tok/s	4.6× Faster Decode
Active VRAM Consumption	~6.4 GB	6.42 GB	6.38 GB	Fits within physical 8GB GDDR6

---

💻 Quick Start & Deployment

1. Hugging Face transformers (Zero Remote Code Required)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Yo-Purple/Meditor-2-3.4B-Universal"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto"  # trust_remote_code=False is default
)

prompt = "<|im_start|>user\nWhat is 15 * 28?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

2. vLLM High-Throughput Serving

Serve an OpenAI-compatible endpoint with standard vLLM:

vllm serve Yo-Purple/Meditor-2-3.4B-Universal --port 8000

---

3. LM Studio / llama.cpp Deployment & GGUF Models

Because Meditor-2-3.4B-Universal conforms to standard qwen2_moe architecture, it runs natively in LM Studio, Ollama, and llama.cpp with zero custom drivers.

Available GGUF Quantizations

Quantization	File	Size	Description
BF16	meditor-2-3.4b-universal-bf16.gguf	6.54 GB	Master uncompressed weights
Q8_0	meditor-2-3.4b-universal-q8_0.gguf	3.48 GB	Near-lossless 8-bit precision
Q6_K	meditor-2-3.4b-universal-q6_k.gguf	2.69 GB	Ultra-high fidelity 6-bit
Q5_K_M	meditor-2-3.4b-universal-q5_k_m.gguf	2.33 GB	High quality 5-bit balanced
Q5_K_S	meditor-2-3.4b-universal-q5_k_s.gguf	2.27 GB	Compact 5-bit small
Q4_K_M	meditor-2-3.4b-universal-q4_k_m.gguf	2.00 GB	Recommended: Peak speed (~76 tok/s on RTX 4060)
Q4_K_S	meditor-2-3.4b-universal-q4_k_s.gguf	1.88 GB	Compact 4-bit small
Q3_K_M	meditor-2-3.4b-universal-q3_k_m.gguf	1.60 GB	Lightweight 3-bit for low memory / laptops

To run in LM Studio:

1. Search for Yo-Purple/Meditor-2-3.4B-Universal or copy the .gguf file to ~/.lmstudio/models/Yo-Purple/Meditor-2-Universal/.
2. Select your desired quantization (e.g. Q4_K_M or Q5_K_M).
3. Enjoy blazing fast local inference!

---

📜 Citation & Credits

• Two-Tier HMoE Architecture & Expert Training: Yo-Purple
• Universal Remapping & Router Compilation: Yo-Purple
• Pretrained Foundation: HuggingFaceTB/SmolLM2-1.7B-Instruct
• License: Apache 2.0