Akahsizrr/Mini-Whale-Flash

Mini-Whale Flash

An 11.5B-parameter Mixture-of-Experts model that grafts 260 coding-specialized experts — surgically extracted from DeepSeek V4 Flash (284B) — onto a Qwen3-4B host, achieving near-frontier coding ability at 1/25th the parameter cost.

Mini-Whale Flash is built on a simple observation: large MoE models like DeepSeek V4 Flash concentrate their coding ability in a small subset of experts. By profiling expert activation patterns across hundreds of coding and non-coding prompts, we identified exactly which experts fire for code, extracted them, and wired them into a compact dense host with learned bridges and per-layer routing. The result: a model that thinks like Qwen3-4B for general tasks and codes like a much larger model when the router detects coding tokens.

---

Architecture

┌────────────────────────────────────────────────────────────┐
│  Qwen3-4B Host (36 layers, frozen)                         │
│  hidden=2560, heads=16, KV heads=4, intermediate=9728      │
│  vocab=151,936, context=40K                                 │
│                                                             │
│  For each augmented layer:                                  │
│                                                             │
│  hidden ──→ Host Attention ──→ residual                     │
│         ──→ Host FFN (shared expert) ──→ residual           │
│         ──→ bridge_in (2560→4096)                           │
│              ──→ Router (sqrtsoftplus scoring)               │
│                   ──→ Top-k Coding Experts (frozen SwiGLU)   │
│                        4096→2048→4096, ~25M params each     │
│              ──→ bridge_out (4096→2560, zero-init)          │
│         ──→ residual                                        │
│                                                             │
│  260 coding experts across 43 layers (avg 6 per layer)      │
│  + 43 shared experts (1 per layer)                          │
└────────────────────────────────────────────────────────────┘

Component	Parameters	Status
Qwen3-4B host	4.0B	Frozen
260 coding experts (from DeepSeek V4 Flash)	6.5B	Frozen
43 shared experts	1.1B	Frozen
Bridges (43 × in + out, 2560↔4096)	~0.9B	Trained
Routers + repair	~0.05B	Trained
Total	~11.5B
Active per token	~5–6B

Expert selection method

Not all experts in DeepSeek V4 Flash are equal. We ran a two-stage causal selection:

1. Activation profiling: 400+ forward passes (200 coding, 200 non-coding prompts) capturing per-expert activation frequency and magnitude across all 43 layers × 256 experts.
2. Causal validation (zero-ablation): For each candidate expert, zero it out and measure the NLL increase on held-out coding prompts. Only experts whose removal hurts coding performance (and does not help non-coding performance) are kept.

Result: 260 experts survived selection. Key coding-heavy layers: L34 (14 experts), L28 (12), L36 (12), L25 (11). Only Layer 9 has zero coding experts.

Routing

Each augmented layer has a lightweight router (Linear(4096, K)) with sqrtsoftplus scoring — matching DeepSeek V4's native routing function. The router is trained to:

• Fire coding experts for coding tokens (high activation)
• Stay silent for general-language tokens (zero or minimal activation)
• Maintain load balance across experts (auxiliary load-balancing loss)

This means non-coding inference pays only the host's dense cost (~4B active params).

---

Precision & Quantization

All released weights are in BFloat16 (BF16). No post-training quantization is applied.

Component	Source precision	Released precision	Notes
Qwen3-4B host	BF16	BF16	Native training precision
Coding experts	FP4 (NVFP4) in DeepSeek V4 Flash	BF16	Dequantized during extraction
Shared experts	FP8 in DeepSeek V4 Flash	BF16	Dequantized during extraction
Bridges & routers	—	BF16	Trained from scratch in BF16

Why BF16?

DeepSeek V4 Flash stores its 11,008 routed experts in FP4 (4-bit floating point, NVIDIA NVFP4 format) and the rest of the model in FP8. This is an inference-time storage optimization for a 284B model.

During extraction, every selected expert's gate_proj, up_proj, and down_proj weights were dequantized from FP4 to full BF16 before saving. This is deliberate:

• FP4 quantization error is acceptable when you have 256 redundant experts per layer and top-6 routing. But we extracted only the best 260 — each one must carry its full representational weight.
• The bridges are trained against these expert outputs. If the experts were kept in FP4, the bridges would learn to compensate for quantization noise rather than learning the true geometric mapping between representation spaces.
• At 6.5B expert params, BF16 costs ~13 GB — well within single-GPU inference range. The precision preservation is worth the storage.

The shared experts (1 per layer in DeepSeek V4 Flash, always active) were similarly dequantized from FP8 to BF16.

For deployment: Community GGUF/AWQ quantization of the final fused model is recommended for memory-constrained setups. The expert weights quantize well to Q4_K_M or Q5_K_S given their SwiGLU structure.

---

Repository contents

├── coding-experts-v1/
│   ├── coding-experts-00000.safetensors   (9.3 GB — layers 0–21)
│   ├── coding-experts-00001.safetensors   (2.9 GB — layers 22–42)
│   └── manifest.json                      (expert indices, shapes, provenance)
├── shared-experts-v1/
│   ├── shared_experts.safetensors         (2.0 GB — 43 shared experts)
│   └── manifest.json
├── fuse2-v1-trained/
│   └── stage1_step1000/trainable_params.pt (bridge alignment checkpoint)
├── fuse2-v2-trained/
│   └── bridge_alignment_step500/trainable_params.pt (v2 bridge checkpoint)
├── profiling/
│   ├── coding_profile.json                (per-expert coding activation stats)
│   ├── noncoding_profile.json             (per-expert non-coding activation stats)
│   ├── specialization_scores.json         (coding − non-coding delta per expert)
│   └── manifest.json
└── README.md

Loading the experts

from safetensors import safe_open

# Load coding experts for a specific layer
with safe_open("coding-experts-v1/coding-experts-00000.safetensors", framework="pt") as f:
    # Expert naming: layer{LL}_expert{EEE}.{proj}.weight
    gate = f.get_tensor("layer00_expert001.gate_proj.weight")  # [2048, 4096]
    up   = f.get_tensor("layer00_expert001.up_proj.weight")    # [2048, 4096]
    down = f.get_tensor("layer00_expert001.down_proj.weight")  # [4096, 2048]

# Each expert is a SwiGLU FFN: output = down(SiLU(gate(x)) * up(x))

Serving (after assembly)

Once the full Fuse2 model is assembled (host + experts + bridges merged into a single checkpoint), it can be served with standard frameworks:

vLLM:

from vllm import LLM, SamplingParams

llm = LLM(
    model="Akahsizrr/Mini-Whale-Flash",
    trust_remote_code=True,
    dtype="bfloat16",
    max_model_len=32768,
    # MoE-aware: only ~5-6B params active per token
    gpu_memory_utilization=0.90,
)
tokenizer = llm.get_tokenizer()

messages = [{"role": "user", "content": "Write a concurrent web scraper in Rust using tokio."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

outputs = llm.generate([prompt], SamplingParams(temperature=0.6, max_tokens=2048))
print(outputs[0].outputs[0].text)

# OpenAI-compatible server
vllm serve Akahsizrr/Mini-Whale-Flash \
    --trust-remote-code \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --port 8000

SGLang:

import sglang as sgl

runtime = sgl.Runtime(
    model_path="Akahsizrr/Mini-Whale-Flash",
    trust_remote_code=True,
    dtype="bfloat16",
    mem_fraction_static=0.88,
)
sgl.set_default_backend(runtime)

@sgl.function
def code_gen(s, prompt):
    s += sgl.user(prompt)
    s += sgl.assistant(sgl.gen("response", max_tokens=2048, temperature=0.6))

state = code_gen.run(prompt="Implement a lock-free concurrent hash map in C++20.")
print(state["response"])
runtime.shutdown()

# SGLang server
python -m sglang.launch_server \
    --model-path Akahsizrr/Mini-Whale-Flash \
    --trust-remote-code \
    --dtype bfloat16 \
    --port 30000

Note: This repository currently contains the model as component weights (experts, bridges, routers, host reference). A merged single-checkpoint release will follow after the final training phase. The serving snippets above apply to the merged release.

---

Training protocol

Three-stage training on the bridges and routers (host + experts frozen):

Stage	What trains	Data	Steps	LR
1 — Bridge alignment	bridge_in + bridge_out	Coding only	1000	1e-4 cosine
2 — Router training	+ routers	50/50 coding/general	2000	5e-5
3 — Joint refinement	+ repair residuals	Full mixed	3000	2e-5 → 1e-6

Preservation KL-divergence penalizes divergence from base Qwen3-4B on general data throughout all stages.

---

Provenance

Source	Model	License	Role
Host	Qwen/Qwen3-4B	Apache 2.0	Dense reasoning backbone
Expert donor	deepseek-ai/DeepSeek-V4-Flash	MIT	Source of 260 coding experts

Extraction performed on 2026-07-30. Source model SHA verified. All expert tensors hash-checked after dequantization.

---

Design philosophy

Mini-Whale Flash is an experiment in knowledge surgery: instead of distilling a large model into a small one (lossy compression), we transplant the exact weights responsible for a capability. The experts are not approximations — they are the literal DeepSeek V4 Flash experts that fire when the model writes code.

The bridges learn the geometric mapping between two representation spaces (Qwen3's 2560-dim and DeepSeek's 4096-dim expert space). The router learns when to call for help. The host never forgets what it knew, because its weights are never touched.

This is cheaper than distillation (hours of bridge training vs. days of full-model training), more faithful (no information loss through a student bottleneck), and more interpretable (you can ablate the coding path entirely and recover the exact base model).

---

Citation

@misc{miniwhaleflash2026,
  title={Mini-Whale Flash: Causal Expert Extraction and MoE Fusion from DeepSeek V4 Flash into Qwen3-4B},
  author={akashizrr},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/akashizrr/Mini-Whale-Flash}
}

---

Limitations

• The model is released as component weights (experts + bridges + routers), not a single merged checkpoint. Assembly requires the Fuse2 model class.
• Router training was conducted on a limited corpus (~10K examples). Routing quality may degrade on out-of-distribution coding languages or highly ambiguous prompts.
• The 2560↔4096 bridge is a single linear layer per direction — a bottleneck that loses some fine-grained expert information. Multi-layer or rank-expanded bridges are a natural next step.
• DeepSeek V4 Flash's experts were trained with a 129K vocabulary; Qwen3-4B uses 151K. The bridge implicitly handles this mismatch, but token-level alignment is not guaranteed for rare tokens.
• No RLHF/DPO was applied. The model inherits Qwen3-4B's alignment and DeepSeek V4 Flash's raw expert knowledge without additional preference tuning.