Pocket MoE
Pocket MoE is a compact top-2 mixture-of-experts classifier. A shared image encoder feeds four specialist MLPs while a learned router activates two experts for each example.
The benchmark compares an MoE and a parameter-matched dense MLP on the same split, then reports expert utilization, routing entropy, and each digit's dominant expert.
Reproduce
uv run python projects/tiny-vision-foundry/prepare_data.py
uv run python projects/pocket-moe/train.py
Sparse routing here demonstrates conditional computation; it does not reduce the stored checkpoint size because all four experts remain available.
Verified results
| Model | Stored parameters | Active parameters | Test accuracy | Macro F1 |
|---|---|---|---|---|
| Dense control | 5,490 | 5,490 | 96.30% | 0.9618 |
| Top-2 Pocket MoE | 5,004 | 3,608 | 96.30% | 0.9625 |
The MoE activates 34.3% fewer parameters per example than the dense control without
losing accuracy. Experts specialized strongly: digit 0 routed 86.5% to expert 3,
digit 2 routed 83.1% to expert 2, and digit 7 routed 90.9% to expert 0.
Utilization was not perfectly balanced: expert shares were 44.2%, 12.4%, 19.6%, and 23.8%, for a coefficient of variation of 0.473. The stored MoE still includes all four experts, so conditional compute should not be confused with checkpoint compression.