MiniMind-Rerank-MoE

A Mixture-of-Experts cross-encoder reranker built on the MiniMind-3-MoE backbone (198M total / 64M active). The MoE counterpart of MiniMind-Rerank-Dense.

📌 How It Works

Same pointwise yes/no design as the Dense version: given a query and document, the model predicts whether the next token is 是 (relevant) or 否 (irrelevant). The softmax probability of 是 is the relevance score.

📊 Evaluation

C-MTEB/T2Reranking (MAP@10, no data leakage)

Model	MAP@10	Accuracy
Rerank-Dense (64M)	0.915	85%
Rerank-MoE (198M-A64M)	0.604	88%

Key Finding: MoE classification > Dense, but ranking < Dense

💡 Counterintuitive: MoE's training accuracy (88%) is higher than Dense (85%), yet its MAP@10 (0.604) is far lower (Dense: 0.915). Why?

• Pointwise ≠ Ranking: MoE excels at binary classification (yes/no), but ranking requires fine-grained discrimination between "very relevant" vs "somewhat relevant." MoE's top-1 routing fragments the sequence across experts, making it harder to form a coherent global relevance signal.
• Consistent with embedding findings: just as MoE underperforms Dense on STS (0.422 vs 0.478), the MoE architecture hurts representation quality at small scales.

Recommendation: for reranking, use MiniMind-Rerank-Dense — smaller, faster, and scores 50% higher.

🎓 Training

Config	Value
Training data	T2Reranking, 80k pairs (80/20 split, no leakage)
Epochs	3
Batch size	16
Learning rate	1e-5 (cosine)
Final loss	0.28
Final accuracy	88%

🏗️ Model Details

Item	Value
Architecture	MiniMind-3-MoE (4 experts, top-1)
Total params	198M
Active params	64M
Scoring	pointwise yes/no
Target tokens	是 (357) / 否 (1332)

🙏 Acknowledgements

• Base: MiniMind by @jingyaogong
• Route: Qwen3-Reranker
• Data: C-MTEB/T2Reranking

📄 License

Apache 2.0