MiniMind-Embedding-Dense

A text embedding model built on the MiniMind-3 small language model (64M params), with the training pipeline fully aligned with Qwen3-Embedding. Part of the minimind-embedding project.

📌 Introduction

This model turns the 64M MiniMind-3 causal language model into a text encoder using three key techniques:

1. Last-Token Pooling

💡 Think of a sentence as a queue of people. The last person (EOS) has "heard" everyone before them through causal attention, so they alone can summarize the whole sentence.

Instead of BERT-style bidirectional attention + mean pooling, we keep causal (unidirectional) attention and take the hidden state at the appended EOS token. The last token aggregates information from all preceding tokens via causal attention, naturally serving as a sentence summary. This maximizes reuse of pretrained weights.

2. InfoNCE + False-Negative Masking

💡 Contrastive learning is like a "find your friend" game: push the correct match closer, push strangers further away. But what if a "stranger" is actually your friend (false negative)? We detect and mask them!

The core loss is InfoNCE (contrastive). Qwen3-Embedding adds a false-negative mask: if a candidate negative is more similar to the query than the positive (beyond margin=0.1), it's likely a false negative and gets masked out.

3. Matryoshka Representation Learning (MRL)

💡 Like Russian nesting dolls — the big doll (768-dim) contains smaller dolls (512/256/128/64-dim), each fully functional at its size.

Trained so the vector works at any truncated dimension. Use 64-dim for fast coarse retrieval, 768-dim for precise re-ranking.

🏗️ Model Details

Item	Value
Architecture	MiniMind-3 (causal decoder-only Transformer)
Parameters	64M
Hidden size	768
Layers	8
Attention heads	8 (query) / 4 (KV, GQA)
Vocab size	6400
Embedding output dim	768 (MRL: truncatable to 512/256/128/64)
Max sequence length	8192 (backbone supports 32768)
Pooling	last-token
Features	L2-normalized; MRL multi-dim

🎓 Training Details (3 Stages)

Stage	Data	Loss	Steps	Final Loss	Status
1. Weakly-supervised	T2Ranking triplets (90k)	Standard InfoNCE	1,413	0.43	✅ Done
2. Supervised	T2Ranking-15 (340k, 15 hard negs)	InfoNCE + false-neg mask + MRL	63,768 (3 epochs)	0.72	✅ Done
3. Model merging	Last 5 Stage-2 checkpoints	SLERP	—	—	✅ Done

Stage 1 (left): 1.16→0.43. Stage 2 (right): 1.47→0.72, 3 epochs with epoch boundaries marked.

Hyperparameter	Value
Temperature τ	0.02
False-neg margin	0.1
Hard negatives	7 per query
Batch size	16
Learning rate	1e-5 (Stage 2), 2e-4 (Stage 1)
MRL dims	[768, 512, 256, 128, 64]

📊 Evaluation

Task	Samples	Score
ATEC	20,000	0.265
BQ	10,000	0.379
LCQMC	12,500	0.631
STSB	1,361	0.637
Average		0.478

Model	Params	C-MTEB STS
This model	64M	0.478
BGE-small-zh	24M	~0.60
BGE-large-zh	326M	~0.66
Qwen3-Embedding-0.6B	600M	~0.66

Honest note: scores are bounded by the small vocab (6400) and limited Stage-1 data (90k vs Qwen3's 150M). The value is the complete, reproducible Qwen3-Embedding pipeline.

🚀 Usage

sentence-transformers

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("Muzian/minimind-embedding-dense")

documents = ["天空呈蓝色是因为阳光散射。", "红烧肉的做法是..."]
doc_embs = model.encode(documents, normalize_embeddings=True)

queries = ["天空为什么是蓝色的"]
query_embs = model.encode(queries, prompt_name="query", normalize_embeddings=True)

scores = cos_sim(query_embs, doc_embs)

Matryoshka (truncate dimensions)

# Use only first 256 dims (smaller index, faster search)
emb_256 = embedding[:, :256]
emb_256 = F.normalize(emb_256, p=2, dim=1)

⚠️ Limitations

1. Small vocabulary (6400) — weak Chinese compression, primary bottleneck
2. Limited Stage-1 data (90k vs Qwen3's 150M) — scaling is the top improvement direction
3. SLERP merging shows minimal gain on small models (Stage2 ≈ Stage3)

🙏 Acknowledgements

• Base: MiniMind by @jingyaogong
• Route: Qwen3-Embedding
• Data: T2Ranking
• Eval: MTEB

📄 License

Apache 2.0