mentee-embed-v4

A 41M-parameter trilingual text embedding model trained entirely from scratch — no pretrained backbone, no BERT, no RoBERTa. Random initialization only.

v4 adds mMARCO Arabic retrieval data (500K triplets) and a third distillation round with hard negatives, yielding significant improvements over v3 across all languages and benchmarks.

Developed by Team MenteE AI (menteeai.org) as part of an ongoing research effort to build competitive multilingual embeddings from the ground up for Arabic, English, and Urdu.

---

Key Facts

Property	Value
Parameters	41M
Embedding dimension	384
Max sequence length	128 tokens
Languages	Arabic · English · Urdu
Architecture	12-layer Transformer, custom BPE tokenizer (50K vocab)
Initialization	Random (trained from scratch)
Training data	~2.6M triplets (NLI + MS-MARCO + OPUS + MIRACL + mMARCO Arabic)
Pooling	Mean pooling
Training objective	Relational distillation + InfoNCE contrastive (teacher: multilingual-e5-base)

---

What "From Scratch" Means

Most embedding models fine-tune an existing pretrained encoder (BERT, RoBERTa, MPNet). mentee-embed-v4 does not. We:

1. Trained a custom BPE tokenizer on Arabic, English, and Urdu text
2. Initialized a 12-layer Transformer with random weights
3. Ran masked language modeling pretraining (50K steps on trilingual corpus)
4. Applied three-round contrastive distillation with hard negative mining

No pretrained checkpoint was used at any stage.

---

v4 vs v3 Improvements

Benchmark	v3	v4	Change
Bench MRR@10 (all)	0.103	0.252	+146%
MIRACL EN MRR@10	0.768	0.916	+19%
MIRACL AR MRR@10	0.447	0.874	+96%
MIRACL UR MRR@10	0.354	0.572	+62%
MS-MARCO MRR@10	0.592	0.706	+19%
STS-B Spearman	0.683	0.725	+6%

Key improvements come from:

• mMARCO Arabic — 500K Arabic retrieval triplets added to training
• 50K MLM steps — 6x more pretraining than v3 (8K steps)
• 3 distillation rounds — extra hard-negative round for better discrimination

---

Benchmark Results

All baselines evaluated under identical conditions on the same hardware (RTX 5090).

MenteE Embed Bench (Custom EN/AR/UR Pilot)

121 queries · 18 domains · dialect Arabic · Roman Urdu · hard negatives

Model	Params	EN MRR@10	AR MRR@10	UR MRR@10	Dialect AR	Roman UR	All MRR@10
mentee-embed-v4	41M	0.369	0.120	0.261	0.095	0.399	0.252
paraphrase-mpnet-base-v2	278M	0.764	0.602	0.611	0.445	0.661	0.661
paraphrase-MiniLM-L12-v2	118M	0.682	0.568	0.519	0.385	0.480	0.592
multilingual-e5-base	278M	0.667	0.352	0.413	0.310	0.430	0.481
all-MiniLM-L6-v2	23M	0.873	0.087	0.289	0.084	0.486	0.425

Protocol A — In-batch Retrieval (MIRACL)

Format: acc@1 / MRR@10

Model	MIRACL-EN	MIRACL-AR	MIRACL-UR
mentee-embed-v4	0.870 / 0.916	0.825 / 0.874	0.475 / 0.572
paraphrase-mpnet-base-v2	0.965 / 0.982	0.860 / 0.898	0.755 / 0.824
paraphrase-MiniLM-L12-v2	0.965 / 0.979	0.815 / 0.868	0.720 / 0.785
multilingual-e5-base	0.980 / 0.990	0.940 / 0.958	0.960 / 0.970
all-MiniLM-L6-v2	0.980 / 0.990	0.005 / 0.027	0.015 / 0.031

v4 achieves strong MIRACL scores despite being 6.8x smaller than mpnet-base.

Protocol C — MS-MARCO Corpus Retrieval (10K passages)

Model	MRR@10	R@5	R@100
all-MiniLM-L6-v2	0.877	0.992	0.992
multilingual-e5-base	0.856	0.992	0.996
paraphrase-mpnet-base-v2	0.803	0.964	0.988
mentee-embed-v4	0.706	0.913	0.980

STS-B (Spearman Correlation)

Model	Spearman
paraphrase-mpnet-base-v2	0.868
bge-small-en-v1.5	0.859
paraphrase-MiniLM-L12-v2	0.844
multilingual-e5-base	0.842
mentee-embed-v4	0.725

---

Speed & Efficiency

Measured on RTX 5090.

Model	Params	Sents/sec (bs=128)	Latency@1 (ms)	VRAM (MB)	Cost/1B sents
mentee-embed-v4	41M	18,115	5.15	175	$7.15
all-MiniLM-L6-v2	23M	12,445	4.71	103	$10.40
multilingual-e5-small	118M	9,749	7.88	482	$13.28
paraphrase-MiniLM-L12-v2	118M	9,456	8.17	573	$13.69
paraphrase-mpnet-base-v2	278M	5,158	8.04	1126	$25.10
multilingual-e5-base	278M	5,379	8.07	1126	$24.07

v4 is the fastest model tested — 3.5x faster than mpnet-base, 45% faster than MiniLM-L12.

Efficiency Metrics

Model	Bench MRR/1M params	Sents/sec per 1M params
mentee-embed-v4	0.525	441.82
paraphrase-MiniLM-L12-v2	0.452	80.34
paraphrase-mpnet-base-v2	0.224	18.55
multilingual-e5-base	0.166	19.35

---

Training Data

Source	Language	Triplets	Type
all-NLI (sentence-transformers)	EN	558K	NLI triplets
XNLI	AR	128K	NLI triplets
XNLI	UR	125K	NLI triplets
OPUS-100 EN-UR	EN/UR	300K	Parallel translation
OPUS-100 AR-EN	AR/EN	300K	Parallel translation
MS-MARCO BM25 triplets	EN	500K	Passage retrieval
MS-MARCO hard negatives	EN	200K	Hard retrieval
mMARCO Arabic	AR	500K	Passage retrieval
MIRACL	EN/AR/UR	~9K	Wikipedia retrieval
Total		~2.6M

---

Training Pipeline

Stage 1 — MLM Pretraining (50K steps)
  Random init → masked language modeling on trilingual corpus
  50,000 steps · batch=64 · vocab=50K BPE

Stage 2 — Teacher Encoding
  Teacher: intfloat/multilingual-e5-base (768-dim)
  Encode all 2.6M triplets to soft labels

Stage 3 — Distillation Round 1 (no hard negatives)
  InfoNCE contrastive + relational distillation
  4,000 steps · batch=512 · temp=0.05

Stage 4 — Hard Negative Mining Round 1
  GPU-accelerated top-5 mining across full 2.6M corpus

Stage 5 — Distillation Round 2 (with hard negatives)
  Same objective + mined hard negatives per anchor
  10,000 steps · batch=512 · temp=0.05

Stage 6 — Hard Negative Mining Round 2 (on improved model)
  Re-mine with the better round-2 model

Stage 7 — Distillation Round 3 (final polish)
  Train on round-2 mined negatives
  15,000 cumulative steps · batch=512 · temp=0.05

---

Usage

# pip install torch transformers tokenizers huggingface_hub
from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "مرحبا، كيف حالك؟",
    "ہیلو، آپ کیسے ہیں؟"
]

embeddings = model.encode(sentences, tokenizer=tok)
print(embeddings.shape)  # torch.Size([3, 384])

trust_remote_code=True is required — standard for custom-architecture models on HuggingFace. The code runs entirely on your machine.

Similarity search

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",   trust_remote_code=True)

query    = model.encode(["What is machine learning?"], tokenizer=tok)
passages = model.encode([
    "Machine learning is a subset of artificial intelligence.",
    "The weather today is sunny.",
    "تعلم الآلة هو فرع من فروع الذكاء الاصطناعي.",
], tokenizer=tok)

scores = query @ passages.T
print(scores)  # tensor([[0.81, 0.60, 0.79]])

---

Limitations

• Custom benchmark gap — v4 scores 0.252 vs 0.661 for mpnet-base on the custom bench; the gap reflects the difference between 2.6M training pairs and web-scale data
• Arabic and Urdu lag behind English on the custom bench, though MIRACL scores are strong (AR 0.874, UR 0.572)
• Vocabulary limited to 50K tokens trained on ~2.6M sentences — rare scripts and dialects may tokenize poorly
• Not evaluated on MTEB full suite yet

---

Citation

@misc{mentee-embed-v4-2026,
  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
             A Compute-Efficient Study of Arabic, English, and Urdu},
  author  = {Shah, Syed Syab Ahmad and Team MenteE AI},
  year    = {2026},
  url     = {https://huggingface.co/MenteEAI/mentee-embed-v4},
  note    = {MenteE AI. Apache-2.0 License}
}

---

About MenteE AI

Built by Syed Syab Ahmad Shah and Team MenteE AI. menteeai.org · syab@menteeai.org