nepali-e5-base-v1

A 278M-parameter, 768-dimensional retrieval embedding model for Nepali, fine-tuned from intfloat/multilingual-e5-base.

Built because there was no Nepali-focused retrieval embedding model. The Nepali models that existed on the Hub were sentence-similarity models, and general multilingual retrievers of comparable quality were twice the size.

Two things it does that nothing else in its class does:

1. On a 50,000-passage Nepali retrieval benchmark it matches bge-m3 and multilingual-e5-large — models with twice the parameters and 1024-dim vectors — at 2.4-4× their indexing throughput.
2. It is the only model tested with any functional handling of romanized Nepali (Nepali typed in Latin script). Every off-the-shelf multilingual retriever scores at or near chance there.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Ganu000/nepali-e5-base-v1")

# The "query: " and "passage: " prefixes are REQUIRED -- they are inherited from e5 and were
# used during fine-tuning. Omitting them costs roughly 10 nDCG points.
queries = model.encode(["query: सगरमाथा कति अग्लो छ?"])
passages = model.encode(["passage: सगरमाथा विश्वको सर्वोच्च शिखर हो। यसको उचाइ ८,८४९ मिटर छ।"])

similarity = model.similarity(queries, passages)   # cosine

• Output: 768-dimensional, cosine similarity
• max_seq_length: 256 tokens (~600 Devanagari characters). Longer input is silently truncated — chunk your documents below this.
• Scripts: Devanagari Nepali and English are both well supported. Romanized Nepali works partially — see the section below before relying on it.

Script support

input	supported	notes
query: सगरमाथा कति अग्लो छ?	✅	primary case, nDCG@10 0.6373
query: How tall is Mount Everest?	✅	64% of training rows have English on one side
query: Sagarmatha kati aglo chha?	⚠️	partial, nDCG@10 0.2983 — read the caveat

Benchmark

92 hand-written natural Nepali questions (e.g. कञ्चनजङ्घा कुन देशको सीमामा छ?, सुगौली सन्धि कहिले भएको थियो?) against a 50,000-passage corpus drawn from Nepali Wikipedia. Every model was evaluated on the identical corpus, capped at the same 256 tokens, with each model's own required prefixes or prompts applied.

The 90 Wikipedia articles supplying gold answers were excluded from this model's training data. Gold articles were excluded whole, not just their lead paragraph, because a later paragraph of the right article frequently also answers the question.

Devanagari queries (50,000-passage corpus)

model	params	dim	nDCG@10	R@1	R@5	R@10	docs/s
multilingual-e5-large	560M	1024	0.6590	0.5326	0.7283	0.7826	604
nepali-e5-base-v1	278M	768	0.6373	0.4783	0.7500	0.8043	2464
bge-m3	568M	1024	0.6287	0.5109	0.7065	0.7609	1033
multilingual-e5-base (our base)	278M	768	0.5993	0.4565	0.6957	0.7500	2409
Yunika/sentence-transformer-nepali	238M	768	0.5872	0.4348	0.6739	0.7609	3270
universalml/Nepali_Embedding_Model	560M	1024	0.5630	0.4022	0.6304	0.7391	1059
jangedoo/all-MiniLM-L6-v2-nepali	23M	384	0.3419	0.1957	0.4130	0.5109	5749
syubraj/sentence_similarity_nepali	82M	768	0.0493	0.0326	0.0543	0.0761	4917

docs/s is batch indexing throughput on an RTX 4050 laptop at 256 tokens.

Romanized queries (same corpus, same gold passages)

model	nDCG@10	answer in top 10
nepali-e5-base-v1	0.2983	35 / 92
bge-m3	0.0738	8 / 92
multilingual-e5-large	0.0205	4 / 92
multilingual-e5-base	0.0000	0 / 92

Every off-the-shelf model is non-functional here. On a 50,092-passage corpus, 0-8 hits out of 92 is at or near chance. None of them has seen romanized Nepali in any form.

⚠️ Read 0.2983 as a ceiling, not a field measurement. The benchmark's romanized queries were produced by the same rule-based transliterator that generated the training anchors, so the model is being tested on spellings it was explicitly taught. Real users spell inconsistently and apply medial schwa deletion the transliterator does not (sagarmatha, not sagaramatha). Performance on human-typed romanized Nepali is unverified and will be lower.

If romanized input matters to your application, the stronger architecture is to transliterate Roman → Devanagari at query time (e.g. with ai4bharat's IndicXlit) and use the Devanagari path, which scores 0.6373. All of the model's pretrained knowledge lives in Devanagari.

What the numbers say

Tied with the 560M/1024-dim models. −2.2 nDCG against e5-large, +0.9 against bge-m3. Both inside the ~4-point noise floor of a 92-query benchmark. Same quality, 2.4-4× the indexing throughput, 25% smaller vectors.

Best Recall@5 and Recall@10 in the field, though by 2.2 points each — real but marginal. If you rerank, Recall@10 is the metric that matters: a reranker can reorder what you retrieved but cannot recover what you missed.

Ahead of every Nepali-specific model. +5.0 nDCG over the best of them. Note that those models' training corpora are undisclosed; since Nepali Wikipedia is the dominant Nepali corpus, they may have trained on the articles this benchmark holds out. The comparison is, if anything, tilted against this model.

Retrieval training beats language training. Untuned multilingual-e5-base, with no Nepali adaptation whatsoever, already matches the best Nepali-specific model. The two sentence_similarity_nepali models score near zero — not because they are bad, but because CosineSimilarityLoss on STS data teaches symmetric similarity, not asymmetric retrieval. Wrong task, not wrong model.

Corpus size changes the answer

The same models on a 4,092-passage version of this benchmark:

model	4k corpus	50k corpus
nepali-e5-base-v1 (Devanagari-only variant)	0.8624	0.6361
multilingual-e5-large	0.8356	0.6590
bge-m3	0.8162	0.6287
multilingual-e5-base	0.8021	0.5993

Every score drops ~20 points and the ranking inverts. A small evaluation corpus flatters every model and can reverse conclusions. Always report corpus size alongside a retrieval score, and treat even 50k as optimistic relative to a production corpus of millions.

Training

Base model	intfloat/multilingual-e5-base (XLM-R base, 278M, 768-dim)
Objective	CachedMultipleNegativesRankingLoss
Training pairs	218,180
Epochs	1 (852 steps)
Batch size	256 effective, mini-batch 4 (gradient cache)
Learning rate	1e-5, cosine schedule, 10% warmup
Optimizer	Adafactor
Precision	bf16
max_seq_length	256
Hardware	1× RTX 4050 Laptop (6 GB), ~2.5 hours

query: and passage: prefixes were applied to anchors and positives during training, so they are mandatory at inference.

The 6 GB VRAM budget drove three choices: gradient-cached MNRL (so a 256-pair batch of in-batch negatives fits with only 4 sentences resident), Adafactor instead of AdamW (whose two fp32 moments alone need ~2.2 GB for 278M parameters), and bf16 autocast.

Data

pair type	count	source
First sentence → rest of paragraph	33,126	Nepali Wikipedia
Article title → lead paragraph	21,080	Nepali Wikipedia
Lead paragraph → later paragraph	5,625	Nepali Wikipedia
Romanized anchor → Devanagari passage	53,677	transliterated from the above
English ↔ Nepali, both directions	104,808	OPUS-100 en-ne
Total	218,180

Romanization was applied to anchors only, never to passages — users type queries in Latin script while documents stay in Devanagari — and only to anchors under 200 characters, since nobody types a paragraph into a search box.

21,080 Nepali Wikipedia articles contributed, out of 32,885 in the 20231101.ne dump — 10,124 were stubs with no paragraph over 120 characters and 791 were list or meta pages. OPUS-100 required ~406,000 streamed rows to yield 52,404 usable unique pairs: 264,094 failed a length-ratio and script filter, and roughly 90,000 more were exact duplicates.

No human labelling. MultipleNegativesRankingLoss needs only positive pairs, with negatives taken from the batch.

Limitations

Wikipedia domain. Training and evaluation are both Nepali Wikipedia — same register, same length, same style. Expect lower numbers on a different domain. Build a small evaluation set from your own documents before trusting these figures in production.

92 evaluation queries. Differences under about 4 nDCG points are not measurable. The confidence interval on any single score here is roughly ±9 points. More queries, not more training data, is the fix.

Romanized support is partial and measured against a synthetic transliteration. See the romanized benchmark section above. Treat 0.2983 as a ceiling.

Query shape mismatch. Training anchors are titles and opening sentences; real queries are questions. Transfer works, but templating declaratives into questions would likely close some of the Recall@1 gap to e5-large.

~4% of Nepali training passages are list-shaped Wikipedia stubs — bot-generated place-name lists rather than prose. Measured and left in, because the one-line fix would also change the evaluation set and break comparability with every number above.

Hard negatives did not help. A variant trained on mined hard-negative triplets scored statistically identically, trading Recall@1 gains for Recall@10 losses at no net benefit. Not published.

Only ~60k of the Devanagari pairs are Nepali-to-Nepali. The majority of the non-romanized signal is cross-lingual alignment, capped by Nepali Wikipedia having only 21k usable articles.

Reproducing

Write-up, full benchmark tables, and method: https://github.com/Ganu000/nepali-embeddings

The whole pipeline runs on a single 6 GB laptop GPU in about three hours including data preparation. The evaluation set is 113 hand-written Nepali questions mapped to Wikipedia articles, of which 92 resolve against the 20231101.ne dump.

Licensing and attribution

• Base model: intfloat/multilingual-e5-base, MIT.
• Nepali Wikipedia text: CC BY-SA 4.0. Attribution to Wikipedia contributors.
• OPUS-100: see the dataset card for its per-corpus licensing.

Weights are released under Apache 2.0. Whether CC BY-SA training data imposes downstream obligations on model weights is legally unsettled; confirm for your own use case.

Citation

@misc{nepali-e5-base-v1,
  title  = {nepali-e5-base-v1: a retrieval embedding model for Nepali},
  author = {Ganesh Sah},
  year   = {2026},
  url    = {https://huggingface.co/Ganu000/nepali-e5-base-v1}
}