mitra-qwen35-embedder

A 9B dense retrieval embedder for classical Buddhist literature, built on Qwen3.5-9B with large-scale domain continued-pretraining, SFT, and contrastive finetuning on cross-lingual parallel data (Sanskrit, Tibetan, Buddhist Chinese, Pāli, English), including hard-negative pairs mined from deep-research usage.

It is the embedding model behind the Dharmamitra search stack. Its defining capability is asymmetric cross-lingual retrieval: matching a single sentence in one classical language to the passage window containing its parallel in another — a task on which strong general-purpose embedders (BGE-M3, Qwen3-Embedding) score near zero.

Prompting template

Queries are wrapped in an instruction template using the two special tokens <instruct> and <query> (added to the tokenizer vocabulary):

<instruct>{instruction}\n<query>{query text}

Passages/documents are embedded as raw text — no template.

The instruction used at training time:

Please find the semantically most similar text.

optionally with a target-language hint, which improves cross-lingual precision:

Please find the semantically most similar text in Tibetan.
Please find the semantically most similar text in Chinese.

Three conventions that materially affect quality:

1. EOS is required on every input (query and passage). The bundled tokenizer has add_eos_token: true, so AutoTokenizer handles this automatically — do not disable it.
2. Left padding + last-token pooling. The embedding is the final hidden state at the last (EOS) position, L2-normalized.
3. Script conventions: Tibetan works in either native Tibetan script or Wylie transliteration — we measure identical retrieval quality for both (the pretraining corpus is native script; the contrastive stage used Wylie). Sanskrit and Pāli should be IAST romanization; Chinese in Chinese script.

Usage

import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("buddhist-nlp/mitra-qwen35-embedder")
tok.padding_side = "left"
model = AutoModel.from_pretrained(
    "buddhist-nlp/mitra-qwen35-embedder", dtype=torch.bfloat16, device_map="cuda"
).eval()

INSTR = "Please find the semantically most similar text."

def embed(texts, is_query=False, batch_size=16):
    if is_query:
        texts = [f"<instruct>{INSTR}\n<query>{t}" for t in texts]
    out = []
    with torch.no_grad():
        for i in range(0, len(texts), batch_size):
            enc = tok(texts[i:i+batch_size], padding=True, truncation=True,
                      max_length=512, return_tensors="pt").to(model.device)
            h = model(**enc).last_hidden_state[:, -1]          # last-token pooling
            out.append(torch.nn.functional.normalize(h.float(), dim=-1).cpu())
    return torch.cat(out)

q = embed(["evaṃ mayā śrutam ekasmin samaye"], is_query=True)
p = embed(["'di skad bdag gis thos pa dus gcig na", "如是我聞一時"])
print(q @ p.T)   # cosine similarities

Evaluation

R@1 on the mitra retrieval benchmark, hardened setting: 25,000 in-language distractor passages added to every candidate pool (this model's own measurements; Tibetan given in native script):

task	this model	gemma-2-mitra-e	BGE-M3
Multilingual parallels (sa/bo/zh)	0.846	0.841	0.001
Multilingual asymmetric (sentence→window)	0.788	0.581	0.000
English → classical	0.876	0.879	0.068
Verse → commentary	0.483	0.384	0.138
English question → passage	0.523	0.473	0.020

This checkpoint's own results on gold pools (per-subtask pools without added distractors):

task	this model	gemma-2-mitra-e	BGE-M3
Multilingual parallels (sa/bo/zh)	0.861	0.850	0.007
Multilingual asymmetric (sentence→window)	0.813	0.654	0.002
English → classical	0.885	0.886	0.204
Verse → commentary	0.506	0.413	0.298
English question → passage	0.574	0.534	0.077

This checkpoint additionally leads the sibling on question→passage retrieval (its distinguishing strength from the deep-research continuation training).

Model details

• Base: Qwen3.5-9B after ~30B tokens of continued pretraining on classical Buddhist corpora and a domain SFT stage
• Contrastive finetuning: LoRA (r=32), InfoNCE with cross-device negatives, temperature 0.02, group size 8, 512-token inputs, followed by a short continuation on hard-negative pairs mined from deep-research citation logs
• Embedding dimension: 4096
• Languages: Sanskrit, Tibetan (Wylie), Buddhist Chinese, Pāli, English

Citation

If you use this model, please cite the Dharmamitra project (https://dharmamitra.org). A technical report is in preparation.