darmm-embed-kazakh-v2

General-purpose Kazakh retrieval embedding model — semantic search, RAG, deduplication, and clustering for Kazakh text, with strong cross-lingual alignment to Russian and English (a Kazakh query finds Russian/English documents by meaning). Supersedes darmm-embedding-multilingual.

Fine-tuned from BAAI/bge-m3 (568M, 1024-dim, up to 8k tokens) on 373k Kazakh pairs from Darmm/darmm-kk-retrieval-pairs: Kazakh Wikipedia structural pairs (title↔lead, section, adjacent paragraphs), kk↔ru (WikiMatrix) and kk↔en (NLLB) parallel sentences, and Kazakh instruction QA pairs.

Evaluation

Held-out sets from the pairs dataset (never trained on). Retrieval: 6,322 article titles as queries against 6,322 lead paragraphs. Cross-lingual: match a Kazakh sentence to its translation in a 2,000-candidate pool.

metric	this model	BAAI/bge-m3	darmm-embedding-multilingual (v1)
kk retrieval Recall@1	0.716	0.666	0.662
kk retrieval Recall@5	0.754	0.714	0.705
kk retrieval Recall@10	0.775	0.730	0.721
kk retrieval MRR@10	0.733	0.687	0.681
kk→ru Accuracy@1	0.668	0.518	0.546
kk→en Accuracy@1	0.918	0.851	0.855

Reproducible with eval_embed.py in this repo.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Darmm/darmm-embed-kazakh-v2")

query = "Астанада ауа райы қандай?"
docs = [
    "Елордада бүгін күн ашық, +25 градус болады.",
    "Погода в столице сегодня солнечная.",
    "Доллар бағамы тағы өзгерді.",
]
q_emb = model.encode([query], normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)
print((q_emb @ d_emb.T))  # cosine similarities

Training

MultipleNegativesRankingLoss (in-batch negatives, no-duplicates sampler), 1 epoch over 373k pairs, batch 32, max_seq 192, lr 2e-5 cosine with 5% warmup, bf16, single A100. Final train loss 0.137.

Limitations

• Trained with 192-token pairs; quality on much longer passages is inherited from bge-m3 rather than tuned.
• kk↔ru pairs come from mined alignments (WikiMatrix) and carry some noise.
• Domain-specific jargon (legal, medical) is underrepresented in Wikipedia-based training data.

Built by Darmm · 2026-09 · Apache 2.0