qwen-indic-v1

A multilingual text embedding model for the 22 scheduled Indic languages, fine-tuned from Qwen/Qwen3-Embedding-8B via a 3-stage LoRA training recipe with frozen-teacher preservation.

• Base model: Qwen/Qwen3-Embedding-8B
• Training: LoRA (r=64, alpha=128), merged into base for release
• Pooling: last-token, L2-normalized
• Embedding dimension: 4096
• Languages: 22 scheduled Indic languages
• License: Apache 2.0 (inherited from base model)

Evaluation

Evaluated on MTEB(Indic, v1) using the official mteb package. Overall task-averaged mean across 20 Indic tasks: 73.80.

Per-category results

Category	# Tasks	Mean
Retrieval	2	94.89
Reranking	1	86.76
PairClassification	1	81.24
BitextMining	2	77.62
Classification	12	70.59
STS	1	61.68
Clustering	1	54.10
Overall (task avg)	20	73.80

Per-task results

Task	Category	Score
XQuADRetrieval	Retrieval	96.49
NepaliNewsClassification	Classification	95.32
BelebeleRetrieval	Retrieval	93.29
MalayalamNewsClassification	Classification	89.98
IN22GenBitextMining	BitextMining	88.46
BengaliSentimentAnalysis	Classification	87.30
WikipediaRerankingMultilingual	Reranking	86.76
GujaratiNewsClassification	Classification	84.58
XNLI	PairClassification	81.24
PunjabiNewsClassification	Classification	80.83
MTOPIntentClassification	Classification	78.09
SentimentAnalysisHindi	Classification	67.70
SanskritShlokasClassification	Classification	67.08
IN22ConvBitextMining	BitextMining	66.78
MultiHateClassification	Classification	63.03
IndicCrosslingualSTS	STS	61.68
SIB200ClusteringS2S	Clustering	54.10
UrduRomanSentimentClassification	Classification	50.89
HindiDiscourseClassification	Classification	42.58
TweetSentimentClassification	Classification	39.73

Full per-task JSON results are in the MTEB results repository.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("LingoIITGN/qwen-indic-v1")

queries = [
    "भारत की राजधानी क्या है?",
    "চা কীভাবে বানানো হয়?",
]
documents = [
    "नई दिल्ली भारत की राजधानी है।",
    "গরম জলে চা পাতা ভিজিয়ে চা তৈরি করা হয়।",
]

query_embs = model.encode(queries, prompt_name="Retrieval")
doc_embs = model.encode(documents)

import numpy as np
sim = (query_embs / np.linalg.norm(query_embs, axis=1, keepdims=True)) @ \
      (doc_embs / np.linalg.norm(doc_embs, axis=1, keepdims=True)).T

For classification and STS, use prompt_name="Classification" or "STS" respectively; the model was trained with per-task-family instruction prefixes. See config_sentence_transformers.json for the full prompt list.

Training procedure

Fine-tuned via a 3-stage LoRA recipe with frozen full-base teacher preservation, adapted from the Harrier-OSS training methodology.

Each stage trains the LoRA student while a frozen copy of the base model serves as a preservation teacher via a relational distillation loss. The recipe uses a gentle contrastive weight and a dominant, rising teacher-preservation weight to prevent catastrophic forgetting of the base model's strong general-purpose capabilities while adding Indic-specific improvements.

Parameter	Stage 1	Stage 2	Stage 3
Data type	Retrieval + bitext	Classification + NLI + intent	Clustering + retrieval
Instruction strategy	none	hard	conditional
Learning rate	2e-5	5e-6	5e-7
Contrastive weight	0.1	0.2	0.3
Hard-negative weight	0.05	0.1	0.1
Teacher-relational weight	1.5	3.0	5.0
Flow weight	0.001	0.003	0.001
Epochs	1	1	1

Effective batch size: ~256 (micro=16, grad-accum=16, DDP across GPUs). Precision: bf16 mixed precision throughout.

Training data

All training data was drawn from train splits only — no test or validation splits from any dataset were used, to avoid contamination of MTEB evaluation tasks. Sources include:

• Retrieval/bitext (Stage 1): Samanantar (ai4bharat), plus curated retrieval pairs with mined hard negatives.
• Classification/NLI (Stage 2): IndicXNLI (entailment as positive, contradiction as hard negative), MASSIVE intent classification, sentiment data, and language-identification triplets constructed from parallel bitext with same-script hard negatives.
• Clustering (Stage 3): IndicGLUE news-genre topic clusters and additional clustering-format data.

All sources were audited to ensure use of training splits only. Data was sanitized to remove control characters and validated for JSONL round-tripping before training.

Model architecture

• Base: Qwen3-Embedding-8B (decoder-only transformer, 4096 hidden dim)
• Pooling: last-token (matches Qwen3 base convention)
• Normalization: L2, applied on the pooled vector
• LoRA targets: all-linear (auto-detected linear projections)
• LoRA rank: 64, alpha: 128, dropout: 0.05
• Merged: yes — the LoRA adapter is merged into the base weights for release

Instruction format

The model uses Qwen3's instruction convention on the query side:

Instruct: {task_description}\nQuery: {query_text}

Documents are encoded without an instruction prefix. Task-family prompt strings are stored in config_sentence_transformers.json and applied automatically by sentence-transformers when prompt_name is passed to encode().

Limitations

• The model inherits Qwen3-Embedding-8B's overall capabilities and biases; Indic-specific fine-tuning does not eliminate base-model behaviors.
• Coverage of very-low-resource Indic languages (Kashmiri, Manipuri, Bodo, Santali) is limited by training data availability; scores on these languages may be substantially lower than on higher-resource languages.
• Clustering performance, while improved over the base model, remains below the current state-of-the-art on MTEB(Indic). Future work will address this.
• 8B parameters — inference requires ~16GB VRAM in bf16.