coild-aikosh/coild-mul-mt

COILD Multilingual Translation (mul-mt)

A COILD model from IIT Patna for Indic-to-Indic machine translation, covering 40 translation directions.

Languages

code	language
asm_Beng	Assamese
ben_Beng	Bengali
brx_Deva	Bodo
doi_Deva	Dogri
gom_Deva	Konkani
guj_Gujr	Gujarati
hin_Deva	Hindi
kan_Knda	Kannada
kas_Arab	Kashmiri (Arabic)
mai_Deva	Maithili
mal_Mlym	Malayalam
mar_Deva	Marathi
mni_Mtei	Manipuri (Meetei Mayek)
npi_Deva	Nepali
ory_Orya	Odia
pan_Guru	Punjabi
sat_Olck	Santali (Ol Chiki)
snd_Deva	Sindhi (Devanagari)
tam_Taml	Tamil
tel_Telu	Telugu
urd_Arab	Urdu

Translation runs directly between these languages, in both directions, without pivoting through English.

Domains

The training data covers 8 domains:

• Agriculture
• Climate
• Education
• Governance
• Healthcare
• Judiciary
• Science & Technology
• Tourism

Translation quality is best on text of this kind; other domains are out of distribution and unmeasured.

Training

Finetuned from the base model on 40 Indic translation directions. The released weights are the best checkpoint by validation loss, selected from a 5-epoch run.

Usage

Install the dependencies (see requirements.txt in this repo):

pip install "transformers<5" sentencepiece IndicTransToolkit

transformers must be < 5. The bundled modeling code uses transformers.onnx (removed in v5) and tokenizer internals that v5 changed, so on transformers 5 the model fails at load with ModuleNotFoundError: No module named 'transformers.onnx'. pip install transformers installs v5 by default, hence the pin. sentencepiece is imported directly by the tokenizer and is not pulled in by anything else.

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from IndicTransToolkit.processor import IndicProcessor

model_id = "coild-aikosh/coild-mul-mt"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda").eval()
ip = IndicProcessor(inference=True)

src_lang, tgt_lang = "hin_Deva", "mni_Mtei"
sentences = ["गेहूँ की बुआई अधिकतर धान के बाद की जाती है।"]

batch = ip.preprocess_batch(sentences, src_lang=src_lang, tgt_lang=tgt_lang)
enc = tok(batch, truncation=True, max_length=256, padding=True, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**enc, num_beams=5, max_new_tokens=256, use_cache=True)
print(ip.postprocess_batch(tok.batch_decode(out, skip_special_tokens=True), lang=tgt_lang))
# ["ꯒꯦꯍꯨ ꯑꯁꯤ ꯑꯌꯥꯝꯕꯅ ꯆꯦꯡꯒꯤ ꯃꯇꯨꯡꯗ ꯊꯥꯏ ꯫"]

inference.py in this repo is the same thing as a command line tool:

python inference.py --src_lang hin_Deva --tgt_lang mni_Mtei --text "गेहूँ की बुआई अधिकतर धान के बाद की जाती है।"
python inference.py --src_lang asm_Beng --tgt_lang hin_Deva --file sentences.txt   # one sentence per line
python inference.py --list_languages

It runs on GPU when one is available and falls back to CPU, and works against a local checkpoint with --model ./path.

Deploy

This is a custom architecture (model_type: IndicTrans, loaded with trust_remote_code=True) that also needs IndicProcessor around generation. Serverless inference and third-party inference providers only run standard architectures, so they cannot serve it. A dedicated Inference Endpoint can: this repo ships handler.py and requirements.txt, which the endpoint picks up automatically.

Deploy → Inference Endpoints on this page, pick a GPU (the model is 2.4 GB in bf16), then call it:

import requests
r = requests.post(
    "https://<your-endpoint>.endpoints.huggingface.cloud",
    headers={"Authorization": "Bearer <hf-token>"},
    json={"inputs": "गेहूँ की बुआई अधिकतर धान के बाद की जाती है।",
          "parameters": {"src_lang": "hin_Deva", "tgt_lang": "mni_Mtei"}},
)
print(r.json())   # [{"translation_text": "ꯒꯦꯍꯨ ꯑꯁꯤ ꯑꯌꯥꯝꯕꯅ ꯆꯦꯡꯒꯤ ꯃꯇꯨꯡꯗ ꯊꯥꯏ ꯫"}]

inputs also accepts a list of up to 64 sentences. parameters takes src_lang and tgt_lang (required, from the table above) plus optional num_beams (5) and max_new_tokens (256).

Notes

• generate() works on transformers >= 4.54 out of the box. The bundled remote code assumes the legacy cache format and would otherwise crash against EncoderDecoderCache; modeling_indictrans.py here opts out of the new cache, so generation runs with use_cache=True and no speed penalty.
• Running on CPU unexpectedly? pip install torch may install a build for a newer CUDA than your driver supports; torch then falls back to CPU silently. Install the build matching your driver from pytorch.org.
• Targets were normalized by IndicProcessor(..., is_target=True) during training, so the model emits normalized text. For Arabic-script targets in particular it does not reproduce vowel diacritics present in raw corpora.

License and attribution

The finetuned weights are released under MIT by the COILD project, IIT Patna, and were produced by finetuning ai4bharat/indictrans2-indic-indic-1B (MIT) on COILD data. Please cite IndicTrans2 alongside this model.

The bundled modeling code (configuration_indictrans.py, modeling_indictrans.py, tokenization_indictrans.py) is not ours: it is redistributed from IndicTrans2 under the Apache License 2.0, Copyright 2023 The IndicTrans2 Authors and AI4Bharat team, and its notices are kept intact. modeling_indictrans.py carries one COILD modification, marked in the file: a compatibility shim so generate() works on transformers >= 4.54 (see Notes). No other bundled file is modified.