Shiksha: A Technical Domain focused Translation Dataset and Model for
  Indian Languages
Authors: name: Advait Joglekar; name: Srinivasan Umesh
Published: 2024-12-12 07:40:55

Abstract: Neural Machine Translation (NMT) models are typically trained on datasets
with limited exposure to Scientific, Technical and Educational domains.
Translation models thus, in general, struggle with tasks that involve
scientific understanding or technical jargon. Their performance is found to be
even worse for low-resource Indian languages. Finding a translation dataset
that tends to these domains in particular, poses a difficult challenge. In this
paper, we address this by creating a multilingual parallel corpus containing
more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
translation pairs across 8 Indian languages. We achieve this by bitext mining
human-translated transcriptions of NPTEL video lectures. We also finetune and
evaluate NMT models using this corpus and surpass all other publicly available
models at in-domain tasks. We also demonstrate the potential for generalizing
to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
average for these Indian languages on the Flores+ benchmark. We are pleased to
release our model and dataset via this link: https://huggingface.co/SPRINGLab.