ShrutamMT-v1 🎧

"श्रुतम्" (Shrutam) = "that which is heard" — a multilingual 11 Indic languages → English neural machine translation model, built as a Transformer trained completely from scratch (no pretrained weights, no fine-tuning shortcuts) on the AI4Bharat Samanantar parallel corpus.

Supports: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu → English, all in a single shared model.

This is a from-first-principles implementation of the original "Attention Is All You Need" (Vaswani et al., 2017) transformer architecture — encoder, decoder, multi-head attention, positional encodings, label smoothing, Noam learning-rate schedule, beam search — all hand-written in PyTorch, scaled up and extended for multilingual training.

Model Details

Architecture	Transformer (encoder-decoder), tied embeddings
Parameters	148.5M
Layers	6 encoder + 6 decoder
d_model	768
Attention heads	12
Feed-forward dim	3072
Vocabulary	64,000 (joint SentencePiece BPE, 11 Indic scripts + English)
Max sequence length	256 tokens
Training data	~49.2M sentence pairs (all 11 Samanantar Indic languages, English side as target)
Training steps	400,000
Optimizer	Adam (β1=0.9, β2=0.98, eps=1e-9)
LR schedule	Noam (warmup 8000 steps)
Label smoothing	0.1
Hardware	Up to 5× NVIDIA RTX 6000 Ada (DataParallel)

Design choices beyond the base Transformer paper

• 3-way tied embeddings (source embedding = target embedding = output projection) — fewer params, better generalization across languages sharing a joint vocabulary
• Source language-tag tokens (<hi>, <bn>, <ta>, ...) prepended to input, so the encoder knows which language it's translating from — same idea used in Google's multilingual NMT and IndicTrans
• Length-ratio filtering during data cleaning (0.3x–3.0x) to remove noisy/misaligned pairs
• Frequent checkpointing (every 500 steps) to survive GPU driver instability during the long training run
• Checkpoint averaging (last 5 checkpoints) as the final released weights

Results

Best result (final model, step 400,000)

Metric	Value
Overall BLEU	29.76
Overall chrF	52.61
Best language (BLEU)	Bengali — 40.91
Val Loss	2.83
Val Perplexity	16.91
Training steps	400,000 / 400,000
Training pairs	49,236,019

Per-language BLEU / chrF

Evaluated on 30 held-out Samanantar test sentences per language (330 total) with beam search (beam size 4):

Language	BLEU	chrF
Assamese (as)	15.70	39.51
Bengali (bn)	40.91	62.00
Gujarati (gu)	35.08	54.89
Hindi (hi)	33.29	58.24
Kannada (kn)	25.05	49.30
Malayalam (ml)	19.66	45.88
Marathi (mr)	26.30	50.68
Odia (or)	40.71	59.12
Punjabi (pa)	31.80	54.46
Tamil (ta)	26.89	51.77
Telugu (te)	35.49	56.02
Overall	29.76	52.61

Training curve (validation loss / perplexity)

Full interactive curves (train loss, val loss, val ppl, learning rate) are on the live Weights & Biases dashboard.

Step	Val Loss	Val PPL
4,000	6.51	674.97
20,000	4.25	70.02
40,000	3.74	42.09
60,000	3.60	36.57
80,000	3.50	33.27
100,000	3.32	27.80
120,000	3.18	24.04
140,000	3.08	21.73
160,000	3.03	20.71
180,000	2.99	19.91
200,000	2.97	19.43
220,000	2.94	18.95
240,000	2.92	18.58
260,000	2.90	18.25
280,000	2.89	18.01
300,000	2.88	17.73
320,000	2.87	17.60
340,000	2.85	17.31
360,000	2.85	17.21
380,000	2.83	16.98
400,000	2.83	16.91

The curve shows the classic Noam-schedule shape: fast initial drop during warmup, a long plateau through the middle of training while LR is still relatively high, then steady improvement as LR decays in the final third — val PPL nearly halved from step 20,000 (70.0) to step 400,000 (16.9).

Lower-resource languages (Assamese: 137k training pairs) score lower than higher-resource ones (Bengali: 8.5M pairs) — expected behavior, directly correlated with per-language data volume in Samanantar.

Sample translations

TA: உள்ளே செல்ல அனுமதி கிடையாது.
REF: Nobody is allowed to go inside.
HYP: Entry is not allowed.

TE: ఏం పని వుండి చేశారో ఏమో!
REF: What have you just done !
HYP: What have you done!

Repository Structure

.
├── checkpoint/
│   └── model_step400000.pt   # final trained weights (step 400,000)
├── tokenizer/
│   ├── spm_joint.model       # SentencePiece BPE model (joint, 64k vocab, 11 scripts + English)
│   └── spm_joint.vocab
└── code/
    ├── config.py              # all hyperparameters
    ├── models/                # encoder/decoder/attention/embedding — full from-scratch architecture
    ├── dataset/                # data loading + token-bucket batching + masks
    ├── prepare_data.py         # builds the multilingual train/val split from Samanantar
    ├── train_tokenizer.py      # SentencePiece training script
    ├── train.py                # training loop (AMP, checkpointing, Noam LR, wandb)
    ├── translate.py             # beam-search inference (single sentence, language-tagged)
    ├── infer_and_compare.py     # per-language BLEU/chrF evaluation
    ├── average_checkpoints.py   # checkpoint averaging utility
    └── run_train_loop.sh        # auto-restart wrapper for long unattended training

Usage

import torch
import sentencepiece as spm
import sys
sys.path.insert(0, "code")

import config
from models.transformer import Transformer
from dataset.masks import make_src_mask, make_tgt_mask

sp = spm.SentencePieceProcessor()
sp.load("tokenizer/spm_joint.model")

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Transformer(
    src_vocab_size=sp.get_piece_size(),
    tgt_vocab_size=sp.get_piece_size(),
    d_model=config.D_MODEL,
    n_heads=config.N_HEADS,
    n_encoder_layers=config.N_ENCODER_LAYERS,
    n_decoder_layers=config.N_DECODER_LAYERS,
    d_ff=config.D_FF,
    dropout=config.DROPOUT,
    max_len=config.MAX_LEN,
    tie_embeddings=config.TIE_EMBEDDINGS,
).to(device)
ckpt = torch.load("checkpoint/model_step400000.pt", map_location=device)
model.load_state_dict(ckpt["model"])
model.eval()

# translate — see code/translate.py for the full beam_search() implementation
# input must be prefixed with a language tag, e.g. "<hi> your text here"

Or run the included script directly:

python3 code/translate.py --text "यह एक परीक्षण है" --lang hi --checkpoint checkpoint/model_step400000.pt --beam_size 4

Training Data

Trained on ShrutamMT-v1-data, a cleaned and multilingual-tagged derivative of AI4Bharat Samanantar — the largest publicly available parallel corpora collection for 11 Indic languages, created by the AI4Bharat lab. All credit for the underlying parallel data collection and curation goes to AI4Bharat; this repository only adds cleaning, multilingual formatting, and language tagging on top of their original release.

Limitations

• Beam search inference in translate.py is not KV-cached / not batched across beams — correctness-oriented, not throughput-optimized.
• Trained on news/formal-register text (Samanantar is largely mined from news and web sources); informal or code-mixed text may translate less reliably.
• Direction is X→English only (not English→X, and not Indic-to-Indic directly).
• Lower-resource languages (Assamese, Odia, Punjabi) have noticeably lower quality than higher-resource ones (Bengali, Telugu, Gujarati), proportional to training data volume.

Citation

This model is entirely dependent on the Samanantar dataset. Please cite AI4Bharat's work if you use this model:

@article{ramesh2021samanantar,
  title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
  author={Ramesh, Gowtham and others},
  journal={Transactions of the Association for Computational Linguistics},
  year={2022}
}