Singla0009/all-indic-transliteration

All Indic Languages Transliteration

Convert between English and 21 Indian languages instantly. Type namaste → get नमस्ते, or reverse it. No PyTorch. No Fairseq. Just one pip install and you're running.

This is a production-ready, optimized version of AI4Bharat's IndicXlit — converted to CTranslate2 for maximum speed and minimum hassle.

---

Why use this instead of the original?

The official IndicXlit repo requires installing PyTorch, Fairseq, and multiple dependencies that frequently break on modern Python versions. We solved all of that.

	Official IndicXlit	This Version
Dependencies	PyTorch + Fairseq + OmegaConf + Hydra	Just ctranslate2 (one package)
Install size	~2 GB+	~90 MB total
Python support	Python 3.8–3.10 only (Fairseq crashes on 3.11+)	Python 3.8–3.13 ✅
Speed	~5–15ms per word	~1ms per word
GPU required?	No (but slow on CPU)	No (blazing fast on CPU)
Fairseq bugs	Mutable dataclass errors, OmegaConf crashes	All fixed and pre-converted
Ready to use in	Python only	Python, Rust, C++, Node.js, Go

---

What's Included

Two ready-to-use CTranslate2 models (FP32 precision):

Folder	Direction	Example	Size
indicxlit_ct2_fp32/	English → Indic	namaste → नमस्ते	~45 MB
indicxlit_indic_en_ct2_fp32/	Indic → English	नमस्ते → namaste	~44 MB

Each folder contains:

model.bin              ← The neural network weights
config.json            ← CTranslate2 engine configuration  
source_vocabulary.json ← Input character vocabulary
target_vocabulary.json ← Output character vocabulary

---

Installation

pip install ctranslate2 huggingface_hub

That's it. No PyTorch, no Fairseq, no CUDA required.

---

Usage

English → Hindi (Forward)

Type in English letters, get back native Hindi script:

import ctranslate2
from huggingface_hub import snapshot_download
import os

# Download the model (only happens once, cached after that)
model_path = snapshot_download(
    repo_id="Singla0009/all-indic-transliteration",
    allow_patterns="indicxlit_ct2_fp32/*"
)
model_dir = os.path.join(model_path, "indicxlit_ct2_fp32")

# Load the engine
translator = ctranslate2.Translator(model_dir, device="cpu")

# Transliterate (Output: नमस्ते)
word = "namaste"
source_tokens = ["__hi__"] + list(word)
results = translator.translate_batch([source_tokens], beam_size=4)
output = "".join(results[0].hypotheses[0])

print(f"{word} → {output}")

Hindi → English (Reverse)

Take native Devanagari script, get back romanized English:

import ctranslate2
from huggingface_hub import snapshot_download
import os

# Download the reverse model
model_path = snapshot_download(
    repo_id="Singla0009/all-indic-transliteration",
    allow_patterns="indicxlit_indic_en_ct2_fp32/*"
)
model_dir = os.path.join(model_path, "indicxlit_indic_en_ct2_fp32")

# Load the engine
translator = ctranslate2.Translator(model_dir, device="cpu")

# Transliterate (Output: namaste)
word = "नमस्ते"
source_tokens = ["__hi__"] + list(word)
results = translator.translate_batch([source_tokens], beam_size=4)
output = "".join(results[0].hypotheses[0])

print(f"{word} → {output}")

Batch Processing (Multiple Words)

For real-world use, process many words at once for maximum throughput:

words = ["namaste", "dhanyavaad", "bharat", "cricket", "youtube"]
batch = [["__hi__"] + list(w) for w in words]

results = translator.translate_batch(batch, beam_size=4)

for word, result in zip(words, results):
    output = "".join(result.hypotheses[0])
    print(f"{word} → {output}")

Switching Languages

Just change the language prefix. That's it — same model, same code:

# Tamil
source_tokens = ["__ta__"] + list("vanakkam")

# Bengali  
source_tokens = ["__bn__"] + list("namaskar")

# Punjabi
source_tokens = ["__pa__"] + list("sat sri akal")

---

Supported Languages (21)

Code	Language	Script	Code	Language	Script
__hi__	Hindi	देवनागरी	__ta__	Tamil	தமிழ்
__bn__	Bengali	বাংলা	__te__	Telugu	తెలుగు
__pa__	Punjabi	ਗੁਰਮੁਖੀ	__kn__	Kannada	ಕನ್ನಡ
__gu__	Gujarati	ગુજરાતી	__ml__	Malayalam	മലയാളം
__mr__	Marathi	मराठी	__or__	Odia	ଓଡ଼ିଆ
__ur__	Urdu	اردو	__as__	Assamese	অসমীয়া
__ne__	Nepali	नेपाली	__sd__	Sindhi	سنڌي
__si__	Sinhala	සිංහල	__sa__	Sanskrit	संस्कृतम्
__ks__	Kashmiri	कॉशुर	__mai__	Maithili	मैथिली
__mni__	Manipuri	মণিপুরী	__gom__	Konkani	कोंकणी
__brx__	Bodo	बड़ो

---

Technical Details

• Original Model: AI4Bharat IndicXlit v1.0 (Transformer, 6 encoder + 6 decoder layers, ~11M parameters)
• Conversion Format: CTranslate2 FP32 (lossless — identical output to the original)
• Architecture: Sequence-to-sequence Transformer with character-level tokenization
• Inference Engine: CTranslate2 v4.x (optimized C++ with CPU vectorization)
• Beam Search: Default beam_size=4 for best accuracy (reduce to 1 for maximum speed)

Bugs We Fixed During Conversion

Converting the original Fairseq checkpoints to CTranslate2 required patching several compatibility issues:

1. Python 3.12 dataclass errors — Fairseq uses mutable default arguments in dataclasses, which Python 3.12+ rejects. We patched configs.py and transformer_config.py.
2. OmegaConf/Hydra initialization crash — We disabled the hydra_init() call in Fairseq's __init__.py that causes validation errors during conversion.
3. PyTorch 2.6+ weight loading — Added weights_only=False to torch.load() calls since modern PyTorch blocks legacy pickle loading by default.
4. Vocabulary alignment — Fixed the lang_list.txt to correctly match all 21 language tokens present in the model weights.

---

Use Cases

• Subtitle Romanization — Convert Hindi/Tamil/Bengali subtitles into readable English characters for Hinglish audiences
• Search & Indexing — Enable multilingual search across Indic content by normalizing to a common script
• Input Method Engines — Power transliteration keyboards and text input tools
• Content Localization — Automatically adapt content between scripts for different regional audiences
• NLP Pipelines — Pre/post-processing step for machine translation, ASR output cleanup, and text normalization

---

Credits & License

This is a converted and optimized redistribution of the AI4Bharat IndicXlit model, originally trained on the Aksharantar dataset containing 26 million transliteration pairs.

Released under the MIT License. Original model by the AI4Bharat team at IIT Madras.

If you find this useful, please consider starring the repository! ⭐