Tamil Tokenizer (BPE)
A high-performance Byte Pair Encoding (BPE) tokenizer for Tamil text, trained on Tamil Wikipedia.
Model Details
- Language: Tamil (
ta) - Tokenizer type: Byte Pair Encoding (BPE)
- Vocab size: 32,015 tokens
- Training data: Tamil Wikipedia (5,000 articles, 128K lines, 38 MB)
- Training merges: 29,999
- Special tokens: 108 (BOS, EOS, PAD, UNK, MASK, question, answer, code, 100 reserved)
Files
| File | Description |
|---|---|
tamil_tokenizer_wiki.json |
Production tokenizer — 32K vocab trained on Tamil Wikipedia |
tamil_tokenizer_large.json |
Smaller tokenizer — 3.1K vocab trained on a synthetic corpus |
tamil_wiki_corpus.txt |
Training data (38 MB of extracted Tamil Wikipedia text) |
Usage
Python (Hugging Face Tokenizers)
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tamil_tokenizer_wiki.json")
encoded = tokenizer.encode("வணக்கம்! இது தமிழ் tokenizer.")
print(encoded.ids)
# [245, 1023, 567, ...]
Rust
use tamil_tokenizer::FastBPEEncoder;
use rustc_hash::FxHashMap;
use std::fs;
let data = fs::read_to_string("tamil_tokenizer_wiki.json").unwrap();
let vocab: FxHashMap<String, u32> = serde_json::from_str(&data).unwrap();
let encoder = FastBPEEncoder::new(vocab);
let token_ids = encoder.encode("வணக்கம்! இது தமிழ் tokenizer.");
println!("{:?}", token_ids);
CLI
# Encode text to token IDs
cargo run --release --bin encode -- tamil_tokenizer_wiki.json "வணக்கம்! இது தமிழ்"
# Decode token IDs back to text
cargo run --release --bin decode -- tamil_tokenizer_wiki.json 245 1023 567
Vocabulary Layout
| Token Range | Count | Description |
|---|---|---|
| 0 – 246 | 247 | Tamil character clusters (consonant+vowel sign) |
| 247 – 503 | 257 | Byte-level tokens (GPT-2 compatible mappings) |
| 504 – 31,906 | ~31K | BPE merge tokens (trained on Tamil Wikipedia) |
| 31,907 – 32,014 | 108 | Special tokens (BOS, EOS, PAD, UNK, MASK, reserved) |
Special Tokens
| Token | ID | Description |
|---|---|---|
<|startoftext|> |
31,907 | Beginning of sequence |
<|endoftext|> |
31,908 | End of sequence |
<|pad|> |
31,909 | Padding token |
<|unk|> |
31,910 | Unknown token |
<|mask|> |
31,911 | Mask token (MLM) |
<|question|> |
31,912 | Question marker |
<|answer|> |
31,913 | Answer marker |
<|code|> |
31,914 | Code block marker |
<|reserved_0|> – <|reserved_99|> |
31,915 – 32,014 | 100 reserved tokens |
Training Details
- Algorithm: Byte Pair Encoding (BPE)
- Pretokenizer: Custom regex splitting for Tamil Unicode (consonants, vowel signs, matras)
- Encoder: Aho-Corasick automaton for O(n) multi-pattern matching
- Parallelism: rayon-based parallel corpus processing during training
Performance
Benchmarked against Python tiktoken on Tamil text:
| Metric | This Tokenizer | tiktoken (cl100k_base) | Speedup |
|---|---|---|---|
| Encoder build | 10.83 ms | 440 ms | 41× |
| Encode throughput | 10.95 MB/s | 1.43 MB/s | 7.7× |
| Compression ratio | 2.57 bytes/token | 2.35 bytes/token | — |
Citation
If you use this tokenizer in your research, please cite:
@software{tamil_tokenizer,
title = {Tamil BPE Tokenizer},
year = {2026},
note = {Trained on Tamil Wikipedia},
}
License
MIT