TralaIndo-32K Tokenizer
A 32K SentencePiece BPE tokenizer for Indonesian.
Main file
Use tokenizer.model.
Special tokens
<unk>= 0<s>= 1</s>= 2<pad>= 3
Training data
Trained on Indonesian text from Wikimedia Wikipedia, AllenAI C4, Wikimedia Wikisource, and FineWeb2.
Dataset users should follow the original dataset licenses and attribution requirements.
21-06-2026: BEAT o200k_base overall! (well, good!) 22-06-2026: TralaIndo has a 0.851x (0.8511685116851169x) fertility rate, meanwhile o200k_base has a 1.175x one (1.1748554913294798x)! TralaIndo is 1.1748554913294798x more efficient than o200k_base.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support