Vagabond404/khmer-tokenizer-sota

🇰🇭 Khmer Tokenizer SOTA v2 (K-Token v2)

State-of-the-Art Khmer Word Segmenter & LLM Sub-word Tokenizer designed to eliminate the "Khmer Tokenizer Tax" and solve broken Unicode rendering.

🌟 Key Highlights in v2

• ⚡ Token Fertility: 0.88 tokens/word (vs 6.20 for Llama 3, ~85.9% reduction in LLM cost and latency).
• 🛡️ Native Unicode Metaspace: 100% clean UTF-8 tokens, zero byte-level Latin character mangling.
• 📚 Official RAC 2022 Dictionary Seeding: 38,700+ validated headwords from the Royal Academy of Cambodia integrated as native tokens.
• 🚀 Rust-backed Hugging Face Speed: Instant drop-in replacement with AutoTokenizer.from_pretrained("Vagabond404/khmer-tokenizer-sota").

📊 Benchmark Comparisons

Model / Tokenizer	Token Fertility (Tokens/Word)	Khmer KCC Safe	Speed (words/sec)	Context Window Utilization
K-Token SOTA v2 (Ours)	0.88	100% (Strict)	> 280,000	4.9x more context
K-Token v1	2.52	100% (Strict)	> 250,000	2.5x more context
Llama 3 Tokenizer	6.20	❌ High risk (Byte split)	Medium	1.0x (Baseline)
Mistral NeMo / v0.3	5.85	❌ Byte fallback	Medium	1.1x
Gemma 2	5.40	❌ Byte fallback	Medium	1.2x

🚀 Quick Start (Python & Hugging Face Transformers)

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Vagabond404/khmer-tokenizer-sota")
text = "ខ្ញុំស្រឡាញ់ប្រទេសកម្ពុជា និងបច្ចេកវិទ្យាបញ្ញាសិប្បនិម្មិត"

tokens = tokenizer.tokenize(text)
print("Tokens:", tokens)
print("Token count:", len(tokens))

🛠️ Standalone Word Segmentation

from khmer_segmenter import KhmerSegmenter

segmenter = KhmerSegmenter(dict_path="rac_dictionary.json")
result = segmenter.segment("សួស្តីពិភពលោក")
print(" | ".join(result))
# Output: សួស្តី | ពិភពលោក

📜 Citation & Credits

• Royal Academy of Cambodia (RAC) Dictionary 2022.
• Asian Language Treebank (ALT) Project.
• Developed by Ange (@Vagabond404).