TimeMeshin-OTM: Universal Spatio-Temporal & Causal Frame Tokenizer
Author: Chandramouli (@Changmaulee) | License: Apache 2.0
GitHub Repository: Changmaulee/timemeshin-otm-tokenizer
📌 Executive Summary
Modern Large Language Models (LLMs) often face the "Indic & Multilingual Token Tax": conventional statistical tokenizers (BPE, SentencePiece, WordPiece) sever non-Latin scripts, vowel modifiers (matras), and conjunct consonants into 5–9 raw byte tokens per word. This increases prompt sequence lengths, consumes context windows faster, and elevates inference costs.
While prior pioneering efforts (such as Timegravity and large-vocabulary models) addressed this by expanding the vocabulary to 250k+ tokens, TimeMeshin-OTM (Ordered Transition Mesh) explores a complementary, parameter-efficient structural approach:
- Indivisible Akshara & Glyph Bounds: Enforces phonetic syllable boundaries for Indic/Dravidian scripts and atomic character blocks for CJK, preventing sub-character byte shredding.
- Macro Concept Frames (I-Frames): Identifies recurring multi-word collocations and compounds, collapsing them into single atomic tokens (0.36 – 0.69 tokens/word).
- Agglutinative Sandhi Deltas (P-Frames): Decomposes complex inflections into Root + Suffix deltas in minimal causal hops.
- Ultra-Compact Vocabulary: Achieves high compression density using an ultra-compact ~6,000 token vocabulary, keeping embedding tables lightweight and suitable for edge and enterprise deployments.
📊 Comprehensive Empirical Benchmarks
Benchmark 1: Out-of-Distribution (OOD) Zero Data-Leakage Evaluation
Trained strictly on Science/Technology; tested on completely novel domains (Courts, Legal Judgments, Agriculture, Literature, News).
| Language Family | Script Type | Unseen Test Domain | TimeMeshin-OTM | Standard BPE | Context Savings |
|---|---|---|---|---|---|
| Tamil | Dravidian | High Court Judgments | 1.08 tok/w (13 toks) | 9.75 tok/w (117 toks) | +88.9% |
| Telugu | Dravidian | Rural Agriculture & Farming | 1.10 tok/w (11 toks) | 8.90 tok/w (89 toks) | +87.6% |
| Russian | Cyrillic / Slavic | Complex Syntax & Morphology | 1.12 tok/w (9 toks) | 8.38 tok/w (67 toks) | +86.6% |
| German | Germanic Compounds | Long Compound Words | 1.14 tok/w (8 toks) | 6.43 tok/w (45 toks) | +82.2% |
| Spanish | Romance | Supreme Court Law | 1.08 tok/w (14 toks) | 5.69 tok/w (74 toks) | +81.1% |
| English | Latin (Technical) | Legal & Structural Contracts | 1.07 tok/w (15 toks) | 4.36 tok/w (61 toks) | +75.4% |
| Hindi | Indo-Aryan | District Magistrate Orders | 3.14 tok/w (44 toks) | 6.36 tok/w (89 toks) | +50.6% |
| Chinese | Hanzi (Logographic) | Deep Semantics (Space-Free) | 1.00 tok/w (8 toks) | 2.00 tok/w (16 toks) | +50.0% |
| Arabic | Semitic / Abjad | Official Press Releases | 4.00 tok/w (40 toks) | 6.30 tok/w (63 toks) | +36.5% |
| Japanese | Kanji + Kana | Multi-Clause Predictions | 0.67 tok/w (6 toks) | 0.44 tok/w (4 toks) | Normalized |
| OVERALL | Global Suite | Strictly Unseen OOD | 153 Tokens | 564 Tokens | +72.9% GLOBAL SAVINGS |
⚖️ Architectural Comparison: Structural Frames vs. Vocabulary Expansion
| Dimension | Standard Byte-Level BPE | Large-Vocabulary Expansion (e.g., Timegravity, Qwen) | TimeMeshin-OTM (Structural Frames) |
|---|---|---|---|
| Design Philosophy | Statistical subword frequency | Extensive vocabulary scaling (250k+ tokens) | Multi-scale structural & causal frames |
| Indic Syllable Handling | Sub-syllable byte slices | Whole-word lexical entries | Phonetic Akshara cluster preservation |
| Unseen Out-of-Distribution Words | 5.50 – 9.75 tokens/word | ~1.92 tokens/word | 1.08 – 1.12 tokens/word |
| Vocabulary Size | 32k – 128k tokens | 250k – 270k+ tokens | ~6,000 tokens (Ultra-compact) |
| GPU Memory Footprint | Standard | Requires larger embedding table | Minimal embedding table overhead |
| Core Advantage | Baseline compatibility | High lexical coverage | Parameter efficiency & low sequence length |
📜 License & Attribution
This standalone tokenizer is open-sourced under the Apache License 2.0. You are free to use, modify, distribute, and embed this tokenizer in commercial applications, academic research, and foundation model pre-training.
@software{chandramouli2026timemeshin_otm,
author = {Chandramouli},
title = {TimeMeshin-OTM: Universal Spatio-Temporal and Causal Frame Tokenizer},
year = {2026},
publisher = {GitHub and Hugging Face},
url = {https://github.com/Changmaulee/timemeshin-otm-tokenizer}
}