- Neuratal NTokenizer v0.3-polished
- Model details
- Tokenizer Benchmark β NTokenizer vs Leading Tokenizers
- Description
- Training and data
- Evaluation
- Compatibility
- Limitations
- License and attribution
- FAQ β honest answers to reasonable skepticism
- "Why should I trust an unknown tokenizer over OpenAI's, Meta's or Google's?"
- "Is the benchmark legit, or was it tuned to make NTokenizer win?"
- "Does higher B/T mean the language model will be better?"
- "What are you hiding? What are the known weaknesses?"
- "Why is English's gain smaller than the other languages?"
- "Is it safe / stable on weird inputs?"
- "Why is it not famous?"
- "Can I use it commercially?"
- "Should I switch from o200k_base today?"
- Links
- Model details
Neuratal NTokenizer v0.3-polished
NTokenizer is an independently developed, multilingual byte-level BPE tokenizer written from scratch in Rust by Neuratal (founder and creator: RΓΌzgar Erdem). NTokenizer is a tokenizer, not a language model.
- Repository (source, Rust + Python bindings): https://github.com/Neuratal/neuratal-tokenizer
- Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
- Organization site: none published yet β unverified look-alike domains should not be trusted.
Model details
- Name: Neuratal NTokenizer v0.3-polished
- Organization: Neuratal (independent AI research and software project; not claimed to be an incorporated company)
- Creator: RΓΌzgar Erdem
- Architecture: byte-level Byte Pair Encoding (BPE), single deterministic merge table
- Vocabulary: 262,144 (256 base byte tokens + 261,881 learned merges + 7 special tokens:
<|bos|>,<|eos|>,<|pad|>,<|unk|>,<|system|>,<|user|>,<|assistant|>) - Native format:
.ntokv3 (NTOK magic + CRC32 + payload) - Implementation language: Rust (reference encoder/decoder); Python bindings via ctypes
- Intended use: text tokenization for language-model training and inference pipelines; research tooling
- Out of scope: serving as a language model, or as a normalization/rewriter of Unicode text
Tokenizer Benchmark β NTokenizer vs Leading Tokenizers
Development/validation benchmark β not a pristine blind evaluation set. 544 files, 207,712,944 bytes, identical raw bytes, no normalization, no chat templates for any candidate. **Bold** = higher B/T (fewer tokens per input byte) than the comparator. Equal-category B/T = arithmetic mean of the 11 bucket-level B/Ts.
| Tokenizer (reference) | Vocabulary | Tokens | Overall B/T | Equal-category B/T | Token reduction vs o200k |
|---|---|---|---|---|---|
| NTokenizer v0.1 | 262,144 | 65,573,261 | 3.1676 | 3.1203 | β28.65% |
| NTokenizer v0.2 | 262,144 | 42,140,674 | 4.9290 | 4.6855 | +17.32% |
| NTokenizer v0.3-original (historical) | 262,144 | 37,397,108 | 5.5543 | 5.1057 | +26.63% |
| NTokenizer v0.3-polished (official) | 262,144 | 37,397,319 | 5.5542 | 5.1056 | +26.63% |
| NTokenizer v0.4-fail (CJK8, archived) | 262,144 | 37,418,652 | 5.5511 | 5.1031 | +26.59% |
| NTokenizer v0.4-fail (Han4, archived) | 262,144 | 37,406,269 | 5.5529 | 5.1081 | +26.61% |
| o200k_base (tiktoken) | 200,019 | 50,971,306 | 4.0751 | 4.0899 | 0.00% |
| Bloom-7b1 | 250,680 | 54,773,073 | 3.7922 | 4.1584 | +7.4% |
| Qwen2.5-7B-Instruct tokenizer | 151,665 | 62,266,982 | 3.3358 | 3.3236 | β22.16% |
| cl100k_base (tiktoken) | 100,256 | 75,061,973 | 2.7672 | 2.8566 | β47.2% |
| Mistral-7B-Instruct-v0.2 | 32,000 | 87,276,011 | 2.3800 | 2.4943 | β71.3% |
| StableLM-base-alpha-3b | 50,277 | 83,207,383 | 2.4963 | 2.6430 | β63.2% |
| Llama-2-7b (community mirror) | 32,000 | 91,853,794 | 2.2613 | 2.3985 | β80.2% |
| CodeLlama-7b | 32,016 | 91,853,791 | 2.2613 | 2.3985 | β80.2% |
| DeepSeek-LLM-7b-base | 100,015 | 93,334,847 | 2.2255 | 2.4470 | β83.1% |
| Falcon-7b | 65,024 | 103,640,810 | 2.0042 | 2.3126 | β103.3% |
| GPT-2 (r50k_base) | 50,256 | 105,229,579 | 1.9739 | 2.1406 | β106.4% |
| DeepSeek-Coder-6.7b-base | 32,022 | 116,855,033 | 1.7775 | 2.1074 | β129.3% |
| MosaicML MPT | n/a | NOT MEASURED (401/404 on the hub) | β | β | β |
| Gemini 3.8 Flash | n/a | NOT MEASURED (no API credential) | β | β | β |
Competitor landscape (properties from public model cards; only the B/T column measured here):
| Tokenizer | Algorithm | Multilingual & code | License | Vocab | Measured B/T |
|---|---|---|---|---|---|
| o200k_base | byte-level BPE (regex) | Multilingual; code/math emphasis | MIT (tiktoken) | 200,019 | 4.0751 |
| cl100k_base | byte-level BPE (regex) | Multilingual (narrower) | MIT (tiktoken) | 100,256 | 2.7672 |
| GPT-2 | byte-level BPE | largely English | MIT (code) | 50,256 | 1.9739 |
| Llama-2 | SentencePiece BPE, byte fallback | Multilingual (Europe-strong) | Llama 2 Community License | 32,000 | 2.2613 |
| CodeLlama-7b | byte-level BPE | code (Llama 2 family) | Llama 2 Community License | 32,016 | 2.2613 |
| Qwen2.5-7B-Instruct | byte-level BPE | Multilingual (29+ languages), code | Apache-2.0 | 151,665 | 3.3358 |
| Mistral-7B-Instruct-v0.2 | byte-level BPE, byte fallback | Multilingual; code | Apache-2.0 | 32,000 | 2.3800 |
| Bloom-7b1 | byte-level BPE (BBPE) | ~46 languages | BigScience Open RAIL-M-1.0 | 250,680 | 3.7922 |
| StableLM-base-alpha-3b | byte-level BPE (GPT-NeoX family) | English + code | CC BY-SA 4.0 | 50,277 | 2.4963 |
| DeepSeek-LLM-7b-base | byte-level BPE (custom) | English/Chinese focus, code | other (see card) | 100,015 | 2.2255 |
| DeepSeek-Coder-6.7b-base | byte-level BPE (custom) | code-focused | other (see card) | 32,022 | 1.7775 |
| Falcon-7b | byte-level BPE | English + European, code | Apache-2.0 | 65,024 | 2.0042 |
| NTokenizer v0.3-polished | byte-level BPE + MathSymbolAware | Multilingual by design (incl. code + CJK) | source MIT/Apache-2.0; model CC BY-SA 4.0 | 262,144 | 5.5542 |
NTokenizer v0.3-polished wins 10 of 11 categories against o200k_base and against Qwen2.5-7B-Instruct (both: loses only the tiny 0.16 MB other_languages bucket). Turkish +35.0%, Arabic +33.8%, Hindi +27.1%, Spanish +12.4%, STEM +17.6%, Programming +10.2%, English +6.9% fewer tokens than o200k_base. Full per-bucket table and methodology: COMPARISON_REPORT.
B/T measures compression efficiency, not language-model quality. No downstream 50M-language-model evaluation exists yet. Do not read this as proof of downstream superiority.
Gemini 3.8 Flash is not measured: no Gemini API credential exists in this environment, and no estimate was substituted. See the linked report for Qwen/o200k pinning, exact file hashes, and limitations.
Description
NTokenizer converts text into token IDs (encode) and reconstructs text/raw bytes from IDs (decode). The byte-level design guarantees that any byte sequence β including invalid UTF-8 β is recoverable exactly: decode_bytes(encode_bytes(x)) == x always holds.
Highlights of this release (v0.3-polished, 2026-11-15):
- A Unicode data-loss bug fixed: combining marks at the start of a string were previously dropped by the production pre-tokenizer family (MathSymbolAware) and by the leading-whitespace pre-tokenizer. They now attach correctly and round-trip byte-exactly. Covered by a regression suite (30 adversarial samples Γ 12 pre-tokenizer modes; ~40 samples Γ 2 production modes; 136/136 Rust tests pass).
- Encoding throughput measured ~1.6β1.7Γ the pre-polish implementation on the same machine (interleaved A/B; 8.9β9.0 β 14.5β15.9 MB/s single-thread; 44.6β50.6 β 92.3β96.6 MB/s at 8 threads; peak encode RSS 415 β 360 MB). These are project test-machine measurements, not universal guarantees.
- Normalization is off by default (
Normalization::None). NFC/NFD/NFKC/NFKD are explicit opt-ins and may alter bytes by design.
Training and data
- Corpus: 1.96 GB, 10,555 files, assembled exclusively from the v0.2 leakage-audited clean tree (
data/v0.2/clean/train). No benchmark file, byte or near-duplicate was included (leakage audit documented in the repository reports). - Composition (bytes): Arabic 15.0%, Turkish 14.6%, Hindi 11.7%, English 11.2%, STEM 9.6%, Korean 9.5%, Spanish 7.5%, Programming 7.2%, Japanese 6.9%, Chinese 6.6%, Other 0.2%.
- Sources: Wikimedia projects (the largest share, mostly CC BY-SA/GFDL), arXiv per-article CC BY 4.0 (187 papers only; the rest were quarantined in v0.2), Project Gutenberg (public domain), enwikinews (CC BY, not ShareAlike), Tatoeba (CC BY 2.0 FR), permissively licensed code (CPython PSF 2.0, TypeScript Apache-2.0, Go BSD-3, Rust MIT/Apache-2.0).
- Recipe:
--pretokenizer math --min-freq 2 --min-word-freq 2 --seed 42, 256K vocabulary. training_manifest.jsonin this repo fingerprints every training file (path, bytes, sha256, source, license). No training corpora are distributed.
Evaluation
Development benchmark (544 files, 207,712,944 bytes; development/validation set β not a pristine blind test set):
| Metric | v0.3-polished |
|---|---|
| Overall byte-weighted B/T | 5.5542 |
| Equal-category B/T | 5.1056 |
| Byte-exact decode round-trip | 544/544 files |
| Used vocabulary (dev benchmark) | 245,334 / 262,144 (93.59%) |
Historical v0.3 record: 37,397,108 tokens / 5.5543 / 5.1057. The polished-tree figure is 37,397,319 / 5.5542; the +211-token difference (+0.00056%, 24 files) predates the polishing cycle and is disclosed in the repository.
Reference points (identical files, identical bytes): o200k_base 4.0751 B/T (50,971,306 tokens, reproduced); Qwen2.5-7B-Instruct tokenizer 3.3358 B/T (62,266,982 tokens, pinned revision a09a35458c702b33eeacc393d103063234e8bc28, tokenizer.json SHA-256 c0382117β¦ β byte-identical to the historical Qwen2.5 measurement file); NTokenizer v0.2 4.9290 B/T. Category-level and per-bucket deltas in the comparison section above; full detail in COMPARISON_REPORT.
B/T measures compression efficiency, not language-model quality. No downstream 50M-language-model evaluation exists yet; do not read this as proof of downstream superiority. Decode throughput ~160β278 MB/s; encode throughput as above (test machine; interleaved A/B).
Also disclosed:
- Chinese 3.54 B/T and Japanese 5.01 B/T remain imperfect; CJK-chunking candidates were rejected (Japanese regression).
- Programming 4.10 B/T, English 4.46 B/T.
- A 2026-11-16 research cycle (v0.4-v2, 64K pilots) found a corpus-expansion arm with +0.55% overall B/T but guardrail regressions; nothing from that cycle was promoted and no v0.4-v2 model exists.
- The 6.00 B/T aspiration was not reached; this release is the polished v0.3 model.
Compatibility
- Native
.ntok(this repository): fully validated (ntokenizer validate), byte-exact both directions, special tokens isolated. tokenizer.json(Hugging Face export, included here as a convenience): loads viatokenizers0.23; byte-exact encodeβdecode in both directions; token-ID parity is only 3/16 (HF ByteLevel pre-tokenization cannot express MathSymbolAware). Decode-exact β ID-parity. The native format is authoritative. Do not claim full HF parity.
Limitations
- The development benchmark is not a blind test; no untouched test set exists in this environment.
- Other-languages bucket (0.16 MB) lags o200k_base.
- ASAN-instrumented fuzzing is blocked by a pre-existing linker issue (not claimed). Load/round-trip fuzzing ran ~1.0M combined cases with zero crashes.
- No fine-grained safety, bias, or fairness evaluation is provided.
- Do not redistribute this model under Apache-2.0 without respecting ShareAlike.
License and attribution
- Source code: MIT OR Apache-2.0.
- This model artifact and the HF export: CC BY-SA 4.0. The model is a machine-learned derivative of mostly CC BY-SA/GFDL text (Wikimedia) and CC BY material (arXiv 187 papers; enwikinews; Tatoeba). ShareAlike applies to adaptations. Attribution follows the pattern
"<project> contributors, <license>, <URL>"(e.g. "Simple English Wikipedia contributors, CC BY-SA 4.0, https://simple.wikipedia.org"). Full attribution set: seeNOTICEin the GitHub repository. - Nothing here licenses the training corpora themselves; they are not distributed.
FAQ β honest answers to reasonable skepticism
NTokenizer is new and independent. We know why you'd hesitate; here are the questions we'd ask in your place, answered with evidence you can check yourself.
"Why should I trust an unknown tokenizer over OpenAI's, Meta's or Google's?"
You shouldn't β trust nothing, verify. Every NTokenizer artifact is SHA-256-pinned; every benchmark number in the docs is reproducible by running one command (./target/release/examples/benchmark_ntok --model output/v0.3/ntokenizer-0.3-256k.ntok --manifest <manifest>); the model round-trips arbitrary bytes byte-exactly (544/544 on the development benchmark); 136/136 tests pass; ~1.0M fuzz cases ran without a crash. Nothing about this project asks you to believe a brand. If verification takes 15 minutes, do it β if it doesn't check out, don't use it.
"Is the benchmark legit, or was it tuned to make NTokenizer win?"
It's the same 544 files, 207,712,944 bytes, used unchanged for every candidate, external tokenizers included; files and manifest are untouched (SHA-pinned). It is a development/validation set used throughout development β not a blind test β and we say so everywhere. Different slices of it will weight differently: English (NTokenizer beats o200k by +6.9%), programming (+10.2%), STEM (+17.6%), Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%), Chinese (+22.3%), Japanese (+31.7%) fewer tokens than o200k_base. If you need numbers for your domain, run the harness on your own files β it takes minutes.
"Does higher B/T mean the language model will be better?"
No β and we say this in every public page. B/T is compression density. It tells you how many tokens you pay for the same bytes. It does not measure next-token loss, task accuracy, fairness or safety. No 50M language model exists yet, and nothing in this repository should be read as a claim that NTokenizer produces better LLMs. What it does demonstrate, carefully scoped, is that NTokenizer compresses multilingual and technical text more efficiently than every public competitor we could measure β e.g. 26.6% fewer tokens than o200k_base, 39.9% fewer than Qwen2.5-7B-Instruct, 50.2% fewer than cl100k_base, 57.2% fewer than Mistral-7B, 59.3% fewer than Llama-2-class, 63.9% fewer than Falcon-7b and 64.5% fewer than GPT-2 (per-file evidence in reports/public-benchmark/STANDARDIZED_RESULTS.csv).
"What are you hiding? What are the known weaknesses?"
Nothing by design; the weaknesses are documented rather than discovered late: other_languages (0.16 MB) loses to o200k_base (3.32 vs 5.19); Chinese and Japanese are improved over v0.2 but Han-chunking candidates were rejected (Japanese regression) and pure Han efficiency is still imperfect; Gemini 3.8 Flash and MosaicML MPT are marked NOT MEASURED (no credential / unavailable models); the 6.00 B/T target was not achieved (the v0.4-v2 cycle is archived, not promoted); the HF export has 3/16 token-ID parity (documented); ASAN-instrumented fuzzing remains blocked (disclosed, not claimed as done).
"Why is English's gain smaller than the other languages?"
English is where the vendor tokenizers concentrate their training; o200k is genuinely strong there, and +6.9% is what an independent multilingual corpus buys. The NTokenizer edge is elsewhere: STEM (+17.6%), code (+10.2%), and the long tail of multilingual technical text β Turkish (+35.0%), Arabic (+33.8%), Hindi (+27.1%), Korean (+34.7%). If your corpus is pure English news, you will not see a dramatic difference; if it is mixed-language or code-heavy, you will see a large one.
"Is it safe / stable on weird inputs?"
Byte-level by construction: any byte sequence β invalid UTF-8 included β is recoverable exactly (decode_bytes(encode_bytes(x)) == x), decoding never fabricates bytes, and the training-time guarantee (136 tests, ~1M fuzz cases, zero crashes) is enforced by CI-style local verification. Combining marks, ZWJ sequences, variation selectors, Arabic harakat, Devanagari matras, planar CJK, unassigned scalars β all round-trip byte-exactly; the regression suite covers 30 adversarial samples Γ 12 pre-tokenizer modes.
"Why is it not famous?"
It is brand-new, independent and distributed under a model license (CC BY-SA 4.0) that doesn't allow advertising without care. Fame requires time and backing this project doesn't have β that is an honest constraint, not a hidden disqualifier. Fame has never been evidence of quality; in this space, it is usually marketing backstopped by a large corpus. Here, the correct evaluation is two commands and 15 minutes of your time.
"Can I use it commercially?"
Source code: MIT OR Apache-2.0 β free for any use, including commercial, with no copyleft. The trained model artifact: CC BY-SA 4.0 β you may use it commercially; adaptations (e.g., redistributed fine-tuned variants of the tokenizer) must be distributed under CC BY-SA 4.0 with attribution. Training corpora are not redistributed. This is a summary, not legal advice; consult counsel for your case.
"Should I switch from o200k_base today?"
You should try NTokenizer on your own data first β that is the only honest answer. If your workload is multilingual, STEM, code or Turkish/Arabic/Hindi/Korean-heavy, expect meaningful token-count reduction and thus cost/throughput gains (independent of any downstream LM claim). If it is monolingual English, o200k is a solid, battle-tested choice and the improvement will be modest. Nothing about either choice is irreversible β the harness prints the numbers for your files in one command.
Links
- Source: https://github.com/Neuratal/neuratal-tokenizer
- Release: https://github.com/Neuratal/neuratal-tokenizer/releases/tag/v0.3.0
- Citation: see
CITATION.cffin the repository (author RΓΌzgar Erdem, Neuratal, 2026)