JamesQuartz/qt-VI.6.4-49k

QT VI.6.4 - 49K Multilingual Tokenizer

QT VI.6.4 is a 49,152-token byte-level BPE tokenizer built for broad multilingual coverage with a strong low-resource tail. It covers all 204 FLORES-200 languages, carries a full agentic / Fill-in-the-Middle control-token layout, and is designed as the shared backbone for an English-first, community-extensible model family: pretrain concentrated on English and code, then fine-tune downstream into other languages without hitting byte-fallback.

Highlights

• 49,152 tokens (a clean 384 x 128 multiple, chosen to keep the lm_head projection aligned to Tensor-Core tile sizes) - under 40% the vocabulary size of the tokenizers it is compared against (Llama 3.2: 128,256; DeepSeek V4: 129,280 as measured here). A smaller vocabulary means a smaller embedding matrix, which is the point: it keeps the embedding footprint low on a small model.
• Matches English compression at a fraction of the budget. 4.63 characters/token versus Llama 3.2's 4.68 - near-identical, at roughly one third the vocabulary size.
• 200 of 204 FLORES-200 languages with real sub-byte coverage - broader than Llama 3.2 (179) and DeepSeek V4 (193), and achieved with less than half their vocabulary.
• Strongest on low-resource scripts. It beats both Llama 3.2 and DeepSeek V4 on 37 languages, concentrated in the underserved scripts where general-purpose tokenizers fall back toward bytes.
• Frozen control-token layout with FIM, so the tokenizer is ready for chat, tool-use, and code-infill fine-tuning without any later retraining.
• Lossless. Byte-level fallback guarantees decode(encode(x)) == x for any input, including code, emoji, and mixed scripts.

Vocabulary size in context

Tokenizer efficiency should be read against vocabulary size, because a larger vocabulary buys compression at the cost of a larger embedding matrix.

Tokenizer	Vocab size	English (chars/token)	Real coverage (of 204)
QT VI.6.4	49,152	4.63	200
Llama 3.2	128,256	4.68	179
DeepSeek V4	129,280	4.88	193

QT VI.6.4 reaches essentially the same English density as Llama 3.2 and covers more languages, using roughly a third of the vocabulary. On a small model that difference is paid straight back into the non-embedding parameter budget. (Llama 3.2's 128,256 is from its published config; the DeepSeek V4 figure is as measured in this benchmark run.)

Benchmark results

Measured on FLORES-200 devtest. The primary metric is bytes per token (higher is better: more source text encoded per token). Global mean across 204 languages: 3.07 bytes/token.

Where VI.6.4 leads (bytes/token, vs Llama 3.2 and DeepSeek V4)

Language	Script	QT VI.6.4	Llama 3.2	DeepSeek V4
Tibetan	Tibetan	6.70	1.44	2.65
Tamil	Tamil	6.18	2.02	5.08
Burmese	Myanmar	6.07	1.47	3.25
Hindi	Devanagari	5.70	4.85	4.46
Malayalam	Malayalam	5.73	1.69	3.45
Telugu	Telugu	5.21	1.56	3.50
Thai	Thai	5.03	-	-

On Tibetan, Burmese, Malayalam, and Telugu the gain over Llama 3.2 is roughly four to five times, because those scripts fall to near byte-level in general-purpose tokenizers and VI.6.4 gives them real subword inventories.

Where the majors stand (characters/token, higher is better)

Language	QT VI.6.4	Llama 3.2	DeepSeek V4
English	4.63	4.68	4.88
German	3.11	3.50	3.75
Russian	2.26	3.16	3.32
Chinese	1.12	1.21	1.70
Arabic	1.91	2.53	2.66

On the high-resource majors, Llama 3.2 and DeepSeek V4 are more efficient. VI.6.4 trades some major-language density for its much broader coverage and its low-resource strength - a deliberate choice for a backbone meant to be fine-tuned across many languages. English stays competitive.

Head-to-head

Comparison	Vocab sizes	VI.6.4 wins
vs Llama 3.2 (190 shared languages)	49K vs 128K	47 (25%)
vs DeepSeek V4 (203 shared languages)	49K vs 129K	41 (20%)

The wins concentrate in the low-resource tail; the losses concentrate in the high-resource majors. Both competitors carry roughly 2.6x the vocabulary, so their major-language edge is bought with a substantially larger embedding matrix.

Special tokens

The control region is a fixed 160-token block. IDs 0-22 are frozen and must not be reordered - downstream chat, tool-use, and infill fine-tunes depend on these exact positions. The remaining slots are reserved for future use and can be renamed in place without shifting any existing ID or changing the vocabulary size.

ID	Token	Purpose
0	<|padding|>	padding
1	<|bos|>	begin of sequence
2	<|endoftext|>	document / sequence boundary
3	<|unk|>	unused (byte-level fallback means no true UNK)
4	<|sep|>	separator
5-6	<|im_start|> <|im_end|>	chat turn boundaries (ends a turn, not the sequence)
7-9	<|system|> <|user|> <|assistant|>	chat roles
10-13	<|tool_call|> <|/tool_call|> <|tool_result|> <|/tool_result|>	tool use
14-15	<|thinking|> <|/thinking|>	reasoning spans
16-17	<|code|> <|/code|>	code spans
18	<|lang:en|>	language tag
19-22	<|fim_prefix|> <|fim_suffix|> <|fim_middle|> <|fim_pad|>	Fill-in-the-Middle (code infill)

Fill-in-the-Middle tokens are included at the tokenizer level because infill is a train-time capability: a model can only learn to place these tokens if they exist in the vocabulary during pretraining.

Usage

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")

ids = tok.encode("The committee recommended further investigation.").ids
print(len(ids), tok.decode(ids))

# lossless on mixed scripts and code
s = "def f(x): return x  中文 العربية 🌍"
assert tok.decode(tok.encode(s).ids) == s

Design notes

• Byte-level BPE with a word-splitting pre-tokenizer and digit grouping, plus a SuperBPE-style superword stage for cross-word-boundary tokens in high-frequency text.
• Coverage before major-language density. The training corpus is weighted so that low-resource scripts keep real subword inventories rather than collapsing to bytes. This is what produces the low-resource wins and the modest major-language cost.
• Lossless by construction. Byte-level fallback means any input round-trips exactly; there is no true out-of-vocabulary case.

Intended use and limitations

• Intended: a shared backbone tokenizer for multilingual models, especially English-first models meant to be fine-tuned into additional languages, and for code models that need Fill-in-the-Middle.
• Not intended as a majors-optimal tokenizer. If your workload is exclusively high-resource languages (English, German, Russian, Chinese), a tokenizer that concentrates its budget there will compress those languages more tightly. VI.6.4 deliberately spreads its budget for coverage.
• Benchmarks are FLORES-200 devtest and reflect tokenizer efficiency, not downstream model quality.

Files

• tokenizer.json - the tokenizer (HuggingFace tokenizers format)
• vocab.json, merges.txt - vocabulary and merge table

Citation

@misc{qt_vi_6_4_2026,
  title  = {QT VI.6.4: A 49K Multilingual Tokenizer with Low-Resource Emphasis},
  author = {James (JamesQuartz)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/JamesQuartz/qt-VI.6.4-49k}}
}

License

Apache-2.0.