TriusLD

Trius's Little Dictionary — a byte-level BPE tokenizer with 64,972 tokens and native coverage of 19 writing systems, built to spend a limited vocabulary on breadth rather than on English alone.

Vocabulary	64,972 (256 bytes + 14 special + 64,702 merges)
Type	byte-level BPE, NFC normalisation
Scripts	Latin, Cyrillic, Greek, Hebrew, Arabic, Thai, Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala, Han, Kana, Hangul
Languages	58

Special tokens:

<|pad|>  <|bos|>  <|eos|>  <|unk|>  <|system|>  <|user|>  <|assistant|>
<|end|>  <|fim_prefix|>  <|fim_middle|>  <|fim_suffix|>  <|tool_call|>
<|tool_result|>  <|image|>

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("TriusAI/TriusLD")

ids = tok("The quick brown fox — 敏捷的棕色狐狸 — 빠른 갈색 여우")
print(tok.decode(ids["input_ids"]))

Without transformers:

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
print(tok.encode("Hello, 世界!").tokens)

Behaviour

• Lossless. The 256-token byte alphabet means any UTF-8 input encodes and decodes exactly, including scripts this vocabulary was not trained on.
• NFC is applied for you. Composed and decomposed forms of the same text produce the same ids. NFKC is deliberately not used: it would fold fullwidth punctuation and ligatures and destroy information.
• No automatic BOS/EOS. Add them explicitly; <|bos|> and <|eos|> exist but are not inserted by encode.
• Chinese, both standards. Simplified and Traditional are distinct code points rather than normalisation variants, and both inventories are carried in the vocabulary. The card's language field can only hold an ISO 639 code, so it says zh; the compression figures below list the two separately.
• add_prefix_space=False, matching training.
• Whitespace is not stripped, and clean_up_tokenization_spaces is disabled — decoding is exact.

Compression

Measured on held-out text (FineWeb-2 test split, 100,000 characters per language). Tokens per character, lower is better.

family	tpc
Latin	0.299
Cyrillic	0.494
Arabic script	0.519
Greek + Hebrew	0.540
Thai	0.595
CJK	0.730
Indic	0.732

Selected languages:

language	tpc	language	tpc
English	0.260	Hindi	0.657
Vietnamese	0.276	Japanese	0.678
Spanish	0.277	Chinese (S)	0.708
German	0.293	Bengali	0.722
Russian	0.461	Korean	0.733
Arabic	0.522	Tamil	0.754
Hebrew	0.556	Chinese (T)	0.802

English is about 3.85 characters per token. For comparison on the same text: cl100k 0.842 average over 58 languages, Llama 3 0.699, Qwen3 0.675, o200k 0.394, Gemma 3 0.392 — against 0.524 here, at a third to a half of their vocabulary.

How it was built

Per-script BPE trained with explicit quotas, then composed by a dependency-respecting union; the top characters of each script are forced to be single tokens by a coverage pass. Details, including two merge-ordering bugs and their symptoms, are in the accompanying article.

Training data: FineWeb (English) and FineWeb-2 (everything else). No training text is redistributed here; this repository contains only tokenizer statistics — a vocabulary and a list of merge pairs.

Licence

Apache License 2.0. Copyright 2026 TriusAI. The full text is in LICENSE, and a NOTICE file records the data provenance. In short: use it for anything, commercially included, provided you retain the copyright and licence notices, state significant changes, and do not use the TriusAI name to endorse your derivative. The licence carries an express patent grant.

A tokenizer's merges are frequency statistics over text rather than a copy of it, and no source text is redistributed here. The training data — FineWeb and FineWeb-2, both published under ODC-By 1.0 — is credited in NOTICE, because the debt is real even where the law is silent.

Limitations

• Korean uses the 2,350 most frequent Hangul syllables plus jamo, not the full 11,172-syllable block; rare syllables cost two or three tokens.
• Indic coverage is real but thin — ten scripts share 2,660 merges. Expect roughly twice the tokens of a 200k-token vocabulary on Hindi, Bengali or Tamil.
• Unsupported scripts fall back to bytes at two to four tokens per character.
• Code is served by the Latin inventory: adequate, not tuned.

Citation

@misc{triusld2026,
  title  = {TriusLD: a compact multilingual byte-level BPE tokenizer},
  author = {TriusAI},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/TriusAI/TriusLD}}
}