BloatTokenizer32K

Compact vocabulary. Efficient tokenization. Lossless decoding.

BloatTokenizer32K is an open-source text tokenizer featuring a 32,000-entry vocabulary, a native C++ encoding engine, and Python bindings.

Designed for language-model training and inference, it supports English, programming languages, mathematics, and conversational text.

On our mixed-text development benchmark, BloatTokenizer32K achieves compression comparable to GPT-4's cl100k_base and GPT-4o's o200k_base, despite using substantially smaller vocabulary capacity.

GitHub Repository

Key features

• 32,000 vocabulary entries, including 16 reserved control tokens.
• 21.96% fewer tokens than GPT-2 on the evaluated mixed corpus.
• 6.48% fewer code tokens than cl100k_base on the code subset.
• 0.26% fewer tokens than cl100k_base overall.
• 17.20 MB/s four-core warm-cache batch encoding.
• 17.10 MB/s four-core fresh-instance batch encoding.
• 3,412/3,412 successful lossless round trips.
• Native C++ engine with configurable parallel batch encoding.

Benchmarks

Token compression

Token compression results

Lower token counts are better.

Tokenizer	Vocabulary	Output tokens
BloatTokenizer32K	32,000	997,416
GPT-4o (o200k_base)	200,000	998,383
GPT-4 (cl100k_base)	100,261	1,000,000
Qwen2.5-Coder	151,665	1,018,726
Mistral NeMo	131,072	1,052,365
Phi-2	50,295	1,159,396
GPT-2	50,257	1,278,138

Performance by domain

Domain	BloatTokenizer32K	GPT-4 tokenizer
English	522,543	500,000
Mathematics	100,805	100,000
Code	374,068	400,000
Total	997,416	1,000,000

The tokenizer's strongest measured compression advantage is on code. English and mathematics require slightly more tokens than cl100k_base on this evaluation.

Encoding performance

Encoding speed

Measured in input MB/s, using matched CPU budgets.

Tokenizer	1-core warm	4-core warm	4-core fresh
BloatTokenizer32K	8.53	17.20	17.10
cl100k_base	10.28	12.37	13.15
o200k_base	13.83	20.91	19.37
Qwen2.5-Coder	2.09	4.03	4.19
GPT-2	9.77	8.00	13.57

In this configuration, BloatTokenizer32K achieved approximately 39% higher warm-cache four-core batch throughput than cl100k_base, and approximately 30% higher fresh-instance batch throughput.

Single-core encoding remains slower than both OpenAI tokenizer implementations.

Evaluation methodology

The benchmark used identical inputs and matched CPU budgets across 13 tokenizers.

Property	Configuration
Excerpts	3,412
Input size	4,427,375 UTF-8 bytes
Reference tokens	1,000,000 (cl100k_base)
Corpus distribution	50% English, 10% math, 40% code
Trials	5 per configuration
Serial CPU budget	1 logical CPU
Batch CPU budget	4 logical CPUs
Decode failures (BloatTokenizer32K)	0

All measurements report median throughput. Fresh-instance measurements exclude tokenizer construction time; warm-cache measurements follow an untimed corpus pass.

Limitations: The evaluation corpus was excluded from tokenizer training but was previously inspected during development. Results are specific to this corpus and hardware configuration. Multilingual performance beyond the evaluated distribution has not been established, and baseline training-data overlap is unknown.

Getting started

The tokenizer artifacts are available in the artifacts/ directory of this repository.

The native encoding engine and Python interface are maintained on GitHub.

Requirements

• Python 3.12 or 3.14 for prebuilt Windows extensions
• Windows x64 for bundled binaries
• C++17 and pybind11 for building from source on other platforms

Download

Install the Hugging Face Hub client:

pip install -U huggingface_hub

Download the tokenizer artifacts:

from huggingface_hub import snapshot_download

path = snapshot_download(
    repo_id="Qvinfor/BloatTokenizer32K",
    repo_type="model",
)

print(path)

Python usage

First obtain the Python/C++ engine from the GitHub repository and follow its installation instructions.

Then load the downloaded Hugging Face artifacts:

from pathlib import Path
from huggingface_hub import snapshot_download
from curriculum_tokenizer import CurriculumTokenizer

repo_path = snapshot_download(
    repo_id="YOUR_HF_USERNAME/BloatTokenizer32K"
)

tokenizer = CurriculumTokenizer.from_pretrained(
    str(Path(repo_path) / "artifacts")
)

text = "Fast, lossless text encoding."

ids = tokenizer.encode(
    text,
    add_special_tokens=False
)

decoded = tokenizer.decode(
    ids,
    skip_special_tokens=False
)

assert decoded == text

print("Token IDs:", ids)
print("Decoded:", decoded)

By default, encode() adds <|endoftext|> and decode() skips control tokens. Use the explicit options above when a round-trip should reproduce the input.

Batch tokenization

texts = [
    "Hello, world!",
    "def add(a, b): return a + b",
    "2 + 2 = 4"
]

batch = tokenizer.encode_batch(
    texts,
    add_special_tokens=False,
    num_threads=4
)

print(batch["input_ids"])

Batch encoding preserves input order.

Compatibility note: The published artifacts use the BloatTokenizer runtime. Direct loading through Hugging Face AutoTokenizer is not currently documented or guaranteed.

Special tokens

The vocabulary includes 16 reserved control tokens:

Token	ID
`<	endoftext
`<	pad
`<	system
`<	user
`<	assistant
`<	end_turn
`<	tool_call
`<	end_tool_call
`<	tool_result
`<	end_tool_result
`<	final
`<	end_final
`<	think
`<	end_think
`<	verify
`<	end_verify

Applications determine how these control markers are used in prompts and structured conversations.

Model compatibility

BloatTokenizer32K is intended for models trained using its vocabulary and token-ID mapping.

It is not a drop-in replacement for unrelated pretrained models. Changing tokenizers requires appropriate model adaptation.

License

Apache License 2.0

The publicly released tokenizer artifacts and runtime are open-source.

The proprietary vocabulary-training algorithm and its training implementation are not included.

Source code and feedback

The full C++ runtime, Python bindings, usage examples, and technical documentation are available on GitHub.

BloatTokenizer32K — GitHub

Bug reports, independent benchmarks, performance comparisons, and community contributions are welcome.