NexTok-32K

NexTok-32K is a custom 32,000-vocabulary byte-level BPE tokenizer developed for a from-scratch language model focused on computer science, programming, mathematics, educational text, and general natural language.

Overview

Property	Value
Tokenizer	Byte-Level BPE
Vocabulary	32,000
Base	UTF-8 bytes
Pre-tokenization	GPT-4-style regex + ByteLevel
Normalizer	None
Decoder	ByteLevel
BOS / EOS	<|endoftext|>
PAD	<|pad|>
Chat start	<|im_start|>
Chat end	<|im_end|>

The tokenizer uses a 32K vocabulary as a deliberate trade-off between tokenization efficiency and vocabulary-dependent model parameter cost.

Training Data

Approximately 1,000,000 samples were used with the following target mixture:

Dataset	Share
FineWeb-Edu	70%
Cosmopedia v2	15%
StarCoderData — Python	8%
OpenWebMath	5%
StackOverflow	2%
Total	100%

The mixture was selected to provide coverage of general educational prose, mathematics, programming, technical discussions, and code.

Tokenizer Configuration

The tokenizer uses the following GPT-4-style regex:

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s

ByteLevel settings:

add_prefix_space = false
use_regex = false

No normalizer is used.

Held-Out Benchmark

NexTok-32K was compared with the Cosmo2 tokenizer (49,152 vocabulary) on held-out benchmark datasets. The evaluation measures tokenizer efficiency, not model accuracy.

Dataset	Samples	NexTok Tokens	Cosmo Tokens	NexTok Tok/Char	Cosmo Tok/Char	Ratio	Avg NexTok	Avg Cosmo	NexTok p95	Cosmo p95	NexTok RT Failures	Cosmo RT Failures	Time
ARC-Challenge	1,172	31,037	30,711	0.2013	0.1992	1.0106	26.48	26.20	57.00	57.00	0	0	2.4s
HellaSwag	5,000	197,748	192,962	0.2333	0.2277	1.0248	39.55	38.59	79.00	77.00	0	0	7.2s
MMLU	5,000	195,215	196,865	0.2235	0.2254	0.9916	39.04	39.37	103.00	104.00	0	0	5.3s
GSM8K	1,319	80,494	81,777	0.2544	0.2585	0.9843	61.03	62.00	106.00	105.00	0	0	2.8s
Winogrande	1,267	29,476	28,237	0.2308	0.2211	1.0439	23.26	22.29	30.70	29.70	0	0	2.0s
Aggregate	13,758	533,970	530,552	0.2302	0.2288	1.0064	—	—	—	—	0	0	19.7s

Ratio interpretation: values below 1.0 mean NexTok-32K used fewer tokens.

Result

On this evaluation, NexTok-32K used 0.64% more tokens overall than Cosmo2.

NexTok-32K used fewer tokens on:

• MMLU: 0.84% fewer
• GSM8K: 1.57% fewer

Cosmo2 used fewer tokens on:

• ARC-Challenge: 1.06% fewer
• HellaSwag: 2.48% fewer
• Winogrande: 4.39% fewer

Both tokenizers had 0 round-trip failures across all 13,758 evaluated samples.

This is a tokenizer-efficiency comparison only. It does not establish downstream model performance. The vocabulary sizes also differ: 32K vs 49,152.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "Abhay2310/NexTok-32K"
)

text = "Explain binary search."

ids = tokenizer.encode(
    text,
    add_special_tokens=False
)

decoded = tokenizer.decode(
    ids,
    skip_special_tokens=False
)

print("Tokens:", tokenizer.tokenize(text))
print("Token IDs:", ids)
print("Token count:", len(ids))
print("Decoded:", decoded)
print("Round-trip:", decoded == text)

Special Tokens

Token	ID	Purpose
<|endoftext|>	0	BOS / EOS
<|pad|>	1	Padding
<|im_start|>	2	Chat turn start
<|im_end|>	3	Chat turn end

The chat tokens are reserved for future instruction or conversational training; they were not evidence that chat-formatted data was used during tokenizer training.

Limitations

• The 32K vocabulary is smaller than Cosmo2's 49,152 vocabulary.
• Training data is strongly oriented toward English educational, mathematical, technical, and programming text.
• The code component was Python-focused; other programming languages have not yet been benchmarked.
• The current held-out evaluation is preliminary and measures tokenization efficiency rather than downstream language-model quality.

Status

Experimental — Active Development

NexTok-32K is the tokenizer component of an ongoing from-scratch language-model project.

Citation

@misc{kashyap2026nextok32k,
  title        = {NexTok-32K},
  author       = {Kashyap, Abhay J.},
  year         = {2026},
  howpublished = {Hugging Face Hub}
}