wikitext2-bpe-30k

A byte-level Byte-Pair Encoding tokenizer trained from scratch on the WikiText-2 (wikitext-2-v1) corpus.

Configuration

Setting	Value
Algorithm	Byte-level BPE
Target vocabulary	30,000
Realised vocabulary	30,000
Minimum pair frequency	2
Normalizer	NFKC (no lowercasing)
Pre-tokenizer	ByteLevel (add_prefix_space=False)
Post-processor	[CLS] $A [SEP]
Special tokens	[PAD], [UNK], [CLS], [SEP], [MASK]
Training rows	21,329 (cleaned and exactly deduplicated)

Preprocessing

wikitext-2-v1 is the word-level WikiText subset, so the training text was cleaned first: <unk> placeholders removed, @-@ / @.@ / @,@ artifacts reversed, detached punctuation and possessives re-attached, whitespace collapsed. Exact deduplication was applied to the training split only.

Evaluation

	validation	test
lines evaluated	2461	2858
total tokens	209667	233147
round-trip exact (%)	98.618	99.895
deterministic (%)	100	100
[UNK] rate (%)	0	0
avg tokens/line	85.2	81.58
median tokens/line	72	66
p95 tokens/line	227	229
chars per token	4.807	4.782
tokens per word	1.197	1.197

Because the base alphabet contains all 256 bytes, [UNK] is never emitted and decoding is lossless.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("PhilixOkigbo/wikitext2-bpe-30k")
print(tok("Byte-pair encoding is a subword tokenization algorithm."))

Trained by Chinedu Okigbo.