Gemma 4 E2B-it, vocabulary pruned to English (m = 1)
Homework 1 (tokenizer pruning), Modern Methods and Algorithms of Generative AI, Skoltech, Fall 2026.
google/gemma-4-E2B-it with its vocabulary pruned from 262,144 to 103,351 tokens (padded to 103,360 rows):
every token that occurs at least once when 12 MB of English Wikipedia (wikimedia/wikipedia, 20231101.en)
is tokenized, plus all special/added tokens, the tokenizer's base alphabet, and the merge closure (route
closure: every piece formed while BPE builds a kept token). No training: the input embedding (tied with the
output head) and the per-layer embedding table are sliced with the id map (pruning.json: new id -> old id);
all token-id fields in the configs are remapped.
| original | pruned | |
|---|---|---|
| vocabulary | 262,144 | 103,351 |
| parameters | 5.10 B | 3.44 B (-32.7 %) |
| bf16 checkpoint | 9.54 GiB | 6.4 GiB |
| held-out bits per byte (English Wikipedia, 1 MB) | 1.495 | 1.526 |
| MMLU-Pro (12,032, direct answer) | 31.55 | 31.16 |
Loads with transformers 5.14 (AutoModelForImageTextToText, AutoTokenizer). Rows 103,351-103,359 are zero
padding: suppress them when generating. Text outside English Wikipedia's vocabulary (other scripts, some
markdown, ALL-CAPS words) is spelled with smaller pieces or bytes and is generated less reliably.
- Downloads last month
- 5