ArminBurkhardt/DeepSeek-V4-Pro-tokenizer-65536

DeepSeek-V4-Pro Tokenizer (Compact, 65536)

Compact remap of DeepSeek-V4-Pro's tokenizer down to a 65,536-entry vocab (fits uint16), for smaller embedding tables.

Contents

• tokenizer.json: BPE model, vocab + merges, pre/post-processing, added tokens.
• tokenizer_config.json: HF PreTrainedTokenizerFast config (special token strings, no auto BOS/EOS).
• config.json: full DeepSeek-V4 model config (architecture, MoE, MLA, YaRN RoPE, FP8 quant) : reference only, not part of tokenizer itself.
• chat_template.jinja: chat prompt template.
• id_remap.json: old-id → new-id mapping (65,536 entries) used to build this compact vocab from the original.

Vocab

• Type: BPE, byte-level fallback pretokenizer, \p{N}{1,3} digit split + CJK isolation + GPT-style punctuation/word split.
• vocab_size: 65536
• base vocab: 64256 tokens + merges
• added/special tokens: 1283 (mostly reserved placeholders <｜place▁holder▁no▁N｜> filling unused ids)

Key special tokens

id	token
0	<｜begin▁of▁sentence｜> (BOS)
1	<｜end▁of▁sentence｜> (EOS / PAD)
2	<｜▁pad▁｜>
65102	<｜end_of_query｜>
65518/65519	<｜rl_image_pad｜> / <｜rl_image_start｜>
65527–65535	vision/grounding tags: <｜polygon｜>, <｜point｜>, <｜box｜>, <｜ref｜>, <｜image｜> (open/close pairs)
64256	<think> (repurposed placeholder)
64257	</think> (repurposed placeholder)

add_bos_token / add_eos_token are both false, the caller must add them explicitly.

Thinking tokens

Two previously-unused placeholder ids were repurposed as reasoning-trace delimiters:

• id 64256: <｜place▁holder▁no▁0｜> → <think>
• id 64257: <｜place▁holder▁no▁1｜> → </think>

Changed in tokenizer.json (added_tokens entries) and tokenizer_config.json (added_tokens_decoder + additional_special_tokens). These ids were not part of model.vocab/merges, so the base BPE vocab is untouched — only the added-token layer changed. chat_template.jinja was not modified; wrap reasoning content in <think>...</think> manually or update the template if the chat format should emit it automatically.

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<this dir>")
tok.encode("hello world")

Notes

• id_remap.json documents provenance only; not needed at inference time; tokenizer.json is self-contained.
• Reserved placeholder ids leave room for future special tokens without reshuffling the vocab.