---
library_name: transformers
tags: []
---

Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens 
(`|||PHONE_NUMBER|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds 
extra tokens to fill gaps in tiktoken `cl100k_base` version.