Slightly modified version of cl100k_base that supports Dolma 1.x special tokens (|||PHONE_NUMBER|||, |||EMAIL_ADDRESS|||, |||IP_ADDRESS|||) as well as adds extra tokens to fill gaps in tiktoken cl100k_base version.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference API

Unable to determine this model’s pipeline type. Check the docs .