allenai
/

dolma2-tokenizer

Inference Endpoints

Model card Files Files and versions Community

soldni commited on Jul 12

Commit

dd6339e

•

1 Parent(s): 58b74ae

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -4,5 +4,5 @@ tags: []
 ---
 Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens
-(`|||IP_ADDRESS|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds
 extra tokens to fill gaps in tiktoken `cl100k_base` version.

 ---
 Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens
+(`|||PHONE_NUMBER|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds
 extra tokens to fill gaps in tiktoken `cl100k_base` version.