nlgp-docstring / added_tokens.json
Geert's picture
add tokenizer
6258c1d
raw
history blame
240 Bytes
{"<|12space|>": 50258, "<|endofcomment|>": 50267, "<|8space|>": 50265, "<|10space|>": 50257, "<|2space|>": 50262, "<|14space|>": 50259, "<|4space|>": 50263, "<|6space|>": 50264, "<|cell|>": 50266, "<|18space|>": 50261, "<|16space|>": 50260}