gpt2_mwp / added_tokens.json
Chakita's picture
add tokenizer
9611727
raw
history blame contribute delete
No virus
90 Bytes
{"<|BOS|>": 50257, "<|PAD|>": 50260, "<|UNK|>": 50259, "<|SEP|>": 50261, "<|EOS|>": 50258}