transfer-learning-gpt / added_tokens.json
AlekseyKorshuk's picture
add tokenizer
09488e2
{"<bos>": 40478, "<eos>": 40479, "<speaker1>": 40480, "<speaker2>": 40481, "<pad>": 40482}