add missing special tokens in tokenizer.json
#10
by
mmoskal
- opened
tokenizer_config.json has <think>, </think> and <tool_response> tokens, which are missing from tokenizer.json. If I understand correctly, the files should match.