fix: update tokenizer config to support `add_generation_prompt=True` and clarify content
#63
by
lamhieu
- opened
Here I have updated it so that for each role it will return the correct tokens id of that role more clearly, trim the spaces and support add_generation_prompt config.