forked from https://huggingface.co/huggyllama/llama-13b/tree/main
This tokenizer supports Birchlabs/llama-13b-stepwise-adapter
.
Adds four new tokens for stepwise reasoning:
<|step_start|>
<|step_end|>
<|answer_start|>
<|answer_end|>
See Birchlabs/llama-13b-stepwise-adapter
for details of how all the parts should be used together.