Risk to Using llama-tokenizer branch?
#9
by
lapp0
- opened
Thanks for creating this fork!
I was wondering, why isn't llama-tokenizer branch the main branch? Is there some risk I should be aware of?
Glad it's helpful!
When I originally posted this I wasn't 100% sure I got the tokenizer converted correctly. I put it on a branch so it could be submitted to the HF leaderboard and evaluated, but left the main branch as is because it was a known working configuration.
I'm now quite confident in the llama-tokenizer branch. It could be moved over to main for sure. But 01-ai now have their official repos llamaized, so there isn't much reason to be using this one anymore - I figured I would leave it as is, as a historical curiosity.