Salesforce

/

xgen-7b-8k-base

like 315

This tokenizer is a tiktoken tokenizer, so it's not using the huggingface tokenizer. If you look at model.config.vocab_size and then len(tokenizer) you'll see that there's a mismatch of vocab sizes. Still working on figuring out what to do about it. But that's why tokenizer.add_special_tokens() doesn't work: the implementation is tiktoken so the functionality is different.

You can take a look at their tokenizer.py file to see how to add tokens and such: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/main/tokenization_xgen.py