Update special_tokens_map.json

#10
by fxmarty - opened

Following https://github.com/huggingface/transformers/pull/23909 it appears that this filed is relied on for special tokens. Not sure how it works internally, but I am not getting a 2 token at the beginning of a sentence with transformers>=4.34 instead of the rightful 1 before.

This update fixes the issue. Other models may be affected.

Ooh, base model must have misconfigured that. Thanks for the fix!

TheBloke changed pull request status to merged

Sign up or log in to comment