Missing "eos_token" key in tokenizer_config.json
#8
by
clyang
- opened
By comparing with xgen-7b-8k, I found that there's a missing key eos_token in tokenizer_config.json. Will this affect my current fine-tuned model?
Thanks, added it here too. But the default value was already set in the tokenizer file, I don't think it had affected.
Either way, I would appreciate it if you could update the locally cached files. Thanks for noticing this!
rooa
changed discussion status to
closed