Missing special tokens
#4
by
narphorium
- opened
Could you please upload the tokenizer with the <NME>, <BEF>, <MSG> and <DFF> tokens so that they can be used in inference.
Could you please upload the tokenizer with the <NME>, <BEF>, <MSG> and <DFF> tokens so that they can be used in inference.
There were some discussions and in the end we didn't add them as special tokens. They are a combination of a few tokens and it still seems to be reasonably capable of recognizing/using them in the prompt completion.
But this is really just a preliminary model and we are currently doing more training and testing.