CarperAI

/

diff-codegen-350m-v1

like 13

Could you please upload the tokenizer with the <NME>, <BEF>, <MSG> and <DFF> tokens so that they can be used in inference.

There were some discussions and in the end we didn't add them as special tokens. They are a combination of a few tokens and it still seems to be reasonably capable of recognizing/using them in the prompt completion.
But this is really just a preliminary model and we are currently doing more training and testing.