Update tokenizer to include [SUFFIX] and [PREFIX] tokens?

by ShiveringSpine - opened May 30

May 30

It looks like codestral had an update to their tokenizer for the fill-in-the-middle functionality. The update now includes the [SUFFIX] and [PREFIX]: https://huggingface.co/mistralai/Codestral-22B-v0.1/discussions/10

Owner May 30

thanks for this, will look now

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment