Special tokens
#90
by
kostyamyasso2002
- opened
Where can I find description of sentinel tokens (I haven't understand them from the paper https://arxiv.org/pdf/2305.06161.pdf). I am interested especially in fim_pad token
The special tokens (like fim_pad) are used for training for the "Fill-in-the-middle" objective (in short FIM).
This paper (https://arxiv.org/pdf/2207.14255.pdf) has more details about the FIM objective.
There's an example in the model's card: https://huggingface.co/bigcode/starcoder#fill-in-the-middle
loubnabnl
changed discussion status to
closed