Why not use DeciCoder tokenizers?
#10
by
pvelosipednikov
- opened
In this example notebook (https://colab.research.google.com/drive/1JCxvBsWCZKHfIcHSMVf7GZCs3ClMQPjs), you use the StarCoder tokenizers. I understand that DeciCoder was trained on a subset of the Starcoder Training dataset. Is the advice not to use a DeciCoder tokenizer and if so, why?
Hi @pvelosipednikov it's fixed now.
harpreetsahota
changed discussion status to
closed