Tokenizer class GPT3 5 Tokenizer does not exist or is not currently imported.
#2
by
GozaloMoreno
- opened
Hi: I am using your tokenizer to avoid tiktoken because I don't have permission to do it. However I always get this warning: "Tokenizer class GPT3 5 Tokenizer does not exist or is not currently imported."
How can I fix it?
Thanks
Gonzalo Moreno
Hi there. You can ignore the warning, because the tokenizer class does not exist in Transformers. Fortunately, it still works because it has a very similar format to existing tokenizers.