Tokenizer isn't available.
#4
by
Jaewoo1
- opened
When I load the tokenizer using the code tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-MT/polylm-1.7b") an error occurs like below.
RuntimeError: Llama is supposed to be a BPE model!
You can load the tokenizer with use_fast=False, i.e. tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-MT/polylm-1.7b", use_fast=False).
I have tried it and no error occurs.
Thanks!