Tokenizer isn't available.

#4
by Jaewoo1 - opened

When I load the tokenizer using the code tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-MT/polylm-1.7b") an error occurs like below.
RuntimeError: Llama is supposed to be a BPE model!

Machine Translation Team at Alibaba DAMO Academy org

You can load the tokenizer with use_fast=False, i.e. tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-MT/polylm-1.7b", use_fast=False).
I have tried it and no error occurs.

image.png

Sign up or log in to comment