CodeLlamaTokenizer does not exist

#13
by saksham-lamini - opened

Getting this error when I try to load the model using HuggingFace:
ValueError: Tokenizer class CodeLlamaTokenizer does not exist or is not currently imported.

You need to use the transformers from mainline and import it, but anyway I used LlamaTokenizer instead (because the other one complains about naming) and it worked just fine :)

the fix is to uninstall tokenizers, transformers, and accelerate. then rerun pip install git+https://github.com/huggingface/transformers.git@main accelerate

Code Llama org

That's exactly right, you need to install transformers from main as explained by @Choms and @cassanof . This is because the changes to support Code Llama have not been published as part of a pip release yet (they were written last Friday πŸ˜…)

I think there is an active issue being discussed here: https://github.com/huggingface/transformers/issues/22222

Oh thanks @pcuenq for that update.

osanseviero changed discussion status to closed

Sign up or log in to comment