LLaMATokenizer renders this incompatible with huggingfaces transformers pipeline

by Alignment-Lab-AI - opened Jun 13, 2023

Jun 13, 2023

in transformers the tokenizer is listed as LlamaTokenizer
attempting to run this through any transformers based system results in an error, rendering it incompatible unless the user downloads it and renames it themselves.

James-WYang

Owner Jun 15, 2023

•

edited Jun 15, 2023

Thanks for your suggestion!
We construct the LLaMA based on the code in the BigTrans, so we import tokenizer as llama.LLaMATokenizer, which may conflict with transformers. Therefore, we recommend downloading the code in the BigTrans and using BigTrans model and tokenizer.

Alignment-Lab-AI

Jun 17, 2023

•

edited Jun 17, 2023

i believe attempting to even download the model from huggingface throws the error, its been a bit over a week since i involved BigTrans in my research but i believe i had to wget each file link to retrieve it

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment