load from local filepath error
#2
by
bdambrosio
- opened
The model loads fine, but when I try to load the tokenizer: transformers.from_pretrained( 'local filepath')
File "/home/bruce/.local/lib/python3.10/site-packages/transformers/models/qwen2/tokenization_qwen2.py", line 179, in init
with open(merges_file, encoding="utf-8") as merges_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
btw, using transformers 4.41.1
I'm not sure what is transformers.from_pretrained( 'local filepath'), but the following should be fine:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("local filepath/")