THUDM

/

visualglm-6b

like 207

After downloading the model via LFS, I changed the path to local according to the demo in the README, but the following error occurs when loading the Tokenizer:
'''
Traceback (most recent call last):
File "demo.py", line 4, in
tokenizer = AutoTokenizer.from_pretrained(ModelPATH, trust_remote_code=True)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 738, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2017, in from_pretrained
return cls._from_pretrained(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2249, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/visualglm-6b/tokenization_chatglm.py", line 196, in init
super().init(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 367, in init
self._add_tokens(
File "/root/miniconda3/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 467, in _add_tokens
current_vocab = self.get_vocab().copy()
File "/root/.cache/huggingface/modules/transformers_modules/visualglm-6b/tokenization_chatglm.py", line 248, in get_vocab
vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
File "/root/.cache/huggingface/modules/transformers_modules/visualglm-6b/tokenization_chatglm.py", line 244, in vocab_size
return self.sp_tokenizer.num_tokens
AttributeError: 'ChatGLMTokenizer' object has no attribute 'sp_tokenizer'
'''