ValueError: Tokenizer class GPTNeoXTokenizer does not exist or is not currently imported.
I tried to use it, but got this exception:
2023-10-15 08:52:03 INFO:Loading rwkv-4-430m-pile...
2023-10-15 08:52:29 ERROR:Failed to load the model.
Traceback (most recent call last):
File "/run/media/knut/HD/text-generation-webui/modules/ui_model_menu.py", line 201, in load_model_wrapper
shared.model, shared.tokenizer = load_model(shared.model_name, loader)
File "/run/media/knut/HD/text-generation-webui/modules/models.py", line 87, in load_model
tokenizer = load_tokenizer(model_name, model)
File "/run/media/knut/HD/text-generation-webui/modules/models.py", line 106, in load_tokenizer
tokenizer = AutoTokenizer.from_pretrained(
File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 749, in from_pretrained
raise ValueError(
ValueError: Tokenizer class GPTNeoXTokenizer does not exist or is not currently imported.
try:
I am quite curious, it should use a custom tokenizer (RWKVWorldTokenizer) here, why would it call GPTNeoXTokenizer?
I can load this model with this adjustment. I get another exception when trying inference but I don't know if it is related to the text-gen-webui or something else. I get the same error with RWKV-5-world-167m after downgrading to transformers-4.33.0. I can load that model, too with that version.
I don't know what's going on. perhaps it doesnt use the custom tokenizer?
Traceback (most recent call last):
File "/run/media/knut/HD/text-generation-webui/modules/callbacks.py", line 56, in gentask
ret = self.mfunc(callback=_callback, *args, **self.kwargs)
File "/run/media/knut/HD/text-generation-webui/modules/text_generation.py", line 349, in generate_with_callback
shared.model.generate(**kwargs)
File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/knut/miniconda3/envs/textgen/lib/python3.10/site-packages/transformers/generation/utils.py", line 1443, in generate
eos_token_id = eos_token_id[0]
IndexError: list index out of range
It has been fixed now!
For subsequent questions, please go directly to the RWKV community, my personal page will not upload newly developed and converted RWKV series models anymore.