abeja/gpt-neox-japanese-2.7b · Can not load the tokenizer with newest transformers

Sep 17, 2022

Traceback (most recent call last):
  File "train.py", line 188, in <module>
    main()
  File "train.py", line 134, in main
    tokenizer = AutoTokenizer.from_pretrained("abeja/gpt-neox-japanese-2.7b")
  File "/local/10511020.1.gpua/work/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 535, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/local/10511020.1.gpua/work/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 725, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/local/10511020.1.gpua/work/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 432, in __getitem__
    raise KeyError(key)
KeyError: 'gpt_neox_japanese'

SO0529

ABEJA, Inc. org Sep 20, 2022

•

edited Sep 20, 2022

@conan1024hao
Thank you for raising an issue. Did you install by pip install git+https://github.com/huggingface/transformers like this colab?

conan1024hao

Sep 22, 2022

@SO0529
I didn't do that, thank you for the response.

conan1024hao changed discussion status to closed Sep 22, 2022