ValueError: Couldn't instantiate the backend tokenizer from one of
Author: brifin
Pull request: no

### System Info

System: Linux x86_64 GNU/Linux
Python=3.9.18
sentence-transformers    2.2.2
sentencepiece            0.1.99
tokenizers               0.14.1
torch                    2.0.0
torchvision              0.15.1
transformers             4.34.0

"tokenizer_class": "GPTNeoXTokenizer"
  "_name_or_path": "EleutherAI/pythia-1.4b-deduped",
  "architectures": [
    "GPTNeoXForCausalLM"
  ],



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**Relative code in example_submission.py：**
```python
    args = parse_args()
    subtrack = args.subtrack
    phase = args.phase
    baseline = args.baseline
    mode = args.mode
    results_path = args.results_path
    device = args.device
    verbose = args.verbose

    # ========== load the tokenizer and the model ========== #
    tokenizer_path = model_path = f"./data/{phase}/{subtrack}/model"
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, padding_side='left')
    tokenizer.add_special_tokens({'pad_token': '
'})
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16).to(device).eval()
```


**Execute：**
python3 example_submission.py --subtrack base --phase dev --baseline PEZ --mode val

**Result:**
Traceback (most recent call last):
  File "/xxx/example_submission.py", line 205, in <module>
    main()
  File "/xxx/example_submission.py", line 116, in main
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, padding_side='left')
  File "/xxx/anaconda3/envs/trogan/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 751, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/xxx/anaconda3/envs/trogan/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2045, in from_pretrained
    return cls._from_pretrained(
  File "/xxx/anaconda3/envs/trogan/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/xxx/anaconda3/envs/trogan/lib/python3.9/site-packages/transformers/models/gpt_neox/tokenization_gpt_neox_fast.py", line 109, in __init__
    super().__init__(
  File "/xxx/anaconda3/envs/trogan/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 120, in __init__
    raise ValueError(
ValueError: Couldn't instantiate the backend tokenizer from one of: 
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

**Failed struggle：**
I have tried ‘pip install sentencepiece’, but it doesn't work for me.

### Expected behavior

Run successfullly

Comments:
Hey! When you say `it does not work for me` what do you mean ? 

> Hey! When you say `it does not work for me` what do you mean ?

which means the result remained the same😢

I solved it. There was some network problem when downloading the model files, and it made the model broken. I redownloaded it then succeeded. 😂
But the error and advice reported actually don't match the situation I met, hope some beneficial changes to be made.