T5Tokenizer from_pretrained Error
Author: TristynAlxander
Pull request: no

### System Info


Operating System: Linux Mint 21.1
Kernel: Linux 6.2.0-36-generic

pip list:

    Package            Version
    ------------------ ------------
    accelerate         0.24.1
    certifi            2023.7.22
    charset-normalizer 3.3.1
    filelock           3.13.1
    fsspec             2023.10.0
    huggingface-hub    0.17.3
    idna               3.4
    Jinja2             3.1.2
    MarkupSafe         2.1.2
    mpmath             1.3.0
    networkx           3.0
    numpy              1.26.1
    packaging          23.2
    Pillow             9.3.0
    pip                22.3.1
    psutil             5.9.6
    PyYAML             6.0.1
    regex              2023.10.3
    requests           2.31.0
    safetensors        0.4.0
    sentencepiece      0.1.99
    setuptools         65.5.0
    sympy              1.12
    tokenizers         0.14.1
    torch              2.1.0+cu118
    torchaudio         2.1.0+cu118
    torchvision        0.16.0+cu118
    tqdm               4.66.1
    transformers       4.34.1
    triton             2.1.0
    typing_extensions  4.8.0
    urllib3            2.0.7

Note: Computer was restarted after sentencepiece was installed. No change.



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code: 

    model_path = "/home/username/downloads/large_language_models/flan-t5-large"
    from transformers import T5Tokenizer, T5ForConditionalGeneration
    tokenizer = T5Tokenizer.from_pretrained(model_path)


### Expected behavior


Actual Behavior:

    Traceback (most recent call last):
      File "/home/username/downloads/large_language_models/temp.py", line 3, in <module>
        tokenizer = T5Tokenizer.from_pretrained(model_path)
      File "/opt/.pyenv_root/versions/LLM/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2017, in from_pretrained
        return cls._from_pretrained(
      File "/opt/.pyenv_root/versions/LLM/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2249, in _from_pretrained
        tokenizer = cls(*init_inputs, **init_kwargs)
      File "/opt/.pyenv_root/versions/LLM/lib/python3.10/site-packages/transformers/models/t5/tokenization_t5.py", line 166, in __init__
        self.sp_model.Load(vocab_file)
      File "/opt/.pyenv_root/versions/LLM/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load
        return self.LoadFromFile(model_file)
      File "/opt/.pyenv_root/versions/LLM/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
        return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    TypeError: not a string


Expected Behavior:

Return of T5Tokenizer. No Error. 

Comments:
Hi @TristynAlxander, thanks for raising this issue! 

Could you provide us with a model path or tokenizer files we can use to replicate this error on our end? 

cc @ArthurZucker 

Greetings, 

The above is the simplified version that reproduces the error. 

The `/home/username/downloads/large_language_models/flan-t5-large` file contains the following:

    -rw-rw-r-- 1 user user        797 Nov  1 13:48 config.json
    -rw-rw-r-- 1 user user 3001197502 Nov  1 13:48 pytorch_model.bin
    -rw-rw-r-- 1 user user       2201 Nov  1 13:48 special_tokens_map.json
    -rw-rw-r-- 1 user user      20771 Nov  1 13:48 tokenizer_config.json
    -rw-rw-r-- 1 user user    2422192 Nov  1 13:48 tokenizer.json

Those files were downloaded with the following code: 

    model_path = "/home/username/downloads/large_language_models/flan-t5-large"
    huggingface_path = "google/flan-t5-large"
    device = "cuda" # for GPU usage or "cpu" for CPU usage
    is_dir = os.path.exists(model_path) and os.path.isdir(model_path)
    if( not is_dir ): 
      from transformers import AutoTokenizer,AutoModel
      # Fetch
      tokenizer = AutoTokenizer.from_pretrained(huggingface_path)
      model = AutoModel.from_pretrained(huggingface_path).to(device)
      # Save 
      tokenizer.save_pretrained(model_path)
      model.save_pretrained(model_path)

It didn't occur to me that this portion might have caused the problem, so I apologize if I've messed something up here.

Hey! Seems like you did not download the full content of the repo. The [spiece.model](https://huggingface.co/google/flan-t5-large/blob/main/spiece.model) is missing, which you absolutely need to use the `T5Tokenizer` class. If you are using `AutoTokenizer` the fast files will be fetched, not the slow ones. 

Unless you add `from_slow = True` in the call to from_pretrained 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.