UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position
Author: PheelaV
Pull request: no

### System Info

Hiya, I am trying to run some pre-trained models that are using this tokenizer as a defualt and I have not had any luck trying to get it going. Is it me or is there more to it? I am too new to transformers to say this is a bug, but given the tests I have done it does seem to be at least inconsistent. It would be great to get either a confirmation or pointers, thank you!

transformers 4.38
python 3.12
AMD64/Ubuntu 22.04
<details><summary>Details</summary>
<p>

```
- `transformers` version: 4.38.0
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.12.2
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

</p>
</details> 

I am normally trying to run this on Python 3.10, just for the sake of sanity I made this minimal repro:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
```

Which produces:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/filip/miniforge3/envs/test/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 825, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/filip/miniforge3/envs/test/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2048, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/filip/miniforge3/envs/test/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2268, in _from_pretrained
    tokenizer_file_handle = json.load(tokenizer_file_handle)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/filip/miniforge3/envs/test/lib/python3.12/json/__init__.py", line 293, in load
    return loads(fp.read(),
                 ^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 196614: invalid start byte
```

Also tested on MacOS Sonoma 14.2/ARM64/M3 where it does work.
<details><summary>Details</summary>
<p>

```
- `transformers` version: 4.28.0.dev0
- Platform: macOS-14.2-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.20.1
- PyTorch version (GPU?): 2.1.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

</p>
</details> 

And so does it for a gpu node on an academic cluster:

<details><summary>Details</summary>
<p>

```
- `transformers` version: 4.32.1
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

</p>
</details> 




### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. install python and transformers into an environment (I am using conda)
2. execute the minimal repro script above

### Expected behavior

succesfully loaded tokenizer

Comments:
Issue solved, likely a corrupted download. Needed to remove cache:
```
rm -rf ~/.cache/huggingface/hub/models--EleutherAI--gpt-neox-20b/
```