Can not load Llama3's tokenizer correctly with sentencepiece installed
Author: smartliuhw
Pull request: no
Labels: name: dependencies

name: bug

### System Info

transformers--4.41.2

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Try to load Llama3's tokenizer and got the message. I have installed sentencepiece and tried to reinstall it, how can I solve this problem?
![image](https://github.com/user-attachments/assets/f6fb52bb-cc33-4c8f-8760-92b3f31ea840)


### Expected behavior

load the tokenizer correctly

Comments:
Is it possible to share the model file @smartliuhw ? If possible, I would like to replicate the error if I encounter.

Sure! I download the model from the [Llama3-8B's huggingface site](https://huggingface.co/meta-llama/Meta-Llama-3-8B). The following are my local files related to tokenizer:
[tokenizer_config.json](https://github.com/user-attachments/files/17350637/tokenizer_config.json)
[tokenizer.json](https://github.com/user-attachments/files/17350638/tokenizer.json)
[special_tokens_map.json](https://github.com/user-attachments/files/17350645/special_tokens_map.json)

Besides, these are my installed packages
```
Package                  Version
------------------------ -----------
absl-py                  2.1.0
accelerate               0.30.1
aiohttp                  3.9.5
aiosignal                1.3.1
annotated-types          0.7.0
async-timeout            4.0.3
attrs                    23.2.0
certifi                  2024.6.2
charset-normalizer       3.3.2
click                    8.1.7
datasets                 2.19.2
deepspeed                0.14.2
dill                     0.3.8
docker-pycreds           0.4.0
docstring_parser         0.16
einops                   0.8.0
filelock                 3.14.0
flash-attn               2.5.9.post1
frozenlist               1.4.1
fsspec                   2024.3.1
gitdb                    4.0.11
GitPython                3.1.43
grpcio                   1.64.1
hjson                    3.1.0
huggingface-hub          0.23.2
idna                     3.7
Jinja2                   3.1.4
Markdown                 3.6
markdown-it-py           3.0.0
MarkupSafe               2.1.5
mdurl                    0.1.2
mpmath                   1.3.0
multidict                6.0.5
multiprocess             0.70.16
networkx                 3.3
ninja                    1.11.1.1
numpy                    1.26.4
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu12   12.1.105
nvidia-cuda-nvrtc-cu12   12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu12        11.0.2.54
nvidia-curand-cu12       10.3.2.106
nvidia-cusolver-cu12     11.4.5.107
nvidia-cusparse-cu12     12.1.0.106
nvidia-nccl-cu12         2.20.5
nvidia-nvjitlink-cu12    12.5.40
nvidia-nvtx-cu12         12.1.105
packaging                24.0
pandas                   2.2.2
pip                      24.0
platformdirs             4.2.2
protobuf                 5.28.2
psutil                   5.9.8
py-cpuinfo               9.0.0
pyarrow                  16.1.0
pyarrow-hotfix           0.6
pydantic                 2.7.3
pydantic_core            2.18.4
Pygments                 2.18.0
pynvml                   11.5.0
python-dateutil          2.9.0.post0
pytz                     2024.1
PyYAML                   6.0.1
regex                    2024.5.15
requests                 2.32.3
rich                     13.7.1
safetensors              0.4.5
sentencepiece            0.2.0
sentry-sdk               2.4.0
setproctitle             1.3.3
setuptools               69.5.1
shtab                    1.7.1
six                      1.16.0
smmap                    5.0.1
sympy                    1.12.1
tensorboard              2.16.2
tensorboard-data-server  0.7.2
tokenizers               0.19.1
torch                    2.3.0
tqdm                     4.66.4
transformers             4.41.2
triton                   2.3.0
trl                      0.8.6
typing_extensions        4.12.1
tyro                     0.8.4
tzdata                   2024.1
unsloth                  2024.5
urllib3                  2.2.1
wandb                    0.18.3
Werkzeug                 3.0.3
wheel                    0.43.0
xxhash                   3.4.1
yarl                     1.9.4
zstandard                0.23.0
```


Hey! 
Not sure I understand what `transformers--4.41.2` means, but try upgrading! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.