Can't load Llama's tokenizer with add_prefix_space=True parameter.
Author: Lyaaaaaaaaaaaaaaa
Pull request: no
Labels: name: bug

### System Info

I'm using transformers 4.44.0 but here is a full list.
I'm on Pop!_OS 22.04 jammy (x86-64), kernel 6.9.3-76060903-generic
```
  Name                     Version       Build                     Channel    
────────────────────────────────────────────────────────────────────────────────
  _libgcc_mutex            0.1           conda_forge               conda-forge
  _openmp_mutex            4.5           2_kmp_llvm                conda-forge
  accelerate               0.33.0        pyhd8ed1ab_0              conda-forge
  aiohttp                  3.9.5         py39hd1e30aa_0            conda-forge
  aiosignal                1.3.1         pyhd8ed1ab_0              conda-forge
  async-timeout            4.0.3         pyhd8ed1ab_0              conda-forge
  attrs                    24.1.0        pyh71513ae_0              conda-forge
  aws-c-auth               0.7.22        h96bc93b_2                conda-forge
  aws-c-cal                0.6.14        h88a6e22_1                conda-forge
  aws-c-common             0.9.19        h4ab18f5_0                conda-forge
  aws-c-compression        0.2.18        h83b837d_6                conda-forge
  aws-c-event-stream       0.4.2         ha47c788_12               conda-forge
  aws-c-http               0.8.1         h29d6fba_17               conda-forge
  aws-c-io                 0.14.8        h21d4f22_5                conda-forge
  aws-c-mqtt               0.10.4        h759edc4_4                conda-forge
  aws-c-s3                 0.5.9         h594631b_3                conda-forge
  aws-c-sdkutils           0.1.16        h83b837d_2                conda-forge
  aws-checksums            0.1.18        h83b837d_6                conda-forge
  aws-crt-cpp              0.26.9        he3a8b3b_0                conda-forge
  aws-sdk-cpp              1.11.329      hba8bd5f_3                conda-forge
  brotli-python            1.1.0         py39h3d6467e_1            conda-forge
  bzip2                    1.0.8         h4bc722e_7                conda-forge
  c-ares                   1.32.3        h4bc722e_0                conda-forge
  ca-certificates          2024.7.4      hbcca054_0                conda-forge
  certifi                  2024.7.4      pyhd8ed1ab_0              conda-forge
  cffi                     1.16.0        py39h7a31438_0            conda-forge
  charset-normalizer       3.3.2         pyhd8ed1ab_0              conda-forge
  colorama                 0.4.6         pyhd8ed1ab_0              conda-forge
  cuda-cudart              12.4.127      0                         nvidia     
  cuda-cupti               12.4.127      0                         nvidia     
  cuda-libraries           12.4.0        0                         nvidia     
  cuda-nvrtc               12.4.127      0                         nvidia     
  cuda-nvtx                12.4.127      0                         nvidia     
  cuda-opencl              12.4.127      0                         nvidia     
  cuda-runtime             12.4.0        ha804496_0                conda-forge
  cuda-version             11.8          h70ddcb2_3                conda-forge
  cudatoolkit              11.8.0        h4ba93d1_13               conda-forge
  cudnn                    8.9.7.29      hbc23b4c_3                conda-forge
  datasets                 2.20.0        pyhd8ed1ab_0              conda-forge
  dill                     0.3.8         pyhd8ed1ab_0              conda-forge
  filelock                 3.15.4        pyhd8ed1ab_0              conda-forge
  freetype                 2.12.1        h267a509_2                conda-forge
  frozenlist               1.4.1         py39hd1e30aa_0            conda-forge
  fsspec                   2024.5.0      pyhff2d567_0              conda-forge
  gflags                   2.2.2         he1b5a44_1004             conda-forge
  glog                     0.7.1         hbabe93e_0                conda-forge
  gmp                      6.3.0         hac33072_2                conda-forge
  gmpy2                    2.1.5         py39h048c657_1            conda-forge
  h2                       4.1.0         pyhd8ed1ab_0              conda-forge
  hpack                    4.0.0         pyh9f0ad1d_0              conda-forge
  huggingface_hub          0.24.5        pyhd8ed1ab_0              conda-forge
  hyperframe               6.0.1         pyhd8ed1ab_0              conda-forge
  icu                      73.2          h59595ed_0                conda-forge
  idna                     3.7           pyhd8ed1ab_0              conda-forge
  jinja2                   3.1.4         pyhd8ed1ab_0              conda-forge
  keyutils                 1.6.1         h166bdaf_0                conda-forge
  krb5                     1.21.3        h659f571_0                conda-forge
  lcms2                    2.16          hb7c19ff_0                conda-forge
  ld_impl_linux-64         2.40          hf3520f5_7                conda-forge
  lerc                     4.0.0         h27087fc_0                conda-forge
  libabseil                20240116.2    cxx17_he02047a_1          conda-forge
  libarrow                 16.1.0        hcb6531f_6_cpu            conda-forge
  libarrow-acero           16.1.0        hac33072_6_cpu            conda-forge
  libarrow-dataset         16.1.0        hac33072_6_cpu            conda-forge
  libarrow-substrait       16.1.0        h7e0c224_6_cpu            conda-forge
  libblas                  3.9.0         23_linux64_openblas       conda-forge
  libbrotlicommon          1.1.0         hd590300_1                conda-forge
  libbrotlidec             1.1.0         hd590300_1                conda-forge
  libbrotlienc             1.1.0         hd590300_1                conda-forge
  libcblas                 3.9.0         23_linux64_openblas       conda-forge
  libcrc32c                1.1.2         h9c3ff4c_0                conda-forge
  libcublas                12.4.2.65     0                         nvidia     
  libcufft                 11.2.0.44     0                         nvidia     
  libcufile                1.9.1.3       0                         nvidia     
  libcurand                10.3.5.147    0                         nvidia     
  libcurl                  8.8.0         hca28451_1                conda-forge
  libcusolver              11.6.0.99     0                         nvidia     
  libcusparse              12.3.0.142    0                         nvidia     
  libdeflate               1.20          hd590300_0                conda-forge
  libedit                  3.1.20191231  he28a2e2_2                conda-forge
  libev                    4.33          hd590300_2                conda-forge
  libevent                 2.1.12        hf998b51_1                conda-forge
  libffi                   3.4.2         h7f98852_5                conda-forge
  libgcc-ng                14.1.0        h77fa898_0                conda-forge
  libgfortran-ng           14.1.0        h69a702a_0                conda-forge
  libgfortran5             14.1.0        hc5f4f2c_0                conda-forge
  libgoogle-cloud          2.24.0        h2736e30_0                conda-forge
  libgoogle-cloud-storage  2.24.0        h3d9a0c8_0                conda-forge
  libgrpc                  1.62.2        h15f2491_0                conda-forge
  libhwloc                 2.11.1        default_hecaa2ac_1000     conda-forge
  libiconv                 1.17          hd590300_2                conda-forge
  libjpeg-turbo            3.0.0         hd590300_1                conda-forge
  liblapack                3.9.0         23_linux64_openblas       conda-forge
  libmagma                 2.7.2         h09b5827_2                conda-forge
  libmagma_sparse          2.7.2         h09b5827_3                conda-forge
  libnghttp2               1.58.0        h47da74e_1                conda-forge
  libnpp                   12.2.5.2      0                         nvidia     
  libnvfatbin              12.4.127      0                         nvidia     
  libnvjitlink             12.4.99       0                         nvidia     
  libnvjpeg                12.3.1.89     0                         nvidia     
  libopenblas              0.3.27        pthreads_hac2b453_1       conda-forge
  libparquet               16.1.0        h6a7eafb_6_cpu            conda-forge
  libpng                   1.6.43        h2797004_0                conda-forge
  libprotobuf              4.25.3        h08a7969_0                conda-forge
  libre2-11                2023.09.01    h5a48ba9_2                conda-forge
  libsentencepiece         0.2.0         he81a138_2                conda-forge
  libsqlite                3.46.0        hde9e2c9_0                conda-forge
  libssh2                  1.11.0        h0841786_0                conda-forge
  libstdcxx-ng             14.1.0        hc0a3c3a_0                conda-forge
  libthrift                0.19.0        hb90f79a_1                conda-forge
  libtiff                  4.6.0         h1dd3fc0_3                conda-forge
  libtorch                 2.3.1         cuda118_h7aef8b2_300      conda-forge
  libutf8proc              2.8.0         h166bdaf_0                conda-forge
  libuv                    1.48.0        hd590300_0                conda-forge
  libwebp-base             1.4.0         hd590300_0                conda-forge
  libxcb                   1.15          h0b41bf4_0                conda-forge
  libxml2                  2.12.7        hc051c1a_1                conda-forge
  libzlib                  1.2.13        h4ab18f5_6                conda-forge
  llvm-openmp              18.1.7        ha31de31_0                conda-forge
  lz4-c                    1.9.4         hcb278e6_0                conda-forge
  markupsafe               2.1.5         py39hd1e30aa_0            conda-forge
  mkl                      2023.2.0      h84fe81f_50496            conda-forge
  mpc                      1.3.1         hfe3b2da_0                conda-forge
  mpfr                     4.2.1         h38ae2d0_2                conda-forge
  mpmath                   1.3.0         pyhd8ed1ab_0              conda-forge
  multidict                6.0.5         py39hd1e30aa_0            conda-forge
  multiprocess             0.70.16       py39hd1e30aa_0            conda-forge
  nccl                     2.22.3.1      hee583db_1                conda-forge
  ncurses                  6.5           h59595ed_0                conda-forge
  networkx                 3.2.1         pyhd8ed1ab_0              conda-forge
  numpy                    1.26.4        py39h474f0d3_0            conda-forge
  openjpeg                 2.5.2         h488ebb8_0                conda-forge
  openssl                  3.3.1         h4bc722e_2                conda-forge
  orc                      2.0.1         h17fec99_1                conda-forge
  packaging                24.1          pyhd8ed1ab_0              conda-forge
  pandas                   2.2.2         py39hfc16268_1            conda-forge
  pillow                   10.3.0        py39h90c7501_0            conda-forge
  pip                      24.2          pyhd8ed1ab_0              conda-forge
  protobuf                 4.25.3        py39h1be52a0_0            conda-forge
  psutil                   6.0.0         py39hd3abc70_0            conda-forge
  pthread-stubs            0.4           h36c2ea0_1001             conda-forge
  pyarrow                  16.1.0        py39h8003fee_1            conda-forge
  pyarrow-core             16.1.0        py39h176f5a7_1_cpu        conda-forge
  pyarrow-hotfix           0.6           pyhd8ed1ab_0              conda-forge
  pycparser                2.22          pyhd8ed1ab_0              conda-forge
  pysocks                  1.7.1         pyha2e5f31_6              conda-forge
  python                   3.9.7         hf930737_3_cpython        conda-forge
  python-dateutil          2.9.0         pyhd8ed1ab_0              conda-forge
  python-tzdata            2024.1        pyhd8ed1ab_0              conda-forge
  python-xxhash            3.4.1         py39hd1e30aa_0            conda-forge
  python_abi               3.9           4_cp39                    conda-forge
  pytorch                  2.3.1         cuda118_py39hd3e083d_300  conda-forge
  pytorch-cuda             12.4          hc786d27_6                pytorch    
  pytorch-mutex            1.0           cpu                       pytorch    
  pytz                     2024.1        pyhd8ed1ab_0              conda-forge
  pyyaml                   6.0.1         py39hd1e30aa_1            conda-forge
  re2                      2023.09.01    h7f4b329_2                conda-forge
  readline                 8.2           h8228510_1                conda-forge
  regex                    2024.7.24     py39hcd6043d_0            conda-forge
  requests                 2.32.3        pyhd8ed1ab_0              conda-forge
  s2n                      1.4.15        he19d79f_0                conda-forge
  safetensors              0.4.3         py39h9fdd4d6_0            conda-forge
  sentencepiece            0.2.0         hf3d152e_2                conda-forge
  sentencepiece-python     0.2.0         py39h4f2c52d_2            conda-forge
  sentencepiece-spm        0.2.0         he81a138_2                conda-forge
  setuptools               72.1.0        pyhd8ed1ab_0              conda-forge
  six                      1.16.0        pyh6c4a22f_0              conda-forge
  sleef                    3.6.1         h3400bea_1                conda-forge
  snappy                   1.2.1         ha2e4443_0                conda-forge
  sqlite                   3.46.0        h6d4b2fc_0                conda-forge
  sympy                    1.13.0        pypyh2585a3b_103          conda-forge
  tbb                      2021.12.0     h434a139_3                conda-forge
  tk                       8.6.13        noxft_h4845f30_101        conda-forge
  tokenizers               0.19.1        py39hc62d755_0            conda-forge
  torchaudio               2.3.1         py39_cpu                  pytorch    
  torchvision              0.18.1        cuda118py39heff4aec_1     conda-forge
  tqdm                     4.66.5        pyhd8ed1ab_0              conda-forge
  transformers             4.44.0        pyhd8ed1ab_0              conda-forge
  typing-extensions        4.12.2        hd8ed1ab_0                conda-forge
  typing_extensions        4.12.2        pyha770c72_0              conda-forge
  tzdata                   2024a         h0c530f3_0                conda-forge
  urllib3                  2.2.2         pyhd8ed1ab_1              conda-forge
  websockets               12.0          py39hd1e30aa_0            conda-forge
  wheel                    0.44.0        pyhd8ed1ab_0              conda-forge
  xorg-libxau              1.0.11        hd590300_0                conda-forge
  xorg-libxdmcp            1.1.3         h7f98852_0                conda-forge
  xxhash                   0.8.2         hd590300_0                conda-forge
  xz                       5.2.6         h166bdaf_0                conda-forge
  yaml                     0.2.5         h7f98852_2                conda-forge
  yarl                     1.9.4         py39hd1e30aa_0            conda-forge
  zlib                     1.2.13        h4ab18f5_6                conda-forge
  zstandard                0.23.0        py39h623c9ba_0            conda-forge
  zstd                     1.5.6         ha6fb4c9_0                conda-forge
```

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=True)
```
This will raise an error `TypeError: not a string`

### Expected behavior

It should load without error, like other models.

I saw a few closed issues (e.g https://github.com/huggingface/transformers/issues/29625) related to this problem, but it's still happening. 

Comments:
Hi @Lyaaaaaaaaaaaaaaa , thanks for the issue! Unfortunately I can't reproduce on my end, the tokenizer is loaded without raising an error and adds prefix space if asked for. A more detailed stack trace would help -which line is causing this error exactly? What other errors precede it?

You can find the real code here: https://github.com/LyaaaaaGames/AIdventure_Server/blob/main/server/model.py#L308 

But in fact, I tried with a modified version of the Llama model. I saved it in a sharded format. Maybe it could be the reason ??

The model is on the hub at https://huggingface.co/LyaaaaaGames/Llama-2-7b

Ah also the transformers and tokenizers packages should be installed with pip! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @Lyaaaaaaaaaaaaaaa , the same issue occurred for me too and I fixed it. Just in case, if anyone needs the solution, I'm posting it in this issue:
Just add `token=access_token` in `from_pretrained`. `access_token` can be retrieved from one's hugging face account settings.

Cheers!

Closing as @RUFFY-369 gave the solution!

> Hi @Lyaaaaaaaaaaaaaaa , the same issue occurred for me too and I fixed it. Just in case, if anyone needs the solution, I'm posting it in this issue: Just add `token=access_token` in `from_pretrained`. `access_token` can be retrieved from one's hugging face account settings.
> 
> Cheers!

Thanks. I tried with your solution, it works. But then, I tried to reproduce my original error once more, and it didn't happen!
Though, my OS got a few updates since by then. Maybe the updates got rid of the problem.
No clue.