Llama3 Tokenizer Decode Removing Space Character
Author: jonathanasdf
Pull request: no
Labels: name: Core: Tokenization

name: bug

### System Info

- `transformers` version: 4.44.0
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.3
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
>>> import transformers
>>> tok = transformers.AutoTokenizer.from_pretrained('baseten/Meta-Llama-3-tokenizer')
>>> tok.decode([1232, 364])
"': '"
>>> tok.decode([364, 1874])
"'search"
>>> tok.decode([1232, 364, 1874])
"':'search"
```

### Expected behavior

Output should be `': 'search`; the space between the colon and quote character should be kept

Comments:
The following code caused this bug:

``       out_string = (
            out_string.replace(" .", ".")
            .replace(" ?", "?")
            .replace(" !", "!")
            .replace(" ,", ",")
            .replace(" ' ", "'")
            .replace(" n't", "n't")
            .replace(" 'm", "'m")
            .replace(" 's", "'s")
            .replace(" 've", "'ve")
            .replace(" 're", "'re")
        )
        return out_string 
``
This code is located in the file transformers\transformers\src\transformers\tokenization_utils_base.py in line 4075
This error can also be reproduced if any word starts with these subwords: m, s, ve, re. 
For example:
`tok.decode([1232, 364, 23129])`
result:
`':'verb`

You basically just have to set `cleanup_tokenization_spaces = False`

```
>>> import transformers
>>> tok = transformers.AutoTokenizer.from_pretrained('baseten/Meta-Llama-3-tokenizer')
>>> tok.decode([1232, 364], cleanup_tokenization_spaces=False)
"': '"
>>> tok.decode([364, 1874], cleanup_tokenization_spaces=False)
"'search"
>>> tok.decode([1232, 364, 1874], cleanup_tokenization_spaces=False)
"':'search"
```

Sorry it's `clean_up_tokenization_spaces`: 
```python
>>> import transformers
>>> tok = transformers.AutoTokenizer.from_pretrained('baseten/Meta-Llama-3-tokenizer')
>>> tok.decode([1232, 364, 1874], clean_up_tokenization_spaces=False)
 "': 'search"
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, which version of transformers has this `clean_up_tokenization_spaces` feature for LLama3 class models?? (it seems to be a no-op on transformers-4.43.1

(possibly, that this issue is not fully addressed by `clean_up_tokenization_spaces` )

For example
```
import llama_models.llama3.api.tokenizer as L3Tok
tk3 = AutoTokenizer.from_pretrained("unsloth/llama-3-8b")
tk4 =  L3Tok.Tokenizer('Meta-Llama-3-8B-Instruct/tokenizer.model')
tk3.decode([600, 662], cleanup_tokenization_spaces=False), tk4.decode([600, 662])
```
results in 
`('unction.', 'unction .')` which seems to be, if not a bug, then a really not well documented feature.

Thanks for reading! @ArthurZucker @jonathanasdf @YOUSEFNANIS 

There is a type, you are looking for `clean_up_tokenization_spaces=False`:
```diff
import llama_models.llama3.api.tokenizer as L3Tok
tk3 = AutoTokenizer.from_pretrained("unsloth/llama-3-8b")
tk4 =  L3Tok.Tokenizer('Meta-Llama-3-8B-Instruct/tokenizer.model')
-tk3.decode([600, 662], cleanup_tokenization_spaces=False), tk4.decode([600, 662])
+tk3.decode([600, 662], clean_up_tokenization_spaces=False), tk4.decode([600, 662])

```

oops, you are right @ArthurZucker Thank you.