Detokenization discrepancy with Llama3.1
Author: AbrahamSanders
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.12.7
- Huggingface_hub version: 0.26.5
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: N/A

### Who can help?

@ArthurZucker @itazap


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Spaces are being stripped from space-prefixed token `Ġ'` when followed by a common abbreviation (e.g., `n't`, `'m`, `'s`, `'ve`), even when not appropriate to do so. This is being caused because `clean_up_tokenization_spaces` is True by default for the Llama 3.1 tokenizer.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

original = " plunged the long 'sword' into"
input_ids = tokenizer.encode(original, add_special_tokens=False)
tokens = tokenizer.convert_ids_to_tokens(input_ids)
decoded = tokenizer.decode(input_ids)
decoded2 = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)

print("token ids:                ", input_ids)
print("tokens:                   ", tokens)
print("original:                ", original)
print("decoded (default):       ", decoded)
print("decoded (clean_up=False):", decoded2)
```

Produces
```text
token ids:                 [75803, 279, 1317, 364, 80138, 6, 1139]
tokens:                    ['Ġplunged', 'Ġthe', 'Ġlong', "Ġ'", 'sword', "'", 'Ġinto']
original:                  plunged the long 'sword' into
decoded (default):         plunged the long'sword' into
decoded (clean_up=False):  plunged the long 'sword' into
```

### Expected behavior

I would expect the `original` string to match the `decoded` string in all cases unless it actually contains "traditional" tokenization spacing (e.g., `it 's` vs `it's`). Perhaps a good approach could be to modify the [clean_up_tokenization](https://github.com/huggingface/transformers/blob/5d7739f15a6e50de416977fe2cc9cb516d67edda/src/transformers/tokenization_utils_base.py#L3909) function to only apply this rule when the common abbreviation is followed immediately by another space.

Comments:
As you mentioned, `clean_up_tokenization_spaces` is set to `True`. It should be set to `False` ! 

Is it a problem to be fixed in the code in master or should we set it manually everytime?

@ArthurZucker yes, same question as @denadai2. 

It seems counterintuitive for the tokenizer's default to be `True`, since most applications won't be using pre-tokenized texts. I think a good default would be `False` in the tokenizer config, and then anyone working with pre-tokenized text can set it to `True` manually in their `decode()` call. Alternatively, `clean_up_tokenization` could be modified as I suggested previously. Any thoughts?

It IS counter intuitive, but we can't easily break stuff in `transformers` this has been `True` for models online for example. 
#31938 made is `False` by default if unset, so `Llama` should not have it to False. But here the model has `"clean_up_tokenization_spaces": true,` in the config. Let's open a pr to change this!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.