add_tokens does not preserve spacing
Author: denizyuret
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.6
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
>>> from transformers import AutoTokenizer
>>> mtok = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1')
>>> str = 'Arthur met Deniz today.'
>>> mtok.decode(mtok.encode(str))
'
 Arthur met Deniz today.'
>>> mtok.add_tokens(['Deniz'])
1
>>> mtok.decode(mtok.encode(str))
'
 Arthur metDeniz today.'
```

### Expected behavior

Spaces shoud be preserved when a text is encoded/decoded.

Comments:
Few things can come into play. By default the token will be normalized, this means that the normalizer will be adding a prefix space to that token. When decoding, that space is removed. You should add the token using `tokenizer.add_tokens(AddedToken("Deniz", normalized = False, special=False))`

This time there is an extra space that appears after the token:
```
>>> from transformers import AddedToken, AutoTokenizer
>>> mtok = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1')
>>> mtok.add_tokens(AddedToken('Deniz', normalized=False, special=False))
1
>>> str = 'Arthur met Deniz today.'
>>> mtok.decode(mtok.encode(str))
'
 Arthur met Deniz  today.'
```

Here are the individual tokens if helps:
```
>>> mtok.convert_ids_to_tokens(mtok.encode(str))
['
', '▁Arthur', '▁met', '▁', 'Deniz', '▁', '▁today', '.']
```

That is also expected, you should either use a slow tokenizer (`use_fast = False`) or follow the fix that is here: #26678. It's a known issue of the normalizer. 

`use_fast=False` has an interesting behavior:
```
>>> mtok = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1', use_fast=False)
>>> mtok.add_tokens([AddedToken('Deniz', normalized=False, special=False)])
>>> mtok.decode(mtok.encode('Arthur met Deniz today.'))
'
Arthur met  Deniz today.'
>>> mtok.convert_ids_to_tokens(mtok.encode('Arthur met Deniz today.'))
['
', '▁Arthur', '▁met', '▁', 'Deniz', '▁', '▁today', '.']
```
i.e. `decode` puts extra space before the added token, however `convert_ids_to_tokens` shows an extra space after the added token.

I tried adding ' Deniz' or '▁Deniz' thinking it would better match the behavior of regular tokens like '▁Arthur' but no success.

So far I haven't been able to find a combination of options that will preserve spacing. Will check out your fix next.

This is the combination that works:
```python 
In [44]: mtok = AutoTokenizer.from_pretrained('mistralai/Mistral-7B-v0.1', use_fast=False, legacy = False)

In [45]: mtok.tokenize('Arthur met Deniz today.')
Out[45]: ['▁Arthur', '▁met', '▁Den', 'iz', '▁today', '.']

In [46]: mtok.add_tokens([AddedToken('Deniz', normalized=False, special=False)])
Out[46]: 1

In [47]: mtok.tokenize('Arthur met Deniz today.')
Out[47]: ['▁Arthur', '▁met', '▁', 'Deniz', '▁today', '.']
```


`mtok.decode` still throws in some extra spaces, but this works for model training etc. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.