The Phi-3 tokenizer is not inverting the chat template correctly.
Author: tbenthompson
Pull request: no

### System Info

transformers version: 4.40.0

### Who can help?

@ArthurZucker

### Reproduction

```
import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct"
)
S = '
\n
\n'
ids = tokenizer.encode(S, add_special_tokens=False)
S_decode = tokenizer.decode(ids)
print(repr(S))
print(repr(S_decode))
```

```
'
\n
\n'
'
 \n
 \n'
```

### Expected behavior

I would expect the decoded string to exactly match the encoded string. This was working correctly as of last week so it likely involves recent updates to the phi-3 tokenizer.

Thanks!!

Comments:
cc @Rocketknight1 

This is a tokenizers issue, not a chat template issue! I can't see what would cause this, though - you can see the full history of updates to Phi-3 [here](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/commits/main).

In general, decoding spaces around special tokens like `
` can be ambiguous. This is because our tokenizers will generally encode a special token with a space as just the single special token, and then add a space when decoding. This results in an incorrect decoding when there was no space in the first place.

My guess is that if the behaviour of Phi-3 changed, it is because `
` is now correctly recognized as a special token. Unfortunately, I don't think we can guarantee perfect decoding of spaces around special tokens - this is just something that's inherent to how we tokenize/detokenize strings!

Ok! If this isn't a bug, feel free to close the issue. I will say that Phi-3 is the only model I've worked with where the chat template fails the encode/decode inversion test above. That's why I reported this - I have code that I use with quite a few other models that threw errors when run on Phi-3. It's no big deal to work around though.  

That is expected it seems:
```python 
        32000: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        32001: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32002: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32003: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32004: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32005: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32006: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32007: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32008: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32009: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32010: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
```
the special tokens are added with `rstrip=True`

Closing as expected!