T5 Tokenizer Prepends Space after Each Added (Extra) Token
Author: ankrgyl
Pull request: no
Labels: name: bug

### System Info

```
$ transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.22.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.10.7
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.0+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@LysandreJik @SaulLu 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('t5-base')
tokenizer.add_tokens(['<']) # '>' is already in the vocab

tokenizer.decode(tokenizer('a>=5').input_ids)
# prints 'a>=5
' as expected (no space after >)
tokenizer.decode(tokenizer('a<=5').input_ids)
# prints 'a< =5
'

### Expected behavior

There shouldn't be a space after the `<` character.

Comments:
In case it helps with debugging, it returns a different result with `use_fast=False`:

```
tokenizer = AutoTokenizer.from_pretrained('t5-base', use_fast=False)
tokenizer.add_tokens(['<'])
tokenizer.decode(tokenizer('a<=5').input_ids)
# 'a < =5
' (notice the space both before and after the `<`)

Maybe also of interest to @ArthurZucker 

I did some digging, and I believe the culprit (in the non-fast, but potentially fast, tokenizer) is this:
https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L433

```
# Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert)
if special_tokens:
    if len(new_tokens) == 1:
        _insert_one_token_to_ordered_list(self.unique_no_split_tokens, new_tokens[0])
    else:
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens)))
else:
    # Or on the newly added tokens
    if len(tokens_to_add) == 1:
        _insert_one_token_to_ordered_list(self.unique_no_split_tokens, tokens_to_add[0])
    else:
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add)))
self._create_trie(self.unique_no_split_tokens)
```

Do you know if that's a fundamental restriction with respect to the models? Or if not, could we potentially expose a flag that disables this behavior?

@ankrgyl  What is the expected behaviour of decode should be ? 

My desired behavior would be that `a<=5` round trips (i.e. encodes and then decodes) to `a<=5`, not `a < =5`

@LysandreJik  @ArthurZucker I debugged this issue. 

Incase of of input is a a>=5
calling self.tokens_trie.split() returns ['a>=5']

Incase of of input is a a<=5
calling self.tokens_trie.split() returns ['a', '<', '=5']

Here '<' is not part of the original vocab is added. 

Hi @raghavanone just to clarify, that has been known the whole time. The specific issue here is that added tokens (in this case `<`) _cannot_ split words (see the code snippet I pasted above).

@ankrgyl Did some more digging, this is being done intentionally when we add new token to tokenizer. Look at _decode function in tokenization_utils.py . I do not think this a bug. 

@ArthurZucker Please validate this understanding.

Hey! @raghavanone you are right, this is not a bug! 
@ankrgyl you can use the following snippet : 
```python 
>>> tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
>>> 'a<=5
'
``` 
(this works for the slow tokenizer, not for the fast, will have a look) tell me if that fixes your issue! 😄 

@ArthurZucker  Just pondering why is slow and fast tokenizer functionally not equal ?

Well.. this is not really intended ^^ But mostly the `fast` is an entire library mostly implemented in `rust`, so we must have forgotten to update this argument when adding it to the `transformers` tokenizers. cc @LysandreJik  and @SaulLu FYI 🤗 

@ArthurZucker confirmed with both the slow and fast tokenizers (`build_tokenizer()` below is a wrapper function in my code that simply adds `<` as a special token):

```
In [2]: tokenizer = model.build_tokenizer()

In [3]: tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
Out[3]: 'a< =5
'

In [4]: tokenizer = model.build_tokenizer(use_fast=False)

In [5]: tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
Out[5]: 'a<=5
'
```

Awesome, closing this issue, will open a PR in tokenizers when I have bandwith to try to match the outputs. 

@ArthurZucker happy to help with this