new T5 tokenisation unexpected behaviour immediately after added special tokens
Author: mickeymickeymonkey
Pull request: no

### System Info

System Info

    transformers version: 4.33.1
    Python version: 3.8.16
    PyTorch version (GPU?): 2.0.1+cu118


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer1 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=True)
tokenizer2 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=False)
add_tokens = ['[EXT1]', '[test_test]']
words = ['terminal', 'many', 'sense'] 
texts = [ '{}', '[EXT1] {} [test_test]', '[EXT1] {} [test_test] {}', '[EXT1] {} [test_test] {} {}', '[EXT1] {}', '{} [EXT1]', '{} [EXT1] {}']
tokenizer1.add_tokens(add_tokens, special_tokens=True)
tokenizer2.add_tokens(add_tokens, special_tokens=True)
for xx, tok in [('Legacy', tokenizer1), ('New', tokenizer2)]:
    print(xx + ' tokenizer')
    for __text in texts:
        for word in words:
            try: text = __text.format(word)
            except: text = __text.format(word, word, word)
            else: text = __text.format(word, word)
            print(f'"{text}" -->') 
            print('\t\t', [tok.decode(t) for t in tok.encode(text)])
    print()
```


```
Legacy tokenizer
"terminal" -->
		 ['terminal', '
']
"many" -->
		 ['many', '
']
"sense" -->
		 ['sense', '
']
"[EXT1] terminal [test_test]" -->
		 ['[EXT1]', 'terminal', '[test_test]', '
']
"[EXT1] many [test_test]" -->
		 ['[EXT1]', 'many', '[test_test]', '
']
"[EXT1] sense [test_test]" -->
		 ['[EXT1]', 'sense', '[test_test]', '
']
"[EXT1] terminal [test_test] terminal" -->
		 ['[EXT1]', 'terminal', '[test_test]', 'terminal', '
']
"[EXT1] many [test_test] many" -->
		 ['[EXT1]', 'many', '[test_test]', 'many', '
']
"[EXT1] sense [test_test] sense" -->
		 ['[EXT1]', 'sense', '[test_test]', 'sense', '
']
"[EXT1] terminal [test_test] terminal terminal" -->
		 ['[EXT1]', 'terminal', '[test_test]', 'terminal', 'terminal', '
']
"[EXT1] many [test_test] many many" -->
		 ['[EXT1]', 'many', '[test_test]', 'many', 'many', '
']
"[EXT1] sense [test_test] sense sense" -->
		 ['[EXT1]', 'sense', '[test_test]', 'sense', 'sense', '
']
"[EXT1] terminal" -->
		 ['[EXT1]', 'terminal', '
']
"[EXT1] many" -->
		 ['[EXT1]', 'many', '
']
"[EXT1] sense" -->
		 ['[EXT1]', 'sense', '
']
"terminal [EXT1]" -->
		 ['terminal', '[EXT1]', '
']
"many [EXT1]" -->
		 ['many', '[EXT1]', '
']
"sense [EXT1]" -->
		 ['sense', '[EXT1]', '
']
"terminal [EXT1] terminal" -->
		 ['terminal', '[EXT1]', 'terminal', '
']
"many [EXT1] many" -->
		 ['many', '[EXT1]', 'many', '
']
"sense [EXT1] sense" -->
		 ['sense', '[EXT1]', 'sense', '
']

New tokenizer
"terminal" -->
		 ['terminal', '
']
"many" -->
		 ['many', '
']
"sense" -->
		 ['sense', '
']
"[EXT1] terminal [test_test]" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]', '
']
"[EXT1] many [test_test]" -->
		 ['[EXT1]', 'man', 'y', '[test_test]', '
']
"[EXT1] sense [test_test]" -->
		 ['[EXT1]', 's', 'ense', '[test_test]', '
']
"[EXT1] terminal [test_test] terminal" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]', 'termin', 'al', '
']
"[EXT1] many [test_test] many" -->
		 ['[EXT1]', 'man', 'y', '[test_test]', 'man', 'y', '
']
"[EXT1] sense [test_test] sense" -->
		 ['[EXT1]', 's', 'ense', '[test_test]', 's', 'ense', '
']
"[EXT1] terminal [test_test] terminal terminal" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]', 'termin', 'al', 'terminal', '
']
"[EXT1] many [test_test] many many" -->
		 ['[EXT1]', 'man', 'y', '[test_test]', 'man', 'y', 'many', '
']
"[EXT1] sense [test_test] sense sense" -->
		 ['[EXT1]', 's', 'ense', '[test_test]', 's', 'ense', 'sense', '
']
"[EXT1] terminal" -->
		 ['[EXT1]', 'termin', 'al', '
']
"[EXT1] many" -->
		 ['[EXT1]', 'man', 'y', '
']
"[EXT1] sense" -->
		 ['[EXT1]', 's', 'ense', '
']
"terminal [EXT1]" -->
		 ['terminal', '[EXT1]', '
']
"many [EXT1]" -->
		 ['many', '[EXT1]', '
']
"sense [EXT1]" -->
		 ['sense', '[EXT1]', '
']
"terminal [EXT1] terminal" -->
		 ['terminal', '[EXT1]', 'termin', 'al', '
']
"many [EXT1] many" -->
		 ['many', '[EXT1]', 'man', 'y', '
']
"sense [EXT1] sense" -->
		 ['sense', '[EXT1]', 's', 'ense', '
']
```


### Expected behavior

the non-legacy T5 tokenizer has unexpected behaviour for words immediately after added special tokens. these were one full token in the legacy tokenizer. e.g. "many", "terminal", "sense", but now become sequence of subwords. but after the first word following special token, it returns to expected behaviour, see the examples. 

Expected behaviour:
tokenization should be the same for words after special tokens. this new behaviour affects ability to use the pretrained knowledge of the LM.  at least a warning should be added for people who want to use their own special tokens seperators. 

Comments:
Hey! This is expected. If you use `tokenizer.tokenize` instead of encode decode you will have a better understanding of what is happening. The `legacy = False` behave properly. And the tokens are by default stripping left and right. 

<details>
  <summary> A small snippet: </summary>

```python 
In [2]: from transformers import AutoTokenizer
   ...: tokenizer1 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=True)
   ...: tokenizer2 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=False)
   ...: add_tokens = ['[EXT1]', '[test_test]']
   ...: words = ['terminal', 'many', 'sense']
   ...: texts = [ '{}', '[EXT1] {} [test_test]', '[EXT1] {} [test_test] {}', '[EXT1] {} [test_test] {} {}', '[EXT1] {}', '{} [EXT1]', '{} [EXT1] {}']
   ...: tokenizer1.add_tokens(add_tokens, special_tokens=True)
   ...: tokenizer2.add_tokens(add_tokens, special_tokens=True)
   ...: for xx, tok in [('Legacy', tokenizer1), ('New', tokenizer2)]:
   ...:     print(xx + ' tokenizer')
   ...:     for __text in texts:
   ...:         for word in words:
   ...:             try: text = __text.format(word)
   ...:             except: text = __text.format(word, word, word)
   ...:             else: text = __text.format(word, word)
   ...:             print(f'"{text}" -->')
   ...:             print('\t\t', tok.tokenize(text))
   ...:     print()
   ...:
/home/arthur/transformers/src/transformers/models/t5/tokenization_t5.py:237: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on t5-base automatically truncating your input to 512 when padding/encoding.
- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
  warnings.warn(
Legacy tokenizer
"terminal" -->
		 ['▁terminal']
"many" -->
		 ['▁many']
"sense" -->
		 ['▁sense']
"[EXT1] terminal [test_test]" -->
		 ['[EXT1]', '▁terminal', '[test_test]']
"[EXT1] many [test_test]" -->
		 ['[EXT1]', '▁many', '[test_test]']
"[EXT1] sense [test_test]" -->
		 ['[EXT1]', '▁sense', '[test_test]']
"[EXT1] terminal [test_test] terminal" -->
		 ['[EXT1]', '▁terminal', '[test_test]', '▁terminal']
"[EXT1] many [test_test] many" -->
		 ['[EXT1]', '▁many', '[test_test]', '▁many']
"[EXT1] sense [test_test] sense" -->
		 ['[EXT1]', '▁sense', '[test_test]', '▁sense']
"[EXT1] terminal [test_test] terminal terminal" -->
		 ['[EXT1]', '▁terminal', '[test_test]', '▁terminal', '▁terminal']
"[EXT1] many [test_test] many many" -->
		 ['[EXT1]', '▁many', '[test_test]', '▁many', '▁many']
"[EXT1] sense [test_test] sense sense" -->
		 ['[EXT1]', '▁sense', '[test_test]', '▁sense', '▁sense']
"[EXT1] terminal" -->
		 ['[EXT1]', '▁terminal']
"[EXT1] many" -->
		 ['[EXT1]', '▁many']
"[EXT1] sense" -->
		 ['[EXT1]', '▁sense']
"terminal [EXT1]" -->
		 ['▁terminal', '[EXT1]']
"many [EXT1]" -->
		 ['▁many', '[EXT1]']
"sense [EXT1]" -->
		 ['▁sense', '[EXT1]']
"terminal [EXT1] terminal" -->
		 ['▁terminal', '[EXT1]', '▁terminal']
"many [EXT1] many" -->
		 ['▁many', '[EXT1]', '▁many']
"sense [EXT1] sense" -->
		 ['▁sense', '[EXT1]', '▁sense']

New tokenizer
"terminal" -->
		 ['▁terminal']
"many" -->
		 ['▁many']
"sense" -->
		 ['▁sense']
"[EXT1] terminal [test_test]" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]']
"[EXT1] many [test_test]" -->
		 ['[EXT1]', 'man', 'y', '[test_test]']
"[EXT1] sense [test_test]" -->
		 ['[EXT1]', 's', 'ense', '[test_test]']
"[EXT1] terminal [test_test] terminal" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]', 'termin', 'al']
"[EXT1] many [test_test] many" -->
		 ['[EXT1]', 'man', 'y', '[test_test]', 'man', 'y']
"[EXT1] sense [test_test] sense" -->
		 ['[EXT1]', 's', 'ense', '[test_test]', 's', 'ense']
"[EXT1] terminal [test_test] terminal terminal" -->
		 ['[EXT1]', 'termin', 'al', '[test_test]', 'termin', 'al', '▁terminal']
"[EXT1] many [test_test] many many" -->
		 ['[EXT1]', 'man', 'y', '[test_test]', 'man', 'y', '▁many']
"[EXT1] sense [test_test] sense sense" -->
		 ['[EXT1]', 's', 'ense', '[test_test]', 's', 'ense', '▁sense']
"[EXT1] terminal" -->
		 ['[EXT1]', 'termin', 'al']
"[EXT1] many" -->
		 ['[EXT1]', 'man', 'y']
"[EXT1] sense" -->
		 ['[EXT1]', 's', 'ense']
"terminal [EXT1]" -->
		 ['▁terminal', '[EXT1]']
"many [EXT1]" -->
		 ['▁many', '[EXT1]']
"sense [EXT1]" -->
		 ['▁sense', '[EXT1]']
"terminal [EXT1] terminal" -->
		 ['▁terminal', '[EXT1]', 'termin', 'al']
"many [EXT1] many" -->
		 ['▁many', '[EXT1]', 'man', 'y']
"sense [EXT1] sense" -->
		 ['▁sense', '[EXT1]', 's', 'ense']
```
</details>

Really inviting you to read the documentation on what is happening here! 🤗 
A fix? 
```python 
from transformers import AutoTokenizer, AddedToken
tokenizer1 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=True)
tokenizer2 = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=False)
add_tokens = [AddedToken('[EXT1]', rstrip = False, lstrip = False), AddedToken('[test_test]', rstrip = False, rstrip= False)]
tokenizer1.add_tokens(add_tokens, special_tokens=True)
tokenizer2.add_tokens(add_tokens, special_tokens=True)
```

ok this is fixed. you are right that the new T5 tokenisation works as expected. the problem is having sentencepiece installed at the same time. 

your fix did not work for me at first, but when I tried on another environment it was ok. after looking, I found out that the 2nd environment where it worked did not have sentencepiece package installed. 

when I installed sentencepiece it asked for protobuf installation, and then the unexpected behaviour happened again. 

in summary: to solve i had to make sure I did not have sentencepiece installed. 

That is very strange. If you do not have sentencepiece, you cannot use the `slow ` tokenizers. If you use the `fast` tokenizer, you will still have the issue (it's expected for now). 
`use_fast = False` would only work if you have sentencepiece, and the `legacy = False` is also only for sentencepiece. 

you're right. I did not notice that. without sentencepiece, the use_fast = False argument is ignored and the fast tokenizer is loaded. 

the AddedToken method is not working for me. it is a good suggestion and seems to be logical way to get such control if this behaviour for special tokens is desired. 

it seems to be because in tokenization_utils.py the Trie's split method returns the word coming immediately after added special tokens without the "▁" already, so the AddedToken check for rstrip/lstrip does not hit. Here's what I mean, e.g.  

```
561                   tokens = self.tokens_trie.split(text)
562                  # added
563                  print('\t\t Text:', text)
564                  print('\t\t Tokens:', tokens)
``` 

```
Input: "terminal [EXT1] terminal" -->
		Text: ▁terminal [EXT1] terminal
		Tokens: ['▁terminal ', '[EXT1]', ' terminal']

Input: "many [EXT1] many" -->
		Text: ▁many [EXT1] many
		Tokens: ['▁many ', '[EXT1]', ' many']

Input: "sense [EXT1] sense" -->
		Text: ▁sense [EXT1] sense
		Tokens: ['▁sense ', '[EXT1]', ' sense']
```



No when you use a fast tokenizer this tire is not used and the stripping / splitting logic is done in rust!  But this issue was fixed by #26538 !

(Note that T5 will always tokenize `'▁'` as `''` so you won't get the space before exist even if you do not strip

ok I will install from source and try it. thank you for the explanations and the work on the tokenizer, you're amazing! 🥳

🤗 thanks for bearing with the changes!

no worries, great work 🤗. just tested with installation from source and it works for me (with slow, and legacy=False). thank you! 🙏