Slow Tokenizer adds whitespace after special token
Author: g588928812
Pull request: no

### System Info

Python 3.10.6
Transformers 4.31.0
<class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>


### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
import transformers

tokenizer = AutoTokenizer.from_pretrained(
	"../models/llama-2-7b",
	use_fast=False,
)

txt="this is one sentence." + tokenizer.eos_token + "this is another sentence." + tokenizer.eos_token + "this is the third sentence." + tokenizer.eos_token

txt_encoded = tokenizer.encode(txt, add_special_tokens=False)
txt_encoded_decoded = tokenizer.decode(txt_encoded)
txt_encoded_decoded_spaces_false = tokenizer.decode(txt_encoded, spaces_between_special_tokens=False)

print(transformers.__version__)
print(tokenizer.__class__)

print(f"INPUT:\n{txt}\n")
print(f"ROUNDTRIP:\n{txt_encoded_decoded}\n")
print(f"ROUNDTRIP w/ spaces_between_special_tokens=F:\n{txt_encoded_decoded}\n")
```

**Output**:
```
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
4.31.0
<class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>
INPUT:
this is one sentence.
this is another sentence.
this is the third sentence.


ROUNDTRIP:
 this is one sentence.
 this is another sentence.
 this is the third sentence.


ROUNDTRIP w/ spaces_between_special_tokens=F:
 this is one sentence.
 this is another sentence.
 this is the third sentence.

```


### Expected behavior

`txt == txt_encoded_decoded`

I expect `text` to be the same as `decode(encode(text))`, however a whitespace is added after each special token (`
`). From what I saw in previous issues, `spaces_between_special_tokens=F` should change that but it does not, whitespaces are still there.

What am I missing?

Thank you for your help and apologies in advance, this issue seems to come up quite often and I spent quite some time going through issues in this repo but nothing solved it for me.


Comments:
Hey! My first suggestion would be to not use the legacy behaviour by setting `legacy = False` when you initialize the tokenizer. 
Second, the `txt_encoded == txt_encoded_decoded` assumption is not always true for all tokenizers. In this case, the decoding adds an extra space, maybe because it is based on the previous legacy behaviour. Will investigate

> My first suggestion would be to not use the legacy behaviour by setting `legacy = False` when you initialize the tokenizer.

thanks! I tried that though and it did not change the output

Ok, the same issue exists with the fast version, but the problem is with the encoding that adds extra spaces between the special tokens.... It's a mess haha

@ArthurZucker
Sorry I can't understand when and why we need to set `legacy=False` , Could you exlpain？
I run the code as follows:
```python
    txt = "one more thing" + "
" + "traditionally" + "
"
    tokenizer1 = LlamaTokenizer.from_pretrained(
        "./resources/models/llama-2-7b-hf", legacy=True, use_fast=False
    )
    tokenizer2 = LlamaTokenizer.from_pretrained(
        "./resources/models/llama-2-7b-hf", legacy=False, use_fast=False
    )

    t1 = tokenizer1.tokenize(txt)
    t2 = tokenizer2.tokenize(txt)

```
Then I got:
```
t1:['▁one', '▁more', '▁thing', '
', '▁tradition', 'ally', '
']
t2:['▁one', '▁more', '▁thing', '
', 'tradition', 'ally', '
']
```
The word starting with a `▁` usually means the start of a new word (as when comparing `▁`more and `ally`).
Even though we don't add a space before "traditionally", it is still considered a new word.
So, seems `tokenizer2` is meaningful?
 

No, words starting with `_` means that these word have a space before them, and thus the token is `_tradition`. While `tradition` is a different token. If you read the documentation that points to the PR  #24565, there is a similar example. 
What's important to understand is the concept of `added tokens`. 

Most often, sentencepiece tokenizers have a vocabulary, but some tokens are added afterwards. This happens with t5 for example. In `transformers`, we do not modify the underlying sentencepiece object. But we still support adding tokens. 

Now imagine if `thin` is part of the sentencpiece vocab, but not `_thin`. If `thin` appears next to a work like `thinking`, is will be tokenized as [`_`, `thin`, `king`], not  [`_`, `thin`, `_king`]. The same applies for any tokens that are originally part of the sentencepiece model. 

In `transformers` all `special tokens` are kind of added to the vocabulary, so we want to reproduce the behaviour and not add extra space. 

PS: please refrain from asking something pretty much unrelated. If you have a question (not a bug) feel free to post it on [the discussion forum](https://discuss.huggingface.co/)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ArthurZucker this should be reopened, right? As stated in your previous response:
> In transformers all special tokens are kind of added to the vocabulary, so we want to reproduce the behaviour **and not add extra space.**

So, basically, there **should not be added space after special tokens**... However, I'm getting the opposite results to [this](https://github.com/huggingface/transformers/issues/25073#issuecomment-1655271420), with `legacy=False` being incorrect.
```py

from transformers import AutoTokenizer
text = "hello world"

# 1. Legacy tokenizer
tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", use_fast=False, legacy=True)
token_ids = tokenizer.encode(text, add_special_tokens=True)
print(f'{token_ids=}')                              # [1, 22172, 3186] (correct)
print(f'{tokenizer.decode(token_ids)=}')            # '
hello world' (correct)

# 2. Non-Legacy tokenizer
tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer", use_fast=False, legacy=False)
token_ids = tokenizer.encode(text, add_special_tokens=True)
print(f'{token_ids=}')                              # [1, 22172, 3186]  (correct)
print(f'{tokenizer.decode(token_ids)=}')            # '
 hello world' (incorrect)
```

(this is also different to the other related issues, since those deals with encoding and not decoding)

Yes, until #26678 is merged 

Just wait a bit!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closed by https://github.com/huggingface/transformers/pull/26678 👍 

Why is this closed while depending on if you set `use_fast` to True or False, the behavior is not the same @ArthurZucker ?

No the reason it's closed is because this has a flag: `legacy`, which can be set to `True` or `False`