🚨🚨🚨 [`SPM`] Finish fix spm models 🚨🚨🚨
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Modifies `Llama` and `T5` other sentencepiece based tokenizer will follow. 

Previous behaviour is always possible with ` tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = True)`
## The goal of `transformers`'s wrapping around `sentencepiece`
To clarify, we want to:
- be able to choose the behaviour of the special/added tokens. This means handling the `stripping`, encoding and decoding of such tokens
- allow users to easily add new tokens, with `tokenenizer.add_tokens(...)` instead of having to load the protobuf file, modify the vocab, save it and reload the sentencepiece processor. 

## The current and past problems with our wrappers
Let's use both T5 and Llama as reference models. Currently, we do not mimic the behaviour of adding words to the actual `sentencepiece` vocabulary. This is an issue for anyone expecting (and rightfully) that adding tokens does not modify the behaviour of the model. 

### Adding a word to sentencepiece's vocab
This can be done using: ([source](https://github.com/google/sentencepiece/issues/121#issuecomment-400362011))

```python 
>>> # wget https://huggingface.co/huggyllama/llama-7b/resolve/main/tokenizer.model
>>> from sentencepiece import sentencepiece_model_pb2 as model
>>> import sentencepiece as spm
>>> sp_model = model.ModelProto()
>>> sp_model.ParseFromString(open('tokenizer.model', 'rb').read())
>>> token = "your_token"
>>> sp_model.pieces.add(piece=f"{token}",score=0.0,type=model.ModelProto.SentencePiece.USER_DEFINED,)
>>> with open('new.model', 'wb') as f:
...     f.write(sp_model.SerializeToString())
```
then load the `sp_model`:
```python 
>>> sp_model = spm.SentencePieceProcessor()
>>> sp_model.Load('new.model')
```

Then, try the following :
```python 
>>> sp_model.encode("your_tokenHello", out_type=str)
["_", "your_token", "Hello"]
```

### Adding a word to a `PretrainedTokenizer
This can be done using `tokenizer.add_tokens(["your_token"])`. It is a lot simpler indeed. 
But the output you will get is: 
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = True, use_fast = False)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["your_token", "_Hello"]

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", legacy = False, use_fast = False)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["your_token", "Hello"]
```

This is because we always split the text on the added tokens, and give the text on the left and right to the `sentencepiece` model. But, most sentencepiece models add a prefix space `_` (the `SPIECE_UNDERLINE` character). Thus, when the `transformers` tokenizers splits `"your_tokenHello"`, it encode `your_token` with the `tokenizer.added_tokens_encoder` and thus does not add a prefix space, and then encode `Hello` with the sentencepiece model, which adds a prefix space and thus outputs `_Hello`. 

Other missmatches: 

```python
# t5-base tokenizer
>>> tokenizer.encode("<extra_id_0>. Hello", add_special_tokens = False)
[32099, 3, 5, 8774] # ['<extra_id_0>', ' ▁', '.', '▁Hello']
# seqio.SentencePieceVocabulary(vocab_path, extra_ids = 300)
>>> processor.encode("<extra_id_0>. Hello")
[32099, 5, 8774] # ['<extra_id_0>', '.', '▁Hello']

``` 


TLDR; this shows the only way we can actually and properly handle added tokens and sentencepiece. We have to disable automatic prefix addition, and always encode with a token that is part of the vocab at the beginning to properly encode the first token, whether it has a prefix space or not. Yes this is dirty and sad, but the previous fix was removing the extra space, which was cleaner but had a corner cases #25176. 

### The same issue happens with fast tokenizers:
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", use_fast = True)
>>> tokenizer.add_tokens(["your_token"])
>>> tokenizer.tokenize("your_tokenHello")
["_your_token", "Hello"]

>>> tokenizer.add_tokens(["your_token_special"], True)
>>> tokenizer.tokenize("your_token_specialHello")
['your_token_special', '▁Hello']
```

### Another issue 😈 

So, here, the issue is that before the special token, even if there is no `rstrip` or `lstrip` (both are set to False), we have very strange behaviours:

```python 
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast = True)

>>> tokenizer.tokenize("
inform
")
# prefix space is eaten
['
', '▁inform', '
']

>>> tokenizer.tokenize("
inform 
")
# prefix space is not eaten for the second 

['
', '▁inform', '▁', '
']

>>> tokenizer.tokenize(" 
inform 
")
# prefix space is not eaten for the second 

['▁▁', '
', '▁inform', '▁', '
']

>>> tokenizer.tokenize(" 
inform 
 ")
# prefix space is not eaten for the first 
, extra space added (known)
['▁▁', '
', '▁inform', '▁', '
', '▁▁']

>>> tokenizer.tokenize("inform 
 ")
# prefix space is added to inform
['▁inform', '▁', '
', '▁▁']
```
Note that `tokenizer.convert_tokens_to_ids("▁▁") = 259` while `tokenizer.convert_tokens_to_ids("▁") = 29871`
Also if we add a prefix space to special tokens the beginning, we are probably gonna break a lot of things

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Will fix the prefixing of special tokens! 

@ArthurZucker any update to this PR? 

Hey @faaany, I am updating it right now! 

Reverted the changes as adding proper support for `add_prefix_space` is actully questionable. The usecase is already wrong as you should be reverse looking for ids not strings. See #24846 (adding prefix space was almost never done properly as the decoders were not updated as well)

pinging @sgugger for a final review !

Will do so in a follow up PR! 

I have added the legacy=True as "enc = AutoTokenizer.from_pretrained(model_path, legacy = True, use_fast=False)" but I have still gotten an error, which is a "Not a string" error. Anyone can give a hint what is going on here?

<img width="1257" alt="截屏2023-08-20 21 40 35" src="https://github.com/huggingface/transformers/assets/32901895/840e31b4-0b56-4615-ba5a-beb31fe6c376">



@zhacmsra the issue is in loading the vocabulary file, not 100% sure it's related to this. Can you open a new Issue with a reproducer please? 

> 

Hi Arthur, you are correct. I figured out that it is not related to this PR. The networking problem broke the input model and result in error inputs. Sorry for this trouble. Thank you for the kind and timely response.