The Phi 1.5 tokenizer is giving different tokens for same word
Author: bnicholl
Pull request: no

### System Info

The Phi 1.5 tokenizer is giving different tokens for same word. For example tokenizing dog twice will give two different tokens.

### Who can help?

@ArthurZucker  @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5", trust_remote_code=True, torch_dtype="auto")
```

```
# example word to tokenize
encoded_inputs = tokenizer(["dog"])
```
`output: {'input_ids': [[9703]], 'attention_mask': [[1]]}`

Now when I use a sentence with dog twice, the tokens for dog are different. Below is an example simply having dog twice in the tokenizer

```
encoded_inputs = tokenizer(["dog dog"])
output: {'input_ids': [[9703, 3290]], 'attention_mask': [[1, 1]]}
```

### Expected behavior

The output for 
`encoded_inputs = tokenizer(["dog dog"])`
should be
`output: {'input_ids': [[9703, 9703]], 'attention_mask': [[1, 1]]}`

assuming the correct token indices for dog is indices 9703

Comments:
The phi-1_5 tokenizer takes the space character in between into account and produces a different token for ` dog`, in this case it is `3290`. 

If you use 
1) encoded_inputs = tokenizer(["dog,dog"])
2) encoded_inputs = tokenizer(["dogdog"])

you will get 
1) `output: {'input_ids': [[9703, 11, 9703]], 'attention_mask': [[1, 1, 1]]}`
2) `output: {'input_ids': [[9703, 9703]], 'attention_mask': [[1, 1]]}`
respectively. 

I am not 100% sure if this was intended. I will look into that.

I have found this 

> the BPE vocabulary of GPT-2 is extended by special tokens representing
> repeating tokens of tabs and white spaces.

in the [paper](https://arxiv.org/pdf/2203.13474.pdf) where the tokenizer is borrowed from. 

> the BPE vocabulary of GPT-2 is extended by special tokens representing
> repeating tokens of tabs and white spaces.

This seems to be for white spaces only though.

Upon further inspection
`tokenizer.convert_ids_to_tokens(9703)` gives the word dog
and 
`tokenizer.convert_ids_to_tokens(3290)` is giving the word 'Ġdog'

also the sentence 
`encoded_inputs = tokenizer(["moose is a dog a good dog"])`
gives output 
`output:  {'input_ids': [[5908, 577, 318, 257, 3290, 257, 922, 3290]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1]]}`
So it seems "dog"(indice 9703) is the token if it is the first word in a sequence(not sentence) separated by a space,  but 'Ġdog'(indice 3290) is used if it is any word other than the first word in a sequence. This also seems to be true for other words as well. 


I'll look at the paper and see if I can figure out if this is expected. 

It's important to note, I found this potential discrepancy because I'm getting very poor results when fine tuning phi 1.5. Not sure if this would be the reason or not. 

Hi there 👋 This is intended behaviour for the tokenizer: i.e., to treat the first word a bit differently. Check out [this answer](https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475/2) by @thomwolf for a full explanation why (TLDR: it performs better this way).

You can see this setting in the [tokenizer_config.json](https://huggingface.co/microsoft/phi-1_5/blob/main/tokenizer_config.json#L2):
```json
{
  "add_prefix_space": false,
  ...
}
```

If you really want to override this, you can set `add_prefix_space` to `True` as follows:
```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5", add_prefix_space=True)
print(tokenizer("dog"))  # {'input_ids': [3290], 'attention_mask': [1]}
print(tokenizer("dog dog"))  # {'input_ids': [3290, 3290], 'attention_mask': [1, 1]}
```

As you can see, the same token is now used for `"dog"` in both cases (the one corresponding to `" dog"`; with a prefix space).

@xenova thanks for the response and the link. I'll close this.