[Phi-3-mini-128k-instruct] Difference of encodings for Slow and Fast Tokenizer
Author: GKIBMNY
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-5.10.223-212.873.amzn2.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0a0+3bcc3cddb5.nv24.07 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA L40S


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-128k-instruct', use_fast=True)
text = '
\n I am good 
\n
'
ids=tokenizer(text).input_ids
print(ids)
print(tokenizer.convert_ids_to_tokens(ids))
print(tokenizer.encode(text))

tokenizer = AutoTokenizer.from_pretrained('microsoft/Phi-3-mini-128k-instruct', use_fast=False)
ids = tokenizer(text).input_ids
print(ids)
print(tokenizer.convert_ids_to_tokens(ids))
print(tokenizer.encode(text))

### Expected behavior

In expected case, both slow and fast tokenizer should return the same IDs

Comments:
Hey @GKIBMNY try using the following script,

Try setting legacy=True when loading the tokenizer:
```python
from transformers import AutoTokenizer

model_name = "microsoft/Phi-3-mini-128k-instruct"
text = "
\n I am good 
\n
"

# Using Fast Tokenizer
fast_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
fast_ids = fast_tokenizer(text).input_ids
print("🔵 Fast Tokenizer IDs:", fast_ids)
print("🔵 Fast Tokenizer Tokens:", fast_tokenizer.convert_ids_to_tokens(fast_ids))
print("🔵 Fast Tokenizer Encoded:", fast_tokenizer.encode(text))

# Using Slow Tokenizer without legacy=True
slow_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False, legacy=False)
slow_ids = slow_tokenizer(text).input_ids
print("🟢 Slow Tokenizer (legacy=False) IDs:", slow_ids)
print("🟢 Slow Tokenizer (legacy=False) Tokens:", slow_tokenizer.convert_ids_to_tokens(slow_ids))
print("🟢 Slow Tokenizer (legacy=False) Encoded:", slow_tokenizer.encode(text))

# Using Slow Tokenizer with legacy=True
slow_tokenizer_legacy = AutoTokenizer.from_pretrained(model_name, use_fast=False, legacy=True)
slow_legacy_ids = slow_tokenizer_legacy(text).input_ids
print("🟠 Slow Tokenizer (legacy=True) IDs:", slow_legacy_ids)
print("🟠 Slow Tokenizer (legacy=True) Tokens:", slow_tokenizer_legacy.convert_ids_to_tokens(slow_legacy_ids))
print("🟠 Slow Tokenizer (legacy=True) Encoded:", slow_tokenizer_legacy.encode(text))
```

Output:
🔵 Fast Tokenizer IDs: [32010, 306, 626, 1781, 29871, 32007, 32000]
🔵 Fast Tokenizer Tokens: ['
', '▁I', '▁am', '▁good', '▁', '
', '
']
🔵 Fast Tokenizer Encoded: [32010, 306, 626, 1781, 29871, 32007, 32000]
🟢 Slow Tokenizer (legacy=False) IDs: [29871, 32010, 306, 626, 1781, 29871, 32007, 32000]
🟢 Slow Tokenizer (legacy=False) Tokens: ['▁', '
', '▁I', '▁am', '▁good', '▁', '
', '
']
🟢 Slow Tokenizer (legacy=False) Encoded: [29871, 32010, 306, 626, 1781, 29871, 32007, 32000]
🟠 Slow Tokenizer (legacy=True) IDs: [32010, 29871, 306, 626, 1781, 29871, 32007, 32000]
🟠 Slow Tokenizer (legacy=True) Tokens: ['
', '▁', '▁I', '▁am', '▁good', '▁', '
', '
']
🟠 Slow Tokenizer (legacy=True) Encoded: [32010, 29871, 306, 626, 1781, 29871, 32007, 32000]



Yep! `legacy` is the most important flag for the behaviour between special tokens and tokens right after them! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.