Phi-3's tokenizer `.convert_ids_to_tokens` returns bytes instead of a string
Author: iPieter
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.0 and 4.44.2
- Platform: Linux-5.4.0-193-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA RTX A5000

### Who can help?

@arthu

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. initialize tokenizer
```python
from transformers import AutoTokenizer.
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-small-8k-instruct")
```
2. call `convert_ids_to_tokens` on a random token id
```python
tokenizer.convert_ids_to_tokens(tokenizer.encode(" a", add_special_tokens=False)[0])
> b' a'
```
3. Observe that the returned object is a byte (array), instead of the documented string (array).

### Expected behavior

The returned object is a byte (array), instead of the [documented string (array)](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.convert_ids_to_tokens). For most models it is actually a string (array), so this would need some conditional conversion logic to call `.decode("utf-8")`. 

Comments:
Hey @iPieter ! I see that this specific Phi-3 model uses a custom tokenizer model (non-`transformers` --> see [here](https://huggingface.co/microsoft/Phi-3-small-8k-instruct/blob/1535ae26fb4faada95c6950e8bc6e867cdad6b00/tokenizer_config.json#L15) the model specified is their custom Phi3SmallTokenizer) so it may be that the authors intended this. `tokenizer.get_vocab()` returns a byte vocab as well.

If it works for your use-case, you can try a Phi-3 model that uses a `transformers` model such as [`microsoft/Phi-3-mini-4k-instruct`](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/5a516f86087853f9d560c95eb9209c1d4ed9ff69/tokenizer_config.json#L127) which uses `LlamaTokenizer`. Please let me know if that would work for you! 🤗 

hmmm ok, I guess it is not really a bug in the transformers library then, but in their implementation. So this issue can be closed. @itazap Thanks for linking the custom tokenizer file.

But perhaps a solution for this problem in general could be to do type checking on the custom files? It does take a lot of debugging to find out that some models suddenly return wildly different types.

And for this specific case I am also not convinced about Phi's implementation. In general, the tokens should stay in the string domain and represent bytes with strings ([as gpt-2 does](https://github.com/openai/gpt-2/blob/master/src/encoder.py#L9)), so this byte-based representation is a bit unconventional at least.