Different tokenizer silently being loaded based on `trust_remote_code`
Author: DarkLight1337
Pull request: no
Labels: name: bug

### System Info

```
python==3.9.20
transformers==4.46.2
```

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm trying to use `Alibaba-NLP/gte-Qwen2-1.5B-instruct` in vLLM, but found that the tokenizer that is loaded from HF Transformers has different `padding_side` based on the `trust_remote_code` setting:

```py
>>> from transformers import AutoTokenizer
>>> AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct", trust_remote_code=False)
Qwen2TokenizerFast(name_or_path='Alibaba-NLP/gte-Qwen2-1.5B-instruct', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '
', 'pad_token': '
', 'additional_special_tokens': ['
', '
']}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
        151643: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        151644: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        151645: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
>>> AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct", trust_remote_code=True)
Qwen2TokenizerFast(name_or_path='Alibaba-NLP/gte-Qwen2-1.5B-instruct', vocab_size=151643, model_max_length=32768, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '
', 'pad_token': '
', 'additional_special_tokens': ['
', '
']}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
        151643: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        151644: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        151645: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
```

This setting significantly impacts the output of the embedding model. Is this intended behavior? I suspect no error is being raised because the name of the custom tokenizer in this HF repo (see [relevant file](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct/blob/main/tokenization_qwen.py)) is `Qwen2TokenizerFast` which is also defined in core Transformers.

### Expected behavior

To avoid accidentally loading the wrong tokenizer, the code should raise an error if a custom tokenizer is defined in the HF repo but `trust_remote_code=False`, regardless of whether there is a tokenizer with the same name defined inside core Transformers.

Comments:
Hey! Sorry I think in general the padding side should be defined in the `config.json` not hardcoded. What you are getting with `trust_remote_code=False` is the `transformers` default, so it's the same for everyone! 

> What you are getting with `trust_remote_code=False` is the `transformers` default

Do you mean that setting `trust_remote_code=False` will automatically fallback to the tokenizer defined inside `transformers`, even if the model repo defines a custom one? Since the custom tokenizer may also be different in other ways, we should still raise an error to get fail-fast behavior instead of silently causing incorrect model output.

No, there is a miss-understanding. In `transformers` we provide an API to allow users to control the padding side. 
If the code on the hub pushed by the maintainers hardcoded a padding side, which does not follow the API we recommend, there is nothing we can really do on our side. 

Best recommendation: open a PR to the hub to change the `tokenizer_config.json` to use  `padding_side='left'`, as this is the API followed by every other model when the default padding side is `left` instead of `right`

I see, thanks for explaining this! I'll open an issue on their side then.