Unsuppressable warning: "<model> will not detect padding tokens in `inputs_embeds`"
Author: naimenz
Pull request: no

### System Info

- `transformers` version: 4.39.3
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.3
- Accelerate version: 0.29.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2 (False)
- Tensorflow version (GPU?): 2.16.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running this example prints a warning for every loop, despite there being no padding.
```python
from transformers import AutoTokenizer, GPT2Model, GPT2ForSequenceClassification

model = GPT2ForSequenceClassification.from_pretrained("gpt2")
assert isinstance(model, GPT2ForSequenceClassification)
model.config.pad_token_id = model.config.eos_token_id

tokenizer = AutoTokenizer.from_pretrained("gpt2")
text = ["Hello, my dog is cute."]
inp = tokenizer(text, return_tensors="pt")
embeds = model.get_input_embeddings()(inp["input_ids"])

for i in range(3):
    out = model(inputs_embeds=embeds)
```
Output:
```
GPT2ForSequenceClassification will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`
GPT2ForSequenceClassification will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`
GPT2ForSequenceClassification will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`
```

### Expected behavior

The warning is printed once, or at least there is some way to disable the warning.

I was split between whether this should be a bug report or a feature request. It makes sense to display this warning, but in my project I need to run on embeddings often and the warning really spams the logs.

For a while I was only running batches  of size `1` so I was suppressing the warning by temporarily setting `model.config.pad_token_id = None`. The problem with this is that I then can't run batches of size `>1` even if I'm careful to make them the same length with no padding tokens.

I'm not sure of the best way to handle this, but either using the `warnings` library to make it only print once and/or allow it to be suppressed would help, or having some flag to disable the warning.

The earliest instance of the string `will not detect padding tokens in` in the codebase I could find was from https://github.com/huggingface/transformers/pull/7501.

Comments:
Hi @naimenz, thanks for opening this issue / feature request! 

As you mentioned, I don't think we'd want to remove the warning completely, it's telling you something quite useful. One option would be to replace `logger.warning` with `logger.warning_once` so that the message is only emitted once per session. If you think this is a suitable solution, feel free to open a PR and I'll be happy to review 🤗 

Thanks @amyeroberts! Sorry for the delay, got a PR up now: #31411