Confusing deprecation / warning messages
Author: patrickvonplaten
Pull request: no

### System Info

```
- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.3
- Safetensors version: 0.4.1
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@amyeroberts for Vision

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When doing:

```py
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
```

I'm getting a couple of confusing error messages since transformers 4.36:

```
`AnnotionFormat` is deprecated and will be removed in v4.38. Please use `transformers.image_utils.AnnotationFormat` instead
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
```

None of these variables (`AnnotionFormat` or `text_config_dict`) are defined anywhere in `diffusers` or in the configs: https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/text_encoder/config.json

It seems like something inside Transformers triggers these deprecation warnings which makes the messages very confusing and non-actionable for users. Also since it happens every time `from_pretrained(...)` is called, it clutters the CLI quite a bit

### Expected behavior

No warnings or clearer instructions and what needs to be changed to remove these warnings

Comments:
@patrickvonplaten Thanks for reporting. I think I know the culprit PR but indeed it shouldn't be triggered here. Looking into it 🕵️ 

Or, at least I know about `AnnotionFormat` - I don't have any immediate ideas regarding `text_config_dict`

~~The `text_config_dict` is from~~

~~https://huggingface.co/openai/clip-vit-large-patch14/blob/main/config.json~~

Opened a PR to address the Annotion format errors. 

For the text_config errors, using the `"runwayml/stable-diffusion-v1-5"`, at least one of the errors is coming from using [this config](https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/safety_checker/config.json). 

As the error mentions, both `text_config_dict` and `text_config` are being used in the model config. The warning is letting you know some of the values in `"text_config"` won't be used e.g. `eos_token_id`, even though `eos_token_id` isn't in `text_config_dict`. This is because `text_config_dict` is used to instantiate a `CLIPTextConfig` class, which will have default values of e.g. `eos_token_id = 49407`.  I think the warning itself is good - it told me exactly what to look for and flags potentially confusing behaviour. I believe the correct resolution would be to add the overwritten values to `text_config`  - @ydshieh is this right? 

@patrickvonplaten would adding a note on how to fix be sufficient or is there more information you'd like to see? 



> I believe the correct resolution would be to add the overwritten values to text_config - @ydshieh is this right?

Yes, if the 2 config (dict) get the same value for a key, then no warning won't be given (for that key).
(or just update the config file to not use `text_config_dict`  with an updated `text_config`)

We can open Hub PRs to the most used CLIP repositories to avoid such warning.


I have three warnings triggered by calling StableDiffusionSafetyChecker.from_pretrained():

```
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
```



> As the error mentions, both text_config_dict and text_config are being used in the model config. The warning is letting you know some of the values in "text_config" won't be used e.g. eos_token_id, even though eos_token_id isn't in text_config_dict. This is because text_config_dict is used to instantiate a CLIPTextConfig class, which will have default values of e.g. eos_token_id = 49407. I think the warning itself is good - it told me exactly what to look for and flags potentially confusing behaviour. I believe the correct resolution would be to add the overwritten values to text_config - @ydshieh is this right?

> Yes, if the 2 config (dict) get the same value for a key, then no warning won't be given (for that key).
(or just update the config file to not use text_config_dict with an updated text_config)

I have some trouble understanding this tbh :sweat_smile: 

What exactly do we need to change here: https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/safety_checker/config.json ? (could you maybe open a PR?)