output_hidden_states is not working correctly with SiglipModel class
Author: hamzaakyildiz
Pull request: no
Labels: name: WIP

name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35
- Python version: 3.12.12
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0+cu126 (CUDA)
- Tensorflow version (GPU?): 2.19.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.7 (gpu)
- Jax version: 0.7.2
- JaxLib version: 0.7.2
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: Tesla T4

### Who can help?

@zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Problem
Even though I pass the `output_hidden_states=True` to the vision and text model through the config, it seems that the `SiglipModel` class does not successfully cascade this config to the vision and text model.

## Reproduction Code
```python
import torch
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoConfig

model_path = "google/siglip-base-patch16-224"

config = AutoConfig.from_pretrained(model_path)
config.output_hidden_states = True
config.vision_config.output_hidden_states = True
config.text_config.output_hidden_states = True

model = AutoModel.from_pretrained(
    "google/siglip-base-patch16-224",             
    config=config
)
processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")


images = [Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)]
texts = ["a photo of 2 cats"]
inputs = processor(text=texts, images=images, padding="max_length", return_tensors="pt")

with torch.no_grad():
    outputs = model.forward(**inputs)

print(outputs.text_model_output.keys()) # odict_keys(['last_hidden_state', 'pooler_output'])
```

### Expected behavior

It should have returned hidden_states also as i already stated in the config file when i am using SiglipModel class.

Comments:
I tried to dig into the problem and found these, it may help solve the bug
- If I initialize the vision or text model separately, it works fine, returning the hidden states successfully.
```python

from PIL import Image
import requests
from transformers import AutoProcessor, SiglipVisionModel, AutoConfig

model_path = "google/siglip-base-patch16-224"

config = AutoConfig.from_pretrained(model_path)
config.vision_config.output_hidden_states = True

model = SiglipVisionModel.from_pretrained(
    "google/siglip-base-patch16-224",             
    config=config.vision_config
)
processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")


images = [Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)]
inputs = processor(images=images, padding="max_length", return_tensors="pt")

with torch.no_grad():
    outputs = model.forward(**inputs)

print(outputs.keys())  # odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states'])
```
- Assuming this module is initialized correctly, I tried to change the `SiglipModel` class instance's text/vision model with this initialized module.
```python
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel, AutoConfig

model_path = "google/siglip-base-patch16-224"

config = AutoConfig.from_pretrained(model_path)
config.output_hidden_states = True
config.vision_config.output_hidden_states = True
config.text_config.output_hidden_states = True

siglip_model = AutoModel.from_pretrained(
    "google/siglip-base-patch16-224",             
    config=config
)
siglip_vision_model = SiglipVisionModel.from_pretrained(
    "google/siglip-base-patch16-224",             
    config=config.vision_config
)
processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")


# This is not the way to go I guess, because in the original initialization it assigns siglip_vision_model.vision_model to siglip_model.vision_model
siglip_model.vision_model = siglip_vision_model
images = [Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)]
texts = ["a photo of 2 cats"]
inputs = processor(text=texts, images=images, padding="max_length", return_tensors="pt")

with torch.no_grad():
    outputs = siglip_model.forward(**inputs)

print(outputs.vision_model_output.keys()) # odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states'])


# If i assign the siglip_vision_model.vision_model to siglip_model.vision_model, then it does not return the hidden_states.
siglip_model.vision_model = siglip_vision_model.vision_model
images = [Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)]
texts = ["a photo of 2 cats"]
inputs = processor(text=texts, images=images, padding="max_length", return_tensors="pt")

with torch.no_grad():
    outputs = siglip_model.forward(**inputs)

print(outputs.vision_model_output.keys()) # odict_keys(['last_hidden_state', 'pooler_output'])
```
- Based on these findings, I believe that even though the config is passed through `SiglipVisionModel` and `SiglipTextModel`, their transformer blocks have a bug that prevents them from taking this config into account.



Can you check the v5 rc0? We had a PR for this issue I believe, see #41930 cc @molbap 

Automappings mapped to non-top modules (i.e. non pretrained modules) which in turn did not allow outputting the hidden states (and attentions). It's a bit awkward that this is even a thing because functionally there is no difference and it only acts as a wrapper class.

Yeah, looks like the same thing. The model calls `text_model.text_model` which is not a pretrained module. Siglip has a super awkward way to define submodules tbh, never got to dig why we wrap the text model inside another text model 🥲 

Still broken on main, the above PR fixed only the vision part apparently

I will fix it in https://github.com/huggingface/transformers/pull/42841, we don't need to have the nedted structure for `SiglipModel`, and instead make use of `base_model_prefix` to adjust model state dict.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @yonigozlan , maybe you can assign this to yourself. You had a PR to refactor all siglip models. I will keep the issue open until then

@zucchini-nlp  Noted, Thanks!