[v5 regression] CLIPVisionModel.forward returns hidden_states=None even when output_hidden_states=True
Author: yukiu00
Pull request: no
Labels: name: bug

### System Info

### Description

I am reporting a potential regression found while testing `transformers` **v5.0.0**.
We noticed that `CLIPVisionModel.forward()` returns `hidden_states=None` even when `output_hidden_states=True` is explicitly passed. This behavior is different from v4.x, where hidden states were correctly returned.

### Reproduction / Context

We encountered this issue during Llava convergence tests in the **Liger Kernel** repository.
Specifically, the issue was identified in:
- **Liger-Kernel PR:** https://github.com/linkedin/Liger-Kernel/pull/1061
- **Related Issue:** https://github.com/linkedin/Liger-Kernel/issues/1011

In our test suite, we noticed the failure when creating a new model instance after reloading modules. The `CLIPVisionModel` fails to return hidden states.

Upon investigating the code changes in `transformers`, it appears that `output_hidden_states` is no longer handled correctly in `ClipEncoder.forward`.
- **v5.0.0 (Current):** [modeling_clip.py#L485](https://github.com/huggingface/transformers/blob/08810b1e278938278c50153ee1edfd7a20a759da/src/transformers/models/clip/modeling_clip.py#L485)
- **v4.57.6 (Previous):** [modeling_clip.py#L506](https://github.com/huggingface/transformers/blob/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/models/clip/modeling_clip.py#L506)

It seems `output_hidden_states` is present in `TransformersKwargs` fields but might not be utilized in the `.forward()` method after the recent modernizing changes.

### Suspected Cause
This might be related to the changes introduced in PR #41546.

Could you please confirm if this is an intended change? If so, what is the recommended way to retrieve hidden states in v5?

Thanks!



### Who can help?

@yonigozlan @molbap @zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import CLIPVisionModel, CLIPVisionConfig

# 1. Setup a minimal CLIPVisionModel
config = CLIPVisionConfig(
    hidden_size=32,
    intermediate_size=64,
    num_hidden_layers=2,
    num_attention_heads=4,
    image_size=30,
    patch_size=10
)
model = CLIPVisionModel(config)

# 2. Create dummy input
pixel_values = torch.randn(1, 3, 30, 30)

# 3. Run forward pass with output_hidden_states=True
outputs = model(pixel_values, output_hidden_states=True)

# 4. Check the result
print(f"Transformers version: {importlib.metadata.version('transformers')}")
print(f"Hidden states: {outputs.hidden_states}")

if outputs.hidden_states is None:
    raise ValueError("Regression: hidden_states is None despite output_hidden_states=True")
```

### Expected behavior

When `output_hidden_states=True` is passed to `model()`, `outputs.hidden_states` should be a tuple of tensors containing the hidden states from all layers. It should not be `None`.

Comments:
Hello @yukiu00 , `output_hidden_states` is indeed removed but not the tracking itself of hidden_states. it is handled through `_can_record_outputs`

```python
    _can_record_outputs = {
        "hidden_states": CLIPEncoderLayer,
        "attentions": CLIPAttention,
    }
```

so it is tracked. What happens is a different issue and is linked to when a model is _reloaded_, I had to dig a bit. Exact reproducer is:

```python
import importlib, torch
from transformers import CLIPVisionConfig, CLIPVisionModel
import transformers.models.clip.modeling_clip as modeling_clip

config = CLIPVisionConfig(hidden_size=32, intermediate_size=64, num_hidden_layers=2, num_attention_heads=4, image_size=30, patch_size=10)
pixel_values = torch.randn(1, 3, 30, 30)

m1 = CLIPVisionModel(config)
o1 = m1(pixel_values=pixel_values, output_hidden_states=True)
print("before reload:", o1.hidden_states is None)

importlib.reload(modeling_clip)

m2 = CLIPVisionModel(config)
o2 = m2(pixel_values=pixel_values, output_hidden_states=True)
print("after reload:", o2.hidden_states is None)
```

The reason why that happens is that `isinstance(module, specs.target_class) ` fails. Opening a PR to fix it

Thanks for the quick investigation @molbap!
To clarify my understanding: the output_hidden_states parameter itself was intentionally removed in v5, and hidden states tracking now works through the new _can_record_outputs mechanism. The actual bug is that isinstance() checks fail after module reload, which breaks this tracking.
Is that correct?
Looking forward to the fix PR. Let me know if there's anything I can help with for testing.

Yes, your understanding is correct. For testing, yes thanks, you may checkout the linked PR and let me know if it solves your issue? I mean just  `git checkout fix_recording_on_reload` and `pip install -e .` to have the local install, if you want to test

@molbap 

Thanks for the quick fix! I tested the `fix_recording_on_reload` branch and confirmed it resolves the issue:

- Minimal reproduction code: ✓ `hidden_states` correctly returned after reload
- Liger-Kernel Llava convergence test: ✓ Passed


This solves a huge pain point for me. Shared my automated deployment scripts for this on my GitHub.

Ig this is fixed now in `main` branch