put output_hidden_states into filter_output_hidden_states
Author: molbap
Pull request: yes

# What does this PR do?

Backbones need all intermediate hidden states to build their feature maps, so each one forced the flag by hand in its forward:

```python
kwargs["output_hidden_states"] = True  # required to extract layers for the stages
```

This PR moves that into the `filter_output_hidden_states` decorator: it forces `output_hidden_states=True` on the wrapped forward, then drops `hidden_states` from the output unless the caller requested them. It's still not particularly memory-efficient but at least not returned when not asked.

Two fixes along the way. The old filter check `k not in ("hidden_states")` was just a susbtring check so only worked by accident; it is now `k != "hidden_states"`. And timm_backbone loses the decorator: it implements `output_hidden_states` itself by switching timm's `return_layers`, so forcing the flag broke every timm-based model.

After internal discussions: VLM `get_image_features` / `get_video_features` use the same forcing pattern but interact with how those methods set `pooler_output`, and how we want to type the outputs of these models. Hence, a bit trickier, so that part will move to a follow-up PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46422). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Oh wow fair enough :D (how could i not see this🤦)

run-slow: beit, convnext, convnextv2, depth_anything, dinov2, dinov2_with_registers, dinov3_vit, dpt,
  grounding_dino, lw_detr, mask2former, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, resnet, rf_detr, rt_detr, sapiens2,
  segformer, swin, timm_backbone, uvdoc, vitdet, vitpose_backbone

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27008060004)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/beit", "models/convnext", "models/convnextv2", "models/depth_anything", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_vit", "models/dpt", "models/grounding_dino", "models/lw_detr", "models/mask2former", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/resnet", "models/rf_detr", "models/rt_detr", "models/sapiens2", "models/segformer", "models/swin", "models/timm_backbone", "models/uvdoc", "models/vitdet", "models/vitpose_backbone"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27008060004)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [638a2c80](https://github.com/huggingface/transformers/commit/638a2c80bb0e3b2abc45cab04e32143922066603) | workflow commit (merge commit) |
| PR | [4dafef1d](https://github.com/huggingface/transformers/commit/4dafef1d31bd64c6d8b6317d762df9047e5b2d0a) | branch commit (from PR) |
| main | [10ed83ff](https://github.com/huggingface/transformers/commit/10ed83ffb9d2b3b6e7f89ace037de7f0d2414d3e) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


run-slow: aya_vision, llava, vipllava, llava_next

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27017945272)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aya_vision", "models/llava", "models/llava_next", "models/vipllava"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27017945272)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [fd95dcdb](https://github.com/huggingface/transformers/commit/fd95dcdb5fc2e6091e2f0281bf4d1ecd8f2b2f9e) | workflow commit (merge commit) |
| PR | [3c4003d5](https://github.com/huggingface/transformers/commit/3c4003d5f888f9b5c7e53d044be55eae60907d3e) | branch commit (from PR) |
| main | [98eea724](https://github.com/huggingface/transformers/commit/98eea7240670acfe8fbb2a0a9a4ce9edb25cd7d9) | base commit (on `main`) |

### Model CI Report

❌ **[2 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/e5b0d30a2bca67bc5be3479f32d66e16d8ee004f/2026-06-05/runs/35829-27017945272/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [llava](https://github.com/huggingface/transformers/actions/runs/27017945272/job/79738174413):
    tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_pixtral_4bit (❌ ⟹ ❌)
    tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_pixtral_batched (❌ ⟹ ❌)



run-slow: aria, aya_vision, beit, convnext, convnextv2, deepseek_vl_hybrid, llava, llava_next, llava_next_video, llava_onevision, lw_detr, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, segformer, swin

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27021398595)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/beit", "models/convnext", "models/convnextv2", "models/deepseek_vl_hybrid", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/lw_detr", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/segformer", "models/swin"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27021398595)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [53f5ed97](https://github.com/huggingface/transformers/commit/53f5ed97fd261c1d27bbbccc216af3177af79daa) | workflow commit (merge commit) |
| PR | [032c0930](https://github.com/huggingface/transformers/commit/032c09304ace799b56948e1bd140fbc8643998f9) | branch commit (from PR) |
| main | [b9d2bb4a](https://github.com/huggingface/transformers/commit/b9d2bb4a44f2bbce1592ad9897ef4f31774d4a0b) | base commit (on `main`) |

⚠️ No test being reported (jobs are skipped or cancelled)!


run-slow: aria, aya_vision, beit, convnext, convnextv2, deepseek_vl_hybrid, llava, llava_next, llava_next_video, llava_onevision, lw_detr, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, segformer, swin

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27021967223)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/beit", "models/convnext", "models/convnextv2", "models/deepseek_vl_hybrid", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/lw_detr", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/segformer", "models/swin"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27021967223)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [1662668b](https://github.com/huggingface/transformers/commit/1662668b51e5813380ba7d266acceb903b61e8a6) | workflow commit (merge commit) |
| PR | [7be62010](https://github.com/huggingface/transformers/commit/7be620107751c5e0b8b6a174b939b2c94ce0c8ce) | branch commit (from PR) |
| main | [bcd12e36](https://github.com/huggingface/transformers/commit/bcd12e3679daedeac7f9ebb34b7896d074786373) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


run-slow: aria, aya_vision, beit, convnext, convnextv2, deepseek_vl_hybrid, depth_anything, dinov2, dinov2_with_registers, dinov3_vit, dpt, grounding_dino, llava, llava_next, llava_next_video, llava_onevision, lw_detr, mask2former, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, resnet, rf_detr, rt_detr, sapiens2, segformer, swin, timm_backbone, uvdoc, vitdet, vitpose_backbone


[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27145914929)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/beit", "models/convnext", "models/convnextv2", "models/deepseek_vl_hybrid", "models/depth_anything", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_vit", "models/dpt", "models/grounding_dino", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/lw_detr", "models/mask2former", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/resnet", "models/rf_detr", "models/rt_detr", "models/sapiens2", "models/segformer", "models/swin", "models/timm_backbone", "models/uvdoc", "models/vitdet", "models/vitpose_backbone"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27145914929)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [e1d26b76](https://github.com/huggingface/transformers/commit/e1d26b76d7e690dacc621f9ca5bce30ca6b88aa6) | workflow commit (merge commit) |
| PR | [eae1637c](https://github.com/huggingface/transformers/commit/eae1637ca1e9dd41724e008bfa1165821e373248) | branch commit (from PR) |
| main | [fb9f5b43](https://github.com/huggingface/transformers/commit/fb9f5b43d0ab4a15269741da8db79014400bec60) | base commit (on `main`) |

⚠️ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/27145914929) for details.


run-slow: aria, aya_vision, beit, convnext, convnextv2, deepseek_vl_hybrid, depth_anything, dinov2, dinov2_with_registers, dinov3_vit, dpt, grounding_dino, llava, llava_next, llava_next_video, llava_onevision, lw_detr, mask2former, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, resnet, rf_detr, rt_detr, sapiens2, segformer, swin, timm_backbone, uvdoc, vitdet, vitpose_backbone

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27191888099)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/beit", "models/convnext", "models/convnextv2", "models/deepseek_vl_hybrid", "models/depth_anything", "models/dinov2", "models/dinov2_with_registers", "models/dinov3_vit", "models/dpt", "models/grounding_dino", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/lw_detr", "models/mask2former", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/resnet", "models/rf_detr", "models/rt_detr", "models/sapiens2", "models/segformer", "models/swin", "models/timm_backbone", "models/uvdoc", "models/vitdet", "models/vitpose_backbone"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27191888099)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [1c1218a6](https://github.com/huggingface/transformers/commit/1c1218a6faee7fc1f72c1b01f12efe15f67541c2) | workflow commit (merge commit) |
| PR | [7e3f47d7](https://github.com/huggingface/transformers/commit/7e3f47d796dcc0ebacc174b847027f36e88423ac) | branch commit (from PR) |
| main | [e3144397](https://github.com/huggingface/transformers/commit/e31443976ba8aca39f28c36c97f1805ba8326f43) | base commit (on `main`) |

⚠️ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/27191888099) for details.


**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46422)

run-slow: beit, convnext, convnextv2, lw_detr, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, segformer, swin, timm_backbone, uvdoc, vitpose_backbone

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27289485472)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/beit", "models/convnext", "models/convnextv2", "models/lw_detr", "models/pp_lcnet", "models/pp_lcnet_v3", "models/pp_lcnet_v4", "models/segformer", "models/swin", "models/timm_backbone", "models/uvdoc", "models/vitpose_backbone"]
quantizations: []

@vasqu @zucchini-nlp I reverted the VLM modifications and updated the PR desc, CI is a bit janky it seems? LMK if ok to merge, would be happy to see this (1st part) through :D 

Let's rather merge tomorrow ig? CI really isn't doing well atm

Not sure what is happening, looks like mostly timeouts as you mentioned @ydshieh ? 
ok to merge later!

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27289485472)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [d0bd150a](https://github.com/huggingface/transformers/commit/d0bd150a0da0319580ffe743965260a10ecc21a3) | workflow commit (merge commit) |
| PR | [f1029db9](https://github.com/huggingface/transformers/commit/f1029db94609a4e2c1b54a3b2e24d7b64bb82eec) | branch commit (from PR) |
| main | [b0a38eb6](https://github.com/huggingface/transformers/commit/b0a38eb6df5ae41e17d6a8a869c383b013e62239) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Gentle ping @molbap let's try to merge/rebase?

woops sorry @vasqu , yes rebasing and seeing if it passes. Wanted to wait a bit for CI to green itself back

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: beit, convnext, convnextv2, lw_detr, pp_lcnet, pp_lcnet_v3, pp_lcnet_v4, segformer, swin, timm_backbone, uvdoc, vitpose_backbone

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46422&sha=ba53ad