Raise early for unsupported QuantizedCache layer types
Author: Sunt-ing
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46781)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46781)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

`QuantizedCache` currently builds one quantized key/value cache layer per hidden layer, without checking `config.layer_types`:

```python
layers = [
    layer_class(nbits, axis_key, axis_value, q_group_size, residual_length)
    for _ in range(config.num_hidden_layers)
]
```

That works for pure attention models, but hybrid models also contain Mamba, linear-attention, or hybrid state layers that are not plain attention key/value caches. Passing `cache_implementation="quantized"` for those models currently lets the invalid cache get created and then fails later with confusing state-cache errors.

This keeps `QuantizedCache` scoped to attention key/value cache layers. If `config.layer_types` contains a layer type that does not map to a key/value cache layer, `QuantizedCache` now raises during initialization with a clear error asking users to use the default cache implementation instead. Pure attention configs keep the previous behavior: every layer still maps to the selected quantized cache layer.

<details>
<summary>Original crash that motivated the guard</summary>

Built each model from its own `ModelTester.get_config()` (legitimate hybrid configs, no hand-written tiny configs), float16, single RTX 4090. Each model was generated greedily with `cache_implementation="quantized"` using both `quanto` and `hqq`.

Environment: `transformers` 5.13.0.dev0, `torch` 2.8.0+cu128, Python 3.12, Linux, single RTX 4090, `optimum-quanto` and `hqq`.

```python
import importlib, torch
from transformers import AutoModelForCausalLM

class P:
    def __getattr__(self, k): return lambda *a, **kw: None

cfg = getattr(importlib.import_module("tests.models.zamba2.test_modeling_zamba2"),
              "Zamba2ModelTester")(P()).get_config()
cfg.use_cache = True
m = AutoModelForCausalLM.from_config(cfg).to(torch.float16).cuda().eval()
ids = torch.randint(5, 200, (1, 10)).cuda()

with torch.no_grad():
    m.generate(ids, max_new_tokens=12, do_sample=False, cache_implementation="quantized",
               cache_config={"backend": "quanto", "nbits": 4})
```

On main, the same root cause shows up across the tested hybrid families:

| model | result on main |
|---|---|
| Zamba2 | `ValueError: You called has_previous_state on layer index N, but this layer is an Attention layer...` |
| LFM2 | `ValueError: You called has_previous_state on layer index N, but this layer is an Attention layer...` |
| Qwen3-Next | `ValueError: has_previous_state can only be called on LinearAttention layers...` |
| Nemotron-H | `ValueError: has_previous_state can only be called on LinearAttention layers...` |
| Falcon-H1 | `ValueError: has_previous_state can only be called on LinearAttention layers...` |
| Jamba | `ValueError: has_previous_state can only be called on LinearAttention layers...` |

The failure happens because state layers that are not key/value cache layers are represented as quantized attention cache layers. This PR now rejects that unsupported cache layout before generation reaches those state accesses.
</details>

Added `test_quantized_cache_rejects_non_quantizable_layer_types` in `tests/utils/test_cache_utils.py`. It asserts that mixed `full_attention`/`mamba`/`hybrid` layer types raise early, while a pure `full_attention` config still builds quantized cache layers.

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://huggingface.co/docs/transformers/contributing) and the
      [Pull Request](https://huggingface.co/docs/transformers/pr_checks) checks?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes according to the [guidelines](https://github.com/huggingface/transformers/tree/main/docs)?
- [x] Did you write any new necessary [tests](https://huggingface.co/docs/transformers/testing)?

## Who can review?

@Cyrilvallez

Comments:
Hey @Sunt-ing, I would just raise early if any layer type cannot be quantized. I don't think quantized cache is really used

Thanks @Cyrilvallez. I updated `QuantizedCache` to raise early for non-quantizable layer types, while keeping pure `full_attention` configs on the existing path. Updated the PR description accordingly.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46781)
**Latest run:** [28623693333:2](https://github.com/huggingface/transformers/actions/runs/28623693333)
**Result:** `success` | **Jobs:** 14 | **Tests:** 47,278 | **Failures:** 0 | **Duration:** 18h 57m

<!-- ci-dashboard-recap:end -->

Superseded by https://github.com/huggingface/transformers/pull/47160