Fix: honor cache_config["max_cache_len"] on the static-cache path
Author: bojanstef
Pull request: yes
Labels: name: Code agent slop

Fixes #46424 x

## What

`GenerationConfig.cache_config` is documented as carrying "arguments used in the key-value cache class", but on the **static**-cache path it is silently ignored. Setting `cache_config={"max_cache_len": N}` together with `cache_implementation="static"` produces a cache sized to `max_length - 1`, not `N`:

```python
gen = GenerationConfig(max_new_tokens=16, cache_implementation="static",
                       cache_config={"max_cache_len": 2048}, do_sample=False)
model.generate(input_ids, generation_config=gen)   # prompt_len = 32
model._cache.max_cache_len      # 47  (== max_length-1), not the requested 2048
```

The practical cost (from the issue): a second `generate()` with a larger `max_new_tokens` pays a full `StaticCache` realloc **+ `torch.compile` recompile**, because the auto-allocated cache was sized for the first call. The documented way to avoid that — pin the size via `cache_config` — is exactly the path that doesn't work today.

## Root cause

In `GenerationMixin._prepare_cache_for_generation` (`src/transformers/generation/utils.py`), `cache_config` is read **only** inside the `elif generation_config.cache_implementation == "quantized"` branch. The static branch sizes the cache straight from the `max_cache_length` argument (`= generation_config.max_length - 1`, computed by the caller) and never consults `cache_config`:

```python
if generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS:
    ...
    model_kwargs[cache_name] = self._prepare_static_cache(
        ...
        max_cache_len=max_cache_length,   # <-- cache_config never read here
    )
elif generation_config.cache_implementation == "quantized":
    ...
    cache_config = generation_config.cache_config if ... else {}   # <-- only reader
```

A runtime trace of the repro confirms it: `max_cache_length` arrives as `47`, the static branch is taken, and **neither** the quantized-branch condition **nor** the `cache_config` read ever executes (trace below).

## When it was introduced (not a regression)

`git bisect` has no "good" commit here — the static path has **never** honored `cache_config`. `cache_config` was introduced as a *QuantizedCache* config object, and every refactor since has kept it quantized-only:

```
$ git log -S cache_config -- src/transformers/generation/utils.py   # earliest
d583f1317b  2024-05-23  Quantized KV Cache (#30483)              # cache_config is born, quantized-only
1687954f5a  2026-02-04  🚨 Generation cache preparation (#43679) # current branch structure
```

At both #43679 and its parent commit, `cache_config` is read only on the quantized branch. So this is a long-standing gap dating to the field's origin (#30483), not a regression.

## Fix

Thread `cache_config["max_cache_len"]` into the static path, guarded with `max()` so the cache is never under-sized for the current call:

```python
if generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS:
    ...
    static_max_cache_len = max_cache_length
    requested_max_cache_len = (generation_config.cache_config or {}).get("max_cache_len")
    if requested_max_cache_len is not None:
        static_max_cache_len = max(max_cache_length, requested_max_cache_len)
    model_kwargs[cache_name] = self._prepare_static_cache(
        ...
        max_cache_len=static_max_cache_len,
    )
```

When `cache_config` is `None` or carries no `max_cache_len`, the value is unchanged — **no behavior change for the default path**. `max(...)` means an explicit-but-too-small `max_cache_len` can never shrink the cache below what the current call needs.

## Tests

Added `tests/utils/test_cache_utils.py::CacheTest::test_static_cache_honors_cache_config_max_cache_len` (in the existing no-download / no-GPU `CacheTest` class): builds a tiny random Llama, runs `generate()` with `cache_implementation="static"` and `cache_config={"max_cache_len": 2048}`, and asserts the allocated cache is sized to `2048`.

- **With this fix:** passes.
- **Without it:** fails with `AssertionError: 47 != 2048` (the bug).

<details>
<summary>Runtime trace — <b>before</b> the fix (the diagnosis)</summary>

Captured by instrumenting `src/transformers/generation/utils.py` and running the repro. `file:line` are pristine source coordinates.

```
#10  utils.py:2185  → generate(…)  generation_config = { "cache_config": {"max_cache_len": 2048}, "cache_implementation": "static", … }
#190 utils.py:2549  max_cache_length = 47                                                       # = max_length - 1
#192 utils.py:1861  → _prepare_cache_for_generation(… max_cache_length=47)
#207 utils.py:1922  generation_config.cache_implementation == "offloaded" = False
#208 utils.py:1925  generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS = True
#209 utils.py:1926  generation_config.cache_implementation in DEPRECATED_STATIC_CACHE_IMPLEMENTATIONS = False
#221 utils.py:1955  self.config.is_encoder_decoder = False
```

The complete set of events between lines 1920–1960 is the four shown above. There is **no event at line 1938** (`elif … == "quantized"`) and **no event at line 1945** (`cache_config = …`): the only code that reads `cache_config` is never reached on the static path, so the requested `2048` is structurally unreachable and `47` sizes the cache.

</details>

<details>
<summary>Runtime trace — <b>after</b> the fix (verification)</summary>

```
#208 utils.py:1925  generation_config.cache_implementation in ALL_STATIC_CACHE_IMPLEMENTATIONS = True
#210 utils.py:1936  static_max_cache_len    = 47       # starts at max_length - 1
#211 utils.py:1937  requested_max_cache_len = 2048     # static path now reads cache_config
#212 utils.py:1938  requested_max_cache_len is not None = True
#213 utils.py:1939  static_max_cache_len    = 2048     # max(47, 2048) → cache sized 2048
```

</details>

## Who can help

@ArthurZucker @Cyrilvallez

Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46700&sha=b2fb78

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46700)

Ready for review. The single failing check (`tests_processors` on CircleCI) is an unrelated network flake, not a regression from this change:

- **Test:** `tests/models/qwen3_vl/test_processing_qwen3_vl.py::Qwen3VLProcessorTest::test_apply_chat_template_video_frame_sampling`
- **Error:** `httpx.ConnectError: [SSL: UNEXPECTED_EOF_WHILE_READING]` while fetching a remote video asset

This diff only touches the static-cache path in `generation/utils.py` and adds a no-download test in `tests/utils/test_cache_utils.py`, with no connection to Qwen3-VL processing or any network I/O. All GitHub Actions checks and the other CircleCI jobs are green. Happy to rebase or re-push if you'd like a fresh CI run.


Immediate anti-spam block, sorry! There was a PR open at https://github.com/huggingface/transformers/pull/46446 for the same issue which your bot didn't bother to check, then it pinged two core maintainers to ensure it wasted the maximum amount of everyone's time.

@Rocketknight1 ah, I thought I checked but I clearly missed #46446 

In my defense I took care to reproduce the bug and include runtime traces

Thanks anways