Prefill-related logic in input preparation for generation
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/41863 and fixes https://github.com/huggingface/transformers/issues/40910

We always have had an imperfect way to infer if we're in prefill or decoding stage, which caused us many bugs in the past. The most reliable way is to check cache position values but it is not compile-compatible and also has an edge case

Recently Manuel merged a PR to split prefill into its own function so now we can benefit from it and know with 100% certainty which stage we're in. This PR adds `is_first_iteration` flag to generation input preparation and replaces existing logic with the flag

Note in some models, we have to keep checking `if cache_position[0] == 0` because first iteration does not mean first in total. We might get cached system prompt and we don't want to call some methods a second time (e.g. Qwen mRoPE) 

Also it adds a test case for the above linked issue

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42088). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Another worm of cans, assisted decoding has no prefill separated out and is causing issues now 😢 

> Another worm of cans, assisted decoding has no prefill separated out and is causing issues now 😢

worm of cans?? 🤣 haha love it

Sooo this already arose on my PR. The main gist is that assisted generate does not prefill with the prompt tokens, but waits for the first batch of candidates and then prefills. Thus, we could not apply the standard prefill. But surely assisted_gen can pass the prefill flag on the first call, or we can also maybe call _prefill with the first batch of candidates.

> assisted_gen can pass the prefill flag on the first call

yeah, this seemed to be the easiest option. The only issue with VLMs is that we should not be passing certain inputs (pixels/etc) after a prefill phase. But with assistant model calling `generate()` many times internally, we end up with several "prefill" phases

Support for `continue_generation_from_past_cache` is making it hard 😭  The concept of `is_prefill` and `is_first_iteration` can be different in some models and require different input preparation

I don't want us to multiplicate number of input args for `prepare_inputs_generation` so I might leave the `cache_position[0] == 0` as the ultimate source of truth for prefill, and pass only `is_first_iteration` as an argument.

@bot /style

Style fix runs successfully without any file modified.

run-slow: git

This comment contains `run-slow`, running the specified jobs: 

models: ["models/git"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19636191473)

### Model CI Report

#### ❌ Failed tests

- [git](https://github.com/huggingface/transformers/actions/runs/19636191473/job/56228051377):
    tests/models/git/test_modeling_git.py::GitModelIntegrationTest::test_batched_generation
    tests/models/git/test_modeling_git.py::GitModelIntegrationTest::test_inference_image_captioning



@vasqu requesting one last review. One tiny thing left to do is to make sure slow GIT tests pass, some expected values are hardware-dependent

Otherwise should be ready, addressed a few comments and answered q above

run-slow: git

This comment contains `run-slow`, running the specified jobs: 

models: ["models/git"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19961857384)

### Model CI Report

#### ❌ Failed tests

- [git](https://github.com/huggingface/transformers/actions/runs/19961857384/job/57244428243):
    tests/models/git/test_modeling_git.py::GitModelIntegrationTest::test_batched_generation



run_slow: git, csm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/csm", "models/git"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20099047354)

✅ No failing test specific to this PR 🎉 !


Will merge when tokenization tests are fixed on main

No worries, kinda been this way the past few days :cry: 

@bot /repo

Repo. Consistency fix runs successfully without any file modified.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, aya_vision, bamba, bloom, chameleon, clvp, cohere2_vision, csm, ctrl, deepseek_vl, deepseek_vl_hybrid, emu3, falcon_h1, falcon_mamba, fast_vlm, florence2