`apply_chat_template(return_assistant_tokens_mask=True)` returns an empty assistant mask for VLM processors when an image is present
Author: qgallouedec
Pull request: no

### System Info

- `transformers` version: 5.10.1
- Platform: Linux
- Python: 3.13

@zucchini-nlp

### Description

For a multimodal processor, `apply_chat_template(..., return_assistant_tokens_mask=True)` produces a correct assistant mask for a text-only conversation, but an **all-zero (empty)** mask as soon as a real image is included in the conversation (even though the assistant turn is unchanged and is correctly wrapped in `{% generation %}` / `{% endgeneration %}` markers)

### Reproduction

```python
from PIL import Image
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

# Minimal chat template with `{% generation %}` markers around the assistant turn.
chat_template = (
    "{% for message in messages %}"
    "
{{ message['role'] }}\n"
    "{% if message['role'] == 'assistant' %}{% generation %}"
    "{% for c in message['content'] %}{% if c['type'] == 'image' %}


"
    "{% elif c['type'] == 'text' %}{{ c['text'] }}{% endif %}{% endfor %}"
    "
\n{% endgeneration %}"
    "{% else %}"
    "{% for c in message['content'] %}{% if c['type'] == 'image' %}


"
    "{% elif c['type'] == 'text' %}{{ c['text'] }}{% endif %}{% endfor %}"
    "
\n{% endif %}"
    "{% endfor %}"
)


def assistant_token_count(messages):
    out = processor.apply_chat_template(
        messages,
        chat_template=chat_template,
        tokenize=True,
        return_dict=True,
        return_assistant_tokens_mask=True,
    )
    return sum(out["assistant_masks"][0])


text_only = [
    {"role": "user", "content": [{"type": "text", "text": "Hi"}]},
    {"role": "assistant", "content": [{"type": "text", "text": "Hello there"}]},
]
with_image = [
    {
        "role": "user",
        "content": [{"type": "image", "image": Image.new("RGB", (56, 56))}, {"type": "text", "text": "Hi"}],
    },
    {"role": "assistant", "content": [{"type": "text", "text": "Hello there"}]},
]

print("assistant tokens (text-only) :", assistant_token_count(text_only))   # 4 (correct)
print("assistant tokens (with image):", assistant_token_count(with_image))  # 0 (BUG)
```

Output:

```
assistant tokens (text-only) : 4
assistant tokens (with image): 0
```

### Expected behavior

The assistant mask should cover the same assistant tokens (`Hello there
\n`) regardless of whether an image is present in an earlier turn.

Confirmed on Qwen2.5-VL; the same offset-misalignment affects other processors that expand image placeholders at `__call__` time (Llava, Gemma3, Qwen2-VL).


Comments:
sorry, actually a duplicate of https://github.com/huggingface/transformers/issues/44521

😄 

yep, it needs a way to tell how many `placeholder` tokens were added to adjust the offsets. I recently added an option for models to return such metadata, which imo is a an easier option than calling `tokenizer.call` twice. Also it will be useful downstream in vllm as well

Still have to update all processors first before jumping to fix the chat template (WIP in https://github.com/huggingface/transformers/pull/46556)