Keep user-typed multimodal placeholders out of the placeholder expansion
Author: souvikDevloper
Pull request: yes
Labels: name: Code agent slop

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47314)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47314)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes #47217. A multimodal placeholder token typed by a user inside a text message is now kept as plain text, instead of being treated as a reference to an image.

```py
messages = [{"role": "user", "content": [
    {"type": "text", "text": "How do I convert this to an <image> tag?"},
    {"type": "image", "url": url},
]}]
processor.apply_chat_template(messages, tokenize=True)
# before: ValueError: The number of images in the text [2] and images [1] should be the same.
# after : the literal "<image>" is tokenized as plain text, the single image is expanded once
```

As @samuelstevens noted in the issue, this affects many VLMs, not just LFM2-VL.

## The problem

`apply_chat_template` renders the conversation to a plain string and hands it to `processor.__call__`, which counts and splits on the placeholder token. Once rendered, the `<image>` a template emitted for a real image and the `<image>` a user typed in their message are the same seven characters, so the second one is expanded as if it referenced an image — and, once past the count check, tokenized as the image token, leaving a placeholder id with no image behind it.

## The fix

Images, videos and audio are passed as their own content blocks, so a placeholder found inside a `"text"` block is text the user meant literally. `ProcessorMixin.apply_chat_template` now escapes those occurrences (and only those) before rendering, and both tokenizer backends encode escaped spans as plain text:

- `tokenization_utils_base.py`: escaping maps each character into a reserved Private Use Area block. It is per-character, so it is length-preserving and character offsets into the rendered prompt stay valid, and the result matches no special token, so processors no longer count or split on it.
- `processing_utils.py`: `apply_chat_template` escapes the tokens of `all_special_multimodal_tokens` found in text blocks, when `tokenize=True`. Messages are copied on write, so the caller's conversation is untouched.
- `tokenization_utils_tokenizers.py` / `tokenization_python.py`: escaped spans are unescaped and encoded with `split_special_tokens=True`, the rest of the sequence as usual — the per-segment tokenization suggested in the issue, without a change to the `apply_chat_template` signature. In the fast backend the chunks are encoded with truncation and padding off, merged, and then post-processed, so special tokens, truncation and padding still apply to the whole sequence.

Because this hangs off `all_special_multimodal_tokens` rather than off any one processor, it holds for the processors that override `__call__` or `apply_chat_template` too (SmolVLM, LFM2-VL, ...).

## Tests

- `tests/test_processing_common.py`: `test_apply_chat_template_literal_image_token`, in the shared mixin, so it runs for every processor. It asserts a literal placeholder adds no image placeholder to the prompt and survives as text.
- `tests/utils/test_tokenization_utils.py`: `LiteralTextEscapingTest` covers the escaping round trip and both backends, including padding and truncation of escaped text.

Verified end-to-end on the models from the issue thread — `LiquidAI/LFM2.5-VL-1.6B`, `Qwen/Qwen3-VL-2B-Instruct`, `HuggingFaceTB/SmolVLM2-256M-Video-Instruct`, `OpenGVLab/InternVL3-1B-hf` — each expands exactly one image and keeps the typed placeholder as text.

## Notes for review

- **Tokenizer hot path.** `__call__` now sniffs the text for escaped spans. Escaped characters are never ASCII and `str.isascii()` is O(1), so ASCII text short-circuits: measured overhead is 0.05% of tokenization time on a 512x1.1k-char ASCII batch, and 3.4% on a non-ASCII (CJK) one, where the scan actually runs. If that is too much, the alternative is to have the processor signal the escaping explicitly, but the sniffing is what makes this work without touching each of the ~100 processors.
- **`tokenize=False` is unchanged.** A plain string cannot carry the distinction, which is the crux of the issue, so escaping is limited to the `tokenize=True` path that produces the model inputs. Text content passed as a bare string (rather than as content blocks) is also left alone, since there a `<image>` may well be the user marking where their image goes.
- Placeholder tokens are assumed to be Latin-1; anything else is left unescaped rather than mangled.

## Who can review?

@zucchini-nlp, as the multimodal maintainer, and cc @samuelstevens / @lcheng321 who diagnosed this in the issue.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: colqwen2

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47314)
**Latest run:** [29321741067:2](https://github.com/huggingface/transformers/actions/runs/29321741067)
**Result:** `success` | **Jobs:** 15 | **Tests:** 171,143 | **Failures:** 0 | **Duration:** 14h 37m

<!-- ci-dashboard-recap:end -->