VL models fail when their image placeholder is in user text.
Author: samuelstevens
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.13.0
- Platform: macOS-26.3.1-arm64-arm-64bit-Mach-O
- Python version: 3.14.3
- Huggingface_hub version: 1.23.0
- Safetensors version: 0.8.0
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): not installed (NA)
- Using distributed or parallel set-up in script?: No

### Who can help?

@zucchini-nlp is the listed multimodal maintainer. 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# /// script
# requires-python = ">=3.14"
# dependencies = [
#     "torchvision>=0.28.0",
#     "transformers>=5.13.0",
# ]
# ///
import transformers


def main() -> None:
    processor = transformers.AutoProcessor.from_pretrained("LiquidAI/LFM2.5-VL-1.6B")

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "How do I convert this to an <image> tag?"},
                {
                    "type": "image",
                    "url": "https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/2b08LKpev0DNEk6DlnWkY.png",
                },
            ],
        }
    ]

    print(processor.apply_chat_template(messages))

    # Throws ValueError: The number of images in the text [2] and images [1] should be the same.
    print(processor.apply_chat_template(messages, tokenize=True))


if __name__ == "__main__":
    main()
```

Run with `uv run repro.py`.

### Expected behavior

No failure, and correct tokenization.

Comments:
Actually, this is a bug with many popular VLMs. There's no way to include the image format in the user messages with the current "structured messages" -> `apply_chat_template()` -> string -> `__call__()` -> tokens` pipeline. See my expanded reproduction script below.

```py
# /// script
# requires-python = ">=3.14"
# dependencies = [
#     "torchvision>=0.28.0",
#     "transformers>=5.13.0",
#     "num2words>=0.5.13",
# ]
# ///
import transformers


def main() -> None:

    models = [
        ("LiquidAI/LFM2.5-VL-1.6B", "<image>"),
        ("Qwen/Qwen3-VL-2B-Instruct", "


"),
        ("Qwen/Qwen3.5-2B", "


"),
        ("HuggingFaceTB/SmolVLM2-256M-Video-Instruct", "<image>"),
        ("google/gemma-4-E2B-it", "
"),
        ("OpenGVLab/InternVL3-1B-hf", "<img><IMG_CONTEXT></img>"),
        ("OpenGVLab/InternVL3_5-1B-HF", "<img><IMG_CONTEXT></img>"),
    ]

    for model, special_tok in models:
        processor = transformers.AutoProcessor.from_pretrained(model)

        messages = [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"How do I format your input with {special_tok}?"},
                    {
                        "type": "image",
                        "url": "https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/2b08LKpev0DNEk6DlnWkY.png",
                    },
                ],
            }
        ]

        chat_msg = processor.apply_chat_template(messages)

        try:
            print(processor.apply_chat_template(messages, tokenize=True))
        except Exception as err:
            print(f"FAILED: Model {model} failed with exception {type(err).__name__}: '{err}'")


if __name__ == "__main__":
    main()
```

I think we could solve this by tokenizing with `split_special_tokens=True` on user message content, and with `split_special_tokens=False` on the rest of the chat template. I don't think I can fix this with a PR; we can instead adjust the special token and chat template to not use `<image>` for the image token in LFM2 and LFM2.5 VL models, which will make it more similar to Qwen and Gemma.

It seems that both Prime Intellect and Thinking Machines developed solutions to these problems (https://www.primeintellect.ai/blog/renderers and https://tinker-docs.thinkingmachines.ai/tutorials/core-concepts/rendering/). Curious if HF is aware of these issues and has any plans?

Confirmed on 5.13.0, CPU only, deterministic repro. Root cause is the lossy messages to string round trip: in processing_lfm2_vl.py the processor string-counts (L103) and string-splits (L154) on the placeholder, so a user-typed `<image>` is indistinguishable from the real one. Narrow fix is swapping the LFM2/LFM2.5-VL placeholder to a non-collidable sentinel; the general fix is tracking image positions structurally instead of via the string. @zucchini-nlp happy to send the narrow PR with a regression test if the direction is welcome.

Hi @samuelstevens @zucchini-nlp — I'd like to work on this if the direction and ownership are welcome.

I see @lcheng321 has already offered a narrow LFM2/LFM2.5-VL fix, so I don't want to duplicate work. My proposed scope would be a general structural fix that keeps user-typed placeholder text distinct from actual multimodal content through `apply_chat_template`, with focused regression tests across the affected processor patterns.

@lcheng321, are you already working on this? If a maintainer prefers the narrow fix, or would like you to own it, I'll defer. I won't start implementation or open a PR until ownership and scope are confirmed.

I don't think you can implement a general structural fix without a change to the function signature. As far as I can tell, you need to somehow get some metadata out of apply_chat_template, but the signature is something like

```py
def apply_chat_template(
    conversation: list[dict[str, Any]] | list[list[dict[str, Any]]],
    ...
    **kwargs: Any,
) -> str | list[int] | list[str] | list[list[int]] | BatchEncoding:
```

And str, list[int], list[str], etc don't offer any way to separate user content from the ChatML-style strings surrounding the user content.

> Hi [@samuelstevens](https://github.com/samuelstevens) [@zucchini-nlp](https://github.com/zucchini-nlp) — I'd like to work on this if the direction and ownership are welcome.
> 
> I see [@lcheng321](https://github.com/lcheng321) has already offered a narrow LFM2/LFM2.5-VL fix, so I don't want to duplicate work. My proposed scope would be a general structural fix that keeps user-typed placeholder text distinct from actual multimodal content through `apply_chat_template`, with focused regression tests across the affected processor patterns.
> 
> [@lcheng321](https://github.com/lcheng321), are you already working on this? If a maintainer prefers the narrow fix, or would like you to own it, I'll defer. I won't start implementation or open a PR until ownership and scope are confirmed.

@souvikDevloper I only floated the diagnosis, haven't started any implementation. Go ahead and take the general structural fix. I'll leave the narrow LFM2 placeholder swap for whoever gets to it, no claim on it from me.

Hey @samuelstevens 

It is not really clear to me why the input might have `<image>`  token embedded instead of letting processor's jinja template do its work. Becuase even if we were to allow this pattern, we still wouldn't be able to use a model's forward as it also checks `num_image_tokens == num_placeholders`

I think the strict validation on number of placeholders vs mm-data is valid and tbh I am pushing for all models to check it, since currently some models just let it happen 

Especially for `<image>`, the user might be asking about `<image>` tags in HTML/XML. But even for `
` and more obscure image tokens, for a sufficiently large VLM, you might chat with the VLM about its own chat template. So I don't think it's impossible that an image placeholder string shows up in the user message. 

> I think the strict validation on number of placeholders vs mm-data is valid and tbh I am pushing for all models to check it, since currently some models just let it happen

My point is that strict validation is also wrong: if a user includes 1 image placeholder token in their message which also includes 1 real image, then there will be 2 placeholder tokens in the rendered chat template and only 1 image, but this *should be valid*, because the user is actually discussing the placeholder token. 

Hmm interesting! @Rocketknight1 will be working on encoding these special tokens separately which should solve the issue