[GLM-4.5V] [vLLM Video inference] 
 token doesn't exist in the inputs after data processing

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info


- `llamafactory` version: 0.9.4.dev0
- Platform: Linux-6.4.3-0_fbk15_hardened_2630_gf27365f948db-x86_64-with-glibc2.34
- Python version: 3.12.11
- PyTorch version: 2.7.1+cu128 (GPU)
- Transformers version: 4.56.0.dev0
- Datasets version: 3.6.0
- Accelerate version: 1.7.0
- PEFT version: 0.15.2
- TRL version: 0.9.6
- GPU type: NVIDIA PG509-210
- GPU number: 8
- GPU memory: 79.14GB
- vLLM version: 0.10.1.1
- Default data directory: detected

### Reproduction

I first added video metadata to the video outputs for GLM-4.5V, as GLM4.5V requires it.
```
        elif batch["videos"][j] is not None:
            video = batch["videos"][j]
            multi_modal_data = {
                "video": template_obj.mm_plugin._regularize_videos(
                    video,
                    image_max_pixels=image_max_pixels,
                    image_min_pixels=image_min_pixels,
                    video_fps=video_fps,
                    video_maxlen=video_maxlen,
                )["videos"]
            }
            if "GLM-4.5V" in model_args.model_name_or_path:
                video_data = multi_modal_data["video"]
                for k, each in enumerate(video_data):
                    video_metadata = {
                        "fps": video_fps,
                        "total_num_frames": len(each),
                        "duration": len(each) / video_fps,
                        "video_backend": "opencv"
                    }
                    video_data[k] = (each, video_metadata)
                multi_modal_data["video"] = video_data
```


Then I ran the following command to do inference.
```
DISABLE_VERSION_CHECK=1 python scripts/vllm_infer.py \
--model_name_or_path=zai-org/GLM-4.5V \
--dataset=mllm_video_demo \
--save_name=/tmp/generations_glm_4.5v.jsonl \
--cutoff_len=8192 \
--max_new_tokens=1024 \
--top_p=0.9 \
--temperature=0.1 \
--template=glm4v_moe \
```


However, it would raise the following error.

```
Traceback (most recent call last):
  File "/data/repos/fbsource/third-party/LLaMA-Factory/scripts/vllm_infer.py", line 309, in <module>
    fire.Fire(vllm_infer)
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/data/repos/fbsource/third-party/LLaMA-Factory/scripts/vllm_infer.py", line 252, in vllm_infer
    results = llm.generate(vllm_inputs, sampling_params, lora_request=lora_request)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/utils/__init__.py", line 1557, in inner
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 488, in generate
    self._validate_and_add_requests(
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1666, in _validate_and_add_requests
    self._add_request(
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 1684, in _add_request
    self.llm_engine.add_request(
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 206, in add_request
    prompt_str, request = self.processor.process_inputs(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/v1/engine/processor.py", line 258, in process_inputs
    processed_inputs: ProcessorInputs = self.input_preprocessor.preprocess(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/inputs/preprocess.py", line 857, in preprocess
    return self._process_decoder_only_prompt(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/inputs/preprocess.py", line 806, in _process_decoder_only_prompt
    prompt_comps = self._prompt_to_llm_inputs(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/inputs/preprocess.py", line 490, in _prompt_to_llm_inputs
    return self._process_tokens(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/inputs/preprocess.py", line 345, in _process_tokens
    inputs = self._process_multimodal(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/inputs/preprocess.py", line 271, in _process_multimodal
    return mm_processor.apply(prompt,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/multimodal/processing.py", line 1711, in apply
    prompt_ids, prompt, mm_placeholders = self._maybe_apply_prompt_updates(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/multimodal/processing.py", line 1667, in _maybe_apply_prompt_updates
    self._validate_mm_placeholders(mm_placeholders, mm_item_counts)
  File "/home/zihaoh/.conda/envs/vllm/lib/python3.12/site-packages/vllm/multimodal/processing.py", line 1619, in _validate_mm_placeholders
    raise RuntimeError(
RuntimeError: Expected there to be 1 prompt updates corresponding to 1 video items, but instead found 0 prompt updates! This is likely because you forgot to include input placeholder tokens (e.g., `<image>`, `
`) in the prompt. If the model has a chat template, make sure you have applied it before calling `LLM.generate`.

```

I checked the processed dataset, and "
" token does NOT exist in the prompts. Only 
 and 
 were there. Even after setting ```expand_mm_tokens``` to True, the error persisted. 

In that case, could you please provide a script for running batch inference of GLM4.5V using vLLM? Thanks!


### Others

_No response_

Labels: solved
State: closed