GLM4.1v  video inference bug
Author: aminer-404
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.1
- Platform: Linux-5.15.0-160-generic-x86_64-with-glibc2.35
- Python version: 3.12.9
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.5.3
- Accelerate version: 1.7.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

Models:
multimodal models: @zucchini-nlp 


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoProcessor, Glm4vForConditionalGeneration
import torch

processor = AutoProcessor.from_pretrained("GLM-4.1V-9B-Thinking")
model = Glm4vForConditionalGeneration.from_pretrained("GLM-4.1V-9B-Thinking", torch_dtype=torch.bfloat16,device_map="cuda:0")

messages = [{"role": "user", "content": [{"type": "video", "url": "/path/to/some.mp4"}, {"type": "text", "text": "describe the video"}]}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=512)

output = processor.decode(generated_ids[0], skip_special_tokens=True)
print(output)
```

it turns out extra numbers at the begin of output
```
01234567891011121314151617181920212223242526272829describe the video
<think>xxxxx</think><answer>xxxxxx
```

### Expected behavior

normal thinking model output 
```
<think>xxxxx</think><answer>xxxxxx
```



Comments:
@aminer-404 those are the timestamps of each video frame, no bug :) Model was trained with timestamps

> [@aminer-404](https://github.com/aminer-404) those are the timestamps of each video frame, no bug :) Model was trained with timestamps

But there are no these tokens in  [tokenizer.json](https://huggingface.co/zai-org/GLM-4.1V-9B-Thinking/blob/main/tokenizer.json). The decode/encode process may cause some issues, especially during training phase using transformers  library.  [ref ](https://github.com/modelscope/ms-swift/blob/bf0257370abf5eddab7814fef166d87d0e25d557/swift/llm/template/template/glm.py#L223)

Moreover, the default output [output = processor.decode(generated_ids[0], skip_special_tokens=True)]  is expected to be more 'clean' output without these extra numbers, just as other MLLMs. 

###  @**_

![Image](https://github.com/user-attachments/assets/f2243e68-4612-41c0-bf91-f354bc0ea986)

_**

@aminer-404 these tokens are added when applying chat template and preprocessing inputs, by manually inspecting each input video. Each video is sampled first and the model needs to know which timestamps are sampled. They cannot be cleaned with `skip_special_tokens` since these are simple numbers and aren't special tokens, thus can be encoded in multiple ways for each timestamp

Technically we can clean up those token with regex, though I didn't quite get how it's affecting training. Could you give an example, maybe it's doable without an update?

> [@aminer-404](https://github.com/aminer-404) these tokens are added when applying chat template and preprocessing inputs, by manually inspecting each input video. Each video is sampled first and the model needs to know which timestamps are sampled. They cannot be cleaned with `skip_special_tokens` since these are simple numbers and aren't special tokens, thus can be encoded in multiple ways for each timestamp
> 
> Technically we can clean up those token with regex, though I didn't quite get how it's affecting training. Could you give an example, maybe it's doable without an update?

I can't train 4.1v since popular framework like 'ms-swift', 'llama-factory' have not support training 4.1v with videos.   And one of the reasons is there maybe bugs in  transforms as noted [here1](https://github.com/modelscope/ms-swift/blob/bf0257370abf5eddab7814fef166d87d0e25d557/swift/llm/template/template/glm.py#L223) and [here2](https://github.com/modelscope/ms-swift/issues/6528).  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.