Fix GLM video frame padding for temporal patches
Author: labAxiaoming
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47141)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47141)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This PR fixes video frame padding in the GLM-family video processors.

The affected processors currently repeat `temporal_patch_size - 1` frames
whenever the number of video frames is not divisible by `temporal_patch_size`.
That only works for the default `temporal_patch_size = 2`; for larger temporal
patch sizes it can still leave the padded frame count indivisible.

For example, with `num_frames = 18` and `temporal_patch_size = 4`, the current
logic pads to `21` frames, which is still not divisible by `4`. The new logic
matches the Qwen2-VL/Qwen3-VL video processors and pads to the next valid
multiple instead:


https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/qwen3_vl/video_processing_qwen3_vl.py#L230-L232
```python
T = patches.shape[1]
if pad := -T % temporal_patch_size:
    repeats = patches[:, -1:].expand(-1, pad, -1, -1, -1)
    patches = torch.cat((patches, repeats), dim=1)
```

This keeps the default GLM configurations unchanged while making the padding
logic correct for non-default temporal patch sizes.

Affected processors:

- `Glm4vVideoProcessor`
- `Glm46VVideoProcessor`
- `GlmgaVideoProcessor`
- `VideoLlama3VideoProcessor`

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


- vision models: @yonigozlan @molbap
- multimodal models: @zucchini-nlp

Comments:
> Can you pls check that no models have incorrect padding, we don't want many PRs fixing the same bug

I checked all `video_processing_*.py` files. Besides the GLM-family processors, I found the same issue in VideoLLaMA3 and fixed it as well.

> > Can you pls check that no models have incorrect padding, we don't want many PRs fixing the same bug
> 
> I checked all `video_processing_*.py` files. Besides the GLM-family processors, I found the same issue in VideoLLaMA3 and fixed it as well.


image has 2 too 
https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/paddleocr_vl/image_processing_paddleocr_vl.py#L193-L195
https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py#L305-L307

in image only 1 frame, so this logit is ok


Hmm, images shouldn't need that part at all as they have no frames. Lets delete it completely from image processors pls 

> Hmm, images shouldn't need that part at all as they have no frames. Lets delete it completely from image processors pls

I have not modify image processors  


For image processors, the similar-looking code is not really padding video frames. The image path creates a fake temporal axis with T=1 because the vision patch format still expects `channel * temporal_patch_size * patch_size * patch_size`. Some processors materialize that by repeating/padding the single image to `temporal_patch_size`; Qwen2-VL does the same thing more directly with `expand`. 

https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/glm46v/image_processing_glm46v.py#L184-L187
https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L210-L219
﻿https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/models/paddleocr_vl/modular_paddleocr_vl.py#L305-L307

So for images:
- there are no variable-length frames to align;
- multi-image batching is on batch dim, not temporal dim;
- T starts as 1, so expand to temporal_patch_size ;  padding repeated `temporal_patch_size - (T % temporal_patch_size)` or padding repeated `temporal_patch_size - 1` are equivalent there.
﻿

ref.
## Qwen2 vl
 https://arxiv.org/pdf/2409.12191 page 5
### UnifiedImageandVideoUnderstanding 
For consistency, each image is treated as two identical frames. 
## GLM
 https://arxiv.org/pdf/2507.01006  page 3    
### 2 Overview and Architecture
This enables temporal downsampling by a factor of two for video inputs,
thereby improving model efficiency. For single-image inputs, the image is duplicated to maintain
consistency.





> Hmm, images shouldn't need that part at all as they have no frames. Lets delete it completely from image processors pls

should i process image preprocess or not 

run-slow: glm46v, glm4v, glmga, video_llama_3

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/29501937507)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/glm46v", "models/glm4v", "models/glmga", "models/video_llama_3"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47141). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/29501937507)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [649ad307](https://github.com/huggingface/transformers/commit/649ad307d1299f77dc8dae8c77f2a1c3409800c5) | workflow commit (merge commit) |
| PR | [e66b135d](https://github.com/labAxiaoming/transformers/commit/e66b135d35f468bbce342739b94e8b1f631978aa) | branch commit (from PR) |
| main | [01b03b2e](https://github.com/huggingface/transformers/commit/01b03b2e9fbc5de0d62d080026be087998035c10) | base commit (on `main`) |

⚠️ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/29501937507) for details.


> run-slow: glm46v, glm4v, glmga, video_llama_3

The two CI failures are maybe unrelated to the preprocessing changes in this PR:

test_multi_gpu_data_parallel_forward –  caused by DataParallel parameter replication issues on multi‑GPU setups.

Glm46VIntegrationTest – a test harness bug that incorrectly casts input_ids to float16 before passing to torch.nn.functional.embedding.

So I rebased  the latest upstream main, avoid may cause by old code

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29548688826) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm46v, glm4v, glmga, video_llama_3

Yeah, seems unrelated though I am not liking the errors on GLM46V. Let's merge and I will check/fix glm46V myself

Update: will ask GLM team to fix on the hub, smth weird with model type

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47141)
**Latest run:** [28928092574:2](https://github.com/huggingface/transformers/actions/runs/29548689362)
**Result:** `success` | **Jobs:** 3 | **Tests:** 93 | **Failures:** 0 | **Duration:** 2m 10s

<!-- ci-dashboard-recap:end -->