Does Qwen2VLImageProcessor treat two consecutive images as one group/feature?
Author: priancho
Pull request: no

When looking at Qwen3-VL model's image processor (which uses Qwen2-VL's one), I found the following lines of code hard to understand.

`L296-300` checks the number of input images (`patches.shape[0]`), and repeat the last one to make it divisible by `temporal_patch_size`.
This make the model processes two consecutive images as a single feature due to the use of 3DConv with temporal_patch_size=2 by default.

https://github.com/huggingface/transformers/blob/76fc50a1527a7db593a6057903b749598f7000a9/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L293-L300

But as I understand, Qwen2-VL paper mentions that it repeats each input image `temporal_patch_size` times.
Did I misunderstand the code?.?

<img width="787" height="205" alt="Image" src="https://github.com/user-attachments/assets/fc697460-e0a2-49fa-99b8-ea3e733bb097" />



Comments:
You're not mistaken. The code actually works as you see it.

The short answer is: The paper and the code you quoted are talking about two different scenarios.

1. What the paper says: "Each image is treated as two identical frames."

Scenario: This is what happens when you feed the model a single static image.

Why: 3D Conv was designed for video (time dimension). If you feed it a single static image, it won't work. The model has to "pretend" to create a short video. By copying that image as [Image, Image] (if temporal_patch_size=2) to enable 3D Conv.

2. What the code does: if patches.shape[0] % temporal_patch_size != 0: (If the number of frames is not a multiple of 2)

Scenario: This is what happens when you input a video with multiple frames (e.g., 3, 5, or 7 frames).

Why: 3D Conv processes the video in chunks of 2 frames each.

If you input 4 frames: [f1, f2, f3, f4] -> the model processes (f1, f2) and (f3, f4) -> complete.

If you input 3 frames: [f1, f2, f3] -> the model processes (f1, f2)... Oh, and what about f3? It's left alone.

The L296-300 code you found "pads" or "fills" the missing frames. It copies the last frame (f3) to complete the block: [f1, f2, f3, f3] so that the model can process (f1, f2) and (f3, f3).

Summary:

Paper: Explains how to handle still image input (copying itself).

Code: Explains how to handle odd-numbered video input (copying the last frame).

You're not mistaken. You found the code for handling "video" while the paper is explaining the code for handling "still images."

```python
import numpy as np
# We use PIL to create dummy images for testing
from PIL import Image
from typing import List, Union

# Define the ChannelDimension enum as used in the original code
class ChannelDimension:
    FIRST = "channels_first"
    LAST = "channels_last"

def robust_qwen_vl_preprocessor(
    processed_images: List[np.ndarray], 
    temporal_patch_size: int = 2,
    data_format: str = ChannelDimension.LAST
) -> np.ndarray:
    """
    Robustly preprocesses a list of images for Qwen-VL 3D Conv.
    
    This function correctly handles both:
    1. Single static images (duplicates them per the paper).
    2. Video frame lists (pads the last frame per the code).
    """
    
    # --- 1. Error Handling (Dักจับ) ---
    # Catch empty or invalid inputs
    if not processed_images or not isinstance(processed_images, list) or len(processed_images) == 0:
        print("Warning: Input is not a valid list of images. Returning an empty array.")
        return np.array([])
    
    print(f"Received {len(processed_images)} frame(s).")
    
    # --- 2. Convert to NumPy array (Base) ---
    try:
        # This is L296 in your code: [Image, Image, ...] -> NumPy array
        patches = np.array(processed_images)
    except Exception as e:
        # This catches errors if images are not uniform in size
        print(f"Error: Could not convert list of images to NumPy array. Check if all images have the same dimensions. Details: {e}")
        return np.array([])

    # Ensure we have a 4D array (N, H, W, C) even for a single image
    if patches.ndim == 3:
        # Input was a single image (H, W, C), add a batch dimension
        patches = np.expand_dims(patches, axis=0)
        
    num_frames = patches.shape[0]

    # --- 3. The Core Logic (Flexible Handling) ---
    # This is where we solve the "Paper vs. Code" confusion.
    
    if num_frames == 1:
        # --- CASE A: STATIC IMAGE (What the paper described) ---
        # "each image is treated as two identical frames"
        # We duplicate the single image to create a "video" of N identical frames.
        print(f"Input is a single image. Repeating it {temporal_patch_size} times (Paper's logic).")
        patches = np.repeat(patches, temporal_patch_size, axis=0)
    
    elif num_frames > 1:
        # --- CASE B: VIDEO (What your code snippet handled) ---
        # We only PAD the end to make the total frame count divisible by temporal_patch_size.
        
        remainder = num_frames % temporal_patch_size
        
        if remainder != 0:
            # The number of frames is odd (e.g., 3, 5)
            print(f"Input is a video with {num_frames} frames. Padding last frame (Code's logic).")
            
            # Calculate how many frames to add
            num_to_add = temporal_patch_size - remainder
            
            # Get the last frame (patches[-1])
            # Add a new axis [np.newaxis] to make it (1, H, W, C) for repeating
            last_frame = patches[-1][np.newaxis] 
            
            # Repeat the last frame 'num_to_add' times
            repeats = np.repeat(last_frame, num_to_add, axis=0)
            
            # Concatenate the original frames with the repeated last frame(s)
            patches = np.concatenate([patches, repeats], axis=0)
        else:
            # The number of frames is already even (e.g., 2, 4, 6)
            print(f"Input is a video with {num_frames} frames. No padding needed.")
    
    # --- 4. Handle Channel Dimension (from user's code) ---
    # This (L297-298) ensures the data is in (N, C, H, W) format,
    # which is often required by PyTorch models.
    if data_format == ChannelDimension.LAST and patches.ndim == 4:
        # Input is (N, H, W, C). Transpose to (N, C, H, W).
        print(f"Data format is (N, H, W, C). Transposing to (N, C, H, W).")
        patches = patches.transpose(0, 3, 1, 2)
    elif data_format == ChannelDimension.FIRST:
         print(f"Data format is already (N, C, H, W). No transpose needed.")

    print(f"Final processed shape: {patches.shape}\n")
    return patches

# --- Example Usage (ตัวอย่างการทดสอบ) ---

# Create dummy images (64x64 pixels, 3 color channels)
img1 = np.random.rand(64, 64, 3).astype(np.float32)
img2 = np.random.rand(64, 64, 3).astype(np.float32)
img3 = np.random.rand(64, 64, 3).astype(np.float32)

# --- Test Case 1: Single Static Image (The "Paper" logic) ---
print("--- TEST CASE 1: Single Image ---")
# What you might get from a single image input
input_single_image = [img1]
processed_single = robust_qwen_vl_preprocessor(input_single_image)
# Expected: Will be repeated. Final shape (2, 3, 64, 64)

# --- Test Case 2: Odd-numbered Video (The "Code" logic) ---
print("--- TEST CASE 2: Odd-Frame Video ---")
# What you get from a 3-frame video
input_video_odd = [img1, img2, img3]
processed_video_odd = robust_qwen_vl_preprocessor(input_video_odd)
# Expected: Will be padded with img3. Final shape (4, 3, 64, 64)

# --- Test Case 3: Even-numbered Video (No padding) ---
print("--- TEST CASE 3: Even-Frame Video ---")
input_video_even = [img1, img2]
processed_video_even = robust_qwen_vl_preprocessor(input_video_even)
# Expected: No change. Final shape (2, 3, 64, 64)

# --- Test Case 4: Empty Input (Error Handling) ---
print("--- TEST CASE 4: Empty Input ---")
input_empty = []
processed_empty = robust_qwen_vl_preprocessor(input_empty)
# Expected: Warning message.
```

Oh, I got it.
So it assumes that the input is VIDEO if the number of input images >= 2.
I thought that it somehow distinguishes input type (image or video) based on a type parameter, since users may use multiple images (e.g., multiple scanned document pages) as input.

Yes, this code uses the Number of input frames to decide whether it is a "still image" or a "video".

If there is 1 frame: it is considered a still image (use the principle in the article: make a copy of that image $\text{2}$ times).
If there are $\ge 2$ frames: it is considered a Video (use the code principle: repeat the last frame if the number of frames is odd).