VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

## Abstract

Abstract Building on the advances of language models, Large Multimodal Models (LMMs) have contributed significant improvements in video understanding.
While the current video LMMs utilize advanced Large Language Models (LLMs), they rely on either image or video encoders to process visual inputs, each of which has its own limitations.
Image encoders excel at capturing rich spatial details from frame sequences but lack explicit temporal context, which can be important in videos with intricate action sequences.
On the other hand, video encoders provide temporal context but are often limited by computational constraints that lead to processing only sparse frames at lower resolutions, resulting in reduced contextual and spatial understanding.
To this end, we introduce VideoGPT+ , which combines the complementary benefits of the image encoder (for detailed spatial understanding) and the video encoder (for global temporal context modeling).
The model processes videos by dividing them into smaller segments and applies an adaptive pooling strategy on features extracted by both image and video encoders.
Our architecture showcases improved performance across multiple video benchmarks, including VCGBench, MVBench and Zero-shot question-answering.
Further, we develop 112K video-instruction set using a novel semi-automatic annotation pipeline which further improves the model performance.
Additionally, to comprehensively evaluate video LMMs, we present VCGBench-Diverse , covering 18 broad video categories such as lifestyle, sports, science, gaming, and surveillance videos.
This benchmark with 4,354 question-answer pairs evaluates the generalization of existing LMMs on dense video captioning, spatial and temporal understanding, and complex reasoning, ensuring comprehensive assessment across diverse video types and dynamics. Code: https://github.com/mbzuai-oryx/VideoGPT-plus .

## 1 Introduction

Existing methods for video understanding often rely solely on either image encoders or video encoders ; ; . Most works focus on image encoders, which encode multiple frames and either fuse the information or concatenate the embeddings before passing them to the LLM. When fusing the information, spatial or temporal pooling is typically used . Spatial pooling has shown minimal effectiveness in capturing video information, whereas temporal pooling retains some spatial information but lacks explicit temporal context. On the other hand, concatenating embeddings without pooling ; ; can rapidly increase computational complexity due to the extended context length required by the LLM, limiting the number of frames that can be processed. While this approach provides better spatial representation, the overall context is still limited to few frames. The limited context results in a poor understanding of the video, especially if a uniform sampling strategy is employed, as it only captures small segments of the video, missing important temporal dynamics.

Figure: Figure 1: Performance comparison of VideoGPT+ with various SoTA models across multiple video benchmarks. VideoGPT+ demonstrates better performance compared to various models ; ; ; on video conversation benchmarks: VCGBench and MVBench , Zero-shot video question answering: MSVD-QA, MSRVTT-QA, ActivityNet-QA. We also evaluate on VCGBench-Diverse that covers 18 broad video categories (across dense captioning, spatial understanding, and reasoning).
Refer to caption: extracted/5664890/Figures/vcg+_intro_radar_plot.png

In order to address these challenges, we propose VideoGPT+ which effectively combines the merits of both image and video encoders (see Fig. 2). By leveraging an image encoder for rich spatial details and a video encoder for global temporal context, our model achieves improved video understanding. To model finegrained temporal dynamics in VideoGPT+ , we use a segment-wise sampling strategy. Unlike uniform sampling used in existing video LMMs , which may miss important temporal dynamics, our approach divides the video into smaller segments and applies segment-wise sampling. This ensures that the model captures representative information from different segments of the video, enabling a more
comprehensive understanding.

To facilitate the integration of image and video features, VideoGPT+ introduces a visual adapter module that combines their complimentary benefits. This module performs projection and pooling operations, mapping both image and video features to a common space while reducing computational complexity. By aligning the features in this manner, the model can effectively utilize the combined spatial and temporal information for improved video understanding.

We demonstrate the effectiveness of VideoGPT+ across multiple video-conversation benchmarks, including VCGBench , MVBench , and Zero-shot question-answering , where it outperforms previous SoTA approaches (see Fig. 1). Further, we develop VCG+ 112K using a novel semi-automatic annotation pipeline (see Fig. 3), which provides dense video captions along with spatial understanding and reasoning-based question-answer (QA) pairs, further enhancing the model’s performance. We also propose VCGBench-Diverse, extending VCGBench  by including videos from 18 different domains to extensively evaluate the video-based conversation models in diverse domains (see Fig. 4).

Our work has three main contributions:

- •
We present  VideoGPT+, the first video-conversation model that benefits from a dual-encoding scheme based on both image and video features. These complimentary sets of features offer rich spatiotemporal details for improved video understanding (Sec. 3).
- •
Addressing the limitations of existing VideoInstruct100K dataset , we develop VCG+ 112K with a novel semi-automatic annotation pipeline, offering dense video captions along with spatial understanding and reasoning-based QA pairs, further improving the model performance (Sec. 4).
- •
Recognizing the lack of diverse benchmarks for video-conversation task, we propose VCGBench-Diverse, which provides 4,354 human annotated QA pairs across 18 video categories to extensively evaluate the performance of a video-conversation model (Sec. 5).

## 2 Related Works

Building on advances in language models, LLMs offer a flexible interface for various multimodal applications. Early efforts in image-based conversation models such as BLIP-2 , MiniGPT-4  and LLaVA ; project image features into the language space through a learnable module and perform instruction tuning for visual conversations capabilities. Other efforts extend these models to visual grounding tasks ; ; , exploring the potential of LLMs in complex vision tasks.

Video Conversation Models: Initial works like Video-ChatGPT  and Video-LLaMA  extend image-based models to the video domain by introducing components to encode temporal features, where frame-level visual features are fed to the LLM. However, this is computationally expensive and quickly fills its context window. To address this issue, Video-ChatGPT  employs spatial and temporal pooling. LLaMA-Vid  proposes representing a single image with two tokens, context and content. IG-VLM  treats a video as a grid of images, while LITA  employs slow-fast token pooling to reduce the number of visual features. Chat-UniVi  uses clustering in both spatial and temporal dimensions to merge tokens, and VideoChat  uses Q-Former  to learn a fixed number of queries by cross-attending to the visual features. MobileVLM ; utilize a lightweight CNN to reduce the spatial dimensions. Other notable methods include ; ; ; ; .

Alternatively, methods such as VideoChat2  use pretrained video encoders. Although video encoders provide temporal context, they are limited by computational constraints, operating with limited frames at lower resolutions, restricting temporal context and spatial understanding. Our VideoGPT+ model addresses these issues by using segment-wise sampling and effectively combining the merits of image and video encoders to capture rich
spatial and temporal details (see Fig. 2).

Video Instruction Tuning Datasets: VideoChat  builds a video-instruction tuning dataset consisting of 7K instructions using videos from WebVid-10M . Video-ChatGPT  introduces a semi-automatic annotation pipeline to generate VideoInstruct100K using videos from ActivityNet . VideoChat2  combines multiple existing image and video datasets to develop a 1.9M joint image-video instruction tuning dataset. In our experiments, we use VideoInstruct100K and a subset of the dataset from VideoChat2. Additionally, addressing the limitations of the VideoInstruct100K dataset , we develop  VCG+ 112K through a novel semi-automatic annotation pipeline, which provides dense video captions along with 112K QA pairs targeting reasoning, spatial and temporal understanding, which further improves model’s understanding of video content (see Fig. 3).

Video Conversation Benchmarks: Video-ChatGPT  introduces VCGBench and Zero-shot QA benchmarks, where VCGBench includes 500 videos with 3000 QA pairs, evaluated using GPT-3.5 across various metrics. Despite its comprehensive evaluation, it only contains videos from the ActivityNet dataset. The Zero-shot evaluation covers MSVD-QA , MSR-VTT-QA , TGIF-QA , and ActivityNet-QA .
MVBench  consists of 4K QA pairs evaluating 20 temporal tasks, though it mostly includes short videos averaging 5-40 seconds. Considering the limitation of existing benchmarks, which often lack focus on generalization and diversity, we propose VCGBench-Diverse, featuring 4,354 QA pairs from 877 videos across 18 domains (see Fig. 4).

## 3 Method

Figure: Figure 2: Overview of VideoGPT+. VideoGPT+ is a large multimodal model for video understanding. It uses a dual-encoder design that combines the complementary strengths of an image encoder and a video encoder. The image encoder captures detailed spatial features, while the video encoder captures temporal dynamics across multiple frames. To retain fine-grained temporal details while ensuring efficiency, we use segment-wise frame sampling instead of random sparse sampling. Both sets of features are then projected into a unified space through Vision-Language (V-L) projection layers and the resulting tokens are pooled and concatenated before being processed by a Large Language Model to generate comprehensive responses to video-based questions. Symbols indicates frozen components, indicates trainable components, and the indicates LoRA-training.
Refer to caption: extracted/5664890/Figures/vcg+_block_diagram_v6.png

For effective video understanding, combining detailed spatial information with explicit temporal context is crucial. To achieve this, we propose VideoGPT+, which features a dual encoder design that leverages the complementary strengths of an image encoder and a video encoder.

Overall Architecture:
The overall architecture consists of (i) segment-wise sampling, (ii) dual visual encoder, (iii) vision-language adapters that project vision features to the language domain and (iv) a large language model. Frames selected through a segment-wise sampling strategy are encoded through a dual encoder consisting of an image and a video encoder. Both sets of features are projected to language space using vision-language (V-L) adapters, and the resulting tokens are pooled through adaptive token pooling and concatenated before being fed to the LLM (see Fig. 2).

Segment-wise Sampling: To extract fine-grained temporal cues, we use a segment-wise frame sampling strategy. Given an input video $\mathbf{V}\in\mathbb{R}^{T\times H\times W\times C}$, we divide it into $K$ segments, where each segment consists of $n=\frac{T}{K}$ frames. Thus, the video can be represented as $\mathbf{V}=[\mathbf{V}_{k}]_{k=1}^{K}$. Each segment $\mathbf{V}_{k}\in\mathbb{R}^{n\times H\times W\times C}$ can be described as a sequence of frames, $\mathbf{X}_{i}$, where $\mathbf{V}_{k}=[\mathbf{X}_{i,j}]_{j=1}^{n}$. The video segments are downsampled to a lower resolution of $n\times h\times w\times c$ for video encoding.

Compared to a uniform sampling, segment-wise sampling better aligns with our dual encoder design. Video encoders often face computational constraints, limiting them to processing only sparse frames. Uniform sampling increases the self-attention computation complexity as it requires attending to features of all frames. Additionally, video encoders are typically trained with sparse frames, and providing more frames can hinder their ability to accurately capture temporal information. In contrast, the segment-wise sampling strategy divides the video into smaller, manageable segments, enabling the video encoder to efficiently capture rich temporal cues within each segment.

Dual Vision Encoder: Our design leverages the complementary strengths of an image encoder that captures detailed spatial features and a video encoder that provides explicit temporal context. The image encoder $g$, processes $T$ frames, $g(\mathbf{X})\in\mathbb{R}^{T\times H_{g}\times W_{g}\times D_{g}}$, producing local features that provide frame-level context. Meanwhile, the video encoder $h$, operates on low-resolution video segments $\mathbf{V}_{k}$, yielding global features that provide segment-wise context, $h(\mathbf{V}_{k})\in\mathbb{R}^{n\times h_{h}\times w_{h}\times D_{h}}$.

The primary goal of VideoGPT+ is to leverage the capabilities of a pre-trained LLM alongside visual modalities from both a pre-trained image encoder and a pre-trained video encoder. Specifically, we utilize the pre-trained CLIP model, ViT-L/14 ($336\times 336$)  as the image encoder, and InternVideo-v2 ($224\times 224$)  as the video encoder. These models are selected for their robust performance and their ability to complement each other in capturing both spatial and temporal information. Both encoders are pre-trained on large-scale datasets in a multimodal setting using contrastive loss, facilitating their integration within our architecture.

Visual Adapter: The output embeddings from the second last layer of both image and video encoders are passed through separate V-L projection layers, $W_{g}$ and $W_{h}$, respectively. These Multi-Layer perceptrons (MLPs) project the visual features into the language space. The projection layers are trainable, while the visual encoders remain frozen, preserving the rich, pre-trained representations. The projected embeddings are reshaped back into their grid forms and subjected to a $2\times 2$ adaptive token pooling, which operates on the spatial dimensions of the local and global features. This pooling reduces the token length by a factor of $4$, thereby allowing to fit in larger visual context within the same LLM context window.
The pooled embeddings from the local features form $\mathbf{E}^{img}\in\mathbb{R}^{T\times h_{g}\times w_{g}\times D_{t}}$, while the pooled embeddings from the global features of each segment form $\mathbf{E}^{vid}\in\mathbb{R}^{n\times h_{h}\times w_{h}\times D_{t}}$.

Large Language Model: We obtain the final representation by concatenating the embeddings $\mathbf{E}^{img}$ with $K$ segment-wise embeddings $\mathbf{E}^{vid}$, such that we have detailed spatial representation across all segments followed by their global temporal context. We then concatenate the text embeddings $\mathbf{E}^{text}\in\mathbb{R}^{L\times D_{t}}$ of the user text query with the visual embeddings,

$$ $\mathbf{E}=[\mathbf{E}^{img},\mathbf{E}_{1}^{vid},\ldots,\mathbf{E}_{K}^{vid}, \mathbf{E}^{text}].$ (1) $$

This integration ensures that the LLM receives a sequence of embeddings that include detailed spatial features from the image encoder and comprehensive temporal context from the video encoder, allowing for robust video understanding. The LLM is fine-tuned using LoRA  in an auto-regressive manner with a next-token prediction loss. Refer to Fig. 2 for detailed illustration.

## 4 Dataset

Figure: Figure 3: Illustration of the semi-automatic annotation process in VCG+ 112K. The figure shows how we use ground-truth video captions and frame-level descriptions to generate a detailed video description. GPT-4 is used to remove irrelevant and conflicting noisy information in the frame-level descriptions to produce a high-quality video description. The semi-automatic annotation process integrates spatial, temporal and event, and reasoning details into the brief information we start with. This dense video description is then used to generate instruction-tuning QA pairs using GPT-3.5. We provide detailed prompts used in both stages in Appendix 11 (see Figs. 8 and 9). We also compare the video description in the VideoInstruct100K dataset to show the improvement in quality achieved by our new annotation pipeline.
Refer to caption: extracted/5664890/Figures/vcg120k_block_diagram_v0.png

Video-ChatGPT  introduces the VideoInstruct100K dataset, which employs a semi-automatic annotation pipeline to generate 75K instruction-tuning QA pairs. To address the limitations of this annotation process, we present VCG+ 112K dataset developed through an improved annotation pipeline. Our approach improves the accuracy and quality of instruction tuning pairs by improving keyframe extraction, leveraging SoTA large multimodal models (LMMs) for detailed descriptions, and refining the instruction generation strategy.

Keyframe Extraction:
VideoInstruct100K uses a fixed number of video keyframes, regardless of video length or dynamics, to generate frame-level dense captions. This often results in both insufficient and redundant information. We address this by first extracting scenes from videos , and then selecting one keyframe/scene. Consequently, we obtain
detailed information for videos with rich content and reduce redundancy for videos with less content. It provides better visual context by extracting more stable keyframes, thus offering a more accurate video representation.

Frame-Level Descriptions:
After extracting keyframes, we use a SoTA image LMM, LLaVA-v1.6 , to generate dense descriptions for each keyframe. These descriptions encompass comprehensive visual details, including spatial attributes, scene context, and object characteristics, which are often absent in concise ground truth captions. While ground truth captions are precise, they lack the granularity to capture intricate visual and spatial information. To address this, we augment them captions with detailed but noisy information from the frame-level descriptions, thus enhancing the quality and accuracy of the subsequent video descriptions.

Detailed Video Descriptions:
VideoInstruct100K  prompts GPT-3.5 directly with frame-level descriptions and concise ground truth captions to generate QA pairs, imposing a significant cognitive load on the model to verify frame-level descriptions with the ground truth. We improve this process by first creating a coherent and detailed video description. We prompt GPT-4 to integrate the detailed frame-level descriptions with the ground truth captions by comparing information and removing any inconsistencies. The resulting detailed descriptions include a timeline of events, actions, object attributes, and scene settings, providing a thorough representation of the video content. This structured input
simplifies the task for LLM, thereby enhancing the generated QA pairs quality.

Improved Instruction Tuning Data:
Using the ground truth captions and detailed video descriptions, we generate two types of high-quality QA pairs using GPT-3.5: descriptive and concise. For descriptive instruction pairs, we focus on three categories: (i) dense captioning, which provides descriptions of the video covering the entire sequence of events and visual details; (ii) detailed temporal information, which addresses the sequence of events and their dependency to learn temporal relationships; and (iii) generic question answering, which involves in-depth questions about different actions, their consequences, and other detailed aspects of the video. For concise instruction pairs, we target (i) spatial reasoning, focusing on understanding and describing spatial details such as scene settings, number of objects, attire, and locations; (ii) reasoning of events, covering the causal relationships between events; and (iii) short temporal questions, addressing specific moments or sequences, such as what happened at the beginning or end.

## 5 Proposed Benchmark

Figure: Figure 4: Illustration of VCGBench-Diverse video conversational benchmark. VCGBench-Diverse comprehensive benchmark is designed to evaluate video LMMs across 18 broad video categories. With 4,354 QA pairs, VCGBench-Diverse tests generalization on dense video captioning, spatial and temporal understanding, and complex reasoning. It covers five video-capturing methods, ensuring diversity and robust generalization and six reasoning complexities, assessing various analytical and comprehension skills.
Refer to caption: extracted/5664890/Figures/vcgbench_block_diag_v6.png

Recognizing the limited diversity in existing video conversation benchmarks, we introduce VCGBench-Diverse to comprehensively evaluate the generalization ability of video LMMs. While VCG-Bench  provides an extensive evaluation protocol, it is limited to videos from the ActivityNet200  dataset. Our benchmark comprises a total of 877 videos, 18 broad video categories and 4,354 QA pairs, ensuring a robust evaluation framework. The detailed breakdown of VCGBench-Diverse is illustrated in Fig. 4, showcasing the distribution of videos across content domains, video capturing methods, and reasoning complexities.

We collect videos from 18 distinct domains, including lifestyle, how-to, science and technology, news, travel, entertainment, film, sports, comedy, activism, gaming, education, surveillance, pets, cooking, music, automobile, and traffic (see Fig. 4). These categories encompass a broad spectrum of real-world scenarios, ensuring that models are evaluated on a diverse set of challenges. In addition to content diversity, VCGBench-Diverse includes a variety of video capture methods, which ensures a comprehensive assessment of robustness to different filming techniques, camera movements, quality levels and lighting. The benchmark covers five video capture methods including static and controlled settings, dynamic and unpredictable settings, fixed camera perspectives, professional and high-quality videos, and uncontrolled and variable quality. Further, the benchmark evaluates models across six reasoning complexities, including sequential understanding, complex action and predictive reasoning, contextual and world knowledge reasoning, causal reasoning, narrative and emotional reasoning, and analytical and critical reasoning, which is crucial for understanding diverse video content.

The videos in VCGBench-Diverse are sourced from HDVILA , MPII , YouCook2 , UCF Crime , and STUD Traffic . The video durations range from 29 sec to 471 sec, with an average of 217 sec. Human annotators are tasked with writing detailed descriptions based on their understanding of both audio and visual elements of the videos. This comprehensive annotation process involves a set of annotators who are provided with an initial set of ten videos each. These annotations undergo a meta-review stage where feedback is provided, and necessary corrections are made to meet the required standards. Following this, annotators receive additional batches, with random samples being selected for quality checks by the meta-reviewer. The final human annotations are utilized to generate QA pairs using GPT-3.5, based on prompts detailed in Fig. 10.

Following VCG-Bench , the evaluation is computed over five different aspects: (i) correctness of information (ii) detail orientation (iii) contextual understanding (iv) temporal understanding and (v) consistency. Additionally, VCGBench-Diverse provides a breakdown of performance across three key aspects: (i) dense video captioning, which assesses the ability to generate detailed and accurate descriptions of the video content, (ii) spatial understanding, which evaluates the capability to understand and describe the spatial relationships and settings within the video, and (iii) reasoning, which tests the adeptness in inferring and explaining causal relationships and actions within the video.

## 6 Experiments

We perform quantitative evaluation of VideoGPT+ on four standard benchmarks: i) VCGBench , ii) VCGBench-Diverse, iii) MVBench  and iv) Zero-shot QA.

Implementation Details: We use CLIP-L/14  as our image encoder, InternVideo-v2  stage-2 1B model as our video encoder in conjunction with Phi-3-Mini-3.8B  based LLM with 4K context window in our experiments. The image encoder operates at $336\times 336$, while the video encoder operates at $224\times 224$ resolution. Our training consists of two pretraining stages and one instruction-tuning stage. In the pretraining stage, we train with only the image encoder and only the video encoder on the CC-595K dataset , with only the visual adapters being learned while the rest of the model is kept frozen. During the instruction-tuning stage, we use LoRA  with $r=64$ for LLM, while visual adapters are fully trained and vision encoders are kept frozen. The LR is set to $1e^{-3}$ during pretraining and $2e^{-4}$ during instruction tuning.

For experiments on VCGBench, VCGBench-Diverse and Zero-shot QA, we sample 16 frames from videos, while for MVBench which consists of relatively shorter videos, we sample 8 frames. We keep the same sampling strategy during inference. For VCGBench and  VCGBench-Diverse, the model is trained on VideoInstruct100K , VCG+ 112K , conversation and caption data from VideoChat  and VQA dataset from WebVid , that combines to approximately 260K single turn conversations. For MVBench, the model is trained on Kinetics-710 , Something-Something-v2 , conversations from VideoChat , CLEVRER , VQA dataset from WebVid  and NExT-QA  datasets, which combines to approximately 330K single turn conversations. We run all trainings for one epoch. Following previous approaches ; ; , we employ GPT-3.5-Turbo-0613 for VCGBench and Zero-shot QA evaluation. However, for our proposed VCGBench-Diverse, we employ the latest GPT-3.5-Turbo-0125 for evaluation.

**Table 1: Performance of VideoGPT+ on VCGBench . All models use 16 frames except Video-ChatGPT and Chat-UniVi which use 100 and 64 frames respectively.**
| Method | CI | DO | CU | TU | CO | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Video-ChatGPT | 2.40 | 2.52 | 2.62 | 1.98 | 2.37 | 2.38 |
| BT-Adapter | 2.68 | 2.69 | 3.27 | 2.34 | 2.46 | 2.69 |
| VTimeLLM | 2.78 | 3.10 | 3.40 | 2.49 | 2.47 | 2.85 |
| Chat-UniVi | 2.89 | 2.91 | 3.46 | 2.89 | 2.81 | 2.99 |
| LLAMA-VID | 2.96 | 3.00 | 3.53 | 2.46 | 2.51 | 2.89 |
| Video-LLaVA | 2.84 | 2.86 | 3.44 | 2.46 | 2.57 | 2.81 |
| VideoChat2 | 3.02 | 2.88 | 3.51 | 2.66 | 2.81 | 2.98 |
| IG-VLM | 3.11 | 2.78 | 3.51 | 2.44 | 3.29 | 3.03 |
| VideoGPT+ (ours) | 3.27 | 3.18 | 3.74 | 2.83 | 3.39 | 3.28 |

VCGBench: The benchmark consists of approximately 3000 QA pairs generated using 500 human-annotated videos from ActivityNet . The benchmark evaluates the responses on five different aspects: i) Correctness of Information (CI), which assesses the correctness of the response to ensure it aligns with the video contents, ii) Detail Orientation (DO), which evaluates the depth of the response, iii) Contextual Understanding (CU), which assesses if the response aligns with the overall context of the video, iv) Temporal Understanding (TU), which assesses the model’s ability to identify temporal sequences accurately, and v) Consistency (CO), which evaluates the consistency in the model response to similar questions. Table 1 compares our model with previous SoTA approaches. VideoGPT+ achieves an average score of 3.28 surpassing previous best method by a margin of 0.25 (5%).

VCGBench-Diverse: We provide a quantitative comparison of VideoGPT+ against previous SoTA approaches on VCGBench-Diverse, which contains 4,354 QA pairs from 877 videos. Following , we evaluate the Correctness of Information (CI), Detail Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU), and Consistency (CO). Additionally, we provide results for dense captioning, spatial understanding, and visual reasoning abilities. The results are presented in Table 2. VideoGPT+ achieves an average score of 2.47 surpassing all previous methods. Further, VideoGPT+ achieves a score of 1.38, 2.80, and 3.63 on dense captioning, spatial understanding, and visual reasoning, respectively. Notably, VideoGPT+ achieves improvements in spatial and temporal understanding, surpassing previous best models by 0.37 (7.4%) and 0.23 (4.6%), respectively. This is attributed to the dual encoder architecture, where the high-resolution image encoder enhances spatial understanding and the video encoder improves temporal accuracy.

**Table 2: Performance of VideoGPT+ on VCGBench-Diverse. All models use 16 frames except Video-ChatGPT and Chat-UniVi, which use 100 and 64 frames, respectively. The good performance of our model on VCGBench-Diverse shows its generalization to diverse scenarios.**
| Method | CI | DO | CU | TU | CO | Avg. | Caption | Spatial | Reasoning |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Video-ChatGPT (ACL 2024) | 2.07 | 2.42 | 2.46 | 1.39 | 2.06 | 2.08 | 0.89 | 2.25 | 3.60 |
| BT-Adapter (CVPR 2024) | 2.20 | 2.62 | 2.59 | 1.29 | 2.27 | 2.19 | 1.03 | 2.35 | 3.62 |
| VTimeLLM (CVPR 2024) | 2.16 | 2.41 | 2.48 | 1.46 | 2.35 | 2.17 | 1.13 | 2.29 | 3.45 |
| Chat-UniVi (CVPR 2024) | 2.29 | 2.56 | 2.66 | 1.56 | 2.36 | 2.29 | 1.33 | 2.36 | 3.59 |
| VideoChat2 (CVPR 2024) | 2.13 | 2.42 | 2.51 | 1.66 | 2.27 | 2.20 | 1.26 | 2.43 | 3.13 |
| VideoGPT+ (ours) | 2.46 | 2.73 | 2.81 | 1.78 | 2.59 | 2.47 | 1.38 | 2.80 | 3.63 |

**Table 3: Performance of VideoGPT+ on MVBench. Following , we evaluate on 20 tasks including AS: Action Sequence, AP: Action Prediction, AA: Action Antonym, FA: Fine-grained Action, UA: Unexpected Action, OE: Object Existence, OI: Object Interaction, OS: Object Shuffle, MD: Moving Direction, AL: Action Localization, ST: Scene Transition, AC: Action Count, MC: Moving Count, MA: Moving Attribute, SC: State Change, FP: Fine-grained Pose, CO: Character Order, EN: Egocentric Navigation, ER: Episodic Reasoning and CI: Counterfactual Inference.**
| Model | AS | AP | AA | FA | UA | OE | OI | OS | MD | AL | ST | AC | MC | MA | SC | FP | CO | EN | ER | CI | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Random | 25.0 | 25.0 | 33.3 | 25.0 | 25.0 | 33.3 | 25.0 | 33.3 | 25.0 | 25.0 | 25.0 | 33.3 | 25.0 | 33.3 | 33.3 | 25.0 | 33.3 | 25.0 | 20.0 | 30.9 | 27.3 |
| GPT-4V | 55.5 | 63.5 | 72.0 | 46.5 | 73.5 | 18.5 | 59.0 | 29.5 | 12.0 | 40.5 | 83.5 | 39.0 | 12.0 | 22.5 | 45.0 | 47.5 | 52.0 | 31.0 | 59.0 | 11.0 | 43.5 |
| Otter-V | 23.0 | 23.0 | 27.5 | 27.0 | 29.5 | 53.0 | 28.0 | 33.0 | 24.5 | 23.5 | 27.5 | 26.0 | 28.5 | 18.0 | 38.5 | 22.0 | 22.0 | 23.5 | 19.0 | 19.5 | 26.8 |
| mPLUG-Owl-V | 22.0 | 28.0 | 34.0 | 29.0 | 29.0 | 40.5 | 27.0 | 31.5 | 27.0 | 23.0 | 29.0 | 31.5 | 27.0 | 40.0 | 44.0 | 24.0 | 31.0 | 26.0 | 20.5 | 29.5 | 29.7 |
| Video-ChatGPT | 23.5 | 26.0 | 62.0 | 22.5 | 26.5 | 54.0 | 28.0 | 40.0 | 23.0 | 20.0 | 31.0 | 30.5 | 25.5 | 39.5 | 48.5 | 29.0 | 33.0 | 29.5 | 26.0 | 35.5 | 32.7 |
| VideoLLaMA | 27.5 | 25.5 | 51.0 | 29.0 | 39.0 | 48.0 | 40.5 | 38.0 | 22.5 | 22.5 | 43.0 | 34.0 | 22.5 | 32.5 | 45.5 | 32.5 | 40.0 | 30.0 | 21.0 | 37.0 | 34.1 |
| VideoChat | 33.5 | 26.5 | 56.0 | 33.5 | 40.5 | 53.0 | 40.5 | 30.0 | 25.5 | 27.0 | 48.5 | 35.0 | 20.5 | 42.5 | 46.0 | 26.5 | 41.0 | 23.5 | 23.5 | 36.0 | 35.5 |
| VideoChat2 | 66.0 | 47.5 | 83.5 | 49.5 | 60.0 | 58.0 | 71.5 | 42.5 | 23.0 | 23.0 | 88.5 | 39.0 | 42.0 | 58.5 | 44.0 | 49.0 | 36.5 | 35.0 | 40.5 | 65.5 | 51.1 |
| VideoGPT+ (ours) | 69.0 | 60.0 | 83.0 | 48.5 | 66.5 | 85.5 | 75.5 | 36.0 | 44.0 | 34.0 | 89.5 | 39.5 | 71.0 | 90.5 | 45.0 | 53.0 | 50.0 | 29.5 | 44.0 | 60.0 | 58.7 |

MVBench: We evaluate VideoGPT+ on MVBench , which provides 4,000 QA pairs from 11 video datasets covering a broad spectrum of scenes, ranging from first-person to third-person and from indoor to outdoor environments. The tasks are categorized into 20 fine-grained temporal understanding tasks. The results presented in Table 3 compare VideoGPT+ with previous methods, indicating an overall improvement of 7.6% compared to the previous best, VideoChat2. Specifically, VideoGPT+ achieves SoTA results in 14 out of 20 tasks and comes second in 4 out of 20 tasks, obtaining an average score of 58.7% across the 20 tasks. Additionally, VideoGPT+ shows significant improvements in the Action Prediction (+12.5%), Object Existence (OE) (+27.5%), Moving Direction (MD) (+17%), Moving Count (MC) (+29%) and Moving Attributes (MA) (+32%) indicating the rich spatial information and temporal context achieved by our model.

Zero-shot Question-Answering: We provide a quantitative comparison of our method on the zero-shot QA task across four open-ended QA datasets, including MSVD-QA , MSRVTT-QA , TGIF-QA , and ActivityNet-QA . Results presented in Table 4 show VideoGPT+ achieves superior performance compared to previous methods, indicating its ability to adapt effectively to unseen videos and generate accurate contextually relevant responses in challenging settings.

**Table 4: Performance of VideoGPT+ on Zero-shot QA. We evaluate accuracy and score on four commonly used datasets. All the models are evaluated in zero-shot setting where none of the videos were included in the training set. VideoGPT+ achieves good results on all datasets.**
| Model | MSVD-QA | MSRVTT-QA | TGIF-QA | ActivityNet-QA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Accuracy | Score | Accuracy | Score | Accuracy | Score | Accuracy | Score |
| FrozenBiLM | 32.2 | – | 16.8 | – | 41.0 | – | 24.7 | – |
| VideoChat | 56.3 | 2.8 | 45.0 | 2.5 | 34.4 | 2.3 | 26.5 | 2.2 |
| LLaMA Adapter | 54.9 | 3.1 | 43.8 | 2.7 | - | - | 34.2 | 2.7 |
| Video-LLaMA | 51.6 | 2.5 | 29.6 | 1.8 | - | - | 12.4 | 1.1 |
| Video-ChatGPT | 64.9 | 3.3 | 49.3 | 2.8 | 51.4 | 3.0 | 35.2 | 2.8 |
| ChatUniVi | 65.0 | 3.6 | 54.6 | 3.1 | 60.3 | 3.4 | 45.8 | 3.2 |
| LLaMA-VID | 70.0 | 3.7 | 58.9 | 3.3 | – | – | 47.5 | 3.3 |
| Video-LLaVA | 70.7 | 3.9 | 59.2 | 3.5 | 70.0 | 4.0 | 45.3 | 3.3 |
| VideChat2 | 70.0 | 3.9 | 54.1 | 3.3 | – | – | 49.1 | 3.3 |
| VideoGPT+ (ours) | 72.4 | 3.9 | 60.6 | 3.6 | 74.6 | 4.1 | 50.6 | 3.6 |

**Table 5: Ablation on Vision Encoder type, Image feature pooling, Video feature pooling, and VCG+ 112K. We evaluate VideoGPT+ training using only an image encoder, video encoder, and both image and video encoders (dual encoder) on VCGBench. Using a dual encoder provides rich semantic and temporal cues.**
|  | Vision Encoder Type | Image Pooling | Video Pooling | VCG+ 112K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Image | Video | Dual | CNN | $\boldsymbol{4\times 4}$ | $\boldsymbol{2\times 2}$ | Time | Space | × | ✓ |  |
| Correctness (CI) | 3.14 | 3.22 | 3.27 | 3.24 | 3.24 | 3.27 | 3.21 | 3.27 | 3.20 | 3.27 |
| Detail (DO) | 3.09 | 3.10 | 3.18 | 3.13 | 3.18 | 3.18 | 3.13 | 3.18 | 3.08 | 3.18 |
| Context (CU) | 3.68 | 3.70 | 3.74 | 3.70 | 3.73 | 3.74 | 3.70 | 3.74 | 3.66 | 3.74 |
| Temporal (TU) | 2.69 | 2.70 | 2.83 | 2.74 | 2.73 | 2.83 | 2.72 | 2.83 | 2.66 | 2.83 |
| Consistency (CO) | 3.26 | 3.31 | 3.39 | 3.41 | 3.39 | 3.39 | 3.36 | 3.39 | 3.28 | 3.39 |
| Average | 3.17 | 3.20 | 3.28 | 3.25 | 3.25 | 3.28 | 3.23 | 3.28 | 3.17 | 3.28 |

Vision Encoder Type: We ablate our dual visual encoder design in VideoGPT+ in on VCGBench with results presented in Table 5. We conduct three experiments: using only the image encoder, only the video encoder, and both encoders. The image encoder alone achieves a score of 3.17, while the video encoder alone achieves a better score of 3.20, indicating the benefits of video-based pretraining. The dual encoder design, combining both spatial and temporal information, achieves the highest score of 3.28, demonstrating enhanced performance in video-conversation tasks.

Pooling Strategy: We ablate different pooling strategies for the image and video encoders in Table 5. The image encoder outputs a $24\times 24$ feature map from a $336\times 336$ input. We compare two downsampling methods: a learnable lightweight CNN (LDPv2 from ) and a non-learnable adaptive average pooling with a $2\times 2$ kernel. Results indicate that adaptive pooling performs better than CNN. A $4\times 4$ adaptive pooling was also tested but showed inferior performance.

Similarly, we ablate the pooling choice for the video encoder, which takes an input of size $T\times 224\times 224\times C$ and outputs a feature map of $T\times 16\times 16\times d$. We compare two pooling strategies: time pooling across the temporal dimension to reduce the feature map to $1\times 16\times 16\times d$, and space pooling across the spatial dimension with a $2\times 2$ kernel. Table 5 shows that space pooling effectively preserves temporal information and yields better results.

**Table 6: Ablation on LLM type. We train and evaluate VideoGPT+ with different LLMs, including vicuna and LLaMA3 , which further improves accuracy.**
| LLM | VCGBench | Avg. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| CI | DO | CU | TU | CO |  |  |
| Phi3-Mini-3.8B | 3.27 | 3.18 | 3.74 | 2.83 | 3.39 | 3.28 |
| Vicuna-7B | 3.22 | 3.14 | 3.69 | 2.65 | 3.46 | 3.23 |
| Vicuna-13B | 3.30 | 3.20 | 3.75 | 2.77 | 3.48 | 3.30 |
| LLaMA3-8B | 3.29 | 3.21 | 3.73 | 2.86 | 3.38 | 3.29 |

VCG+ 112K: To demonstrate the effectiveness of VCG+ 112K, we train VideoGPT+ with and without it. As shown in Table 5, VCG+ 112K improves performance, particularly in detail orientation (DO) and temporal understanding (TU). This improvement can be attributed to our novel semi-automatic annotation pipeline and the enhanced instruction tuning data, which focuses on generating both detailed and concise instruction pairs. Refer to Fig. 3 for qualitative visualization of the data.

LLM Type: We train VideoGPT+ with different LLMs including Vicuna 7B and 13B  and LLaMA-3 8B  and shows results in Table 6. We observe slight improvements in VCGBench scores when training using better LLMs, including Vicuna 13B and LLaMA-3 8B models.

## 7 Conclusion

In this work, we introduce  VideoGPT+, a novel video conversation model that leverages the complementary benefits of image and video encoders to achieve enhanced video understanding. VideoGPT+ demonstrates better performance across multiple video benchmarks, owing to its dual-encoder design, lightweight visual adapters that map image/video features to a common space and a segment-wise sampling strategy that retains fine-grained temporal information.
We also develop VCG+ 112K, a 112K video-instruction set using a resource-efficient semi-automated annotation pipeline that delivers further gains. Lastly, we propose VCGBench-Diverse, a diverse benchmark covering 18 video categories, to comprehensively evaluate video LMMs. Despite reported improvements, video LMMs still find challenges in precise action localization, understanding very long videos, and navigating long paths; areas where major improvements can unlock new applications.

Figure: Figure 5: Qualitative comparison of VideoGPT+ with VideoChat2. Our VideoGPT+ demonstrates superior temporal understanding by correctly identifying multiple events in the video, effective reasoning in generating a creative advertisement, and accurate spatial understanding by identifying the SPF value and brand name of the sunscreen.
Refer to caption: extracted/5664890/Figures_Supp/demo_vcg+_main.png

Figure: Figure 6: Qualitative comparison from VCGBench-Diverse of VideoGPT+. We show qualitative comparison of VideoGPT+ with VideoChat2 and propriety models GPT-4V and Gemini-1.5-Pro-V from three different categories including traffic, education and surveillance from VCGBench-Diverse.
Refer to caption: extracted/5664890/Figures_Supp/demo_vcg+_full_part1.jpg

Figure: Figure 7: Qualitative comparison from VCGBench-Diverse of VideoGPT+. We show qualitative comparison of VideoGPT+ with VideoChat2 and propriety models GPT-4V and Gemini-1.5-Pro-V from three different categories including sports, news and automobiles videos from VCGBench-Diverse.
Refer to caption: extracted/5664890/Figures_Supp/demo_vcg+_full_part2.jpg

## 8 Qualitative Results

We provide a qualitative comparison of our VideoGPT+ with the previous state-of-the-art approach, VideoChat2 , in Fig. 5. The example shows an advertisement video for sunscreen, where multiple scene changes are present. The video starts with a close-up view of the sunscreen, followed by a woman applying sunscreen on her hand, then applying sunscreen near a beach. The woman is then seen applying sunscreen on her arms, and finally, the video shows the key ingredients of the sunscreen and ends with the cover of the sunscreen.

As shown in Fig. 5, our VideoGPT+ correctly identifies the events present in the video and provides a detailed and accurate description. On the other hand, VideoChat2 struggles to accurately capture all the events. Further, our model generates an advertisement post highlighting one of the unique features of the sunscreen shown in the video, namely that it functions as both sunscreen and moisturizer. Lastly, our VideoGPT+ correctly identifies the SPF value and brand name of the sunscreen, while VideoChat2 struggles to correctly identify the brand name. We present further comparison in Fig. 7.

## 9 Additional Implementation Details

In this section, we provide additional implementation details regarding our training setup and compute requirements. All of our experiments are conducted using 8xA100 40GB GPUs. The training for VCGBench experiments takes around 12 hours to complete, while the training for MVBench experiments finishes in around 10 hours. We use the model trained for the VCGBench task to evaluate on VCGBench-Diverse and zero-shot question-answering benchmarks. All of our training and evaluation codes, pretrained models and dataset will be publicly released.

## 10 Additional Ablations

**Table 7: Ablation on Feature Concatenation Strategy. Performance comparison between interleaved and sequential feature concatenation strategies. The sequential feature concatenation performs better.**
| Feature | VCGBench | Avg. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Concatenation | CI | DO | CU | TU | CO |  |
| Interleaved | 3.25 | 3.17 | 3.72 | 2.78 | 3.39 | 3.26 |
| Sequential | 3.27 | 3.18 | 3.74 | 2.83 | 3.39 | 3.28 |

Feature concatenation strategy: We conduct an ablation study to determine the optimal order in which image and video features should be input to the LLM. Specifically, we perform two experiments. In the first experiment, image and video features are extracted for each video segment and concatenated in an interleaved manner before sending as input to the LLM. For example, the video is divided into segments of equal size, and then the image and video features from each segment are concatenated and input to the LLM.
In the second experiment, we first place all the image features followed by all the video features. The results shown in Table 7, indicate that the sequential design, where the image features are placed first followed by the video features, yields better performance. This can be justified by the fact that we use different visual adapters for image and video features, so interleaving the features from both modalities can create a larger distribution shift, hindering the learning process.

Figure: Figure 8: Prompt for Dense Video Captions Generation for VCG+ 112K. We use GPT-4 to generate detailed video captions using concise ground truth and frame-level detailed captions.
Refer to caption: extracted/5664890/Figures_Supp/promtp_1_data.png

Figure: Figure 9: Prompt for Question-answer generation for VCG+ 112K. We use GPT-3.5 to generate question-answer pairs for instruction tuning using the concise video ground truths and detailed video descriptions.
Refer to caption: extracted/5664890/Figures_Supp/promtp_2_data.png

Figure: Figure 10: Prompt for Question-Answer Generation for VCGBench-Diverse. We use GPT-3.5 to generate temporal, spatial, and reasoning question-answer pairs.
Refer to caption: extracted/5664890/Figures_Supp/promtp_3_bench.png

## 11 GPT Prompts

In this section, we provide the GPT prompts used for the following tasks: (i) Dense video description generation for VCG+ 112K, (ii) Question-answer generation for VCG+ 112K and (iii) Question-answer generation for VCGBench-Diverse.

Dense Video Description Generation for VCG+ 112K: To generate dense video captions, we provide GPT-4 with a concise ground truth caption of the video and detailed frame-level captions of the key-frames generated from LLaVA-v1.6 . GPT-4 is then prompted to combine this information into a detailed caption for the entire video. As illustrated in Fig. 8, the prompt includes clear instructions to eliminate any conflicting information, ensuring an accurate and detailed caption.

Question-answer generation for VCG+ 112K: After generating detailed video descriptions using GPT-4, we use GPT-3.5 to create question-answer pairs for instruction tuning. Fig. 9 shows the prompt to generate detailed summary question-answer pair using the ground truth caption and the dense description of the video.

Question-Answer Generation for VCGBench-Diverse: We provide prompts used to generate comprehensive question-answer pairs for VCGBench-Diverse. As illustrated in Fig. 10, the questions are generated in three categories: temporal, spatial, and reasoning. Similar prompts are used to generate consistency and summary questions, offering an extensive evaluation protocol for VCGBench-Diverse.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. What is the instructions provided to the annotators to obtain good captions?
2. Why not directly use the training data from plenty of existing works and see the effectiveness of the idea of integrating image encoder and video encoder?
3. In terms of zero-shot question-answering, is Activitynet data excluded from the training set?
4. Which results does Line 470-475 refer to? I don't see a cross-reference in the paper for this.
5. Are results in tables 463 and 490 comparable? I understand from Lines 480-482 that these numbers might be based on different samples. Please correct me if I'm taking it wrong.
6. By designing the new benchmarks, what are the insights and new conclusions for model design?
soundness: 2
strengths: 1. The idea of combining a high-resolution image encoder and a low-resolution sparse video encoder makes sense. It can somehow reduce the model training cost and inference latency, compared to full high-resolution dense video input.
2. The paper introduces new training data curated by GPT-3.5, which might serve as new training data to feed the community.
3. The paper introduces a new Video-QA benchmark using GPT-3.5 called VCGBench-Divese, which is based on human-curated descriptions.
4. Experiments on various Video-QA benchmarks show the effectiveness of the trained model.
summary: The paper claims that the image encoder could capture rich spatial details while the video encoder could provide temporal contexts with sparse frames as input. Based on such observation, the authors propose to integrate both image encoder embeddings and video encoder embeddings to take both the benefits for video-language understanding. The performance on VCGBench, VCGBench-Diverse, MVBench and VideoMME shows the superiority of the trained model. The paper also proposes a new benchmark, called VCG-Diverse, posing its efforts towards question-answering of diversified video topics.
weaknesses: 1. Related recent literature is missing. The paper has two efforts, one is towards better video understanding with LLM, e.g., [1][2][3], the other one is towards video-language benchmark and training data curation [4][5]. The paper lacks good literature reviews in both directions.
2. Training data curation - necessity and fairness: To illustrate the effectiveness of the method and design. I think the experiments should be at least conducted on same data source and annotation approach. Curating new data  could be a good community contribution, however, the training data from existing works should be used to experiment to show the effectiveness of arch design.
3. Inference data curation - video sources: The usage of HDVILA is doubtable. HDVILA was introduced as a pretraining dataset and an important source of training data. Using it for general Video-QA inference purposes may lead to leakage.
4. Inference data curation - QA quality control: The VCGBench-Diverse QA is generated by GPT. How to control the quality? Is there any manual check involved?
4. Experiments - Fairness: Missing some recent works. Please include the works for fair comparison.
5. Experiments - Ablation: Adequate ablation experiments should be performed on the effectiveness of image encoder branch and video encoder branch on common video question answering benchmarks. In such way, the necessity of the dual branches can be validated.
6. Format - the tables on Pages 9 and 10 don't have captions (excluding Table 4).
7. The authors try to fit too much content into a single paper, which make the contribution of this paper ambiguous. For example, the integration of the image encoder and video encoder doesn't seem to be correlated to the data annotation process and new benchmark. I treasure their efforts, however, this might not be a good practice for me.

[1] Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models
[2] Pllava: Parameter-free llava extension from images to videos for video dense captioning
[3] Slowfast-llava: A strong training-free baseline for video large language models
[4] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions
[5] Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What is the sampling frame rate for the image encoder and the video encoder? How many frames are input to each encoder in total?

2. Have the authors verified if the performance improvement is due to the increased number of vision tokens? What is the performance difference between a pure image encoder and a hybrid encoder with an equivalent number of tokens?
soundness: 2
strengths: 1. The paper is well-written, providing clear explanations of both the model architecture and its rationale.

2. The introduction of VCGBench-Diverse adds a valuable resource for evaluating video models across diverse domains.

3. The model demonstrates improved performance on key benchmarks, indicating robust capabilities in video understanding tasks.
summary: The paper presents VideoGPT+, a multimodal large language model (LMM) designed to enhance video understanding by integrating image and video encoders. VideoGPT+ combines an image encoder for spatial details and a video encoder for temporal context. Using a segment-wise sampling approach and adaptive pooling, the model improves on multiple benchmarks, demonstrating significant gains in video captioning, temporal understanding, and question-answering tasks. Additionally, the authors introduce VCG+112K, a semi-automatically annotated dataset, and VCGBench-Diverse, a comprehensive benchmark spanning 18 video categories, to further assess model generalization in video conversation tasks​.
weaknesses: 1. The approach might be considered trivial, primarily relying on combining two powerful existing encoders without introducing new theoretical insights to advance the field.

2. The paper lacks comparisons with significant baselines, specifically LongVA [1], InternVL [2] and Kangaroo [3], which could provide a clearer context for evaluating its contributions​.


[1] Zhang, Peiyuan, et al. "Long Context Transfer from Language to Vision." arXiv preprint arXiv:2406.16852 (2024). 


[2] Chen, Zhe, et al. "InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks." arXiv preprint arXiv:2312.14238 (2023). 

[3] Mingo Hoffman, Enrico, et al. "Modeling and Numerical Analysis of Kangaroo Lower Body based on Constrained Dynamics of Hybrid Serial-Parallel Floating-Base Systems." arXiv preprint arXiv:2312.04161 (2023).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Since your main modeling contribution is the use of dual encoders, for fair comparisons, could you use the data recipe and model design from an existing work (e.g., VideoLLaVA) and apply your dual encoders to demonstrate the effectiveness of your dual-encoder design across evaluation benchmarks? Similarly, could you train another model architecture on VCG+112K to demonstrate the effectiveness of VCG+112K? That would make the paper's claims more reliable.

2. What is the video source for VCG+112K? Is it ActivityNet? Are the videos in VCG+112K the same set of videos as those in VideoInstruction100K? How many videos are included in VCG+112K?

3. Lines 267–269 mention that the detailed video descriptions generated by GPT-4 include a timeline of events, actions, object attributes, and scene settings. Could you qualitatively show some examples of video descriptions obtained after this step? Since a timeline of events is included, does this imply that a subset of the resulting VCG+112K data could be used to train models like Grounded-VideoLLM for tasks such as temporal grounding? Would it be advisable to choose the dense captions from VCG+112K over other existing video description/caption datasets for training a video description/caption model?

4. Lines 473–475 mention that the video-only model performs better than the image-only model on MVBench action categories. How does the dual-encoder design perform in comparison?

5. For the ablation 'without segment-wise sampling,' did you use uniform frame sampling instead? For the ablation 'without adaptive token pooling,' it is mentioned that this restricts the model to fewer frames. How many frames were used for this ablation? More implementation details are needed for the ablations.

6. In the table showing the results of the VCG+112K ablation study, does the second row indicate the use of VCG+112K in training? The table design is problematic because the second row shows better performance but is marked with a cross mark, suggesting that VCG+112K was not used.

7. Line 524 mentions a combined dataset. Does this mean that you combined all the training data listed in lines 353 to 364? For datasets with multiple splits mentioned in lines 353 to 364, did you use only the training split? Did you use the full dataset for each one mentioned in lines 353 to 364, or was any sampling performed when designing your training data recipe?

8. How is VCGBench-Diverse unique compared to existing benchmarks?

9. How do the new design choices affect the training and inference costs? How does the training data affect the training time?

Minor comments:

In Figure 1 caption: "VideoGPT+ permors better" -> "VideoGPT+ performs better"
soundness: 3
strengths: 1. The model VideoGPT+ is a 3.8B-scale model. The VideoGPT+ model along with the VCG+112K dataset will be valuable resources once refined and released. 

2. The primary design choice in this work—using both an image encoder and a video encoder to develop a video large language model—is reasonable and straightforward.
summary: The authors of the paper propose a new video large language model, VideoGPT+, by (1) using two encoders—an image encoder and a video encoder—to extract visual features, and (2) introducing a new video instruction tuning dataset called VCG+112K, which contains 112K samples generated using a semi-automatic annotation pipeline based on PySceneDetect for keyframe extraction, LLaVA-v1.6 for frame description generation, GPT-4 for detailed video description generation, and finally GPT-3.5 for QA pair generation.

Additionally, they introduce a new benchmark called VCGBench-Diverse, consisting of 4,354 question-answer pairs for 877 videos sourced from HDVILA, MPII, YouCook2, UCF Crime, and STUD Traffic. A human annotation process assisted by GPT-3.5 is used to obtain the QA pairs for this benchmark.
weaknesses: 1. **The novelty and technical depth of this paper are limited. While the paper follows empirical best practices for constructing a multimodal large language model, it lacks the original and novel ideas expected for a conference like ICLR.** The main architectural change introduced in VideoGPT+ is the use of a dual-encoding scheme; however, this approach is neither particularly innovative nor original. In the multimodal large language model domain, the use of two or multiple encoders to extract visual features has been widely explored [1-5]. In terms of ideas, I don’t see a fundamental difference, notable innovation, or new insights.

2. **(a) The introduced benchmark, VCGBench-Diverse, is limited: it is small (containing only 877 videos) and supports only open-ended question answering.** The authors claim that a multiple-choice question-answering format introduces bias and fails to capture the model's true understanding (Lines 136-138), which I disagree with, as open-ended QA, relying on LLM-as-a-judge, often leads to more problematic evaluations compared to multiple-choice QA. In multimodal large language model evaluation, the trend favors using multiple-choice QA over open-ended QA for robust and reliable benchmarking, or incorporating multiple task formats (e.g., TempCompass).
**(b) There are many benchmarks for video conversation models nowadays. It is also unclear how VCGBench-Diverse is unique compared to existing benchmarks**, including those mentioned in this paper, such as VCGBench, MVBench, and VideoMME, as well as others not mentioned, such as TempCompass [6], CVRR-ES [7], Vinoground [8], VideoVista [9], and more [10-14].

3. **State-of-the-art (SOTA) works**, such as LLaVA-NeXT-Video [15], LLaVA-OneVision [16], PLLaVA [17], SlowFast-LLaVA [18], Tarsier [19], MiniGPT4-Video [20], ShareGPT4Video [21], VideoLLaMA 2 [22], etc., **are missing**. The SOTA model comparisons are unconvincing due to the absence of these SOTA models and different training data recipes are used across models. Moreover, the **ablation studies are performed on a selectively chosen and inconsistent subset of evaluation benchmarks**.

4. **Latency is not discussed.**

5. **Clarity of the paper could be improved.**


References:

[1] BRAVE: Broadening the Visual Encoding of Vision-Language Models

[2] Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

[3] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

[4] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

[5] Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

[6] TempCompass: Do Video LLMs Really Understand Videos?

[7] How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs

[8] Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos

[9] VideoVista: A Versatile Benchmark for Video Understanding and Reasoning

[10] AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering

[11] VideoBench: https://github.com/PKU-YuanGroup/Video-Bench

[12] MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding

[13] LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

[14] TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

[15] LLaVA-NeXT: Tackling Multi-Image, Video, and 3D in Large Multimodal Models

[16] LLaVA-OneVision: Easy Visual Task Transfer

[17] PLLaVA: Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

[18] SlowFast-LLaVA- A Strong Training-Free Baseline for Video Large Language Models

[19] Tarsier: Recipes for Training and Evaluating Large Video Description Models

[20] MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

[21] ShareGPT4Video- Improving Video Understanding and Generation with Better Captions

[22] VideoLLaMA 2 Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Are there any studies comparing the effects of using only an image encoder versus using both an image and a video encoder?

2. Can you compare the performance effects between having information interaction and not having information interaction between them?
soundness: 3
strengths: 1. The paper's strengths include its innovative dual-encoder design that effectively combines image and video encoders for richer spatiotemporal understanding of videos. 
2. The introduction of the VCG+ 112K dataset, developed through a semi-automatic annotation pipeline, enhances the model's performance by providing dense video captions and reasoning-based QA pairs. 
3. Furthermore, the proposal of the VCGBench-Diverse benchmark allows for a more comprehensive evaluation of video LMMs across diverse video types and dynamics.
summary: The paper introduces VideoGPT+, a novel video understanding model that integrates image and video encoders to enhance video comprehension by leveraging detailed spatial information and temporal context. It presents VCG+ 112K, a new dataset developed through a semi-automatic annotation pipeline, which improves model performance. Additionally, the paper proposes VCGBench-Diverse, a benchmark covering 18 video categories, for a comprehensive evaluation of video LMMs.
weaknesses: The paper has the following weaknesses.
1. The simultaneous use of both an image and a video encoder increases the number of parameters in the model, making it difficult to determine whether the effectiveness of the paper is accurate or if the performance improvement is simply due to the increased parameters. Are there any studies comparing the effects of using only an image encoder versus using both an image and a video encoder?

2. The image encoder and the video encoder are independent of each other, with no interaction between them. Can you compare the performance effects between having information interaction and not having information interaction between them?

3. The paper lacks some theoretical derivations.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Other suggestions for improvement:
- Additional ablations that would be helpful: use one of the base models mentioned in the paper (e.g. video-ChatGPT) without modification and augment its training with your new instruction tuning data. This would help showing the benefits of the new instruction tuning data.
- Line 053: “uniform sampling” -> uniform frame sampling
- Line 160: ”uniform sampling” -> dense uniform frame sampling
- Line 184: “trained with sparse frames” -> do you mean they are trained with a small number of frames? Or with a small frame rate? This could be made more clear.
- Split experiments in two subsections: first subsection would contain performance evaluation of the full pipeline (model+data) on various existing benchmarks, plus the new benchmark proposed here (lines 340-462). The second subsection would contain all ablations (lines 463-526).
- Section 4: This section does not clearly state what is the source of videos for VCG+ 112K.
soundness: 2
strengths: The paper studies an interesting problem, video question-answering. This is a very active topic in the community that’s advancing rapidly. The potential applications of this technology, when successful, will be interesting.

The paper is generally easy to follow, but I do have some comments on some presentation choices, as outlined later in this review.

The first half of the Experiments section shows strong empirical performance on various benchmarks.
summary: This paper studies the problem of video question-answering (video QA).

The authors claim three novelties/contributions:
1. A video-language model that integrates frame-level image features, clip-level video features and an LLM (VideoGPT+) 
2. A ‘novel semi-automatic annotation pipeline’ to generate instruction-tuning data (VCG+ 112K)
3. A new evaluation benchmark for video QA (VCGBench-Diverse)
weaknesses: My first key concern is the lack of acknowledgement from the authors that the idea of combining video features and image features has been studied in the vision community for a very long time. There is absolutely no discussion on any prior work that has taken this approach, which was very surprising to me. This may give the impression that the authors are first to combine these two feature types, but this is absolutely not the case in the video understanding literature. On the other hand, this paper may indeed be the first in which a video+LLM method combines image and video features, but given the vast amount of prior work that have studied such combinations without LLMs, it makes this contribution not very surprising/significant. 

Here are some examples of methods that combine image and video features for video understanding. I’m sure the list could be made much better with some more time.
* [Two-Stream Convolutional Networks for Action Recognition in Videos. Simonyan & Zisserman. 2014]
* [SlowFast Networks for Video Recognition. Feichtenhofer, Fan, Malik & He. 2019]
* [Camera Motion and Surrounding Scene Appearance as Context for Action Recognition. Heilbron, Thabet, Niebles & Ghanem. 2014]
* [A Closer Look at Spatiotemporal Convolutions for Action Recognition. Tran, Wang, Torresani, Ray, LeCun & Paluri. 2017]
* [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification. Xie, Sun, Huang, Tu & Murphy. 2018]
* [Action Recognition and Detection by Combining Motion and Appearance Features. Wang, Qiao, Tang. 2014]
* [Joint Action Recognition and Pose Estimation From Video. Nie, Xiong &  Zhu. 2015.]



My second key concern with this paper is its lack of focus. The three contributions are somewhat independent, so in a sense it almost feels like they belong to three different papers. The story of image+video features for the model is disconnected from the story of the instruction tuning data, and these two stories are disconnected from the story of the new evaluation benchmark. There is no cohesiveness, so it is unclear what the key research question and hypothesis is here.

In the experiments, there are ablations for the image+video feature combination (463-469). This study includes the new instruction tuning data, so it is easy to compare performance with other work: is the difference in performance due to the feature integration or because this model has additional training data? Here, contributions 1 (model) and 2 (instruction tuning data) are entangled.

There is also an ablation on the instruction tuning data (lines 506-518). Here the comparison is based on the full model that integrates video+image features. Again, is the model without the data competitive? This is not clearly presented here, which makes these two contributions entangled.

While the experiments entangle contributions 1 and 2, their design and implementation are actually disconnected. Or at least, there’s no clear presentation in the paper of how model design and instruction data design influence each other, which makes the reader think they are actually unrelated.

A similar problem happens with the evaluation benchmark. The paper is lacking in discussing how this benchmark relates to the rest of the paper. To be clear, there are cases in which a paper sets up a new problem, and thus needs to introduce a benchmark to evaluate the problem, as well as a method to tackle the problem. Here, the task is well studied (video QA), the key idea of the model innovation is about image+video features, and the benchmark seems to be about domain diversity. In that sense, it is unclear to me why this benchmark and model belong together in the same manuscript.


Experimental support and validation of claims:
1. Claim 1: image+video feature fusion. The authors design a way to fuse image and video features which seems reasonable to me. There’s no discussion on any other prior work that fuses video and image features, and no comparison to alternative design choices for this fusion. The only supporting evidence is an ablation on using image-only, video-only and dual features (lines 463-496) without comparison to alternative designs.
2. Claim 2: ‘novel semi-automatic annotation pipeline’ (line 088) to construct an instruction tuning dataset. There is no discussion in the paper of what makes this data pipeline novel. I understand it’s new data, but the claim here is that the pipeline is novel. If the claim is instead about the generated data instead, then there should be evaluations on the data quality, but these are not provided either. Section 4 seems to emphasize the ‘improvement’, ‘high-quality’ of this data, so not having a direct quality evaluation makes this claim weak. I acknowledge there is an ablation on removing this set from the instruction tuning phase (lines 506-518), but since training data size is not controlled, the performance differences could be attributed to the change in training data size instead of the actual quality of the data. 
3. Claim 3: New benchmark. This is useful for the community, but there is very little information on what makes this evaluation dataset new/novel. I acknowledge the discussion on 18 distinct domains (line 301-319), but there’s no quantitative comparison on what and how many domains are covered on previous datasets. Since the claim here appears to be about increased diversity, there should be some metric and comparison to support it.

Discussion of Related Work:

Missing from the related work section is discussion about: (a) methods for fusion of multiple features, (b) methods that combine image+video features for video understanding, (c) methods for semi-automated dataset annotation.

Another issue in the related work section is at times (for instance lines 120-125) simply a list of prior work without clear insights of what’s their limitation, or how they relate to this paper. Creating a relationship to prior work is important for the reader to better appreciate the work.

Presentation:
1. Paragraph 1 from the introduction (lines 037-048) makes assertions that are written generally about “video understanding”, but instead apply only to video-language models. To correct this, I suggest modifying the first sentence to include the scope more clearly on ‘video-language’, rather than general video understanding.
2. Through the paper, the authors talk about ‘local features’ and ‘global features’ (line 051, Figure 2, line 205, etc). My opinion is that this is a misnomer. What the authors name as ‘local features’ are really static frame-level features, and ‘global features’ are really dynamic clip-level features. That is, these dynamic clip-level features are not really global for the reasons claimed by the authors: 1) the frame resolution is not too different from the static image features (336x336 vs 224x224, line 346), and these features also come in a spatial grid so they are not spatially global either. I suggest removing the local/global name and instead use static/dynamic or image/clip features.