Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

## Abstract

Abstract Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM , a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM , we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM ’s temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.

## 1 Introduction

Multi-modal Large Language Models (MLLMs) have made remarkable progress in image-level understanding (Liu et al., ; Dai et al., ; Li et al., ). However, extending their capabilities to the video domain poses distinct challenges. Unlike static images, the temporal nature of videos challenges models to process not only visual content but also the sequence and timing of events. While current Video-LLMs (Xu et al., ; Li et al., ; Zhang et al., ; Lin et al., ) are capable of capturing global visual semantics and generating coarse-grained captions for short clips, they struggle with fine-grained video understanding (Liu et al., ; Wang et al., ), which requires decomposing the video along the temporal axis to accurately perceive and reason over specific moments, such as subtle actions, transitions, and events that unfold over time.

Figure: Figure 1: Grounded-VideoLLM enables Temporal Referring/Localizing/Reasoning for MLLM.
Refer to caption: x1.png

Previous research efforts (Ren et al., ; Huang et al., ; Qian et al., ; Huang et al., ; Guo et al., ) have explored temporal grounding to improve fine-grained video understanding. However, two main challenges impede their potential for achieving effective fine-grained temporal grounding: (1) Models like Video-ChatGPT (Maaz et al., ), P-LLaVA (Xu et al., ), and Video-LLAMA (Zhang et al., ) typically sample multiple frames from a video and encode each frame independently using an image encoder, followed by a feature projector (e.g., sliding Q-former (Ren et al., ), slot-based token compression (Guo et al., ), or visual adapter (Huang et al., )). This approach focuses primarily on spatial details while potentially neglecting the temporal relationships between frames (Maaz et al., ). (2) Current models also struggle with timestamp representation, which is crucial for pinpointing specific moments in time for fine-grained understanding. Models such as TimeChat (Ren et al., ) and VTimeLLM (Huang et al., ) represent timestamps as plain texts, for example, ["from 102.3 to 120.1 seconds"]. Despite being straightforward, this strategy needs to tokenize continuous floating-point values, which is inefficient for LLMs since their next-token prediction paradigm struggles with handling numerical data (Schwartz et al., ; Frieder et al., ).

To further improve video comprehension, we propose to sharpen the model with fine-grained temporal grounding, which allows the model to not only recognize what happens but also pinpoint when it happens in a fine-grained manner. We identify three essential capabilities of fine-grained temporal grounding, as illustrated in Figure 1, including Temporal Referring (Huang et al., ), Localizing (Gao et al., ; Caba Heilbron et al., ), and Reasoning (Xiao et al., ; Qian et al., ). Referring involves the model precisely describing events within user-specified time intervals. Localizing challenges the model to identify the timestamps of a given query and construct a coherent storyline by locating a sequence of events within the video. Reasoning combines both Referring and Localizing capabilities and further requires the ability to answer questions by understanding temporal dynamics and applying wrold knowledge. Achieving these tasks necessitates both effective temporal modeling and robust timestamp representation.

Targeting these goals, we introduce Grounded-VideoLLM, a novel Video-LLM that can perceive and reason over specific video moments with fine-grained precision. From the perspective of model architecture, Grounded-VideoLLM is built upon two key innovations: (1) Two-Stream Encoding: We decompose each segment of the video into spatial and temporal components, and encode each with an expert encoder respectively. The temporal stream extracts motion representations from dense frames and complements the spatial stream which captures appearance representations. This dual-stream approach forms comprehensive video representations enriched with both temporal and spatial information. (2) Temporal Tokens: We extend the LLM’s vocabulary by introducing discrete tokens specifically crafted for timestamp representation. These temporal tokens denote relative time positions within a video and share a unified embedding space with the LLM. This integration allows Grounded-VideoLLM to avoid the inefficiency of tokenizing numerical text, enabling seamless prediction of both timestamps and textual outputs in a single sequence of discrete tokens. From the perspective of training and datasets, given that such a unified model requires extensive pre-training, we start with an image-based MLLM (Microsoft, ) as the foundation and adopt a three-stage training strategy to ensure efficiency. We meticulously select different tasks for each stage, and progressively refine the model in a “coarse-to-fine” manner, transitioning from image understanding to video comprehension, and ultimately to fine-grained temporal grounding. Furthermore, we enhance the model’s temporal reasoning capability by curating 17K grounded VideoQA (Xiao et al., ) samples with the assistance of GPT-4. Extensive experiments demonstrate that Grounded-VideoLLM shows promising results over existing Video-LLMs not only in traditional video temporal grounding tasks but also in general video understanding benchmarks.

In summary, we make the following contributions: (1) We present key insights into fine-grained temporal grounding for Video-LLMs and introduce Grounded-VideoLLM, a Video-LLM adept at perceiving and reasoning over specific video moments. This is achieved through a two-stream architecture for effective temporal modeling and the temporal tokens for efficient timestamp representation. (2) We propose a step-by-step training strategy that progressively adapts an image-based MLLM into a robust Video-LLM, while curating a grounded VideoQA dataset to further enhance the temporal reasoning capability. (3) We conduct extensive experiments across various tasks, including temporal sentence grounding, dense video caption, VideoQA, and general Video-LLM benchmarks, demonstrating the superiority of Grounded-VideoLLM for fine-grained video understanding.

## 2 Related Work

Video Large Language Models have caught a growing interest with the advancements in image-based MLLMs (Zhang et al., ; Lin et al., ; Maaz et al., ; Luo et al., ). Despite promising results, current Video-LLMs, such as P-LLaVA (Xu et al., ) and Video-ChatGPT (Maaz et al., ), often struggle with temporal understanding (Liu et al., ) and exhibit temporal hallucination (Wang et al., ) when answering questions about specific video moments. These models encode each frame independently using a pre-trained image encoder, concatenating the frame embeddings to create a video representation. This late fusion method can result in video representations that lack inherent temporal information and heavily rely on the position embeddings of LLM (Su et al., ) for temporal understanding, limiting the model’s capability to perform fine-grained temporal grounding and understanding. In contrast to these studies, we employ a two-stream architecture that integrates a video expert to extract motion features to complement the appearance features during the early encoding process. Additionally, we employ a progressive training strategy that gradually adapts an image-based MLLM for fine-grained video understanding. Unlike concurrent studies such as SlowFast-LLaVA (Xu et al., ) and VideoGPT+ (Maaz et al., ), which also utilize a two-stream architecture, we specifically targets fine-grained temporal grounding through a unique encoding/pooling/training strategy for dense frames and grounding design.

Video Temporal Grounding (VTG) aims to associate specific video moments with their corresponding timestamps. Traditional VTG tasks include Temporal Sentence Grounding (Gao et al., ; Hendricks et al., ) and Dense Video Captioning (Caba Heilbron et al., ; Zhou et al., ). Other tasks, such as Grounded VideoQA (Xiao et al., ), emphasize reasoning over videos with grounded information. Given the emerging reasoning capabilities of Video-LLMs, many studies have investigated how to adapt them for VTG tasks. For example, TimeChat (Ren et al., ), VTimeLLM (Huang et al., ), and LITA (Huang et al., ) perform temporal grounding using a fully text-to-text approach through instruction-tuning datasets. Momentor (Qian et al., ) introduces a temporal perception module to address the quantization errors associated with time tokens, while VTG-LLM (Guo et al., ) incorporates a limited set of absolute-time tokens to handle timestamp knowledge. Compared to these studies, we avoid textual representation of timestamps and instead introduce discrete and relative temporal tokens for more efficient timestamp encoding. While Vid2Seq (Yang et al., ) also employs specialized tokens to indicate temporal positions, it relies heavily on large-scale pre-training from scratch using noisy transcribed speech and is limited to dense video captioning. Our two-stream architecture and progressive training strategy, however, enable MLLMs to efficiently comprehend videos and effectively handle diverse tasks of fine-grained temporal referring, localizing, and reasoning.

Figure: Figure 2: Overview of Grounded-VideoLLM. For temporal modeling, we employ a segment-wise encoding strategy by decomposing each segment into a spatial part and a temporal part and encoding each respectively. For timestamp representation, we introduce additional special temporal tokens sharing a unified embedding space with LLM.
Refer to caption: x2.png

## 3 Model Architecture

Given that current MLLMs already exhibit strong image-understanding capabilities, our architecture aims to sharpen temporal awareness by capturing motion dynamics across frames, which serve as a vital supplement to spatial content. As shown in Figure 2, we develop Grounded-VideoLLM upon a well-established MLLM for spatial comprehension and integrate an expert video encoder for temporal comprehension. Additionally, to avoid tokenizing numerical texts, we incorporate temporal tokens into the LLM’s vocabulary for efficient and unified timestamp representation.

### 3.1 Two-Stream Encoding

To effectively model the long-range temporal structure, given a video $\mathcal{V}$ with $T$ frames, we divide it into $K$ segments and employ a segment-wise encoding strategy (Wang et al., ). Due to the inherent redundancy of consecutive frames in videos, each segment can be naturally represented from two perspectives: spatial and temporal. The spatial representation of each segment is derived from an individual keyframe, capturing the primary appearance semantics, while the temporal representation is learned from multiple frames depicting the motion evolution within the segment. We discuss the details of our two-stream segment encoding as follows.

Spatial Stream. We sample the middle frame from each segment as the keyframe and extract its spatial features using the original image encoder from the MLLM (Radford et al., ), resulting in spatial features $\textbf{F}_{S}\in\mathbb{R}^{H_{S}\times W_{S}\times D_{S}}$, where $H_{S}$, $W_{S}$, $D_{S}$ denote the height, width and dimension of the spatial features. Since dense frames are crucial for fine-grained temporal grounding, an appropriate pooling strategy is required to reduce token length for efficient computation. As indicated by Xu et al. () and Yao et al. () that a 2D average pooling is both efficient and robust for spatial downsampling, we employ a 2D pooling kernel with a size $\sigma_{S}\times\sigma_{S}$ over the feature map and gets $\textbf{F}_{S}\in\mathbb{R}^{N_{S}\times D_{S}}$ as the feature for spatial stream, where $N_{S}=\frac{H_{S}}{\sigma_{S}}\times\frac{W_{S}}{\sigma_{S}}$.

Temporal Stream. Traditional two-stream networks (Simonyan & Zisserman, ; Feichtenhofer et al., ) typically encode the optical flow as the temporal stream. However, given the scale of data and parameters involved in MLLMs, extracting optical flow is computationally expensive and impractical. Consequently, we resort to a strong and well pre-trained video encoder, InternVideo2 (Wang et al., ), to extract motion representations for each segment, using a lower resolution but more frames.
Specifically, we input each segment, containing $\frac{T}{K}$ frames, into the video encoder to obtain the segment-level features $\textbf{F}_{T}\in\mathbb{R}^{\frac{T}{K}\times H_{T}\times W_{T}\times D_{T}}$, where $H_{T}$, $W_{T}$, $D_{T}$ denote the height, width and dimension of each frame feature. Similar to the spatial stream, we apply a 2D average pooling strategy to downsample $\textbf{F}_{T}$. However, as the temporal stream focuses primarily on temporal modeling, we retain the complete temporal information by only pooling along the spatial dimensions. Specifically, we aggressively downsample $\textbf{F}_{T}$ using a kernel with a larger size of $\sigma_{T}\times\sigma_{T}$, resulting in the compressed $\textbf{F}_{T}\in\mathbb{R}^{\frac{T}{K}\times N_{T}\times D_{T}}$ for temporal stream, where $N_{T}=\frac{H_{T}}{\sigma_{T}}\times\frac{W_{T}}{\sigma_{T}}$.

To get the complete segment-level representation, we flatten the features of the spatial stream and temporal stream and concat them together :

$$ $\displaystyle\textbf{F}_{Seg}=\operatorname{Concat}\left[\operatorname{Flatten }(f(\textbf{F}_{S}));\operatorname{Flatten}(g(\textbf{F}_{T}))\right],\textbf{ F}_{Seg}\in\mathbb{R}^{(N_{S}+\frac{T}{K}\cdot N_{T})\times D}$ (1) $$

where $f(\cdot)$ and $g(\cdot)$ are two MLPs that project the visual features to LLM’s dimension $D$. The final video representation is formed by concatenating the $K$ segment-level representations $\textbf{F}_{Seg}$, resulting in $\textbf{F}_{Vid}\in\mathbb{R}^{K\cdot(N_{S}+\frac{T}{K}\cdot N_{T})\times D}$. This representation retains detailed spatial information across all segments along with their global temporal contexts, while maintaining a manageable token length. The combined video representation $\textbf{F}_{Vid}$ is then fed into the LLM serving as soft prompts, alongside the word embeddings of the instruction text $\textbf{F}_{Text}$ to generate the target response $\mathcal{A}$. The model is trained using the cross-entropy loss function with trainable parameters $\theta$:

$$ $\displaystyle\mathcal{L}=-\sum_{t=1}^{L_{a}}logP_{\theta}(\mathcal{A}_{t}| \mathcal{A}_{<t},\textbf{F}_{Vid},\textbf{F}_{Text})$ (2) $$

where $\mathcal{A}_{t}$ is predicted autoregressively at position $t$, and $L_{a}$ is the sequence length of the ground truth answer text $\mathcal{A}$.

### 3.2 Unified Temporal Tokens

Given a text span depicting a particular video clip and its associated timestamps, we employ a relative time representation that converts continuous timestamps into a sequence of discrete temporal tokens. For a video $\mathcal{V}$ with a duration of $L$ seconds, we evenly divide $\mathcal{V}$ into $M$ equal-length chunks, and then define $M+1$ anchor points (ranging from <0> to <M>) across $\mathcal{V}$, representing relative temporal positions. Each anchor point corresponds to a specific timestamp of $\mathcal{V}$ and is encoded as a temporal token. For instance, <0> denotes the very start of $\mathcal{V}$ while <M> indicates the end. These $M+1$ tokens are added to the LLM’s vocabulary to enable unified modeling alongside text. A specific continuous timestamp $\tau$ can be easily converted to a temporal token <t> and vice verse:

$$ $\displaystyle t=\operatorname{Round}(M\cdot\frac{\tau}{L}),\ \ \ \tau=L\cdot \frac{t}{M}$ (3) $$

While this may introduce minor quantization errors, these can be minimized by selecting an appropriate $M$. We then organize the text span and its corresponding temporal tokens in a unified format. Both text tokens and temporal tokens are mapped to embeddings through the extended word embedding layer of LLM. For example, one input representation is as follows:

where 
 and 
 indicate start- and end-of-sequence, <video> and </video> represent the beginning and end of encoded video representations. <grounded> is a special token to tell the model should output the grounded timestamps. This strategy avoids the need to tokenize and process numerical values, which has been identified as a limitation of LLMs (Schwartz et al., ), and it greatly simplifies the representation of timestamps within the unified embedding space of LLMs. Consequently, text and timestamps can be jointly decoded as a single sequence, following the objective outlined in Eq. (2).

## 4 Progressive Training

Different from previous methods (Zhang et al., ; Lin et al., ) that train models from scratch using mixed image and video datasets, we start with a pre-trained image-based MLLM (Microsoft, ) and progressively enhance its fine-grained temporal grounding capabilities. This strategy can be applied to any off-the-shelf MLLM and is more efficient. Table 1 enumerates the datasets used at different training stages, and Table 9 lists the prompts for different tasks.

Stage-1: Video-Caption Alignment. Feature alignment is widely used to improve training efficiency (Liu et al., ). In this stage, we leverage approximately 1.28 million video-text pairs sampled from diverse sources (Wang et al., ; Bain et al., ; Chen et al., ) to align the video encoder’s features with the MLLM. This alignment allows the MLLM, which was pre-trained solely on images, to gain a foundational understanding of videos. Since 2D down-sampling has been applied to the visual features, only the two projectors $f(\cdot)$ and $g(\cdot)$ are set to be trainable, while the video encoder, image encoder, and LLM remain frozen. As this stage does not involve any video temporal grounding tasks, the temporal tokens described in Sec.3.2 are not yet incorporated.

Stage-2: Temporal Token Alignment. While video-caption alignment effectively connects videos and the MLLM at a coarse semantic level, a gap persists between this alignment and fine-grained temporal grounding. To address this, we introduce the temporal tokens described in Sec.3.2 and continue pre-training the model on a diverse range of grounding datasets (Huang et al., ; Qian et al., ; Wang et al., ), focusing on tasks such as Temporal Sentence Grounding, Dense Video Captioning, and Temporal Referring, which enables the model to refer to and localize temporal information effectively. Since new tokens are introduced, the trainable parameters in this stage include the two projectors, $f(\cdot)$ and $g(\cdot)$, the word embedding matrix, and the final classifier head of the LLM. This step enhances the model’s ability to comprehend multiple events and aligns the temporal tokens with both the video timelines and the LLM’s semantic space.

Stage-3: Multi-Task Instruction Tuning.
Following the initial two stages of pre-training, the model has developed a basic understanding of video content and the ability to refer to and locate specific timestamps. In this stage, we will further enhance the model’s fine-grained temporal grounding while improving its responsiveness to diverse user instructions. To achieve this, we gather two types of datasets: (1) We compile a wide range of public datasets for video temporal grounding tasks, similar to Time-IT (Ren et al., ) and VTG-IT (Guo et al., ), which include tasks of dense video captioning, temporal sentence grounding, and grounded VideoQA. (2) We incorporate a selection of instructional video-to-text datasets from VideoChat2 (Li et al., ), which feature tasks such as conversations, classification, question answering, and captioning. Additionally, we include ShareGPT-4Video (Chen et al., ) to further enhance the model’s ability to generate detailed video captions. By utilizing these diverse datasets, which encompass both temporal grounding and video instruction tasks, Grounded-VideoLLM excels in temporal referring, localization, reasoning, and general comprehension of video content. In this stage, the trainable parameters remain the same as in Stage 2, with the addition of LoRA parameters (Hu et al., ) for the LLM.

**Table 1: Datasets used at three training stages. Tasks with gray background consist of datasets regarding temporal grounding.**
| Training Stage | Task | # of Samples | Datasets |
| --- | --- | --- | --- |
| Video-Caption Alignment | Video Captioning | 1.28M | WebVid-10M, Panda-70M, InternVid-10M |
| Temporal Token Alignment | Temporal Sentence Grounding | 149K | VTimeLLM-Stage2 |
| Dense Video Captioning | 92K | VTimeLLM-Stage2, Moment-10M, InternVid-G |  |
| Temporal Referring | 95K | VTimeLLM-Stage2, InternVid-G |  |
| Multi-Task Instruction Tuning | Grounded Conversation | 442K | ANet-RTL, Moment-10M |
| Temporal Sentence Grounding | 84K | DiDeMo, HiREST, QuerYD, VTG-IT |  |
| Dense Video Caption | 41K | COIN, ViTT, YouCook2, VTG-IT |  |
| Grounded VideoQA | 17K | Self Collected |  |
| Converstation | 233K | VCG-Plus-112K, Videochatgpt-100K, Videochat2-Conv |  |
| VideoQA | 282K | EgoQA, NExT-QA, Intent-QA, STAR, CLEVRER, WebVid-QA |  |
| Classification | 66K | SthSthV2, Kinetics |  |
| Video Captioning | 136K | TextVR, YouCook2, WebVid, ShareGPT4Video |  |

Figure: Figure 3: Examples of annotation pipeline and generated data for Grounded VideoQA.
Refer to caption: x3.png

## 5 Grounded VideoQA Dataset Generation

Grounded VideoQA requires the model to not only answer questions but also identify relevant video moments that support predicted answers, thereby demonstrating the model’s temporal reasoning abilities. The NExT-GQA dataset (Xiao et al., ) was manually developed by extending NExT-QA (Xiao et al., ) with temporal labels for start and end timestamps. However, annotating these temporal labels is labor-intensive and time-consuming, which has limited NExT-GQA only to QA pairs for the validation and test sets. To create a scalable training dataset, we utilized OpenAI GPT-4 (Achiam et al., ) to assist in constructing training sets for the grounded VideoQA task. These sets were built on public datasets that already contain temporal labels, such as ActivityNet-Caption (Caba Heilbron et al., ) and QVHighlights (Lei et al., ). We framed the task as a multiple-choice problem using a two-round conversational format, as depicted in Figure 3.

Specifically, we input event descriptions along with their timestamps into GPT-4 and prompted it to generate corresponding question-answer pairs, as shown in Table 8. To create distractor options for the multiple-choice questions, we retrieved the top 50 questions most similar to the generated question, based on cosine similarity using an embedding model (Reimers, ). The answers to these 50 retrieved questions served as candidates for distractors. From this pool, we randomly sampled four distractors with cosine similarities to the correct answer ranging from 0.2 to 0.9, ensuring that the distractors were contextually relevant but not overly similar to the correct answer. The ground-truth timestamps for answering each question were derived from the timestamps of the associated event descriptions. The constructed dataset comprises 17K samples, which have been incorporated into the training sets for Stage 3, further enhancing the model’s temporal reasoning performance.

## 6 Experiments

### 6.1 Experiment Setting

Implementation Details.
We select the Phi3.5-Vision-Instruct-3.8B (Microsoft, ) as the base MLLM for our Grounded-VideoLLM. For temporal stream encoding, we adopt InternVideo2-1B (Wang et al., ) as the video encoder. Each video is sampled as a sequence of $T=96$ frames, which are evenly divided into $K=12$ segments. We set the pooling size $\sigma_{S}=2$ for the spatial stream while $\sigma_{T}=4$ for the temporal stream respectively, which results in $N_{S}=144$ tokens per frame for the spatial stream while $N_{T}=16$ tokens per frame for the temporal stream. Moreover, we introduce $M=300$ temporal tokens into the LLM’s vocabulary for timestamp representation. More implementation details can be found in Appendix A.1.

Tasks and Benchmarks.
To thoroughly evaluate Grounded-VideoLLM in fine-grained temporal grounding, we assess it across three video temporal grounding tasks: Temporal Sentence Grounding, Dense Video Captioning, and Grounded VideoQA, utilizing datasets such as Charades-STA (Gao et al., ), ActivityNet-Captions (Caba Heilbron et al., ), and NExT-GQA (Xiao et al., ). We also show its reasoning capability by the task of Open-Ended VideoQA with datasets including MSVD-QA, MSRVTT-QA (Xu et al., ), and ActicityNet-QA (Yu et al., ). Additionally, to evaluate the model’s general video understanding capabilities, we benchmark Grounded-VideoLLM against existing models using modern Video-LLM benchmarks including VCG-Bench (Maaz et al., ) and MVBench (Li et al., ).

**Table 2: Zero-shot results on temporal sentence grounding and dense video captioning tasks.**
| Model | LLM | Charades-STA | ActivityNet-Grounding | ActivityNet-Captions |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Scale | R@0.3 | R@0.5 | R@0.7 | mIoU | R@0.3 | R@0.5 | R@0.7 | mIoU | SODA_c | METEOR |  |
| Video-LLaMA (Zhang et al., ) | 7B | 25.2 | 10.6 | 3.4 | 16.8 | 21.9 | 10.8 | 4.9 | 16.5 | 1.9 | 1.9 |
| SeViLA (Yu et al., ) | 3B | 27.0 | 15.0 | 5.8 | 18.3 | 31.6 | 19.0 | 10.1 | 23.0 | - | - |
| Video-ChatGPT (Maaz et al., ) | 7B | 27.2 | 6.2 | 1.9 | 19.7 | 19.5 | 10.6 | 4.8 | 14.2 | 1.9 | 2.1 |
| Valley (Luo et al., ) | 7B | 28.4 | 1.8 | 0.3 | 21.4 | 30.6 | 13.7 | 8.1 | 21.9 | 0.3 | 0.8 |
| VideoChat2 (Li et al., ) | 7B | 38.0 | 14.3 | 3.8 | 24.6 | 40.8 | 27.8 | 9.3 | 27.9 | - | - |
| VideoChat (Li et al., ) | 7B | 32.8 | 8.6 | 0.0 | 25.9 | 23.5 | 12.6 | 6.0 | 17.4 | 0.9 | 0.9 |
| Momenter (Qian et al., ) | 7B | 42.6 | 26.6 | 11.6 | 28.5 | 42.9 | 23.0 | 12.4 | 29.3 | 2.3 | 4.7 |
| VTimeLLM (Huang et al., ) | 7B | 51.0 | 27.5 | 11.4 | 31.2 | 44.0 | 27.8 | 14.3 | 30.4 | 5.8 | 6.8 |
| TimeChat (Ren et al., ) | 7B | - | 32.2 | 13.4 | - | - | - | - | - | - | - |
| VTG-LLM (Guo et al., ) | 7B | - | 33.8 | 15.7 | - | - | - | - | - | - | - |
| HawkEye (Wang et al., ) | 7B | 50.6 | 31.4 | 14.5 | 33.7 | 49.1 | 29.3 | 10.7 | 32.7 | - | - |
| Grounded-VideoLLM | 4B | 54.2 | 36.4 | 19.7 | 36.8 | 46.2 | 30.3 | 19.0 | 36.1 | 6.0 | 6.8 |

**Table 3: Results on NExT-GQA. Acc@GQA is defined as the percentage of questions that are both correctly answered and visually grounded with IoP $\geq$ 0.5.**
| Model | Acc@GQA | mIoP | IoP@0.3 | IoP@0.5 | mIoU | IoU@0.3 | IoU@0.5 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| VIOLETv2 (Fu et al., ) | 12.8 | 23.6 | 25.1 | 23.3 | 3.1 | 4.3 | 1.3 |
| Temp[CLIP] NG+ (Xiao et al., ) | 16.0 | 25.7 | 31.4 | 25.5 | 12.1 | 17.5 | 8.9 |
| SeViLA (Yu et al., ) | 16.6 | 29.5 | 34.7 | 22.9 | 21.7 | 29.2 | 13.8 |
| HawkEye (Wang et al., ) | - | - | - | - | 25.7 | 37.0 | 19.5 |
| LangRepo (Kahatapitiya et al., ) | 17.1 | 31.3 | - | 28.7 | 18.5 | - | 12.2 |
| FrozenBiLM NG+ (Yang et al., ) | 17.5 | 24.2 | 28.5 | 23.7 | 9.6 | 13.5 | 6.1 |
| VideoStreaming (Qian et al., ) | 17.8 | 32.2 | - | 31.0 | 19.3 | - | 13.3 |
| LLoVi (Zhang et al., ) | 24.3 | 37.3 | - | 36.9 | 20.0 | - | 15.3 |
| Grounded-VideoLLM | 26.7 | 34.5 | 42.6 | 34.4 | 21.1 | 30.2 | 18.0 |

Evaluation Metrics.
For temporal sentence grounding, we report the metric of Intersection over Union (IoU) (Gao et al., ) between the timestamps predicted by the model and the ground truth, including Recall at IoU thresholds of {0.3, 0.5, 0.7} and their mean IoU. For dense video captioning, we use metrics including SODA_c (Fujita et al., ) which is specifically tailored for the video’s storyline, and METEOR score (Banerjee & Lavie, ), which is the average of traditional METEOR scores that are calculated based on matched pairs between generated events and the ground truth across IoU thresholds of {0.3, 0.5, 0.7, 0.9}. For Visually-grounded VideoQA, we calculate both the Intersection of Prediction (IoP) (Xiao et al., ) and Intersection of Union (IoU), and use Acc@GQA (Xiao et al., ) to measure the percentage of questions that are both correctly answered and visually grounded with IoP $\geq$ 0.5. The responses of Open-Ended VideoQA and VCG-Bench are evaluated by GPT-3.5 with the prompts introduced by Video-ChatGPT (Maaz et al., ). More evaluation details are in Appendix A.3.

### 6.2 Main Results

Temporal Sentence Grounding requires the model to identify the precise time interval corresponding to a given query sentence. As shown in Table 2, Grounded-VideoLLM achieves performance on “mIoU” with 36.8 for the Charades-STA (Gao et al., ) and 36.1 for ActivityNet-Grounding (Caba Heilbron et al., ) respectively, surpassing previous SoTA end-to-end Video-LLMs, i.e., HawkEye (Wang et al., ), by a significant margin (+3.4). It is worth mentioned that the promising performance of “mIoU” are largely attributed to the signigicant gains in terms of ”R@0.7” compared with other thresholds, demonstrating that Grounded-VideoLLM is more advanced in localizing specific moments within videos with finer granularity.

Dense Video Captioning involves generating descriptions for all events in a video, along with their corresponding start and end timestamps. We evaluated Grounded-VideoLLM on the ActivityNet-Captions (Caba Heilbron et al., ), and the results in Table 2 show that our method achieves the highest SODA_c score (6.0), which demonstrates that, thanks to the Temporal Token Alignment training stage, Grounded-VideoLLM is highly effective in identifying the multi-event structure of the video and capturing complete storylines. The highest METEOR score (6.8) also indicates that Grounded-VideoLLM provides more detailed event descriptions compared with other Video-LLMs.

**Table 4: Results on zero-shot Open-Ended VideoQA and VCG-Bench. VCG-Bench contains five aspects: Correctness of Information (CI), Detail Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU), and Consistency (CO).**
| Model | MSVD-QA | MSRVTT-QA | ANet-QA | VCG-Bench |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc. | Score | Acc. | Score | Acc. | Score | CI | DO | CU | TU | CO | Avg. |  |
| Video-LLMs w/o temporal grounding capability. |  |  |  |  |  |  |  |  |  |  |  |  |
| Video-LLaMA (Zhang et al., ) | 51.6 | 2.5 | 29.6 | 1.8 | 12.4 | 1.1 | 1.96 | 2.18 | 2.16 | 1.82 | 1.79 | 1.98 |
| Video-ChatGPT (Maaz et al., ) | 64.9 | 3.3 | 49.3 | 2.8 | 35.2 | 2.7 | 2.50 | 2.57 | 2.69 | 2.16 | 2.20 | 2.42 |
| Vista-LLaMA (Ma et al., ) | 65.3 | 3.6 | 60.5 | 3.3 | 48.3 | 3.3 | 2.44 | 2.64 | 3.18 | 2.26 | 2.31 | 2.57 |
| MovieChat (Song et al., ) | 75.2 | 3.8 | 52.7 | 2.6 | 45.7 | 3.4 | 2.76 | 2.93 | 3.01 | 2.24 | 2.42 | 2.67 |
| LongVLM (Weng et al., ) | 70.0 | 3.8 | 59.8 | 3.3 | 47.6 | 3.3 | 2.76 | 2.86 | 3.34 | 2.39 | 3.11 | 2.89 |
| VideoChat2 (Li et al., ) | 70.0 | 3.9 | 54.1 | 3.3 | 49.1 | 3.3 | 3.02 | 2.88 | 3.51 | 2.66 | 2.81 | 2.98 |
| Chat-UniVi (Jin et al., ) | 65.0 | 3.6 | 54.6 | 3.1 | 45.8 | 3.2 | 2.89 | 2.91 | 3.46 | 2.89 | 2.81 | 2.99 |
| P-LLaVA-7B (Xu et al., ) | 76.6 | 4.1 | 62.0 | 3.5 | 56.3 | 3.5 | 3.21 | 2.86 | 3.62 | 2.33 | 2.93 | 3.12 |
| ST-LLM (Liu et al., ) | 74.6 | 3.9 | 63.2 | 3.4 | 50.9 | 3.3 | 3.23 | 3.05 | 3.74 | 2.93 | 2.81 | 3.15 |
| VideoGPT+ (Maaz et al., ) | - | - | - | - | - | - | 3.27 | 3.18 | 3.74 | 2.83 | 3.39 | 3.28 |
| Video-LLMs w/ temporal grounding capability. |  |  |  |  |  |  |  |  |  |  |  |  |
| Momentor (Qian et al., ) | 68.9 | 3.6 | 55.6 | 3.0 | 40.8 | 3.2 | - | - | - | - | - | - |
| VTimeLLM (Huang et al., ) | - | - | - | - | - | - | 2.78 | 3.10 | 3.40 | 2.49 | 2.47 | 2.85 |
| LITA (Huang et al., ) | - | - | - | - | - | - | 2.94 | 2.98 | 3.43 | 2.68 | 3.19 | 3.04 |
| Grounded-VideoLLM | 76.3 | 4.1 | 60.3 | 3.6 | 56.8 | 3.5 | 3.34 | 2.94 | 3.66 | 3.12 | 3.14 | 3.24 |

**Table 5: Results on MVBench multi-choice question answering.**
| Model | Avg. | AS | AP | AA | FA | UA | OE | OI | OS | MD | AL | ST | AC | MC | MA | SC | FP | CO | EN | ER | CI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VideoChatGPT (Maaz et al., ) | 32.7 | 23.5 | 26.0 | 62.0 | 22.5 | 26.5 | 54.0 | 28.0 | 40.0 | 23.0 | 20.0 | 31.0 | 30.5 | 25.5 | 39.5 | 48.5 | 29.0 | 33.0 | 29.5 | 26.0 | 35.5 |
| VideoLLaMA (Zhang et al., ) | 34.1 | 27.5 | 25.5 | 51.0 | 29.0 | 39.0 | 48.0 | 40.5 | 38.0 | 22.5 | 22.5 | 43.0 | 34.0 | 22.5 | 32.5 | 45.5 | 32.5 | 40.0 | 30.0 | 21.0 | 37.0 |
| VideoChat (Li et al., ) | 35.5 | 33.5 | 26.5 | 56.0 | 33.5 | 40.5 | 53.0 | 40.5 | 30.0 | 25.5 | 27.0 | 48.5 | 35.0 | 20.5 | 42.5 | 46.0 | 26.5 | 41.0 | 23.5 | 23.5 | 36.0 |
| TimeChat (Ren et al., ) | 38.5 | 40.5 | 36.0 | 61.0 | 32.5 | 53.0 | 53.5 | 41.5 | 29.0 | 19.5 | 26.5 | 66.5 | 34.0 | 20.0 | 43.5 | 42.0 | 36.5 | 36.0 | 29.0 | 35.0 | 35.0 |
| Video-LLaVA (Lin et al., ) | 43.0 | 46.0 | 42.5 | 56.5 | 39.0 | 53.5 | 53.0 | 48.0 | 41.0 | 29.0 | 31.5 | 82.5 | 45.0 | 26.0 | 53.0 | 41.5 | 33.5 | 41.5 | 27.5 | 38.5 | 31.5 |
| P-LLaVA-7B (Xu et al., ) | 46.6 | 58.0 | 49.0 | 55.5 | 41.0 | 61.0 | 56.0 | 61.0 | 36.0 | 23.5 | 26.0 | 82.0 | 39.5 | 42.0 | 52.0 | 45.0 | 42.0 | 53.5 | 30.5 | 48.0 | 31.0 |
| VideoChat2 (Li et al., ) | 51.1 | 66.0 | 47.5 | 83.5 | 49.5 | 60.0 | 58.0 | 71.5 | 42.5 | 23.0 | 23.0 | 88.5 | 39.0 | 42.0 | 58.5 | 44.0 | 49.0 | 36.5 | 35.0 | 40.5 | 65.5 |
| ShareGPT4Video (Chen et al., ) | 51.2 | 49.5 | 39.5 | 79.5 | 40.0 | 54.5 | 82.5 | 54.5 | 32.5 | 50.5 | 41.5 | 84.5 | 35.5 | 62.5 | 75.0 | 51.0 | 25.5 | 46.5 | 28.5 | 39.0 | 51.5 |
| ST-LLM (Liu et al., ) | 54.9 | 66.0 | 53.5 | 84.0 | 44.0 | 58.5 | 80.5 | 73.5 | 38.5 | 42.5 | 31.0 | 86.5 | 36.5 | 56.5 | 78.5 | 43.0 | 44.5 | 46.5 | 34.5 | 41.5 | 58.5 |
| VideoGPT+ (Maaz et al., ) | 58.7 | 69.0 | 60.0 | 83.0 | 48.5 | 66.5 | 85.5 | 75.5 | 36.0 | 44.0 | 34.0 | 89.5 | 39.5 | 71.0 | 90.5 | 45.0 | 53.0 | 50.0 | 29.5 | 44.0 | 60.0 |
| Grounded-VideoLLM | 59.4 | 76.0 | 75.5 | 77.0 | 48.0 | 67.5 | 85.5 | 77.0 | 34.5 | 39.5 | 59.5 | 86.5 | 44.5 | 60.5 | 79.0 | 51.5 | 49.0 | 46.0 | 35.0 | 42.5 | 54.0 |

NExT-GQA (Xiao et al., ) is quite challenging since it requires the model to not only correctly answer questions but also provide timestamps that support the answers, highlighting the temporal reasoning capability. According to Table 3, Grounded-VideoLLM achieves the highest Acc@GQA (26.7, +2.4) and delivers comparable IoU and IoP scores to models such as SeViLA (Yu et al., ) and LLoVi (Zhang et al., ), which use specialized grounding modules or rely on proprietary large language models (Achiam et al., ). The highest Acc@GQA score further demonstrates Grounded-VideoLLM’s capability in both fine-grained temporal grounding and high-level reasoning.

Open-Ended VideoQA.
As shown in Table 4, Grounded-VideoLLM achieves state-of-the-art or comparative performance across MSVD-QA, MSRVTT-QA (Xu et al., ), and ActivityNet-QA (Yu et al., ), highlighting its advancements in general video question answering.

General Video-LLM Benchmarks.
While Grounded-VideoLLM excels in fine-grained temporal grounding, we aim to ensure that it maintains robust performance in general video understanding. Therefore, we conducted a comprehensive evaluation using VCG-Bench (Maaz et al., ) and MVBench (Li et al., ). As shown in Table 4, Grounded-VideoLLM achieves promising results in VCG-Bench, with an average score of 3.24, outperforming other Video-LLMs with temporal grounding capabilities (e.g., LITA, VTimeLLM). Notably, Grounded-VideoLLM surpasses all other Video-LLMs on the TU (Temporal Understanding) task, with a score of 3.12 (+7%), demonstrating its superior temporal understanding, which can be attributed to the two-stream architecture that can capture motion dynamics. For MVBench which provides 4,000 QA pairs spanning a wide range of scenes categorized into 20 fine-grained tasks, the results, presented in Table 5, show that Grounded-VideoLLM achieves an average score of 59.4, surpassing other Video-LLMs. Notably, it achieves top performance in several critical tasks requiring perceiving and reasoning over specific video moments, including Action Sequence (AS), Action Prediction (AP), Action Localization (AL), and State Change (SC), with scores of 76.0 (+10%), 75.0 (+26%), 59.5 (+43%), and 51.5 (+14%), respectively, demonstrating significant advancements on fine-grained video understanding.

### 6.3 In-Depth Analysis

Figure: Figure 4: We visualize the attention weights of the LLM when generating the temporal tokens.
Refer to caption: x4.png

Two-stream Encoding. We conduct ablations to our two-stream encoding. Specifically, we set two variants by removing the temporal stream while only retaining the spatial stream, where all frame embeddings are concatenated as the video representation: (1) w/o temporal-stream (dense) feeds $T=96$ frames with a pooling size $\sigma_{S}=4$ (36 tokens per frame), resulting in a total of $36\times 96=3456$ tokens. (2) w/o temporal-stream (sparse) feeds $T=24$ frames with a pooling size $\sigma_{S}=2$ (144 tokens per frame), also resulting in a total of $144\times 24=3456$ tokens. Both variants have a close number of tokens for video representation compared to our two-stream encoding ($12\times(144+\frac{96}{12}\times 16)=3264$ tokens for $\textbf{F}_{Vid}\in\mathbb{R}^{K\cdot(N_{S}+\frac{T}{K}\cdot N_{T})\times D}$). Table 6 shows that both variants result in a significant performance drop on temporal grounding tasks. Interestingly, the dense frame variant performs slightly better than the sparse frame variant, suggesting that dense temporal information is more critical for grounding tasks than spatial details. Our two-stream architecture strikes a balance by effectively capturing dense motion dynamics while maintaining essential appearance details.

**Table 6: Impact of two-stream encoding and alignment stage.**
| Model | C-STA | ANet-G | ANet-Cap |  |
| --- | --- | --- | --- | --- |
| mIoU | mIoU | SODA_c | METEOR |  |
| Grounded-VideoLLM | 36.8 | 36.1 | 6.0 | 6.8 |
| w/o temporal-stream (sparse) | 30.4 ($\downarrow$ 6.4) | 28.0 ($\downarrow$ 8.1) | 4.9 ($\downarrow$ 1.1) | 5.5 ($\downarrow$ 1.3) |
| w/o temporal-stream (dense) | 34.3 ($\downarrow$ 2.5) | 29.2 ($\downarrow$ 6.9) | 5.4 ($\downarrow$ 0.6) | 6.2 ($\downarrow$ 0.6) |
| w/o temporal token alignment | 27.5 ($\downarrow$ 9.3) | 23.1 ($\downarrow$ 13.0) | 4.7 ($\downarrow$ 1.3) | 6.4 ($\downarrow$ 0.4) |

Temporal Token Alignment. We further investigate the role of temporal tokens by ablation of the second training stage, Temporal Token Alignment. Quantitative results in Table 6 reveal a sharp performance drop across all tasks, particularly in temporal sentence grounding tasks. Upon analyzing the outputs, we found that the model often produces time intervals spanning nearly the entire video (e.g., from <0> to <300>), neglecting the association between specific moments and temporal tokens, which leads to misalignment. Qualitatively, we visualize the attention weights of the last layer in the LLM to demonstrate how temporal tokens attend to corresponding video moments. Details of the visualizations are provided in Appendix A.4. As shown in Figure 4 (a), when generating the temporal token, e.g. <241> or <271>, the attention weights are higher and more focused on their corresponding video moments. Conversely, in Figure 4 (b), when the model is trained without the Temporal Token Alignment stage, the attention weights of temporal tokens become significantly dispersed across irrelevant moments. This illustrates that our multi-stage training strategy is essential for achieving proper alignment between temporal tokens and the video timeline. We also visualize the embedding distribution of temporal tokens in Appendix A.5.

**Table 7: Impact of grounded VideoQA dataset.**
| Model | NExT-GQA |  |  |
| --- | --- | --- | --- |
| Acc@GQA | mIoP | mIoU |  |
| Grounded-VideoLLM | 26.7 | 34.5 | 21.1 |
| w/o grounded VideoQA | 18.1 ($\downarrow$ 8.6) | 22.2 ($\downarrow$ 12.3) | 12.9 ($\downarrow$ 8.2) |

Grounded VideoQA Dataset.
We validate the role of our constructed grounded-VideoQA dataset by removing it from the training sets of stage-3. Since the model without training on our dataset usually generates free-form texts when asked to output the timestamps supporting the answer, we reformulate it as a temporal sentence grounding task, where we combine the predicted answer and question into a single sentence and ask the model to localize its timestamps. Table 7 suggests that there is a significant performance decrease with regard to Acc@GQA ($\downarrow$ 8.6), mIoP ($\downarrow$ 12.3), and mIoU ($\downarrow$ 8.2), from which we can conclude that our Grounded VideoQA dataset is essential to further enhance the model’s temporal reasoning capability.

## 7 Conclusion

We present Grounded-VideoLLM, a Video-LLM capable of fine-grained perception and reasoning over specific video moments. This is achieved through a novel model architecture that incorporates two-stream encoding for effective temporal modeling, along with the temporal tokens for efficient timestamp representation. We employ a multi-stage training scheme, starting with an image-based MLLM and progressively equipping it with fine-grained temporal grounding capabilities. Additionally, we curate a grounded-VideoQA dataset to further enhance the model’s temporal reasoning ability. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in video temporal grounding tasks but also performs strongly on general video understanding benchmarks.

## Appendix A Appendix

### A.1 More Implementation Details

Phi3.5-Vison-Instruct (Microsoft, ) consists of a CLIP style ViT image encoder (Radford et al., ), an MLP projector $f(\cdot)$, and the large language model Phi3.5-mini-3.8B (Abdin et al., ). Each video is sampled as a sequence of $T=96$ frames, which are evenly divided into $K=12$ segments. For the spatial stream encoded by the ViT in Phi3.5-V, we adopt a higher resolution 336$\times$336, but a lower resolution 224$\times$224 for the temporal stream encoded by InternVideo-2.

We set the pooling size $\sigma_{S}$ to be 2 while $\sigma_{T}$ to be 4 respectively. For the spatial stream, each frame takes up $24\times 24=576$ tokens before while $12\times 12=144$ tokens after pooling. For the temporal stream, each frame takes up $16\times 16=256$ tokens before while $4\times 4=16$ tokens after pooling. Therefore, we have an overall of $K\times(144+\frac{T}{K}\times 16)=3264$ tokens in total.

During training, we use the AdamW optimizer with a cosine learning rate decay and set the learning rate as 2e-5 and 1e-3 for projector $f(\cdot)$ and $g(\cdot)$ in stage-1. During stage-2 and stage-3, we set the learning rate for both projectors and word embeddings as 2e-5, while 2e-4 for LoRA parameters ($r$ = 128 and $\alpha$ = 256). All experiments are conducted on 8 NVIDIA A100/H800 GPUs.

### A.2 Instructions for Each Task

The quality and diversity of instructions are essential in the training process. We manually write well-designed instructions for some tasks, combined with some templates in Time-IT (Ren et al., ). Table 9 lists the prompts for different tasks.

### A.3 Evaluation Process

For the evaluation of the temporal sentence grounding task, we directly input the prompt [”At which time interval in the video can we see $<query>$ occurring?”] in Table 9 to get the response ["From $<start>$ to $<end>$"], and then calculate the predicted timestamps with the Equation (3) to get the IoU metrics.

For the evaluation of the dense video captioning task, we directly input the prompt [”Detect and report the start and end timestamps of activity events in the video, along with descriptions.”] in Table 9 to get the response ["From $<start_{1}>$ to $<end_{1}>$, $<caption_{1}>$. From $<start_{2}>$ to $<end_{2}>$, $<caption_{2}>$. $\cdots$"], and then calculate the SODA_c (Fujita et al., ) and Meteor scores (Banerjee & Lavie, ).

For the evaluation of the visually-grounded VideoQA task, we adopt a two-round conversation evaluation as follows:

In the first round, we input the question and options into the model and get the answer. In the second round, we input the question, options, and predicted answer as the contexts, together with the prompt ["Provide the timestamps that correspond to your answer."], into the model to get the predicted timestamps. With both the predicted answers and timestamps, we can calculate the metrics including IoU, IoP, and Acc@GQA (Xiao et al., ).

For the evaluation of the Open-ended VideoQA and VCG-Bench, we employed GPT-3.5 turbo to juxtapose model outputs with ground truth data as introduced by Video-ChatGPT (Maaz et al., ), subsequently computing both accuracy and a score. To ensure a fair and consistent comparison with the results presented in Video-ChatGPT, we adopted the same prompt for our evaluations. For MVBench, we directly compute the accuracy of multiple-choice questions.

### A.4 Visualization Process

We visualize the attention weights from the last layer of the LLM during the generation of a new temporal token. Since the full video representation consists of a total of $K\times(N_{S}+\frac{T}{K}\times N_{T})$ tokens—where $T$, $K$, $N_{S}$, and $N_{T}$ denote the number of frames, number of segments, number of tokens per frame for the spatial stream, and number of tokens per frame for the temporal stream, respectively—we obtain an attention weight vector with the shape $[K\times(N_{S}+\frac{T}{K}\times N_{T}),1]$. First, we discard the spatial stream portion, focusing only on the temporal information, which results in a new vector with the shape $[K\times\frac{T}{K}\times N_{T},1]$. We then reshape this vector to the form $[T,N_{T},1]$ and average it along the spatial dimension, yielding $[T,1]$, which represents the final attention weights corresponding to each frame when generating a new token.

### A.5 Distribution of Temporal Tokens

We visualize the embeddings of the $M=300$ temporal tokens to investigate their distribution in embedding space. We employ PCA (Abdi & Williams, ) to reduce the dimensionality of the temporal tokens to 1D, 2D, and 3D representations. For all reductions, we use the reduced values as coordinates, incorporating a gradient color scheme in which the color of the data points changes progressively with the token index, as illustrated in Figure 5. Our observations reveal that temporal tokens with similar indices tend to cluster together, exhibiting a continuous transition from tokens with smaller indices (light colors) to those with larger indices (darker colors).

Figure: Figure 5: Visualization of temporal tokens with PCA.
Refer to caption: x5.png

Table: Table 8: Prompts used to generate visually-grounded VideoQA samples with GPT-4.

Table: Table 9: Prompts used for different tasks.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you discuss the computational environment and time cost of finetuing this large models?
soundness: 3
strengths: 1. They introduce Grounded-VideoLLM, a Video-LLM adept at perceiving and reasoning over specific video moments.
2.They propose a step-by-step training strategy that progressively adapts an image-based MLLM into a robust Video-LLM, while curating a grounded VideoQA dataset to further enhance the temporal reasoning capability.
summary: This paper introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments
in a fine-grained manner. To solve the limitations of video LLM for fine-grained video understanding since they lack effective temporal modeling and timestamp representation, they propose  an additional temporal stream to encode the relationships between
frames and  discrete temporal tokens enriched with specific time knowledge to represent timestamps.To optimize the training of Grounded-VideoLLM, they employ a multi-stage training scheme, beginning with simple video-captioning tasks
and progressively introducing video temporal grounding tasks of increasing complexity.A grounded VideoQA dataset by an automatic annotation pipeline is build.
weaknesses: 1.The table 2 shows the zero shot result. However, it seems bring ths knowledge about ActivityNet-Captions datasets in training stages. It should be explained detailed.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1) In light of the gravitation towards large language models (LLMs) and their multimodal variants (MLLMs), the proposed approach addresses two really important research questions about the effectiveness of such models in understanding temporal information in the visual domain as well as generating more interpretable responses with regards to multimodal queries for videos. 

2) In terms of clarity, the paper is well-written and motivated. The paper is well-organized, with clear explanations of the two-stream architecture, temporal tokens, and multi-stage training strategy. The model figures are informative and especially helpful in helping the reader to understand the different stages of the data curation process as well as the intuition behind each stage.
summary: The paper introduces Grounded-VideoLLM, a Video Large Language Model (Video-LLM) that is focused on fine-grained temporal grounding in videos. The proposed Grounded-VideoLLM approach utilizes a dual-stream architecture, where one stream encodes spatial features that represent the appearance and the other temporal features for motion. Furthermore, the authors propose to use temporal tokens to represent timestamps in the latent language space. Specifically, the model incorporates discrete temporal tokens, which allows for unified processing of temporal information and text within the model. The paper also leverages a progressive training scheme, which begins with simpler video-caption alignment and eventually advancing to complex temporal grounding tasks. Empirically, the authors demonstrate the effectiveness of their Grounded-VideoLLM approach, where it outperforms state-of-the-art approaches on multiple tasks including temporal sentence grounding, dense video captioning, and grounded video question answering.
weaknesses: 1) The proposed approach is relatively lacking in novelty, given its stated contributions. To begin, the benefits of the two-stream encoding mechanism used in the Grounded-VideoLLM approach have been well-documents by prior works such as the SlowFast and Two-stream Convolutional Networks. Additionally, the introduction of unified temporal tokens to convert continuous timestamp information into the latent space of word token representations in LLMs mirrors that of prior work [1, 2]. At  a high-level, the proposed approach appears to be simply combining different aspects of prior video understanding models. One possibility is to  provide a more detailed comparison with the above-mentioned work(SlowFast, Two-stream Convolutional Networks, LITA, and Vid2Seq) to help clarify the paper's novelty.

[1] LITA: Language Instructed Temporal-Localization Assistant, ECCV 2024
[2] Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, CVPR 2023

2) Given point (1) and the demonstrated performance gain of Grounded-VideoLLM over LITA and VTimeLLM in Table 4 on video question answering, it is not clear from the experiments if the gains are due to differences in the base language models used in the different approaches. It would be very helpful to include this ablation in the empirical analysis. For example, you can compare the proposed approach using the same base language model as LITA or VTimeLLM.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	It is better to provide the evaluation on other challenging video benchmarks like **long video datasets**, e.g., EgoSchema [3], MovieChat [4] 

[3] Egoschema: A diagnostic benchmark for very long-form video language understanding. 

[4] Moviechat: From dense token to sparse memory for long video understanding.
soundness: 3
strengths: 1. The paper is well written and easy to follow.
2. Grounded-VideoLLM performs well on both general Open-Ended VideoQA benchmarks and temporal grounding benchmark.
summary: This paper introduces Grounded-VideoLLM, an advanced Video Large Language Model designed to enhance fine-grained temporal grounding. The authors identify limitations in existing Video-LLMs related to temporal modeling and timestamp representation. They propose a solution by integrating a temporal stream for frame relationship encoding and discrete temporal tokens for timestamp representation. Their multi-stage training approach, along with the creation of a grounded VideoQA dataset, demonstrates that Grounded-VideoLLM significantly improves performance in fine-grained tasks, showcasing its potential as a versatile video assistant.
weaknesses: The main weekness of this paper is the lack of more fair comparison of both the design of two-stream encoder and temporal token with previous works to demonstrate its effectiveness.
1. **The effectiveness of two-stream encoder**: Does the two-stream design necessary? Temporal modeling is important for video time grounding tasks. However, ablations in Table 6 only consider to remove the temporal stream, what if removing the spatial stream? The experiments in Table 6 should consider two more settings: 

(i) replacing the image encoder in baseline (1) and (2) with video encoders to extract features.

(ii) only use temporal stream, for key frame with 144 tokens per frame, while non-key frame with 8 tokens.

What is the advantages of the proposed two-stream design over the above two time modeling settings?

2. Table 6 only shows the performance on time grounding related benchmark, does the two-stream encoder also perform better on general Open-Ended VideoQA benchmarks?
3. **Comparison of different timestamp representations**. 1) Lack fair comparison between the proposed temporal tokens and **the plain texts** used in previous works. Actually, the choice of timestamp representation in video is similar to the bounding box representation in spatial grounding tasks, but in Table 2 of [1] has demonstrated that numerical representation seems to perform better than adding vocabulary. So I wonder does it have different conclusion on timestamp representation. 2) **Compare with other temporal modeling method**, e.g., Temporal Perception Module in Momentor [2].

[1] Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic

[2] Momentor: Advancing video large language model with fine-grained temporal reasoning.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Below is the summary of the review. 
Please consider the weaknesses and the summary below for rebuttal.

**Summary**

Overall, the proposed architectures and training scheme seem to be an effective way to enhance a VideoLLMs’ fine-grained understanding ability, validated on multiple benchmarks. 

However, the technical contribution seems limited, since the idea of the temporal stream is already well-known in the field of video understanding, and [1, 2] has already proposed an idea identical to the unified temporal tokens.

Also, the authors should better discuss the additional inference cost induced by adopting a temporal stream.

**References**

1. Qian et al., Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning, ICML 2024
2. Yang et al., Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning, CVPR 2023
soundness: 3
strengths: 1. Proposed architectures (two-stream encoding, unified temporal tokens) and training scheme (three-stage training including GPT-generated dataset) seem natural
2. Performance of Grounded-VideoLLM is validated on multiple temporal video understanding tasks, showing promising results compared to previous methods. Notably, Grounded-VideoLLM also shows satisfactory performances on general video understanding tasks, but not only on fine-grained tasks.
3. The paper is well-written and easy to follow, allowing one to understand key points
summary: In this paper, the authors proposed Grounded-VideoLLM, which is a VideoLLM with superior capability in fine-grained video understanding which previous VideoLLMs lack.
To do so, a two-stream encoding network equipped with a conventional spatial encoder and a newly adopted temporal encoder is adopted.
Also, unified temporal tokens are used, which are tokens dedicated to timestamp representation, making it easier to represent timestep-related information.
With proposed methods and collected training dataset, including ones automatically created with the aid of GPT, Grounded-VideoLLM shows satisfactory performance on multiple fine-grained video understanding tasks, while also showing satisfactory performances on conventional coarse-grained tasks.
weaknesses: 1. For me, a temporal stream seems like a process of simply sampling more frames and pooling them in a way to preserve the temporal axis. Since the idea of pooling (or aggregating) information in a temporal axis for better temporal modeling is widely adopted, the novelty seems to be limited, although this work might be the first work to adopt the idea in VideoLLMs.
2. The idea of adopting temporal tokens dedicated to timestamp representation has already been proposed in previous works [1, 2]. Unified temporal tokens proposed in this work seem to have no significant difference from those works.
3. By adopting a temporal stream, a large number of tokens are additionally fed into LLMs, which may significantly increase the computation cost considering the decoder-only LLMs have time complexity quadratic to the number of tokens. A comparison of the number of tokens being fed into LLMs and the overall model throughput with SOTA VideoLLMs may be required.
4. Grounded-VideoLLM is built upon Phi3.5-Vision-Instruct, which is a relatively recent baseline. Since recent MLLMs may show better performance compared to MLLMs that baseline VideoLLMs have been built upon, performances of those VideoLLMs might not be comparable. Therefore, implementing the idea of Grounded-VideoLLM on MLLMs identical (or similar) to those of previous VideoLLMs might better show the effectiveness of the proposed methods.