TRACE: Temporal Grounding Video LLM via Causal Event Modeling

## Abstract

Abstract Video Temporal Grounding (VTG) is a crucial capability for video understanding models and plays a vital role in downstream tasks such as video browsing and editing.
To effectively handle various tasks simultaneously and enable zero-shot prediction, there is a growing trend in employing video LLMs for VTG tasks. However, current video LLM-based methods rely exclusively on natural language generation, lacking the ability to model the clear structure inherent in videos, which restricts their effectiveness in tackling VTG tasks. To address this issue, this paper first formally introduces causal event modeling framework, which represents videos as sequences of events, and predict the current event using previous events, video inputs, and textural instructions. Each event consists of three components: timestamps, salient scores, and textual captions. We then propose a novel task-interleaved video LLM called TRACE to effectively implement the causal event modeling framework in practice.
The TRACE processes visual frames, timestamps, salient scores, and text as distinct tasks, employing various encoders and decoding heads for each. Task tokens are arranged in an interleaved sequence according to the causal event modeling framework’s formulation.
Extensive experiments on various VTG tasks and datasets demonstrate the superior performance of TRACE compared to state-of-the-art video LLMs. Our model and code are available at https://github.com/gyxxyg/TRACE .

## 1 Introduction

Video Temporal Grounding (VTG) is an important ability for video understanding models (Lin et al., ), and has becoming the base of a series of downstream tasks like moment retrieval (Fabian Caba Heilbron & Niebles, ; Gao et al., ; Oncescu et al., ), dense video caption (Zhou et al., ; Tang et al., ), video highlight detection (; Liu et al., ), and video summarization (Song et al., ; Gygli et al., ).
While non-generative models excel in moment retrieval and video highlight detection (Lei et al., ; Han et al., ; Wang et al., ), they are inflexible, task-specific, and demand substantial fine-tuning for optimal performance.
To tackle these challenges, recent research employs video LLMs as versatile models, integrating timestamp information into visual inputs, and fine-tuning them on VTG tasks (Ren et al., ; Huang et al., ; Wang et al., ; Qian et al., ) to enhance their performance and facilitate zero-shot prediction.

##### Challenges posed by videos’ inherent structures.

Despite reflecting human intent, current video LLM-based approaches rely on pure natural language generation, which, as illustrated in Figure 1(a), lacks a clear structure and indiscriminately blends information like timestamps and text captions.
In contrast, videos have an inherent structure, consisting of sequential events over time, each with distinct timestamps, captions, and salient scores (for tasks like video highlight detection and video summarization).
Consequently, the gap between videos and current video LLMs undermines the ability of video LLMs to effectively model video events, potentially making video LLMs difficult to achieve satisfactory results (Figure 1(b)) on VTG tasks.

##### Causal event modeling as a solution.

In this paper, our primary goal is to develop a novel video LLM approach for resolving the mismatch between language modeling of LLMs and videos’ inherent structure.
Specifically, we concentrate on tackling two main challenges: (1) developing a theoretical framework that shifts from causal language modeling to structured video modeling, and (2) constructing a practical video LLM based on the theoretical framework to provide an effective solution.
To accomplish this, we first introduce the causal event modeling framework, where videos are represented as sequences of events, each containing timestamps, salient scores, and textual captions. The next events are predicted based on video inputs, text instructions, and preceding events.
To effectively implement the causal event modeling framework in practice, we present a novel task-interleaved video LLM, TempoRAl grounding via Causal Event modeling (TRACE), as illustrated in Figure 2.
The TRACE treats visual frames, timestamps, salient scores, and text as separate tasks, utilizing diverse encoders and decoding heads for each task, with task tokens sequenced in an interleaved manner.
Furthermore, we develop an adaptive head-switching method for improved generation. Our numerical results across various VTG tasks reveal the superior performance of TRACE in comparison to state-of-the-art (SOTA) video LLMs.

Our key contributions are summarized as follows:

- •
We model the videos by a series of events, and propose causal event modeling framework to capture videos’ inherent structure. We then present a novel task-interleaved video LLM model, TRACE, tailored to implement the causal event modeling framework through the sequential encoding/decoding of timestamps, salient scores, and textual captions.
- •
We conduct comprehensive experiments on multiple VTG tasks and datasets to verify the effectiveness of TRACE. The results reveal significant improvements of TRACE in comparison to SOTA video LLMs. Notably, TRACE improves zero-shot performance by 3.1 and 4.9% on Youcook2 (CIDEr and F1 Score), by 6.5% and 3.7% in Recall (IOU = {0.5, 0.7}) on Charades-STA, and by 10.3% and 9.2% for mAP and HIT@1 on QVHighlights. Moreover, surpassing existing video LLMs, TRACE achieves comparable performance to traditional non-generative and task-specific methods after fine-tuning, highlighting the potential of video LLMs to excel in VTG tasks.

Figure: (a) Video Structure
Refer to caption: x1.png

Figure: Figure 2: Overview of the training process of TRACE model. We employ a variety of encoders and heads to handle time, score, and text inputs and outputs. The timestamps of the sampled frames are converted into time tokens and subsequently integrated into the visual tokens of each frame. In the answer section, time tokens, score tokens, and text tokens are inserted in a sequential manner. The generation process of TRACE is summarized in Figure 4.
Refer to caption: x3.png

## 2 Related Works

##### Video temporal grounding.

Video Temporal Grounding (VTG) tasks aim to precisely identify the timestamps of events within a given video (Lin et al., ). This includes various tasks such as moment retrieval (Gao et al., ; Zala et al., ; Oncescu et al., ; Hendricks et al., ; Boris et al., ), dense video caption (Zellers et al., ; Zala et al., ; Tang et al., ; Fabian Caba Heilbron & Niebles, ; Kim et al., ), video summarization (Song et al., ; Gygli et al., ; Hua et al., ), and video highlight detection (Lei et al., ; Xiao et al., ).
For tasks such as moment retrieval, video summarization, and video highlight detection, traditional approaches primarily use large-scale video-text pre-training (Xu et al., ; Wang et al., ; Yan et al., ; Li et al., ; Chen et al., ; Tong et al., ; Zhao et al., ). Subsequently, they fine-tune the pretrained models by incorporating task-specific prediction heads.
While these methods have demonstrated satisfactory results, they are resource-intensive for pre-training, lack zero-shot capabilities, can only handle one specific task per model, and often require additional fine-tuning for numerous downstream tasks.
For the dense video caption task, Vid2Seq employs special time tokens to represent timestamps (Yang et al., ). Some approaches integrate additional input information, such as text queries from training datasets (Kim et al., ), while other models utilize different decoding heads to decode timestamps and textual captions (Wang et al., , ) in parallel. However, these architectures are specifically designed for the dense video caption task, cannot be easily adapted to LLM structures to fully harness the capacity of pretrained LLMs, and also lack zero-shot capabilities.

##### Video LLMs for video temporal grounding.

Large language models (LLMs) (Kaplan et al., ; Achiam et al., ; Touvron et al., ) have demonstrated significant potential in acquiring knowledge and addressing real-world challenges using a zero-shot approach. Recent research has focused on integrating knowledge from other modalities, such as vision (Liu et al., ; Li et al., ) and audio (Ghosal et al., ), to bolster the capabilities of LLMs. Within the visual domain, video large language models (video LLMs) have emerged as a crucial research area (Lin et al., ; Maaz et al., ; Zhu et al., ).
Traditional video LLMs (Zhang et al., ; Lin et al., ; Li et al., , ; Cheng et al., ; Yao et al., ) have made considerable performance improvements in tasks such as video question answering, reasoning, and video captioning. However, these methods encounter difficulties in precisely pinpointing event timestamps within videos. To address this issue, TimeChat (Ren et al., ), VTimeLLM (Huang et al., ), and Hawkeye (Wang et al., ) have attempted to overcome this limitation by fine-tuning the video LLMs on VTG datasets.
More recently, LITA (Huang et al., ) introduces fast-slow visual tokens and incorporates time tokens into LLM tokenizers. Momentor (Qian et al., ) suggests a time encoder to address time token quantization errors.
VTG-LLM (Guo et al., ) integrates special time tokens and time position embeddings to improve the ability of video LLMs in comprehending timestamps.
However, these methods do not take into account the inherent structure of videos and still cannot achieve satisfactory performance. In this paper, we propose the causal event modeling framework to model video structure and design the TRACE model to address the proposed framework. Numerical results demonstrate significant performance gains of TRACE over existing video LLMs on VTG tasks.

## 3 TRACE

In this section, we aim to develop a novel video LLM that aligns well with video structures, addressing two questions: (1) how to model video structures, and (2) how to implement theoretical models. We start by proposing *causal event modeling* framework to tackle "how to model". Then, we introduce TRACE to address "how to implement".

### 3.1 Modeling the Inherent Structures of Videos

##### Formulating videos by events.

Given a video $\mathbf{V}$, we represent the video $\mathbf{V}$ as a series of events $\{e_{1},e_{2},\cdots,e_{K}\}$, with each event $e_{k}=(t_{k},s_{k},c_{k})$ encompassing timestamps $t_{k}$, salient scores $s_{k}$, and textual captions $c_{k}$. In summary, we have

$$ $\displaystyle\mathbf{V}=\{e_{1},e_{2},\cdots,e_{K}\}=\{(t_{k},s_{k},c_{k})|1 \leq k\leq K\}\,.$ (1) $$

##### Causal event modeling framework.

To effectively utilize the knowledge of pretrained LLMs, the design of causal event modeling shares the underlying intuition of causal language modeling, as formulated in the subsequent equation (^1^11Theoretically, the order of time, score, and text will not impact the results. We select one order here.).

$$ $\displaystyle\mathcal{P}(e_{k}|e_{1:k-1},\mathbf{I},\mathbf{F})$ $\displaystyle=\mathcal{P}(t_{k},s_{k},c_{k}|e_{1:k-1},\mathbf{I},\mathbf{F})\,,$ $\displaystyle=\mathcal{P}(t_{k}|e_{1:k-1},\mathbf{I},\mathbf{F})\mathcal{P}(s_ {k}|t_{k},e_{1:k-1},\mathbf{I},\mathbf{F})\mathcal{P}(c_{k}|s_{k},t_{k},e_{1:k -1},\mathbf{I},\mathbf{F})\,,$ (2) $$

where $\mathbf{I}$ indicates the textural instructions, and $\mathbf{F}$ indicates the visual frame inputs. The next event $e_{k}$ is determined by textural instructions, visual inputs, and previous events.
We can find that causal event modeling framework aligns well with the video structure (Figure 1(a)):
(1) timestamps, salient scores, and textual captions are sequentially decoded within each event;
(2) events are then ordered by timestamps.

### 3.2 TRACE: Task-Interleaved Temporal Grounding Video LLM

In Eq. 2, we introduce a formal causal event modeling framework to tackle the challenge of modeling video structures. This section illustrates the design of TRACE to implement the causal event modeling framework (Figure 2).

##### Overview of TRACE.

As illustrated in Eq. 2, the causal event modeling framework necessitates encoding/decoding of visual frames ($\mathbf{F}$), text ($\mathbf{I}$ and $c_{k}$), timestamps ($t_{k}$), and scores ($s_{k}$). Consequently, the TRACE considers these elements as distinct tasks and employs the following design to efficiently manage them.

- •
Separated multi-task processing. TRACE utilizes separate encoders and decoding heads for each task to convert task inputs into task tokens and decode task tokens back to outputs (Sec. 3.2.1).
- •
Task-interleaved sequence modeling. Task tokens are sequenced in an interleaved manner according to Eq. 2 in TRACE and fed into LLM backbones (Sec. 3.2.2).
- •
Adaptive head-switching mechanism for generation. During generation, we implement an adaptive head-switching mechanism to select the appropriate decoding head for producing the next token (Sec. 3.2.3).

#### 3.2.1 Separated Multi-Task Processing

TRACE consists of four unique tasks: visual frames, text, timestamps, and scores. Regarding text, we directly utilize the text tokenizer and LLM head of the LLM backbone (Mistral-7B-v0.2 (Jiang et al., )).
Moreover, we added a special token $\langle sync\rangle$ for indicating the end of text tasks.
The processing for the other tasks is detailed below.

##### Timestamps and scores processing.

For processing timestamps and score information, we employ two separate encoders and decoding heads, both of which share the same architecture. Specifically, each encoder is initialized with a tokenizer containing 13 tokens: 11 number tokens $\langle 0\rangle,\cdots,\langle 9\rangle,\langle.\rangle$ for representing timestamps/scores, $\langle sep\rangle$ to mark the end of each timestamp/score, and $\langle sync\rangle$ to signify the end of the current task. Token embeddings are initialized using LLM token embeddings.

In accordance with the research in VTG-LLM (Guo et al., ), we format each timestamp/score to the same length, comprising 4 whole-number parts, 1 dot, and 1 fractional part (^2^22Different from timestamps, scores will be encoded to 3 score tokens, including 1 whole-number parts, 1 dot, and 1 fractional part.). Subsequently, $\langle sep\rangle$ is inserted between timestamps/scores, and $\langle sync\rangle$ is appended at the end of each timestamp/score input sequence. For instance, the timestamp inputs [10.23, 125.37] will be tokenized into the following sequence: $\langle 0\rangle\langle 0\rangle\langle 1\rangle\langle 0\rangle\langle.
\rangle\langle 2\rangle\langle sep\rangle\langle 0\rangle\langle 1\rangle
\langle 2\rangle\langle 5\rangle\langle.\rangle\langle 4\rangle\langle sync\rangle$.

##### Visual frames processing.

Given a $T$-frame video, we initially encode the frames using the pretrained CLIP ViT-L (Radford et al., ), with each frame being encoded into 576 visual tokens. Subsequently, we employ Slot-Based Compression (Guo et al., ) to reduce the number of visual tokens to 8 per frame.
Moreover, to integrate temporal information into the visual inputs, we use a time encoder to encode the timestamps of each sampled frame and remove the $\langle sync\rangle$ and $\langle sep\rangle$ tokens, resulting in 6 time tokens for each frame. Finally, we concatenate the 8 visual tokens with the 6 time tokens to form the visual inputs for each frame.

Figure: Figure 3: Illustration of token sequence of TRACE. Following Eq 2, the sequence begins with visual frame tokens ($\mathbf{F}$) followed by instruction tokens ($\mathbf{I}$). Event ($e$) tokens are structured in the sequence of time tokens ($t$), score tokens $s$, and text tokens $c$, with events ordered chronologically based on their occurrence time.
Refer to caption: x4.png

#### 3.2.2 Task-interleaved sequence modeling

Utilizing the processed task tokens, we construct the sequence following Eq. 2. The token sequence order is illustrated in Figure 3.

##### Inter-event sequence order.

The sequence commences with visual frame tokens $\mathbf{F}$ followed by textual instruction tokens $\mathbf{I}$. For the events section, event tokens are sequenced according to the events’ occurrence time to align with the causal event modeling formula $\mathcal{P}(e_{k}|e_{1:k-1},\mathbf{I},\mathbf{F})$.

##### Intra-event sequence order.

For each event, in accordance with Eq. 2, tokens are arranged sequentially by time tokens ($\mathcal{P}(t_{k}|e_{1:k-1},\mathbf{I},\mathbf{F})$), score tokens ($\mathcal{P}(s_{k}|t_{k},e_{1:k-1},\mathbf{I},\mathbf{F})$), and text tokens ($\mathcal{P}(c_{k}|s_{k},t_{k},e_{1:k-1},\mathbf{I},\mathbf{F})$). Consequently, the causal event modeling framework (Eq. 2) emerges as a specialized autoaggressive model, featuring a unique sequence order that closely aligns with video structures.

#### 3.2.3 Adaptive Head-Switching Mechanism for Generation

##### Using ⟨ s ⁢ y ⁢ n ⁢ c ⟩ delimited-⟨⟩ 𝑠 𝑦 𝑛 𝑐 \langle sync\rangle ⟨ italic_s italic_y italic_n italic_c ⟩ token for adaptive head switching.

Since TRACE employs distinct decoding heads for various tasks during training, selecting the appropriate decoding head during generation based on previously decoded tokens is crucial. This selection is facilitated by the $\langle sync\rangle$ token. As illustrated in Figure 4, TRACE generates tokens in the sequence of time, score, and text tokens. Detection of the $\langle sync\rangle$ token prompts TRACE to switch decoding heads accordingly. The heads are cycled switched in the order of time head - score head - text head.

### 3.3 Training Strategy and Data Preparation

This section outlines the TRACE training process, which includes two stages.
For the stage 1, task modules such as the vision compression layer, task encoder, and task heads are trained for initialization. For the stage 2, the LLM backbone is fine-tuned while keeping the task modules tuned. Detailed settings and datasets are presented below.
Due to the page limitation, detailed annotation examples for each task, and details about data filtering and processing are provided in Appendix A.

##### Stage 1: Initialization of task modules.

In stage 1, task modules such as the vision compression layer, time encoder/head, and score encoder/head are trained while the vision encoder and LLM backbone remain fixed.
As shown in Table 1, stage 1 primarily utilizes two groups of datasets.

- •
Image and video caption datasets for initializing the visual compression layer. This group of datasets including Valley (Luo et al., ), LLaVA_Image (Liu et al., ), TextVR (Wu et al., ), and a randomly sampled subset of ShareGPT4Video (Chen et al., ) datasets.
- •
VTG datasets for task encoder/head initialization. We use VTG-IT dataset in this group.

For stage 1 training, we uniformly sample 128 frames from each video. The learning rate is set to 1e-3, and models are trained for one epoch with a batch size of 128.

##### Stage 2: Instruction tuning for enhancing VTG capacity.

In Stage 2, the LLM backbone and task modules are fine-tuned, with only the vision encoder remaining fixed.
As shown in Table 1, stage 2 primarily utilizes three groups of datasets.

- •
VTG instruction tuning datasets for enhancing VTG capacity. We use VTG-IT (Guo et al., ), ActivityNet Captions (Fabian Caba Heilbron & Niebles, ), and a subset of InternVid (Wang et al., ), resulting in a total of 635K data samples. Low-quality samples were filtered out, and the VTG-IT-VHD and VTG-IT-VS datasets were re-annotated. Additional details can be found in Appendix A.
- •
Video caption datasets for maintaining the quality of the visual compression layers. We use parts of the video data from stage 1, such as Valley (Luo et al., ), TextVR (Wu et al., ), and ShareGPT4Video (Chen et al., ) datasets. These datasets are compressed by retaining only one sample for samples with identical videos but different instructions, yielding 284K data.
- •
Video question answering datasets to enhance TRACE’s reasoning capabilities. We use VideoChatGPT (Maaz et al., ) and Next-QA (Xiao et al., ) in this part. For TRACE-uni, we have incorporated the YouTube and perception-test components from LLaVA-Video-178K (Zhang et al., ).

For each video, the content is uniformly divided into 128 clips, with one frame randomly sampled from each clip. The learning rate is set to 5e-6, and the models are trained for two epochs using a batch size of 128.

Figure: Figure 4: Generation process of TRACE. The TRACE generate tokens following the order of time tokens, score tokens, and text tokens. The decoding heads are switched when $\langle sync\rangle$ tokens are generated.
Refer to caption: x5.png

**Table 1: Datasets used for TRACE training process. "Compressed" indicates that datasets are condensed by retaining only one sample for samples with identical videos but varying instructions. The transparent text denotes the setting of TRACE-uni, which involves the inclusion of additional training data pertaining to general video understanding tasks.**
| Stage | Datasets | Quantity |
| --- | --- | --- |
| Stage 1 | Valley, LLaVA_Image, TextVR, ShareGPT4Video, VTG-IT | 1.9M |
| Stage 2 | Valley (Compressed), TextVR (Compressed), ShareGPT4Video (Compressed), VTG-IT, ActivityNet Captions, VideoChatGPT, InternVid, Next-QA, LLaVA-Video-178K | 0.9M (1.77M) |

## 4 Experiments

Detailed experimental settings and hyper-parameters can be found in Appendix B.1. Case studies can be found in Appendix B.3

### 4.1 Evaluation Datasets, Metrics, and Baseline Models.

We evaluate the model performance on three different tasks:

- •
Dense video caption.
We use Youcook2 (Zhou et al., ) and ActivityNet Captions (Fabian Caba Heilbron & Niebles, ) datasets as the evaluation datasets. The evaluation metrics include CIDEr (Vedantam et al., ), METEOR (Banerjee & Lavie, ), and SODA_c (Fujita et al., ) for assessing the quality of the captions. These metrics are averaged under different IoU thresholds $\{0.3,0.5,0.7,0.9\}$, following previous studies (Ren et al., ; Huang et al., ). Additionally, we report the F1 score to measure the model’s ability to accurately locate timestamps.
- •
Moment retrieval.
We utilize test set of Charades-STA (Gao et al., ) for the moment retrieval task and report the recall at IOU thresholds of 0.5 and 0.7. Additionally, we present the mIOU results.
- •
Video highlight detection. We employ the validation set of the QVHighlights dataset (Lei et al., ) and report the mean average precision (mAP) with IOU thresholds of 0.5 and 0.75, as well as the HIT@1, which represents the hit ratio of the highest scored clip.

For baseline models, we select Valley (Luo et al., ), VideoChat (Li et al., ), Video-ChatGPT (Maaz et al., ), and Video-LLaMA (Zhang et al., ) as examples of traditional video LLMs. For video LLMs specifically designed for VTG tasks, we choose TimeChat (Ren et al., ), VTimeLLM (Huang et al., ), Momentor (Qian et al., ), HawkEye (Wang et al., ), and VTG-LLM (Guo et al., ).

### 4.2 Performance of TRACE

##### Superior zero-shot performance of TRACE over other video LLMs.

In Table 2, we show the zero-shot performance of TRACE compare to SOTA video LLM baselines.
The results show that

- •
Suprior zero-shot performance. As shown in Table 2, TRACE significantly outperforms other video LLMs by a substantial margin across all three datasets. Notably, it achieves a 3.1 and 4.9% performance improvement on the Youcook2 dataset using the CIDEr and F1 Score metrics; a 6.5% and 3.7% performance increase in Recall with IOU = {0.5, 0.7} thresholds on the Charades-STA dataset; and a 10.3% and 9.2% performance gain for the mAP and HIT@1 metrics on the QVHighlights dataset.
- •
Better performance than task-specific models and larger LLMs. As shown in Table 2, as a generalist model capable of handling various tasks, the performance of TRACE surpasses that of task-specific models like HawkEye (Wang et al., ). Furthermore, the 7B TRACE model outperforms the VTimeLLM (13B) model (Huang et al., ), further validating the advantages of the TRACE architecture.
- •
Effectiveness of the TRACE architecture in addressing general video understanding tasks. As illustrated in Table 6 and 2, we observe that: (1) despite not being trained on extensive multi-task datasets, TRACE is still effective in managing general QA tasks; (2) by incorporating additional training data for general video understanding tasks, the TRACE-uni achieves superior performance compared to TRACE in both VTG tasks and general video understanding tasks.

**Table 2: Zero-shot performance of algorithms over various tasks. We evaluated the performance of VTG-LLM using the Youcook2, Charades-STA, and QVHighlights datasets. We highlight the best results for each block using bold font. The Valley, VideoChat-Embed, and Video-LLaMA results are elaborated from previous studies (Ren et al., ; Huang et al., ; Qian et al., ). The results with transparent text indicates unfair comparison (13B).**
| Model | Youcook2 | Charades-STA | QVHighlights |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mAP | HIT@1 |  |
| Traditional Video LLMs |  |  |  |  |  |  |  |
| Valley (7B) | 0.1 | 0.0 | 1.5 | 4.7 | 1.6 | 10.9 | 15.2 |
| VideoChat (7B) | 0.2 | 0.6 | 3.4 | 3.2 | 1.4 | 13.1 | 18.1 |
| Video-LLaMA (7B) | 0.0 | 0.0 | 0.1 | 2.7 | 1.2 | 11.3 | 15.6 |
| Temporal Grounding Video LLMs |  |  |  |  |  |  |  |
| TimeChat (7B) | 1.2 | 3.4 | 12.6 | 32.2 | 13.4 | 14.5 | 23.9 |
| VTimeLLM (7B) |  |  |  | 27.5 | 11.4 |  |  |
| VTimeLLM (13B) |  |  |  | 34.3 | 14.7 |  |  |
| Momentor (7B) |  |  |  | 26.6 | 11.6 | 7.6 |  |
| HawkEye (7B) |  |  |  | 31.4 | 14.5 |  |  |
| VTG-LLM (7B) | 1.5 | 5.0 | 17.5 | 33.8 | 15.7 | 16.5 | 33.5 |
| TRACE (7B) | 2.2 | 8.1 | 22.4 | 40.3 | 19.4 | 26.8 | 42.7 |
| TRACE-uni (7B) | 2.3 | 8.6 | 22.4 | 43.7 | 21.0 | 27.5 | 43.9 |

##### Performance of TRACE on ActivityNet Captions dataset.

In Table 4, we show the performance of TRACE on ActivityNet Captions dataset. All the reported algorithms except for Momentor (Qian et al., ) have incorporated the ActivityNet Captions dataset as part of the training data. Results show that the TRACE attains the best performance in both moment retrieval tasks and dense video caption tasks.

**Table 3: Ablation studies of TRACE. All the algorithms solely utilize VTG-IT (Guo et al., ) during fine-tuning for efficient evaluation. The "w/o causal event modeling " approach indicates the use of natural language-style inputs similar to previous studies (Guo et al., ; Ren et al., ). The "w/o independent encoder/heads" approach signifies directly adding new tokens to the LLM tokenizer instead of employing separate encoders/heads for different tasks. We highlight the best results using bold font for each block.**
| Models | Frame Number | Youcook2 | Charades-STA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ |  |  |
| Ablation Studies on Architecture |  |  |  |  |  |  |
| w/o causal event modeling | 96 | 1.4 | 4.3 | 17.2 | 29.7 | 14.0 |
| w/o independent encoder/heads | 64 | —-Fail to Follow Instruction—- |  |  |  |  |
| TRACE (VTG-IT) | 64 | 1.9 | 6.9 | 21.4 | 37.0 | 17.0 |
| Ablation Studies on Frame Number |  |  |  |  |  |  |
| TRACE (VTG-IT) | 8 | 1.4 | 5.0 | 18.6 | 28.8 | 13.6 |
| TRACE (VTG-IT) | 64 | 1.9 | 6.9 | 21.4 | 37.0 | 17.0 |
| TRACE (VTG-IT) | 128 | 2.1 | 7.5 | 21.4 | 41.2 | 20.0 |

Figure: (a) Performance on Youcook2
Refer to caption: x6.png

### 4.3 Ablation Studies of TRACE.

##### The causal event modeling framework enhances model performance in VTG tasks.

In the ’Ablation Studies on Architecture’ section of Table 3, we conducted experiments without utilizing the causal event modeling framework. The results indicate that employing the causal event modeling framework significantly improves model performance, and TRACE can achieve better results even when sampling fewer video frames.

##### Using different encoders and decoding heads for different tasks is essential for TRACE to achieve the best result.

In the "w/o independent ecoder/heads" part of Table 3, we performed ablation studies by not utilizing separate encoders and decoder heads for different tasks. Instead, we directly incorporated time tokens and score tokens into the text tokenizers. The results suggest that using shared encoder/decoding heads for causal event modeling framework significantly disrupts the prelearned knowledge of LLMs, leading to irrelevant and meaningless responses.

##### The performance of TRACE improves with the increase in the number of frames.

We conducted ablation studies on the number of sampled frames, as presented in Table 3. The results show that (1) the performance of TRACE enhances as the number of sampled frames increases; (2) the performance of TRACE is comparable or even superior to SOTA video LLMs like VTG-LLM and TimeChat when sampling just 8 frames, demonstrating the effectiveness of the TRACE model architecture.

##### Incorporating InternVid (Wang et al., 2023b ) and ActivityNet Captions (Fabian Caba Heilbron & Niebles, 2015 ) datasets boost TRACE performance on long videos.

As illustrated in Figure 5, we carried out ablation studies by exclusively using VTG-IT as the training data for VTG tasks. The results indicate that the performance of TRACE on long videos improves when incorporating internVid and ActivityNet Captions datasets, leading to enhanced performance on Youcook2, QVHighlights, and ActivityNet Captions datasets. Conversely, the performance of TRACE on short videos slightly decreases (Charades-STA), suggesting that the annotations in the internVid and ActivityNet Captions datasets may not be as accurate as those in short video annotations.

### 4.4 Fine-tuned Performance of TRACE.

##### Competitive performance of TRACE to traditional methods after fine-tuning.

In Table 5, we fine-tune the TRACE for 3 epochs on Youcook2 and Charades-STA datasets (^3^33Results on QVHighlights can be found in Appendix B.2.). The results indicate that

- •
TRACE significantly outperform generalist baselines. In contrast to TimeChat and VTG-LLM, which struggle to attain satisfactory performance even after fine-tuning, the TRACE derives significant benefits from fine-tuning and achieves notably better performance than generalist baselines. These results further substantiate that our enhancements to the model architecture are crucial for VTG tasks.
- •
TRACE achieve comparable performance to non-generative and task-specific SOTAs. As depicted in Table 5, the TRACE achieves new SOTA results on Youcook2 (without audio inputs). Furthermore, the performance of TRACE on the Charades-STA dataset is also competitive with non-generative models such as InternVideo2 and VDI. *However, these methods cannot handle various tasks simultaneously and lack zero-shot capability – the contribution of TRACE herein.*

**Table 4: Performance of TRACE on ActivityNet Captions dataset. The evaluation of TimeChat’s and VTG-LLM’s results was conducted using the official provided checkpoints. The ^∗ indicates zero-shot evaluation. We highlight the best and the second best results using bold and underline.**
| Models | Dense Video Caption | Moment Retrieval |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| METEOR | SODA_c | CIDEr | F1 Score | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mIOU |  |
| VTimeLLM | 6.8 | 5.8 | 27.6 |  | 29.5 | 14.2 | 31.4 |
| Momentor^∗ | 4.7 | 2.3 | 14.9 |  | 23.0 | 12.4 | 29.3 |
| TimeChat | 5.7 | 4.7 | 19.0 | 36.9 | 4.6 | 2.0 | 6.9 |
| VTG-LLM | 5.9 | 5.1 | 20.7 | 34.8 | 8.3 | 3.7 | 12.0 |
| TRACE | 6.4 | 6.0 | 25.9 | 39.3 | 37.7 | 24.0 | 39.0 |
| TRACE-uni | 6.9 | 6.4 | 29.2 | 40.4 | 38.2 | 24.7 | 39.4 |

**Table 5: Fine-tuned performance of TRACE. We fine-tune the TRACE for 3 epochs on the Youcook2 and Charades-STA datasets. We emphasize the best and second best results using bold font and underline. For Youcook2 dataset, we choose Vid2Seq (Yang et al., ), PDVC (Wang et al., ), and CM^2 (Kim et al., ) as task-specific baselines. The results depicted in gray indicate unfair comparisons due to additional audio inputs and different architectures. For charades-STA dataset, we choose InternVideo2-6B (Wang et al., ), VDI (Luo et al., ), and Moment-DETR (Lei et al., ) as examples of non-generative models.**
| Model | Youcook2 |  |  |
| --- | --- | --- | --- |
| SODA_c | CIDEr | F1 Score |  |
| Task-Specific Models |  |  |  |
| PDVC | 4.4 | 22.7 |  |
| Vid2Seq (Audio Input) | 7.9 | 47.1 | 27.3 |
| Vid2Seq | 5.7 | 25.3 | 23.5 |
| CM^2 | 5.3 | 31.7 | 28.4 |
| Generalist Models |  |  |  |
| TimeChat | 3.4 | 11.0 | 19.5 |
| VTG-LLM | 3.6 | 13.4 | 20.6 |
| TRACE | 6.7 | 35.5 | 31.8 |

**Table 6: Performance of TRACE on general video understanding tasks.**
| Models | MVBench (Li et al., ) | VideoMME (w/o subtitle) (Fu et al., ) |
| --- | --- | --- |
| Video-LLaVA (Lin et al., ) | 41.0 | 39.9 |
| LLaVA-NeXT-Video (Liu et al., ) | 46.5 | 33.7 |
| VideoChat2 (Li et al., ) | 51.1 | 33.7 |
| VideoLLaMA 2 (Cheng et al., ) | 54.6 | 44.6 |
| TRACE | 48.1 | 43.8 |
| TRACE-uni | 53.8 | 49.6 |

## 5 Conclusion and Future Works

In this paper, our goal is to address the mismatch between video structure and video LLMs on VTG tasks, and propose a causal event modeling framework and the TRACE model as a solution. Numerical results indicate the superior zero-shot performance of TRACE compared to other video LLM baselines, and TRACE also achieves competitive performance relative to traditional non-generative and task-specific models after fine-tuning.
By overcoming the inherent limitations of video LLM architectures, TRACE demonstrates the potential of video LLMs on VTG tasks, and we believe that the TRACE could be a strong foundation for future research on video LLMs in VTG tasks.

However, there are future works that can further enhance the capabilities of TRACE. For instance, incorporating additional training data on general tasks to improve the performance of TRACE in general video understanding tasks and expanding the VTG dataset.
Furthermore, considering that current annotations for general video understanding tasks do not include timestamp and score information, expanding the annotation of these tasks by incorporating the occurrence timestamps of QA pairs and the matching score between questions and answers could significantly improve the overall performance of TRACE.

## Appendix A Dataset Preparation

### A.1 Details of data format

In this section, we introduce the details of the data format we used while training TRACE. For the VTG datasets, in addition to ActivityNet Captions dataset and InternVid dataset, we directly use the annotation collected by VTG-IT (Guo et al., ).
In detail, the annotations can be categories into the following four types:

- •
General tasks (Figure 6). For general tasks such as video captioning, image captioning, and video question answering, the answer component of the data does not include timestamps or scores. Consequently, we employ a single token $\langle sync\rangle$ as a placeholder for timestamps and scores, signifying an empty response for this part of response. This kind of annotation including LLaVA_Image (Liu et al., ), Valley (Luo et al., ), TextVR (Wu et al., ), ShareGPT4Video (Chen et al., ), VideoChatGPT (Maaz et al., ), and Next-QA (Xiao et al., ) datasets.
- •
Dense video caption task (Figure 7). The Dense Video Captioning task solely comprises timestamps and textual captions responses. As a result, we use a single token $\langle sync\rangle$ as a placeholder for scores. The datasets of this task include HiRESTstep (Zala et al., ), COIN (Tang et al., ), ActivityNet Captions (Fabian Caba Heilbron & Niebles, ), VTG-IT-DVC (Guo et al., ), and InternVid (Wang et al., ) datasets.
- •
Moment retrieval task (Figure 8). Similiar to dense video caption task, moment retrieval solely comprises timestamps and textual captions responses. The moment retrieval task including HiRESTgrounding (Zala et al., ), QuerYD (Oncescu et al., ), DiDeMo (Hendricks et al., ), VTG-IT-MR (Guo et al., ), and InternVid (Wang et al., ) datasets.
- •
Video highlight detection task (Figure 9). For the video highlight detection task, we utilize the query as the textual response for all highlight moments. In this case, we employ the VTG-IT-VHD (Guo et al., ) dataset.
- •
Video summarization task (Figure 10). The video summarization task employs the caption of each event as the textual response. We use the VTG-IT-VS (Guo et al., ) dataset here.

Figure: Figure 6: Annotation example of video caption task.
Refer to caption: x10.png

Figure: Figure 7: Annotation example of dense video caption task.
Refer to caption: x11.png

Figure: Figure 8: Annotation example of moment retrieval task.
Refer to caption: x12.png

Figure: Figure 9: Annotation example of video highlight detection task.
Refer to caption: x13.png

Figure: Figure 10: Annotation example of video summarization task.
Refer to caption: x14.png

### A.2 Processing InternVid

##### Dense video caption data.

The dense video caption data is constructed based on the InternVid-Full annotations. For each video, we opt not to use the annotation of that video as dense video caption data if any of the following checklist items are met:

- •
There exists a caption with fewer than 5 words.
- •
There exists a caption very similar to another caption within one video. We use fuzzywuzzy package here and set the threshold to 70.
- •
The number of events is less than 5 or greater than 50.
- •
There are special characters present, excluding letters, spaces, and dots.

##### Moment retrieval data.

We discovered that directly utilizing the InternVid-Full annotations to construct the moment retrieval data leads to suboptimal performance, likely due to the imprecise timestamp annotations. Consequently, we employ the InternVid-10M-FLT-INFO annotation to construct the moment retrieval data, which is a filtered subset of the InternVid-Full annotation provided by the authors.

### A.3 Processing VTG-IT

In this section, we describe the processing of the VTG-IT dataset. For the dense video caption and moment retrieval tasks, we directly utilize the annotations provided by VTG-IT. However, the data for video highlight detection and video summarization tasks supplied by VTG-IT often have uniform salient scores within each video. As a result, our goal here is to enhance the quality of data for these two tasks, specifically video highlight detection and video summarization.

- •
Video highlight detection task. For each event in the dense video caption task, we initially divide each event into a maximum of 20 clips. Subsequently, we compute the similarity between the frames within each clip and the event captions using ViT-G/14 from EVA-CLIP (Sun et al., ). The similarity scores are then normalized to a Gaussian distribution to serve as the highlight score. In detail, the clips with scores higher than [2.275%, 3.593%, 5.480%, 8.076%, 11.507%, 15.866%, 21.186%, 27.425%, 34.458%, 42.074%, 50.000%, 57.926%, 65.542%, 72.575%, 78.814%, 84.134%, 88.493%, 91.924%, 94.520%, 96.407%, 97.725%] of other clips will be assign scores [1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.2, 3.4, 3.6, 3.8, 4.0, 4.2, 4.4, 4.6, 4.8, 5.0].
- •
Video summarization task. The data for the video summarization task is built upon the foundation of the video highlight detection task and the dense video caption task. Specifically, for each event in the dense video caption task data, we select the clip with the highest score to serve as the summarization clip for that particular event.

## Appendix B Experiments

### B.1 Detailed Experimental Settings

We report the detailed model architecture design and training hyper-parameters in Table 7. The training takes about 5 days for stage 1 and 5 days for stage 2.

**Table 7: Detailed training setting and hyper-parameters.**
| Setting | Stage 1 | Stage 2 |
| --- | --- | --- |
| Computation | 16 ATN 910B | 16 ATN 910B |
| Vision Encoder | openai/clip-vit-large-patch14-336 | openai/clip-vit-large-patch14-336 |
| DeepSpeed Stage | Zero3 Offload | Zero3 Offload |
| LLM | Mistral-7B-v0.2 | Mistral-7B-v0.2 |
| Batch Size | 128 | 128 |
| Num Frames | 128 | 128 |
| Frame Sample | Uniform | Split to 128 clips then random within clip |
| Train Epochs | 1 | 2 |
| Learning Rate | 1e-3 | 5e-6 |
| LR Scheduler | Cosine | Cosine |
| Model Max Length | 4096 | 4096 |

### B.2 Additional Experiment Results

**Table 8: Ablation studies on training data (Youcook2 and Charades-STA).**
| Models | Youcook2 | Charades-STA |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SODA_c | CIDEr | F1 Score | METEOR | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mIOU |  |
| TRACE (VTG-IT) | 2.1 | 7.5 | 21.4 | 2.6 | 41.2 | 20.0 | 38.9 |
| TRACE | 2.2 | 8.1 | 22.4 | 2.8 | 40.3 | 19.4 | 38.7 |

**Table 9: Ablation studies on training data (ActivityNet Captions).**
| Models | Dense Video Caption | Moment Retrieval |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SODA_c | CIDEr | F1 Score | METEOR | $\text{R@1}_{\text{(IOU=0.5)}}$ | $\text{R@1}_{\text{(IOU=0.7)}}$ | mIOU |  |
| TRACE (VTG-IT) | 5.8 | 24.7 | 38.9 | 6.0 | 19.2 | 9.3 | 25.0 |
| TRACE | 6.0 | 25.9 | 39.3 | 6.4 | 37.7 | 24.0 | 39.0 |

##### Ablation studies on training data.

In Tables 8 and  9, we report the performance of TRACE using different training data. The results show that (1) the performance of TRACE on long videos increase while using the original TRACE setting. (2) The performance of TRACE slightly reduced on short videos (Charades-STA).

**Table 10: Fine-tuned performance of algorithms on QVHighlights datasets. We fine-tune the Algorithm on QVHighlights datasets.**
| Model | QVHighlights |  |
| --- | --- | --- |
| mAP | HIT@1 |  |
| TimeChat | 21.7 | 37.9 |
| VTG-LLM | 24.1 | 41.3 |
| TRACE | 31.8 | 51.5 |

##### Fine-tuned performance on QVHighlights.

In Table 10, we show the performance of TRACE on QVHighlights dataset after fine-tuning. The results indicate that TRACE significantly outperform other video LLMs by a large margin.

### B.3 Case Studies

We present the case studies of TRACE in Figure 11. The results demonstrate that TRACE can accurately identify the events within the given video and is also proficient in performing traditional video captioning tasks.

Figure: Figure 11: Case study of TRACE.
Refer to caption: x15.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: No additional questions. Please see the "Weaknesses" section for areas needing clarification.

### Recommendations for Improvement:
- **Refine Prompt Design Explanation:** Providing specific strategies or insights on prompt design tailored for VTG tasks would enhance the paper's originality and usefulness for future researchers.
  
- **Explore Custom Scene Parsing Techniques:** Introducing refined parsing methods could strengthen TRACE's robustness and accuracy in multi-modal alignment.

This structured feedback should provide the authors with a comprehensive view of the strengths and areas for enhancement in their paper on TRACE.
soundness: 3
strengths: 1. The paper proposes TRACE, a framework leveraging causal event modeling to generate structured video representations through large language models (LLMs). This approach addresses structural gaps in video data, making it valuable for multi-modal research and practical applications in video analysis.
   
2. TRACE maximizes the potential of pre-trained LLMs by adopting causal event modeling, which decomposes video inputs into frames and aligns them with textual prompts. The temporal segmentation and alignment methods allow videos to be broken down into events with associated timestamps, salient scores, and captions. This granularity is crucial for precise video event parsing and presents a significant step forward in video understanding with LLMs.

3.TRACE outperforms the existing Video-LLMs on three pivotal Video Temporal Grounding (VTG) benchmarks—Charades-STA, QV Highlights, and YouCook2—underscoring its efficacy and robustness in handling video temporal grounding tasks. This achievement underscores TRACE's ability to accurately capture and model the intricate temporal dynamics across a spectrum of video datasets.
summary: This paper proposes a new method for Video Temporal Grounding (VTG) tasks, named TRACE. TRACE uses a causal event modeling framework to represent videos as a sequence of events with timestamps, salient scores, and textual descriptions. The paper designs a task-interleaved video large language model to address the limitations of traditional video LLMs in handling the inherent structure of videos. The TRACE model utilizes different encoders and decoding heads to process visual frames, timestamps, and text inputs, enabling more effective event sequencing and causal modeling. Experimental results demonstrate that TRACE achieves state-of-the-art zero-shot performance on various VTG tasks and, after fine-tuning, can match the performance of traditional non-generative, task-specific models.
weaknesses: 1.	While causal event modeling is presented as a core contribution of this work, the related work section does not address any prior research on similar methodologies. It would be helpful to clarify whether comparable approaches have been explored in the field of video understanding, or if this approach is entirely novel within this domain. Providing this context could strengthen the argument for the method’s originality and situate it more clearly within existing research.

2.	It is unclear whether compressing visual features to 8 tokens is sufficient for preserving critical information in complex video scenes. The paper does not provide an analysis or experimental results on the trade-off between the number of tokens and model performance, which would be valuable in understanding the potential impact of this compression choice.

3.	There are several grammatical and spelling errors throughout the manuscript, which impact readability and may detract from the paper’s clarity. For example: Line 22: "processes" should be corrected to "process". Line 44-45: The phrase "...which,..." should be rephrased, and "lacks" should be changed to "which lack".

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: 1. The presentation and illustration are quite clear and easy to follow.
2. The motivation of causal event modeling is quite intuitive and the design is straightforward and yet effective.
3. The zero-shot performance is superior compared to previous video LLM methods.
summary: The paper introduces a task-interleaved video LLM, TRACE, which incorporates a newly-designed causal event modeling framework for VTG task. The TRACE employs multiple encoders for different inputs, while the task tokens are arranged in an interleaved manner. TRACE demonstrates SOTA performance on various VTG datasets compared to previous video LLMs.
weaknesses: 1. While the paper compares TRACE with other video LLMs, it presents limited comparison and may not adequately address how it stands against traditional non-generative and task-specific models.
2. The extent to which TRACE can be applied to other types of video tasks beyond VTG is unclear. Its design may be highly specialized, which could limit its applicability across diverse video understanding tasks. Authors should present more results on other video-understanding tasks since the design seems generalizable by building such causal event relations.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1) Video Temporal Grounding (VTG) is a crucial task, yet current Video-LLMs underperform in this area. Techniques aimed at improving temporal grounding for these models are highly valuable to advance the field.
2) The causal event modeling framework fits well with the next-token prediction paradigm of large language models (LLMs), offering an intuitive way to model video structures in sequential tasks.
3) TRACE demonstrates consistent performance improvements over prior Video-LLMs across three key VTG benchmarks (Charades-STA, QVHighlights, and YouCook2), underscoring its effectiveness.
summary: This paper addresses the task of Video Temporal Grounding (VTG) and introduces TRACE, a task-interleaved Video-LLM designed for enhanced VTG performance. The authors highlight limitations in current Video-LLMs, which rely solely on natural language generation without considering the inherent temporal structure of videos. To address this, they propose a novel causal event modeling framework, decomposing videos into sequences of events defined by timestamps, salient scores, and textual captions. Extensive experiments on datasets such as Charades-STA, QVHighlights, and YouCook2 demonstrate the superior zero-shot performance of TRACE compared to existing Video-LLMs.
weaknesses: 1) While the motivation for TRACE is clear, the use of multiple task-specific heads may limit the model’s generalization. A primary appeal of Video-LLMs lies in their ability to handle a variety of tasks without specific fine-tuning. TRACE’s focus on VTG may narrow its versatility, making it less effective for general video understanding tasks. In most cases, lightweight VTG-specific models with stronger performance could be more suitable for VTG scenarios.
2) Some clarity is not clear. For example, the paper does not adequately explain slot-based compression, which is not a widely known technique. Moreover, compressing each frame to just 8 visual tokens might lead to significant information loss, raising concerns about the trade-off between efficiency and accuracy.
3) It is unclear whether the same set of number tokens is used for both timestamps and scores. If so, this could blend the two types of information, contradicting the authors' claim (lines 45–46) that the model preserves the distinct structure of video events.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: My main concern is with the autoregressive modeling approach, and if the authors can provide a reasonable explanation, I am willing to consider raising my score, as I believe this work could provide inspiration for future work.
soundness: 3
strengths: The author first employs a causal modeling method in the grounding of VLLM, achieving causal probability modeling through the sequence of input tokens. This approach will provide inspiration for future work on video understanding tasks using VLLM.
summary: This paper proposes a new Video Temporal Grounding (VTG) method that addresses the shortcomings of existing LLM in handling VTG tasks by modeling causal events.
weaknesses: 1. **Autoregressive modeling.**

One of my major concerns is that the authors have only used the earlier events $e_{1:k-1}$ in their modeling of causal relationships between events through autoregression, without incorporating the equally known $e_{k+1:K}$. I believe this approach may be unreasonable since it is likely that the same events may occur earlier while the current event is different due to unrelated pretexts. However, this issue can be avoided by modeling different subsequent events simultaneously. Besides, most current video understanding researchers have modeled multiple events by utilizing all contextual events that occur before and after them [1-4]. This may require the authors to provide further explanation.

[1] Liang C, et al. Visual abductive reasoning[C]. CVPR 2022.

[2] Lam T E, et al. CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes. NeurIPS 2024.

[3] Chen T, et al. MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning. NeurIPS 2024.

[4] Du Y, et al. Towards Event-oriented Long Video Understanding. ArXiv 2024.

2. **Inference speed.**

The authors have adopted a form similar to autoregression, and I would like to understand if there is a time overhead in comparing their model's inference speed to that of current mainstream LLMs.

3. **LLM backbone.**

I noticed that the authors used Mistral-7B as the LLM backbone, however, in other comparison methods, Timechat used LLaMA-2, while HawkEye, Momentor, and VTimeLLM used Vicuna. I would like to know if the authors have conducted experiments with LLaMA-2 or Vicuna as the LLM backbone, to ensure that the superior performance is not due to the better LLM backbone but rather the causal modeling.

Meta-review: The paper receives 3 positive and 1 negative ratings after rebuttal, with 3 upgraded scores. Initially, the reviewers had several concerns about some technical clarity, motivations of using timestamps and scores, more contexts with the relevant work, more analysis on model parameters, extending to other video tasks, comparisons with multimodal LLMs (e.g., VideoLLama2). In the post-rebuttal discussion period, three reviewers were satisfactory with the authors' comments and raised the rating. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback of the proposed method being effective and significant as a LLM foundation for video temporal grounding. Therefore, the AC recommends the acceptance rating.

Decision: Accept (Poster)