TRACE: Temporal Grounding Video LLM  via Causal Event Modeling
Authors: Yongxin Guo
Jingyu Liu
Mingda Li
Qingbin Liu
Xi Chen
Xiaoying Tang
Venue: ICLR 2025 Poster (2025)
Keywords: video large language model
video temporal grounding
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper aims to address the mismatch between video structure and video LLMs on video temporal grounding tasks,  and propose a causal event modeling framework and the TRACE model as a solution.

Abstract: Video Temporal Grounding (VTG) is a crucial capability for video understanding models and plays a vital role in downstream tasks such as video browsing and editing. 
To effectively handle various tasks simultaneously and enable zero-shot prediction, there is a growing trend in employing video LLMs for VTG tasks. However, current video LLM-based methods rely exclusively on natural language generation, lacking the ability to model the clear structure inherent in videos, which restricts their effectiveness in tackling VTG tasks. To address this issue, this paper first formally introduces causal event modeling framework, which represents video LLM outputs as sequences of events, and predict the current event using previous events, video inputs, and textural instructions. Each event consists of three components: timestamps, salient scores, and textual captions. We then propose a novel task-interleaved video LLM called TRACE to effectively implement the causal event modeling framework in practice. 
The TRACE process visual frames, timestamps, salient scores, and text as distinct tasks, employing various encoders and decoding heads for each. Task tokens are arranged in an interleaved sequence according to the causal event modeling framework's formulation.
Extensive experiments on various VTG tasks and datasets demonstrate the superior performance of TRACE compared to state-of-the-art video LLMs. Our model and code are avaliable at \url{https://github.com/gyxxyg/TRACE}.

Published as a conference paper at ICLR 2025
TRACE: TEMPORAL GROUNDING VIDEO LLM VIA CAUSAL EVENT MODELING
Yongxin Guo"
Jingyu Liu? Mingda Li
Qingbin Liu Xi 2hen Xiaoying T3
13,4,,
Schooo of Science and Engineering, The Chinese University of Hong Kong, Shenzhen 518172, China
Teencett PCG
Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), Shenzhen, China *Guangdong Provincial Key Laboratory of Future Networks of Intelligence, Shenzhen, China
ABSTRACT
Video Temporal Grounding (VTG) is crucial capability for video understanding models and plays a vital role in downstream tasks such as video browsing and editing. To effectively handle various tasks simultaneously and enable zero-shot prediction, there is a growing trend in employing video LLMs for VTG tasks. However, current video LLM-based methods rely exclusively on natural language generation, lacking the ability to model the clear structure inherent in videos, which restricts their effectiveness in tackling VTG tasks. To address this issue, this paper first formally introduces causal event modeling framework, which rep- resents video LLM outputs sequences of events, and predict the current using previous events, video inputs, and textural instructions. Each event consists of three components: timestamps, salient scores, and textual captions. We then propose a novel task-interleaved video LLM called TRACE to effectively imple- ment the causal event modeling framework in practice. The TRACEprocess visual frames, timestamps, salient scores, and text as distinct tasks, employing various encoders and decoding heads for each. Task tokens are arranged in an interleaved sequence according to the causal event modeling framework's 's formulation. Ex- tensive experiments on various VTG tasks and datasets demonstrate the superior performance of TRACE compared to state-of-the-art video LLMs. Our model and code are avaliable at https:/ / giihhu... com/ gwwxyg/xxxygggoooo
Seyxxyy/TRRCEE... TRACE.
INTRODUCTION
Video Temporal Grounding (VTG) is an important ability for video understanding models (Lin et al., 2023b), and has becoming the base of a series of downstream tasks like moment retrieval (Fabian Caba Heilbron & Niebles, 2015; Gao et al., 2017; Oncescu et al., 2021), dense video caption (Zhou et al., 2018; Tang et al., 2019), video highlight detection (Lei et al.; Liu et al., 2022), and video summarization (Song et al., 2015: Gygli et al., 2014). While non gon-geeertt models excel in moment retrieval and video highlight detection (Lei et al., 2021; Han et al., 2024; Wang et al., 2024a), they inflexible, task-specific, and demand substantial fine-tuning for optimal performance. To tackle these challenges, recent research employs video LLMs as versatile models, integrating timestamp information into visual inputs, and fine-tuning them on VTG tasks (Ren al., 2023; Huang al., 2023; Wang et al., 2024b; Qian et al., 2024; Wang et al., 2024c; Wu et al., 2024) to enhance their performance and facilitate zero-shot prediction.
Challenges posed by videos' inherent structures. Despite reflecting human intent, current video LLM based approaches rely on pure natural language generation. As illustrated Figure 1(a), this approach lacks a clear structure and indiscriminately blends information, such as timestamps and text captions. In contrast, videos possess an inherent structure that transcends mere textual descrip- tion. To accurately describe or reason from a video, it is insufficient to rely solely on natural lan- guage text. Instead, the corresponding timestamps and salient scores are also essential components. Together, these elements provide more comprehensive and structured understanding of the video content. Consequently, the gap between videos' structure and current video LLMs undermines the
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
Performance
TRACE
35-
diii serodkring ant was tayy deferlattin prosess ng tham watar with Futt frush acoad
sscrdds
lindch
Natural Language
Evont
Start Time 1s:
End Time 63)
Capsion
Insoouerg and summar
using poda:
3
43.0
78.0
cuees and scedin them woter Prail
Netrinaal
Video Caption
Video Hightight Detection
Video Structure
(a) Vidco Structure
(h) Performance Gap between Models
Figure Challenges posed by videos' inherent structures. Figure ((a) shows the difference between natural language and video structure, while Figure 1(b) highlights the performance gap between SOTA video LLMs (Ren al., 2023; Guo al., 2024) and TRACE. We present zero-shot performance results for video LLM approaches. Specifically, we report the performance of models using the CIDEr metric the dense video captioning task on the Youcook2 datasct, R@ lou-0, for the moment retrieval task on the Charades STA dataset, and HIT@ for the highlight detection task on the QVHighlights dataset.
ability of video LLMs effectively model video events, potentially making video LLMs difficult to achieve satisfactory results (Figure 1(b)) on VTG tasks.
Causal event modeling solution. In this paper, primary goal is develop novel video LLM approach for resolving the mismatch between language modeling of LLMs and videos' inher- structure. Specifically, we concentrate on tackling oretical framework that shifts from causal language modeling to structured event-based modeling, and (2) constructing a practical video LLM based on the thcoretical framework provide an effec- tive solution. To accomplish this, we first introduce the causal event modeling framework, where video LLM outputs are represented as sequences of events, each containing timestamps, salient scores, and textual captions. The next events are predicted based on video inputs, text instructions, and preceding events. To effectively implement the causal event modeling framework in practice, present novel task-interleaved video LLM, TempoRAl grounding via Causal Event modeling (TRACE), as illustrated Figure 2. The TRACE treats visual frames, timestamps, salient scores, and text separate tasks, utilizing diverse encoders and decoding heads for each task, with task tokens sequenced an interlcaved manner. Furthermore, we develop an adaptive head-switehing method for improved generation. Our numerical results across various VTG tasks reveal the superior performance of TRACE comparison to state-of-the-art (SOTA) video LLMs.
main challenges: (1) developing the-
Our key contributions
summarized as follows:
We model the videos by a series events, and propose causal event modeling framework to capture videos' inherent structure. We then present novel task-interleaved video LLM model, TRACE, tailored to implement the causal event modeling framework through the sequential en- coding/decoding of timestamps, salient scores, and textual captions.
We conduct comprehensive experiments on multiple VTG tasks and datasets to verify the ef- fectiveness of TRACE. The results reveal significant improvements of TRACE in comparison to SOTA video LLMs. Notably, TRACE improves zero-shot performance by 3.1 and 4.9% on Youcook2 (CIDEr and F1 Score). by 6.5% and 3.7% Recall (IOU {0.5, 0.7)) on Charades- STA, and by 10.3% and 9.2% for mAP and HIT@ I QVHighlights. Moreover, surpassing existing video LLMs, TRACE achieves comparable performance traditional non-generative and task-specific methods after fine-tuning, highlighting the potential of video LLMs to excel in VTG tasks.
RELATED WORKS
Video temporal grounding. Video Temporal Grounding (VTG) tasks aim to precisely identify the timestamps of events within given video (Lin al., 2023b). This includes various tasks such as
2

---

Published as a conference paper at ICLR 2025
Time Head
Score Head
Text Head
Large
Language Model
Compression
Time Encoder
Score Encoder
Text Tokenizer
Vision Encoder
Timestamp
Score
0.0s,
Text Inputs
locats,a cerles events video, tinestares.of evens, describe each sentences... saaances...cynne. ci 3 cyycxx the vegetables and croutons to the saladesyne>...
5.0s
10.0s
15.0s
Figure 2: Overview of the training process of TRACE model. We employ variety of encoders and heads to handle time, score, and text inputs and outputs. The timestamps of the sampled frames are converted into time tokens and subsequently integrated into the visual tokens of each frame. In the answer section, time tokens, tokens, and text tokens inserted in sequential manner. The generation process of TRACE summarized Figure 4.
moment retrieval (Gao al., 2017: Zala et al., 2023b; Oncescu et al., 2021; Hendricks et al., 2018a;
Boris et al., 2024), dense video caption (Zellers et al., 2021: Zala et al., 2023b; Tang et al., 2019;
Fabian Caba Heilbron & Niebles, 2015; Kim et al., 2024), video summarization (Song et al., 2015;
Gygli et al., 2014; Hua et al., 2024), and video highlight detection (Lei et al., 2021; Xiao et al., 2023). For tasks such as moment retrieval, video summarization, and video highlight detection, traditional approaches primarily
large-scale video-text pre- training (Xu et al., 2021; Wang et al., 2022; Yan et al., 2022; Li et al., 2023d; Chen et al., 2024h; Tong et al., 2022; Zhao et al., 2024). Subsequently. they fine-tune the pretrained models by incorporating task-specific prediction heads. While these methods have demonstrated satisfactory results, they are resource-intensive for pre-training, lack zero-shot capabilities, can only handle one specific task per model, and often require additional fine- tuning for numerous downstream tasks. For the dense video caption task, Vid2Seq employs special time tokens represent timestamps (Yang et al.. 2023). Some approaches integrate additional input information, such as text queries from training datasets (Kim et al., 2024), while other models utilize different decoding heads to decode timestamps and textual captions (Wang al., 2021; 2023a) in parallel. However, these architectures are specifically designed for the dense video caption task, cannot easily adapted to LLM structures to fully harness the capacity pretrained LLMs, and also lack zero-shot capabilities.
Video LLMs for video temporal grounding. Large language models (LLMs) (Kaplan et al., 2020; Achiam et al., 2023; Touvron et al., 2023) have demonstrated significant potential in acquiring knowledge and addressing real-world challenges using zero-shot approach. Recent research has focused on integrating knowledge from other modalities, such as vision (Liu et al., 2024; Li et al., 2023a) and audio (Ghosal et al., 2023), bolster the capabilities of L.LMs. Within the visual domain, video large language models (video LLMs) have emerged
crucial research area (Lin et al., 2023a; Maaz al., 2023; Zhu et al., 2023; Song et al., 2024a;b). Traditional video LLMs (Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023b; 2024; Cheng et al., 2024; Yao al., 2024) have made considerable performance improvements in tasks such as video question answering, reasoning, and video captioning. However, these methods encounter difficulties in precisely pinpointing event timestamps within videos. To address this issue, TimeChat (Ren et al., 2023), VTimeLLM (Huang al., 2023), and Hawkeye (Wang et al., 2024b) have attempted overcome this limitation by
3

---

Published as a conference paper at ICLR 2025
fine-tuning the video LLMs VTG datasets. More recently, LITA (Huang et al., 2024) introduces fast-slow visual tokens and incorporates time tokens into LLM tokenizers. Momentor (Qian et al., 2024) suggests time encoder to address time token quantization errors. VTG-LLM (Guo et al., 2024) integrates special time tokens and time position embeddings to improve the ability of video LLMMs in comprehending timestamps. However, these methods do not take into account the inherent structure of videos and still cannot achieve satisfactory performance. In this paper, we propose the causal event modeling framework to provide structured video LLM outputs and design the TRACE model to address the proposed framework. Numerical results demonstrate significant performance gains TRACE over existing video LLMs on VTG tasks.
TRACE
In this section, we aim to develop a novel video LLM that aligns well with video structures, ad- dressing two questions: (I) how to model the structured video LLM outputs that align well with video structures, and (2) how to implement theoretical models. We start by proposing causal event modeling framework to tackle "how model". Then, introduce TRACE to address "how to implement". We have included a detailed discussion about our framework in Appendix C.
3.1 MODELING THE INHERENT STRUCTURES OF VIDEOS
Formulating outputs of video LLMs by events. Given the instruction and video visual inputs F, represent the outputs of video LLMs R as series of events {c1, C2, ,ek), ). with each event 6₁ = (fx, Sk, (t) encompassing timestamps t. salient scores Sk, and textual captions cj. In summary, we have
R = fe1. ". ,ek) ((t,, Sp, cx)l1 S K).
Causal event modeling framework. the design of causal event modeling shares the underlying intuition of causal language modeling, as formulated in the subsequent equation
To effectively utilize the knowledge of pretrained LLMs,
P(ealer:-11. = PP(k, Sk, cgletx-1.111)
Pttx F)P(sssk CI:--1, F)PPP(C 8k, 4t 1,,k-1,,,F)
(2)
The next event é₁ determined by textural instructions, visual inputs, and previous events. We can find that causal event modeling framework aligns well with the video structure (Figure 1(a)): (1) timestamps, salient scores, and textual captions are sequentially decoded within each event; (2) events are then ordered by timestamps.
3.2 TRACE: INSK-INTERLEAVED TEMPORAL GROUNDING VIDEO LLM
In Eq. 2, we introduce a formal causal event modeling framework to tackle the challenge of modeling structured video LLM outputs. This section illustrates the design of TRACE implement the causal event modeling framework (Figure 2).
Overview of TRACE. encoding/decoding of visual frames (F), text (I and cx), timestamps (t&), and scores (sk). Conse- quently, the TRACE considers these elements as distinct tasks and employs the following design to efficiently manage them.
As illustrated in Eq. 2, the causal event modeling framework necessitates
Separated multi-task processing. TRACE utilizes separate encoders and decoding heads for each task convert task inputs into task tokens and decode task tokens back outputs (Sec. 3.2. Task-interleaved sequence modeling. Task tokens are sequenced in an interleaved manner ac- cording to Eq. 2 in TRACE and fed into LLM backbones (Sec. 3.2.2).
Adaptive head-switching mechanism for generation. During generation, we implement an adap- tive head-switching mechanism to select the appropriate decoding head for producing the next token (Sec. 3.2.3).
Theoretically, the order of time, score, and text will not impact the results. We select order here.
4

---

Published as a conference paper at ICLR 2025
visual token
time token
score
token
S
G
PG (LF)
1-)
P(g (.,.FF)
Ple. (q,LF)
Figure 3: Illustaaioon of token sequence of TRACE. Following Eq the sequence begins with visual frame
tokens (F) followed by instruction tokens (I). Event (e) tokens are structured the sequence of time tokens (0), score tokens s, and text tokens c, with events ordered chronologically based on their occurrence time.
3.2.1 SEPARATED MULTI- MASK PROCESSING
TRACE consists of four unique tasks: visual frames, text, timestamps, and scores. Regarding text, we directly utilize the text tokenizer and LLM head of the LLM backbone (Mistral-7B-vol (Jiang et al., 2023)). Moreover, we added special token (sync) for indicating the end of text tasks. The processing for the other tasks is detailed below.
Timestamps and scores processing. For processing timestamps and score information, em- ploy two separate encoders and decoding heads, both of which share the same architecture. Specifically, each encoder is initialized with tokenizer containing 13 tokens: 11 number tokens ........ (9), (-) for representing timestamps/scores, (sep) to mark the end of each timestamp/score, and (syne) to signify the end of the current task. Token embeddings are initialized using LLM token embeddings.
In accordance with the research in VTG-LLM (Guo al., 2024), format each timestamp/score to the same length, comprising 4 whole-number parts, dot, and fractional part 2. Subsequently, (sep) is inserted between timestamps/scores, and (sync) appended the end of each times- tamp/score input sequence. For instance, the timestamp inputs [10.23, 125.37] will be tokenized into the following sequence: (0)
(sep) (5) (symc).
Visual frames processing, Given a T'frame video, we initially encode the frames using the pre- trained CLIP ViT-L (Radford al., 2021), with each frame being encoded into 576 visual tokens. Subsequently, we employ Slot-Based Compression (Guo et al., 2024) to reduce the number of visual tokens to 8 per frame. Moreover, to integrate temporal information into the visual inputs, we use time encoder to encode the timestamps of each sampled frame and remove the (symc) and (sep) tokens, resulting in 6 time tokens for each frame. Finally, we concatenate the 8 visual tokens with the 6 time tokens form the visual inputs for each frame.
3.2.2 TASK-INTERLEAVVD SEQUENCE MODELING
Utilizing the processed task tokens, we construct the sequence following Eq. 2. The token sequence order is illustrated in Figure 3.
Inter-event sequence order. The sequence commences with visual frame tokens F followed by textual instruction tokens I. For the events section, event tokens are sequenced according to events' occurrence time to align with the causal event modeling formula I,
Intra-event sequence order. For each event, in accordance with Eq. tokens are arranged se- quentially by time tokens (P(tyler:x-,, I, F)), score tokens (P(s)lta. 61:k-1, I, F)), and text tokens (P(ckl te 61:4-1, F)).. Consequently, the causal event modeling framework (Eq. 2) emerges as specialized autoaggressive model, featuring a unique sequence order that closely aligns with video structures.
Different from timestamps, scores will be encoded to 3 score tokens, including whole-number parts, dot, and fractional part.

---

Published as a conference paper at ICLR 2025
Table 1: Datasets used for TRACE training process. "Compressed" indicates that datasets are condensed by retaining only one sample for samples with identical videos but varying instructions.
Stage
Datasets
Quantity
Stage
Valley, LLaVA_ TextVR, ShareGPT4Video, VTG-IT
1.9M
Valley (Compressed), TextVR (Compressed), ShadeGPT4Video (Compressed), VTG-IT, ActivityNet Captions, VideoChatGPT, In- tern Vid, Next-QA
Stage 2
0.9M
3.2.3 ADAPTIVE HEAD-SWITCHING MECHANISM FOR GENERATION
Using (sync) token for adaptive head switching. Since TRACE employs distinct decoding heads for various tasks during training, selecting the appropriate decoding head during generation based on previously decoded tokens is crucial. This selection facilitated by the (symc) token. As illustrated in Figure 4, TRACE generates tokens in the sequence of time, score, and text tokens. Detection of the (sync) token prompts TRACE to switch decoding heads accordingly. The heads are cycled switched in the order of time head score head text head.
3.3 TRAINING STRATEGY AND DATA PREPARATION
This section outlines the TRACE training process, which includes stages. For the stage 1, task modules such as the vision compression layer, task encoder, and task heads are trained for initialization. For the stage 2, the LLM backbone is fine-tuned while keeping the task modules tuned. Detailed settings and datasets are presented below. Due to the page limitation, detailed annotation examples for each task, and details about data filtering and processing provided in Appendix A.
Stage 1: Initialization of task modules. In stage 1, task modules such as the vision compression layer, time encoder/head, score encoder/head, and text tokenizer/head are trained while the vision encoder and LLM backbone remain fixed. As shown in Table 1, stage I primarily utilizes two groups of datasets.
Image and video caption datasets for initializing the visual compression layer. This group of datasets including Valley (Luo et al., 2023b), Image (Liu et al., 2024), TextVR (Wu al., 2025), and randomly sampled subset of ShareGPT4Video (Chen et al., 2024a) datasets. VTG datasets for task encoder/head initialization. We use VTG-IT dataset in this group.
For stage 1 training, uniformly sample 128 frames from each video. The learning rate is set to 1e-3, and models are trained for one epoch with batch size of 128.
Stage 2:
Instruction tuning for enhancing VTG capacity. In Stage 2, the LLM backbone and task modules fine-tuned, with only the vision encoder remaining fixed. As shown in Table 1, stage 2 primarily utilizes three groups of datasets.
VTG instruction tuning datasets for enhancing VTG capacity. We use VTG-IT (Guo al., 2024), ActivityNet Captions (Fabian Caba Heilbron & Niebles, 2015), and a subset of Intern Vid (Wang al., 2023b), resulting in total of 635K data samples. Low-quality samples were filtered out, and the VTG-IT-VHD and VTG-IT-VS datasets were re-annotated. Additional details can be found in Appendix A.
Video caption datasets for maintaining the quality of the visual compression layers. We use parts of the video data from stage such as Valley (Luo et al., 2023b), TextVR (Wu et al., 2025), and ShareGPT4Vides (Chen et al., 2024a) datasets. These datasets compressed by retaining only one sample for samples with identical videos but different instructions, yielding 284K data.
Video question answering datasets enhance TRACE's reasoning capabilities. We use VideoChatGPT (Maaz et al., 2023) and Next-QA (Xiao et al., 2021) in this part.
For each video, the content is uniformly divided into 128 clips, with frame randomly sampled from each clip. The learning rate is set to 5c-6, and the models are trained for two epochs using a batch size of 128.

---

Published as a conference paper at ICLR 2025
Model Response
Large Language Model
token
Figure 4: Generation process of TRACE. The TRACE generate tokens following the order of time tokens, score tokens, and text tokens. The decoding heads are switched when (sync) tokens are generated.
4 EXPERIMENTS
Detailed experimental settings and hyper- parameters be found in Appendix B.I. Numerical results on more video understanding benchmarks and more ablation studies can be found in Ap- pendix B.2. Case studies be found in Appendix B.3.
4.1 EVALUATION DATASETS, METRICS, AND BASELINE MODELS.
We evaluate the model performance three different tasks:
Dense video caption. We use Youcook2 (Zhou et al., 2018) and ActivityNet Captions (Fabian Caba Heilbron & Niebles, 2015) datasets the evaluation datasets. The evaluation metrics include CIDEr (Vedantam et al., 2015), METEOR (Banerjee & Lavie, 2005), and SODA.c (Fujita al., 2020) for assessing the quality of the captions. These metrics are averaged under different IoU thresholds {0.3, 0.5, 0.7, 0.9), following previous studies (Ren et al., 2023; Huang et al., 2023). Additionally, we report the Fl score to measure the model's ability to accurately locate timestamps.
Moment retrieval. We utilize test set of Charades-STA (Gao al., 2017) for the moment retrieval task and report the recall at IOU thresholds of 0.5 and 0.7. Additionally, we present the mIOU results.
Video highlight detection. We employ the validation set of the QVHighlights dataset (Lei et al., 2021) and report the mean average precision (mAP) with IOU thresholds of 0.5 and 0.75, as well the HIT @1 1, which represents the hit ratio of the highest scored clip.
For baseline models, we select Valley (Luo et al., 2023b), VideoChat (Li et al., 2023b), Video- ChatGPT (Maaz et al., 2023), and Video-ILaMA (Zhang al., 2023) examples of traditional video LLMs. For video L.LMs specifically designed for VTG tasks, we choose TimeChat (Ren al., 2023), V. (Huang al., 2023), Momentor (Qian et al., 2024), HawkEye (Wang et al., 2024b), and VTG-LLM (Guo et al., 2024).
4.2 PERFORMANCE OF TRACE
Superior zero-shot performance of TRACE over other video LLMs. In Table 2, we show the zero-shot performance of TRACE compare SOTA video LLM baselines. The results show that
Suprior zero-shot performance. As shown in Table 2, TRACE significantly outperforms other video LLMs by substantial margin across all three datasets. Notably, it achieves a 3.1 and 4.9% performance improvement on the Youcook2 dataset using the CIDEr and FI Score metrics; 6.5% and 3.7% performance increase Recall with IOU (0.5, 0.7) thresholds the Charades- STA dataset; and 10.3% and 9.2% performance gain for the mAP and HIT@@ metrics on the QVHighigghts dataset.
Better performance than task-specific models and larger LLMs. As shown in Table 2, generalist model capable of handling various tasks, the performance TRACE surpasses that of task-spec models like HawkEye (Wang al., 2024b). Furthermore, the 7B TRACE model outperforms the VTimeLLM (13B) model (Huang et al., 2023), further validating the advantages of the TRACE architecture.

---

Published as a conference paper at ICLR 2025
Table Zero-shot performance of algorithms over various tasks. We evaluated performance TRACE using the Youcook2, Charades-STr. STA. and QVHighlights datasets. We highlight the best results for each block using bold font. The Valley, VidooChat-Embed, and Video-LLaMs results are elaborated from previous stud- ies (Ren et al., 2023; Huang al., 2023; Qian al., 2024). The results with transparent text indicates unfair comparison (13B).
Youcook2
Charades-STD
QVilihhights
Model
SODA_c CIDEr
FI Score
R (1ou-0.5
R@ (qu0-a7)
mAP HIT@@
Traditional Video L.Lss
Valley (7B)
0.1
0.0
1.5
4.7
1.6
10.9
15.2
VideoChat (7B)
0.2
0,6
3,4
3,2
1.4
13.1
18.1
Vidco-LLaMA (7B)
0.0
0.0
0.1
2.7
1.2
11.3
15.6
Temporal Grounding Video LLMs
TimeChat (7B)
1.2
3,4
12.6
32.2
13.4
14.5
23.9
V (7B)
27.5
11.4
VT (13B)
34.3
14.7
Momentor (7B)
26.6
11.6
7.6
HawkEye (7B)
31.4
14.5
VIG-LLM
1.5
5.0
17.5
33.8
15.7
16.5
33.5
TRACE (7B)
2.2
8,1
22.4
40.3
19.4
26.8
42.7
Performance of TRACE ActivityNet Captions dataset. In Table show the performance of TRACE on ActivityNet Captions dataset. All the reported algorithms except for Momentor (Qian al., 2024) have incorporated the ActivityNet Captions dataset part of the training data. Results show that the TRACE attains the best performance in moment retrieval tasks and demonstrates comparable results VTimeLLM in dense video caption tasks.
4.3 ABLATION STUDIES OF TRACE.
The causal event modeling framework enhances model performance in VTG tasks. In the Ablltioon Studies Architecture' section of Table 3, conducted experiments without utilizing the causal event modeling framework. The results indicate that employing the causal event modeling framework significantly improves model performance, and TRACE can achieve better results even when sampling fewer video frames.
Using different encoders and decoding heads for different tasks is essential for TRACE to achieve the best result. In the "w/o independent ecoder/heads" part of Table 3, we performed ablation studies by not utilizing separate encoders and decoder heads for different tasks. Instead, we directly incorporated time tokens and score tokens into the text tokenizers. The results suggest that using shared encoder/decoding heads for causal event modeling framework significantly disrupts the prelearned knowledge of LLMss, leading irrelevant and meaningless responses.
The performance of TRACE improves with the increase in the number of frames. We con- ducted ablation studies on the number of sampled frames, as presented in Table 3. The results show that (1) the performance of TRACE enhances as the number of sampled frames increases; (2) the performance of TRACE is comparable or even superior to SOTA video LLMs like VTG-LLM and ThmeChat when sampling just 8 frames, demonstrating the effectiveness of the TRACE model architecture.
Incorporating Intern Vid (Wang et al., 2023b) and ActivityNet Captions (Fabian Caba Heil- bron & Niebles, 2015) datasets boost TRACE performance on long videos. As illustrated in Figure 5, we carried out ablation studies by exclusively using VTG-IT as the training data for VTG tasks. The results indicate that the performance of TRACE on long videos improves when incorpo- rating intern Vid and ActivityNet Captions datasets, leading to enhanced performance on Youcook2, QVHighlights, and AetivityNet Captions datasets. Conversely, the performance of TRACE short videos slightly decreases (Charades- Sarrddes-TTA,, suggesting that the annotations in the intern Vid and Ac- Nev Captions datasets may not be accurate those in short video annotations.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation studies TRACE. All algorithms solely utilize VTG-IT (Guo al., 2024) during fine-tunin tine-tuning for efficient evaluation. The "w/o causal event modeling approach indicates the use of natural language-style imputs similar to previous studies (Guo et al., 2024; Ren et al., 2023). The "w/o independent encoder/heads" approach signifies directly adding tokens to the L.M tokenizer instead of employing sep- eneoders/heads for different tasks. We highlight the best results using bold font for each black.
Youcuok2
Charades-STI
Models
Frame Number
SODA_c
CIDEr Fl Score R loocans) R Igou=o.7)
Ablation Studies Architecture w/o causal event modeling
96
1.4
17.2
w/o independent encodcr/heads
64
to Follow Instruction
TRACE (VTG-IT)
64
1.9
6.9
21.4
37.0
17.0
Ablation Studies Frame Number
TRACE (YTG-IT)
1.4
5.0
18.6
28.8
TRACE (VTG-IT)
64
1.9
6.9
21.4
37.0
17.0
TRACE (VTG-IT)
128
2.1
7.5
21.4
41.2
20.0
Ril
WWV
(a) Performance on Youcook2
(b) Performance on Charades-STI
Pefomance
Petomance www.eaoooooooooooooo Actalyee Clapee
YN/O (MC
177 Good
NeySBull
Kghemee
Rige merii
440
(c) Performance QVHighlights
(d) Performance on ActivityNet Captions
Figure 5: Ablation studies data utilized while training TRACE. We conduct experiments solely utilizing VTG-IT and compare its performance with that of the original TRACE.
4.4 FINE-TUNED PERFORMANCE OF TRACE.
Competitive performance of TRACE to traditional methods after fine-tuning. In Table we fine-tune the TRACE for 3 epochs Youcook2 and Charades-STA datasets The results indicate that
TRACE significantly outperform generalist baselines. In contrast TimeChat and VTG-LLM, which struggle to attain satisfactory performance even after fine-tuning, the TRACE derives sig- nificant benefits from fine-tuning and achieves notably better performance than generalist base- lines. These results further substantiate that our enhancements to the model architecture are crucial for VTG tasks.
TRACE achieve comparable performance to non-generative and task-specific SOTAs. As de- picted in Table 5, the TRACE achieves new SOTA results on Youcook2 (without audio inputs). Furthermore, the performance of TRACE on the Charades-STi STA dataset is also competitive with non-generative models such as InternVide22 and VDL. However, these methods cannot handle various tasks simultaneously and lack zero-shot capability the contribution of TRACE herein.
Results QViighighhts be found Appendix B.2.

---

Published as a conference paper at ICLR 2025
Table Performance TRACE on ActivityNet Captions dataset. The evaluation of Timechat's and VTG- LLM's results was conducted using the official provided checkpoints. The indicates zero-shot evaluation. We highlight the best and the second best results using bold and underline.
Dense Video Caption
Moment Retrieval
Models
METEOR SODA
CIDEr
Fl Score R@ (000=005)
R@ (000-077
mIOU
VTimeLLM
6.8
5.8
27.6
29.5
14.2
31.4
Momentor
4.7
2.3
14.9
23.0
12.4
29.3
TimeChat
5.7
4.7
19.0
36.9
4.6
2.0
6.9
VTG-LLM
5.9
5.1
20.7
34.8
8.3
12.0
TRACE
6.4
6.0
39.3
37.7
39.0
Table 5: Fine-tuned performance TRACE. We fine-tune the TRACE for 3 epochs the Youcook2 and Charades-STA datasets. We emphasize the best and second best results using bold font and underline. For Youcook2 dataset, we choose Vid2Seq (Yang et al., 2023), PDVC (Wang al., 2021), and CM2 (Kim et al., 2024) as task-specific baselines. The results depicted in gray indicate unfair comparisons due to additional audio inputs and different architectures. For charades-STA dataset, choose Intern Video2-6B (Wang et al., 2024a). VDI (Luo al., 2023a), and Moment- DETR (Lei al., 2021) examples of hon-generative models.
Youcook2
Charades-STA
Model
Model
c CIDEr Fl Score
R @ laou-o5)
R@ lgou=o7)
Task Specifpe Models
Non-Generative Models
PDVC
4.4
22.7
Intern Video2-6B
70.0
49.0
Vid2Seq (Audio Input)
7.9
47.1
27.3
VDI
52.3
31.4
Vid2Seq
5.7
25.3
23.5
Moment-DEN DETR
55.7
34.2
CM2
5.3
31.7
28.4
Generative Models
Generalist Models
HawkEye
58.3
28.8
TimeChat
3.4
11.0
19.5
TimeChat
46.7
23.7
VTG-LLM
3.6
13.4
20.6
VTG-LLM
57.2
33.4
TRACE
6.7
35.5
31.8
TRACE
61.7
41.4
CONCLUSION AND FUTURE WORKS
In this paper, our goal address the mismatch between video structure and video LLMs on VTG tasks, and propose causal event modeling framework and the TRACE model as a solution. Numer- ical results indicate the superior zero-shot performance of TRACE compared other video LLM baselines, and TRACE also achieves competitive performance relative to traditional non-generative and task-specific models after fine-tuning. By overcoming the inherent limitations of video LLM architectures, TR ARACE demonstrates the potential of video LLMs on VTG tasks, and we believe that the TRACE could be a strong foundation for future research on video LLMs VTG tasks.
However, there are future works that can further enhance the capabilities of TRACE. For instance, TRACE relies on the pre-trained decoder-only LLMs, and only using previous events to predict the next event, which may not discover the complex event relationships as pointed out by previous stud- ies (Yi et al., 2019; Girdhar Ramanan, 2019; Li et al., 2020), As a remedy, we can use the outputs of causality discovery models (Liang et al., 2022; Chen et al., 2024c) supplementary inputs for TRACE to provide a more comprehensive understanding of video contents. Furthermore, expanding the annotation of more video understanding tasks by incorporating the occurrence timestamps of QA pairs and the matching score between questions and answers could significantly improve the overall performance of TRACE.
ACKNOWLEDGMEENTS
This work is supported in part by the funding from Shenzhen Institute of Artificial Intelligence and Robotics for Society, in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low- Carbon Energy Network (Grant No. 1BSSYS2020006100000000000 in part by Shenzhen Stability
10

---

Published as a conference paper at ICLR 2025
Science Program 2023, and in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 202281212010001..
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 08774, 2023.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on infrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Meinardus Boris, Batra Anil, Rohrbach Anna, and Rohrbach Marcus. The surprising effectiveness of multimodal large language models for video moment retrieval. arXiv preprint arXiv:2406.18113, 2024.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint arXiv:2406.04322 2024a.
Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: omni-modality foundation model and dataset. Advances in Neural
A iasion-audio-subtitee Information Processing Systems, 36, 2024b.
Tieyuan Chen, Huabin Liu, Tianyao He, Yihang Chen, Chaofan Gan, Xiao Ma, Cheng Zhong, Yang Zhang, Yingxue Wang, Hui Lin, et al. Meed: Unlocking multi-event causal discovery in video reasoning. arXiv preprint arXiv:2409. 17647, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.074766 2024.
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, and Ji- Ri-Rong Wen. Towards event-oriented long video understanding. arXiv preprint arXiv:2406. 141.441112 14129, 2024.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet:
A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961-970, 2015.
Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. Soda: Story oriented dense video captioning evaluation framework. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 517-531. Springer, 2020.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gen- eration using instruction-tuned Imm and latent diffusion model. arXiv preprint urXiv:2304. 13731, 2023.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and tem- poral reasoning. arXiv preprint 110.4:1910004444, 2019.
Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Xi Chen, and Bo Zhao. Vtg-Ilm: Integrating timestamp knowledge into video lims for enhanced video temporal grounding. arXiv preprint arXiv:2405.11 13382, 2024.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Lue Van Gool. Creating summaries from user videos. In Computer EisionEECCy 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VII 13, pp. 505-520. Springer, 2014.
11

---

Published as a conference paper at ICLR 2025
Donghoon Han, Seunghyeon Seo, Eunhwan Park, Seong-Uk, Nam, and Nojun Kwak. Unleash the potential of clip for video highlight detection. arXiv preprint arXiv:2404.017453 2024.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In Empirical Methods in Natural Language Processing (EMNLP), 2a.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In Empirical Methods in Natural Language Processing (EMNLP), 201
Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-Ilm: Cross-modal video summa- rization with temporal prompt instruction tuning. arXiv preprint arXiv:2404. 12353, 2024.
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower IIm grasp video moments. arXiv preprint arXiv:23/1. 2(3):9, 2023.
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. arXiv preprint 0rXiv:2403.19046 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Yang Jin, Linchao Zhu, and Yadong Mu. Complex video action reasoning via learnable markov logic network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3242-3251, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836) 2020.
Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Do you remember? dense video captioning with cross-modal memory retrieval. arXiv preprint arXiv:2404.0700 076100 2024.
J Lei, TL Berg, and M Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language (201). URL https://axxiv. org/abs/2107.09609
Jie Lei, Tamara L. Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 14:1181661155,, 2021.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint (:Xiv:2408.03322 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video under- standing benchmark, 2023c.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked efficient video foundation models. In Proceedings of the IEEE/CVF
teacher: Towards erainint-efficien International Conference Computer Vision, pp. 19948-19960, 2023d.
12

---

Published as a conference paper at ICLR 2025
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discov- ery in physical systems from videos. Advances in Neural Information Processing Systems, 33: 9180-9192, 2020.
Chen Liang, Wenguan Wang. Tianfei Zhou, and Yi Yang. Visual abductive reasoning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15565-15575. 2022.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023a.
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 2794-2804, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open-ended event-level video-language understanding. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Ye Liu, Siyuan Li, Yang Wu, Chang-Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3042-3051, 2022.
Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, and Yang Liu, Towards generalisable video moment retrieval: Visual-dynamic injection to image-text pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23045-23055, 2023a.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint 20%00 2023b.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05522, 2023.
Audreea-Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: video dataset with high-quality text and audio narrations. ICASSP 2021-2021 IEEE Internutional Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 2265-2269. IEEE, 2021.
Paritosh Parmar, Eric Peh, Ruirui Chen, Ting En Lam, Yuhan Chen, Elston Tan, and Basura Fer- nando. Causalchaos! dataset for comprehensive causal action question answering over longer causal chains grounded in dynamic visual scenes. arXiv preprint vrXiv:2404.012999 2024.
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint arXiw:211..02051, 2023.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang. Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18232 2024a.
13

---

Published as a conference paper at ICLR 2025
Enxin Song. Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-awe aware sparse memory for long video question answering. arXiv preprint 17X11:2404.17176. 2024b.
Yale Song. Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pp. 5179-5187, 2015.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 2023.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: large-scale dataset for comprchensive instructional video analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 35:10078-10099 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: consensus-basee image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575 2015.
Teng Wang. Ruimao Zhang. Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6847-6857 2021.
Teng Wang, Jinrui Zhang, Feng Zheng, Wenhao Jiang, Ran Cheng, and Ping Luo. Learning grounded vision-language representation for versatile understanding in untrimmed videos. arXiv preprint arXiv:2303.06377, 2023a.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjee Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 22 2022.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understand- ing and generation. arXiv preprint arXiv:23377.06942 2023b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video 2024a.
understanding. arXiv preprint 4rXhv:2403.15332
Yueqian Wang, Xiaojun Meng, Jianxin Liang. Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawk- eye: Training video-text lims for grounding text in videos. arXiv preprint arXiv:2403. 2024b.
Yuxuan Wang. Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, Yang Liu, and Zilong Zheng. Efficient temporal extrapolation of multimodal large language models with temporal grounding bridge. arXiv preprini 4rXiv:2402.16055 2024c.
Weijia Wu, Yuzhong Zhao, Zhuang Li, Jiahong Li, Hong Zhou, Mike Zheng Shou, and Xiang Bai. A large cross-modal video retrieval dataset with reading comprehension. Pattern Recognition, 157:1 10818, 2025.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. arXiv preprint arXiv:2411.1033 103332, 2024.
14

---

Published as a conference paper at ICLR 2025
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Hengwei Bian, Yatai Ji, Yujuu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. arXiv preprint arXiv:2311.16460 2023.
Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109, 14084, 2021.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04999, 2022.
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Fuse-Tuset, Ivan Laptev, Josef Sivie, and Cordelia Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10714-10726, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm your phone. arXiv preprint arXiv:2408.00 8000, 2024.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.0144.. 2019.
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mo- hit Bansal. Hicrarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23056-23065, 2023a.
Abhay 7ala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mohit Bansal, Hierarchical video-moment retrieval and step-captioning. In CVPR, 2023b.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In Advances in Neural Infor- mation Processing Systems 34, 2021.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2.306. 02858, 2023.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint N.Xv:22400002713 2024.
Long Zhao, Nitesh Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint arXiv:2402. 13217, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing langunel understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
A Dataset Preparation
16
A.I
Details of data format
16
A.2
Processing Intern
16
A.3
Processing VTG-IT
18
Experiments
20
B.1
Detailed Experimental Settings
20
B.2
Additional Experiment Results
20
B.3
Case Studies
21
C Discussion on Causal Event Modeling
22
C.1
Causal Language Modeling VS. Causal Event Modeling.
23
C.2 Causal Event Modeling VS. Complete Causal Relationship Modeling/Discovvyy
24
A DATASET PREPARATION
A.1 DETAILS DATA FORMAT
In this section, introduce the details of the data format used while training TRACE. For the VTG datasets, in addition to ActivityNet Captions dataset and Intern VidernVid dataset, we directly use the annotation collected by VTG-IT (Guo al., 2024). In detail, the annotations be categories into the following four types:
General tasks (Figure 6). For general tasks such as video captioning, image captioning, and video question answering, the answer component of the data does not include timestamps or scores. Consequently, we employ a single token (sync) as a placeholder for timestamps and scores, signifying empty response for this part of response. This kind of annotation in- cluding LLaVA Image (Liu et al., 2024), Valley (Luo et al., 2023b), TextVR (Wu et al., 2025), ShareGPT4Vides (Chen et al., 2024a), VideoChatGPT (Maaz al., 2023), and Next-QA (Xiao et al., 2021) datasets.
Dense video caption task (Figure 7). The Dense Video Captioning task solely comprises times- tamps and textual captions responses. As a result, we use single token (sync) as a placeholder for scores. The datasets of this task include HiREST step (Zala et al., 2023a), COIN (Tang et al., 2019), ActivityNet Captions (Fabian Caba Heilbron Niebles, 2015), VTG-IT-DVC (Guo al., 2024), and Intern Vid (Wang al., 2023b) datasets.
Moment retrieval task (Figure 8). Similiar to dense video caption task, moment retrieval solely comprises timestamps and textual captions responses. The moment retrieval task includ- ing HiREST grounding (Zala al., 2023a). QuerYD (Onceseu et al., 2021), DiDeMo (Hendricks et al., 2018b), VTG-IT-MR (Guo et al., 2024), and Intern Vid (Wang et al., 2023b) datasets.
Video highlight detection task (Figure 9). For the video highlight detection task, we utilize the query as the textual response for all highlight moments. In this case, employ the VTG-IT- VHD (Guo et al., 2024) dataset.
Video summarization task (Figure 10). The video summarization task employs the caption of each event as the textual response. We use the VTG-IT-VS (Guo al., 2024) dataset here.
A.2 PROCESSING INTERNVID
Dense video caption data. The dense video caption data is constructed based on the Intern Vid- Full annotations. For each video, opt to the annotation of that video dense video caption data if any of the following checklist items are met:
There exists caption with fewer than 5 words.
There exists a caption very similar to another caption within one video. We use fuzzywuzzy package here and set the threshold 70.
16

---

Published as a conference paper at ICLR 2025
"id": "0", "video": wwaleyy/BF110075555///22224....9 "conversations".
"fron": "human",
"value": "<video>\nMrin terse but informative sumnary of the following video
clip."
b
fron"" "gpt",
juarez, mexico circa 1970: mexican tourists on the square of the cathedral of our lady of the assumption in the city of oaxaca. archival of mexico in оaxaca state in the 1970s."
"value": "sync>xtimmxxccoeeeaxxee
"times":
[1
"scores": [
[]
Figure Annotation example of video caption task.
'videa": "attivityyee//vides///_BB http:/sssww_ttuuutMM........ 'id': 378958,
"canversation':
'fron': 'human', 'value": 'evideo> Inscrutiin the video and determine multiple occurrences, providing their initial final timestamps well summary action."
'fron': 'gpt', 'value': otee ott
owcast logo is the screen.<syne>ctt ierttieeextinociinexttieeeeeee core>two men are sitting in middle drums explaining about instruments. cwa.uueennttccyccttttttt..o csteeccceeeetctettteiiee ineeettteeetiieeiiieeeee.. it setimmectimee in cttimmeectt in me>cscore>the the men begin playing the drums. <scnnccctieecttiii sinteeesttteesttteeetttteeeccce ore>the go back talking
ctmettemmmmmmmeeemmmmmm ime> <tm
drums."
"scores : [1, [I, [I,
"tiees::
[e, 5.42], [5.42, 86.09], [86.77, 120.67], [120.67, 135.58]
Figure 7: Annotation example of dense video caption task.
17

---

Published as a conference paper at ICLR 2025
"video": seddecchttgpt_uun//__olSssseeee
"id": 8, "conversations":
"fron": "human",
"value": "<video>\néive you a textual query: 'a young woman is seen standing in roon and leads into her dancing. when does the described content occur the video? Please return the timestamp in seconds."
"fron": "gpt",
"value":
young woman is seen standing in a roon and leads into her dancing."
}
"scores":
[1
"times":
[0.83, 19.86]
Figure 8: Annotation example of moment retrieval task.
The number of events is less than 5 or greater than 50.
There are special characters present, excluding letters, spaces, and dots.
Moment retrieval data. We discovered that directly utilizing the Intern annotations to construct the moment retrieval data leads to suboptimal performance, likely due the imprecise annotation to con-
timestamp annotations. Consequently, we employ the Intern struct the moment retrieval data, which is filtered subset of the Intern Intern FullFull annotation provided by the authors.
A.3 PROCESSING VTG-IT
this section, we describe the processing the VTG-IT dataset. For the dense video caption and moment retrieval tasks, directly utilize the annotations provided by VTG-IT. However, the data for video highlight detection and video summarization tasks supplied by VTG-IT often have uniform salient scores within each video. As a result, goal here is to enhance the quality of data for these two tasks, specifically video highlight detection and video summarization.
Video highlight detection task. For each event in the dense video caption task, we initially divide each event into maximum of 20 clips. Subsequently, we compute the similarity be- tween the frames within each clip and the event captions using ViT-G/14 from EVA-CLIP (Sun al., 2023). The similarity scores then normalized to Gaussian distribution to serve the highlight score. In detail, the clips with scores higher than [2.275% %. 3.593%, 5.480%, 8.076%, 11.507%, 15.866%, 21. $1.1865 186%, 27. 27.425% 34.458%. 42.074%. 50.000% 57.9 57.926%, 65.544 65.542%, 72.575%. 78.814%, 84. 134%, 88.493% 91.924%, 94. 94.520% 96.407% 97.725% other clips will be assign scores [1.0, 1.2, 1.4, 1.6, 1.8, 2.0. 2.2, 2.4, 2.6. 2.8, 3.0, 3.2, 3.4, 3.6, 3.8, 4.0, 4.2, 4.4, 4.6, 4.8, 5.0].
Video summarization task. The data for the video summarization task is built upon the founda- tion of the video highlight detection task and the dense video caption task. Specifically, for each event in the dense video caption task data, we select the clip with the highest score to serve as the summarization clip for that particular event.
18

---

Published as a conference paper at ICLR 2025
"video": wwt.aeelleessssaaee///rrrrrr///r 2.0
"id": 1, "conversations":
"From": "human",
"value": "<video>\nIn the video presented, seek moments that are a perfect
match with 'Man in baseball cap eats before doing his interview." It's vital to notate their timestamps and to score each based on their level of saliency."
b
"fram": "gpt",
"value":
in
baseball cap eats before doing his interview. interview.csynncxtiimee http://cweectttcceeeeeeeeaaeeeen
in baseball cap eats before doing his
in baseball cap eats before doing his <synn>>timm><timmee http://cc.tttttttiooooeeeeeeeeee
in baseball cap eats before doing his interview.<sync>xtiine xtmemeeeeeeeeeeeeeeettttee in baseball eats before doing his ttgp: seteeieeeeeiiiiiiieeecooooooeeee
in baseball cap eats before doing his interview.csync><e in baseball eats before doing his interview.csyne><tt in baseball eats before doing his unterview.<synee htw.://www....tt.eeeeeeeeeeeettn
in baseball eats before doing his interview."
"scores": [
[2. 66666666666666666666 [3.0], [2.8], [1.66]]]_6611010 616666666666666666671 [2. 626666666666666666666
[2. 2233333333333333335 [2.0], 6666666666666677) [ (2.3333333333333355]
"times":
[96], [98], [lee], [102], [184], [106], [108], [110], [112]
]
Figure Annotation example of video highlight detection task.
'video': vytemmorrl/viiees///eeeeettttt/i 'id': 370616, ccnverrations':
'faa'' 'human', 'value': (ciideo-\nSummarize video by pinpointing key frames encompass the storyline."
from': 'gpt', 'value':
and
materials needed for the DIY disco wtw.mmccttieetttttetttteeeee.e.. wtwp://wweetttttteeeeeeeeeeeeeee the CDs into small pieces using scissors."
: [5.0], [4.6]
"tines":
(34.534498828324)
(31.36466992338334)
Figure 10: Annotation example of video summarization task.
19

---

Published as a conference paper at ICLR 2025
Table Detailed training setting hyper-parameters.
Setting
Stage 1
Stage 2
Computation
16 ATN 910B
16 ATN 910B
Vision Encoder
http--vvi-argg-paach------3 Zero3 Offload Mistral-7B-v0.2
DeepSpeed Stage
Zero3 Offload
LLM
Mistral-7B-v0.2
Batch Size
128
128
Num Frames
128
128
Frame Sample
Uniform
Split to 128 clips then random within clip
Train Epochs
1
2
Learning Rate
le-3
5e-6
LR Scheduler
Cosine
Cosine
Model Max Length
4096
4096
Table 7: Ablation studies on training data (Youcook2 and Charades-STA))
Youcook2
Charades-STA
Models
SODA.c
CIDEr
Fl Score
METEOR R @ lyoυ=0.5)
R@ Laoυ=07)
mIOU
TRACE (VTG-IT)
2.1
7.5
21.4
2.6
41.2
20.0
38.9
TRACE
2.2
8.1
22.4
2.8
40.3
19.4
38.7
TRACE-uni
2.3
8.6
22.4
2.9
43.7
21.0
41.5
Table 8: Ablation studies on training data (ActivityNet Captions).
Dense Video Caption
Moment Retrieval
Models
SODA.c
CIDEr
FI Score
METEOR @ (yoU=).)
R@ [gou-oo) mIOU
TRACE (VTG-IT)
5.8
24.7
38.9
6.0
19.2
9.3
25.0
TRACE
6.0
25.9
39.3
37.7
24.0
39.0
TRACE-uni
6.4
29.2
40.4
6.9
38.2
24.7
39.4
B EXPERIMENTS
B. DETAILED EXPERIMENTAL SETTINGS
We report the detailed model architecture design and training hyper-parameters in Table 6. The training takes about days for stage and days for stage 2.
B.2 ADDITIONAL EXPERIMENT RESULTS
Ablation studies on training data. Tables 7 and 8, we report the performance of TRACE using different training data. The TRACE-uni indicates incorporating additional general video understand- (Zhang et al., 2024) (specifically the perceptiontest and YouTube parts). The results show that (1) the performance of TRACE long videos increase while using the original TRACE setting. (2) The performance of TRACE slightly reduced on short videos (Charades-STAA while using the original TRACE setting: (3) Although adding more VTG data, TRACE-uni outperforms trace in both VTG tasks and general video understanding tasks.
ing data from a subset of LLaVA-Video-
Fine-tuned performance QVHighlights. In Table show the performance of TRACE on QVHighlights dataset after fine-tuning. The results indicate that TRACE significantly outperform other video L.LMs by large margin.
20

---

Published as a conference paper at ICLR 2025
Table Fine-tuned performance of algorithms on @VHighlights datasets. We fine-tune the Algorithm on QVHighlights datasets.
QVHighlights
Model
mAP
HIT@@ 1
TimeChat
21.7
37.9
VTG-LLM
24.1
41.3
TRACE
31.8
51.5
Table 10: Performance on general video understanding tasks.
Model
MVBench
VideoMME
Vide2LLaMA2 54.6
46.6
TRACE
48.1
43.8
TRACE-uni
53.8
49.6
Table 11: Performance on E.T.Bench.
E.T.Bench Metric
RAR
BCA
RVQ
TVG F1
EPM
TAL.
EVS
VHD
DVC
DVC
SLC
SLC
TEM
GVQ
Acc
F1
F1
Sim
Sim
Rec
Acc
Lamal (7B)
28.8
27.4
28.0
0.1
0.0
0.0
0.0
1.5
0.6
14,5
0.0
15.2
0.0
Qwen2-VL (7B)
39.4
34.8
42.2
3.9
0.4
20.6
0.0
6.6
55.9
GPT-4o
27.8
27.3
57.7
40.4
20.0
17.6
56.9
46.9
22.3
14.9
13.6
TRACE (7B)
29.4
28.8
42.6
46.8
12.3
21.6
26.6
45.2
45.7
24.0
27.3
17.7
17.8
52.4
Effectiveness of TRACE general video understanding tasks. We evaluate TRACE on gen- eral video understanding tasks, and the results in Tables 10, 11, and 12 show that the TRACE archi- tecture is still capable of handling general video understanding tasks and excel VTG tasks:
Despite not being trained on extensive multi-task datasets, TRACE is still highly effective in handling general video understanding tasks. For example, the TRACE outperform generalist video LLMss like VideoChat2, ShareGPT4Video, and ST-LLM on VideoMME. benchmark.
On the E.T.Bench (Liu et al.). TRACE outperforms VideoLlama2 across all tasks; achieves per- formance comparable GPT-4o RAR, ECA, RVQ, and DVC tasks; achieves similar perfor- mance to Qwen2-VI. on RVQ and GVQ tasks; and outperforms both GPT-4o and Qwen2-VI. on TVG, EPM, TAL, EVS, SLCC, and TEM tasks.
We train TRA by incorporating additional general video understanding data from subset of I.LVA- 78k (Zhang et al., (0eecsipafff the perceptiontest and YouTube parts). TRACE-uni shows both improved general video understanding and stronger VTG performance without additional VTG training data.
Notably, TRACE-uni performs on par with, the same LLM backbone and vision encoder (VideoLlama2) using only about 2M training data. Additionally, TRACE-uni surpasses TRACE in VTG performance across all three evaluation datasets.
outperforms, general video LLMs that
Performance of TRACE with different number of slots per frame. Tables 13 and 14, we adopted the same settings as in Table 3, using VTG-IT only in Stage 2 and sampling 64 frames. Our findings are as follows: Increasing the number of slots per token significantly enhances TRACE's performance. Therefore, if computational or efficiency constraints are not concern, we recommend using a larger number of slots per frame.
B.3 CASE STUDIES
We present the case studies of TRACE in Figure 11. The results demonstrate that TRACE can accurately identify the events within the given video and also proficient in performing traditional video captioning tasks.
21

---

Published as a conference paper at ICLR 2025
Zero-shot performance of algorithms over various tasks. evaluated performance TRACE using the Youcook2, Charades-STA, STA, and QVHighlights datasets. We highlight the best results for each block using bold font. The Valley. AddoChat-Embees and Lideo-LLaMA results are elaborated from previous studies (Ren et al., 2023; Huang et al., 2023; Qian et al., 2024). The results with transparent text indicates unfair comparison (13B). We train TRACE-uni by incorporating additional general video understanding data from subset of LLaVA --V--------- 178k (Zhang et al., 2024).
Table 12:
Youcook2
Charades -SA
QVHighlights
Model
SODA CIDEr Score Lgou=n.5)
R@ Inou=0.)
mAP IIIT
Traditional Video LLMs
Valley (7B)
0.1
0.0
1.5
4.7
1.6
10.9
15.2
VideoChat (7B)
0.2
0.6
3,4
3.2
1.4
13.1
18.1
Video-LLaMA (7B)
0.0
0.1
1.2
11.3
15.6
Temporal Grounding Video LLMs
TimeChat (7B)
1.2
3.4
12.6
32.2
13.4
14.5
23.9
LLM (7B)
27.5
11.4
VL (13B)
14.7
Momentor (7B)
26.6
11.6
7.6
HawkFye (7B)
31.4
14.5
VTG-LLM (7B)
1.5
5,0
17.5
33.8
15.7
16.5
33.5
TRACE (7B)
2.2
22.4
40.3
19.4
26.8
42.7
TRACE-uni (7B)
2.3
8.6
22.4
43.7
21.0
27.5
43.9
Table 13: Performance of TRACE with different number of slots for each token Youcook2 dataset. We only VTG-IT for training in Stage 2, and sample 64 frames for each video.
Frame Num
Slot Num per Frame
SODA
CIDEr
F1 Score
64
8
1.9
6.9
21.4
16
2.1
7.3
22.1
Table 14: Performance of TRACE with different number of slots for each token Youcook2 dataset. only use VTG-IT for training in Stage 2, and sample 64 frames each vidco.
Frame Num
Slot Num per Frame
RØ 1ou
R@ rou=0.7
64
37.0
17.0
64
16
41.9
20.1
C DISCUSSION ON CAUSAL EVENT MODELING
In this section, we provide deeper analysis of the causal modeling approach presented in this paper. We also offer a comprehensive discussion and comparison of the following three key concepts:
Causal Language Modeling. This well-established approach for decoder-only large lan- guage models (LLMs) and have become basis of current video LLMs (Li et al., 2023b: Ren et al., 2023; Huang et al., 2023)
Causal Event Modeling. In contrast, TRACE introduces causal event modeling, which struc- tures the responses of video LLMs by event triplets, providing novel framework for understand- ing video content.
Complete Causal Relationship Modeling/Discovery, Traditional video understanding mod- els (Liang et al., 2022; Parmar et al., 2024; Chen et al., 2024c; Du et al., 2024; Yi et al., 2019; Girdhar & Ramanan, 2019; Li et al., 2020: Jin et al., 2022) focus extensively discovering and analyzing the complex relationships between events within vidcos, offering a more comprehen- sive understanding of video content.
22

---

Published as a conference paper at ICLR 2025
0.0s
14.5s
29.00
43.5s
58.0s
72.58
87.0s
101.5s
Write single-sentence overview of the video, paying special attention the text and its role in the video.
Human
The and the chick playing together couch.
22
TRACE
Localize series activity events in the output start end timestamp for each describe each event with sentences.
Human
(limestamps': [[0.0, 54.6), 116.8]]. 'scores': 'captions": ['a small kilten laying the floor while chicken stands above it.,, 'the kitten then lays on the chicken and seen playing with it.
TRACE
Figure 11: Case study of TRACE.
We would like to first discuss how causal event modeling excels in video understanding compared to causal language modeling. Next, will explore the benefits, limitations, and potential integration of TRACE compared to complete causal relationship modeling/discoovry methods.
C.1 CAUSAL LANGUAGE MODELING VS. CAUSAL EVENT MODELING.
We will first provide clear comparison between causal language modeling and causal event mod- cling, highlighting their similarities and differences. Then, we will delve into the benefits of using causal event modeling in more detail.
Similarities. Both causal language modeling and causal event modeling rely on decoder-only LLMs, which video visual content (F) and instructions (I) inputs. The models then gen- erate responses based on the observations from both and F.
Differences. The key differences from the output formats.
For causal language modeling, the response can be seen as a whole text content, i.e.,
P Il plylu: < y. F,T)
(3)
Here, timestamps, scores, and text are all represented by text tokens, and ordered following the natrual language structure.
For causal event modeling, the responses are formatted to series of event triplets e; (t,, S4, ot), i.e.
p R.F.
Il p F,I)
II 66
p(llens, F, ()p(8 Cuss F, T)p(c1188 4, G1s F,I).
(4)
Key Improvements. We can find that the key improvement of causal event modeling over causal language modeling comes from the format model responses.
Clear inter- and intra- event triplets correlation. The inter-event relationships are modeled through the next event prediction formulation. For intra-event relationships, observe that causal connection exists between t; + 8;, and between (t;, 8;) - c;. In this context, t, and s; also be viewed kind of thought chain when generating G.-
23

---

Published as a conference paper at ICLR 2025
Enable independent timestamps, scores, and text modeling. We separate the modeling of timestamps t,. scores 8;, and text C,, and independently model each component using different encoders and decoders. As directly adding new tokens to the text tokenizer may significantly disrupting the pretrained LLMs (Guo et al., 2024), such a decomposition helps to eliminate this issue, and the TRACE architecture retain the capacity on handling general video understanding tasks, as evaluated in Table 10.
C.2 CAUSAL EVENT MODELING VS. COMPLETE CAUSAL RELATIONSHIP MODELINGG/DISCORR
In this section, we will discuss the difference between causal event modeling and studies (Liang al., 2022; Parmar et al., 2024; Chen et al., 2024c; Du al., 2024; Yi et al., 2019; Girdhar & Ramanan, 2019; Li et al., 2020; Jin et al., 2022) focused on discovering or modeling complete causal relationships. Additionally, we will explore potential future work on integrating these two approaches.
Discussion on studies that discovering or modeling complete causal relationships. Based on the existing studies in this area, we classify the research into three main groups, which we will discuss in detail:
Building complete causal relationships to solve the video understanding problems. VAR (Liang et al., 2022) uses an encoder-decoder architecture that first encodes video frames into event representations, then decodes each event into text captions. Similarly, Jin et al. (2022) constructs a learnable Markov Logic Network for action reasoning. Both of these approaches model complex causal relationships between events before generating answers, thereby enhanc- ing the reasoning capacity of the models.
Introducing Benchmark Datasets for Complex Causal Reasoning. Several studies have intro- duced benchmark datasets to evaluate models' ability to perform causal reasoning (Parmar et al., 2024; Yi al., 2019; Girdhar & Ramanan, 2019; Du al., 2024).
Discovering Causality from Videos. Some research aims to automatically build causality graphs uncover the relationships between events, providing a more comprehensive understanding of video content (Chen et al., 2024c; Li et al., 2020).
Evaluation results of TRACE on causality reasoning benchmarks. In Table 15, we present the performance of TRACE on the Event-Bench (Du et al., 20e4).The results show that, despite being trained on less causal reasoning data (NexIQA) compared to other video LLMs (Li et al., 2023c), TRACE outperforms open-source video LLMsand achieves performance comparable to GPT-4o on event description, contextural reasoning, and episodic reasoning tasks.
Potential future improvements to TRACE through integration with causality discovery mod- els. Current video LLMs typically generate answers directly from text prompts and visual frames without explicitly modeling the causal relationships between events. While TRACE improves upon this limitation by representing model responses as event triplets, it still relies on previously generated events to produce subsequent triplets, due to the inherent architecture of decoder-only LLM back- bones. To address this issue, further improvements could be made by integrating causality discovery methods:
Utilizing the outputs of causality discovery models as inputs for video LLMs. For instance, we can encode the causality graph produced by Chen et al. (2024c); Li et al. (2020) as part of the model's inputs. For TRACE, this involves modifying p( F, I) to p( R F, I, I,C), where C rep- resents the generated causality graph. By incorporating the causality graph an additional input, the model would have access to richer context, enabling it to generate more accurate responses.
Utilizing the outputs of causality discovery models construct Chain-of-Thought exam- ples. We can also guide video L.MSs to first generate causality graphs before answering ques- tions, similar to the approach used by Jin et al. (2022). For TRACE, we can modify p( FRRFFF CC,, R, I) = F(C L)p(R[CC F,
Utilizing the outputs of causality discovery models to modify the attention masks of visual inputs. Currently, the attention masks for visual inputs are typically designed the same text
24

---

Published as a conference paper at ICLR 2025
Table 15: Evaluation results on Event-Bench.
Composile
Avg
Description
Temporal Resenning
Crasil Retooning
Avg
Counter Ressoning
Contestial Retsoning Fgiandic Ressoning
Avg
Video LLaVA (7B)
12.82
5.50
0.00
2.15
5.17
2.78
7.20
5.05
5.87
VideoChur2
33.76
16.74
(7R)
VIM (7D)
18.08
$1.25
61.25
56.25
22.91
32,66
18.67
25.71
41.64
Gemini-1.5PPo
TRACE (78)
55.56
49.25
$4.50
51.88
52.42
46.08
43.00
46.64 50.46
tokens, i.e., using causal attention masks. However, by incorporating the causality graph, we enhance the attention masks by masking visual tokens that causally related to the current reasoning task, thereby improving model focus on relevant events.
25


Reviews:
[{"review_id": "Sh1itr2J38", "reviewer": "Reviewer_KYwb", "summary": "This paper proposes a new method for Video Temporal Grounding (VTG) tasks, named TRACE. TRACE uses a causal event modeling framework to represent videos as a sequence of events with timestamps, salient scores, and textual descriptions. The paper designs a task-interleaved video large language model to address the limitations of traditional video LLMs in handling the inherent structure of videos. The TRACE model utilizes different encoders and decoding heads to process visual frames, timestamps, and text inputs, enabling more effective event sequencing and causal modeling. Experimental results demonstrate that TRACE achieves state-of-the-art zero-shot performance on various VTG tasks and, after fine-tuning, can match the performance of traditional non-generative, task-specific models.", "questions": "No additional questions. Please see the \"Weaknesses\" section for areas needing clarification.\n\n### Recommendations for Improvement:\n- **Refine Prompt Design Explanation:** Providing specific strategies or insights on prompt design tailored for VTG tasks would enhance the paper's originality and usefulness for future researchers.\n  \n- **Explore Custom Scene Parsing Techniques:** Introducing refined parsing methods could strengthen TRACE's robustness and accuracy in multi-modal alignment.\n\nThis structured feedback should provide the authors with a comprehensive view of the strengths and areas for enhancement in their paper on TRACE.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes TRACE, a framework leveraging causal event modeling to generate structured video representations through large language models (LLMs). This approach addresses structural gaps in video data, making it valuable for multi-modal research and practical applications in video analysis.\n   \n2. TRACE maximizes the potential of pre-trained LLMs by adopting causal event modeling, which decomposes video inputs into frames and aligns them with textual prompts. The temporal segmentation and alignment methods allow videos to be broken down into events with associated timestamps, salient scores, and captions. This granularity is crucial for precise video event parsing and presents a significant step forward in video understanding with LLMs.\n\n3.TRACE outperforms the existing Video-LLMs on three pivotal Video Temporal Grounding (VTG) benchmarks—Charades-STA, QV Highlights, and YouCook2—underscoring its efficacy and robustness in handling video temporal grounding tasks. This achievement underscores TRACE's ability to accurately capture and model the intricate temporal dynamics across a spectrum of video datasets.", "weaknesses": "1.\tWhile causal event modeling is presented as a core contribution of this work, the related work section does not address any prior research on similar methodologies. It would be helpful to clarify whether comparable approaches have been explored in the field of video understanding, or if this approach is entirely novel within this domain. Providing this context could strengthen the argument for the method’s originality and situate it more clearly within existing research.\n\n2.\tIt is unclear whether compressing visual features to 8 tokens is sufficient for preserving critical information in complex video scenes. The paper does not provide an analysis or experimental results on the trade-off between the number of tokens and model performance, which would be valuable in understanding the potential impact of this compression choice.\n\n3.\tThere are several grammatical and spelling errors throughout the manuscript, which impact readability and may detract from the paper’s clarity. For example: Line 22: \"processes\" should be corrected to \"process\". Line 44-45: The phrase \"...which,...\" should be rephrased, and \"lacks\" should be changed to \"which lack\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1D9NaLtOv", "reviewer": "Reviewer_p7QK", "summary": "The paper introduces a task-interleaved video LLM, TRACE, which incorporates a newly-designed causal event modeling framework for VTG task. The TRACE employs multiple encoders for different inputs, while the task tokens are arranged in an interleaved manner. TRACE demonstrates SOTA performance on various VTG datasets compared to previous video LLMs.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The presentation and illustration are quite clear and easy to follow.\n2. The motivation of causal event modeling is quite intuitive and the design is straightforward and yet effective.\n3. The zero-shot performance is superior compared to previous video LLM methods.", "weaknesses": "1. While the paper compares TRACE with other video LLMs, it presents limited comparison and may not adequately address how it stands against traditional non-generative and task-specific models.\n2. The extent to which TRACE can be applied to other types of video tasks beyond VTG is unclear. Its design may be highly specialized, which could limit its applicability across diverse video understanding tasks. Authors should present more results on other video-understanding tasks since the design seems generalizable by building such causal event relations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xn5uU63nyR", "reviewer": "Reviewer_MMw2", "summary": "This paper addresses the task of Video Temporal Grounding (VTG) and introduces TRACE, a task-interleaved Video-LLM designed for enhanced VTG performance. The authors highlight limitations in current Video-LLMs, which rely solely on natural language generation without considering the inherent temporal structure of videos. To address this, they propose a novel causal event modeling framework, decomposing videos into sequences of events defined by timestamps, salient scores, and textual captions. Extensive experiments on datasets such as Charades-STA, QVHighlights, and YouCook2 demonstrate the superior zero-shot performance of TRACE compared to existing Video-LLMs.", "questions": "See Weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Video Temporal Grounding (VTG) is a crucial task, yet current Video-LLMs underperform in this area. Techniques aimed at improving temporal grounding for these models are highly valuable to advance the field.\n2) The causal event modeling framework fits well with the next-token prediction paradigm of large language models (LLMs), offering an intuitive way to model video structures in sequential tasks.\n3) TRACE demonstrates consistent performance improvements over prior Video-LLMs across three key VTG benchmarks (Charades-STA, QVHighlights, and YouCook2), underscoring its effectiveness.", "weaknesses": "1) While the motivation for TRACE is clear, the use of multiple task-specific heads may limit the model’s generalization. A primary appeal of Video-LLMs lies in their ability to handle a variety of tasks without specific fine-tuning. TRACE’s focus on VTG may narrow its versatility, making it less effective for general video understanding tasks. In most cases, lightweight VTG-specific models with stronger performance could be more suitable for VTG scenarios.\n2) Some clarity is not clear. For example, the paper does not adequately explain slot-based compression, which is not a widely known technique. Moreover, compressing each frame to just 8 visual tokens might lead to significant information loss, raising concerns about the trade-off between efficiency and accuracy.\n3) It is unclear whether the same set of number tokens is used for both timestamps and scores. If so, this could blend the two types of information, contradicting the authors' claim (lines 45–46) that the model preserves the distinct structure of video events.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IepmOIBZmp", "reviewer": "Reviewer_YJgy", "summary": "This paper proposes a new Video Temporal Grounding (VTG) method that addresses the shortcomings of existing LLM in handling VTG tasks by modeling causal events.", "questions": "My main concern is with the autoregressive modeling approach, and if the authors can provide a reasonable explanation, I am willing to consider raising my score, as I believe this work could provide inspiration for future work.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The author first employs a causal modeling method in the grounding of VLLM, achieving causal probability modeling through the sequence of input tokens. This approach will provide inspiration for future work on video understanding tasks using VLLM.", "weaknesses": "1. **Autoregressive modeling.**\n\nOne of my major concerns is that the authors have only used the earlier events $e_{1:k-1}$ in their modeling of causal relationships between events through autoregression, without incorporating the equally known $e_{k+1:K}$. I believe this approach may be unreasonable since it is likely that the same events may occur earlier while the current event is different due to unrelated pretexts. However, this issue can be avoided by modeling different subsequent events simultaneously. Besides, most current video understanding researchers have modeled multiple events by utilizing all contextual events that occur before and after them [1-4]. This may require the authors to provide further explanation.\n\n[1] Liang C, et al. Visual abductive reasoning[C]. CVPR 2022.\n\n[2] Lam T E, et al. CausalChaos! Dataset for Comprehensive Causal Action Question Answering Over Longer Causal Chains Grounded in Dynamic Visual Scenes. NeurIPS 2024.\n\n[3] Chen T, et al. MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning. NeurIPS 2024.\n\n[4] Du Y, et al. Towards Event-oriented Long Video Understanding. ArXiv 2024.\n\n2. **Inference speed.**\n\nThe authors have adopted a form similar to autoregression, and I would like to understand if there is a time overhead in comparing their model's inference speed to that of current mainstream LLMs.\n\n3. **LLM backbone.**\n\nI noticed that the authors used Mistral-7B as the LLM backbone, however, in other comparison methods, Timechat used LLaMA-2, while HawkEye, Momentor, and VTimeLLM used Vicuna. I would like to know if the authors have conducted experiments with LLaMA-2 or Vicuna as the LLM backbone, to ensure that the superior performance is not due to the better LLM backbone but rather the causal modeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)