TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
Authors: Xiangyu Zeng
Kunchang Li
Chenting Wang
Xinhao Li
Tianxiang Jiang
Ziang Yan
Songze Li
Yansong Shi
Zhengrong Yue
Yi Wang
Yali Wang
Yu Qiao
Limin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Long Video Understanding; Temporal Grounding; Multimodal Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to peform detailed video descriptions with the corresponding time stamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively. In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models.

Published as a conference paper at ICLR 2025
TIMESUITE: IMPROVING MLLMS FOR LONG VIDEO UNDERSTANDING VIA GROUNDED TUNING
Xiangyu Zengl.²
Kunchang Li Chenting Wang 6.2 Xinhao Li Tianxiang Jiang
Ziang Yan Songze Li
Yansong Shi Yu Qiao² Limin Wang 1,2.1
Zhengrong Yue Yi Wang?
Yali Wang 3.2
Nanjing University 'Shanghai Al Laboratory Unnivessity of Science and Technology of China
SIAT. Chinese Academy of Sciences
*Zhejiang University
"Shanghai Jiao Tong University FFudan University
8 Shanghaii Innovation Institute
Xi lnggyuZeng2001 @ut look. com.
Imwange r ju. www.nngnnuueu....om
MVBandd
ThineChtt
VideeChuzz Verwher
MesisChet
platts
VidcoMNE
Pemperai Grmaning
Luny Frike QM
At what line the vóeo does the non dribcle the bsalic
What sid the man do ofter throwing enty the booketbul?
giiagllights
assEsting recioning
10th viseo.
The spoko,
the
The given query hapsens betweon secceds
The givan query hoppore. betweee 10.7 sacands.
cwayy the etoried
Egnerhemss
ChoradeedtTA
VideoCtast
TireeTorr
VideoChont
Figure 1: VideoChat-T demonstrates high performance for both long- video question answering and temporal grounding. Our TimeSuite presents a collection new designs to enhance the long video understanding capability of MLLMs. It will implicitly endow the MLLM with ability of correctly attending the visual segments when generating answers, thus relieving the hallucinations.
ABSTRACT
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MILMS for long video understanding, including simple yet efficient framework to pro- cess long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and carefiully-desigeed instruction tuning task to explicitly incorpo- rate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, comprehen- sive grounding-cente instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design new instruction tun- ing task type, called Temporal Grounded Caption, to perform detailed video de- scriptions with the corresponding timestamps prediction. This explicit tempo- ral location prediction will guide MLLM to correctly attend on the visual con- tent when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short- form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME,, respectively. addition, VideoChat-T exhibits ro- bust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models. Our code and dataset are available at https: : github. com http//penGVaaaba wtw.eaaaaaaimmtttt.....
denotes the corresponding author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Multimodal Large Language Models (MLLMs) have demonstrated impressive video understanding performance by following the general human instructions to interpret the visual content (Li et al., 2023b; Zhang et al., 2023; Lin al., 2023a; Jin al., 2024; Wang et al., 2024e). However, these MLLMS still struggle in long video understanding, as long video sequence may contain various dynamic actions and complex temporal relationships, making it difficult for MLLMs to effectively locate the key segments related to questions. When humans watch long videos, their attention is consciously focused on prominent segments, which may occur within a few seconds. NEXT-GQA (Xiao et al., 2024) has also verified the relevance of temporal grounding for accurately answering video QA tasks. Therefore, a natural question arises: Can we enhance long video understanding by using temporal grounding as auxiliary task?
Previously, some works have made progress in temporal grounding task by using general MLLMs. They often enhance the temporal grounding capability of video MLLMs by designing specialized modules and perform specific supervised fine-tuning (Ren et al., 2024; Huang et al., 2024a;b). How- ever, these overly specialized designs significantly impair the general QA capabilities of video MLLMs, resulting in great performance drop on the video QA task (as illustrated by TimeChat in Figure 1). Meanwhile, current research on long video understanding primarily focuses on archi- tecture design, such as long-context LLMs (Liu et al., 2024a) and token compression (Song et al., 2024a). They can only capture holistic semantics in videos without the ability of localizing fine- grained information, leading to poor performance in temporal grounding tasks (as illustrated by MovieChat in Figure 1). So far, it is still challenging to build a video MLLM that is good at both tasks of temporal grounding and long video QA. We argue long video understanding could be as- sisted by explicitly performing temporal grounding, as grounding supervision enables MLLM to establish the detailed correspondance between the visual segments and fine- grained semantics. This fine-grained alignment would guide the MLLM to attend correctly video segments when generating answers and thus relieve the hallucination risk caused by the LLM.
Based on the above analysis, in this paper, we propose TimeSuite, a collection of new designs to improve the long video understanding capability of the existing fhort-form MLLMs, with a focus on incorporating grounding supervision in instruction tuning process. First, to address the high compu- tational cost caused by the excessive number of visual tokens in long videos, we propose a simple Token Shuffle scheme to compress visual tokens, allowing the LLM to process more frame inputs. We also propose TAPE generate adaptive position encodings, enhancing the temporal awareness of visual representations. The proposed structure does not introduce overly complex proprictary designs, which could be efficiently initialized with the parameters of short video MILMs, with- out damaging the original performance of pre-trained MILM. Second, to naturally incorporate the grounding ability into our MLLMs and yet still to preserve its original general QA capability, we design new instruction tuning task, called Temporal Grounded Caption. This new task requires generating detailed segment-level description with corresponding timestamp prediction. Tuning on this new task will not only endow the MLLM with the extra grounding ability but also enhance its original long video QA performance, thanks to the requirement of building correspondence between grounded segments and detailed captions. Finally, we collect a comprehensive grounding-centric instruction tuning dataset for post-training our designed MILMs, which is composed of 349K high- quality annotations covering 9 tasks. Based on this new dataset, we are able to perform grounded tuning with detailed captions on our proposed MLLMs (coined as VideoChat-T),
We verify the effectiveness of TimeSuite design through extensive experiments on the tasks of long video understanding and temporal grounding. VideoChat-T demonstrates a significant improvement in accuracy over baseline for long video understanding, with a 5.6% increase on Egoschema (Man- galam et al., 2023) and a 6.8% increase on VideoMME (Fu et al., 2024). Additionally, VideoChat- exhibits robust zero-shot temporal localization capabilities on Charades-STA (Gao et al., 2017) and QVHighlights (Lei et al., 2021a). a). Our VideoChat-T outperforms the sate-of-the-an temporal grounding MLLM of TimeChat from 50% to 100% for different metrics. After fine-tuning on the training set of temporal grounding benchmarks, the performance of VideoChat-T is par with the state-of-the-ar supervised expert models. The experiments demonstrate that our VideoChat- T is the first end-to-enu end MLLM that is able to perform well on both temporal grounding and general video QA. In particular, we show that grounded tuning with explicit location prediction can facilitate the long video understanding and relieve the hallucination risk.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Video MLLMss With the advancement of open-sourced LLMs (Chiang et al., 2023; Touvron et al., 2023; Jiang al., 2023), video MILMs have emerged by utilizing projection bridges link vision foundation models with LLMs (Li et al., 2023b; 2024b; Zhang et al., 2023; Li et al., 2024a). Limited by the training context length, thought these methods perform well with small number of frame inputs, they meet significant challenges when processing long videos. The longer video length usually implies longer temporal relationships and more redundancies, resulting in the difficulty of extracting key clues (Zhou et al., 2024). Recently, several methods for long video handling have been proposed, such as exploiting long context LLM (Liu et al., 2024a; Zhang et al., 2024b; Xue et al., 2024; Wang et al., 2024d) and token compression (Li et al., 2023d: Song et al., 2024a; Zhang al., 2024a) for enabling more visual inputs and agents for task decomposition or retrieval (Fan et al., 2024; Wang et al., 2024cth). MovieChat (Song et al., 2024a) supports more frames by applying short-term and long-term memory to merge similar visual tokens. Yet, studies in learning objectives for long videos are less explored, making difficult alleviate the frequent hallucination of LLMs in long context reasoning. Our proposed TimeSuite leverages cmmporall-ccnttre tasks to unlock the temporal perception potential MLLMs, anchoring responses to the most relevant video segments.
Temporal Grounding. Temporal grounding a fundamental capability in video understanding, associating semantics to specific clips with corresponding timestamps. Typical expert models (Lei et al., 2021b; Moon et al., 2023a;b; Lin et al., 2023b; Zeng et al., 2024) have been developed by formulating it into timestamp regression from visual inputs and user queries. Most existing video MLLMs fail to address it compared with expert models, while some remedy its temporal grounding by specifically designed architectures and data (Huang et al., 2024a; Wang et al., 2024f; Li et al., 2024c; Wang et al., 2024g; Huang et al., 2024b; Qu et al., 2024). Timechat (Ren et al., 2024) binds visual features of images with timestamps and uses sliding window to handle variable token length. From the perspective of training data, an instruction-tuning dataset TimelT is constructed. Despite impressive improvements in temporal performance, these MILMs still lag behind expert models and compromise general video dialogue capabilities. In this paper, we explore how to enhance the temporal grounding of MLLMs while preserving their original capabilities.
3 METHOD
In this section, we detail the proposed TimeSuite, a new collection of designs for improving short video MLLMs. Specifically, our TimeSuite includes long video modeling framework, high- quality video dataset for grounded tuning, and a carefully-designed instruction tuning task. With this new TimeSuite design, we are able adapt the short-form video MLLM, obtaining significant performance improvements on two types of long video understanding tasks: traditional long video QA and temporal video grounding.
3.1 VIDEOCHAT-T
We first describe the architecture of our proposed long video modeling framework. Specifically, built upon VideoChat2 (Li et al., 2024b), we devise long-video version of VideoChat-T. Our VideoChat-T is composed of video backbone for extracting visual representations, a visual-language connector to compress visual tokens and bridge the visual and languages modalities, L.LM follow human instructions interpret the video content.
The architecture of VideoChat-Tis illustrated in Figure 2. Its workflow has three stages. the first stage, long videos are evenly segmented into clips and the clips are embedded by the Video Encoder and Q-Former (Li et al., 2023a). Then, for compressing visual token number and highlighting crucial ones, token shuffling is employed to merge adjacent tokens, and TAPE is used to add temporal adaptive positional encodings. Finally, the compressed video token sequence is fed to the LLM to generate accurate responses that adhere user requirements.
3

---

Published as a conference paper at ICLR 2025
Visual Encoder
Token Shuffle
Projection
citps
Qormer
LCCO
LLM
LCCC
Krx Query
clips
The 8-th video dip contains Fromes of
TАPE
LoRA
Temporal Instruction
User Instruction
&
You one given long video. Please find the visual event described sentence in
Prezen
The video contains frames sompled of 0, 1, .., seconds.
the videa, defermining its starting and anding times. The format should be: "The event
Trainable
in the short hime and hime'.
Figure 2: Overall Architecture of CideoChat-T. First, long videos are segmented into clips, which then transformed into feature embeddings by video encoder and time-aware Qformer. Next, all visual tokens undergo Token Shuffle to compress overly long tokens, and generate adaptive posi- tional encodings through TAPE. Finally, the long video tokens are concatenated with the user query, serving as the input of LLM,, thereby generating appropriate responses.
3.1.1 BACKBONE DESIGN
Video clip encoding. For the given long video, we perform uniform sampling (Wang et al., 2019) to obtain K xT" frames. We divide these frames into K video segments in chronological order, and sample T frames from each segment. Next, we use the video encoder and its visual-linguistic connector (Q-Former here) to encode each segment into N tokens. After the aforementioned pro- cessing, the entire video is encoded into sequence of visual tokens, denoted by V, E RLXC, where Cg is the dimension of output token by the Q-Former and L KxN is the total number of tokens for the entire video.
Large Language Model. According to previous research, images and visual cues are projected into the same feature space of the LLM. The LLM acts as an interaction interface in the MLLMs, being used process multimodal inputs, parse user instructions, and generate appropriate responses. To afford the processing of long video sequence, we need to design an efficient compression module between the visual encoder and LLMs.
3.1.2 VL-CONNECTOR: TOKEN SHUFFLE
The increased number of sampled frames in long videos leads to larger number of encoded visual tokens, causing a significant rise the computational complexity and memory consumption of LLMs. Therefore, it is crucial to keep the number of visual tokens within acceptable range. Some works have proposed various token compression schemes, such as clustering (Jin et al., 2024) and pooling (Huang et al., 2024b). However, clustering methods often struggle maintain the temporal consistency, and pooling methods usually result in a certain loss of overall performance.
To address this, we propose simple token shuffling compression scheme that ensures the tempo- ral consistency of video tokens before and after compression while avoiding excessive performance loss. Previous methods often used projector achieve dimensional conversion. However, pro- jecting visual encoding vectors from low to high dimensions does not increase information density. Therefore, we propose rearrange multiple visual tokens along the channel dimension. Specifi- cally, for the long video V, = [0g ..., o E LxC,, we concatenate m adjacent tokens along
the channel dimension to obtain the reshaped visual feature V.. Jums U Um Um where each merged token oy is represented as:
R L XmC.
e = Concat ()mmm))m+))) +1 (i (i-m))m+22
jem i 1,2, 1,2,...,
Next, linear projection layer is applied the merged visual feature V generating the visual token sequences V, E IR xx;;
xC: input into the LLM, where C1 represents the token channel di-
4

---

Published as a conference paper at ICLR 2025
mension of the LLM. This scheme effectively reuses the projector of base model by replicating the original linear layer parameters m times along the channel dimension, achieving an initialization equivalent to mean pooling with window length of This design avoids introducing additional randomly initialized parameters that might disturb the original model, thus preserving the its original capabilities. Additionally, compared to directly using pooling, this method offers higher flexibility for fine-tuning to achieve better results (see ablation study, Table 4).
3.1.3 TEMPORAL ADAPTIVE POSITION ENCODING
To bind temporal positional information to visual tokens, we propose an adapter called Temporal Adaptive Position Encoding (TAPE). Inspired by CPVT (Chu et al., 2021), our TAPE uses padding at both ends of the convolution as anchors, and gradually transmits relative positional en- coding information. Without the need to add any special time tokens, TAPE automatically perceive the relative temporal positions of the token sequence and generate temporal embeddings.
Specifically, the long video token sequence Vg is first compressed in the channel dimension by lin- car layer and further compressed in sequence length by a pooling layer. Next, we use a U-Net-like structure composed of one-dimensional depthwise separable convolutions progressively down- sample the sequence, obtaining three one-dimensional temporal feature sequences with different resolutions. Subsequently. convolution with sufficiently long window is applied to the shortest temporal feature sequence, using zero padding at both ends as anchors to encode the relative tem- poral position of each token in the sequence (Chu et al., 2021). Then, we progressively upsample and restore the temporal feature sequences from short to long, using residual connections to retain temporal features at different scales. Finally, the temporal feature sequences are restored to the same length as Vi and aligned in the channel dimension by a linear layer, thereby obtaining the temporal features V, output by the TAPE. For detailed implementation of TAPE, please refer to Appendix A.
Our proposed TAPE offers a plug plug-and-pla and-play module. which could be easily integrated into the net- work structure via residual connections, adding temporal position information to video tokens with- out disrupting the distribution of other trainable parameters. With appropriate training strategies, TAPE effectively preserves the model's generalization capabilities and enhances its temporal sensi- tivity (see ablation study, Table 3), which is important for temporal grounding task.
3.2 TIMEPRO: TEMPORAL GROUNDED INSTRUCTION DATA
Traditional temporal grounding datasets only contain monotonous ground truth, i.e., the start and end times of the target period. This data format performs well in training the classic expert models, but is difficult to unleash the potential of LLMs. Although several temporal grounding-centric datasets have been released for MLLM fine-tuning (Ren et al., 2024; Huang et al., 2024b), they still have deficiencies in data quantity, data quality, and task diversity. Thus, it is necessary to build a more comprehensive temporal dataset designed for the tuning of MILMs.
Based on the criteria of diversity, length, and difficulty, we collect and clean several existing high- quality grounding- centric datasets (Ren et al., 2024; Huang et al., 2024a;b), and create two new datasets, resulting in the TimePro. Compared to previous temporal grounding-centric datasets, TimePro offers a larger volume of data, a broader distribution, and a higher task diversity, facili- tating the learning of more generalizable temporal representations for MLLMs.
As shown in Figure 3(a). TimePro contains 9 task types from 15 datasets that are highly relevant to temporal grounding, containing approximately 349K high-quality temporal grounding annotations. The 9 tasks are specified as follows. Temporal Video Grounding involves identifying the start and end times of video content based on a natural language query (Anne Hendricks et al., 2017; Oncescu et al.. 2021; Zala et al., 2023). Dense Video Captioning requires detecting events within a video and providing corresponding timestamps and descriptions (Krishna et al., 2017; Huang al., 2020; Zhou et al., 2018). Video Summarization focuses on determining key frames or clips in the form of timestamps rather than semantic summaries (Song et al., 2015; Gygli et al., 2014). Step Localization aims to segment and describe important steps in long video (Tang et al., 2019; Zala et al., 2023). Transcribed Speech Generation predicts speech content and its timestamps from visual signals (Zellers et al., 2022). Reasoning Temporal Localization combines timestamps with explanatory answers (Huang et al., 2024b). Multi-format Temporal Grounding includes single-turn and multi-turn dialogues with diverse question types (Huang et al., 2024a). Highlight
5

---

Published as a conference paper at ICLR 2025
Terporal Locakzztion
Video Groundim
wid steme like happort Plste valiriee colk vour on chiing pood
The givn eary hoppen betweee ofor thep andd timer secnds strect cpptore
Example
Speech Generation
48,009
Temporal Grnanded Caption
R0.18
whur Tarta
cartann riee
Please discriee ddtal contenn paraod
TimePro
The givn euery
22795
Maits-errat Tempira Gecunding
intturs rature The shovel curfan right mose tycle bofoww apir cyca begie.
Vidao Caprisning
10, 840
Generation cipture Siaushangg
9,45
Detect an
chack
Highlagh Detectios
Video
Localzation
Tasks of TimePro
(b) Details of Temporal Grounded Caption
Figure 3: (a) The proposed temporal centric iestruction-tuning dataset, TimePro. This dataset con- tains approximately 349K high-quality and strongly temporally correlated data. (b) The proposed Temporal Grounded Caption the-tuning data paradigm. effectively reducing the occurrence of hallucinations. We employ 4-stage processing pipeline to ensure the quality of the generated data.
Detection identifies the most significant moments in video based on a query (Lei al., 202 la). Temporal Grounded Caption uses a brief scene title to output both the time period and fine- tne-grained description for the scene. More detailed information about TimePro is available in the appendix B. It should be noted that Temporal Grounded Caption is our newly-designed task that can help our model establish fine-grained correspondence between visual segment and linguistic description.
3.3 TEMPORAL GROUNDED CAPTION TASK
Some studies have shown that MLLMs often exhibit severe hallucinations when dealing with fine- grained perception tasks (Ji et al., 2023; Huang et al., 2023; Golkar et al., 2023). Since our VideoChat-T directly regresses the timestamps corresponding the text queries using MLLMs, it is more susceptible to hallucinations compared to methods that use external expert models as de- coders (Wu et al.. 2024). By forcing the video MLLMs predict the event occurrence time and simultaneously describe the visual content evidence, we attempt to anchor these queries to the rel- evant time segments within the video, rather than generating hallucinations originating from LLM itself. Based on this analysis, we design the Temporal Grounded Caption task.
The top of Figure 3(b) illustrates the definition of Temporal Grounded Caption. We use brief scene title of the video segment the query, requiring the model to simultaneously respond with the precise start and end times of the video segment and provide a detailed description of that seg- ment. While the content in the scene title may leak into the detailed caption response, most of the missing detailed information must be correctly described by attending the corresponding segment. Moreover, temporal grounding and detailed captioning other, preventing caption model from hallucinations from unrelated visual or linguistic contexts and helping grounding model to regress the timestamp more accurately.
serve regularization task for each
The process for collecting our Temporal Grounded Caption data is described at the bottom of Figure 3(b). In the first stage, we use a detailed caption dataset with timestamps as our data source. We remove data with targer grounding time intervals that are too short or too long and ensure that the scenes in the video are as diverse as possible. In the second stage, we use a LLM to summarize scene titles. To prevent excessive semantics of video segments from being leaked from the query to the MILM,, we try to retain the minimal subset of key features that are sufficient to distinguish the video segments. In the third stage, to avoid overly similar or identical content appearing at different temporal intervals in the video, we perform similarity filtering on the data annoations. Based on the scene titles and video features, we calculate the similarity between different segments of the same video and remove data with excessively high similarity. In the fourth stage, we randomly sample the generated data and manually assess its quality. Based on human feedback, we refine the threshold parameters for data filtering used in the first three stages to yield the final Temporal Grounded Caption dataset. This new dataset plays important role in grounded tuning.

---

Published as a conference paper at ICLR 2025
Charades-STA R
QVHighlight
Method
LLM Size
RAI (.00
(1OU-
mAP
HITA1
MovieChat (Song et sl., 2024a)
7B
8.8
2.9
1.3
11.7
16.1
GroundingGPT (Li al., 2024c)
7B
29.6
11.9
VTimeLLM (Huang al., 2024a)
7B
51.0
27.5
11.4
HawkEye (Wang et al., 2024f)
7B
50.6
31.4
14.5
TimeChat (Ren al., 2024)
7B
32.2
13.4
14.5
23.9
CharVTG (Qu al., 2024)
7B
52.7
33.0
15.9
VideoChar2 a al., 2024h)
78
9.6
3.4
1.4
13.4
18.6
VideoChat-T
7B
69.9 (+60.3)
48.7
(+45.3)
24.0
(+22.6)
26.5
(+13.1)
54.1
(+35.5)
QD-DETR (FTT (Moon al., 2023b)
57.3
32.6
38.9
64.2
UnLoc-La (FT) (Yan 2023)
60.8
38.4
HawkEye (FT) (Wang ul., 20240)
72.5
58.3
28.8
Timechat (FT) (Ren al., 2024)
46.7
23.7
21.7
37,9
VideoChat-T (FT)
7B
79.4
67.1
43.0
27.0
55.3
Performance of VideoChat-T on temporal grounding and highlight detection tasks. (FT) indicates the model fine-tuned on training set of the evaluation benchmark, with the respective text marked in gray. Classic supervised expert models are marked with st.
Table 1:
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Built upon VideoChat2, we use UMT-L (Li et al., 2023c) and Mistral-7B (Jiang et al., 2023) as the video encoder and LLM, respectively. Except for the TAPE, all components are initialized from the pre-trained model of MideoChat2-Mistree For the TAPE, we use random initialization, set the initial values of the final linear layer to zero, and freeze during the first epoch of training. We set the frame count T for each clip to 8, so the number of clips K for long video is equal to the total frame count divided by T. We fine-tune the model for 3 epochs using the TimePro with 349K instances and general QA task dataset with 82K instances. To ensure the stability of model training, use 192-frame input for the first epoch. In the second and third epochs, we unfreeze the TAPE and adjust the model input to 128 frames. All experiments conducted on 16 A100 GPUs.
4.2 PERFORMANCE ON TEMPORAL GROUNDING
We evaluate our method using two commonly used temporal localization tasks, i.e., Temporal Grounding and Highlight Detection. The performance comparison between VideoChat-T and other models is shown in Table 1. Our method's zero-shot performance surpasses all previous LLM- based methods and after fine-tuning, VideoChat-T even exceeds some classic expert models on the temporal grounding task.
This task aims to identify the start and end timestamps of the video content described by the query sentence, using Charades-STA achieves an accuracy of 48.7 in the R@ 1 (IOU=0.5) metric, significantly surpassing the previous state-of-the-aar MILMM method, namely TimeChat, by 16.5 points. Additionally, it outperforms the fine-tuned version of TimeChat on the training set of the evaluation benchmark by 2.0%. Further- more, the performance of VideoChat-T fine-tuned on the evaluation benchmark training set reaches 67.1 R@] at 1oU=0.5, surpassing most state-of-the-art classic supervised expert models.
Temporal Grounding.
the evaluation benchmark. VideoChat-T
Highlight Detection. We QVHighlights the evaluation benchmark. For given query, this task requires outputting all timestamps of highlight moments and their corresponding saliency scores. Since there could be many sparse highlight moments in a video, this task requires finer- grained video understanding at the frame level. VideoChat-T achieves mAP of 26.5, significantly surpassing the previous MILMM method of TimeChat by 13.0 points, and also outperforms its fine- tuned version by 4.8 points. We observe that after fine-tuning on the corresponding training set, VideoChat-T shows almost no performance improvement. This may be due to the bottleneck in language representation of LLMs. The Highlight Detection task requires outputting (timestamp, saliency score) pair for each highlight moment, and video may contain dozens of discrete highlight moments, making it challenging for the model to correctly respond with dozens to hundreds of numbers in a language format. The precise numerical salience score output is very difficult for L.MMs, and VideoChat-TT can only respond well to queries with fewer highlight moments. Due the

---

Published as a conference paper at ICLR 2025
Long Video
Short Video
Method
LLM Size
Egoschema
VideoMME
MVbench
Subsct
Full
subs
w/o subs (Long)
Avg
VideoAgent (Wang et al., 2024c)
GPT-4
60.2
54.1
VideoAgent (Fan et al., 2024)
GPT-4
62.8
TimeChat (Ren et al., 2024)
7B
33.0
30.2
26.1
38.5
LLAMA-Vid (Li al., 2023d)
7B
38.5
41.9
MovicChat (Song ul., 2024a)
53.5
38.2
33.4
55.1
MovieChat+ (Song et al., 2024b)
56.4
Chat-UniVi (Jin al., 2024)
7B
40.6
35.8
VidcoChat2 (Li ct al., 2024b)
7B
63.6
54.4
39.5
33.2
60.4
VideoChat-T
7B
68.4 (+4.8)
60.0
(+5.6)
46.3
(+6.8)
41.9 (+8.7)
59,9 (-0.5)
Table 2: Performance of VideoChat-T and other methods on video question answering tasks. By upgrading VideoChat2 with TimeSuite, VideoChat-T demonstrates significant improvements across multiple long video benchmarks.
specific architectural design, classic supervised expert models have natural advantage in handling such tasks, and VideoChat-IT still has a performance gap compared to expert models.
4.3 PERFORMANCE ON GENERAL VIDEO QA
In addition test the grounding ability of our VideoChat-T, we also want to verify its general video question answering performance. According to mainstream evaluation standards, we use both long video and short video QA to assess the general video understanding capability of VideoChat-T. Table 2 shows the performance of VideoChat-T on the video QA evaluation benchmarks.
Long Video QA. We use Egoschema (Mangalam et al., 2023) and VideoMME (Fu et al., 2024) to evaluate the long video capabilities of VideoChat-T. In conjunction with our proposed architectural improvements, we incremental fine-tune VideoChat2 using only 432K data points. VideoChat-T demonstrates outstanding performance on the Egoschema, achieving accuracy of 68.4% on the test subset and 60.0% on the entire test set. Compared to VideoChat2, VideoChat-T obtains im- provements of 4.8% and 5.6% the subset and the full test set, respectively. Additionally, for the VideoMME benchmark, VideoChat-T achieves an accuracy of 46.3% by solely analyzing the visual content without using subtitles, representing 6.8% improvement over VideoChat2. On the long video data division of VideOMME, VideoChat-T achieves an accuracy of 41.9%, which is an 8.7% improvement compared to VideoChat2. The upgraded VideoChat-T demonstrated significant performance improvements on long video QA benchmarks. This indicates the potential of leverag- ing grounding-centre video tasks to enhance the temporal awareness of MI.LMs, thereby further improving long video understanding capabilities.
Short Video QA. We use MVBench (Li et al., 2024b) to evaluate the general short video under- standing capabilities of VideoChat-T. VideoChat-T achieves an overall average accuracy of 59.9% on MVBench, which is 0.5% decrease compared to VideoChat2. It important to note that achiev- ing minimal performance loss is a challenging task. According to previous experiences in the field of incremental learning (Van de Ven et al., 2022), models inevitably forget old knowledge while learning new knowledge. VideoChat2 is fine-tuned with 2M data, whereas VideoChat-T is fine- tuned with only 432K data, where 349K annotations are temporal grounding centric, resulting in only 0.5% accuracy loss. Previous temporal MILMS like TimeChat (Ren et al., 2024), although achieving strong temporal localization capabilities, yield much weaker general video QA capability, with accuracy of only 38.5% on MVBench. This demonstrates that the design of our TimeSuite enhances new capabilities for the model while still preserving the original general video understand- ing capabilities. For detailed analysis of the performance degradation of MV MVBench, please refer to Appendix F.2.
4.4 QUALITATIVE ANALYSIS
Figure presents a qualitative comparison between our model and other methods. In the example on the left, VideoChat-T is capable of answering more complex long video reasoning questions. Our model accurately identifies the temporal location of the "light cigarette" event and determines the correct key clue "the person in white coat" based on the video content. This leads the inference
8

---

Published as a conference paper at ICLR 2025
00:
Timd
03:15
00:52
Time
Tareporal Gressstigg
Tampural Groundieg
When the indoo the maris it while sut lglningo dipanctic by the plano?
wher the vodoo black men and womeon faag cach other?
The queryy
42, secords
The query hoppenshrtteen 17.6 secoeds
Jung Praker
Long Ville
are stringo in the video hut unough to light digentt??
What the two men in the video outting brielsase car?
fut enough eigendte
made
Hammmbble materis..
putting
The stringg are hot because the man white coutt is loying the prano very fost and pressing the lass veryy hand..
blonay.
User
TirseChat
VidesOut
Figure 4: Qualitative comparison between VideoChat-T T and other methods. VideoChat-T not only possesses temporal fine- -rainrdi perception capabiliitess but also can perform accurate long video reasoning. Green text indicates correct answers, while red text indicates inappropriate answers.
Model
Egnschema Full
VideoMME
CharadesSTA
UVHighlight Haart
Model
Egoschema
VidesMME
Charades-STA
OVHighlights
wio subs
RGI 10L-0.5
Full
w/o subs
RG 10U-0.5
Hi:
VideoChat-T (Ours)
68.0
46.3
48.7
54.1
VideoChar T(Oun)
60.0
46.3
48.7
54.1
wes TAPF
99.1
45.9
47.1
de pooling
59.8
44.8
40.3
47.3
wlo frz
89.0
45.2
52.4
53.7
de clactesin
59.5
45.0
39.8
40.1
wio
57.4
Table 3: Performance results of the ablation study on the TAPE. Here, w/o adapter refers to removing our proposed TAPE, and w/o frz refers not using the training method where the TAPE is frozen during the first epoch.
Table 4: Performance results of the ablation study on the Token Shuffle. Here, r/w refers to replacing Token Shuffle with the other compo- nent, and w/o init refers to removing the effi cient initialization.
that "playing the piano very fast and pressing the keys very hard" are the true reasons. The example on the right demonstrates our model's fine-gru anneggained perception ability. The appearance of "money in the briefcase" " is very brief, and most models easily overlook this detail. Thanks to its strong fine-grained perception ability, our model precisely captures this visual content.
4.5 ABLATION STUDY
Role of TAPE. To verify the performance improvement brought by TAPE, ablation experiments were conducted. Table lists the performance results of the conducted adapter-related ablation experiments. It can be observed that when the TAPE is removed, the model's 's performance on long video understanding and temporal grounding benchmarks decreases. TAPE can adaptively embed positional encodings into video tokens, and the absence of TAPE leads to certain loss in temporal awareness capability. When we unfroze the TAPE in the first epoch, the performance improved on the temporal grounding task but declined the long video QA task. This is because the TAPE is highly suited for tasks with strong temporal dependencies. If unfrozen too early, the model may become biased towards fitting temporal grounding tasks. Freezing the TAPE during the first epoch allows the model to first optimize and learn relatively generalized feature representation, thereby balancing the performance across different tasks.
Effectiveness of Token Shuffle. To verify the effectiveness of token shuffle, we conducted ablation experiments. Table presents the results of these ablation experiments. We compared token shuffle with conventional methods such as pooling and clustering, and also observed the results after re- moving efficient initialization. When we replaced token shuffle with pooling or clustering methods, the model's 'ss performance declined. This is because the efficient initialization of the linear layer in token shuffle makes the initial values of the module equivalent to average pooling, which gradually optimizes better solutions during training. Therefore, On the other hand, clustering often fails to maintain the spatial/temporal consistency of the video, leading to temporal confusion. When removed the efficient initialization of the linear layer, the negative impact of random initialization severely damaged the model's 's original performance.
method is inherently superior to pooling.
Effect of TimePro.
We conducted ablation studies to evaluate the effectiveness of the TimePro data components. As shown in Table 5, by gradually adding subsets of TimePro. we observed the model's performance changes across various temporal grounding-centric instruction-tuning data, As pro-

---

Published as a conference paper at ICLR 2025
Egoschema
VideoMME
Charades-STA @VHighlights
Normal TimelT TGC HD MTG RTT.
Full
wlo subs
R@ IOU=0.5
Hit@
56.6
42.6
8.0
24.4
57.8
43.6
32.2
25.2
58.3
44.0
39.1
33.9
59.8
44.9
41.9
43.8
60.0
45.1
45.8
48.3
60.0
46.3
48.7
54.1
Table Performance results of the ablation study on different components of TimePro. We use 82K normal training data the baseline. TimelT refers to the training data with five task types from Ren al. (2024), TGC refers to Temporal Grounded Caption, HD refers Highlight Detection, MTG refers to Multi-format Temporal Grounding, and RTL refers to Reasoning Temporal Localization.
Eguschema (Tull)
MVBench (Avg)
Charadees-TTi STA (R@1 10U)-055)
403
23.5
46.1
54.5
50
174
34
30
24
P96
F128
F64
F96
P128
+ PideoCht Choooohhuu
"AdeoCCaat
Figure 5: Performance of VideoChat-T with varying input frame numbers. As the number of input frames increases, the performance VideoChat-T shows an upward trend in both long video QA and temporal grounding tasks. Due the over low temporal grounding performance of VideoChat2, its curve is omitted.
gressively added subsets of TimePro, not only did the model's performance on temporal grounding tasks show stable and significant improvement, but also observed noticeable upward trend in performance on long video benchmarks. This to some extent corroborates that temporal grounding centric tasks have positive impact on long video understanding.
Impact of frames. To investigate the impact of input frame count on model performance, we con- ducted an ablation study. Figure 5 illustrates the scalability of our model's performance with respect to input frame count. VideoChat-T demonstrates good stability as the input frame count varies, and its performance long video QA and temporal grounding tasks improves with an increase in frame count. In contrast, the baseline model, VideoChar2, exhibited catastrophic performance degradation when the frame count was significantly increased. As the input frame count increases, the number of visual encoding tokens grows linearly. Excessive visual token input imposes additional computa- tional burden on the temporal modeling of the LLM. TimeSuite mitigates this by employing Token Shuffle to reduce the number of tokens, ensuring the stable operation of the model.
CONCLUSION
In this paper, have introduced TimeSuite, collection of new designs from perspectives of ef- ficient architecture, high-quality data, and new instruction tuning task, to achieve long video un- derstanding by fine-tuning short video MILMS with temporal grounding-centric data. We address the computational challenges of processing long videos by introducing token shuffle to compress visual tokens. We also propose the TAPE for adaptive position encoding, enhancing the temporal awareness of visual representation. Additionally, our designed Temporal Grounded Caption train- ing task ensure MILMs to build correspondence between grounded segments and detailed caption, while the TimePro dataset provide comprehensive instruction tuning data for learning more effective temporal perception capability. Experimental results demonstrate that VideoChat-T significantly im- proves long video understanding, with notable performance gains on Egoschema and VideoMME.. Furthermore, VideoChat-T exhibits strong zero-shot temporal grounding capabilities, significantly outperforming the previous MLLMs temporal grounding. Overall, TimeSuite provides effec- tive designs for short MLLMs to enhance their performance on temporal grounding and long video QA. We hope our TuitS could yield some insights on designing long video MLLMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China (No. 20222D0160900, the Fundamental Research Funds for the Central Universities (No. 0202143801199 Jiangsu Frontier Technology Research and Development Program (No. BF2024076,, and the Collaborative Innova- tion Center of Novel Software Technology and Industrialization.
REFERENCES
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. Proceedings of the IEEE international
conference on computer vision, pp. 1803-5812,22 2017.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint 2330 22966, 2023.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint urXiv:2406.044 04325, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang. and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv22102.10822 2021.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. arXiv preprint arXiv:2403. 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims video analysis. arXiv preprint arXiv:2405.21075 2024.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference computer vision, pp- 5267-5275, 2017.
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A con- tinuous number encoding for large language models. arXiv preprini arXiv:2310.0200 2023.
Michael Gygli, Helmut Grabner, Hayko Riemenschneiderr, and Lue Van Gool. Creating summaries from user videos. Computer Vision-ECcy 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VII 13, pp. 505-520. Springer, 2014.
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Ilm to grasp video moments. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271-14280, 2024a.
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz, Lita: Language instructed temporal-localization assistant. arXiv preprint Stti:2400319944 2024b.
Gabriel Huang. Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video captioning. arXiv preprint arXiv:2011.11766 2020.
Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arX1v:2311.05222, 2023.
11

---

Published as a conference paper at ICLR 2025
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp- 706-715, 2017.
Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-11858, 202
Jie Lei, Tamara L Berg. and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-11858 2021b.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19742. PMLR, 2023a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.063: 2023b.
Kunchang Li, Yali Wang, Yizhuo Li, Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficien video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19948-19960, 2023c.
Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195 -2200, 2024b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17004 2023d.
Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. Groundinggpt: Language enhanced multi-modal grounding model. CoRR, 2024c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023a.
Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2794-2804, 2023b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08226. 2024a.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. arXiv preprint arXiv:2404 003008 2024b.
12

---

Published as a conference paper at ICLR 2025
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arxiv preprint (rXfv:23006.05444, 2023.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very fong-form video language understanding. Advances in Neural Information Process- ing Systems, 36, 2023.
WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae-Pil Heo. dependency calibration in video representation learning for temporal grounding. arXiv preprint arXiv:231 1.08335, 2023a.
Correlation-guider query-
WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Queey-dependent video representation for moment retrieval and highlight detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23023-23033, 2023b.
Andreea- Maria Oncescu, Joao F Henriques, Yang Liu, Andrew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP 2021-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2265-2269. IEEE, 2021.
Mengxue Qu, Xiaodong Chen, Wu Liu, Alicia Li, and Yao Zhao. Chatvig: Video temporal ground- ing via chat with video dialogue large language models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pallern Recognition, pp. 1847-1856, 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14313-14322 2024.
Share. Sharegemini: Scaling up video caption data for multimodal large language models, June 2024. URL https:/ // ttttt//gittttt github..t............ com/ Sharel 4/ ShareGemini.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024a.
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang. Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404. 17176, 2024b.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 5179-51 77775187,22 2015.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng. Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(1 4(12):: 12: 2022.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE Trans. Pattern Anal. Mach. Intell., (1 (11)::7440-275. 2019.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languee language model's 's perception of the world at any resolution. arXiv preprint a4X9 /2/9/, 2024a.
13

---

Published as a conference paper at ICLR 2025
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024b.
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: 1.17. understanding with large language model as agent. arXiv preprint arXiv:2403.10511 2024c.
Long-form video
Xidong Wang. Dingjie Song, Shumian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:240,.02899 2024d.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, al. Internvide22: Scaling video foundation models for multimodal video understanding. arXiv preprint 2rXiv:2403.1337 2024e.
Yueqian Wang, Xiaojun Meng, Jianxin Liang. Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawk- eye: Training video-text Ilms for grounding text in videos. arXiv preprint arXiv:2403. 10228, 2024f.
Yuxuan Wang. Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, Yang Liu, and Zilong Zheng. Efficient temporal extrapolation of multimodal large language models with temporal grounding bridge. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9972-9987, 2024g.
Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jachong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024h.
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406. 08334, 2024.
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13204 13214, 2024.
Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloe: unified framework for video localization tasks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13623-13633, 2023.
En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang, et al. Merlin: Empowering multimodal Ilms with foresight minds. arXiv preprint 2rXiv:2312.005899 2023.
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad, and Mo- hit Bansal. Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23065, 2023.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 16375-16387) 2022.
Xiangyu Zeng, Mingzhu Xu, Yijun Hu, Haoyu Tang, Yupeng Hu, and Liqiang Nie. Adaptive edge- aware semantic interaction network for salient object detection in optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 2023.
Yingsen Zeng, Yujic Zhong, Chengjian Feng. and Lin Ma. Unimd: Towards unifying moment retrieval and temporal action detection. arXiv preprint 2404:2404.04995 2024.
14

---

Published as a conference paper at ICLR 2025
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-uus tuued audio-visual language model for video understanding. arXiv preprint arXiv:23306. 02858, 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.0800 08085, 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. /6852, 2024b.
Junjie Zhou, Yan Shu, Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint WxXiv:2406.04244 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
A IMPLEMENTATION OF TAPE
Algorithm PyTorch snippet of TAPE.
Initialize related package
class
Tempora L'dapter Modcle):
def
init leelf, merge ler, elip ingut din, mid din, cutpot din, serple_rate)
supez init
se A.gPool nn, Adghoolldineree len, stride menge len)
sclt. selt.apsampic ar, tesaapisitssictooooo
sampia_r3te)
iinaar_iapu: ti. (insut din, aid_din) self. iinesroouuut on nn.init.com ww..int.constat.... www.coontttttseeeeeeeee.oo linser. output. weigut. 0)
Linea inneariiidd_din didd cutput_dim)
on wni.i.connttatt......m iselt, ouneerroutput.teee
sclr. Dww.eamppcc.opptheeee.. cer____epraals_eaaee
an.8oquantiel(
an.Convld
mid,ii,, lenieelen221l, stridersagle paldingenerge.len groupa*
mis dim)
m..omviattnie inviamidddda did,din, 11, transposelayertions (mid_dim), an,CBall (),
self, Self.Dooee Doxnsanplo Deptbiiieeppiisee. htt___eettsss__sparaaaeeee Soparassie Ceav2 an.Saquantial
en donvidimid din, mid mid_din, merge lene2+l, stri stride=sergle e rate, paddingemerge..n len, groups* mid dim),
nn. an.oonid inid mid_dim, 11, Transposelaye wow.layyytlammmmmm.m (niimmiin
te = an Segieniial.
an.convid an din, eiig_nunti, striderl, puddingeelignnnm///3
transposelayerlionn mid diml, an.CBall (),
self. Conv2 de..ooenn:ill
nn convidioiddddddc.. din, din, did_din, merge_ienel, stridoni, padding mic (nid did,in, 1I,
trantpeselayerliom. (mid_dim), nn. mn.csull),
wllf Convl Be.
nn conviaidddd din, mid dim, merge_lentl, striden), padeing-nerge_le// groipermaa_dimi nn. mn.onvila din, mid dim, 11, pranspaselayerioon mid dim), an.csall
def fozward(self, smoct_toxenal:
time ad
volee1irear ingut rinput ingutiintt_.ooeee tokens) transpose (ly 2)
time_ad1
seeof.vaPool (tims_ad)
time_ad2
sclf.Donssampcc httpcc_ptththo_eoeeeeeeee Sobarsti__Coapp (tins_ad1)
sime_ad3
self.howmssmplee httpppiiweepthhhee__eeee
ad33
sel(_fe(Lin__ad]
Limsudd
time seir.Conv2 (cin_______
tine_ad2
(eeie ed21 time_ad)
time_adi
(zime_ad1)
time ad net - self.iineer porpat (tine adl transppse (1, 20)
retorn time_ad_out
Algorithm details the implementation process of TAPE in code form. Specifically, the long video token sequence input_tokes tooeens first compressed in the channel dimension by a linear layer to obtain time_ad, and the sequence length is compressed through a pooling layer. Next, structure composed of one-dimensional depthwise separable convolutions to progressively down- sample the sequence, obtaining three one-dimensional temporal feature sequences with different
a U-Net- -ike
15

---

Published as a conference paper at ICLR 2025
time resolutions, namely time_adl, time_ad2, and time_ad3. Subsequently, convolution with sufficienlly long window applied to the shortest temporal feature sequence time_ad3, using zero padding at both ends anchors to encode the relative temporal position of each token in the quence. Then, we progressively upsample the temporal feature sequences from short to long, using residual connections to preserve temporal features different scales. Finally, the temporal feature sequence aime_ad_out is restored to the same length as the video features after token shuffling and aligned in the channel dimension through linear layer.
B INSTRUCTION-TUNING DATA
We fine-tuned VideoChat-T using 432K data, which includes 349K instances from TimePro and 82K instances from normal data. All videos were sampled from existing open-source video datasets, with specific information about the relevant data provided in Table
Set
Task
Sourse
Instance Num
DideMo
32,944
Temporal Video Grounding
QueryD
14,602
HiREST- grroudiggunning
459
ActivityNet-Captinn
10,009
Dense Video Captioning
VITT
5,086
Youcook2
8,700
TVSum
50
Video Summarization
TimePro
SumMe
25
COIN
9,026
Step Localization and Captioning
HiREST-step
459
Transcribed Speech Generation
YT-Temporal
31,190
Reasoning Temporal Localization
ActivityNet-RTL
33,557
Multi-format Temporal Grounding
IntemVVd-VTime
100,000
Highlight Detection
ActivityNet-HL
10,340
Temporal Grounded Caption
CosMo-TGC
93,118
VideoCharGPP
13,303
Conversation
VideoChat
13,884
EgoQA
7.813
Normal
Video QA
MovieChat-QA
808
Reasoning
STAR
45,731
Caption
MovieChat-Caption
808
Table The complete instruction fine-tuning data used for training. We utilized a total of approxi- mately 432K data points, which can be divided into 349K instances of TimePro and 82K instances of regular video data, covering 13 tasks across 21 datasets.
We evaluate the quality of the data from three perspectives: diversity, length, and difficulty. We strive include different datasets for various tasks, and the distribution of videos the datasets is as broad as possible. The length of the videos should be controlled within an appropriate range, as excessively long or short videos may pose challenges for training. Each query should clearly describe the video content of the target time segment and avoid corresponding to multiple time segments in the video. Based on these principles, we have screened and integrated existing high-quality datasets, which significantly contribute to enhancing the model's temporal awareness capabilities.
TimePro encompasses a series of open-source temporal grounding datasets that we have integrated, cleaned, and refined, such as TimelT (Ren al., 2024), ANet-RTL (Huang et al., 2024b), and ViderVVid-Va VTime (Huang et al., 2024a). These high-quality open-source datasets have been experi- mentally validated by us. We also added two new self-made datasets, AN and CosMo-TGC.
Temporal Video Grounding, This task involves providing a natural language query and requires outputting the corresponding video's 's start and end times. The datasets include DiDeMo (Anne Hen-
16

---

Published as a conference paper at ICLR 2025
dricks et al., 2017), QuerYD (Oncescu al., 2021), and HiREST-grounding (Zala et al., 2023), aiming to achieve precise temporal localization during user interaction with natural language.
Dense Video Captioning. This task requires the model to detect series of events occurring in given video and output the corresponding timestamps and coarse-grained descriptions. The datasets for this part include ActivityNet-Captin (Krishna al., 2017), ViTT Huang et al. (2020), and YouCook2 (Zhou et al., 2018), which help the model learn the temporal relationships between dif- ferent events within the video.
The goal of this task is not to summarize at the semantic level of natural language, but determine set of compressed frames or clips in the form of timestamps, repre- senting the most informative content in a given video. Our datasets include TVSum (Song et al., 2015) and SumMe (Gygli et al., 2014), which effectively combine the model's temporal perception capabilities with its semantic content inference abilities.
Video Summarization.
Step Localization and Captioning This task differs from dense video captioning as it is designed to segment and describe the important steps within a long video. We have integrated two datasets, COIN (Tang et al., 2019) and HiREST-step (Zala al., 2023), which can help the model learn the procedural temporal logic relationships of different steps within a single event.
Transcribed Speech Generation. The purpose this task to predict speech content and corre- sponding start and end timestamps based on visual signals in the video. Including the YT-Temporal (Zellers et al., 2022) dataset, this task can be viewed as a weakly supervised event localization and description task.
Reasoning Temporal Localization. The answers to the questions in this task include both times- tamps and explanations. We used the ANet-RTL (Huang et al., 2024b) dataset as training data for this task. By combining temporal localization and reasoning, we can more specifically enhance the model's 's temporal perception capabilities.
Multi-format Temporal Grounding. This task includes both single-turn and multi-turn dialogues, with variety of question types. We use the Intern Vid- VTime (Huang et al., 2024a) dataset for training this task. The broader range of task types and more diverse output formats can effectively enhance the model's temporal generalization capabilities.
Highlight Detection. Unlike video summarization, this task identifies only the most salient mo- ments of video response to natural language query, without covering the entire scope of the original video (Lei et al., 2021a). We used a custom dataset, ANet-HL, derived from temporal local- ization data. We extract video segments between the and end times of the target's appearance and use CLIP to calculate the similarity between each frame's 's scene and the target. This is converted into discrete saliency levels ranging from to 5, intervals of 0.5. This task effectively enhances the model's ss temporal perception capabilities for specific events.
Temporal Grounded Caption. This task involves using scene titles as queries, requiring the model to output both the time segments when the scenes appear and the fine-grained subtitles for those segments. We used our custom dataset, CosMo-TGE TGC. This task format, which combines tempo- ral localization and semantic understanding, can effectively prevent large language models from focusing on irrelevant video segments, thereby improving the quality of the model's responses to questions.
We also used normal data comprising four tasks and six different data sources. These general data help prevent the model from overfitting temporal grounding-relater tasks during training, thereby preserving the model's 's original capabilities.
C COMPUTATIONAL EFFICIENCY
By applying Token Shuffle, we further reduced the computational cost VideoChat-T, giving significant computational advantage over high-performance models like LLaVA-OneVision (Li et al., 2024a) and Qwen2-VL. VL (Wang et al., 2024a). Under the same settings, VideoChat-T uses only 3 tokens per frame, with flops consumption at just 5.1% of LLaVA-One Vision. Its inference time on single A100 is only 0.63 seconds, reaching sal-time response levels, making it highly suitable for applications requiring rapid response, such online video understanding.
17

---

Published as a conference paper at ICLR 2025
Method
Token sum
per
Bops
12M
Inference Fame
A GPU
Charade-STA
OVHighligh
MV WVBench
Eguschem
Full
VidosMEME
Qwen2-VL (Wing et al. 20048)
138
9
Out Of Menuey
15.0
13.0
67.0
66.7
63.3
LLJNA OueNsios (Li at al., 2024g)
196
6
4.95
7.3
14.98
56.7
6011
58.2
ValeoChat-T
35.5T
Table 7: Comparison the computational efficiency and performance of VideoChat-T with other methods. Our approach achieves relatively impressive performance with extremely low computa- tional cost.
In terms of performance, VideoChat-T significantly outperforms LLaVA-OneVision in temporal grounding tasks. It has slight advantage on MVBench; both perform comparably on Egoschema; but VideoChat-T performs worse on VideoMME. Given the substantial savings computational resources with VideoChat-T, we consider the disadvantages some datasets to be acceptable.
Moreover, our model's ability to maintain reasonable performance under high compression ratios suggests that the token embedding spaces of contemporary models may be characterized by con- siderable feature redundancy. This observation presents a promising avenue for future research, efficient techniques for compressing discarding redundant features could substantially reduce computational costs without sacrificing model performance, enabling longer context reasoning.
D DETAILS OF HYPERPARAMETERS
config
epochl
epoch2&3
input frame
192
128
max text length
1536
1024
freeze TAPE
True
False
learning rate
2e-5
1.5e-5
input resolution
224
clip frame
merge lenth
QFormer token (per clip)
96
Iora rank
Iora alpha
32
lora dropout
0.1
batch size (per GPU)
optimizer
Adam W
optimizer momentum
0.9, 0.999
weight decay
0.02 cosine decay
learning rate schedule
Table 8: Hyper-parameter Settings During the Training Process VideoChat-T.
Table lists the hyperparameters used during different epochs of the training process. In the first epoch, we used a larger number of input frames and froze the TAPE. At the beginning of the second epoch, we unfroze the TAPE and fixed the model's input frames 128. Following the settings of VideoChat2, we integrated the lora module into the LLM and applied flash attention to accelerate the training process.
E FULL PERFORMANCES
Model
LLM Avg AS
AP
AA
FA
CA
OR
01
08
MD
AL
ST
AC
MC 8C C0 EN RR CI
VideoChatGPPT (Maz et at, 2023)
7B 32.7
23.5
26.062.0
LMA (Zhing al., 2023)
34.1 27.5
25.0
48.0
VideoChat (Li al, 2023b)
35.5
26.5
40.5
ST-LLM 2024b)
VideoChat2 (Li al., 2024b)
58.0
74,5
MideoChat-T
59.9
68.5
80.5
Table The full performance of VideoChat-T on MVBench. VideoChat-T still demonstrates strong
performance, effectively prevents catastrophic forgetting caused by incremental fine-tuning.
18

---

Published as a conference paper at ICLR 2025
The performance of VideoChat-T MVBench is shown in Table 9. Compared to VideoChat2, VideoChat-T only experienced 0.3% accuracy loss. This indicates that our method effectively pre- serves the capabilities of the base model, preventing catastrophic forgetting caused by incremental fine-tuning. For detailed analysis of the performance degradation of MVBench, please refer to Appendix F.2. For the Action Localization (AL. task, which requires the model determine the coarse-grained temporal position of events, the test accuracy improved from 44.0% to 56.5%. This indirectly confirms that our method significantly enhances the model's temporal awareness capabil- ities.
Overall (%)
Short Video (%)
Medium Video (%)
Long Video (%)
Model
L.M size
w/o subs
w subs
w/o subs
w subs
w/o subs
w subs
w/o subs
w subs
ST-LLM (Liu al., 2024b)
7B
37.9
42.3
45.7
48.4
36.8
41.4
31.3
36.9
Video-LLaVA al., 2023a)
39.9
46.1
36.2
ShareGPT4Viddo (Chen et al., 2024)
8B
39.9
43.6
48.3
53.6
36.3
39.3
35.0
37,9
Chut-UniVi-v1.5 (Jin et al., 2024)
7B
40.6
45.9
45.7
51,2
40.3
44.6
35.8
41.8
Qwen-VL-Chat (Bai al., 2023)
7B
41.1
41.9
46.9
47.3
38.7
40.4
37.8
37,9
ShareGemmni (Share, 2024)
7B
43.2
47.9
49.1
52.8
41.3
47.3
39.1
43.4
VideoChar2 al., 2024b)
7B
39.5
43.8
48.3
37.0
33.2
VideoChat-T
7B
46.3
53.3
59.9
43.8
41.9
53.4
Table 10: The full performance of VideoChat-T on VideoMME. VideoChat-T achieved significant performance improvements, particularly in the long video subset.
The overall performance of our model on VideoMME is presented in Table 10. VideoChat-T achieved significant improvements on both evaluation benchmarks of VideoMME, which include watching videos only and videos with subtitles. The improvements are particularly notable in the long video subset.
F EXTRA ABLATION
F.1 DOMAIN CORRELATION OF DATA
Model
Charades-STA(R@1 1OU=0.5)
MVBench(avg)
VidcoChat-T
48.7
59.9
STAR
47.5 (-1.2)
59.4 (-0.5)
Table 11: The performance changes of the model after removing STAR. Although the video sources of STAR may have some domain correlation with those Charades-STA and MVBench, the per- formance of our model is minimally affected by STAR.
We found that the video sources in the STAR dataset might have some domain correlation with the video sources in MVBench and Charades-STA. Therefore, we removed STAR from the training set while keeping other training settings consistent with the original. The performance on benchmarks where the video sources might have domain correlation is shown in Table 11. The model's accuracy on Charades-STA (R @1 IOU=0.5) decreased by 1.2%, and the average accuracy MVBench de- creased by 0.5%. This indicates that the domain correlation of video sources did not significantly impact performance for model. Notably, after removing STAR, normal data volume was reduced to approximately 36K. This implics that, with sufficiently parameter-efficieet initialization and appropriate training strategies, using only small amount of high-quality normal data is suffi- cient to retain the model's original capabilities.
F.2 DEEPER INVESTIGATION OF THE PERFORMANCE DROP ON MVBENCH
We conducted a deeper investigation into the performance decline on MVBench. Through additional ablation experiments (as shown in Tabel 12). identified two main factors contributing to the performance drop.
Architectural Discrepancy: The original VideoChat2 model was designed to process only 16 frames, leading to a mismatch the learned feature distribution compared the architecture of VideoChat- T. As shown in the first rows of the table, increasing the input frame number for VideoChar2
19

---

Published as a conference paper at ICLR 2025
Method
post ft data
data size
frame num
token num (per frame)
MVBench((AGG)
VideoChar2
16
12
60.4
VideoChar2
128
12
42.1
VideoChut-T T (Common_Jni)) Init]
128
3
25.3
VideoChat T (Ours)
128
3
48.6
VideoChat T (Ours)
TimePro-Normal (Ours)
0.43M
128
3
59.9
VideoChat-T (Ours)
YimeePo-FFulVssdohhha2
2M
128
3
62.9
Table 12: Performance of VideoChat2 and VideoChat-T on MVBench under different settings.
resulted in significant performance drop (from 60.4 to 42.1). When initializing VideoChat-T with VideoChat2, performance was close to random (25.3) due to the newly introduced randomly ini- tialized layers. By applying efficient initialization these new layers, partially recovered the original capabilities of the model, bringing the MVBench performance of the un un-trained VideoChat- T back 48.6, representing improvement of 6.5 compared to the 128-frame VideoChat2. After further fine-tuning, the short-video processing capability of VideoChat-T improved significantly, reaching 59.9.
Fine-tuning Data Discrepancy: We fine-tuned VideoChat-T using only 432K data, significantly less than the 2M non-grounded regular data used for training VideoChat2. The fine-tuning data for VideoChat2 primarily consisted of short videos of around ten seconds, which closely matched the length distribution of the MVBench evaluation videos, playing crucial role in improving MVBench performance. To validate our hypothesis, we conducted additional experiments by training our VideoChat-T model using the TimePro and full VideoChat2 training data. can be observed that VideoChat-T showed slight improvement in performance the MVBench dataset, achieving an accuracy of 62.9, which is an increase 2.5 compared to the original VidcoChat2.
Based on the above, we can conclude the fundamental reasons affecting the model's foundational generalization capabilities. When a model undergoes adjustments, the learned original distribution may not perfectly match the architecture, making the efficient initialization of new layers cial. The features learned from the original dataset might be forgotten due to changes various parameters. Utilizing more comprehensive and diverse dataset for fine-tuning restore and even further enhance performance.
F.3 ASSOCIATION BETWEEN PERFORMANCE AND MODEL DESIGN
Charades-STA
QVHigglight mAP
MVBench
Egoschema
VideoMME
Method
FT Data
I0U0.5
Avg
Full
w/o subs
TimeChat
Valley+V
32.2
14.5
38.5
33.0
30.2
TimeChat
TimcPro-Normal
34.2
16.3
41.6
38.9
33.4
VideoChar-T
NimmPal
48.7
26.5
59.9
60.0
46.3
Table 13: Comparison of other model architectures trained on our dataset with our method, demon- strating the impact of the overall model structure design.
To eliminate the influence of training data and auxiliary tasks, and to more clearly evaluate the asso- ciation between performance and model design, we fine-tuned TimeChat using the full set of fine- tuning data and auxiliary tasks from VideoChat-T. Table 13 presents the performance of TimeChat, fine-tuned with our data, across five datasets. data, shows improvements across all benchmarks. However, its performance still lags significantly behind VideoChat-T. This indicates that an efficient fine tine-tuning architecture design and sugh-quultty, diverse datasets are both essential and complementary.
be observed that TimeChat, fine-tuned with our
F.4 VALIDATION OF TRANSFERABILITY
To verify the robustness of our TimeSuite for other MLLMs, we transferred our method to Llava- One Vision (Li al., 2024a). Table 14 shows the performance changes of Llava-OneVisioo after applying our TimeSuite. It can be seen that when we apply the full set of methods in TimeSuite to Llava-One Visiin,, the model's performance on two different long-video evaluation benchmarks
20

---

Published as a conference paper at ICLR 2025
Charades-STA
QVHighlight mAP
VideoMME
MLVU
MVBench
Method
10L0.5
w/o subs
Avg
Avg
Glava-OneVssion (baseline)
7.3
15.0
58.2
64.7
56.7
Llava-One Vision-T (Ours)
42.5
21.7
61.4
69.4
56.1
Table 14: Performance comparison of Tumee migration to other MLLMs. The application of our method shows a certain improvement in long video comprehension, demonstrating the transferabilitty of our approach.
improves (+3.2 on VideoMME and +4.7 on MLVU), effectively demonstrating the robustness of our TimeSuite for different MLLMs.
F.5 EXPLORATIONS OF DATA CONFIGRATIONS OF TIMEPRO
MVBench
Egoschema Full
VideoMME
Charades-STA
QVHighlight
Method
Avg
w/o subs
10U-0.5
mAP
TImePro615KK-NermallS3 (old version)
60.0
61.0
46.3
45.4
25.7
"inePPr35KKKNNerailSXX (Ours)
59.9
60.0
46.3
48.7
26.5
Table 15: Comparison of different versions of our proposed TimePro. More data does not necessarily lead to higher overall performance, highlighting the importance of data quality.
In the early version of TimePro, we employed datasets comprising 309K Multi-format Temporal Grounding instances, 150K Temporal Grounded Caption instances and other data. Through exten- sive experimentation (as shown in Tabel 15), we discovered that removing low-quality data while retaining high-quality instances could significantly reduce training time without compromising per- formance. Consequently, we pruned these two part datasets to 100K and 93K instances, respectively. The data distribution presented in the paper represents the optimized and relatively balanced config- uration arrived at.
G DISCUSSION
G.1 CAN THE OVERALL PERFORMANCE MLLMS BE ENHANCED BY CONTINUOUSLY INTEGRATING EXPERT TASKS?
By appropriately fine-tuning the Multimodal Large Language Model (MLLM), we have developed a general MILM with powerful zero-shot temporal grounding capabilities. Its performance, after fine- tuning on the training set of evaluation benchmarks, can rival the current state-of th-the-rrt supervised expert models. Based on these results, we can boldly speculate whether it is possible to internalize the capabilities of expert models such as spatial grounding, tracking and detection (Zeng et al., 2023) into the MILLM itself, without using any external expert decoders, to enhance the comprehensive understanding performance of the MLLM and achieve a unified generalist MLLM for multiple tasks.
Merlin (Yu et al., 2023) and VisionLLM (Wang et al., 2024b) have already attempted something similar, but its performance is limited by the reasoning capabilities and language representation bottlenecks of the LLM. There is still significant gap between its performance and that of ex- pert models for various tasks. We observed similar phenomena in our experiments. The temporal grounding task only requires outputting two timestamps. and the task format relatively simple, so our model achieved good results. However, the highlight detection task requires outputting multiple discrete timestamps and their corresponding saliency scores. The model needs to accurately predict dozens of numbers in language form to answer the question correctly. Our model performed well only on data with fewer timestamps. Therefore, how to simplify the complex output format of ex- pert tasks into the language representation of LLMs, to design special processing procedures to simplify complex expert tasks, is a question worth exploring.
Moreover, designing diverse data formats is also crucial for enhancing the expert capabilities of MLLMs. Compared to classic expert models, MLLMs have a natural advantage in task type diversity and can enhance their performance through various different variants tasks of single capability.
21

---

Published as a conference paper at ICLR 2025
For temporal grounding tasks, found that enhancing task diversity has significant effect improving the model's temporal perception generalization ability. We can boldly speculate that there sufficiently diverse training data task types, most tasks with relatively simple output formats can achieve results comparable to expert models through appropriate instruction fine-tuning.
Through the integration of diverse expert tasks and the optimization of language representations, MLLMs can achieve substantial improvements in their overall capabilities. This allows them to effectively comprehend and address complex tasks, rivaling even exceeding the performance of specialized expert models within specific domains. Looking ahead, MLLMs have the potential to evolve into highly versatile Al models, transcending traditional conversational and QA capabilities. They will be equipped to handle wide range of complex expert tasks across various domains, such as vision, language, and reasoning.
G.2 WHY DOES TEMPORAL GROUNDING DATA LEAD TO ACCURACY LOSS IN SHORT-TERM VIDEOS?
We conducted ablation experiments using different combinations of temporal grounding data and regular data. The accuracy of VideoChat-T on MVBench after fine-tuning with various data combi- nations is shown in Table 16.
FT Data
MVBench (AVG)
TimeIT
54.7
Timela+Normal
55.3
Normal TimePro
56.1
57.4
TimcPro+Normal (Ours)
59.9
Table 16: Performance VideoChat-T on MVBench under different fine-tuning data settings.
The diversity of grounding data formats in the past has often been limited, which can lead to over- fitting on Temporal Grounding tasks and cause the model lose its general question-answering capability. We compared the Timel' dataset proposed in TimeChat (Ren et al., 2024) with our TimePro dataset on MVBench. As shown the Table 16, fine-tuning with only TimeIT resulted in the lowest accuracy, and the combined use of TimelT+Normal also performed slightly worse than using Normal alone. This indicates that monotonous grounding data indeed damages the model's original performance (as shown in Figure 1 at the beginning of the paper, TimeChat loses some of its general question-answering capability after fine-tuning, where it outputs localization times for general questions).
In contrast, our TimePro dataset includes diverse data, encompassing 9 different task types from 15 datasets, which helps mitigate the generalization loss caused by homogeneous grounding data types. Additionally, our dataset integrates Grounding with various general tasks. For instance, Grounded Caption requires detailed descriptions of corresponding video segments, while Reasoning Temporal Localization demands the model to reason about questions. This approach significantly enhances the model's 's generalization ability and minimizes the impact on its original capability (e.g., short video accuracy). As demonstrated in the Table 16, the performance of using only TimePro exceeds that of using Normal alone, and the combined use of TimePro and Normal far surpasses all other combinations. This also confirms that our TimePro effectively preserves the model's original performance.
Overall, using a single type of expert task training data can easily lead model overfitting, resulting in significant loss of the model's 's original capabilities. To preserve the model's foundational gener- alization abilities, it is essential various types and distributions, such as text, images, and videos, can further enhance the model's generalization capabilities.
diversified training data. Additionally, incorporating data of
22

---

Published as a conference paper at ICLR 2025
G.3 COULD TRAINING THE MODEL ON BOTH TEMPORAL AND NON-TEMPORAL GROUNDING DATA MITIGATE PERFORMANCE LOSS IN SHORT-TERE VIDEOS?
To address this question, we conducted additional ablation experiments. By training VideoChat-T with different combinations of temporal and non -emporll grounding data, we were able to clearly observe the effects of both types of data on the model's performance. The results of the experiments are shown in the Table 17.
MVBench
VideoMME
Charades-STA
FT Data
Avg
w/o subs
RI @..5
Normal
56.1
42.6
8.0
TimePro
57.4
46.0
45.6
TimePro+Normal (Ours)
59.9
46.3
48.7
Performance comparison of VideoChat-T using different combinations of temporal grounding and non-temporal grounding data.
Table 17:
can be observed that the combined use of TimePro+Normaal for VideoChat-T achieves the highest performance in short video QA, long video QA, and temporal grounding tasks. This not only demon- strates that using both temporal grounding and non-temporal grounding data can reduce performance loss in short vidcos, but also reveals that the effects of temporal and non-temporal grounding data are complementary across various tasks. The distinct differences between temporal grounding and non-temporal grounding tasks can respectively compensate for the model's shorteomings in dif- ferent task perspectives and feature distributions. The simultaneous use of both types of data can effectively enhance the model's overall capabilities.
H
CASE STUDY
H.1 MORE QUALITATIVE ANALYSIS
To further qualitatively analyze our model, we supplemented it with three types of examples. These examples are about long video QA, short video QA, and captioning tasks, all of which include temporal grounding.
More qualitative comparisons about long video QA shown in Figure 6. VideoChat-T effectively handles various questions across different domains. By better perceiving the temporal relationships of different events occurring in long videos, it can more accurately and deeply understand the de- tailed content of the entire video.
More qualitative comparisons about short video QA shown in Figure 7. VideoChat-T effectively retains the original capabilities of the base model. Through parameter-effiiien initialization meth- ods and appropriate training strategies, we minimize the damage the base model's capabilities caused by new architectures and data.
More qualitative comparisons about captioning are shown in Figure 8. Although VideoChat2 de- scribes more local details in some scenarios compared to VideoChat-T, VideoChat-T focuses more on series of temporal events, which aligns better with how humans typically describe videos.
H.2 SHORTCOMINGS
We also conducted qualitative analysis of the shortcomings of VideoChat-T through examples. As shown in Figure 9, VideoChat-T performs poorly on examples with complex logic. In the left example, although VideoChat-T acccurCatt identified the timing of the event, failed to fully explain the motivation behind the man opening the isolation door, which was "to fight the hijackers of the space elevator, seize the controller, and thus save the people in the entire space elevator." example, VideoChat-T correctly identified the event where Mr. Bean reached out to touch his desk mate's 's table, but incorrectly explained the true reason for this action, which was "to cover up the fact that he was copying his desk mate's exam by pretending wipe dust off the desk."
the right
23

---

Published as a conference paper at ICLR 2025
00:13
00:30
Time
00.07
When in the vólco did plasst wear mg the white Josip, No. 15, male sucooulld
When difthe person the viteo the staio?
The given quary happprm benween 20.8 24.2
gi quary hoppars between 22.5 24.2 seconds
from 13.2 to 16. seconce.
From 7 loso seeondd.
Aang Vidie Gut
Long
Acconting on pame, color clothes womentss
The video does ror show what color destres ald the ouing wornen's bashetbel toarn weas
The persor the video grabbed tapa measure.
The women's betballl
dother.
The the video wallad and grabbedd tnalboo.
Croundiag
Trapural Grasedio
the videe the mon open ouptoand?
lody mioophone sings aga not groon beciground?
giver quers hapoens octeroon
glven quary heppenchetueen
From 20.0 23 useends
From 200 to 220 secords.
What dii the man put bowl fis??
what foshof the song the lody singing doo?
The mas put the chosoiate chiss no thelowl
Theleds in the vidco is siging sung.
The man into beaffree
The beaii
Figure 6: More qualitative comparisons in temporal grounding & long video QA.
Teeevent (i (revediy
Time
Prepuirel Groencting
Time
sun aupoar painting
The betwoer 5.7 20.2 sccends.
The auscared
secards.
Pudio
siwn the man use paint videc?
video?
pon.
The man uoes ha trgger paint theiaad.
The mar in the video addeci to pan.
Figure 7: More qualitative comparisons in temporal grounding short video QA.
Due to the preponderance of single-turn, perceptual questions in training data and the lack of multi-step reasoning data with complex logic, our model struggles to handle more challenging sce- narios that demand intricate logical reasoning. To address this limitation, propose constructing data in a chain-of-thought format to guide the model through multi-step reasoning, enabling it to delve deeper into the underlying motivations and causal relationships within video.
24

---

Published as a conference paper at ICLR 2025
Temporal Croundig
Tampandi
The
secends.
secends.
Dcsoribe be thi video decall
Desoribe 1tis video in detall
street rain. Sie wearing black coat and carvie block bog. The curnero to lows her aushe wills drwn the street woman waa tigg sunglasssaas and has long bacc rair. The struntio wet and there are buildings then side
The itarts
woman sailding
rocket ts carviing The sane lite shown in ortic aound the farth. The vitio shows the Earthffom spece.
The video reciet inunching
womon nuntinuesty to walk downche
The video enewn rodeet munching building and then ctiton docding with lecintemniin statoor The rudket is white and with und star The space dtation whos and gray with solar panats. The video Dhreese and there ane no people
eomua seen walling down strect in erain while woa junglasess and
leather jacket. contiruss wallinng down
loolding disanco.
Figure 8: More qualitative comparisons in temporal grounding & captioning.
5o:00
C1:12
01:17
moment pimure the and to priot and monouttige unllormazen door?
poier estenst hand dendinstee?
1100
Piace
hntatiin doo??
Depoopleessts.
brown u. adwuded Fond Nideebnate to Nim
paper
Figure 9: Examples of poor performance by VideoChat-T, While it accurately identifies the time of events, it struggles answer questions that involve more complex logic.
25


Reviews:
[{"review_id": "OSgbkIxwrQ", "reviewer": "Reviewer_61dM", "summary": "The paper introduces TimeSuite, a set of designs to adapt short-form video multimodal large language models (MLLMs) for long video understanding. It includes a new framework for processing long video sequences, a high-quality dataset (TimePro) for grounded tuning, and an instruction tuning task to incorporate grounding supervision. The proposed model, VideoChat-T, uses token shuffling for compression and Temporal Adaptive Position Encoding (TAPE) for enhanced temporal awareness. A new task type, Temporal Grounded Caption, is introduced to improve video descriptions and timestamp prediction. Experimental results show significant improvements on benchmarks and robust zero-shot temporal grounding capabilities.", "questions": "Could training the model on both temporal and non-temporal grounding data mitigate performance loss in short-term videos? Why does temporal grounding data lead to accuracy loss in short-term videos? Despite this, why is short-term accuracy on VideoMME still improved?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is highly complete. The contributions are multiple folds: VideoChat-T framework, TimePro dataset, Temporal Grounded Caption task, and extensive experiments and analysis.\n\n2. Significant improvements on long video processing shown in Table 2.\n\n3. Interesting TAPE, utilizing zero-padding anchors and gradual transmission of relative temporal positional encoding.", "weaknesses": "1. The VideoChat2 baseline shows a weak performance with 39.5 accuracy in VideoMME. It would be more convincing to use a stronger 7B model for evaluation, such as onevision-7B, which achieves 58.2 accuracy.\n\n2. The short-term performance on MVbench drops require a deeper investigation rather than mere explanations.\n\n3. Instead of using implicit TAPE, consider implementing a more explicit solution like 3D RoPE (e.g., Qwen2-VL) and conduct an ablation study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1dVUV3wnTx", "reviewer": "Reviewer_N2GP", "summary": "This paper introduces TimeSuite, a collection of designs focusing on efficient architecture, high-quality data, and a novel instruction-tuning task. Building on TimeSuite, the authors propose a long-video multimodal large language model (MLLM) named VideoChat-T, which demonstrates robust zero-shot temporal grounding capabilities and significantly outperforms existing state-of-the-art MLLMs. After fine-tuning, it performs comparably to traditional supervised expert models.", "questions": "1. Why is the operation of “concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer” referred to as “token shuffle”?\n2. Can you use the same training data (e.g., TimePro) to train other VideoLLMs (e.g., TimeChat) and then compare their performance? This would help eliminate the impact of training data, allowing for a clearer assessment of how performance is associated with model design.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written. The proposed method is well-illustrated and easy to follow.\n2. The contributions include a strong video LLM (VideoChat-T) and a temporal-centric instruction-tuning dataset (TimePro), both of which are substantial advancements in the field.\n3. The authors conduct a thorough evaluation and ablation study to validate the effectiveness of the proposed model.", "weaknesses": "While I do not see major flaws in this paper, I note that the technical novelty is relatively limited. \n\nThere are two main modules in VideoChat-T: (1) the VL-connector with token shuffling, and (2) temporal adaptive position encoding (TAPE). \nHowever, the VL-connector's approach of concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer has been previously utilized in Qwen2-vl (I am unsure who proposed this operation first; please correct me if you know). For TAPE, the design of the position embedding is derived from CPVT. These factors somewhat limit the technical novelty of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgNrqcPVgN", "reviewer": "Reviewer_M5C1", "summary": "The paper introduces TimeSuite, a framework designed to enhance Multimodal Large Language Models (MLLMs) for long video understanding. It proposes a new long-video MLLM called VideoChat-T, featuring token shuffling and Temporal Adaptive Position Encoding (TAPE) for better temporal awareness. TimeSuite includes a specialized instruction tuning dataset, TimePro, comprising 349,000 annotations across nine tasks, with a focus on Temporal Grounded Captioning for accurate video descriptions with timestamps. Experiments show that TimeSuite improves long video comprehension, achieving notable performance gains on benchmarks and demonstrating strong zero-shot grounding capabilities, rivaling traditional supervised models after fine-tuning.", "questions": "See weaknesses", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper propose TimeSuite, a collection of new designs to improve the long video understanding capability of the existing short-form MLLMs", "weaknesses": "- The paper does not adequately compare its results against notable existing works, such as Qwen2-VL and Llava-OneVision. A comprehensive comparison would help contextualize the contributions and highlight any advantages or shortcomings.\n- The dataset used in the experiments, TimePro is highly imbalanced (Figure 3). It is unclear whether the authors have explored different data ratios or configurations. Investigating this aspect could provide insights into the robustness of the proposed method and its adaptability to varying data distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Etbg7C8jt1", "reviewer": "Reviewer_x14u", "summary": "This paper presents TimeSuite, a novel approach designed to enhance the understanding of long-form videos using existing short-form MLLMs. TimeSuite includes an efficient framework for processing long video sequences, a high-quality dataset for grounded tuning, and a new instruction tuning task to integrate grounding supervision in a traditional QA format. Experimental results demonstrate the advantage of  the proposed method.", "questions": "It would be beneficial to include a more in-depth discussion of theories and principles in the manuscript.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method is both systematic and effective. The experiments are comprehensive and detailed, and the results of the method are convincing.", "weaknesses": "Although the proposed method includes specific designs tailored to the problem, the novelty is limited. It is a relatively engineering-focused paper. The systematic description is quite detailed, but it lacks a deeper discussion of the underlying ideas, theories and principles. Additionally, due to the design of auxiliary tasks and datasets, the comparison with baseline methods may be somewhat unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EmltQBpe2S", "reviewer": "Reviewer_epk2", "summary": "The paper introduces TimeSuite, a novel framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) for understanding long videos through grounded tuning. TimeSuite comprises three key components: a token shuffling scheme to compress visual tokens, reducing computational load; Temporal Adaptive Position Encoding (TAPE) to boost the temporal sensitivity of visual representations; and a new instruction tuning task, Temporal Grounded Caption, which integrates timestamp prediction to guide MLLMs to focus on relevant visual content. The authors also present TimePro, a high-quality dataset with 349k grounded annotations across 9 tasks, aimed at improving MLLMs' temporal perception. Experiments demonstrate that TimeSuite significantly improves performance on long video understanding benchmarks like Egoschema and VideoMME, with improvements of 5.6% and 6.8% respectively, and exhibits robust zero-shot temporal grounding abilities, outperforming existing MLLMs. The paper concludes that TimeSuite provides effective designs for MLLMs to enhance their performance on temporal grounding and long video question answering.", "questions": "1. **Token Shuffle Mechanism**: Could the authors elaborate on the decision to use a token shuffling mechanism over other compression techniques, and provide a comparison of its performance against alternatives in terms of temporal consistency and computational efficiency?\n\n2. **Temporal Adaptive Position Encoding (TAPE)**: It would be beneficial if the authors could discuss the potential limitations of TAPE in handling videos with highly variable or complex temporal dynamics, and whether there are any specific domains where TAPE excels or falls short.\n\n3. **Data Diversity and Model Generalization**: How does the choice of datasets for TimePro impact the model's generalization capabilities? Are there any biases introduced by the current dataset composition that could limit the model's applicability to unseen video domains?\n\n4. **Zero-Shot Performance vs. Fine-Tuning**: The paper mentions robust zero-shot capabilities. Could the authors provide insights into why there is a performance gap between zero-shot and fine-tuned models, and what aspects of the model or training process could be improved to bridge this gap?\n\n5. **Long-Video Understanding Limitations**: Are there specific scenarios or types of long videos where VideoChat-T struggles? If so, could the authors suggest potential improvements or future work to address these limitations?\n\n6. **Integration of Expert Model Capabilities**: The authors propose a future direction of integrating expert model capabilities into MLLMs. What are the authors' thoughts on the feasibility of this approach, and what challenges need to be overcome to achieve a unified generalist MLLM?\n\n7. **Complex Output Formats**: For tasks requiring complex outputs, such as highlight detection, how can MLLMs be adapted to handle multiple discrete timestamps and saliency scores effectively? Are there plans to modify the model architecture or training process to better accommodate such tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Originality**: The paper introduces TimeSuite, a novel framework that enhances Multimodal Large Language Models (MLLMs) for long video understanding through grounded tuning. It proposes creative solutions like Token Shuffle and Temporal Adaptive Position Encoding (TAPE), which are innovative approaches to compress visual tokens and enhance temporal awareness in video representations.\n\n- **Quality**: The research demonstrates high quality through rigorous experimentation and ablation studies, showing significant improvements over existing benchmarks like Egoschema and VideoMME. The paper also presents a comprehensive dataset, TimePro, with 349k high-quality annotations.\n\n- **Clarity**: The paper is well-structured and clearly articulated. Complex concepts such as Temporal Grounded Caption and TAPE are explained with clarity, making the paper accessible to readers. Figures and tables are effectively used to convey key results and comparisons.\n\n- **Significance**: The proposed solutions for temporal grounding could have broad implications for video understanding tasks, and the zero-shot capabilities of VideoChat-T are particularly noteworthy, showing the potential to rival supervised expert models.", "weaknesses": "- **Generalization**: The paper primarily focuses on video datasets that are thematically similar. It is unclear how well TimeSuite generalizes to videos with significantly different content or from other domains, such as OpenEQA and CinePile. Expanding the dataset to include more diverse domains could strengthen the paper's claims.\n\n- **Scalability and Efficiency**: The computational requirements for processing long videos are high. The paper could provide more insights into the scalability of TimeSuite and its efficiency, especially when dealing with very long videos or a large number of frames.\n\n- **Longitudinal Performance**: The paper does not address how the model's performance degrades over time or with an increasing amount of video data. It would be beneficial to include studies on the long-term sustainability of the model's performance.\n\n- **Qualitative Analysis**: While quantitative results are provided, a deeper qualitative analysis of the model's outputs, especially in cases of failure, could offer actionable insights into the model's reasoning process and areas for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)