TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

Authors: Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, Limin Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Long Video Understanding; Temporal Grounding; Multimodal Large Language Model

Abstract:
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to peform detailed video descriptions with the corresponding time stamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively. In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models.

Decision: accept (poster)
Reviewer scores: 6, 6, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730115554859

thread_type:
official_review

<thread_object>
creator: reviewer_epk2

<content>
summary:
The paper introduces TimeSuite, a novel framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) for understanding long videos through grounded tuning. TimeSuite comprises three key components: a token shuffling scheme to compress visual tokens, reducing computational load; Temporal Adaptive Position Encoding (TAPE) to boost the temporal sensitivity of visual representations; and a new instruction tuning task, Temporal Grounded Caption, which integrates timestamp prediction to guide MLLMs to focus on relevant visual content. The authors also present TimePro, a high-quality dataset with 349k grounded annotations across 9 tasks, aimed at improving MLLMs' temporal perception. Experiments demonstrate that TimeSuite significantly improves performance on long video understanding benchmarks like Egoschema and VideoMME, with improvements of 5.6% and 6.8% respectively, and exhibits robust zero-shot temporal grounding abilities, outperforming existing MLLMs. The paper concludes that TimeSuite provides effective designs for MLLMs to enhance their performance on temporal grounding and long video question answering.

strengths:
- **Originality**: The paper introduces TimeSuite, a novel framework that enhances Multimodal Large Language Models (MLLMs) for long video understanding through grounded tuning. It proposes creative solutions like Token Shuffle and Temporal Adaptive Position Encoding (TAPE), which are innovative approaches to compress visual tokens and enhance temporal awareness in video representations.

- **Quality**: The research demonstrates high quality through rigorous experimentation and ablation studies, showing significant improvements over existing benchmarks like Egoschema and VideoMME. The paper also presents a comprehensive dataset, TimePro, with 349k high-quality annotations.

- **Clarity**: The paper is well-structured and clearly articulated. Complex concepts such as Temporal Grounded Caption and TAPE are explained with clarity, making the paper accessible to readers. Figures and tables are effectively used to convey key results and comparisons.

- **Significance**: The proposed solutions for temporal grounding could have broad implications for video understanding tasks, and the zero-shot capabilities of VideoChat-T are particularly noteworthy, showing the potential to rival supervised expert models.

weaknesses:
- **Generalization**: The paper primarily focuses on video datasets that are thematically similar. It is unclear how well TimeSuite generalizes to videos with significantly different content or from other domains, such as OpenEQA and CinePile. Expanding the dataset to include more diverse domains could strengthen the paper's claims.

- **Scalability and Efficiency**: The computational requirements for processing long videos are high. The paper could provide more insights into the scalability of TimeSuite and its efficiency, especially when dealing with very long videos or a large number of frames.

- **Longitudinal Performance**: The paper does not address how the model's performance degrades over time or with an increasing amount of video data. It would be beneficial to include studies on the long-term sustainability of the model's performance.

- **Qualitative Analysis**: While quantitative results are provided, a deeper qualitative analysis of the model's outputs, especially in cases of failure, could offer actionable insights into the model's reasoning process and areas for improvement.

questions:
1. **Token Shuffle Mechanism**: Could the authors elaborate on the decision to use a token shuffling mechanism over other compression techniques, and provide a comparison of its performance against alternatives in terms of temporal consistency and computational efficiency?

2. **Temporal Adaptive Position Encoding (TAPE)**: It would be beneficial if the authors could discuss the potential limitations of TAPE in handling videos with highly variable or complex temporal dynamics, and whether there are any specific domains where TAPE excels or falls short.

3. **Data Diversity and Model Generalization**: How does the choice of datasets for TimePro impact the model's generalization capabilities? Are there any biases introduced by the current dataset composition that could limit the model's applicability to unseen video domains?

4. **Zero-Shot Performance vs. Fine-Tuning**: The paper mentions robust zero-shot capabilities. Could the authors provide insights into why there is a performance gap between zero-shot and fine-tuned models, and what aspects of the model or training process could be improved to bridge this gap?

5. **Long-Video Understanding Limitations**: Are there specific scenarios or types of long videos where VideoChat-T struggles? If so, could the authors suggest potential improvements or future work to address these limitations?

6. **Integration of Expert Model Capabilities**: The authors propose a future direction of integrating expert model capabilities into MLLMs. What are the authors' thoughts on the feasibility of this approach, and what challenges need to be overcome to achieve a unified generalist MLLM?

7. **Complex Output Formats**: For tasks requiring complex outputs, such as highlight detection, how can MLLMs be adapted to handle multiple discrete timestamps and saliency scores effectively? Are there plans to modify the model architecture or training process to better accommodate such tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730425589957

thread_type:
official_review

<thread_object>
creator: reviewer_x14u

<content>
summary:
This paper presents TimeSuite, a novel approach designed to enhance the understanding of long-form videos using existing short-form MLLMs. TimeSuite includes an efficient framework for processing long video sequences, a high-quality dataset for grounded tuning, and a new instruction tuning task to integrate grounding supervision in a traditional QA format. Experimental results demonstrate the advantage of  the proposed method.

strengths:
The proposed method is both systematic and effective. The experiments are comprehensive and detailed, and the results of the method are convincing.

weaknesses:
Although the proposed method includes specific designs tailored to the problem, the novelty is limited. It is a relatively engineering-focused paper. The systematic description is quite detailed, but it lacks a deeper discussion of the underlying ideas, theories and principles. Additionally, due to the design of auxiliary tasks and datasets, the comparison with baseline methods may be somewhat unfair.

questions:
It would be beneficial to include a more in-depth discussion of theories and principles in the manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703604456

thread_type:
official_review

<thread_object>
creator: reviewer_m5c1

<content>
summary:
The paper introduces TimeSuite, a framework designed to enhance Multimodal Large Language Models (MLLMs) for long video understanding. It proposes a new long-video MLLM called VideoChat-T, featuring token shuffling and Temporal Adaptive Position Encoding (TAPE) for better temporal awareness. TimeSuite includes a specialized instruction tuning dataset, TimePro, comprising 349,000 annotations across nine tasks, with a focus on Temporal Grounded Captioning for accurate video descriptions with timestamps. Experiments show that TimeSuite improves long video comprehension, achieving notable performance gains on benchmarks and demonstrating strong zero-shot grounding capabilities, rivaling traditional supervised models after fine-tuning.

strengths:
- This paper propose TimeSuite, a collection of new designs to improve the long video understanding capability of the existing short-form MLLMs

weaknesses:
- The paper does not adequately compare its results against notable existing works, such as Qwen2-VL and Llava-OneVision. A comprehensive comparison would help contextualize the contributions and highlight any advantages or shortcomings.
- The dataset used in the experiments, TimePro is highly imbalanced (Figure 3). It is unclear whether the authors have explored different data ratios or configurations. Investigating this aspect could provide insights into the robustness of the proposed method and its adaptability to varying data distributions.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705281546

thread_type:
official_review

<thread_object>
creator: reviewer_n2gp

<content>
summary:
This paper introduces TimeSuite, a collection of designs focusing on efficient architecture, high-quality data, and a novel instruction-tuning task. Building on TimeSuite, the authors propose a long-video multimodal large language model (MLLM) named VideoChat-T, which demonstrates robust zero-shot temporal grounding capabilities and significantly outperforms existing state-of-the-art MLLMs. After fine-tuning, it performs comparably to traditional supervised expert models.

strengths:
1. The paper is well-written. The proposed method is well-illustrated and easy to follow.
2. The contributions include a strong video LLM (VideoChat-T) and a temporal-centric instruction-tuning dataset (TimePro), both of which are substantial advancements in the field.
3. The authors conduct a thorough evaluation and ablation study to validate the effectiveness of the proposed model.

weaknesses:
While I do not see major flaws in this paper, I note that the technical novelty is relatively limited. 

There are two main modules in VideoChat-T: (1) the VL-connector with token shuffling, and (2) temporal adaptive position encoding (TAPE). 
However, the VL-connector's approach of concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer has been previously utilized in Qwen2-vl (I am unsure who proposed this operation first; please correct me if you know). For TAPE, the design of the position embedding is derived from CPVT. These factors somewhat limit the technical novelty of this work.

questions:
1. Why is the operation of “concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer” referred to as “token shuffle”?
2. Can you use the same training data (e.g., TimePro) to train other VideoLLMs (e.g., TimeChat) and then compare their performance? This would help eliminate the impact of training data, allowing for a clearer assessment of how performance is associated with model design.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731259484589

thread_type:
official_review

<thread_object>
creator: reviewer_61dm

<content>
summary:
The paper introduces TimeSuite, a set of designs to adapt short-form video multimodal large language models (MLLMs) for long video understanding. It includes a new framework for processing long video sequences, a high-quality dataset (TimePro) for grounded tuning, and an instruction tuning task to incorporate grounding supervision. The proposed model, VideoChat-T, uses token shuffling for compression and Temporal Adaptive Position Encoding (TAPE) for enhanced temporal awareness. A new task type, Temporal Grounded Caption, is introduced to improve video descriptions and timestamp prediction. Experimental results show significant improvements on benchmarks and robust zero-shot temporal grounding capabilities.

strengths:
1. This work is highly complete. The contributions are multiple folds: VideoChat-T framework, TimePro dataset, Temporal Grounded Caption task, and extensive experiments and analysis.

2. Significant improvements on long video processing shown in Table 2.

3. Interesting TAPE, utilizing zero-padding anchors and gradual transmission of relative temporal positional encoding.

weaknesses:
1. The VideoChat2 baseline shows a weak performance with 39.5 accuracy in VideoMME. It would be more convincing to use a stronger 7B model for evaluation, such as onevision-7B, which achieves 58.2 accuracy.

2. The short-term performance on MVbench drops require a deeper investigation rather than mere explanations.

3. Instead of using implicit TAPE, consider implementing a more explicit solution like 3D RoPE (e.g., Qwen2-VL) and conduct an ablation study.

questions:
Could training the model on both temporal and non-temporal grounding data mitigate performance loss in short-term videos? Why does temporal grounding data lead to accuracy loss in short-term videos? Despite this, why is short-term accuracy on VideoMME still improved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731884863874

thread_type:
public_comment

<thread_object>
creator: ~shraman_pramanick1

<content>
title:
Questions about Potential Data Leakage for Charades-STA Temporal Grounding Evaluation

comment:
Dear Authors,

Thank you for the excellent work. After reading the paper, I have a question regarding data leakage from the evaluation set during the instruction fine-tuning. I will request the authors to provide detailed clarification on this critical issue.

In Table 6, the authors mention using all 45,731 samples from the STAR [1] dataset. However, the STAR dataset is constructed with the videos from Charades' training and test set. After a quick check, I see that almost half of the videos from the Charades-STA test set appear in the STAR training set. Moreover, as the authors use **all 45,731 samples** from STAR, there is no filtering to alleviate data leakage. Hence, this leakage can be hugely responsible for the strong reported results on the Charades-STA zero-shot setting, where the authors report a massive 8.1 R1@0.7 points improvement over the previous SOTA. Though used in a different task setup, the same videos appearing in fine-tuning and evaluation would hugely affect the zero-shot performance.

It would be helpful to clarify the issue. Thanks again for your work!

[1] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. NeurIPS, 2021.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732776082349

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Comment by Authors

comment:
We sincerely thank all the reviewers for their constructive comments. We are delighted to receive positive feedback such as "highly complete" (61dM), "well-structured" (N2GP, epk2), "easy to follow" (N2GP), "contributions are multiple folds" (61dM), "innovative approaches" (epk2), "substantial advancements" (N2GP), "significant improvements" (61dM, epk2), "systematic and effective" (x14u), "experiments are comprehensive and detailed" (N2GP, x14u), "convincing" (x14u), "broad implications" (epk2).


We have carefully addressed all the concerns raised by the reviewers point by point in the individual response section. In addition, we have supplemented our work with extensive and detailed ablation experiments to further investigate specific characteristics of our proposed method. Most of these additional experiments are included in the appendix section of the revised manuscript.


**Below is a summary of the eight new experiments added during the rebuttal phase:**

1. Comparison of the computational efficiency of VideoChat-T with other MLLMs. This shows that our model has a relatively good performance while consuming very minimal computational resources. (61dM, N2GP, M5C1, x14u, epk2)

2. Validation experiments of TimeSuite's transferability. By applying our full set of methods to LLaVA-OneVision, we observed an improvement in long video benchmarks. (61dM, M5C1)

3. In-depth exploration of the performance degradation of MVBench. This reveals two main reasons for the performance loss on MVBench. (61dM, epk2)

4. Comparison of TAPE with other position embedding methods. This verifies the superiority of TAPE for our model structure. (61dM)

5. Performance comparison of using temporal grounding and non-temporal grounding data both simultaneously and separately. This verifies the complementary functionality of temporal grounding and non-temporal grounding data. (61dM)

6. Analysis experiments on the reasons for performance loss caused by temporal grounding data. This elucidates the advantages of our data over previous temporal grounding data. (61dM, epk2)

7. Training TimeChat with our data and comparing it with our VideoChat-T. This further verifies the effectiveness of our model structure by eliminating training data interference. (N2GP, x14u)

8. Exploratory experiments on the data ratio of TimePro. This verifies the rationality of the current version of TimePro's composition. (M5C1)


We are deeply grateful to all the reviewers for their insightful comments and invaluable contributions, which have greatly improved the quality of our manuscript. If there are any remaining questions or concerns, we would be most delighted to engage in further discussions and address them to the best of our ability.
</content>
</thread_object>

</thread>


Metareview:
metareview:
After discussion, this submission received 3 positive scores and a negative score. The reviewer who assigned the negative score has no response to the author’s rebuttal. After reading the paper, the review comments and the rebuttal, the AC thinks that the major concerns about technical novelty of the proposed approach and experimental evaluation were comprehensively solved. The AC supports the acceptance of this submission as a poster paper in ICLR.