TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

## Abstract

Abstract Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs.
This paper proposes TimeSuite , a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format.
Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation.
Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations.
Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to perform detailed video descriptions with the corresponding timestamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs.
Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively.
In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs.
After fine-tuning, it performs on par with the traditional supervised expert models.

## 1 Introduction

Multimodal Large Language Models (MLLMs) have demonstrated impressive video understanding performance by following the general human instructions to interpret the visual content (Li et al., ; Zhang et al., ; Lin et al., ; Jin et al., ; Wang et al., ).
However, these MLLMs still struggle in long video understanding, as a long video sequence may contain various dynamic actions and complex temporal relationships, making it difficult for MLLMs to effectively locate the key segments related to questions.
When humans watch long videos, their attention is consciously focused on prominent segments,
which may occur within a few seconds.
NExT-GQA (Xiao et al., ) has also verified the relevance of temporal grounding for accurately answering video QA tasks.
Therefore, a natural question arises: Can we enhance long video understanding by using temporal grounding as a auxiliary task?

Previously, some works have made progress in temporal grounding task by using general MLLMs. They often enhance the temporal grounding capability of video MLLMs by designing specialized modules and perform specific supervised fine-tuning (Ren et al., ; Huang et al., ; ). However, these overly specialized designs significantly impair the general QA capabilities of video MLLMs, resulting in great performance drop on the video QA task (as illustrated by TimeChat in Figure 1). Meanwhile, current research on long video understanding primarily focuses on architecture design, such as long-context LLMs (Liu et al., ) and token compression (Song et al., ). They can only capture holistic semantics in videos without the ability of localizing fine-grained information, leading to poor performance in temporal grounding tasks (as illustrated by MovieChat in Figure 1). So far, it is still challenging to build a video MLLM that is good at both tasks of temporal grounding and long video QA. We argue long video understanding could be assisted by explicitly performing temporal grounding, as grounding supervision enables MLLM to establish the detailed correspondance between the visual segments and fine-grained semantics. This fine-grained alignment would guide the MLLM to attend correctly video segments when generating answers and thus relieve the hallucination risk caused by the LLM.

Based on the above analysis, in this paper, we propose TimeSuite, a collection of new designs to improve the long video understanding capability of the existing short-form MLLMs, with a focus on incorporating grounding supervision in instruction tuning process. First, to address the high computational cost caused by the excessive number of visual tokens in long videos, we propose a simple Token Shuffle scheme to compress visual tokens, allowing the LLM to process more frame inputs. We also propose TAPE to generate adaptive position encodings, enhancing the temporal awareness of visual representations. The proposed structure does not introduce overly complex proprietary designs, which could be efficiently initialized with the parameters of short video MLLMs, without damaging the original performance of pre-trained MLLM.
Second, to naturally incorporate the grounding ability into our MLLMs and yet still to preserve its original general QA capability, we design a new instruction tuning task, called Temporal Grounded Caption. This new task requires generating detailed segment-level description with corresponding timestamp prediction. Tuning on this new task will not only endow the MLLM with the extra grounding ability but also enhance its original long video QA performance, thanks to the requirement of building correspondence between grounded segments and detailed captions.
Finally, we collect a comprehensive grounding-centric instruction tuning dataset for post-training our designed MLLMs, which is composed of 349K high-quality annotations covering 9 tasks. Based on this new dataset, we are able to perform grounded tuning with detailed captions on our proposed MLLMs (coined as VideoChat-T).

We verify the effectiveness of TimeSuite design through extensive experiments on the tasks of long video understanding and temporal grounding. VideoChat-T demonstrates a significant improvement in accuracy over baseline for long video understanding, with a 5.6% increase on Egoschema (Mangalam et al., ) and a 6.8% increase on VideoMME (Fu et al., ). Additionally, VideoChat-T exhibits robust zero-shot temporal localization capabilities on Charades-STA (Gao et al., ) and QVHighlights (Lei et al., ). Our VideoChat-T outperforms the state-of-the-art temporal grounding MLLM of TimeChat from 50% to 100% for different metrics. After fine-tuning on the training set of temporal grounding benchmarks, the performance of VideoChat-T is on par with the state-of-the-art supervised expert models. The experiments demonstrate that our VideoChat-T is the first end-to-end MLLM that is able to perform well on both temporal grounding and general video QA. In particular, we show that grounded tuning with explicit location prediction can facilitate the long video understanding and relieve the hallucination risk.

## 2 Related Work

### 2.1 Video MLLMs

With the advancement of LLMs (Chiang et al., ; Touvron et al., ; Jiang et al., ), video MLLMs have emerged by utilizing projection bridges to link vision encoders with LLMs (Li et al., ; Zhang et al., ; Wang et al., ). Limited by the training context length, thought these methods perform well with a small number of frame inputs, they meet significant challenges when processing long videos.
The longer video length usually implies longer temporal relationships and more redundancies, resulting in the difficulty of extracting key clues (Zhou et al., ). Recently, several methods for long video handling have been proposed, such as exploiting long context LLM (Liu et al., ; Zhang et al., ; Xue et al., ; Wang et al., ) and token compression (Li et al., ; Song et al., ; Zhang et al., ) for enabling more visual inputs and agents for task decomposition or retrieval (Fan et al., ; Wang et al., ; ). MovieChat (Song et al., ) supports more frames by applying short-term and long-term memory to merge similar visual tokens. Yet, studies in learning objectives for long videos are less explored, making it difficult to alleviate the frequent hallucination of LLMs in long context reasoning.
Our proposed TimeSuite leverages temporally-centric tasks to unlock the temporal perception potential of MLLMs, anchoring responses to the most relevant video segments.

### 2.2 Temporal Grounding

Temporal grounding is a fundamental capability in video understanding, associating video semantics to clips with corresponding timestamps. Typical expert models have been developed to it (Lei et al., ; Moon et al., ; ; Lin et al., ; Zeng et al., ) by formulating it into a timestamp regression from visual inputs and user queries. Most existing video MLLMs fail to address it compared with expert models, while some remedy its temporal grounding by specifically designed architectures and data (Huang et al., ; Wang et al., ; Li et al., ; Huang et al., ; Qu et al., ). Timechat (Ren et al., ) binds visual features of images with timestamps and uses a sliding window to handle variable token length. From the perspective of training data, an instruction-tuning dataset TimeIT is constructed. Despite impressive improvements in temporal performance, these MLLMs still lag behind expert models and compromise general video dialogue capabilities. In this paper, we explore how to enhance the temporal grounding of MLLMs while preserving their original capabilities.

## 3 Method

In this section, we detail the proposed TimeSuite, a new collection of designs for improving short video MLLMs.
Specifically, our TimeSuite includes a long video modeling framework, a high-quality video dataset for grounded tuning, and a carefully-designed instruction tuning task. With this new TimeSuite design, we are able to adapt the short-form video MLLM, obtaining significant performance improvements on two types of long video understanding tasks: traditional long video QA and temporal video grounding.

### 3.1 VideoChat-T

Figure: Figure 2: Overall Architecture of VideoChat-T. First, long videos are segmented into clips, which are then transformed into feature embeddings by video encoder and time-aware Qformer. Next, all visual tokens undergo Token Shuffle to compress overly long tokens, and generate adaptive positional encodings through TAPE. Finally, the long video tokens are concatenated with the user query, serving as the input of LLM, thereby generating appropriate responses.
Refer to caption: x2.png

We first describe the architecture of our proposed long video modeling framework. Specifically, built upon VideoChat2 (Li et al., ), we devise long-video version of VideoChat-T. Our VideoChat-T is composed of a video backbone for extracting visual representations, a visual-language connector to compress visual tokens and bridge the visual and languages modalities, a LLM to follow human instructions to interpret the video content.

The architecture of VideoChat-T is illustrated in Figure 2. Its workflow has three stages. In the first stage, long videos are evenly segmented into clips and the clips are embedded by the Video Encoder and Q-Former (Li et al., ). Then, for compressing visual token number and highlighting crucial ones, token shuffling is employed to merge adjacent tokens, and TAPE is used to add temporal adaptive positional encodings. Finally, the compressed video token sequence is fed to the LLM to generate accurate responses that adhere to user requirements.

#### 3.1.1 Backbone design

Video clip encoding. For the given long video, we perform uniform sampling (Wang et al., ) to obtain $K{\times}T$ frames. We divide these frames into $K$ video segments in chronological order, and sample $T$ frames from each segment. Next, we use the video encoder and its visual-linguistic connector (Q-Former here) to encode each segment into $N$ tokens.
After the aforementioned processing, the entire video is encoded into a sequence of visual tokens, denoted by $\mathrm{\mathbf{V}}_{q}\;{\in}\;\mathbb{R}^{L{\times}C_{q}}$, where $C_{q}$ is the dimension of output token by the Q-Former and $L=K{\times}N$ is the total number of tokens for the entire video.

Large Language Model. According to previous research, images and visual cues are projected into the same feature space of the LLM. The LLM acts as an interaction interface in the MLLMs, being used to process multimodal inputs, parse user instructions, and generate appropriate responses. To afford the processing of long video sequence, we need to design an efficient compression module between the visual encoder and LLMs.

#### 3.1.2 VL-Connector: Token shuffle

The increased number of sampled frames in long videos leads to a larger number of encoded visual tokens, causing a significant rise in the computational complexity and memory consumption of LLMs.
Therefore, it is crucial to keep the number of visual tokens within an acceptable range.
Some works have proposed various token compression schemes, such as clustering (Jin et al., ) and pooling (Huang et al., ). However, clustering methods often struggle to maintain the temporal consistency, and pooling methods usually result in a certain loss of overall performance.

To address this, we propose a simple token shuffling compression scheme that ensures the temporal consistency of video tokens before and after compression while avoiding excessive performance loss.
Previous methods often used a projector to achieve dimensional conversion. However, projecting visual encoding vectors from low to high dimensions does not increase information density. Therefore, we propose to rearrange multiple visual tokens along the channel dimension. Specifically, for the long video $\mathrm{\mathbf{V}}_{q}=[v_{q}^{1},v_{q}^{2},...,v_{q}^{L}]\in\mathbb{R}^{L
\times C_{q}}$, we concatenate $m$ adjacent tokens along the channel dimension to obtain the reshaped visual feature $\mathrm{\mathbf{V}}_{m}=[v_{m}^{1},v_{m}^{2},...,v_{m}^{\frac{L}{m}}]\in
\mathbb{R}^{{\frac{L}{m}}\times mC_{q}}$, where each merged token $v_{m}^{i}$ is represented as:

$$ $v_{m}^{i}=\mathrm{Concat}(v_{q}^{(i-1)*m+1},v_{q}^{(i-1)*m+2},...,v_{q}^{i*m}) \;\;\;\forall\;i=1,2,...,\scriptstyle\frac{L}{m}\;.$ $$

Next, a linear projection layer is applied to the merged visual feature $\mathrm{\mathbf{V}}_{m}$, generating the visual token sequences $\mathrm{\mathbf{V}}_{l}\in\mathbb{R}^{\frac{L}{m}\times C_{l}}$ as input into the LLM, where $C_{l}$ represents the token channel dimension of the LLM.
This scheme effectively reuses the projector of base model by replicating the original linear layer parameters $m$ times along the channel dimension, achieving an initialization equivalent to mean pooling with a window length of $m$. This design avoids introducing additional randomly initialized parameters that might disturb the original model, thus preserving the its original capabilities. Additionally, compared to directly using pooling, this method offers higher flexibility for fine-tuning to achieve better results (see ablation study, Table 4).

#### 3.1.3 Temporal Adaptive Position Encoding

To bind temporal positional information to visual tokens, we propose an adapter called Temporal Adaptive Position Encoding (TAPE).
Inspired by CPVT (Chu et al., ), our TAPE uses zero padding at both ends of the convolution as anchors, and gradually transmits relative positional encoding information.
Without the need to add any special time tokens, TAPE can automatically perceive the relative temporal positions of the token sequence and generate temporal embeddings.

Specifically, the long video token sequence $\mathrm{\mathbf{V}}_{q}$ is first compressed in the channel dimension by a linear layer and further compressed in sequence length by a pooling layer. Next, we use a U-Net-like structure composed of one-dimensional depthwise separable convolutions to progressively downsample the sequence, obtaining three one-dimensional temporal feature sequences with different resolutions. Subsequently, a convolution with a sufficiently long window is applied to the shortest temporal feature sequence, using zero padding at both ends as anchors to encode the relative temporal position of each token in the sequence (Chu et al., ). Then, we progressively upsample and restore the temporal feature sequences from short to long, using residual connections to retain temporal features at different scales. Finally, the temporal feature sequences are restored to the same length as $\mathrm{\mathbf{V}}_{l}$ and aligned in the channel dimension by a linear layer, thereby obtaining the temporal features $\mathrm{\mathbf{V}}_{t}$ output by the TAPE. For detailed implementation of TAPE, please refer to Appendix A.

Our proposed TAPE offers a plug-and-play module, which could be easily integrated into the network structure via residual connections, adding temporal position information to video tokens without disrupting the distribution of other trainable parameters. With appropriate training strategies, TAPE effectively preserves the model’s generalization capabilities and enhances its temporal sensitivity (see ablation study, Table 4), which is important for temporal grounding task.

### 3.2 TimePro: Temporal Grounded Instruction data

Traditional temporal grounding datasets only contain monotonous ground truth, i.e., the start and end times of the target period. This data format performs well in training the classic expert models, but is difficult to unleash the potential of LLMs. Although several temporal grounding-centric datasets have been released for MLLM fine-tuning (Ren et al., ; Huang et al., ), they still have deficiencies in data quantity, data quality, and task diversity. Thus, it is necessary to build a more comprehensive temporal dataset designed for the tuning of MLLMs.

Based on the criteria of diversity, length, and difficulty, we collect and clean several existing high-quality grounding-centric datasets (Ren et al., ; Huang et al., ; ), and create two new datasets, resulting in the TimePro. Compared to previous temporal grounding-centric datasets, TimePro offers a larger volume of data, a broader distribution, and a higher task diversity, facilitating the learning of more generalizable temporal representations for MLLMs.

As shown in Figure 3(a), TimePro contains 9 task types from 15 datasets that are highly relevant to temporal grounding, containing approximately 349K high-quality temporal grounding annotations. The 9 tasks are specified as follows.
Temporal Video Grounding involves identifying the start and end times of video content based on a natural language query (Anne Hendricks et al., ; Oncescu et al., ; Zala et al., ).
Dense Video Captioning requires detecting events within a video and providing corresponding timestamps and descriptions (Krishna et al., ; Huang et al., ; Zhou et al., ).
Video Summarization focuses on determining key frames or clips in the form of timestamps rather than semantic summaries (Song et al., ; Gygli et al., ).
Step Localization aims to segment and describe important steps in a long video (Tang et al., ; Zala et al., ).
Transcribed Speech Generation predicts speech content and its timestamps from visual signals (Zellers et al., ).
Reasoning Temporal Localization combines timestamps with explanatory answers (Huang et al., ).
Multi-format Temporal Grounding includes single-turn and multi-turn dialogues with diverse question types (Huang et al., ).
Highlight Detection identifies the most significant moments in a video based on a query (Lei et al., ).
Temporal Grounded Caption uses a brief scene title to output both the time period and fine-grained description for the scene.
More detailed information about TimePro is available in the appendix C. It should be noted that Temporal Grounded Caption is our newly-designed task that can help our model to establish fine-grained correspondence between visual segment and linguistic description.

Figure: Figure 3: (a) The proposed temporal centric instruction-tuning dataset, TimePro. This dataset contains approximately 349K high-quality and strongly temporally correlated data. (b) The proposed Temporal Grounded Caption fine-tuning data paradigm. It effectively reducing the occurrence of hallucinations. We employ a 4-stage processing pipeline to ensure the quality of the generated data.
Refer to caption: x3.png

### 3.3 Temporal Grounded Caption Task

Some studies have shown that MLLMs often exhibit severe hallucinations when dealing with fine-grained perception tasks (Ji et al., ; Huang et al., ; Golkar et al., ). Since our VideoChat-T directly regresses the timestamps corresponding to the text queries using MLLMs, it is more susceptible to hallucinations compared to methods that use external expert models as decoders (Wu et al., ). By forcing the video MLLMs to predict the event occurrence time and simultaneously describe the visual content evidence, we attempt to anchor these queries to the relevant time segments within the video, rather than generating hallucinations originating from LLM itself. Based on this analysis, we design the Temporal Grounded Caption task.

The top of Figure 3(b) illustrates the definition of Temporal Grounded Caption. We use a brief scene title of the video segment as the query, requiring the model to simultaneously respond with the precise start and end times of the video segment and provide a detailed description of that segment. While the content in the scene title may leak into the detailed caption response, most of the missing detailed information must be correctly described by attending the corresponding segment. Moreover, temporal grounding and detailed captioning can serve as regularization task for each other, preventing caption model from hallucinations from unrelated visual or linguistic contexts and helping grounding model to regress the timestamp more accurately.

The process for collecting our Temporal Grounded Caption data is described at the bottom of Figure 3(b). In the first stage, we use a detailed caption dataset with timestamps as our data source. We remove data with target grounding time intervals that are too short or too long and ensure that the scenes in the video are as diverse as possible. In the second stage, we use a LLM to summarize scene titles. To prevent excessive semantics of video segments from being leaked from the query to the MLLM, we try to retain the minimal subset of key features that are sufficient to distinguish the video segments. In the third stage, to avoid overly similar or identical content appearing at different temporal intervals in the video, we perform similarity filtering on the data annoations. Based on the scene titles and video features, we calculate the similarity between different segments of the same video and remove data with excessively high similarity. In the fourth stage, we randomly sample the generated data and manually assess its quality. Based on human feedback, we refine the threshold parameters for data filtering used in the first three stages to yield the final Temporal Grounded Caption dataset. This new dataset plays an important role in our grounded tuning.

## 4 Experiments

### 4.1 Implementation Details

Built upon VideoChat2, we use UMT-L (Li et al., ) and Mistral-7B (Jiang et al., ) as the video encoder and LLM, respectively. Except for the TAPE, all components are initialized from the pre-trained model of VideoChat2-Mistral. For the TAPE, we use random initialization, set the initial values of the final linear layer to zero, and freeze it during the first epoch of training. We set the frame count $T$ for each clip to 8, so the number of clips $K$ for a long video is equal to the total frame count divided by $T$. We fine-tune the model for 3 epochs using the TimePro with 349K instances and a general QA task dataset with 82K instances. To ensure the stability of model training, we use 192-frame input for the first epoch. In the second and third epochs, we unfreeze the TAPE and adjust the model input to 128 frames. All experiments are conducted on 16 A100 GPUs.

### 4.2 Performance on temporal Grounding

We evaluate our method using two commonly used temporal localization tasks, i.e., Temporal Grounding and Highlight Detection. The performance comparison between VideoChat-T and other models is shown in Table 1. Our method’s zero-shot performance surpasses all previous LLM-based methods and after fine-tuning, VideoChat-T even exceeds some classic expert models on the temporal grounding task.

**Table 1: Performance of VideoChat-T on temporal grounding and highlight detection tasks. (FT) indicates the model fine-tuned on training set of the evaluation benchmark, with the respective text marked in gray. Classic supervised expert models are marked with ※.**
| Method | LLM Size | Charades-STA | QVHighlight |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $R@1_{(IOU=0.3)}$ | $R@1_{(IOU=0.5)}$ | $R@1_{(IOU=0.7)}$ | $mAP$ | $HIT@1$ |  |  |
| MovieChat (Song et al., ) | 7B | 8.8 | 2.9 | 1.3 | 11.7 | 16.1 |
| GroundingGPT (Li et al., ) | 7B | - | 29.6 | 11.9 | - | - |
| VTimeLLM (Huang et al., ) | 7B | 51.0 | 27.5 | 11.4 | - | - |
| HawkEye (Wang et al., ) | 7B | 50.6 | 31.4 | 14.5 | - | - |
| TimeChat (Ren et al., ) | 7B | - | 32.2 | 13.4 | 14.5 | 23.9 |
| ChatVTG (Qu et al., ) | 7B | 52.7 | 33.0 | 15.9 | - | - |
| VideoChat2 (Li et al., ) | 7B | 9.6 | 3.4 | 1.4 | 13.4 | 18.6 |
| VideoChat-T | 7B | 69.9 (+60.3) | 48.7 (+45.3) | 24.0 (+22.6) | 26.5 (+13.1) | 54.1 (+35.5) |
| QD-DETR※ (FT) (Moon et al., ) | - | - | 57.3 | 32.6 | 38.9 | 64.2 |
| UnLoc-L※ (FT) (Yan et al., ) | - | - | 60.8 | 38.4 | - | - |
| HawkEye (FT) (Wang et al., ) | 7B | 72.5 | 58.3 | 28.8 | - | - |
| Timechat (FT) (Ren et al., ) | 7B | - | 46.7 | 23.7 | 21.7 | 37.9 |
| VideoChat-T (FT) | 7B | 79.4 | 67.1 | 43.0 | 27.0 | 55.3 |

Temporal Grounding.
This task aims to identify the start and end timestamps of the video content described by the query sentence, using Charades-STA as the evaluation benchmark. VideoChat-T achieves an accuracy of 48.7 in the R@1 (IOU=0.5) metric, significantly surpassing the previous state-of-the-art MLLM method, namely TimeChat, by 16.5 points. Additionally, it outperforms the fine-tuned version of TimeChat on the training set of the evaluation benchmark by 2.0%. Furthermore, the performance of VideoChat-T fine-tuned on the evaluation benchmark training set reaches 67.1 R@1 at IoU=0.5, surpassing most state-of-the-art classic supervised expert models.

Highlight Detection.
We use QVHighlights as the evaluation benchmark. For a given query, this task requires outputting all timestamps of highlight moments and their corresponding saliency scores. Since there could be many sparse highlight moments in a video, this task requires finer-grained video understanding at the frame level. VideoChat-T achieves mAP of 26.5, significantly surpassing the previous MLLM method of TimeChat by 13.0 points, and also outperforms its fine-tuned version by 4.8 points. We observe that after fine-tuning on the corresponding training set, VideoChat-T shows almost no performance improvement. This may be due to the bottleneck in language representation of LLMs. The Highlight Detection task requires outputting a (timestamp, saliency score) pair for each highlight moment, and a video may contain dozens of discrete highlight moments, making it challenging for the model to correctly respond with dozens to hundreds of numbers in a language format. The precise numerical salience score output is very difficult for LLMs, and VideoChat-T can only respond well to queries with fewer highlight moments. Due to the specific architectural design, classic supervised expert models have a natural advantage in handling such tasks, and VideoChat-T still has a performance gap compared to expert models.

### 4.3 Performance on General Video QA

In addition to test the grounding ability of our VideoChat-T, we also want to verify its general video question answering performance.
According to mainstream evaluation standards, we use both long video and short video QA to assess the general video understanding capability of VideoChat-T. Table 2 shows the performance of VideoChat-T on the video QA evaluation benchmarks.

**Table 2: Performance of VideoChat-T and other methods on video question answering tasks. By upgrading VideoChat2 with TimeSuite, VideoChat-T demonstrates significant improvements across multiple long video benchmarks.**
|  |  | Long Video | Short Video |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | Egoschema | VideoMME | MVbench |  |  |
| Method | LLM Size | Subset | Full | w/o subs | w/o subs (Long) | Avg |
| VideoAgent (Wang et al., ) | GPT-4 | 60.2 | 54.1 | - | - | - |
| VideoAgent (Fan et al., ) | GPT-4 | 62.8 | - | - | - | - |
| TimeChat (Ren et al., ) | 7B | - | 33.0 | 30.2 | 26.1 | 38.5 |
| LLAMA-Vid (Li et al., ) | 7B | - | 38.5 | - | - | 41.9 |
| MovieChat (Song et al., ) | 7B | - | 53.5 | 38.2 | 33.4 | 55.1 |
| MovieChat+ (Song et al., ) | 7B | - | 56.4 | - | - | - |
| Chat-UniVi (Jin et al., ) | 7B | - | - | 40.6 | 35.8 | - |
| VideoChat2 (Li et al., ) | 7B | 63.6 | 54.4 | 39.5 | 33.2 | 60.4 |
| VideoChat-T | 7B | 68.4 (+4.8) | 60.0 (+5.6) | 46.3 (+6.8) | 41.9 (+8.7) | 59.9 (-0.5) |

Long Video QA.
We use Egoschema (Mangalam et al., ) and VideoMME (Fu et al., ) to evaluate the long video capabilities of VideoChat-T. In conjunction with our proposed architectural improvements, we incremental fine-tune VideoChat2 using only 432K data points.
VideoChat-T demonstrates outstanding performance on the Egoschema, achieving an accuracy of 68.4% on the test subset and 60.0% on the entire test set. Compared to VideoChat2, VideoChat-T obtains improvements of 4.8% and 5.6% on the subset and the full test set, respectively.
Additionally, for the VideoMME benchmark, VideoChat-T achieves an accuracy of 46.3% by solely analyzing the visual content without using subtitles, representing a 6.8% improvement over VideoChat2. On the long video data division of VideoMME, VideoChat-T achieves an accuracy of 41.9%, which is an 8.7% improvement compared to VideoChat2.
The upgraded VideoChat-T demonstrated significant performance improvements on long video QA benchmarks. This indicates the potential of leveraging grounding-centric video tasks to enhance the temporal awareness of MLLMs, thereby further improving long video understanding capabilities.

Short Video QA.
We use MVBench (Li et al., ) to evaluate the general short video understanding capabilities of VideoChat-T.
VideoChat-T achieves an overall average accuracy of 59.9% on MVBench, which is a 0.5% decrease compared to VideoChat2. It is important to note that achieving minimal performance loss is a challenging task. According to previous experiences in the field of incremental learning (Van de Ven et al., ), models inevitably forget old knowledge while learning new knowledge, necessitating specific research to deal with catastrophic forgetting. VideoChat2 is fine-tuned with 2M data, whereas VideoChat-T is fine-tuned with only 432K data, where 349K annotations are temporal grounding centric, resulting in only a 0.5% accuracy loss. Previous temporal MLLMs like TimeChat (Ren et al., ), although achieving strong temporal localization capabilities, yield much weaker general video QA capability, with an accuracy of only 38.5% on MVBench. This demonstrates that the design of our TimeSuite enhances new capabilities for the model while still preserving the original general video understanding capabilities.

### 4.4 Qualitative analysis

Figure: Figure 4: Qualitative comparison between VideoChat-T and other methods. VideoChat-T not only possesses temporal fine-grained perception capabilities but also can perform accurate long video reasoning. Green text indicates correct answers, while red text indicates inappropriate answers.
Refer to caption: x4.png

Figure 4 presents a qualitative comparison between our model and other methods. In the example on the left, VideoChat-T is capable of answering more complex long video reasoning questions. Our model accurately identifies the temporal location of the “light a cigarette” event and determines the correct key clue “the person in a white coat” based on the video content. This leads to the inference that “playing the piano very fast and pressing the keys very hard” are the true reasons. The example on the right demonstrates our model’s fine-grained perception ability. The appearance of “money in the briefcase” is very brief, and most models easily overlook this detail. Thanks to its strong fine-grained perception ability, our model precisely captures this visual content.

### 4.5 Ablation Study

**Table 3: Performance results of the ablation study on the TAPE. Here, w/o adapter refers to removing our proposed TAPE, and w/o frz refers to not using the training method where the TAPE is frozen during the first epoch.**
| Model | Egoschema | VideoMME | Charades-STA | QVHighlights |
| --- | --- | --- | --- | --- |
| Full | w/o subs | R@1 IOU=0.5 | Hit@1 |  |
| VideoChat-T (Ours) | 60.0 | 46.3 | 48.7 | 54.1 |
| w/o TAPE | 59.1 | 45.9 | 47.1 | 50.4 |
| w/o frz | 59.0 | 45.2 | 52.4 | 53.7 |

Role of TAPE. To verify the performance improvement brought by TAPE, ablation experiments were conducted. Table 4 lists the performance results of the conducted adapter-related ablation experiments.
It can be observed that when the TAPE is removed, the model’s performance on long video understanding and temporal grounding benchmarks decreases.
TAPE can adaptively embed positional encodings into video tokens, and the absence of TAPE leads to a certain loss in temporal awareness capability.
When we unfroze the TAPE in the first epoch, the performance improved on the temporal grounding task but declined on the long video QA task. This is because the TAPE is highly suited for tasks with strong temporal dependencies. If unfrozen too early, the model may become biased towards fitting temporal grounding tasks. Freezing the TAPE during the first epoch allows the model to first optimize and learn a relatively generalized feature representation, thereby balancing the performance across different tasks.

Effectiveness of Token Shuffle. To verify the effectiveness of token shuffle, we conducted ablation experiments. Table 4 presents the results of these ablation experiments. We compared token shuffle with conventional methods such as pooling and clustering, and also observed the results after removing efficient initialization.
When we replaced token shuffle with pooling or clustering methods, the model’s performance declined. This is because the efficient initialization of the linear layer in token shuffle makes the initial values of the module equivalent to average pooling, which gradually optimizes better solutions during training. Therefore, our method is inherently superior to pooling. On the other hand, clustering often fails to maintain the spatial/temporal consistency of the video, leading to temporal confusion. When we removed the efficient initialization of the linear layer, the negative impact of random initialization severely damaged the model’s original performance.

**Table 5: Performance results of the ablation study on different components of TimePro. We use 82K normal training data as the baseline. TimeIT refers to the training data with five task types from Ren et al. (), TGC refers to Temporal Grounded Caption, HD refers to Highlight Detection, MTG refers to Multi-format Temporal Grounding, and RTL refers to Reasoning Temporal Localization.**
| Normal | TimeIT | TGC | HD | MTG | RTL | Egoschema | VideoMME | Charades-STA | QVHighlights |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Full | w/o subs | R@1 IOU=0.5 | Hit@1 |  |  |  |  |  |  |
| ✓ |  |  |  |  |  | 56.6 | 42.6 | 8.0 | 24.4 |
| ✓ | ✓ |  |  |  |  | 57.8 | 43.6 | 32.2 | 25.2 |
| ✓ | ✓ | ✓ |  |  |  | 58.3 | 44.0 | 39.1 | 33.9 |
| ✓ | ✓ | ✓ | ✓ |  |  | 59.8 | 44.9 | 41.9 | 43.8 |
| ✓ | ✓ | ✓ | ✓ | ✓ |  | 60.0 | 45.1 | 45.8 | 48.3 |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 60.0 | 46.3 | 48.7 | 54.1 |

Effect of TimePro. We conducted ablation studies to evaluate the effectiveness of the TimePro data components. As shown in Table 5, by gradually adding subsets of TimePro, we observed the model’s performance changes across various temporal grounding-centric instruction-tuning data.
As we progressively added subsets of TimePro, not only did the model’s performance on temporal grounding tasks show a stable and significant improvement, but we also observed a noticeable upward trend in performance on long video benchmarks. This to some extent corroborates that temporal grounding centric tasks have a positive impact on long video understanding.

Figure: Figure 5: Performance of VideoChat-T with varying input frame numbers. As the number of input frames increases, the performance of VideoChat-T shows an upward trend in both long video QA and temporal grounding tasks. Due to the over low temporal grounding performance of VideoChat2, its curve is omitted.
Refer to caption: x5.png

Impact of frames.
To investigate the impact of input frame count on model performance, we conducted an ablation study. Figure 5 illustrates the scalability of our model’s performance with respect to input frame count. VideoChat-T demonstrates good stability as the input frame count varies, and its performance in long video QA and temporal grounding tasks improves with an increase in frame count. In contrast, the baseline model, VideoChat2, exhibited catastrophic performance degradation when the frame count was significantly increased. As the input frame count increases, the number of visual encoding tokens grows linearly. Excessive visual token input imposes an additional computational burden on the temporal modeling of the LLM. TimeSuite mitigates this by employing Token Shuffle to reduce the number of tokens, ensuring the stable operation of the model.

## 5 Conclusion

In this paper, we have introduced TimeSuite, a collection of new designs from perspectives of efficient architecture, high-quality data, and new instruction tuning task, to achieve long video understanding by fine-tuning short video MLLMs with temporal grounding-centric data.
We address the computational challenges of processing long videos by introducing token shuffle to compress visual tokens. We also propose the TAPE for adaptive position encoding, enhancing the temporal awareness of visual representation. Additionally, our designed Temporal Grounded Caption training task ensure MLLMs to build correspondence between grounded segments and detailed caption, while the TimePro dataset provide comprehensive instruction tuning data for learning more effective temporal perception capability. Experimental results demonstrate that VideoChat-T significantly improves long video understanding, with notable performance gains on Egoschema and VideoMME. Furthermore, VideoChat-T exhibits strong zero-shot temporal grounding capabilities, significantly outperforming the previous MLLMs on temporal grounding. Overall, our TimeSuite provides effective designs for short MLLMs to enhance their performance on temporal grounding and long video QA. We hope our TimeSuite could yield some insights on designing long video MLLMs.

## Appendix A Implementation of TAPE

Figure: Algorithm 1 PyTorch snippet of TAPE.

Algorithm 1 details the implementation process of TAPE in code form. Specifically, the long video token sequence $input_tokens$ is first compressed in the channel dimension by a linear layer to obtain $time_ad$, and the sequence length is compressed through a pooling layer. Next, we use a U-Net-like structure composed of one-dimensional depthwise separable convolutions to progressively downsample the sequence, obtaining three one-dimensional temporal feature sequences with different time resolutions, namely $time_ad1$, $time_ad2$, and $time_ad3$. Subsequently, a convolution with a sufficiently long window is applied to the shortest temporal feature sequence $time_ad3$, using zero padding at both ends as anchors to encode the relative temporal position of each token in the sequence. Then, we progressively upsample the temporal feature sequences from short to long, using residual connections to preserve temporal features at different scales. Finally, the temporal feature sequence $time_ad_out$ is restored to the same length as the video features after token shuffling and aligned in the channel dimension through a linear layer.

## Appendix C Instruction-tuning data

We fine-tuned VideoChat-T using 432K data, which includes 349K instances from TimePro and 82K instances from normal data. All videos were sampled from existing open-source video datasets, with specific information about the relevant data provided in Table 6.

**Table 6: The complete instruction fine-tuning data used for training. We utilized a total of approximately 432K data points, which can be divided into 349K instances of TimePro and 82K instances of regular video data, covering 13 tasks across 21 datasets.**
| Set | Task | Sourse | Instance Num |
| --- | --- | --- | --- |
| TimePro | Temporal Video Grounding | DideMo | 32,944 |
| QueryD | 14,602 |  |  |
| HiREST-grounding | 459 |  |  |
| Dense Video Captioning | ActivityNet-Captions | 10,009 |  |
| ViTT | 5,086 |  |  |
| Youcook2 | 8,700 |  |  |
| Video Summarization | TVSum | 50 |  |
| SumMe | 25 |  |  |
| Step Localization and Captioning | COIN | 9,026 |  |
| HiREST-step | 459 |  |  |
| Transcribed Speech Generation | YT-Temporal | 31,190 |  |
| Reasoning Temporal Localization | ActivityNet-RTL | 33,557 |  |
| Multi-format Temporal Grounding | InternVid-VTime | 100,000 |  |
| Highlight Detection | ActivityNet-HL | 10,340 |  |
| Temporal Grounded Caption | CosMo-TGC | 93,118 |  |
| Normal | Conversation | VideoChatGPT | 13,303 |
| VideoChat | 13,884 |  |  |
| Video QA | EgoQA | 7,813 |  |
| MovieChat-QA | 808 |  |  |
| Reasoning | STAR | 45,731 |  |
| Caption | MovieChat-Caption | 808 |  |
|  |  |  |  |

We evaluate the quality of the data from three perspectives: diversity, length, and difficulty. We strive to include different datasets for various tasks, and the distribution of videos in the datasets is as broad as possible. The length of the videos should be controlled within an appropriate range, as excessively long or short videos may pose challenges for training. Each query should clearly describe the video content of the target time segment and avoid corresponding to multiple time segments in the video. Based on these principles, we have screened and integrated existing high-quality datasets, which significantly contribute to enhancing the model’s temporal awareness capabilities.

TimePro encompasses a series of open-source temporal grounding datasets that we have integrated, cleaned, and refined, such as TimeIT (Ren et al., ), ANet-RTL (Huang et al., ), and InternVid-VTime (Huang et al., ). These high-quality open-source datasets have been experimentally validated by us. We also added two new self-made datasets, ANet-HL and CosMo-TGC.

Temporal Video Grounding.
This task involves providing a natural language query and requires outputting the corresponding video’s start and end times. The datasets include DiDeMo (Anne Hendricks et al., ), QuerYD (Oncescu et al., ), and HiREST-grounding (Zala et al., ), aiming to achieve precise temporal localization during user interaction with natural language.

Dense Video Captioning.
This task requires the model to detect a series of events occurring in a given video and output the corresponding timestamps and coarse-grained descriptions. The datasets for this part include ActivityNet-Caption (Krishna et al., ), ViTT Huang et al. (), and YouCook2 (Zhou et al., ), which help the model learn the temporal relationships between different events within the video.

Video Summarization.
The goal of this task is not to summarize at the semantic level of natural language, but to determine a set of compressed frames or clips in the form of timestamps, representing the most informative content in a given video. Our datasets include TVSum (Song et al., ) and SumMe (Gygli et al., ), which effectively combine the model’s temporal perception capabilities with its semantic content inference abilities.

Step Localization and Captioning.
This task differs from dense video captioning as it is designed to segment and describe the important steps within a long video. We have integrated two datasets, COIN (Tang et al., ) and HiREST-step (Zala et al., ), which can help the model learn the procedural temporal logic relationships of different steps within a single event.

Transcribed Speech Generation.
The purpose of this task is to predict speech content and its corresponding start and end timestamps based on visual signals in the video. Including the YT-Temporal (Zellers et al., ) dataset, this task can be viewed as a weakly supervised event localization and description task.

Reasoning Temporal Localization.
The answers to the questions in this task include both timestamps and explanations. We used the ANet-RTL (Huang et al., ) dataset as training data for this task. By combining temporal localization and reasoning, we can more specifically enhance the model’s temporal perception capabilities.

Multi-format Temporal Grounding.
This task includes both single-turn and multi-turn dialogues, with a variety of question types. We use the InternVid-VTime (Huang et al., ) dataset for training this task. The broader range of task types and more diverse output formats can effectively enhance the model’s temporal generalization capabilities.

Highlight Detection.
Unlike video summarization, this task identifies only the most salient moments of a video in response to a natural language query, without covering the entire scope of the original video (Lei et al., ). We used a custom dataset, ANet-HL, derived from temporal localization data. We extract video segments between the start and end times of the target’s appearance and use CLIP to calculate the similarity between each frame’s scene and the target. This is converted into discrete saliency levels ranging from 1 to 5, at intervals of 0.5. This task effectively enhances the model’s temporal perception capabilities for specific events.

Temporal Grounded Caption.
This task involves using scene titles as queries, requiring the model to output both the time segments when the scenes appear and the fine-grained subtitles for those segments. We used our custom dataset, CosMo-TGC. This task format, which combines temporal localization and semantic understanding, can effectively prevent large language models from focusing on irrelevant video segments, thereby improving the quality of the model’s responses to questions.

We also used normal data comprising four tasks and six different data sources. These general data help prevent the model from overfitting to temporal grounding-related tasks during training, thereby preserving the model’s original capabilities.

## Appendix E Details of Hyperparameters

**Table 7: Hyper-parameter Settings During the Training Process of VideoChat-T**
| config | epoch1 | epoch2&3 |
| --- | --- | --- |
| input frame | 192 | 128 |
| max text length | 1536 | 1024 |
| freeze Tape adapter | True | False |
| learning rate | 2e-5 | 1.5e-5 |
| input resolution | 224 |  |
| clip frame | 8 |  |
| merge lenth | 4 |  |
| QFormer token (per clip) | 96 |  |
| lora rank | 16 |  |
| lora alpha | 32 |  |
| lora dropout | 0.1 |  |
| batch size (per GPU) | 2 |  |
| optimizer | AdamW |  |
| optimizer momentum | 0.9, 0.999 |  |
| weight decay | 0.02 |  |
| learning rate schedule | cosine decay |  |
|  |  |  |

Table 7 lists the hyperparameters used during different epochs of the training process. In the first epoch, we used a larger number of input frames and froze the Tape adapter. At the beginning of the second epoch, we unfroze the Tape adapter and fixed the model’s input frames to 128. Following the settings of VideoChat2, we integrated the lora module into the LLM and applied flash attention to accelerate the training process.

## Appendix G Full performances

**Table 8: The full performance of VideoChat-T on MVBench. VideoChat-T still demonstrates strong performance, effectively prevents catastrophic forgetting caused by incremental fine-tuning.**
| Model | LLM | Avg | AS | AP | AA | FA | UA | OE | OI | OS | MD | AL | ST | AC | MC | MA | SC | FP | CO | EN | ER | CI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VideoChatGPT (Maaz et al., ) | 7B | 32.7 | 23.5 | 26.0 | 62.0 | 22.5 | 26.5 | 54.0 | 28.0 | 40.0 | 23.0 | 20.0 | 31.0 | 30.5 | 25.5 | 39.5 | 48.5 | 29.0 | 33.0 | 29.5 | 26.0 | 35.5 |
| VideoLLaMA (Zhang et al., ) | 7B | 34.1 | 27.5 | 25.5 | 51.0 | 29.0 | 39.0 | 48.0 | 40.5 | 38.0 | 22.5 | 22.5 | 43.0 | 34.0 | 22.5 | 32.5 | 45.5 | 32.5 | 40.0 | 30.0 | 21.0 | 37.0 |
| VideoChat (Li et al., ) | 7B | 35.5 | 33.5 | 26.5 | 56.0 | 33.5 | 40.5 | 53.0 | 40.5 | 30.0 | 25.5 | 27.0 | 48.5 | 35.0 | 20.5 | 42.5 | 46.0 | 26.5 | 41.0 | 23.5 | 23.5 | 36.0 |
| ST-LLM (Liu et al., ) | 7B | 54.9 | 66.0 | 53.5 | 84.0 | 44.0 | 58.5 | 80.5 | 73.5 | 38.5 | 42.5 | 31.0 | 86.5 | 36.5 | 56.5 | 78.5 | 43.0 | 44.5 | 46.5 | 34.5 | 41.5 | 58.5 |
| VideoChat2 (Li et al., ) | 7B | 60.4 | 75.5 | 58.0 | 83.5 | 50.5 | 60.5 | 87.5 | 74.5 | 45.0 | 47.5 | 44.0 | 82.5 | 37.0 | 64.5 | 87.5 | 51.0 | 66.5 | 47.0 | 35.0 | 37.0 | 72.5 |
| VideoChat-T | 7B | 59.9 | 83.5 | 68.5 | 80.5 | 44.0 | 61.0 | 71.0 | 84.0 | 35.5 | 48.0 | 56.5 | 87.0 | 46.0 | 56.5 | 78.0 | 49.5 | 59.0 | 46.0 | 37.0 | 40.0 | 66.5 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

The performance of VideoChat-T on MVBench is shown in Table 8. Compared to VideoChat2, VideoChat-T only experienced a 0.5% accuracy loss. This indicates that our method effectively preserves the capabilities of the base model, preventing catastrophic forgetting caused by incremental fine-tuning. For the Action Localization (AL) task, which requires the model to determine the coarse-grained temporal position of events, the test accuracy improved from 44.0% to 56.5%. This indirectly confirms that our method significantly enhances the model’s temporal awareness capabilities.

**Table 9: The full performance of VideoChat-T on VideoMME. VideoChat-T achieved significant performance improvements, particularly in the long video subset.**
| Model | LLM size | Overall (%) | Short Video (%) | Medium Video (%) | Long Video (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| w/o subs | w subs | w/o subs | w subs | w/o subs | w subs | w/o subs | w subs |  |  |
| ST-LLM (Liu et al., ) | 7B | 37.9 | 42.3 | 45.7 | 48.4 | 36.8 | 41.4 | 31.3 | 36.9 |
| Video-LLaVA (Lin et al., ) | 7B | 39.9 | 41.6 | 45.3 | 46.1 | 38.0 | 40.7 | 36.2 | 38.1 |
| ShareGPT4Video (Chen et al., ) | 8B | 39.9 | 43.6 | 48.3 | 53.6 | 36.3 | 39.3 | 35.0 | 37.9 |
| Chat-UniVi-v1.5 (Jin et al., ) | 7B | 40.6 | 45.9 | 45.7 | 51.2 | 40.3 | 44.6 | 35.8 | 41.8 |
| Qwen-VL-Chat (Bai et al., ) | 7B | 41.1 | 41.9 | 46.9 | 47.3 | 38.7 | 40.4 | 37.8 | 37.9 |
| ShareGemini (Share, ) | 7B | 43.2 | 47.9 | 49.1 | 52.8 | 41.3 | 47.3 | 39.1 | 43.4 |
| VideoChat2 (Li et al., ) | 7B | 39.5 | 43.8 | 48.3 | 52.8 | 37.0 | 39.4 | 33.2 | 39.2 |
| VideoChat-T | 7B | 46.3 | 55.8 | 53.3 | 59.9 | 43.8 | 54.0 | 41.9 | 53.4 |
|  |  |  |  |  |  |  |  |  |  |

The overall performance of our model on VideoMME is presented in Table 9. VideoChat-T achieved significant improvements on both evaluation benchmarks of VideoMME, which include watching videos only and videos with subtitles. The improvements are particularly notable in the long video subset.

## Appendix I Extra ablation

**Table 10: The performance changes of the model after removing STAR. Although the video sources of STAR may have some domain correlation with those of Charades-STA and MVBench, the performance of our model is minimally affected by STAR.**
| Model | Charades-STA(R@1 IOU=0.5) | MVBench(avg) |
| --- | --- | --- |
| VideoChat-T | 48.7 | 59.9 |
| w/o STAR | 47.5 (-1.2) | 59.4 (-0.5) |
|  |  |  |

We found that the video sources in the STAR dataset might have some domain correlation with the video sources in MVBench and Charades-STA. Therefore, we removed STAR from the training set while keeping other training settings consistent with the original. The performance on benchmarks where the video sources might have domain correlation is shown in Table 10. The model’s accuracy on Charades-STA (R@1 IOU=0.5) decreased by 1.2%, and the average accuracy on MVBench decreased by 0.5%. This indicates that the domain correlation of video sources did not significantly impact performance for our model. Notably, after removing STAR, our normal data volume was reduced to approximately 36K. This implies that, with sufficiently parameter-efficient initialization and appropriate training strategies, using only a small amount of high-quality normal data is sufficient to retain the model’s original capabilities.

## Appendix K Case study

Figure: Figure 6: More qualitative comparisons in temporal grounding & long video QA.
Refer to caption: x6.png

Figure: Figure 7: More qualitative comparisons in temporal grounding & short video QA.
Refer to caption: x7.png

Figure: Figure 8: More qualitative comparisons in temporal grounding & captioning.
Refer to caption: x8.png

To further qualitatively analyze our model, we supplemented it with three types of examples. These examples are aimed at long video QA, short video QA, and captioning tasks, all of which include temporal grounding.

More qualitative comparisons about long video QA are shown in Figure 6. VideoChat-T effectively handles various questions across different domains. By better perceiving the temporal relationships of different events occurring in long videos, it can more accurately and deeply understand the detailed content of the entire video.

More qualitative comparisons about short video QA are shown in Figure 7. VideoChat-T effectively retains the original capabilities of the base model. Through parameter-efficient initialization methods and appropriate training strategies, we minimize the damage to the base model’s capabilities caused by new architectures and data.

More qualitative comparisons about captioning are shown in Figure 8. Although VideoChat2 describes more local details in some scenarios compared to VideoChat-T, VideoChat-T focuses more on a series of temporal events, which aligns better with how humans typically describe videos.

## Appendix M Discussion

By appropriately fine-tuning the Multimodal Large Language Model (MLLM), we have developed a general MLLM with powerful zero-shot temporal grounding capabilities. Its performance, after fine-tuning on the training set of evaluation benchmarks, can rival the current state-of-the-art supervised expert models. Based on these results, we can boldly speculate whether it is possible to internalize the capabilities of expert models such as spatial grounding, tracking and detection (Zeng et al., ) into the MLLM itself, without using any external expert decoders, to enhance the comprehensive understanding performance of the MLLM and achieve a unified generalist MLLM for multiple tasks.

Merlin (Yu et al., ) and VisionLLM (Wang et al., ) have already attempted something similar, but its performance is limited by the reasoning capabilities and language representation bottlenecks of the LLM. There is still a significant gap between its performance and that of expert models for various tasks. We observed similar phenomena in our experiments. The temporal grounding task only requires outputting two timestamps, and the task format is relatively simple, so our model achieved good results. However, the highlight detection task requires outputting multiple discrete timestamps and their corresponding saliency scores. The model needs to accurately predict dozens of numbers in language form to answer the question correctly. Our model performed well only on data with fewer timestamps. Therefore, how to simplify the complex output format of expert tasks into the language representation of LLMs, or to design special processing procedures to simplify complex expert tasks, is a question worth exploring.

Moreover, designing diverse data formats is also crucial for enhancing the expert capabilities of MLLMs. Compared to classic expert models, MLLMs have a natural advantage in task type diversity and can enhance their performance through various different variants tasks of a single capability. For temporal grounding tasks, we found that enhancing task diversity has a significant effect on improving the model’s temporal perception generalization ability. We can boldly speculate that if there are sufficiently diverse training data task types, most tasks with relatively simple output formats can achieve results comparable to expert models through appropriate instruction fine-tuning.

We have reason to believe that with the continuous enhancement of MLLM capabilities in various aspects, most expert model tasks will be integrated into generalist MLLMs, making MLLMs more flexible with multiple expert capabilities, thereby enhancing their comprehensive understanding ability.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could training the model on both temporal and non-temporal grounding data mitigate performance loss in short-term videos? Why does temporal grounding data lead to accuracy loss in short-term videos? Despite this, why is short-term accuracy on VideoMME still improved?
soundness: 3
strengths: 1. This work is highly complete. The contributions are multiple folds: VideoChat-T framework, TimePro dataset, Temporal Grounded Caption task, and extensive experiments and analysis.

2. Significant improvements on long video processing shown in Table 2.

3. Interesting TAPE, utilizing zero-padding anchors and gradual transmission of relative temporal positional encoding.
summary: The paper introduces TimeSuite, a set of designs to adapt short-form video multimodal large language models (MLLMs) for long video understanding. It includes a new framework for processing long video sequences, a high-quality dataset (TimePro) for grounded tuning, and an instruction tuning task to incorporate grounding supervision. The proposed model, VideoChat-T, uses token shuffling for compression and Temporal Adaptive Position Encoding (TAPE) for enhanced temporal awareness. A new task type, Temporal Grounded Caption, is introduced to improve video descriptions and timestamp prediction. Experimental results show significant improvements on benchmarks and robust zero-shot temporal grounding capabilities.
weaknesses: 1. The VideoChat2 baseline shows a weak performance with 39.5 accuracy in VideoMME. It would be more convincing to use a stronger 7B model for evaluation, such as onevision-7B, which achieves 58.2 accuracy.

2. The short-term performance on MVbench drops require a deeper investigation rather than mere explanations.

3. Instead of using implicit TAPE, consider implementing a more explicit solution like 3D RoPE (e.g., Qwen2-VL) and conduct an ablation study.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why is the operation of “concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer” referred to as “token shuffle”?
2. Can you use the same training data (e.g., TimePro) to train other VideoLLMs (e.g., TimeChat) and then compare their performance? This would help eliminate the impact of training data, allowing for a clearer assessment of how performance is associated with model design.
soundness: 3
strengths: 1. The paper is well-written. The proposed method is well-illustrated and easy to follow.
2. The contributions include a strong video LLM (VideoChat-T) and a temporal-centric instruction-tuning dataset (TimePro), both of which are substantial advancements in the field.
3. The authors conduct a thorough evaluation and ablation study to validate the effectiveness of the proposed model.
summary: This paper introduces TimeSuite, a collection of designs focusing on efficient architecture, high-quality data, and a novel instruction-tuning task. Building on TimeSuite, the authors propose a long-video multimodal large language model (MLLM) named VideoChat-T, which demonstrates robust zero-shot temporal grounding capabilities and significantly outperforms existing state-of-the-art MLLMs. After fine-tuning, it performs comparably to traditional supervised expert models.
weaknesses: While I do not see major flaws in this paper, I note that the technical novelty is relatively limited. 

There are two main modules in VideoChat-T: (1) the VL-connector with token shuffling, and (2) temporal adaptive position encoding (TAPE). 
However, the VL-connector's approach of concatenating tokens in the channel dimension and then compressing the channel dimension with a linear layer has been previously utilized in Qwen2-vl (I am unsure who proposed this operation first; please correct me if you know). For TAPE, the design of the position embedding is derived from CPVT. These factors somewhat limit the technical novelty of this work.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: See weaknesses
soundness: 2
strengths: - This paper propose TimeSuite, a collection of new designs to improve the long video understanding capability of the existing short-form MLLMs
summary: The paper introduces TimeSuite, a framework designed to enhance Multimodal Large Language Models (MLLMs) for long video understanding. It proposes a new long-video MLLM called VideoChat-T, featuring token shuffling and Temporal Adaptive Position Encoding (TAPE) for better temporal awareness. TimeSuite includes a specialized instruction tuning dataset, TimePro, comprising 349,000 annotations across nine tasks, with a focus on Temporal Grounded Captioning for accurate video descriptions with timestamps. Experiments show that TimeSuite improves long video comprehension, achieving notable performance gains on benchmarks and demonstrating strong zero-shot grounding capabilities, rivaling traditional supervised models after fine-tuning.
weaknesses: - The paper does not adequately compare its results against notable existing works, such as Qwen2-VL and Llava-OneVision. A comprehensive comparison would help contextualize the contributions and highlight any advantages or shortcomings.
- The dataset used in the experiments, TimePro is highly imbalanced (Figure 3). It is unclear whether the authors have explored different data ratios or configurations. Investigating this aspect could provide insights into the robustness of the proposed method and its adaptability to varying data distributions.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 2
questions: It would be beneficial to include a more in-depth discussion of theories and principles in the manuscript.
soundness: 3
strengths: The proposed method is both systematic and effective. The experiments are comprehensive and detailed, and the results of the method are convincing.
summary: This paper presents TimeSuite, a novel approach designed to enhance the understanding of long-form videos using existing short-form MLLMs. TimeSuite includes an efficient framework for processing long video sequences, a high-quality dataset for grounded tuning, and a new instruction tuning task to integrate grounding supervision in a traditional QA format. Experimental results demonstrate the advantage of  the proposed method.
weaknesses: Although the proposed method includes specific designs tailored to the problem, the novelty is limited. It is a relatively engineering-focused paper. The systematic description is quite detailed, but it lacks a deeper discussion of the underlying ideas, theories and principles. Additionally, due to the design of auxiliary tasks and datasets, the comparison with baseline methods may be somewhat unfair.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. **Token Shuffle Mechanism**: Could the authors elaborate on the decision to use a token shuffling mechanism over other compression techniques, and provide a comparison of its performance against alternatives in terms of temporal consistency and computational efficiency?

2. **Temporal Adaptive Position Encoding (TAPE)**: It would be beneficial if the authors could discuss the potential limitations of TAPE in handling videos with highly variable or complex temporal dynamics, and whether there are any specific domains where TAPE excels or falls short.

3. **Data Diversity and Model Generalization**: How does the choice of datasets for TimePro impact the model's generalization capabilities? Are there any biases introduced by the current dataset composition that could limit the model's applicability to unseen video domains?

4. **Zero-Shot Performance vs. Fine-Tuning**: The paper mentions robust zero-shot capabilities. Could the authors provide insights into why there is a performance gap between zero-shot and fine-tuned models, and what aspects of the model or training process could be improved to bridge this gap?

5. **Long-Video Understanding Limitations**: Are there specific scenarios or types of long videos where VideoChat-T struggles? If so, could the authors suggest potential improvements or future work to address these limitations?

6. **Integration of Expert Model Capabilities**: The authors propose a future direction of integrating expert model capabilities into MLLMs. What are the authors' thoughts on the feasibility of this approach, and what challenges need to be overcome to achieve a unified generalist MLLM?

7. **Complex Output Formats**: For tasks requiring complex outputs, such as highlight detection, how can MLLMs be adapted to handle multiple discrete timestamps and saliency scores effectively? Are there plans to modify the model architecture or training process to better accommodate such tasks?
soundness: 2
strengths: - **Originality**: The paper introduces TimeSuite, a novel framework that enhances Multimodal Large Language Models (MLLMs) for long video understanding through grounded tuning. It proposes creative solutions like Token Shuffle and Temporal Adaptive Position Encoding (TAPE), which are innovative approaches to compress visual tokens and enhance temporal awareness in video representations.

- **Quality**: The research demonstrates high quality through rigorous experimentation and ablation studies, showing significant improvements over existing benchmarks like Egoschema and VideoMME. The paper also presents a comprehensive dataset, TimePro, with 349k high-quality annotations.

- **Clarity**: The paper is well-structured and clearly articulated. Complex concepts such as Temporal Grounded Caption and TAPE are explained with clarity, making the paper accessible to readers. Figures and tables are effectively used to convey key results and comparisons.

- **Significance**: The proposed solutions for temporal grounding could have broad implications for video understanding tasks, and the zero-shot capabilities of VideoChat-T are particularly noteworthy, showing the potential to rival supervised expert models.
summary: The paper introduces TimeSuite, a novel framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) for understanding long videos through grounded tuning. TimeSuite comprises three key components: a token shuffling scheme to compress visual tokens, reducing computational load; Temporal Adaptive Position Encoding (TAPE) to boost the temporal sensitivity of visual representations; and a new instruction tuning task, Temporal Grounded Caption, which integrates timestamp prediction to guide MLLMs to focus on relevant visual content. The authors also present TimePro, a high-quality dataset with 349k grounded annotations across 9 tasks, aimed at improving MLLMs' temporal perception. Experiments demonstrate that TimeSuite significantly improves performance on long video understanding benchmarks like Egoschema and VideoMME, with improvements of 5.6% and 6.8% respectively, and exhibits robust zero-shot temporal grounding abilities, outperforming existing MLLMs. The paper concludes that TimeSuite provides effective designs for MLLMs to enhance their performance on temporal grounding and long video question answering.
weaknesses: - **Generalization**: The paper primarily focuses on video datasets that are thematically similar. It is unclear how well TimeSuite generalizes to videos with significantly different content or from other domains, such as OpenEQA and CinePile. Expanding the dataset to include more diverse domains could strengthen the paper's claims.

- **Scalability and Efficiency**: The computational requirements for processing long videos are high. The paper could provide more insights into the scalability of TimeSuite and its efficiency, especially when dealing with very long videos or a large number of frames.

- **Longitudinal Performance**: The paper does not address how the model's performance degrades over time or with an increasing amount of video data. It would be beneficial to include studies on the long-term sustainability of the model's performance.

- **Qualitative Analysis**: While quantitative results are provided, a deeper qualitative analysis of the model's outputs, especially in cases of failure, could offer actionable insights into the model's reasoning process and areas for improvement.

Meta-review: After discussion, this submission received 3 positive scores and a negative score. The reviewer who assigned the negative score has no response to the author’s rebuttal. After reading the paper, the review comments and the rebuttal, the AC thinks that the major concerns about technical novelty of the proposed approach and experimental evaluation were comprehensively solved. The AC supports the acceptance of this submission as a poster paper in ICLR.

Decision: Accept (Poster)