Temporal Reasoning Transfer from Text to Video

## Abstract

Abstract Video Large Language Models (Video LLMs) have shown promising capabilities in video comprehension, yet they struggle with tracking temporal changes and reasoning about temporal relationships.
While previous research attributed this limitation to the ineffective temporal encoding of visual inputs, our diagnostic study reveals that video representations contain sufficient information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs’ temporal reasoning capability stems from the underlying LLM’s inherent difficulty with temporal concepts, as evidenced by poor performance on textual temporal question-answering tasks.
Building on this discovery, we introduce the T extual T emporal reasoning T ransfer ( T3 ).
T3 synthesizes diverse temporal reasoning tasks in pure text format from existing image-text datasets, addressing the scarcity of video samples with complex temporal scenarios.
Remarkably, without using any video data , T3 enhances LongVA-7B’s temporal understanding, yielding a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark, which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video samples.
Additionally, the enhanced LongVA-7B model achieves competitive performance on comprehensive video benchmarks. For example, it achieves a 49.7 accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B.
Further analysis reveals a strong correlation between textual and video temporal task performance, validating the efficacy of transferring temporal reasoning abilities from text to video domains. 1 1 1 Project page: https://video-t3.github.io

## 1 Introduction

The rapid development of large language models (LLMs) (OpenAI, ; Gemini Team, ) has sparked significant interest in video large language models (Video LLMs) (Zhang et al., ; Lin et al., ) due to their impressive generation and reasoning capabilities.
Current approaches typically use pre-trained vision encoders (Radford et al., ) combined with powerful LLMs (Touvron et al., ; Chiang et al., ; Yang et al., ) as the starting point for Video LLMs.
These models employ various strategies to handle multiple video frames (Li et al., ; Tan et al., ), and are then further trained on curated instruction-tuning datasets (Chen et al., ), demonstrating promising abilities in video comprehension tasks (Fu et al., ; Zhou et al., ).

Despite the progress in video comprehension, Video LLMs often struggle with temporal reasoning, which is essential for truly interpreting video content (Li et al., ; Tang et al., ).
Specifically, video temporal reasoning involves the ability to track changes over time, comprehend event sequences, and relate objects and actions to specific moments in a video (Mangalam et al., ; Li et al., ).
As illustrated in Figure 1, two strong Video LLMs, LongVA-7B (Zhang et al., ) and VILA-8B (Lin et al., ), both failed to answer basic questions about the chronological order of events in synthesized videos, whereas humans can predict correctly without difficulty.
This significant gap between human performance and current Video LLMs in temporal reasoning tasks motivates us to explore the underlying reasons for this discrepancy.

Previous research has largely attributed temporal reasoning deficiencies in Video LLMs to ineffective video encodings, leading to various temporal aggregation module developments (Ren et al., ; Jin et al., ; Tan et al., ).
Our paper takes a different approach by decomposing Video LLMs into two parts and asking a fundamental question: What is the bottleneck of this limitation?
Is it due to limitations in the vision encoder, or, surprisingly, shortcomings in the LLM itself?
We conduct probing experiments using synthesized videos for basic temporal-related video question-answering (QA) tasks, allowing full control over temporal aspects. Our method involves:
(i) Training small probe classifiers on video representations in the Video LLM embedding space to assess temporal information captured by visual encoders and aggregation modules.
(ii) Transforming synthesized videos into textual descriptions using commercial visual language models (e.g., GPT-4V) to analyze how standalone LLMs process temporal information.
By comparing the performance of these components with full Video LLMs, we precisely locate the bottleneck in temporal understanding.

Our experiments reveal a striking contrast in temporal reasoning capabilities between different components of Video LLMs.
Probe classifiers trained on video embeddings achieve near-perfect accuracy ($>$ 90% in most cases), indicating that these embeddings successfully capture sufficient temporal information. Even simple neural models like LSTM (Hochreiter & Schmidhuber, ) can accurately extract temporal relationships from these embeddings.
Conversely, despite their significantly larger scale, LLMs struggle to process this temporal information effectively. They exhibit relatively low probing accuracy across various temporal aspects, highlighting their difficulty in handling temporal relationships.
These findings provide compelling evidence that the LLM component, rather than the visual encoding, is the primary bottleneck in the temporal reasoning of current Video LLMs,
motivating us to enhance LLMs’ ability to reason about temporal information.

Figure: Figure 1: Two popular Video LLMs struggle with basic temporal reasoning (left). We mitigate this issue via textual temporal transfer (middle), which demonstrates consistent improvement (right).
Refer to caption: x1.png

Building on our insights, we propose enhancing Video LLMs’ temporal understanding by focusing on the LLM component. Our approach leverages existing image-text datasets to generate diverse temporal reasoning tasks in *pure text format*, overcoming the scarcity of video samples with complex temporal scenarios (§3).
Remarkably, without using any video data, our text-only synthesized dataset enables LongVA-7B to outperform ShareGPT4Video-8B (trained on 28,000 video samples) on the TempCompass benchmark (Liu et al., ).
Moreover, as shown in the right of Figure 1, our enhanced model demonstrates competitive results on various video-understanding benchmarks. It improves temporal reasoning accuracy by 12.4 points on Video-MME (Fu et al., ) and increases average accuracy from 56.4 to 58.1 on MLVU (Zhou et al., ), surpassing larger models such as InternVL-Chat-V1.5-20B (Chen et al., ) and VILA-1.5-40B (Lin et al., ).
Furthermore, we observe high correlations between performance on our textual temporal validation set and benchmark results, e.g., Pearson $r=0.89$ and $r=0.85$ on TempCompass and MLVU, respectively. Analysis reveals the crucial role of self-attention modules in temporal reasoning transfer, and our method improves the utilization of more video frames.
These findings underscore the effectiveness of our textual temporal reasoning transfer, offering valuable insights for future Video LLMs.

## 2 Pinpointing Video LLM Temporal Reasoning Bottleneck

In this section, we seek to examine and pinpoint the temporal reasoning bottleneck of Video LLMs.
Video LLMs typically consist of two essential components for temporal understanding tasks: a vision encoder and an LLM decoder, where the former extracts visual features from video frames and the latter is responsible for integrating this information with textual instructions to complete the end task.
We aim to answer two questions: (1) Can existing Video LLMs understand the temporal information in videos? (2) If not, which component—the vision encoder or the LLM decoder—is the bottleneck?
To address these questions, we design different tasks to test the full Video LLMs and these two components separately(§2.1), incorporating various aspects of temporal understanding abilities (§2.2), and discuss our findings (§2.3).

### 2.1 Task Formulation for Different Video LLM Components

Full Video LLM. We evaluate the full Video LLMs through multi-choice video question answering. Specifically, we uniformly sample eight frames from videos and present them to the model along with a multiple-choice question. To encourage the Video LLM to directly output an option, we append the prompt “Answer the option only.” after the question.

LLM Decoder. In terms of the LLM decoder, we replace the video frames with detailed frame captions generated by GPT-4o. In this way, we assess the ability of LLM decoder to understand temporal information in the textual context. The multi-choice questions are identical to those employed in testing the full Video LLM. To ensure that the temporal understanding questions can indeed be addressed using these frame captions, we carefully design the prompting strategy to incorporate all essential information in the captions (please refer to the Appendix A.2 for the details).

Visual Features. Unlike the full Video LLM and LLM decoder, the quality of visual features cannot be directly tested via question answering. To determine whether the visual features contain sufficient information to differentiate between different temporal dynamics in videos (e.g., brightening versus darkening), we employ the “classifier probe” technique proposed by (Alain & Bengio, ).
Assuming that there are $c$ categories of contradicting temporal dynamics, we train a “probe” $f()$ that maps a set of visual features $\mathbf{V}\in\mathcal{R}^{n\times d_{v}}$ to a probability distribution $p\in\mathcal{R}^{c}$. In practice, $f()$ is a single-layer LSTM model (Hochreiter & Schmidhuber, ) for capturing sequential correlation, and we employ the visual features that are down-sampled and projected into LLM embedding space. The probe is tested on the same set of videos as the full Video LLM but is trained on a different set of videos. More details of the visual classifier probe can be found in Appendix A.3.

### 2.2 Evaluation Data Collection

Figure: Figure 2: Example of videos and questions that focus on different temporal reasoning abilities.
Refer to caption: x2.png

Temporal Reasoning Abilities.
Temporal reasoning encompasses several key aspects. Based on recent Video LLM benchmarks (Fu et al., ; Wu et al., ), we focus on four critical dimensions: (1) Order: comprehending the sequential arrangement of events; (2) Attribute: perceiving changes in environmental or object attributes over time; (3) Temporal Referring: formulating questions based on specific temporal positions within a video; and (4) Temporal Grounding: identifying the temporal location of specific elements in a video.
While these aspects are a limited approximation of the full spectrum of temporal understanding, our study reveals that they are sufficient to expose significant deficiencies in current Video LLMs.

Collecting Videos.
Existing video benchmarks are unsuitable for our analysis due to two main limitations: (1) the lack of video clusters with contrasting temporal dynamics needed for visual feature probing, and (2) the inability to fully eliminate single-frame or language bias (Liu et al., ).
Consequently, we synthesize custom videos to isolate different aspects of temporal understanding.
Figure 2 illustrates examples of our synthesized videos. For the *Order* aspect, we concatenate different source videos (e.g., person, cat, flower) temporally, creating various categories based on concatenation order. The *Attribute* aspect focuses on shape (flower blooming videos and their reversals) and brightness (gradually altering pixel values of static images). *Temporal Referring* and *Temporal Grounding* reuse videos from the Order aspect with three concatenated items. Appendix A provides detailed information on the video creation process and data distribution.

Constructing Questions and Answers.
As depicted in Figure 2, we design multi-choice question templates for each temporal aspect. The questions remain consistent across videos within each aspect, while the correct answers vary based on the specific video content.
This approach ensures a controlled evaluation of temporal reasoning capabilities.

Figure: Figure 3: Temporal probing results for LongVA (upper) and VILA (lower). The probe on visual representations achieve $>90$ accuracy in most cases, while the LLM decoders still have large room for improvement even with textual inputs, leading to the poor temporal understanding ability of Video LLMs. Detailed results of the sub-categories are reported in Appendix A.4.
Refer to caption: x3.png

### 2.3 Results

We examine two advanced Video LLMs that utilize different LLM backbones on our synthesized probing datasets: LongVA-7B (Zhang et al., ), which is based on Qwen2-7B (Yang et al., ), and VILA-8B (Lin et al., ), which is built upon LLaMA3-8B (Dubey et al., ). The results are visualized in Figure 3, with detailed scores provided in Appendix A.4. Our analysis reveals several key findings:

Performance of Video LLMs: Video LLMs demonstrate relatively poor performance, struggling to reach 70 accuracy across all temporal understanding tasks. This suggests a potential limitation in their ability to process and reason about temporal information in video content.

Efficacy of Visual Representations:
Notably, small classifiers trained on the visual representations in the LLMs’ embedding space achieve near-perfect accuracy.
This finding suggests that these visual representations encapsulate rich information, sufficient to distinguish between temporally contradicting videos, even with a simple probe classifier.
Consequently, we can conclude that the input processing is not the primary limitation in temporal reasoning tasks.

LLM Backbone Performance: To our surprise, moderate-sized LLM backbone decoders such as Qwen2-7B and LLaMa3-8B fail to answer a considerable number of questions related to Referring, Grounding and Order aspects, which are intuitively very easy for SoTA LLMs given that the frame captions are provided in textual format.
Worse still, these LLM decoders perform only at the level of random guessing when it comes to the Attribute aspect. These findings indicate that the LLM decoders face substantial challenges even in the context of textual temporal understanding, posing a major limitation to the video temporal understanding capabilities of Video LLMs.
In comparison, larger LLMs with over 70 billion parameters significantly outperform their smaller counterparts on these tasks, suggesting that (i) the generated frame captions are accurate and contain sufficient information for temporal reasoning; (ii) the textual temporal understanding ability emerges when the text decoder scale exceeds certain thresholds.
We hypothesize that this discrepancy may be due to the sparse expression of temporal concepts in pre-trained corpora. Consequently, smaller-scale models might not have sufficient exposure to learn these temporal reasoning abilities effectively. This sparsity could explain why only very large language models ($>$70B parameters) demonstrate proficiency in temporal reasoning tasks.

Our findings ultimately suggest that the temporal reasoning ability in current Video LLMs is primarily constrained by the LLM decoder rather than the quality of visual representations. This conclusion highlights a critical area for improvement in the development of more effective Video LLMs, motivating us to enhance the temporal reasoning capabilities from the textual side.

## 3 Textual Temporal Understanding Transfer

In this section, we explore strategies to mitigate the deficiency of Video LLMs in temporal reasoning.
A straightforward approach would be to create a temporal-oriented instruction-tuning dataset from videos. However, existing approaches to video instruction-tuning data generation, whether through human annotation or automatic generation via rules/models, present significant challenges.
Human annotation of video datasets is resource-intensive, demanding considerable time and financial investment.
Synthetic video instruction tuning (Maaz et al., ; Zhang et al., ), while more efficient and scalable, is constrained by the content of the videos themselves.
For instance, if our goal is to enhance a particular aspect of temporal reasoning, it is crucial to ensure that the videos contain information relevant to this aspect.
This limitation makes it difficult to create instruction-tuning data that targets specific abilities.

Motivated by (1) the limitations of video instruction-tuning and (2) our finding that the LLM backbone is the primary bottleneck of temporal understanding, we investigate the feasibility of enhancing video temporal reasoning through a novel perspective: using synthesized textual temporal reasoning data.
Our method uses sequences of image captions as proxies for video frames, allowing us to create temporal-oriented question-answering pairs without relying on actual video content.
This text-only approach offers two key advantages: (i) Scalability: The abundance of available image-caption pairs (Changpinyo et al., ; Liu et al., ) allows for easy expansion of the dataset. (ii) Flexibility: We can precisely control the targeted ability and sample complexity by adjusting the content and number of image captions. To address the four aspects of temporal understanding (Order, Attribute, Referring and Grounding), we design heuristics to construct textual contexts and generate question-answer pairs solely from caption sequences with examples visualized in Figure 4. The specific generation processes are as follows.

Figure: Figure 4: Temporal-oriented question-answering pairs with textual image captions as context.
Refer to caption: x4.png

Order.
This aspect focuses on understanding the sequential order of image captions. To construct the textual context for question answering, we randomly sample 3$\sim$6 captions from a caption pool. The questions are created using two methods: On the one hand, we provide GPT-4-turbo with examples and prompt it to generate order-related questions, which we refer to as Order-GPT. On the other hand, we employ predefined templates and heuristic rules to create questions. These template questions require rearranging shuffled sequences according to the textual context of image captions. These sequences comprise of (a) complete image captions, (b) phrases within captions or (c) phrases with prefix identifiers (e.g., (1)(2)(3)(4) as shown in the example in Figure 4). We denote the template-based questions as Order-Template (X), where $\text{X}\in\{\text{sentence},\text{phrase},\text{prefix}\}$.

Attribute.
This aspect involves recognizing how specific attributes of objects or scenes change throughout the video. To achieve this, we first prompt GPT-4-turbo to generate new image captions by modifying particular attributes in the original captions. Specifically, we consider five types of attributes: color, light condition, size & shape, posture, and emotion. Subsequently, we employ GPT-4-turbo again to generate questions that focus on the attribute changes between pairs of captions.

Temporal Referring.
This aspect requires understanding questions that refer to particular temporal locations (e.g., begin, middle and end). To increase the difficulty, we employ GPT-4-turbo to generate three similar image captions that only differ in certain aspect, such as object, action or attribute. These three captions are then placed respectively at the beginning, middle, and end of the textual context. For the questions, we first generate a question for each caption without referring to temporal locations. Then, temporal references are added to the questions according to the specific temporal location of the corresponding caption.

Temporal Grounding.
This aspect is a complementary angle to temporal referring, aiming to identify the temporal location of an element (a phrase describing an object, action or attribute) of interest. The textual context reuses the same image captions as in temporal referring. To formulate the questions, we first prompt GPT-4-turbo to generate a declarative statement for each caption (e.g., “the coins are black”). These statements are then incorporated into the temporal grounding question templates, as illustrated in Figure 4.

We enhance the basic textual context (i.e., captions relevant to the question) by incorporating “distractor captions” sampled from the caption pool and inserted between the original captions.
This approach challenges the model to identify relevant temporal information amidst irrelevant context, enhances the robustness of its temporal understanding abilities, and presents a more realistic scenario mimicking the complexity of real-world temporal reasoning tasks.
To maintain question clarity, we ensure distracting captions do not share nouns with the original captions. Table 1 gives a summary of our synthesized textual temporal QA datasets. For each task, we also create a validation set consisting of 500 samples for later verification. Appendix B provides a detailed and formalized description of the data construction process, as well as the ablation study for distractor captions and a comparison of temporal reasoning transfer via multiple images (Appendix B.4).

**Table 1: Summary of our textual temporal reasoning datasets where $\mathbf{X}\in\{\text{phrase, prefix, sentence}\}$ for Order-Template ($\mathbf{X}$). Detailed data statistics are provided in Appendices B.3 and C.1.**
| Dataset | #Relevant Captions | #Distractor Captions | Description |
| --- | --- | --- | --- |
| Order-GPT ($N\times$) | 2$\sim$4 | $N\times 100\pm 50$, $N\in\{1,2,4,8\}$ | Order-related questions generated by GPT-4. |
| Attribute ($N\times$) | 2 | $N\times 100\pm 50$, $N\in\{1,2,4,8\}$ | Attribute-related questions. |
| Order-Template ($\mathbf{X}$) | 3$\sim$6 | 200$\pm$50 | Order-related questions based on templates $\mathbf{X}$ |
| Referring | 3 | 200$\pm$50 | Temporal referring questions. |
| Grounding | 3 | 200$\pm$50 | Temporal grounding questions. |

## 4 Experiments

This section details our experiments exploring the textual temporal understanding transfer.
We begin by describing the experimental setup and training protocols (§4.1). Next, we report the transfer results of textual temporal understanding and key findings (§4.2) and provide analysis results (§4.3).

### 4.1 Experimental Settings

Benchmarks. To establish a connection between textual temporal reasoning ability and video comprehension, we adopt the fine-grained temporal understanding benchmark, TempCompass(Liu et al., ) (Multiple-Choice subset), diagnosing multi-facet basic video temporal understanding abilities.
For a comprehensive assessment of general video understanding tasks, we additionally incorporate Multi-Task Long Video Understanding (MLVU) (Zhou et al., ) and Video-MME (Fu et al., ).
For MLVU, we select the following tasks covering three aspects: holistic video understanding (Topic Reasoning (TR) and Anomaly Reasoning (AR)), single-detail understanding (Needle Question-Answering, Ego Reasoning (ER) and Plot Question Answering), and multi-detail understanding (Action order (AO) and Action Count (AC)).
For Video-MME, which consists of 2700 video QA pairs spanning primary visual domains and three video duration types (Short, Medium, and Long), we focus on two temporal-oriented tasks (i.e., Temporal Perception and Temporal Reasoning) and overall performance.

Compared Methods.
We evaluate our temporal textual augmentation approach against two baselines:
(i) LLaVA-Next: Continually fine-tuning on the original image-text instruction dataset used by LongVA.(^2^22[https://github.com/xiaoachen98/Open-LLaVA-NeXT](https://github.com/xiaoachen98/Open-LLaVA-NeXT))
(ii) HotpotQA (Yang et al., ): Fine-tuning on a multi-document QA dataset to assess enhancements in textual locating ability.
We also report the performance of recent Video LLMs, including Video-ChatGPT (Maaz et al., ), Video-LLaVA-7B (Lin et al., ), LLaVA-Next-Video-7B-DPO (Liu et al., ), MA-LMM-7B (He et al., ), ShareGPT4Video-8B (Chen et al., ), LLama-3-VILA1.5-8B (Lin et al., ), VILA1.5-40B (Lin et al., ), and InternVL-CHat-V1.5-20B (Chen et al., ).
GPT-4o (OpenAI, ) is included to represent commercial model results.
These models serve as reference points and we report the highest scores published to represent optimal performance.

Training Details.
We adopt LongVA (Zhang et al., ) as our backbone model and continually fine-tune the official checkpoint. Despite being trained exclusively on text and image data, LongVA demonstrates exceptional zero-shot video understanding capabilities and can effectively handle long videos.
We strictly adhere to the original LongVA implementation and follow the official fine-tuning protocol.
This involves using Adam (Kingma & Ba, ) as the optimizer, with learning rates of 2e-6 for the visual encoder and 1e-5 for the rest part of the model.
For the exploration of textual temporal reasoning transfer, we use 22k samples for fine-tuning across different augmentation datasets. We further scaled the dataset up to transfer to holistic video understanding benchmarks, where we find it necessary to incorporate the original instruction tuning data to maintain visual perception ability.
According to our textual validation accuracy, we set the ratio of textual temporal QA and original data to 1:2 and the total samples to 200k.
This mix applies to the Hotpot QA as well for a fair comparison.
Appendix C provides the details of dataset composition and mixture configurations in our training.
The model is trained on the corresponding dataset for one epoch, a process that can be completed within 5 hours using 8 H100 GPUs.

### 4.2 Results

**Table 2: Fine-grained video temporal understanding evaluation on TempCompass benchmark. Without training on any videos, our T3 helps LongVA-7B outperform ShareGPT4Video-8B trained on 28k video samples. Best results are shown in bold.**
| Method | Action | Direction | Speed | Order | Attribute Change | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Video-ChatGPT-7B (Maaz et al., ) | 61.5 | 28.9 | 29.0 | 36.1 | 30.9 | 37.7 |
| Video-LLaVA-7B (Lin et al., ) | 76.0 | 35.2 | 35.7 | 37.8 | 41.0 | 45.6 |
| LLaVA-NeXT-Video-7B-DPO (Liu et al., ) | 87.6 | 35.8 | 41.3 | 39.7 | 45.8 | 50.6 |
| Llama-3-VILA1.5-8B (Lin et al., ) | 92.9 | 33.7 | 44.2 | 50.0 | 60.1 | 56.4 |
| ShareGPT4Video-8B (Chen et al., ) | 87.6 | 34.6 | 47.5 | 62.9 | 64.2 | 59.4 |
| LongVA-7B (32 frm) | 92.3 | 37.3 | 42.0 | 54.3 | 51.7 | 55.9 |
| + LLaVA-Next | 92.0 | 36.4 | 43.2 | 55.6 | 51.0 | 56.0 |
| + Hotpot QA | 92.3 | 37.0 | 39.4 | 53.0 | 50.7 | 54.9 |
| + Temporal Change (1x) | 91.1 | 37.3 | 39.8 | 65.2 | 63.9 | 59.5 |
| + Temporal Change (2x) | 89.6 | 39.1 | 40.4 | 66.9 | 61.8 | 59.6 |
| + Temporal Change (4x) | 90.5 | 43.0 | 39.4 | 64.2 | 64.2 | 60.4 |
| + Temporal Change (8x) | 90.2 | 42.4 | 39.8 | 64.2 | 68.1 | 61.0 |
| + Temporal Change (1x, 2x, 4x and 8x) | 90.5 | 38.2 | 39.8 | 65.6 | 64.2 | 59.7 |
| + Order-Template | 89.9 | 35.8 | 46.1 | 58.3 | 54.5 | 57.2 |
| + Temporal Referring | 91.4 | 33.7 | 46.1 | 46.7 | 51.0 | 54.2 |
| + Temporal Grounding | 63.0 | 31.6 | 29.3 | 40.7 | 38.9 | 41.0 |
| + Order-Template + Temporal Change (1x) | 90.2 | 40.6 | 42.9 | 63.6 | 65.3 | 60.6 |
| + Temporal Grounding+ Temporal Change (1x) | 91.1 | 38.8 | 41.6 | 62.9 | 62.9 | 59.6 |
| + Temporal Referring + Temporal Change (1x) | 90.8 | 38.2 | 42.3 | 62.9 | 64.6 | 59.8 |
| + T3 (all tasks) | 90.8 | 39.7 | 41.6 | 65.9 | 68.1 | 61.2 |
| GPT-4o | 98.2 | 52.8 | 52.1 | 73.2 | 78.5 | 71.0 |

Our results aim to answer the following three research questions: (1) Can textual temporal reasoning ability be effectively transferred to video temporal reasoning? and (2) Do these two abilities correlate well? and (3) Does this transfer also reflect on holistic video understanding benchmarks?

Textual temporal reasoning transfer:
Table 2 presents the evaluation results of models trained on various compositions of our datasets.
Continual training with the original LLaVA-Next dataset yields negligible improvements, while HotpotQA training surprisingly degrades performance, despite its similarity to temporal location tasks. These findings underscore the non-trivial nature of enhancing temporal understanding in Video LLMs.
Our textual temporal transfer approach, by contrast, demonstrates significant improvements. Regarding the different tasks, we find that: (i)
The Temporal Change (1x) subset which combines Order (1x) and Attribute (1x), enhances Order accuracy from 54.3 to 65.2 and Attribute accuracy from 51.7 to 63.9. Increasing the number of distractor captions (1x to 8x) leads to generally better performance.
(ii)
Order-Template and Temporal Referring excel in the Speed dimension, while Temporal Grounding negatively influences all aspects.
(iii)
Combining different aspects often leads to synergistic improvements. For example, supplementing Temporal Change (1x) with other tasks leads to better results than training with the set alone.
This suggests complementary benefits across diverse temporal reasoning skills. Finally,
despite without any video data, our T3 composition with all synthesized tasks, helps LongVA-7B achieve the best overall accuracy, even outperforming ShareGPT4Video-8B trained on 28,000 video samples annotated by GPT-4V.
These results highlight the effectiveness of our approach in improving Video LLMs’ temporal reasoning across various dimensions.

Figure: Figure 5: Textual temporal reasoning accuracy correlates positively with video understanding results on three benchmarks.
Refer to caption: x5.png

Correlation between textual and video temporal understanding: To assess transferability more intuitively, we calculate correlations between textual validation accuracy and benchmark scores of models trained on our textual samples.
Figure 5 illustrates a strong correlation with TempCompass overall accuracy (Pearson $r=0.89$, $p<0.01$), and MLVU accuracy (Pearson $r=0.85$, $p<0.01$). It also positively correlates with Video-MME performances (Pearson $r=0.59$, $p<0.05$).
These findings validate the feasibility of enhancing video temporal understanding from the LLM side.

Holistic video understanding evaluation:
Based on previous results, we adopt the T3 composition, which achieves the highest textual validation scores, to explore transfer effects on comprehensive video understanding benchmarks.
Table 3 shows detailed task accuracy results on MLVU. Further training on the original image-text instruction dataset or Hotpot QA decreased overall performance, indicating that video temporal understanding cannot be enhanced through continued fine-tuning or long-context training alone.
In contrast, mixing our T3 with LLaVA-Next improves performance across most tasks.
Notably, the two highly temporal-oriented tasks, Action Count (AC) and Action Order (AO), see a substantial average gain of 8.3 points. Specifically, AC scores increased by 3.9 points (25.2 $\rightarrow$ 29.1), while AO scores improved by 12.7 points (41.7 $\rightarrow$ 54.4).
Table 4 demonstrates substantial improvement in Video-MME’s temporal reasoning subtask (37.3 $\rightarrow$ 49.7), even outperforming larger models such as InternVL-Chat-V1.5-20B and VILA1.5-40B.
Besides, compared to fine-tuning with LLaVA-Next and Hotpot QA, our method yields the best overall performance at 55.0.
These results validate that enhanced textual temporal reasoning in LLM backbones effectively transfers to holistic video understanding, confirming our approach’s efficacy.

**Table 3: MLVU evaluation results. Our textual temporal reasoning transfer achieves the best overall performance, with significant gains in temporal-related aspects over the backbone model. TR: Topic Reasoning, AR: Anomaly Recognition, ER: Ego Reasoning, AO: Action Order, AC: Action Count. * denotes temporal-related dimensions. Best results are in bold.**
| Model | AC^∗ | ER | Needle QA | AO^∗ | Plot QA | AR | TR | Macro Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Video-ChatGPT-7B (Maaz et al., ) | 31.1 | 42.0 | 40.3 | 25.1 | 29.9 | 24.0 | 26.9 | 31.3 |
| Video-LLaVA-7B (Lin et al., ) | 35.9 | 45.2 | 53.2 | 20.1 | 48.4 | 57.0 | 71.6 | 47.3 |
| MA-LMM-7B (He et al., ) | 24.3 | 38.9 | 43.1 | 25.1 | 35.8 | 35.5 | 51.9 | 36.4 |
| Llama-3-VILA1.5-8B (Lin et al., ) | 0.0 | 24.7 | 32.4 | 6.6 | 20.0 | 27.0 | 46.2 | 22.4 |
| VILA1.5-40B (Lin et al., ) | 11.7 | 35.8 | 38.3 | 34.3 | 62.0 | 56.4 | 84.7 | 46.2 |
| InternVL-Chat-V1.5-20B (Chen et al., ) | 13.3 | 24.5 | 40.0 | 14.3 | 42.0 | 51.3 | 80.2 | 37.9 |
| LongVA-7B (128 frm) | 25.2 | 48.6 | 70.4 | 41.7 | 68.1 | 58.5 | 82.2 | 56.4 |
| + LLaVA-Next | 11.7 | 20.5 | 36.6 | 17.8 | 45.3 | 17.5 | 72.7 | 31.7 |
| + Hotpot QA w/ LLaVA-Next | 13.1 | 27.6 | 40.6 | 19.7 | 45.3 | 21.5 | 72.4 | 34.3 |
| + T3 w/ LLaVA-Next (Ours) | 29.1 | 48.9 | 72.1 | 54.4 | 69.4 | 51.0 | 81.4 | 58.1 |
| GPT-4o | 46.3 | 57.1 | 64.8 | 56.7 | 65.1 | 74.5 | 87.4 | 64.6 |

**Table 4: Video-MME evaluation results. Our method enhances LongVA-7B’s accuracy across various video durations, even outperforming VILA1.5-40B in temporal reasoning.**
| Method | Temporal Perception | Temporal Reasoning | Short | Medium | Long | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Video-LLaVA-7B (Lin et al., ) | - | - | 45.3 | 38.0 | 36.2 | 39.9 |
| LLaVA-NeXT-Video-7B-DPO (Liu et al., ) | 40.0 | 29.4 | 48.9 | 42.0 | 35.6 | 42.1 |
| Llama-3-VILA1.5-8B (Lin et al., ) | 50.9 | 41.2 | 56.1 | 42.1 | 39.6 | 45.9 |
| VILA1.5-40B (Lin et al., ) | 60.0 | 40.7 | 72.0 | 61.2 | 53.8 | 62.3 |
| InternVL-Chat-V1.5-20B (Chen et al., ) | 45.5 | 33.3 | 60.2 | 46.4 | 45.6 | 50.7 |
| LongVA-7B (128 frm) | 58.2 | 37.3 | 61.1 | 50.4 | 46.2 | 52.6 |
| + LLaVA-Next | 54.6 | 37.3 | 61.2 | 50.6 | 44.9 | 52.2 |
| + Hotpot QA w/ LLaVA-Next | 65.5 | 39.6 | 60.2 | 50.9 | 45.6 | 52.2 |
| + T3 w/ LLaVA-Next (Ours) | 60.0 | 49.7 | 63.3 | 54.8 | 46.8 | 55.0 |
| GPT-4o | 74.1 | 59.4 | 80.0 | 70.3 | 65.3 | 71.9 |

### 4.3 Analysis

Critical LLM components for temporal understanding:
We evaluate the impact of selectively unfreezing LLM components across TempCompass, MLVU, and Video-MME datasets using the T3 /w LLaVA-Next dataset.
Figure 6(a) illustrates that fully unfreezing the language model yields optimal performance.
For TempCompass,
the fully unfrozen model achieves 58.2% accuracy, outperforming self-attention-only (57.7%) and MLP-only (55.1%) configurations.
Notably, unfreezing self-attention modules consistently surpasses unfreezing MLP layers, with MLVU showing a substantial 4.2 percentage point difference (55.8% v.s. 51.6%) and a 2.3 absolute accuracy gap on Video-MME (52.4% v.s. 50.1%).
This suggests that self-attention modules play a more crucial role in transferring temporal understanding capabilities.
These findings align with the understanding that self-attention modules act as dynamic information aggregators (Wang et al., ), while MLP layers primarily serve as static knowledge banks (Geva et al., ).
The superior performance of unfrozen attention modules implies that the ability to dynamically focus on and relate relevant temporal information is more critical for temporal reasoning tasks than accessing static knowledge.

Figure: (a) Compared to MLP modules, self-attention modules are more important for temporal reasoning.
Refer to caption: x6.png

Enhanced LLMs’ utilization of increased video frames:
We examine whether temporal-enhanced LLMs better leverage
increased input video frames by
comparing performance gains when
increasing frames from 32 to 128,
using models trained on Temporal Change sets.
Figure 6(b) shows that our textual temporal reasoning-enhanced models benefit more from increased input frames compared to the baseline LongVA.
On TempCompass, the Temporal Change (1x) model gains 1.7% accuracy with 128 frames, versus 0.6% for LongVA. Similarly, on MLVU, enhanced models achieve nearly double the gain of the baseline. These results indicate that our approach enables better utilization of additional input frames. We also observe that training on longer inputs with more distractor captions generally yields larger gains, though the optimal length varies between datasets.

## 5 Related Work

Video Large Language Models (Video LLMs).
Video LLMs, integrating LLMs and visual encoders, have shown promising results on diverse video tasks (Tang et al., ).
These models typically leverage open-source LLMs (Touvron et al., ; Yang et al., ) for generation and reasoning capabilities.
Recent architectural explorations have focused on efficient video encoding, including Video Transformer and Q-Former (Li et al., ; Ren et al., ), spatial and temporal QFormer (Zhang et al., ; Tan et al., ), and memory bank for long video frames (He et al., ).
Other approaches like Video-LLaVA (Lin et al., ), Chat-UniVi (Jin et al., ), VILA-series (Lin et al., ), and LLaVA-series (Liu et al., ; Zhang et al., ) demonstrate effective transfer from image to video tasks with image-video unification.
In contrast to these studies, our work focuses on probing and enhancing the temporal understanding ability of Video LLMs, identifying the LLM’s poor grasp of temporal concepts as a key bottleneck. Our textual-only temporal reasoning transfer effectively addresses this issue without using any image/video instruction tuning data.

Temporal Understanding Benchmarks for Video LLMs.
Video temporal understanding benchmarks have evolved rapidly to guide Video LLM development. Pilot studies incorporate existing tasks (Wang et al., ; Li et al., ) into comprehensive assessments (Li et al., ; Ning et al., ; Li et al., ).
As Video-LLMs become stronger, benchmarks such as EgoSchema (Mangalam et al., ), Neptune (Nagrani et al., ), Video-MME (Fu et al., ) and MLVU (Zhou et al., ), focus on stress-testing models with diverse and challenging tasks of long videos.
Specific benchmarks addressing temporal understanding (Li et al., ; Liu et al., ) have highlighted limitations in current Video LLMs.
Our work not only diagnoses the temporal understanding bottleneck in Video LLMs but also demonstrates a correlation between textual and fine-grained video temporal understanding.
Importantly, our proposed methods show improvements on challenging benchmarks such as Video-MME and MLVU, advancing the field of video temporal comprehension.

## 6 Conclusions and Limitations

Our work investigates the temporal reasoning bottleneck of Video LLMs, identifying the language model backbone as the primary source.
To address this, we develop a textual temporal reasoning transfer framework that synthesizes QA pairs on various temporal concepts from image-text pairs.
Experimental results validate the correlation between textual and visual temporal understanding, demonstrating the efficacy of our method on comprehensive video understanding benchmarks.
By providing a scalable and efficient solution for enhancing temporal reasoning capabilities, our work offers valuable insights for the future development of Video LLMs.
Limitations:
(i) Limited temporal concept scope: The covered four key dimensions are only an approximation of core abilities and may not encompass the full spectrum of temporal reasoning.
(ii) Limited gain for stronger LLMs: Our synthesized dataset might be too straightforward for advanced language models, potentially limiting observable performance gains. Future work could address this by iteratively creating more complex datasets that better challenge and reflect real-world temporal reasoning.

## Appendix A More Details of Video LLM Temporal Analysis

### A.1 Videos

Section 2.2 describes our basic idea to create temporally contradicting videos for different temporal aspects. In this section, we provide more details of the source videos and the composition of synthesized videos. Table 6 summarizes the video composition for different temporal aspects.

#### A.1.1 Order

Source Videos. This aspect encompasses three categories of source videos: people, cats, and flowers. We obtained these videos from the ShutterStock(^3^33https://www.shutterstock.com) platform. To create distinct training and test sets for the classifier probe, we collected videos of each category with both black and white backgrounds. The black background videos were used for training the probe, while the white background videos were reserved for evaluation.

Composition of Synthesized Videos. The synthesized video for this aspect concatenate two or three source videos. As illustrated in Table 6, the “Two Events” videos comprise two categories, showcasing cats and people in reversing sequential orders. The “Three Events” videos expand to six categories, covering all possible permutations of cat, person, and flower sequences.

#### A.1.2 Attribute

Source Videos. This aspect is further divided into Shape and Brightness. For Shape, we collected videos from ShutterStock that illustrate the process of flower blooming. Similar to the Order aspect, we gathered flowers with both black and white backgrounds to create separate training and testing sets for the classifier probe. For Brightness, we collected static images from the COCO dataset (Lin et al., ) and adjusted pixel values to synthesize videos with brightness variations. Car images were used to create training videos, while cat images were used for testing videos.

Composition of Synthesized Videos. Videos of Shape consist of two categories: flower blooming and its reversed process, with the latter created by inverting the source videos. Brightness videos also comprise two categories: brightening and darkening.

#### A.1.3 Temporal Referring

Source Videos. This aspect utilizes the “Three Events” videos previously collected for the Order aspect.

Composition of Synthesized Videos. The Temporal Referring questions are categorized into three types, each referring to a specific part of the video: the beginning, middle, and end. For instance, a typical question might be, “Which item is shown at the beginning/middle/end of the video?” For each of these temporal reference types, the videos are further classified into three categories, yielding different possible answers: a person, a cat, or a flower.

#### A.1.4 Temporal Grounding

Source Videos. This aspect also employs the “Three Events” videos originally gathered for the Order aspect.

Composition of Synthesized Videos. In this aspect, three question types are formulated, i.e., “In which part of the video can we see a person/cat/flower?” For each question type, the videos are divided into three categories, resulting in different answers grounding to the video’s different temporal locations: beginning, middle, or end.

### A.2 Frame Captions

To evaluate the textual temporal understanding capabilities of LLMs, we must ensure that frame captions contain adequate information about relevant temporal aspects. For example, in videos depicting a transition from a cat to a person, the initial frame captions should clearly identify the subject as a cat, while later frame captions should describe the person.

In our approach, we employ a “differential captioning strategy” for aspects such as Order, Shape Attribute, Temporal Referring and Temporal Grounding, drawing inspiration from ShareGPT4Video (Chen et al., ). As outlined in Table 7,we begin by generating a caption for the first frame. For subsequent frames, captions are created based on both the visual features of the current frame and the captions of preceding frame. Table 9, 10, 11 demonstrate that this method produces captions with sufficient and accurate per-frame information to comprehend these temporal aspects, thanks to the powerful visual perception ability of GPT-4o.

In terms of the Brightness Attribute, we find that simple differential captioning cannot accurately reflect brightness changes across video frames due to the lack of a consistent brightness standard. To address this issue, we developed a chain-of-thought (CoT) captioning strategy. This approach first prompts GPT-4o to categorize frame brightness into four levels (very dark, slightly dark, normal, and bright) before generating the caption. As illustrated in Table 10, the resulting frame captions accurately reflect brightness changes throughout the video.

### A.3 Classifier Probe Training

Model Architecture. Our classifier probe is built upon a single-layer unidirectional LSTM model with a hidden dimension of 128. This probe is trained on visual features that are down-sampled and projected into the LLM embedding space, i.e., the embedding of visual tokens for Video LLMs. Let $\mathbf{V}\in\mathcal{R}^{n\times d_{v}}$ represent these visual features, where $n=l\times h\times w$ and $l,h,w,d_{v}$ correspond to the temporal, height, width and channel dimensions, respectively. For an 8-frame input video, the default setup of LongVA-7B is $n=1152,d_{v}=3584$, while for VILA-8B, it is $n=1568,d_{v}=4096$. We further down-sample $\mathbf{V}$ to $\mathbf{V}^{\prime}\in\mathcal{R}^{n^{\prime}\times d^{{}^{\prime}}_{v}}$ using bi-linear interpolation, where $n^{\prime}=128$ and $d^{{}^{\prime}}_{v}=1024$. The LSTM probe is then trained on this down-sampled representation $\mathbf{V}^{\prime}$.

Data. Table 6 provides a comprehensive breakdown of the training and test video compositions. For the Brightness Attribute, we generated training videos using car images, while the test videos are constructed from cat images. Regarding other temporal aspects, we created training videos with a black background, contrasting with the white background used in the test videos.

Training Hyper-parameters. Table 12 shows the training hyper-parameters for the LSTM classifier probe. For relatively easy temporal aspects, i.e., the “Two Events” Order, we train the probe for only 15 epoches. For Attribute and other aspects, we increase the training epoch to 80 and 120, respectively.

### A.4 Results

Figures 7 and 8 provide a comprehensive breakdown of our temporal analysis results. The data reveals significant variations in model performance across different sub-categories within particular temporal aspects. In the Order aspect, we observe a stark contrast in difficulty between two-event and three-event sequences. The task of discerning the order of three events proves substantially more challenging, with LongVA-7B and VILA-8B experiencing dramatic accuracy drops of 41.8 and 51.1 points, respectively. Regarding Temporal Referring, the accuracy is significantly lower when referring to the middle of the video. The deficiency of the LLM decoder in temporal understanding is also more pronounced in these challenging scenarios.

## Appendix B More Details of Textual Temporal Understanding Data

### B.1 Caption Pool

The contextual information of our textual QA data are sourced from the detailed image captions from the LLaVA-ReCap-558K dataset (^4^44https://huggingface.co/datasets/lmms-lab/LLaVA-ReCap-558K). These captions are generated by the LLaVA-Next-34B model Liu et al., based on images from the BLIP558K dataset. We only retain the first sentence of the detailed image captions to form the caption pool $\mathbf{C}_{\text{pool}}$.

### B.2 Textual Temporal Reasoning Data Construction

The construction process of our textual temporal reasoning datasets are illustrated in detail in Algorithm 1, 2, 3, 4, 5. The prompts used to generate question-answer pairs and captions are shown in Table 14, 15, 16, 17, 18.

### B.3 Dataset Statistics

Table 13 presents detailed statistics of our textual temporal datasets and the two baseline datasets. The information includes the number of samples, the number of relevant and distractor caption, the number of input/output tokens, and the involved modalities.

### B.4 Ablation Study of Textual Temporal Reasoning Transfer

**Table 5: Ablation study of the distractor captions and temporal reasoning transfer via images instead of captions.**
| Method | Action | Direction | Speed | Order | Attribute Change | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| LongVA-7B (32 frm) | 92.3 | 37.3 | 42.0 | 54.3 | 51.7 | 55.9 |
| + Temporal Change (w/o Distractor) | 90.8 | 40.3 | 41.0 | 61.6 | 60.4 | 59.0 |
| + Temporal Change (1x) | 91.1 | 37.3 | 39.8 | 65.2 | 63.9 | 59.5 |
| + Temporal Change (2x) | 89.6 | 39.1 | 40.4 | 66.9 | 61.8 | 59.6 |
| + Temporal Change (4x) | 90.5 | 43.0 | 39.4 | 64.2 | 64.2 | 60.4 |
| + Temporal Change (8x) | 90.2 | 42.4 | 39.8 | 64.2 | 68.1 | 61.0 |
| + Image Order | 86.4 | 36.1 | 40.4 | 55.3 | 56.9 | 55.2 |
| + Textual Order | 91.4 | 38.2 | 41.6 | 63.6 | 58.7 | 58.9 |

##### Effect of distractor captions.

We maintain the same setting as in our main paper and compare the results of the Temporal Change sets with and without distractor captions. As shown in the middle block of Table 5, incorporating distractor captions yields a more pronounced improvement in the targeted Order and Attribute Change aspects. Moreover, increasing the number of distractor captions generally leads to better performance. These results validate our approach of inserting confusing captions to enhance textual temporal reasoning transfer.

##### Transfer via images versus captions.

To investigate this difference, we replace the caption with the original image in our Order set, keeping all other settings constant. As shown in the bottom block of Table 5, textual temporal reasoning outperforms the corresponding transfer set using images (Image Order). Notably, transfer via images only marginally increases the Order performance from 54.3 to 55.4, while text format transfer significantly boosts it to 63.6.
This substantial gap corroborates our findings in the main paper that the temporal reasoning bottleneck lies on the LLM’s side, and therefore, the textual format is more effective in enhancing temporal reasoning capability.

**Table 6: Details of the videos used for our temporal analytical study in Section 2. “A$\rightarrow$B” denotes a synthesized video first showing A and then B. (black) and (white) indicate videos with black and white background, respectively. (car) and (cat) indicate videos showing cars and cats, respectively. xN denotes the number of videos. “Train Videos” is only used when training the classifier probe.**
| Dataset | # Classes | Train Videos | Test Videos |
| --- | --- | --- | --- |
| Order |  |  |  |
| Two<br>Events | 2 | c1: person$\rightarrow$cat (black) x400<br>c2: cat$\rightarrow$person (black) x400 | c1: person$\rightarrow$cat (white) x400<br>c2: cat$\rightarrow$person (white) x400 |
| Three<br>Events | 6 | c1: person$\rightarrow$cat$\rightarrow$flower (black) x400<br>c2: person$\rightarrow$flower$\rightarrow$cat (black) x400<br>c3: cat$\rightarrow$person$\rightarrow$flower (black) x400<br>c4: cat$\rightarrow$flower$\rightarrow$person (black) x400<br>c5: flower$\rightarrow$person$\rightarrow$cat (black) x400<br>c6: flower$\rightarrow$cat$\rightarrow$person (black) x400 | c1: person$\rightarrow$cat$\rightarrow$flower (white) x100<br>c2: person$\rightarrow$flower$\rightarrow$cat (white) x100<br>c3: cat$\rightarrow$person$\rightarrow$flower (white) x100<br>c4: cat$\rightarrow$flower$\rightarrow$person (white) x100<br>c5: flower$\rightarrow$person$\rightarrow$cat (white) x100<br>c6: flower$\rightarrow$cat$\rightarrow$person (white) x100 |
| Attribute |  |  |  |
| Shape | 2 | c1: blooming (black) x177<br>c2: unblooming (black) x177 | c1: blooming (white) x100<br>c2: unblooming (white) x100 |
| Brightness | 2 | c1: brightening (car) x955<br>c2: darkening (car) x955 | c1: brightening (cat) x394<br>c2: darkening (cat) x394 |
| Temporal Referring |  |  |  |
| Begin | 3 | c1: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>person$\rightarrow$flower$\rightarrow$cat (black) x400}<br>c2: {cat$\rightarrow$person$\rightarrow$flower (black) x400,<br>cat$\rightarrow$flower$\rightarrow$person (black) x400}<br>c3: {flower$\rightarrow$person$\rightarrow$cat (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400} | c1: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>person$\rightarrow$flower$\rightarrow$cat (white) x100}<br>c2: {cat$\rightarrow$person$\rightarrow$flower (white) x100,<br>cat$\rightarrow$flower$\rightarrow$person (white) x100}<br>c3: {flower$\rightarrow$person$\rightarrow$cat (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100} |
| Middle | 3 | c1: {cat$\rightarrow$person$\rightarrow$flower (black) x400,<br>flower$\rightarrow$person$\rightarrow$cat (black) x400}<br>c2: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400}<br>c3: {person$\rightarrow$flower$\rightarrow$cat (black) x400,<br>cat$\rightarrow$flower$\rightarrow$person (black) x400} | c1: {cat$\rightarrow$person$\rightarrow$flower (white) x100,<br>flower$\rightarrow$person$\rightarrow$cat (white) x100}<br>c2: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100}<br>c3: {person$\rightarrow$flower$\rightarrow$cat (white) x100,<br>cat$\rightarrow$flower$\rightarrow$person (white) x100} |
| End | 3 | c1: {cat$\rightarrow$flower$\rightarrow$person (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400}<br>c2: {person$\rightarrow$flower$\rightarrow$cat (black) x400,<br>flower$\rightarrow$person$\rightarrow$cat (black) x400}<br>c3: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>cat$\rightarrow$person$\rightarrow$flower (black) x400} | c1: {cat$\rightarrow$flower$\rightarrow$person (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100}<br>c2: {person$\rightarrow$flower$\rightarrow$cat (white) x100,<br>flower$\rightarrow$person$\rightarrow$cat x100}<br>c3: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>cat$\rightarrow$person$\rightarrow$flower (white) x100} |
| Temporal Grounding |  |  |  |
| Person | 3 | c1: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>person$\rightarrow$flower$\rightarrow$cat (black) x400}<br>c2: {cat$\rightarrow$person$\rightarrow$flower (black) x400,<br>flower$\rightarrow$person$\rightarrow$cat (black) x400}<br>c3: {cat$\rightarrow$flower$\rightarrow$person (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400} | c1: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>person$\rightarrow$flower$\rightarrow$cat (white) x100}<br>c2: {cat$\rightarrow$person$\rightarrow$flower (white) x100,<br>flower$\rightarrow$person$\rightarrow$cat (white) x100}<br>c3: {cat$\rightarrow$flower$\rightarrow$person (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100} |
| Cat | 3 | c1: {cat$\rightarrow$person$\rightarrow$flower (black) x400,<br>cat$\rightarrow$flower$\rightarrow$person (black) x400}<br>c2: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400}<br>c3: {person$\rightarrow$flower$\rightarrow$cat (black) x400,<br>flower$\rightarrow$person$\rightarrow$cat (black) x400} | c1: {cat$\rightarrow$person$\rightarrow$flower (white) x100,<br>cat$\rightarrow$flower$\rightarrow$person (white) x100}<br>c2: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100}<br>c3: {person$\rightarrow$flower$\rightarrow$cat (white) x100,<br>flower$\rightarrow$person$\rightarrow$cat (white) x100} |
| Flower | 3 | c1: {flower$\rightarrow$person$\rightarrow$cat (black) x400,<br>flower$\rightarrow$cat$\rightarrow$person (black) x400}<br>c2: {person$\rightarrow$flower$\rightarrow$cat (black) x400,<br>cat$\rightarrow$flower$\rightarrow$person (black) x400}<br>c3: {person$\rightarrow$cat$\rightarrow$flower (black) x400,<br>cat$\rightarrow$person$\rightarrow$flower (black) x400} | c1: {flower$\rightarrow$person$\rightarrow$cat (white) x100,<br>flower$\rightarrow$cat$\rightarrow$person (white) x100}<br>c2: {person$\rightarrow$flower$\rightarrow$cat (white) x100,<br>cat$\rightarrow$flower$\rightarrow$person (white) x100}<br>c3: {person$\rightarrow$cat$\rightarrow$flower (white) x100,<br>cat$\rightarrow$person$\rightarrow$flower (white) x100} |

Table: Table 7: The prompt used to generate frame captions for videos of Order, Shape Attribute, Temporal Referring and Temporal Grounding.

Table: Table 8: The prompt used to generate frame captions for the videos of Brightness Attribute. We prompt GPT-4o to determine the level of brightness before generating the frame caption, which ensures information of brightness change is reflected in the frame captions (Table 10 shows an example).

Table: Table 9: Example of a video focusing on the Order aspect and corresponding frame captions used to evaluate the LLM decoder. We show four frames here but in practice eight frames are used. The words in blue highlight the major subject depicted in each frame.

Table: Table 10: Example of a video focusing on the Brightness Attribute aspect and corresponding frame captions used to evaluate the LLM decoder. The key information of brightness change is highlighted in blue.

Table: Table 11: Example of a video focusing on the Shape Attribute aspect and corresponding frame captions used to evaluate the LLM decoder. The key information of shape change is highlighted in blue.

**Table 12: Training hyper-parameters for the classifier probe.**
| Temporal Aspect | Learning Rate | Batch Size | #Epoch | Optimizer |
| --- | --- | --- | --- | --- |
| Order (Two Events) | 5e-5 | 64 | 15 | Adam |
| Order (Three Events) | 5e-5 | 64 | 120 | Adam |
| Attribute | 5e-5 | 64 | 80 | Adam |
| Temporal Referring | 5e-5 | 64 | 120 | Adam |
| Temporal Grounding | 5e-5 | 64 | 120 | Adam |

**Table 13: Dataset statistics. 1x$\sim$8x indicates the length of the textual context, controlled by the number of distractor captions. #tokens are counted using LongVA-7B tokenizer.**
| Dataset | #Samples | #Relevant<br>Captions | #Distractor<br>Captions | #Input<br>Tokens | #Output<br>Tokens | Modalities |
| --- | --- | --- | --- | --- | --- | --- |
| Order-GPT (1x) | 16k | 2$\sim$4 | 100$\pm 50$ | 1.8k | 13.6 | Text |
| Order-GPT (2x) | 15k | 2$\sim$4 | 200$\pm 50$ | 3.5k | 13.6 | Text |
| Order-GPT (4x) | 15k | 2$\sim$4 | 400$\pm 50$ | 6.9k | 13.5 | Text |
| Order-GPT (8x) | 15k | 2$\sim$4 | 800$\pm 50$ | 13.7k | 13.6 | Text |
| Attribute (1x) | 34k | 2 | 100$\pm 50$ | 1.8k | 8.4 | Text |
| Attribute (2x) | 15k | 2 | 200$\pm 50$ | 3.5k | 8.4 | Text |
| Attribute (4x) | 15k | 2 | 400$\pm 50$ | 6.9k | 8.3 | Text |
| Attribute (8x) | 15k | 2 | 800$\pm 50$ | 13.7k | 8.3 | Text |
| Order-Template (phrase) | 30k | 3$\sim$6 | 200$\pm 50$ | 3.6k | 17.1 | Text |
| Order-Template (prefix) | 30k | 3$\sim$6 | 200$\pm 50$ | 3.6k | 15.8 | Text |
| Order-Template (sentence) | 30k | 3$\sim$6 | 200$\pm 50$ | 3.6k | 77.1 | Text |
| Referring | 22k | 3 | 200$\pm 50$ | 3.5k | 4.5 | Text |
| Grounding | 22k | 3 | 200$\pm 50$ | 3.5k | 8.4 | Text |
| Hotpot QA | 90k | - | - | 1.4k | 4.0 | Text |
| LLaVA-Next | 1M | - | - | 36.5 | 57.7 | Text+Image |

Figure: Figure 7: Detailed temporal analysis results of LongVA.
Refer to caption: x11.png

Figure: Figure 8: Detailed temporal analysis results of VILA.
Refer to caption: x12.png

Figure: Algorithm 1 Textual temporal QA generation for the Order aspect using GPT-4-turbo. $\operatorname{ExtCont(\mathbf{C}_{\text{pool}},\mathbf{C},n)}$ inserts relevant captions $\mathbf{C}$ in between $n$ distractor captions and ensures that every pair of relevant and distractor caption do not share common nouns.

Figure: Algorithm 2 Textual temporal QA generation for the Order aspect using templates.

Figure: Algorithm 3 Textual temporal QA generation for the Attribute aspect.

Figure: Algorithm 4 Textual temporal QA generation for the Temporal Referring aspect.

Figure: Algorithm 5 Textual temporal QA generation for the Temporal Grounding aspect.

Table: Table 14: The prompt used to generate textual temporal QA for the Order aspect.

Table: Table 15: The prompt used to generate similar captions by modifying attributes in the original caption.

Table: Table 16: The prompt used to generate textual temporal QA for the Attribute aspect.

Table: Table 17: The prompt used to generate similar captions for the Temporal Referring aspect.

Table: Table 18: The prompt used to generate declarative statements for the Temporal Grounding aspect.

## Appendix C Training Details

### C.1 Training Datasets

**Table 19: Datasets used in our textual temporal transfer explorations in Table 2.**
| Dataset | # samples | Description |
| --- | --- | --- |
| Open-LLaVA-Next | 22k | LLaVA-Next image-text SFT dataset. |
| HotpotQA | 22k | Multiple-document question answering dataset. |
| Temporal Change (w/o Distractor) | 22k | Balanced mixture of Order-GPT and Attribute without interesting distractor captions |
| Temporal Change (1x) | 22k | Balanced mixture of Order-GPT (1x) and Attribute (1x) |
| Temporal Change (2x) | 22k | Balanced mixture of Order-GPT (2x) and Attribute (2x) |
| Temporal Change (4x) | 22k | Balanced mixture of Order-GPT (4x) and Attribute (4x) |
| Temporal Change (8x) | 22k | Balanced mixture of Order-GPT (8x) and Attribute (8x) |
| Temporal Change (1-8x) | 22k | Balanced mixture of Temporal Change 1x, 2x, 4x and 8x |
| Order-Template | 22k | Balanced Mixture of Order-Template (phrase), (prefix) and (sentence) |
| Temporal Referring | 22k | Synthesized textual samples for enhancing Referring |
| Temporal Grounding | 22k | Synthesized textual samples for enhancing Grounding |
| Order-Template + Temporal Change (1x) | 22k | Balanced mixture of Order-Template and Temporal Change (1x) |
| Temporal Grounding + Temporal Change (1x) | 22k | Balanced mixture of Temporal Grounding and Temporal Change (1x) |
| Temporal Referring + Temporal Change (1x) | 22k | Balanced Mixture of Temporal Referring and Temporal Change (1x) |
| T3 | 22k | Balanced Mixture of all our synthesized subtasks. |

**Table 20: Datasets used for transferring to holistic video understanding benchmarks in Table 3 and Table 4.**
| Dataset | # samples | Description |
| --- | --- | --- |
| LLaVA-Next | 200k | LLaVA-Next image-text SFT dataset. |
| HotpotQA w/ LLaVA-Next | 200k | Hotpot QA (66.7k) + + LLaVA-Next (13.3k) |
| T3 w/ LLaVA-Next (Ours) | 200k | Temporal All (66.7k) + LLaVA-Next (13.3k) |

Table 19 provides details of the training datasets used in our main paper. To ensure fair comparison across all compositions, we set the number of training samples to 22k, which is the maximum available in the smallest subsets (Referring and Grounding).
For the transfer evaluation on MLVU and Video-MME, we combine the original LLaVA-Next image-text SFT dataset with our textually synthesized samples, as shown in Table 20. The mixing ratio and total sample size are determined based on the exploration results discussed in the following subsection.

### C.2 Dataset Mixing Explorations

Figure: Figure 9: (Left) Exploration of mixing ratios between image-text instruction tuning and textual temporal QA. (Right) Dataset scaling analysis.
Refer to caption: x13.png

Our preliminary study revealed that training the model exclusively on large-scale textual temporal QA samples led to catastrophic forgetting of visual perception abilities, degrading it to a text-only model.
To mitigate this issue, we explore the integration of original instruction tuning datasets to enhance video understanding transfer performance.
Specifically, we combine the original LLaVA-Next dataset with textual temporal QA pairs from our All Temporal subset, varying the mixing ratio from 1 to 20, while fixing total samples to 100k for fair comparison.
We use textual validation accuracy as our selection criterion, as it correlates well with temporal understanding ability, as discussed in our main paper.

Figure 9 (left) illustrates that performance on the textual component gradually decreased as more image-text instruction tuning samples are included. Based on these results, we selected a ratio of 2, as it maintains relatively high textual validation accuracy (¿ 95%) while incorporating sufficient visual-text samples to preserve the model’s visual capabilities.

We further explore the optimal sample size for the mixed dataset, with results shown in Figure 9 (right). As expected, textual accuracy continuously improved with larger training samples. Consequently, we set the total sample size to 200k, as this threshold first exhibits saturated textual validation accuracy while keeping the sample size manageable for efficient training.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - Are the temporal-oriented questions generated from a different distribution from the answer choice distribution of the video benchmarks? For example, are the orders in which the objects appear in the temporal-oriented questions not in the benchmark datasets? Does this still improve the performance when the training data is manipulated to be strictly out-of-distribution to the benchmark data?
- What is the performance of the fine-tuned video LLMs on the existing video benchmarks, when they are only provided the text part of the question? 
- If there is an analysis of the number of data points provided and the performance gain, it could provide a better insights to the validity of this approach.

The first two questions were critical in my rating decision.
soundness: 2
strengths: - Interesting approach to tackle temporal understanding of Video LLMs, that the LLM fails to understand the temporal behavior of the text prompts. 
- Utilizing only text data proposes an efficient and scalable framework, although the gains are questionable for larger models.
- The performance gains are quite significant
summary: This paper addresses the limitations of video LLMs in temporal reasoning, which involves tracking objects or events in a sequence of video frames. The authors discover and point out the deficiencies stemming from the limitations of LLM processing. Inspired by this, the authors present a text-based framework which can be used to finetune the LLM part of the video LLM, which can result in improved temporal understanding from text-only data. This approach enables smaller models to outperform larger, video-trained models on benchmarks such as TempCompass and Video-MME.
weaknesses: - The biggest limitation of this work lies in the scope of the temporal concepts, but not simply because they were not covered in the paper. For example, consider rotation, direction, counting the number of a certain event, and relative velocity of an object relative to another object. There are some temporal concepts that cannot be represented in discrete sentences - for example, how would you describe the number of rotations of a diver in the form of text? How would you describe that one person is running faster than the other one without giving out the information explicitly?
- The above concern also brings up the question of whether the success of this approach is largely due to how video LLMs are tested on current video benchmarks: given a few discrete frames, the models are asked to answer a question. For example, a good video LLM would solve problems in "Order" when given a frame of book cover, die, school bus and then split-screen (example in figure 4). If the vision encoder encodes this information but the output quality is bottlenecked by the LLM, this method would improve the model. However, it is unclear whether this method will apply to questions that require virtually the entire video frames to be answered.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See comments above.
soundness: 3
strengths: 1. Show that visual features contain necessary information for order understanding task.
2. Writing is clear.
summary: This paper tried to use text-only reordering style data to fine-tune MLLM to enhance the temporal ordering performance.
weaknesses: 1. Why choosing LongVA? LongVA is only trained on images and text, therefore, evaluating and improving LongVA but evaluating on video understanding is not proper. Because everything is zero-shot manner. Instead, choosing backbones like LLaVA-onevision, phi-3.5-vision, or Qwen2-VL is proper. Those models have videos as the training data. If authors still show that there is clear improvement with their text-only ordering data, then the claim can be supported. 
2. Probing visual features is unfair. For LLM (decoder only), authors did not fine-tune. For visual encoder features, authors fine-tuned. Since the task diversity is limited, therefore, overfitting to this fixed task is not hard. This also explains that a LSTM can overfit this task. The interesting thing should be: if you use LSTM after LLM features, can LSTM also overfits? 
3. The empirical part of this paper demonstrate one thing: using text-only ordering data can improved relatively small LLM backbones. A more foundational question is: since large LLM can already solve the ordering task pretty well, is authors’ approach just a temporary  solution? Can authors’ approach improves Qwen-2-VL-72B?
4. Evaluation limited. Evaluations on benchmarks like NextQA, ActivityNet, etc is still necessary. This is because ordering task is naturally just a small subset of temporal reasoning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1) In terms of significance, this paper addresses the problem of video and language reasoning in multimodal Large Language Models (MLLMs) from a new perspective. It attributes the limitation of existing (MLLMs) to the LLM component instead of the video or adaptor functions. This is a very interesting perspective since it proposes an approach that improves the LM's ability to understand temporal concepts through text. The proposed Textual Temporal Reasoning Transfer (T3) approach creatively uses text-only datasets to improve temporal reasoning without requiring any video-language annotations.

2) The quality of the paper is relatively high, especially with regards to the empirical analysis of the bottleneck in video and language reasoning. The experiments conducted in the analysis are well motivated. The finding that the performance of existing MLLMs is constrained by their LLM decoders instead of the visual encoding function is insightful.

3) In terms of clarity, the paper is well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the differences between questions that focus on different temporal reasoning abilities in videos.
summary: The paper attempts to address current limitations in temporal reasoning of Video Large Language Models (Video LLMs). Current models still often struggle with understanding temporal sequences, despite advances in video comprehension. Through a detailed analysis, the authors empirically demonstrate that the bottleneck primarily lies in the LLM's capability to understand temporal concepts. This is in contrast to a multitude of concurrent work that often aim to improve the temporal encoding or adaptor functions. To address this limitation, this paper also introduces the Textual Temporal Reasoning Transfer (T3) approach, that generates temporal reasoning tasks solely in textual format. The experiments reported in the paper demonstrates the effectiveness of the approach where it obtains performance gains in multiple video question answering benchmarks, without requiring additional video and language annotations for training.
weaknesses: 1) The text-based tasks used in T3 are largely devised based on templates and may not always be similar to the language concepts that are contained in real-world videos. For example, such templated sentences may lack the subtle temporal cues that naturally occur in descriptions of complex events. This might affect the model's ability to generalize to different downstream tasks.

2) In this work, the authors only address four temporal aspects: order, attribute change, temporal referring, and grounding. However, this excludes some other common aspects of temporal reasoning such as understanding causal relationships and durations. Including more training samples that require more complex forms of reasoning could improve robustness of the trained models across a broader spectrum of temporal tasks.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the questions I listed before, there are two additional questions for the authors:

1. How to explain the strong temporal reasoning ability of GPT-4o? Is it only because 4o has a very strong temporal reasoning ability from the language side, not related to its vision encoder or alignment? 

2. Is there any way you can conduct the probing experiments on real datasets, not the synthesized ones?
soundness: 2
strengths: 1. The paper is well structured and written, making it easy to understand its key points. 

2. The enhanced model, built on LongVA-7B, has shown strong performances, surpassing large-scale models such as InternVLChat-V1.5-20B and VILA1.5-40B.
summary: The paper studies the video's large language models, particularly its temporal reasoning ability. It first designs probing experiments using synthesized videos and analyzes the bottleneck in temporal understanding: primarily on LLMs but not the vision encoder.  To mitigate the issue, the authors propose a simple text data augmentation method to leverage the image-text datasets to generate diverse
temporal reasoning tasks and enhance the video LLMs training. The experiments based on LongVA-7B show its competitive performances across different datasets.
weaknesses: 1. The claims of the paper do not fully convince me of the paper made. I agree that most of the vision encoders used today do not help much with temporal reasoning, even though they are trained with millions of videos. However, this can not be claimed as "the temporal reasoning most of rely on the temporal reasoning ability from LLMs. There might be reasons that the temporal reasoning also replies on the model alignment ability between the text and the images/videos. The authors should design extensive prompting experiments on more models and datasets to support their strong claims. 

2. The synthesized tasks are mainly based on GPT-4o. And for me, how to design the synthesized tasks seems very arbitrary and empirical. Are there any particular reasons you designed these tasks and how can you ensure its generalization on more domain/datastes? The authors should have more deep analysis and experiments on those two issues.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Could you report results on larger LLMs (e.g., Qwen2-72B)? Although the performance gain is noted to be limited, these results could be informative for future research.
2. Could you provide examples of captions generated by GPT-4o as mentioned in L-133? It would help clarify whether specific attribute information is inadequately encoded in the caption, potentially leading to near-random results in LLM responses. (You partial addressed this in L216-L217, but the performance achieved by larger model is still 20% percent lower than the performance achieved by LSTM, which makes it not convincing).
3. Minor issue: It appears LLaVA-ReCap-558K is derived from LCS-558K (a subset of the LAION/CC/SBU dataset) rather than BLIP558K (as stated in lines 861-862) [1].
4. I question whether the performance improvement is due to enhanced instruction-following ability (e.g., the ability to "Answer the option only") rather than true temporal reasoning. Could you report the portion of mis-formatted or invalid responses across all experiments to clarify this?

[1] Section on Open Source Release: https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/
soundness: 3
strengths: 1. The authors provide a well-defined motivation for their work, verifying through experiments that the LLM, rather than the VLM, is the limiting factor in video understanding tasks, and proposed solutions.
2. The authors introduce an innovative fine-tuning technique that only tunes the LLM within a VLM, bypassing the need for visual data. This approach yields significant performance improvements across several video understanding benchmarks.
3. The dataset is generated in a fully automated manner, adding to the scalability of their approach.
4. Extensive experiments and in-depth analysis are provided.
summary: This paper addresses the limitations in Vision Language Models (VLMs) for temporal reasoning tasks in video understanding. Through detailed probing experiments, the authors identify the primary bottleneck as the LLM component rather than the visual encoder. To address this, they introduce an approach called T3, which fine-tunes only the LLM component on text-only data, enhancing performance on several temporal reasoning benchmarks.
weaknesses: 1. As mentioned in L-538, the approach shows minimal improvement when applied to larger LLMs, which could restrict its broader applicability.
2. Several areas of the paper would benefit from further clarification (see questions below).

Meta-review: This work presented an innovative perspective about the temporal reasoning ability for multimodal LLMs. The authors found that the drawback of temporal reasoning ability mainly stems from the weak performance of LLMs in this aspect. Starting from this observation, the authors curated a new text-only training data to teach the LLMs for temporal reasoning, and thus benefit the following multimodal temporal reasoning for video inputs. The results in the experiments and rebuttal session are solid and persuasive to demonstrate the effectiveness of the proposed method.

The main contribution of this work lies on the new insightful observations that the temporal reasoning capability of multimodal LLMs stems from that of LLMs. The authors solidly explained the methodology and conducted extensive experiments to demonstrate the effectiveness of the proposed method. After reading through the whole discussions between the authors and reviewers, the ACs think this is a great work to point out how to improve the highly-demanding capability in current multimodal LLMs for video understanding.

A few drawbacks were pointed out by the reviewers, and some of them are crucial. One of them is whether the LLMs do learn the temporal reasoning ability rather than memrozing the texts corpus. The authors did a great job to show that it is due to the latter reason for which we see the improvement of temporal reasoning capability. The auhors was initially challenged whether the proposed method can generalize to other benchmarks and LLM backbones. These were later also addressed well with solid experimental results. In the end, most of the concerns are addressed well, which lead to all positive ratings on this work.

To conclude, the ACs agree with the reviewers that this work shed some new lights on how to enhance the temporal reasoning capability for multimodal LLMs, which as we all know are crucial. Instead of approching this problem by constructing image-text instruction tuning data, the authors started from some intriguing observations and build a new method from the langauge perspective, whcih is quite insightful. As such, the ACs recommend an acceptance to this work.

Decision: Accept (Poster)