Temporal Reasoning Transfer from Text to Video
Authors: Lei Li
Yuanxin Liu
Linli Yao
Peiyuan Zhang
Chenxin An
Lean Wang
Xu Sun
Lingpeng Kong
Qi Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Video Large Language Models
Temporal Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We identified the LLM backbone is the bottleneck of Video LLMs for temporal reasoning and developed a textual temporal reasoning transfer method.

Abstract: Video Large Language Models (Video LLMs) have shown promising capabilities in video comprehension, yet they struggle with tracking temporal changes and reasoning about temporal relationships.
While previous research attributed this limitation to the ineffective temporal encoding of visual inputs, our diagnostic study reveals that video representations contain sufficient information for even small probing classifiers to achieve perfect accuracy.
Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning capability stems from the underlying LLM's inherent difficulty with temporal concepts, as evidenced by poor performance on textual temporal question-answering tasks.
Building on this discovery, we introduce the Textual Temporal reasoning Transfer (T3). 
T3 synthesizes diverse temporal reasoning tasks in pure text format from existing image-text datasets, addressing the scarcity of video samples with complex temporal scenarios. 
Remarkably, without using any video data, T3 enhances LongVA-7B's temporal understanding, yielding a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark, which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video samples.
Additionally, the enhanced LongVA-7B model achieves competitive performance on comprehensive video benchmarks. For example, it achieves a 49.7 accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. 
Further analysis reveals a strong correlation between textual and video temporal task performance, validating the efficacy of transferring temporal reasoning abilities from text to video domains.

Published as a conference paper at ICLR 2025
TEMPORAL REASONING TRANSFER FROM TEXT TO VIDEO
Lei Life Yuanxin Liu
Linli Yao
Peiyuan Zhang Lingpeng Kong' Qi
Chenxin An
Lean Wang² Xu Sun
The University of Hong Kong Peking University University of California, San Diego nlp lilei@gmaillll com {licyuanxin, linliyao yyo)estu pku. eada.ddu.com cn pez010@ucsd, edu exan23@conneod hku. hk
[lean,
en {1pk, liugi Jocs jec.ijk..hhhhh
ABSTRACT
Video Large Language Models (Video LLMs) have shown promising capabilities in video comprehension, yet they struggle with tracking temporal changes and reasoning about temporal relationships. While previous research attributed this limitation to the ineffective temporal encoding of visual inputs, our diagnostic study reveals that video representations contain sufficient information for even small probing classifiers to achieve perfect accuracy. Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning capability stems from the underlying L.M's inherent difficulty with temporal concepts, as evidenced by poor performance on textual temporal prestion-answeringg tasks. Building on this discovery, introduce the Textual Temporal reasoning Transfer (T3). T3 synthe- sizes diverse temporal reasoning tasks in pure text format from existing image-text datasets, addressing the scarcity of video samples with complex temporal sce- narios. Remarkably, without using any video data, T3 enhances LongVA-7B s temporal understanding, yielding 5.3 absolute accuracy improvement on the challenging TempCompass benchmark, which enables our model to outperform ShareGPT4Video--a trained on 28,000 video samples. Additionally, the enhanced LongVA-7B model achieves competitive performance on comprehensive vidco benchmarks. For example, achieves 49.7 accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful large-scale models such as Intern Chat-V Soat-V1.5-2000 and VILAI.5-40B. Further analysis reveals a strong correlation between textual and video temporal task performance, validating the efficacy of transferring temporal reasoning abilities from text to video domains."
INTRODUCTION
The rapid development of large language models (LLMs) (OpenAI, 2024; Gemini Team, 2024) has sparked significant interest video large language models (Video LLMs) (Zhang et al., 2023; Lin al., 2023b) due to their impressive generation and reasoning capabilities. Current approaches typi- cally use pre-trained vision encoders (Radford et al., 2021) combined with powerful L.LSs (Touvron al., 2023; Chiang et al., 2023; Yang al., 2024) the starting point for Video LLMs. These models employ various strategies to handle multiple video frames (Li et al., 2023b; Tan et al., 2024), and are then further trained on curated instruction-tuning datasets (Chen al., 2024), demonstrating promising abilities in video comprehension tasks (Fu et al., 2024; Zhou et al., 2024).
Despite the progress in video comprehension, Video LLMs often struggle with temporal reasoning, which is essential for truly interpreting video content (Li et al., 2023c; Tang et al., 2023b). Specifically, video temporal reasoning involves the ability track changes over time, comprehend event sequences, and relate objects and actions to specific moments in a video (Mangalam et al., 2023; Li et al., 2023d). As illustrated in Figure 1, two strong Video LLMs, Long VA-7B (Zhang et al., 2024b) and VILA- 8B (Lin al., 2023b), both failed to answer basic questions about the chronological order of events
"Equal contribution.
Project page: https::/ https://video-tad tttp:///viooo....tttu.......

---

Published as a conference paper at ICLR 2025
Textual Temporal Training
The irage antia paran daciig This imsia prese no city stue.
Vileo HAME
the vdec?
sposo
Qusetian Arange the follow hame connct oidar sever corket pemor,,
person can Nower I70 centas, liwser cor (P) Nownr
room parion parrecs roor Etrstvel rooms piition SAXT
:10
Lengla-78.
Langhh-78-13 (Dum) pripetoo
Wieee Po4P
JkM
VILA-LLMAA-- MB The cornect ordir that too tern
opper 1hs who or, frowr swaas X
Video Temporal Inference
MUN
Longi (Dam) X4 veL panee, Brver.
Longuh-Tb17 (Dum))
Two popular Video L.MSs struggle with basic temporal reasoning (left). We mitigate this issue via textual temporal transfer (middle), which demonstrates consistent improvement (right).
Figure I:
in synthesized videos, whereas humans can predict correctly without difficulty. This significant gap between human performance and current Video LLMs temporal reasoning tasks motivates us to explore the underlying reasons for this discrepancy.
Previous research has largely attributed temporal reasoning deficiencies in Video LLMs to ineffective video encodings, leading to various temporal aggregation module developments (Ren et al., 2023; Jin et al., 2023; Tan et al., 2024). Our paper takes a different approach by decomposing Video LLMs into two parts and asking a fundamental question: What is the bottleneck of this limitation? Is it due to limitations in the vision encoder, or, surprisingly. shortcomings in the LLM itself? We conduct probing experiments using synthesized videos for basic cemporal-related video question-ane -enswiinnnnsweiing (QA) tasks, allowing full control over temporal aspects, Our method involves: (i) Training small probe classifiers on video representations in the Video LLM embedding space assess temporal information captured by visual encoders and aggregation modules. (ii) Transforming synthesized videos into textual descriptions using commercial visual language models (e.g.. GPT-4o) to analyze how standalone LLMs process temporal information. By comparing the performance of these components with full Video LLMss we could precisely locate the bottleneck in temporal understanding.
Our experiments reveal a striking contrast in temporal reasoning capabilities between different com- ponents of Video LLMs. Probe classifiers trained on video embeddings achieve near near-perfect accuracy (> 90% in most cases), indicating that these embeddings successfully capture sufficient temporal in- formation. Even simple neural models like LSTM (Hochreiter Schmidhuber, 1997) can accurately extract temporal relationships from these embeddings. Conversely, despite their significantly larger scale, LLMs struggle to process this temporal information effectively. They exhibit relatively low probing accuracy across various temporal aspects, highlighting their difficulty in handling temporal relationships. These findings provide compelling evidence that the LLM component, rather than the visual encoding, is the primary bottleneck in the temporal reasoning of current Video LLMs, motivating us to enhance LLMs' ability to reason about temporal information.
Building on our insights, we propose enhancing Video LLMs' temporal understanding by focusing on the LLM component. Our approach leverages existing image-text datasets to generate diverse temporal reasoming tasks in pure text text format, overcoming the scarcity of video samples with complex temporal scenarios ($3). Remarkably, without using any video data, our text-only synthesized dataset enables Long VA-7B to outperform ShareGPT4Video-Se
(trained on 28, 880000 video samples) on the TempCompass benchmark (Liu et al., 2024b). Moreover, as shown in the right of Figure 1, our enhanced model demonstrates competitive results on various video-unde dndeo-nnderttnndnng benchmarks. improves temporal reasoning accuracy by 12.4 points on Video-MME (Fu et al., 2024) and increases average accuracy from 56.4 to 58.1 on MLVU (Zhou et al., 2024), surpassing larger models such as Intern VL-Chat- V1.5-20B (Chen et al., 2023) and VILA-1.5-40B (Lin et al., 2023b). Analysis reveals the crucial role of self-attention modules in temporal reasoning transfer, and our method improves the utilization of more video frames and transfers to subtle temporal aspects well (Shangguan et al., 2024; Zhang et al., 2024a).
PINPOINTING VIDEO LLM TEMPORAL. REASONING BOTTLENECK
In this section, we seek to examine and pinpoint the temporal reasoning bottleneck of Video LLMs. Video LLMs typically consist of two essential components for temporal understanding tasks: vision
2

---

Published as a conference paper at ICLR 2025
encoder and an LLM decoder, where the former extracts visual features from video frames and the latter is responsible for integrating this information with textual instructions to complete the end task. We aim to answer two questions: (1) Can existing Video LLMs understand the temporal information in videos? (2) If not, which component-the vision encoder or the I.LM decoder-is is the bottleneck? To address these questions, we design different tasks to test the full Video L.LMs and these two components separately ($2.1), incorporating various aspects of temporal understanding abilities ($2.2), and discuss our findings ($2.3).
2.1 TASK FORMULATION FOR DIFFERENT VIDEO LLM COMPONENTS
Full Video LLM. We evaluate the full Video LLMs through multi-choice video question answering. Specifically, we uniformly sample cight frames from videos and present them to the model along with multiple-choice question. To encourage the Video LLM directly output an option, we append the prompt "Answer the option only." after the question.
LLM Decoder. We replace the video frames with detailed frame captions generated by GPT-4o. In this way, we assess the ability of LLM decoder to understand temporal information in the textual context. The multi-choice questions identical to those employed in testing the full Video LLM. To ensure that the temporal understanding questions can indeed be addressed using these frame captions, we carefully design the prompting strategy to incorporate all essential information in the captions (please refer to Appendix A.2 for the implementation details and examples of frame captions.)
Visual Features. Unlike the full Video LLM and LLM decoder, the quality of visual features cannot be directly tested via question answering. To determine whether the visual features contain sufficient information to differentiate between different temporal dynamies in videos (e.g., brightening versus darkening), we employ the "classifier probe" technique proposed by (Alain & Bengio, 2017). Assuming that there are e categories of contradicting temporal dynamics, we train a "probe" s() that maps set of visual features V E Raxd, to probability distribution p RS In practice, f() is single-layer LSTM model (Hochreiter & Schmidhuber, 1997) for capturing sequential correlation, and we employ the visual features that are down-sampled and projected into LLM embedding space. The probe is tested on the same set of videos as the full Video LLM but is trained on a different set of videos. More details of the visual classifier probe can be found in Appendix A.3.
2.2 EVALUATION DATA COLLECTION
Temporal Reasoning Abilities. Temporal reasoning encompasses several key aspects. Based on recent Video LLM benchmarks (Fu et al., 2024; Wu et al., 2024), we focus on four critical dimensions: (I) Order: comprchending the sequential arrangement of events; (2) Attribute: perceiving changes in environmental or object attributes over time; (3) Temporal Referring: formulating questions based on specific temporal positions within video; and (4) Temporal Grounding: identifying the
temporal location of specific elements in a video. While these aspects are a limited approximation of the full spectrum of temporal understanding, our study reveals that they sufficient expose significant deficiencies in current Video LLMs.
Collecting Videos. Existing video benchmarks are unsuitable for our analysis due two main limitations: (I) the lack of video clusters with contrasting temporal dynamics needed for visual feature probing, and (2) the inability fully eliminate single-frame or language bias (Liu et al., 2024b). Consequently, we synthesize custom videas to isolate different aspects of temporal understanding. Figure 2 illustrates examples of our synthesized videos. For the Order aspect, we concatenate different source videos (e.g., person, cat, flower) temporally, creating various categories based on concatenation order. The Attribute aspect focuses on shape (flower blooming videos and their reversals) and brightness (gradually altering pixel values of static images). Temporal Referring and Temporal Grounding reuse videos from the Order aspect with three concatenated items. Appendix A provides detailed information on the video creation process and data distribution.
Constructing Questions and Answers. As depicted in Figure 2, we design multi-choice question templates for each temporal aspect. The questions remain consistent across videos within each aspect, while the correct answers vary based on the specific video content. This approach ensures a controlled evaluation of temporal reasoning capabilities.
3

---

Published as a conference paper at ICLR 2025
Order
Feets
Terpporal Referring
Figure 2: Example of videos and questions focusing on different temporal reasoning abilities.
Attribute
Referring
Grounding
Order
Lar Giferiile
Accumity
Figure 3: Temporal probing results for Long VA (upper) and VILA (lower). The probe visual representations achieves 90 accuracy in most cases, while the LLM decoders still have large room for improvement with textual inputs, leading to the poor temporal understanding ability of Video LLMs. Detailed results the sub-categories are reported in Appendix A.4.
2.3 RESULTS
We examine two advanced Video LLMs that utilize different LLM backbones probing datasets: LongVA-7B (Zhang et al., 2024b), which based on Qwen2-7B (Yang et al., 2024), and VILA-8B (Lin al., 2023b), which is built upon LLaMA3-8B (Dubey al., 2024). The results are visualized in Figure 3, with more results reported in Appendix A.4. Our analysis reveals several
synthesized
key findings:
Video LLMs demonstrate relatively poor performance, struggling to reach 70 accuracy across all temporal understanding tasks. This suggests potential limitation in their ability to process and reason about temporal information in video content.
Performance of Video LLMs:
Efficacy of Visual Representations: Notably, small classifiers trained the visual representations in the LLMs' embedding space achieve near-perfect accuracy. This finding suggests that these visual representations encapsulate rich information, sufficient distinguish between temporally contradicting videos, even with a simple probe classifier. Consequently, we can conclude that the input processing is not the primary limitation in temporal reasoning tasks.
LLM Backbone Performance: To our surprise, moderate-sized LLM backbone decoders such as Qwen2-7B and L.LaMa3-BB fail to answer a considerable number of questions related Referring, Grounding and Order aspects, which are intuitively very easy for SoTA LLMs given that the frame captions are provided textual format. Worse still, these LLM decoders perform only at the level of random guessing when it comes the Attribute aspect. These findings indicate that the LLM decoders face substantial challenges even in the context of textual temporal understanding, posing major limitation to the video temporal understanding capabilities of Video LLMs. In comparison, larger LLMs with over 70 billion parameters significantly outperform their smaller counterparts these tasks, suggesting that (i) the generated frame captions accurate and contain
4

---

Published as a conference paper at ICLR 2025
Order
Attribute
Textua Conkest:
The image cboaca hook couer with a predoninarty dark oslor otherta
I arape shons akded
colorti Bust ontiontiii achool
ur ondgas
The shows young whiigg the roor,
Terhua Conhect:
child
anage aia opiiviraan comporioon fach incuudd
Hone doea Fhr lutting
pound?
Vi hacomes brightr menoine langged becomnet
What the order of imagee
the video?
actool
Techual Conhest:
hook nover die, school bus. opliniziion
servo loas difo actson doci soind
T1 kmage showw peraer weating a light litinn frie polo atart
Q Armuga the images
wit acconding the
Now dons the polo change??
(1ppiie Bi P4
Fn
Temporal Referring
Temporal Grounding
Tecthul Contest::
Confext:
The image ensuee fun oquans bioc coine with rammont angrosed on then
The image chour fio aggrrs trakk coine with numbort or organved on them cosigr puetad
smage anone len pliler with ciatgma panked uru
The imag onon lic unail stow with etccams.co
round
them.
When the
(N bock coliee plate appiid
(M the bag
(1) the in call and
oteaota gee shown iihee meddre
(6) midill
What sdpada shown image and wole??
Cr When the video dan adeance the appleer are mech ongiin (8) midil und
(V brick coloe 30) platic (2) Ind applis barounee
Figure 4: Temporal-oriented guestion-answering pairs with textual image captions as context.
sufficient information for temporal reasoning: (ii) the textual temporal understanding ability emerges when the text decoder scale exceeds certain thresholds. We hypothesize that this discrepancy may be due to the sparse expression of temporal concepts in pre-trained corpora. Consequently, smaller-scale models might not have sufficient exposure to learn these temporal reasoning abilities effectively. This spansity could explain why only very large language models (> (0BB parameters) demonstrate proficiency in temporal reasoning tasks.
Our findings ultimately suggest that the temporal reasoning ability in current Video LLMs is primarily constrained by the LLM decoder rather than the quality of visual representations. This conclu- sion highlights critical area for improvement in the development of more effective Video LLMs, motivating us to enhance the temporal reasoning capabilities from the textual side.
3 TEXTUAL TEMPORAL UNDERSTANDING TRANSFER
In this section, we explore strategies to mitigate the deficiency of Video L.MSs in temporal reasoning. straightforward approach would to create temporal-oriented instruction-tuning dataset from videos. However, existing approaches to video instruction-tuning data generation, whether through human annotation or automatic generation via rules/models, present significant challenges. Human annotation of video datasets is resource-intensive, demanding considerable time and financial in- vestment. Synthetic video instruction tuning (Maaz et al., 2024; Zhang et al., 2024c), while more efficient and scalable, is constrained by the content of the videos themselves. For instance, if our goal is to enhance a particular aspect of temporal reasoning, it is crucial to ensure that the videos contain information relevant this aspect. This limitation makes it difficult to create instruction -ttnung data that targets specific abilities.
Motivated by (1) the limitations of video instruction-tuning and (2) our finding that the LLM backbone is the primary bottleneck of temporal understanding, we investigate the feasibility of enhancing video temporal reasoning through a novel perspective: using synthesized textual temporal reasoning data. Our method uses sequences of image captions proxies for video frames, allowing us to create cemporal-oriented question-amee -anseernn pairs without relying on actual video content. This text-only approach offers two key advantages: (i) Scalability: The abundance of available image- caption pairs (Changpinyo et al., 2021; Liu et al., 2023) allows for easy expansion of the dataset. (ii) Flexibility: We can precisely control the targeted ability and sample complexity by adjusting the content and number of image captions. To address the four aspects of temporal understanding (Order, Atrriute, Referring and Grounding), we design heuristics to construct textual contexts and generate question-answe answer pairs solely from caption sequences with examples visualized in Figure 4. follows.
The specific generation processes
Order: This aspect focuses on understanding the sequential order of image captions. To construct the textual context for question answering, randomly sample 3~66 captions from caption pool.
5

---

Published as a conference paper at ICLR 2025
Table 1: Summary of our textual temporal reasoning datasets where X E {phrase, prefix, sentence] for Order-Temp late (X). Detailed data statistics are provided in Appendices B.3 and C.I.
Dataset
#Relevant Captions
#Distractor Captions
Deseription
Order-GPT (Nx
2nu4
N x 100 = 50, N (1. 4,8)
Order-related questions generated GPT-4. Attribate-relatdd questions.
Attribute (N x)
N x 100 + 50, N E 4,8)
Order-Template
3-66
200::50
Order-related questions based on templates Temporal referring questions.
Referring
3
200+
Grounding
200:1:00
Temporal grounding questions.
The questions are created using two methods: On the one hand, we provide GPT-4-turbo with examples and prompt it to generate order-related questions, which we refer to as Order-GPT. On the other hand, we employ predefined templates and heuristic rules to create questions. These template questions require rearranging shuffled sequences according to the textual context of image captions. These sequences comprise of (a) complete image captions, (b) phrases within captions or (c) phrases with prefix identifiers (e.g., (1)(2)(3)(4) as shown in the example in Figure 4). We denote the template-based questions as Order-Template (X), where X E {sentence, phrase, prefix).
Attribute. This aspect involves recognizing how specific attributes of objects or scenes change throughout the video. To achieve this, we first prompt GPT-44turbo to generate new image captions by modifying particular attributes in the original captions. Specifically, we consider five types of attributes: color, light condition, size & shape, posture, and emotion. Subsequently, we employ GPT-4-turbo again to generate questions that focus on the attribute changes between pairs of captions.
Temporal Referring. This aspect requires understanding questions that refer to particular temporal locations (e.g., begin, middle and end). To increase the difficulty, we employ GPT-4-turbo to generate three similar image captions that only differ in certain aspect, such as object, action or attribute. These three captions then placed respectively at the beginning, middle, and end of the textual context. For the questions, we first generate a question for each caption without referring to temporal locations. Then, temporal references are added to the questions according to the specific temporal location of the corresponding caption.
Temporal Grounding. This aspect a complementary angle temporal referring, aiming identify the temporal location of an element (a phrase describing an object, action or attribute) of interest. The textual context reuses the same image captions in temporal referring. To formulate the questions, we first prompt GPT-4-turbo to generate a declarative statement for each caption (e.g.. "the coins are black"). These statements are then incorporated into the temporal grounding question templates, as illustrated in Figure 4.
We enhance the basic textual context (i.e., captions relevant to the question) by incorporating "distractor captions" sampled from the caption pool and inserted between the original captions. This approach challenges the model to identify relevant temporal information amidst irrelevant context, enhances the robustness of its temporal understanding abilities, and presents more realistic scenario mimicking the complexity of real-world temporal reasoning tasks. To maintain question clarity, we ensure distracting captions do not share nouns with the original captions. Table 1 gives summary of our synthesized textual temporal QA datasets. For each task, we also create a validation set consisting of 500 samples for later verification. Appendix B provides detailed and formalized description of the data construction process, as well as the ablation study for distractor captions and a comparison of temporal reasoning transfer via multiple images (Appendix B.4).
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Benchmarks. To establish a connection between textual temporal reasoning ability and video comprehension, we adopt the fine-grained temporal understanding benchmark, TempCompass (Liu et al., 2024b) Multiple-Choic subset), diagnosing multi- basic video temporal understanding abilities. For comprehensive assessment of long-form video understanding capabilities, we choose two challenging benchmarks: MLVU (Zhou et al., 2024) and Video-MME (Fu et al., 2024). These benchmarks specifically designed to test temporal reasoning across diverse domains. For MLVU,

---

Published as a conference paper at ICLR 2025
Table TempCompass evaluation results. Without training on any videos, our T3 helps Long VA-7B outperform ShareGPT4Video-88 trained on 28k video samples. Temporal Change combines Order-GPT and Attribute. The best results are shown bold.
Method
Action
Direction
Speed
Order
Attribute Change
Average
Video CharGPT:7B (Maaz et 2024)
61.5
28.9
29.0
36.1
30.9
37.7
Video-LLaVA--7 (Lin al., 2023a)
76.0
35.2
35.7
37.8
41.0
45.6
LLaVA LLaVA-NeXT NeXT Video- 7B DPO (I.iu et al., 2024a)
87.6
35.8
41.3
39.7
45.8
50.6
Jama---VILAL.--BB (Lin al., 20236)
92.9
44.2
50.0
60.1
56.4
ShareGPT4Video SB (Chen al,, 2024)
87.6
34.6
47.5
62.9
64.2
59.4
VA-BB (32 frm)
92.3
37.3
42.0
54.3
51.7
55.9
LLaVA-Nexi
92.0
36.4
43.2
55.6
51.0
56.0
Hotpot
92.3
53.0
50.7
$4.9
Temporal Change (Ix)
91.1
37.3
39.8
65.2
63.9
59.5
Temporal Change (2x)
89.6
39.1
40.4
61.8
59.6
Temporal Change (4x)
90.5
39.4
64.2
64.2
60.4
Temporal Change (8x)
90.2
42.4
39.8
64.2
68.1
61.0
Temporal Change (Ix, 2x, 4x 8x)
90.5
65.6
64.2
59.7
Order- Template
89.9
35.8
46.1
58.3
54.5
57.2
Temporal Referring
91.4
33.7
46.1
46.7
$1.0
54.2
Temporal Grounding
Order-Templane Temporal Change (Ix)
90.2
40.6
42.9
63.6
65.3
60.6
Temporal Grounding+ Temporal Change (Ix)
91.1
38.8
41.6
62.9
62.9
59.6
Temporal Referring Temporal Change (1x)
90.8
62.9
64.6
59.8
T3 (all tasks)
90.8
39.7
41.6
65.9
68.1
61.2
GPT-4o
98.2
52.8
52.1
73.2
78.5
71.0
Table 3: MLVU evaluation results. Our textual temporal reasoning transfer achieves the best overall performance, with significant gains temporal-related aspects over the backbone model. TR: Topic Reasoning, AR: Anomaly Recognition, ER: Ego Reasoning, AO: Action Order, AC: Action Count. denotes temporal-related dimensions. Best results are in bold.
Model
AC
ER
Needle QA
AO"
Plot QA
AR
TR
Macro Average
Video-ChatGP1-799 (Mazz et al., 2024)
31.1
42.0
40.3
25.1
29.9
24.0
26.9
31.3
Video LLaVA 7B (Lin
2023a)
35.9
45.2
53,2
20.1
48.4
$7.0
71.6
47.3
MA-LMM-7B (He et al., 2024)
24.3
38.9
43.1
25.1
35.8
35.5
51.9
36.4
Llama-3-VILAI.55
(Lin
2023b)
0.0
24.7
32.4
20.0
27.0
46.2
22.4
VILAI.5-40B (Lin et al. 2023b)
11.7
35.8
38.3
34.3
62.0
56.4
84.7
46.2
InterVL. Chat V1.5 (Chen al., 2023)
13.3
$1.3
80.2
Long VA-7B frm)
25.2
48.6
70.4
58.5
82.2
+ LLaVA Next
11.7
20.5
36.6
17.8
45.3
17.5
72.7
31.7
Heepot QA VV..Nee-eest
13.1
21.5
72.4
T3 w/ LLaVA-Next (Ours)
29.1
48.9
72.1
54.4
69.4
51.0
81.4
58.1
GPT-4o
46.3
57.1
64.8
56.7
65.1
74.5
87.4
64.6
select the following tasks covering three aspects: holistic video understanding (Topic Reasoning (TR) and Anomaly Reasoning (AR)), single-detail understanding (Needle Question-Answering Ego Reasoning (ER) and Plot Question Answering). and multi-detail understanding (Action order (AO) and Action Count (AC)). For Video-MME, which consists of 2700 video QA pairs spanning primary visual domains and three video duration types (Short, Medium, and Long), we focus on two temporal-oriented tasks (i.e., Temporal Perception and Temporal Reasoning) and overall performance.
Compared Methods. We evaluate our temporal textual augmentation approach against baselines: (i) LLa VA-Next: Continually fine-tuning on the original image-text instruction dataset used by LongVA VA (Chen & Xing. 2024). (ii) HotpotQA (Yang et al., 2018): Fine-tuning on multi-document QA dataset to assess enhancements in textual locating ability. We also report the performance of recent (Liin et al., 2023a), (Liu et al., 2024a), MA-LMM-7B (He et al., 2024), ShareGPT4Vides 8B (Chen et al., 2024). LLama-3-VILAI 5.BB (Lin et al., 2023b), VILAI.5-40B (Lin et al., 2023b), and Intern VL-CHaL- 1.B (Chen et al., 2023). GPT-40 (OpenAI, 2024) is included represent commercial model results. These models serve as reference points and we report the highest scores published to represent optimal performance.
Video I.LMs, including Cadeo-ChatGPT (Maaz et al., 2024), Video-I.LaVA-1
VeLVA--eet-Viieo-77---A
Training Details. We adopt LongVA VA (Zhang et al., 2024b) as our backbone model and continually fine-tune the official checkpoint. Despite being trained exclusively text and image data, Long
7

---

Published as a conference paper at ICLR 2025
Table 4: Video-MME evaluation results. Our method enhances Long VA-7B's accuracy across various video durations, even outperforming VILAI15-40B in temporal reasoning. Best results are in bold.
Model
Temporal Perception
Tempural Reasoning
Short
Medium
Long
Overall
Video-LLaVA LLaVA-P 7B (Lin al., 2023s)
45.3
38.0
36.2
39,9
LeaVA.NeXT-Va Video-7B-DPe DBDD
2024a)
L VILAA.. (Lin et al., 20236)
50.9
41.2
56.1
42.1
39.6
45,9
VILAA15-10B (Lin et 2023b)
60.0
40.7
72,0
61.2
53.8
62.3
InterWVL-Chat-VV C1.5--V1.5208 (Chen 2023)
45,5
33.3
60.2
46.4
45.6
50.7
LongVA-7B (128 frm)
58.2
Lest
54.6
37.3
61.2
50.6
44.9
52.2
Hotput QA w/ LLaVA-Nest
65.5
39.5
60.2
50.9
45.6
52.2
T3 w/ LLaVA-Next (Ours)
60.0
49.7
63.3
54,8
46.8
55.0
GPT-4o
74.1
70.3
71.9
demonstrates exceptional zero-shot video understanding capabilities and can effectively handle long videos. We strictly adhere to the original LongVA VA implementation and follow the official fine-tuning protocol. This involves using Adam (Kingma & Ba, 2015) the optimizer, with learning rates of 2e-6 for the visual encoder and le-5 for the rest part of the model. For the exploration of textual temporal reasoning transfer, further scale the dataset up to transfer to holistic video understanding benchmarks, where we find it necessary to incorporate the original instruction tuning data maintain visual perception ability. According to our textual validation accuracy, we set the ratio of textual temporal QA and original data to 1:2 and the total samples to 200k. This mixing ratio applies to Hotpot QA for a fair comparison. Appendix C provides the details of dataset composition mixture configurations, and the dataset scaling effect. The model is trained on the corresponding dataset for one epoch, a process that can be completed within 5 hours using 8 H100 GPUs. We apply our recipe to a larger Qwen2-VL-72B backbone and observe consistent improvements for temporal reasoning (Appendix D.I).
use 22k samples for fine-tuning across different augmentation datasets. We
4.2 RESULTS
Our results aim to answer the following three research questions: (1) Can textual temporal reasoning ability be effectively transferred to video temporal reasoning? and (2) Do these two abilities correlate well? and (3) Does this transfer also reflect on holistic video understanding benchmarks?
Textual temporal reasoning transfer: Table 2 presents the evaluation results of models trained on various compositions of our datasets. Continual training with the original LLa dataset yields negligible improvements, while HotpotQA training surprisingly degrades performance, de- spite its similarity to temporal location tasks. These findings underscore the non-trivial nature of enhancing temporal understanding in Video LLMs. Our textual temporal transfer approach, by contrast, demonstrates significant improvements. Regarding the different tasks, we find that: (i) The Temporal Change (Ix) subset which combines Order-GPT (Ix) and Attribute (Ix), enhances Order accuracy from 54.3 to 65.2 and Attribute accuracy from 51.7 to 63.9. Increasing the number of distractor captions (Ix to 8x) leads to generally better performance. (ii) Order-Template and Temporal Referring excel in the Speed dimension, while Temporal Grounding negatively influences all aspects. (iii) Combining different aspects often leads to synergistic improvements. For example, supplementing Temporal Change (1x) with other tasks leads to better results than training with the set alone. This suggests complementary benefits across diverse temporal reasoning skills. Finally, despite without any video data, our T3 composition with all synthesized tasks, helps Long VA-7B achieve the best overall accuracy, even outperforming ShareGPT4Video--5 trained on 28,000 video samples annotated by GPT-4V. These results highlight the effectiveness of our approach in improving Video LLMs' temporal reasoning across various dimensions.
Holistic video understanding evaluation: Based on previous results, we adopt the T3 composition, which achieves the highest textual validation scores, to explore transfer effects on comprehensive video understanding benchmarks. Table 3 shows detailed task accuracy results on MLVU. Further training on the original image-text instruction dataset or Hotpot QA decreased overall performance, indicating that video temporal understanding cannot be enhanced through continued fine-tuning or long-context training alone. In contrast, mixing our T3 with LLaVA-Next improves performance across most tasks. Notably, the two highly camporal-orientee Order (AO), see a substantial average gain of 8.3 points. Specifically, AC scores increased by 3.9 points (25.2 > 29.1). while AO scores improved by 12.7 points (41.7 54.4). Table 4 demonstrates
tasks, Action Count (AC) and Action
8

---

Published as a conference paper at ICLR 2025
substantial improvement in Video-MME's temporal reasoning subtask (37.3 49.7), even outper- forming larger models such as AnterVL-Chhan V1.5at- and VILAI1.540B. Besides, compared to fine-tuning with LLaVA-Nec and Hotpot QA, method yields the best overall performance at 55.0. Further analysis on distribution overlapping between our synthesized samples and downstream tasks justifies the performance gains are not sourced from simply memorizing the distribution of questions and answer choices D.2. We also observe strong correlations between the textual and video temporal reasoning scores across benchmarks Appendix D.3. These results validate that enhanced textual temporal reasoning in LLM backbones effectively transfers to holistic video understanding.
Critical LLM components for temporal understanding: We evaluate the im- pact of selectively unfreezing LLM components across TempCompass, MLVU, and Video-MME datasets using the T3 /w LLaVA-Next dataset. Figure 5 il- lustrates that fully unfreezing the language model yields optimal performance. For TempCompass, the fully unfrozen model achieves 58.2% accuracy, outperforming self- attention-only (57.7%) and MLP-only (55.1 %) con- figurations. Notably, unfreezing self-attention mod- ules consistently surpasses unfreezing MLP layers, with MLVU showing a substantial 4.2 percentage point difference (55.8% v.s. 51.6%) and 2.3 abso- lute accuracy gap on Video-MME (52.4% v.s. 50.1% 30.1%) This suggests that self-attention modules play more crucial role in transferring temporal understanding capabilities. These findings align with the under- standing that self-attention modules act as dynamic information aggregators (Wang et al., 2023b), while MLP layers primarily serve as static knowledge banks (Geva et al., 2021).
Taned LLM
All
S14
Figure Compared to MLP modules, self- attention modules are more important for tem- poral reasoning.
Enhanced utilization of increased video frames: We examine whether temporal-enhancee LLMs
could better leverage increased input video frames by comparing performance gains when increasing frames from 32 to 128, using models trained Temporal Change sets. Figure shows that our textual temporal reusoning-enhancee models benefit more from increased input frames compared the baseline Long VA. On TempCompass, the Temporal Change (1x) model gains 1.7% accuracy with 128 frames, versus 0.6% for LongVA. Similarly, on MLVU, enhanced models achieve nearly double the gain of the baseline. These results indicate that our approach enables better utilization of additional input frames. Training on longer inputs with more distractor captions generally yields larger gains, though the optimal length varies between datasets.
Transfer to Other Temporal Concepts: Our de- signed textual temporal tasks focus on four fundamen- tal temporal aspects inspired by previous diagnosis benchmarks (Liu et al., 2024b; Fu al., 2024). While these aspects may appear basic, they form essential building blocks for complex temporal reasoning and particularly challenging for current Video LLMs to master, as evidenced in our previous analysis. To rigorously evaluate the generalization capability of our method, particularly to temporal concepts that challenging describe textually (e.g., precise rotation counts, relative motion speeds). we test our models on two recent benchmarks with complemen- tary temporal characteristics: TOMATO (Shangguan al., 2024) and Vinoground (Zhang et al., 2024a). leverage more input video frames better.
Seatite Vil-300
Neattr 9E90 MI30
30-
Ameryy Gale
1.5
1.-
Figure 6: Our temporal-enhanced models
TOMATO evaluates fine- grained motion understand- ing that goes beyond discrete textual descriptions, testing continuous aspects of motion dynamics. Our T3-enhanced model shows significant improvements in velocity and frequency understanding
(Vel. & Freq) (39.9 vs. 23.1, 16.8) and Shape & Trend comprehension (30.5 vs. 22.4, + 8.1). The decreased performance in Rotation (16.6 vs. 26.4, 9.8) reveals inherent limitation: purely

---

Published as a conference paper at ICLR 2025
Table 5: Generalization evaluation on TOMATO and Vinoground benchmarks.
Model
TOMATO (Shanggran etal. X011
Vinegrunn (Chang mall 30M6)
Rousiu
Direction
& Freg.
Shage
Veed
Actio Count
Text
Video
Group
TongVa-51
26.4
21.6
25.1
22.4
386
145
313
21
54
24.6 (0
31.3 99
3601+110 (0.(+
40/re1-) (66
(9 164(+17)
810
46608)
-TI -T (0und)
16869990
(1
(-16))
24304300
(..(110.))
900(-6-0
(18
text-based training may not fully capture precise rotational movements that require continuous, quantitative understanding. On the other hand, Vinoground examines temporal reasoning ability in real-world scenarios through carefully curated video-caption pairs that share identical static content but differ in temporal action sequences. For instance, distinguishing between cut then turn versus turn then cut requires precise understanding of temporal order. Our method substantially improves LongVA, VA-7B's performance across all metrics: Text (31.6, +10.4), Video (30.0, +8.4), and Group (11.8, +6.4). These comprehensive evaluations demonstrate that our T3 develops robust temporal reasoning abilities that generalize well to continuous motion understanding (TOMATO) and different task formats (Vinoground).
RELATED WORK
Video Large Language Models (Video LLMs). Video LLMs, integrating LLMs and visual encoders, have shown promising results on diverse video tasks (Tang et al., 2023a). These models typically leverage open-source L.LMs (Touvron et al., 2023; Yang et al., 2024) for generation and reasoning ca- pabilities. Recent architectural explorations have focused on efficient video encoding, including Video Transformer and Q-Former (Li al., 2023b; Ren et al., 2023), spatial and temporal QFormer (Zhang et al., 2023; Tan al., 2024), and memory bank for long video frames (He et al., 2024). Other approaches like Video-LLaVA (Lin et al., 2023a), Chat-UniVi (Jin et al., 2023), VILA-series (Lin al., 2023b), and LLaVA- (Liu et al., 2024a; Zhang et al., 2024b) demonstrate effective transfer from image to video tasks with image-video unification. In contrast to these studies, our work focuses on probing and enhancing the temporal understanding ability of Video LLMs, identifying the LLM's poor grasp of temporal concepts as a key bottleneck. Our textual-only temporal reasoning transfer effectively addresses this issue without using any image/video instruction tuning data.
Temporal Understanding Benchmarks for Video LLMs. Video temporal understanding bench- marks have evolved rapidly to guide Video LLM development. Pilot studies incorporate existing Li et al., 2021) into comprehensive assessments (Li et al., 2023c; Ning et al., 2023; Li et al., 2023a). As Video-LLMS become stronger, benchmarks such as EgoSchema (Man- galam et al., 2023), Neptune (Nagrani et al., 2024), Video-MME (Fu et al., 2024) and MLVU (Zhou et al., 2024), focus on stress-testing models with diverse and challenging tasks of long videos. Specific benchmarks addressing temporal understanding (Li et al., 2023d; Liu et al., 2024b; Wang et al., 2023a; Shangguan et al., 2024; Zhang et al., 2024a) have highlighted limitations in current Video LLMs. Our work not only diagnoses the temporal understanding bottleneck in Video LLMs but also demonstrates a correlation between textual and fine-grad -raingr video temporal understanding. Impor- tantly, our proposed methods show improvements on challenging benchmarks such Video-MME, advancing the field of video temporal reasoning.
tasks (Wang et al., 2019;
6 CONCLUSIONS AND LIMITATIONS
Our work investigates the temporal reasoning bottleneck of Video LLMs, identifying the language model backbone as the primary source. To address this, we develop a textual temporal reasoning transfer framework that synthesizes QA pairs on various temporal concepts from image-text pairs. Experimental results validate the correlation between textual and visual temporal understanding, demonstrating the efficacy of our method on comprehensive video understanding benchmarks.
Limitations: Limited temporal concept scope: While our framework addresses four key temporal dimensions and shows potential across temporal tasks, temporal reasoning aspects such motion- related changes still need further investigation. (ii) Scalability with model size: Although T3 improves even large models such as Qwen2-VL-722,, scales and tasks. Future work could explore more complex temporal scenarios and develop advanced augmentation pipelines for larger models.
VL-72B, the gains vary across the backbone model
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We gratefully acknowledge the HKU-NLP group and PKU Lanco group for their constructive feedback, and thank the anonymous reviewers for their insightful comments that substantially improved this work. This research was partially supported by the National Natural Science Foundation of China under Grant No. 92470205 and No. 62176002, and the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N HKU714/21.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR (Workshop), 2017.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long tail visual concepts. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 3558-3568, 2021. doi: 11w.1////VVA4437.721111111
Lin Chen and Long Xing. Open-llava-next: An open-source implementation of llava-next series for fa- cilitating the large multi-modal model community. petpss//ggithe com/ Open- LLaVA- NeXT, -XXT, 2024.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. ArXiv preprint, abs/2406.04322 2024.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. ArXiv preprint, abs/23 12.12222.4423.. 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783. 2024.
Chaoyou Fu, Yuhan Dai. Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv preprint, abs/2405.21075, 2024.
11

---

Published as a conference paper at ICLR 2025
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint. abs/2403.00550) 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021. doi: 10.18653/v1/2022 .e200000m..mmmmmmmii.44....
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Memory- augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514 2024.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. ArXiv preprint, abs/231 1. 108046, 2023.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. ArXiv preprint, abs/2307. 16125, 2023a.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. ArXiv preprint, 005/23355063555 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: comprehensive multi-modal video understanding benchmark. ArXiv preprint, 2023c.
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al. Value: multi-task benchmark for video-and-language understanding evaluation. ArXiv preprint, 2021.
Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. ArXiv preprint. abs/2311. 1e//2311117404 2023d.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. ArXiv preprint, 0bbs/2111.10122 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Haotian Liu, Chunyuan Li, Yuheng L.,, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACI. 2024, pp. 8731-8777 2024b.
12

---

Published as a conference paper at ICLR 2025
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- chatttt Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, and Tobias Weyand. Neptune: The long orbit to benchmarking long video understanding. 2024.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. ArXiv preprint, 2023.
OpenAI. Gpt-4o system card, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763,8773,,280 2021.
Shuhuai Ren, Linli Yao. Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. ArXiv preprint, abs/2311.00051,, 2023.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint arXiv:2410.23200 2024.
Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanich Deilamsalchy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key trame-conditioned long video-Ilm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13581-13591, 2024.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey, ArXiv preprint, abs/23 122111...4339 17432, 2023a.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song. Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu et al. Video understanding with large language models: A survey, ArXiv preprint. 1bs/2312.11442,, 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971. 2023.
Jianghui Wang, Yuxuan Wang, Dongyan Zhao, and Zilong Zheng. Moviepuzzle: Visual narrative reasoning through multimodal order learning. arXiv preprint arii::200000000002202. 2023a.
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng. Jie Zhou, and Xu Sun. Label words anchors: An information flow perspective for understanding in-context learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 0840-98555 2023b. doi: 10. 18.18v/3/v1 wea.emmnl-mmiin.66
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision languaee model's perception of the world at any resolution. arXiv preprint 1rX1v:2409.12191 2024.
13

---

Published as a conference paper at ICLR 2025
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, quality multilingual dataset for video-and- -nd-uagguggee research. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pp. 4580-4590, 2019. doi: 10.1 10.1109/CCV22211900000 2.000066...
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. ArXiv preprint, abs/2407.15754, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. ArXiv preprint, 0bs/2447.11077. 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning- HotpotQA: dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018. doi: 10. 18653v/ 1tt0//11/11122555 1259.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tunel audio-visual language model for video understanding. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.
543-553, 2023. doi: 10. N0.18653/v1/20 1/202/ .m
Jianrui Zhang, Cai MU, and Yong Jae Lee. Vinoground: Scrutinizing Imms over dense temporal reasoning with short videos. arXiv preprint arXiv:2410.02763, 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. ArXiv preprint, abs/2406. 16852, 2024b.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chun- yuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimization of video large multimodal models from language model reward. ArXiv preprint, abs/2404.01258, 2024c.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024d. URL https: / /arxivv.... org/ wttg/bs//////oooooeo..oooeaaeere 2aabs/2410 2410 0. 02713.
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning, 2024. URL htwps: /
orgg abs/24/8 05517.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. ArXiv preprint, abs/2406. 04264, 2024.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF VIDEO LLM TEMPORAL ANALYSIS
A.1 VIDEOS
$2.2 describes our basic idea to create temporally contradicting videos for different temporal aspects. In this section, we provide more details of the source videos and the composition of synthesized videos. Table summarizes the video composition for different temporal aspects.
A.1.1 ORDER
Source Videos. This aspect encompasses three categories of source videos: people, cats, and flowers. We obtained these videos from the ShutterStock? platform. To create distinct training and test for the classifier probe, we collected videos of each category with both black and white backgrounds. The black background videos were used for training the probe, while the white background videos were reserved for evaluation.
Composition of Synthesized Videos. The synthesized video for this aspect concatenate two or three source videos. As illustrated in Table 7, the "Two Events" videos comprise two categories, showcasing cats and people in reversing sequential orders. The "Three Events" videos expand categories, covering all possible permutations of cat, person, and flower sequences.
A.1.2 ATTRIBUTE
Source Videos. This aspect is further divided into Shape and Brighiness. For Shape, we collected videos from ShutterStock that illustrate the process of flower blooming. Similar to the Order aspect, gathered flowers with both black and white backgrounds to create separate training and testing sets for the classifier probe. For Brighmess, we collected static images from the COCO dataset (Lin al., 2014 4) and adjusted pixel values to synthesize videos with brightness variations. Car images were used to create training videos, while cat images were used for testing videos.
Composition of Synthesized Videos. Videos of Shape consist of two categories: flower blooming and its reversed process, with the latter created by inverting the source videos. Brightness videos also comprise two categories: brightening and darkening.
A.1.3 TEMPORAL REFERRING
Source Videos. This aspect utilizes the "Three Events" videos previously collected for the Order aspect.
Composition of Synthesized Videos. The Temporal Referring questions are categorized into three types, each referring to specific part of the video: the beginning, middle, and end. For instance, a typical question might be, "Which item is shown at the beginning/midddleennd of the video?" For each of these temporal reference types, the videos are further classified into three categories, yielding different possible answers: person, a cat, or a flower.
A.1.4 TEMPORAL GROUNDING
Source Videos. This aspect also employs the "Three Events" videos originally gathered for the Order aspect.
Composition of Synthesized Videos. In this aspect, three question types are formulated, i.e., "In which part of the video can we see person/cal/llower? For each question type, the videos divided into three categories, resulting in different answers grounding to the video's o' different temporal locations: beginning, middle, end.
A.2 FRAME CAPTIONS
To evaluate the textual temporal understanding capabilities of LLMSs, must that frame captions contain adequate information about relevant temporal aspects. For example, in videos
hhtts:/wwww.hhuttrsttkk.com
15

---

Published as a conference paper at ICLR 2025
depicting transition from cat to person, the initial frame captions should clearly identify the subject as a cat, while later frame captions should describe the person.
In approach, der, Shape Attribute, Temporal Referring and Temporal Grounding, drawing inspiration from ShareGPT4Video (Chen et al., 2024). As outlined in Table 8,we begin by generating a caption for the first frame. For subsequent frames, captions are created based on both the visual features of the current frame and the captions of preceding frame. Table 10, 11, 12 demonstrate that this method produces captions with sufficient and accurate per-frame information to comprehend these temporal aspects, thanks to the powerful visual perception ability of GPT-4o.
employ "differential captioning strategy" for aspects such as Or-
In terms of the Brightess Attribute, we find that simple differential captioning cannot accurately reflect brightness changes across video frames due the lack of a consistent brightness standard. To address this issue, we developed chain-of -otthhuhhtttttttt (Co'T) captioning strategy. This approach first prompts GPT-4o categorize frame brightness into four levels (very dark, slightly dark, normal, and bright) before generating the caption. As illustrated in Table 11, the resulting frame captions accurately reflect brightness changes throughout the video.
A.3 CLASSIFIER PROBE TRAINING
Model Architecture. Our classifier probe is built upon a single-layer unidirectional LSTM model with hidden dimension of 128. This probe trained on visual features that are down-sampled and projected into the LLM embedding space, i.e., the embedding of visual tokens for Video L.Mss. Let V E R. represent these visual features, where = x w and 1, h, w, d, correspond to the temporal, height, width and channel dimensions, respectively. For an 8-frame input video, the default setup of Long VA-7B n = 1152, dy = 3584, while for VILA-SB, it is n = 1568, d, = 4096. We further down-sample V V' E R= xd, using bi-linear interpolation, where n' 128 and d, = 1024. The LSTM probe then trained on this down-sampled representation V".
Data. Table provides comprehensive breakdown of the training and test video compositions. For the Brightness Attribute, we generated training videos using car images, while the test videos are constructed from cat images. Regarding other temporal aspects, we created training videos with a black background, contrasting with the white background used in the test videos.
Table 13 shows the training hyper- parameters for the LSTM classifier probe. For relatively easy temporal aspects, i.e., the "Two Events" Order, we train the probe for only 15 epoches. For Attribute and other aspects, we increase the training epoch to 80 and 120, respectively.
Training
A.4 RESULTS
A.4.1 SUB-CATEGON RESULTS
Figures 7 and 8 provide comprehensive breakdown of our temporal analysis results. The data reveals significant variations in model performance across different sub-categories within particular temporal aspects. In the Order aspect, we observe stark contrast in difficulty between two-event and three-event sequences. The task of discerning the order of three events proves substantially more challenging, with LA- and VILA-8B experiencing dramatic accuracy drops of 41.8 and 51.1 points, respectively. Regarding Temporal Referring, the accuracy is significantly lower when referring to the middle of the video. The deficiency of the LLM decoder in temporal understanding is also more pronounced in these challenging scenarios.
A.4.2 RESULTS OF PROBING LLM FEATURES
We also investigate the performance of probing LLM last-layer features, instead of probing the visual features in the embedding space. Specifically, we input videos into VA-TV and extract the hidden states from the last layer of the LLM decoder. Two types of probes were then employed: LSTM Probe: Trained on all hidden state features, this probe assesses whether temporal information retained across all the last-layer LLM features. Linear Probe: Trained on the hidden state of the last time step, this probe evaluates if the temporal information persists in single hidden state.
16

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of the distractor captions and temporal reasoning transfer via images instead of captions.
Method
Action
Direction
Speed
Order
Attribute Change
Avg.
LongVA-7B (32 frm)
92.3
37.3
42.0
54.3
51.7
55.9
Temporal Change (w/o Distractor)
90.8
40.3
41.0
61.6
60.4
59.0
+ Temporal Change (1x)
91.1
37.3
39.8
65.2
63.9
59.5
+ Temporal Change (2x)
89.6
39.1
40.4
66.9
61.8
59.6
+ Temporal Change (4x)
90.5
43.0
39.4
64.2
64.2
60.4
Temporal Change (8x)
90.2
42.4
39.8
64.2
68.1
61.0
+ Image Order
86.4
36.1
40.4
55.3
56.9
55.2
+ Textual Order
38.2
41.6
63.6
58.7
58.9
Table 14 presents the probing results. We can see that the LSTM probe's accuracy on LLM features is consistently lower than that on visual features across all temporal aspects, especially for the Attribute aspect (88.9% vs. 73.9%). The accuracy of the linear probe is even lower. These findings demonstrate that considerable portion of the temporal information lost when transitioning from visual features to the final-layer LLM features. Furthermore, we observe a substantial gap between the performance of the last-layer LLM features and the final output of Long VA. This suggests that even when relevant information is present in the LLM hidden states, the Vidco LLM can still produce incorrect answers. Investigating the causes of this discrepancy is left for future work.
B DETAILS OF TEXTUAL TEMPORAL UNDERSTANDING DATA
B. CAPTION POOL
The contextual information of our textual QA data are sourced from the detailed image captions from the LLa LA---eeaa--558 dataset These captions are generated by the LLaVA-Next-34B model (Liu et al., 2024a) based on images from the LCS-558K (a subset the LAION/CC/SBU dataset). We only retain the first sentence of the detailed image captions to form the caption pool Cpool.
B.2 TEXTUAL TEMPORAL REASONING DATA CONSTRUCTION
The construction process of our textual temporal reasoning datasets are illustrated in detail in Algorithm 2, 3, 4, The prompts used to generate question-answer pairs and captions are shown in Table 16, 17, 18, 19, 20.
B.3 DATASET STATISTICS
Table 15 presents detailed statistics of our textual temporal datasets and the two baseline datasets. The information includes the number of samples, the number of relevant and distractor caption, the number of input/output tokens, and the involved modalities.
B.4 ABLATION STUDY OF TEXTUAL TEMPORAL REASONING TRANSFER
Effect of distractor captions. We maintain the same setting as in our main paper and compare the results of the Temporal Change sets with and without distractor captions. As shown in the middle block of Table 6, incorporating distractor captions yields a more pronounced improvement in the targeted Order and Attribute Change aspects. Moreover, increasing the number of distractor captions generally leads better performance. These results validate our approach of inserting confusing captions to enhance textual temporal reasoning transfer.
Transfer via images versus captions. To investigate this difference. we replace the caption with the original image in our Order set, keeping all other settings constant. As shown in the bottom block of
17

---

Published as a conference paper at ICLR 2025
Order Events)
Brightness
Begin
Referring Middle
Referiing End
Events)
Blooming
Grounding Person
Grounding
Grounding
Acornry
Figure 7: Detailed temporal analysis results of LongVA.
Order Events)
Brightnass
Referring Referring Middle
Referring End
Arcumtyy
Events)
Blooming
Grounding
Grounding Flower
3
Accuray
23
Figure 8: Detailed temporal analysis results of VILA.
Table 6, textual temporal reasoning outperforms the corresponding transfer set using images (Image Order), Notably, transfer via images only marginally increases the Order performance from 54.3 to 55.4, while text format transfer significantly boosts to 63.6. This substantial gap corroborates findings in the main paper that the temporal reasoning bottleneck lies the LLM's side, and therefore, the textual format is more effective in enhancing temporal reasoning capability.
Textual temporal QA generation for the Order aspect using GPT-4-turbo. ExtCont Cpaot: C, n) inserts relevant captions C in between distractor captions and ensures that every pair of relevant and distractor caption not share common nouns.
Algorithm 1:
Input: Caption pool C Cpoot- number of relevant captions Recaps number of distractor captions Oakaps large language model LLM(), function extend context ExiCont ( C, n).
QA generation prompt p
Output: SFT data sample (Xmm Xour
Sample relevant captions
C, {C,, Cr,... Shrop } Cpeod
# Create ext extended context
Ceett = ExtCont(C Cpool, Cr, Ddcap)
Generate questions and answers
Q, A = LLM p)
Concatenate context and question
X Cext ( Q Xnai A
18

---

Published as conference paper at ICLR 2025
Table Details of the videos used for our temporal analytical study Section 2. "A-+B" denotes synthesized video first showing A and then B. (black) and (white) indicate videos with black and white background, respectively. (car) and (cat) indicate videos showing cars and cats, respectively. xN denotes the number of videos. "Train Videos" is only used when training the classifier probe.
Dataset
Classes
Train Videos
Test Videos
Order
Two
cl: person->cat (black) x400
el: person->cat (white) x400
Events
c2: cat->person (black) x400
c2: cat->person (white) x400
cl: person-rrat--fowee (black) x400
cl: person-rcat- (white) x100
c2: penson--llower-vca (black) x400
c2: person-:flower-ccu (white) x100
Three
c3: cat-tperoon-ffloer (black) x400
c3: cat-tperson--flower (white) x100
Events
c4: cat-ffloerr-perron (black) x400
c4: cat-viloerr-perron (white) x100
c5: flower-person- (black) x400
c5: flower-peesson (white) x100
c6: Howersocat- (hlack) x400)
c6: flower-vcat-pperson (white) x x100
Attribule
cl: blooming (black) x177
cl: blooming (white) x100
Shape
c2: unblooming (black) x177
c2: unblooming (white) x100
c1: brightening (car) x955
cl: brightening (cat) x394
Brightness
c2: darkening (car) x955
c2: darkening (cat) x394
Temporal Referring
cl: (person-rcat-floower (black) x400,
cl: (person-xcat (white) x100.
person--flowe--ccal (black) x400)
person-tflower-ccat (white) x100)
c2: (eat-tperson- (black) x400,
c2: (est-tperson-ffooer (white) x100,
Begin
cat--flower-pperron (black) x400]
cat-rffowwrrpperron (white) x100)
c3: -Bower--persoo---all (black) x400.
c3: (lower--persom--eut (white) x100.
flower-vcat-ppersonn (black) x4003
flowe--catt-perron (white) x100}
cl: (cal=-personn-ylowwer (black) x400,
cl: fcat->person (white) x100.
flower-persoon-vcal (black) x400)
floee- (white) x100}
c2: (person--cat--llweer (black) x400.
c2: [person (person-xcatfflweer (white) x100.
Middle
fower->cat--perron (black) x400]
fower->cat-pperson (white) x x100)
c3: (persson--ffoweer-eeaa (black) x400,
c3: fferson-ffoower--catt (white) x100.
cal-eflower-perron (black) x400)
cat-sflower-pperson (white) x100}
cl: (cattttfooowrr-ssrsss (black) x400,
cl: (cat-<flower-pposon (white) x100,
flower- (black) x400)
floeer (white) x100}
c2: (person-fllwerr-eall (black) x400,
c2: (person--lower-eeall (white) x100.
End
flower-ppesonn-ical (black) x400)
flower-sperson--caa x100)
c3: (person (person-paat--fooee (black) x400,
c3: {person (person-rcat--flowerr (white) x100.
cat->person (black) x400)
cat-rpenson->ffower (white) x100)
Temporal Grounding
cl: [person->cat--floweee (black) x400,
cl: {person-scat-flooee (white) x100,
penson-fllowr-teal (black) x400)
person-sflower-teat (white) x100)
c2: (cat-tperson--flowee (black) x400,
c2: (cat--person (white) x100,
Person
flower-persnn-ccat (black) x400)
flower-ppenon--caat (white) x100)
c3: (cat-flower--person (black) x400,
c3: (cat-ffloerr-peeson (white) x100,
flower-bcat-pperson (black) x400)
flower-icat-pprrson (white) x x100}
cl: (cat--person---flowee (black) x400,
cl: (eat-rpersonn~<flowe (white) x100,
cal cat-sflower-pperson (black) x400)
cat-sflower--poersnn (white) x100)
c2: (person--cat--flowee (black) x400.
c2: (person-scat-ffowee (white) x100,
Cat
Hower->catt-persoon (black) x400)
Hoowr--cat--person (white) x100)
c3: (person-=ffover--caa (black) x400,
c3: (person-ffover-rcatt (white) x100,
flower--person--caallllaalaaa x400)
flower-ipnnoon-vcal (white) x100}
cl: (fower--perss--ccctt (black) x400.
cl: (fowerr-perroo--ccul (white) x100.
flower-scat-ppersonn (black) x400)
lower--cat--person (white) x100)
c2: [person>>flower (black) x400,
c:: [person->llower (white) x100.
Flower
cat cal-:flooer-pprrson (black) x400)
eat ooleellowr--personn (white) x100)
c3: {person -person--cat--ffooee (black) x400,
c3: {person-<ce ppessnn-cat--foowee (white) x100,
cat cat->person (black) x400)
cat cat-sperson->ffom fflower (white) x x100)
19

---

Published as a conference paper at ICLR 2025
Table 8: The prompt used generate frame captions for videos of Order, Shape Attribute, Temporal Referring and Temporal Grounding.
First Frame:
You are an advanced Al visual assistant. You will be provided with the first frame extracted from a video clip. Your task describe this frame in much detail as possible, focusing on the following elements:
1. ** **Key Objects**; Identify and mention the main objects or subjects present in the frame. Be specific and provide relevant details about each object.
2. V*Visual Attributes**; Describe the visual characteristies of the key objects, such their color, size, shape, texture, or any other notable features. Pay special attention to the overall brightness lighting conditions in the frame.
3. **Location*** Specify the location or setting of the frame, including the background, environment, any identifiable landmarks or surroundings.
4. **Potential Action ** If applicable, describe any actions or activities that the key objects might be engaged in or are likely to perform based on their positioning, pose, or context within the frame.
** **Movemnt**: If there any visible or implied movement in the frame, describe the direction, trajectory, nature of the movement for the relevant objects.
Ensure your description accurately reflects only the contents of this frame. Do not reference any part of this prompt in your response. Begin with "This frame". The caption should be written in present tense and should not exceed 2 sentences.
Other Frames:
You are an advanced Al visual assistant tasked with describing frames extracted from a video clip. When provided with frame, describe in detailed and accurate terms, focusing on the changes of following elements:
1. ** "Key Objects**: Identify and mention the main objects or subjects present in the frame. Be specific and provide relevant details about each object.
2. Visual Attributes**; Describe the visual characteristics of the key objects, such their color, size, shape, texture, or any other notable features. Pay special attention to the overall brightness or lighting conditions in the frame.
3. **Location *#; Specify the location or setting of the frame, including the background, environment, any identifiable landmarks or surroundings.
**Potentia Action** If applicable, describe any actions or activities that the key objects might be engaged in or likely to perform based on their positioning, pose, context within the frame.
# ** If there any visible or implied movement in the frame, describe the direction, trajectory, nature of the movement for the relevant objects.
While your primary focus should be the current frame, you can reference the provided caption of the preceding frame describe any relevant relationships between the two frames. Ensure your description reflects only the contents of the current frame and do not include any elements of this prompt in your response. Begin with "This frame". The caption should be written in present tense and should not exceed 2 sentences.
The current frame is uploaded as an image. The caption of the preceding frame is provided below:
[previous wwrrious_frame.cco frame_caption]
C TRAINING DETAILS
C.1 TRAINING DATASETS
Table 21 provides details of the training datasets used in our main paper. To ensure fair comparison across all compositions, we the number of training samples to 22k, which is the maximum
20

---

Published as a conference paper at ICLR 2025
Table 9: The prompt used to generate frame captions for the videos of Brightness Attribute. We prompt GPT-4o to determine the level of brightness before generating the frame caption, which ensures information of brightness change is reflected in the frame captions (Table 11 shows an example).
Analyze the provided image and categorize its overall brightness level into one of the following categories: "bright', 'normal', 'slighlly dark', 'very dark'. Based on this assessment, craft a detailed description of the image, reflecting the main elements presented in the image.
Here are two examples of classifed brightness and corresponding descriptions: ("brightness": "bright",
"description": "A person is sitting on a bench with a book in a bright environment." ("brightness": "slightly dark", "description": "The image shows an slightly dark room with red balloon."
Now please give your classification result and description, following the above JSON format:
Table 10: Example of a video focusing on the Order aspect and corresponding frame captions used to evaluate the LLM decoder. We show four frames here but in practice eight frames used. The words in blue highlight the major subject depicted in each frame.
teucksssskk
SI shipe stuketscck
Frame Captions:
Frame 1: This frame shows fluffy white and gray kitten lying on its side against a plain white background, playing with a small purple toy. The kitten is holding the toy with its front paws and appears to be a playful mood, with some motion blur suggesting movement.
Frame 3:This frame depicts the same fluffy white and gray kitten, now lying on its side on the plain white background. The kitten's 's eyes are still closed, and its head is resting on the ground, continuing to appear relaxed or asleep.
Frame 5: This frame shows a woman with smooth blue skin and dark hair tied back, gazing directly the camera. She is positioned against plain gray background with visible light reflecting from her skin, and she appears calm and serene based on her expression and posture.
Frame 7: This frame shows a smiling woman with smooth blue skin and dark, straight hair tied back, facing slightly away from the camera with her head turned toward the viewer. She is set against plain gray background, and her joyful expression and body posture convey sense of happiness and contentment.
available in the smallest subsets (Referring and Grounding). For the transfer evaluation on MLVV and Video-MME, combine the original LLa VA-Nex image- text SFT dataset with our textually synthesized samples, as shown in Table 22. The mixing ratio and total sample size are determined based the exploration results diseussed in the following subsection.
21

---

Published as a conference paper at ICLR 2025
Table 11: Example of video focusing on the Brightness Attribute aspect and corresponding frame captions used to evaluate the LLM decoder. The key information of brightness change is highlighted in blue.
Frame Captions:
Frame 1: The image shows very dark scene with black and white cat lying top of laptop keyboard. The cat's s face and whiskers are slightly visible, but the overall image is hard due to the low brightness.
Frame 3: The image shows black and white cat lying top of laptop keyboard. The overall scene is very dark, making it difficult to see details clearly.
Frame 5: The image shows a black and white cat lying on top of laptop keyboard in slightly dark environment.
Frame The image shows a black and white cat lying on top of laptop keyboard. The cat appears be comfortably settled in the space between the laptop screen and keyboard. The overall brightness of the image is normal, allowing clear visibility of the cat's fur and the laptop's keys.
96.8%
Textual Valttiaiion Accuracy (%6
97.5%
Textual Valicittion Accuratyy (86)
96.8%
95
90.99
90
92.1%
90.3%
88
75
Ratio (LLaVA Textual
Size
Figure 9: (Left) Exploration of mixing ratios between image-text instruction tuning and textual temporal QA. (Right) Dataset scaling analysis.
C.2 DATASET MIXING EXPLORATIONS
Our preliminary study revealed that training the model exclusively on large-scale textual temporal QA samples led to catastrophic forgetting of visual perception abilities, degrading it to text-only model. To mitigate this issue, we explore the integration of original instruction tuning datasets to enhance video understanding transfer performance. Specifically, we combine the original LLa dataset with textual temporal QA pairs from our All Temporal subset, varying the mixing ratio from I to 20, while fixing total samples to 100k for fair comparison. We use textual validation accuracy as selection criterion, as it correlates well with temporal understanding ability, as discussed in our main paper.
22

---

Published as a conference paper at ICLR 2025
Table 12: Example of video focusing on the Shape Attribute aspect and corresponding frame captions used to evaluate the LLM decoder. The key information of shape change is highlighted in blue.
shutters sck
shutters sck
shutters sck
shutters sck
Frame Captions:
Frame 1: This frame shows blue bud at the end of green cotton swab positioned vertically against a plain white background. The cotton bud appears clean and unused, with no visible movement action taking place.
Frame 3: This frame shows the same blue bud the end of green cotton swab, still positioned vertically against plain white background. The cotton on the bud appears slightly more compressed and misshapen compared to the preceding frame.
This frame shows a blue flower with five petals on a green stem, against white background. The flower appears to be blooming mid-stage with slightly spread petals, and the stem is straight and upright, positioned centrally in the frame.
Frame 5:
Frame This frame features the same blue flower with six petals on a green stem, now slightly the right against white background. The petals still widely spread, and the stem remains upright, suggesting no noticeable movement or change from the previous frame
Table 13: Training arperpparametrs for the classifier probe.
Temporal Aspect
Learning Rate
Batch Size
#Epoch
Optimizer
Order (Two Events)
5e-5
64
15
Adam
Order (Three Events)
5e-5
64
120
Adam
Attribute
5e-5
64
80
Adam
Temporal Referring
5e-5
64
Adam
Temporal Grounding
5c-5
64
120
Adam
Table 14: Results of probing LLM features versus visual features.
Model
Attrihute
Referring
Grounding
Order
Average
LSTM Probe Visual Features
88.9
92.7
98.2
96.4
94.0
LSTM Probe LLM Features
73.9
87.2
91.9
94.2
86.8
Linear Probe LLM Features
72.7
80.0
80.4
80.4
78.4
LongVA-1 VA-7B
49.6
70.5
70.9
71.1
65.5
Figure 9 (left) illustrates that performance the textual component gradually decreases more image-text instruction tuning samples are included. Based on these results, we selected a ratio of 2, as it maintains relatively high textual validation accuracy (> 95%) while incorporating sufficient visual-text samples to preserve the model's visual capabilities.
We further explore the optimal sample size for the mixed dataset, with results shown in Figure (right). As expected, textual accuracy continuously improved with larger training samples. This trend holds for downstream tasks such MLVU and Video-MME as well shown in Table 23. Consequently, we set the total sample size to 200k, as this threshold first exhibits saturated textual validation accuracy while keeping the sample size manageable for efficient training.
23

---

Published as a conference paper at ICLR 2025
Table 15:
Dataset statistics. 1x~8x indicates the length of the textual context, controlled by the number of distractor captions. #tokens are counted using LongVA-7B tokenizer.
#Relevant Captions
#Distractor Captions
#Input Tokens
#Output Tokens
Dataset
#Samples
Modalities
Order-GPT (1x)
16k
2~4
100::50
1.8k
13.6
Text
Order-GPT (2x)
15k
200+
3.5k
13.6
Text
Order-GPT (4x)
15k
2~4
400+550
6.9k
13.5
Text
Order-GPT (8x)
15k
2-4
800++50
13.7k
13.6
Text
Attribute (1x)
34k
100::50
1.8k
8.4
Text
Attribute (2x)
15k
200+
3.5k
8.4
Text
Attribute (4x)
1.5k
400++50
6.9k
8.3
Text
Attribute (8x)
15k
800::50
13.7k
8.3
Text
Order-Templan (phrase)
30k
3~6
200+
3.6k
17.1
Text
Order-Templat (prefix)
30k
3-66
200+:50
3.6k
15.8
Text
Order-Templat (sentence)
30k
3-66
200+H50
3.6k
77.1
Text
Referring
22k
200
3.5k
4.5
Text
Grounding
22k
200+H50
3.5k
8.4
Text
Hotpot QA
90k
1.4k
4.0
Text
ALaVA-Next
1M
36.5
57.7
Text+Image
D ADDITIONAL EXPLORATIONS
D.1 RESULTS WITH QWEN2- VL-72B
To validate method's effectiveness state-of-the-art video language models, evaluate T3 on Qwen2-VL-72B (Wang et al., 2024), a state-of-the-as Video LLMs trained on large-scale video-text data. Following the best practices provided for fine-tuning, fine-tune the model using the ms-swift framework (Zhao et al., 2024) with the default LoRA setup. The model is trained with a batch size of 64 for 1000 steps. To preserve the general video understanding capabilities of the model, (Zhang et al., 2024d) in a 1:2 ratio, consistent with
combine our T3 dataset with LLaVA-Video-S0 our previous recipe. As shown in Tables 24 and 25, T3 yields consistent improvements across both 16-frame and 32-frame settings. The gains are particularly pronounced for temporal reasoning tasks on MLVU, e.g., Action Count: +9.2% and Action Order: +8.1%. These results demonstrate that T3's benefits extend to large-scale models, providing complementary temporal reasoning capabilities even when applied to models already pre-trained on video data. The consistent improvements across model scales highlight the generality and effectiveness of our approach.
D.2 ANALYSIS OF PERFORMANCE GAINS
To validate that our T3 method's improvements stem from enhanced temporal reasoning rather than domain- specific training, we conduct two critical analyses: (i) examination of potential dataset distribution overlap and (ii) investigation of model performance without visual inputs.
First, we analyze the data distribution between our T3 dataset and the video benchmarks, in terms of the entire questions and choices along, respectively. Specifically, our analysis consists of the following steps: (1) We utilize our validation set (each subset contains 500 examples) and randomly sample 500 examples from the three video benchmarks. (2) Similarity is measured using Jaccard Similarity:
(() [Bo (TW) (T2) where "BoW" refers the bag-of-words representation of piece of text, excluding stop words. (3) For each validation subset (e-g., Order, Referring, etc), we calculate the percentage of examples with Jaccard Similarity greater than 0.5 compared any example in the video benchmark. The results are shown Table 26, from which we can see that only small fraction of T3 dataset examples have similar counterparts in the video benchmarks. This indicates that the performance improvement of our method is due to enhanced temporal reasoning capabilities, rather than simply memorizing the distribution of questions and answer choices.
Second, we evaluate model performance in text-only setting by removing video inputs. As shown in Tables 27, 29, and 28, LongVA-7B3 VA-7B's performance drops significantly without visual context (e.g..
24

---

Published as a conference paper at ICLR 2025
Algorithm 2: Textual temporal QA generation for the Order aspect using templates.
Input:
Caption pool Cpool, number of relevant captions Recaps number distractor captions Makapp large language model LLM(), target to shuffle bg {sentence, phrase, function to extend context Fxt Cont Cpoot; C, n), function to extract noun phrase from a sentence ExtrePhrase((C C",, order-related question templates Q*P, prefix templates
peenne, prefiat fix)
Preffe
Output: SFT data sample (Xm. Xour
Sample relevant captions
C = [C, -...., G" pooo
Create extended context
Cezt ExtCont(C C, Mddap)
Sample & question from the templates, e.g., "Reorder the
following captions according to the above video."
Q ~ Q'r
lg = sentence then
Q Q @ Shuffle(C.) (C,f Add shuffled captions after the question
C
10
end
if tg = phrase or tg == prefiz then
12
Extract noun phrases from the captions
P (ExtrePhraseCC.)IC E C.}
14
Shuffle the phrases
P Pahuf = Shufile(P)
16
ig = prefix then
Add prefix before the phrases, e.g., (1) (2) (3)
P Pahuf = {Ppfo @ Ppha E Phaff Ppfa E Ppreffrr)
18
19
(Ppfæ + Ppha Ppha P,fp E Ppreffin
end
21
Add shuffled phrases after the question
22
Pshaf
23
Sample three other permutations for form the options
24
P @ Samppee(ermutations([) (P3))
25
Q Q @ Shuflee()
26
end
28
Concatenate context and question
29
Xm
Ceet + Q, Xout
from 55.9% 41.5% TempCompass). Notably, T3 training provides improvement in this text-only scenario (41.3%). This finding strongly suggests that our method's effectiveness comes from genuine enhancement of temporal reasoning capabilities rather than memorization of answer patterns or question distributions.
D.3 CORRELATION BETWERN TEXTUAL AND VIDEO TEMPORAL UNDERSTANDING
To assess transfcrability more intuitively, we calculate correlations between textual validation ac- curacy and benchmark scores of models trained strong correlation with TempCompass overall accuracy (Pearson r = 0.89, p < 0.01), and MLVU accuracy (Pearson 0.85, 0.01). It also positively correlates with Video-MME performances (Pearson = 0.59, < 0.05). These findings validate the feasibility of enhancing video temporal understanding from the LLM side.
textual samples. Figure 10 illustrates
25

---

Published as a conference paper at ICLR 2025
Algorithm 3: Textual temporal QA generation for the Attribute aspect.
Input: Caption pool ppeo. number of distractor captions deeap large language model LLM(), set of attributes a {color, light, shape, posture, emotion), attribute modification prompt Pa. QA generation prompt Pqa. function to extend context ExtCont( ExtCont(Cpool. C.n) n)
Output: SFT data samples (Xi. Xoui
Sample caption from the caption pool
Cr Cooo
Generate similar captions by modifying color, light condition, shape, posture, or emotion
Cattr = (Caur lattr E a) = LL, Pa)
Create extended context
Cext (Cattr attr a), where Cart = Cxtoont(Cpooll Cp @ Cattr, Rdeap)
Generate questions and answers
Q, = L Pga). where Q (Quttrlatth a), A (Aattrlattr a) Concatenate context and question
10
Xin
(Catt @ Qattrlattr E a). Xaut A
Algorithm 4: Textual temporal QA generation for the Temporal Referring aspect.
Input: Caption pool Cpoo number of distractor captions Ricea- large language model LLM().
function to extend context ExtCont (Cpool C, n). element types
c (object, action, attribute). temporal location reference
t = at the begin, at the middle, at the end}, caption generation prompt Pa. QA
generation prompt Pqa
Output: SFT data sample (Xim: Xou
Sample a caption from the caption pool
Cr Cpoot
Generate three similar captions for each element
C (C = IIM Pe), where Ccl 3
Create extended context
Cext = (C E e}, where C = ExtCont Cpeol: Ce Rieaa)
Generate questions and answers. For each element, the three captions share the same question but have different answers.
Q A LLM((C, P.a); where Q (Qele e),, A ({AAA A.) E e), =3
Add temporal reference to the questions and concatenate the context
10 X {(C @ Qu Ot E)leett X.m X.m A
Temp Compass Accurcy (%))
61
Pauson i: 0.89
Pausson 1:
Perssnn 0.59
60
P-saue I. 16e-00
MINU MI Accurac (16)
P-yplee 4.31000
Viden-MME lecameyy
P-value 4.32=02
05
80
Textual Validation Accuracy (%)
Textual Validation A Accuracy (%)
Textual Validation. Accuracy (%)
Figure 10: Textual temporal reasoning accuracy correlates positively with video understanding results on three benchmarks.
26

---

Published as a conference paper at ICLR 2025
Algorithm 5: Textual temporal QA generation for the Temporal Grounding aspect.
Input: Element types = ffobject, action, attribute), ), similar captions C = (C. E e). extended context C Ce le E e), questions for the captions Q = (Qale E e). answers to the questions A = {{A]A He e), large language model LLM(), declarative statement generation prompt Pas grounding question templates
Q = =Q'" !9')
Ji=1
Output: SFT data sample (Xm. X } Initialize statement set
S S=
for E
Generate three declarative statements, corresponding to three answers in A
S. = LLM((Qe. Ags P.), where [S.] =3
S @ S.
end
Initialize SFT data sample set
Xan Xout = {}.(} 10 for E do
11
for S. do
# Sample grounding question template, e.g., "In whi ch part of the video can we observe [X] ?" and insert the statement it. Xin Xin U (Catt @ Replace(Q, S)), where Q dF
Xau = X (4), where {at the begin, at the middle, at the end} is determined by
14
the location of corresponding caption in C
end
end
27

---

Published as a conference paper at ICLR 2025
Table 16: The prompt used to generate textual temporal QA for the Order aspect.
You will be presented with a list of captions describing keyframes of video. Your task is to generate five multi-choice questions (and corresponding answers) based on the captions. The questions should focus on the sequential order of the keyframe contents. Make sure that the questions are related to the order of the keyframes and diverse.
Here an example captions and the corresponding questions and answers:
Captions:
The image shows a person playing basketball.
2. The image shows dog running the grass.
3. The image about a beautiful flower on the table.
4. The image illustrates bustling city street.
Questions and Answers:
""gas":
"questton"": "Sort the events from the video by their chronological order. (1) city street; (2) dog running on the grass; (3) person playing basketball; (4) flower on the table.", "options": (3))) ((3)))((4)(1)" "())())(4)33)" "(1)(4)(3)(2))
"answer": ((3)))(4)(I))
"question": "Organize the listed events from the video according to their time sequence: (1) city street (2) dog running on the grass (3) person playing basketball (4) flower the table".
"options": [ "'city street - dog running the grass person playing basketball - flower on the table", "person playing basketball + dog running on the grass - flower on the table city street", "dog running the grass - city street flower the table person playing basketball", "city street + flower on the table + person playing basketball - dog running on the grass"
"answer": "person playing basketball - dog running on the grass -> flower on the table > city street"
"question"" "What is the correct order that objects appear in the video?".
"options": "doo, flower, person, street", "person, dog, flower, street", "street, flower, person, dog". "flower, street, dog, person"
"person, dog, flower, street" ""question": : "In what sequence do the events occur in the video?".
"answer":
"options": "a dog running and then a person playing baskethall", "a city street is shown and then flower is shown". "a flower appears followed by city street", "a city street appears followed by a dog runnin ], "answer": "a flower appears followed by city street") }
Now please generate five question-answer pairs based on the following captions. Ensure your resonse follows the above ISON format and style of the example question-answer pairs.
Captions: limage.captions'
Questions and Answers:
28

---

Published as a conference paper at ICLR 2025
Table 17: The prompt used to generate similar captions by modifying attributes in the original caption.
You will be presented with original image caption. Your task is enrich this caption with details regarding the attributes of objects or environment. The attributes could include but not limited these aspects: 'light condition', "color', "size & shape', 'emotion' and 'posture'. You are required to create two distinct captions for each attribute. These two captions should contrast each other in terms of the corresponding attribute (c.g.. black versus white for color', small versus big for 'size & shape").
Here are some examples of original caption and enriched captions related to different aspects:
Original Caption: The image shows a person sitting the chair.
Enriched Captions:
("captions": : "light condition": [ ["The image shows person sitting the chair with beam of light illuminating his face.", "The image shows a person sitting on the chair. His appearance is barely recognizable in the dim environment." ]. "emotion": [ "'The image shows a person, with big smile, sitting on the chair", "The image shows an angry person sitting on the chair" "posture": T'hhe image shows person sitting relaxing on the chair.", "The image shows a person standing straight in fromt of the chair."" J. "size & shape": "TThe image shows person sitting on a round, cylindrical chair.", "The image shows a person sitting on a square- square-shaped chair." ]
Original Caption: The image shows an apple on the table.
Enriched Captions:
("captions": { "color": [ "Th image shows red apple the table.". "The image shows green apple on the table." "size & shape": ''The image shows a big ripe apple on the table.". "The image shows rotten apple on the table."
Original Caption: The image illustrates air balloon.
Enriched Captions:
("captions": ('light condition": [Thhe image illustrates an air balloon in dark room.", ", "The image illustrates an air balloon in bright room.". J. "size & shape": ['The image illustrates deflated air balloon.", "The image illustrates an inflated air balloon." J. "color": "The image illustrates a light blue air balloon.", "The image illustrates an air balloon in yellow color.") ]
Now please generate enriched captions based on the following original caption. Ensure your response follows the JSON format of the above examples. Original Caption:
[original caption]
Enriched Captions:
29

---

Published as a conference paper at ICLR 2025
Table 18: The prompt used to generate textual temporal QA for the Attribute aspect.
You will presented with several pairs of image captions. Each pair captions depicts two keyframes in video. Your task is to generate multi-choice questions (and corresponding answers) for each pair of captions. The questions should focus on the change of attribute between the keyframe contents. Ensure that the questions are diverse and distinct from each other in wordings.
Here are some examples of caption pairs and generated question-answer pairs:
Caption Pair 1:
The image shows person sitting on the chair with beam of light illuminating his face. 2. The image shows person sitting on the chair. His appearance is barely recognizable in the dim environment.
Caption Pair 2:
1. The image shows person, with big smile, sitting on the chair. The image shows an angry person sitting on the chair.
Questions and Answers:
("qas": "caption pair I": { ("question": "How does the light condition change in the video?", "options": "rrmmininn stable", "turning darker", "turning brighter" J. "annser"" "turning darker" }. "caption_pair _air_2"; : ""question": "What change occurs the person in the video?", "options": [ ('changing from smiling to being angry", "changing from being angry smiling", "changing from feeling shy to being angry". "changing from feeling awkward to smiling" 1. "answer": "changing from smiling to being angry"
Caption Pair 1:
The image illustrates an air balloon in a dark room.
2. The image illustrates air balloon in bright room.
Caption Pair 2:
The image illustrates a deflated air balloon.
2. The image illustrates inflated air balloon.
Caption Pair 3:
1. The image illustrates a light blue air balloon. The image illustrates an air balloon yellow color.
Questions and Answers:
"question": "What transformation is occurring in the brightness the video?", "options": ""increasing", "staying the same", "decreasing" ]. "answer": "increasing" "caption_pair_2"
("qas": { craption_pair_1""
"question": "What is happening to the shape of the air balloon?". "options": [ ('it is getting bigger", "it is getting smaller", "its size and shape remains consistent" "answer": "it is getting bigger" }. "caption pair 3": { ""question": "How can we describe the change happening to the air balloon?", "options": [ "its color changes from grey to yellow", "its color changes from light blue to yellow", "its color changes from yellow to light blue", "its color changes from yellow to green" J. "answer": "its color changes from light blue yellow")
Now please generate question-ane answer pairs based on the following caption pairs. Ensure your response follows the above JSON format and style of the example (caption pairs]
pairs.
Questions and Answers:
30

---

Published as a conference paper at ICLR 2025
Table 19: The prompt used to generate similar captions for the Temporal Referring aspect.
will presented with an original image caption. Your task is to modify this caption by changing the original elements including object, action and attribute. Ensure that the modified element is distincl from the original ones.
Here is an example of original caption and modified captions:
Original Caption:
The image shows person sitting the chair.
Modified Captions:
("captions": ("change_ohject", "TThe image shows sitting the chair.", "The image shows dog sitting on chair.". "The image shows a cup placed on the chair." J. "change action": "'The image shows a person standing next to the chair.", "The image shows a person sleeping on the chair.", .attribute": "change.attribute" person sitting the chair.". "The image shows short person sitting the chair.". "The image shows strong person sitting the chair."
"The image shows person dancing nearby the chair." J. "change
"The image shows tall
Now please generate modified captions based on the following original caption. Ensure your response follows the JSON format of the above example. Original Caption: coriginal.caption)
Table 20: The prompt used to generate declarative statements for the Temporal Grounding aspect.
You will be given question paired with several answers. Your task is to reformulate each answer into simple declarative statement.
###Example1
Question: What is the color of the cat?
Answer 1: white
Answer 2: orange
Answer 3: black
Declarative Statement I: the cat is white
Declarative Statement 2: the cat is orange
Declarative Statement 3: the cat is black
###Exampee2
Question: What is the person doing?
Answer 1: running
Answer 2: playing guitar
Answer 3: swimming
Declarative Statement 1: the person running
Declarative Statement 2: the person is playing guitar
Declarative Statement 3; the person swimming
Now please reformulate the following question and answers according to the above examples. [question cnnestion_and..o [nnsers)
31

---

Published as a conference paper at ICLR 2025
Table 21: Datasets used in our textual temporal transfer explorations in Table 2.
samples
Description
Open-LLaVA.Nett
22k
LLaVA Next image-lett dataset.
HotporQA
Multiple-dccmmet question answering,
Temporal Change (wl Distractor)
22k
Balanced mixture of Orde-GPT and Attribute without interesting distractor captions
Temporal Change (Ix)
22k
Balanced mixture of Orde--GPT (Ix) and Attribure (Ix)
Temporal Change (2x)
Halanced mixture of Order-GPT (2x) and Atrrbbte (2x)
Temporal Clange (4x)
22k
Balanced mixture of OrdeeGGIT (4x) and Altributc (4x)
Temporal Change (8x)
22k
Balanced mixture of Order-GPT (8x) and Attribate (8K)
Temporal Change
misture Temporal Change
Order Template
22k
Balanced Mixture of Order Templatee (phrase), (prefix) and (sentence)
Temporal Referring
22k
Symthsi zed textual samples for enhancing Referring
Temporal Grounding
Symthisized icxiual samples for cuhancing Grounding
Order Template + Temporal Change (1x)
22%
Balanced mixture of Order Templatee and Temporal Change (1x)
Temporal Grounding Temporal Change (lx)
22k
Balanced misture of Temporal Grounding and Temporal Change (1x)
Temporal Referring Temporal Change
Balanced
Temporal Referring and Temporal Change (Ix)
T3
22k
Balanced Mixture of our synthesized subinsks.
Table 22:
Datasets used for transferring to holistic video understanding benchmarks in Table 3 and Table 4.
Dataset
samples
Description
LLaVA-Next
200k
LLaVA-Ne image-text SFT dataset.
HotpotQA w/ LLLVV-Nextt T3 w/ LLaVA-Noo Next (Ours)
200k
Horpot QA (66.7k) + + LLaVA--Next (133.3k) Temporal All (66.7k) + LLaVA-Nexx -Next (133.3k)
200k
Table 23: Performance continually increases with more training samples Video-MME and MLVU benchmarks
Model
Vidco-MME (Short Medium Long/ Overall)
MLVU (Macro-Avg)
Long
61.1 50.4/ 46.2 52.6
56.4
100K T3 w/ -LaVA-Next
/ 51.0/ 45.7 / 52.8
56.9
200K T3 w/ LLaVA-Next
63.3 / 54.8 46.8 / 55.0
58.1
Table 24: Qwen2-VL-72B results on MLVU with different frame settings.
MLVU
Action Count
Ego Reasoning
Necile QA
Action Order
Plot QA
Anooaaly Recognition
Topic Reasoning
Macro- Average
Downn--VL-724 frame)
52.3
46.7
62.0
65.5
+ T3 w/ LLaVA-Video (Ouns)
(+11.2))
55.7 (+3.1)
67.0 (+11.2)
51.1 (+7.4)
62.0
61.0 (-1.5)
86.3 (+0.7)
59.7 (+1.2)
Quen2-VLL-28 VL-72B (32 frame)
22,8
54.8
65.1
46.3
63.6
71.0
87.5
58.7
LLLVAA-VVV (Oun)
(2.02)
(+5.1)
(45.3)
54.4 (+8.1)
(4.0(4044)
(-2.0)
Table 25: Qwen2-VL-72B VL-72B results on Video-MME with different frame settings.
Video-MME
Short
Medium
Long
Overall
Qwen2-VL VL-72B (16 frame)
72.3
60.2
55.8
62.8
LLaVA- VA-Viee (Ours)
73.8 (+1.5)
62.6 (+2.4)
54.9 (-0.9)
63.7 (+0.9)
Qwen2-VL 2-B77- (32 frame)
75.8
62.9
56.6
65.1
LLaVA- (Ours)
78.3 (+2.5)
64.3 (+1.4)
57.1 (+0.5)
66.6 (+1.5)
Table 26: Percentage of T3 validation examples that similar (>0.5 Jaccard Similarity) to any examples in the video benchmark, in terms of Question and Choice. The results are averaged across five times of random sampling with the standard deviation reported in the parentheses.
Question
Choice
TempCompass
Video-MME
MLVU
TempCompass
Video-MME
MLVU
Attribute
0.0(0.0)
0.0(0.0)
0.04(0.08)
1.5(0.4)
0.1(0.1)
2.7(0.5)
Temelaeemp (phrase)
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0)
Order-Template (prefix)
0.0(0.0)
0,0(0.0)
0.0(0.0)
0.0(0.0)
0.0(0.0
0.0(0.0)
Order- Template (sentence)
0.0(0.0)
0.0(0.0)
0.0(0.0)
Order
0.0(0.0)
0.8(0.2)
0.3(0.1)
0.7(0.4)
0.8(0.2)
0.3(0.1)
Grounding
0.0(0.0)
0.0(0.0)
0.0(0.0)
0.7(1.4)
0.8(1.6)
0.0(0.0)
Referring
0.0(0.0)
0.0(0.0)
0.2(0.2)
2.7(0.4)
0.5(0.2)
4.2(0.5)
32

---

Published as a conference paper at ICLR 2025
Table 27: Comparison results of video inputs effect on TempCompass.
TempCompass
Action
Direction
Speed
Order
Attribute .change
Average
LongVA-7B w/ video input
92.3
37.3
42.0
54.3
51.7
$5.9
Long VA-7B w'o video input
50.3
36.1
42.0
39.7
38.9
41.5
T3 (Ours) w/o video input
48.8
36.7
44.5
38.4
37.2
41.3
Table 28: Comparison results of video inputs effect on Video-MME.
Video MME
Short
Medium
Long
Overall
LongVA VA-7 7B video input
61.1
50.4
46.2
$2.6
VB w/o video input
34.9
35.2
37.6
35.9
T3 (Ours) w/ video input
34.8
35.3
34.4
34.9
Table 29: Comparison results of video inputs effect MLVU.
MLVU
Action Count
Ego Reasoning
Needle QA
Action Onler
Plot
Anomaly Recognition
Topic Reasoning
Macro-
Average
Long VA-7B w/ video imput
25.2
48.6
70.4
41.7
68.1
58.5
82.2
Long VA-7B w/o video input
33.0
34.4
50.7
36.3
28.0
37,5
T3 (Ours) wio vidco input
32.5
34.1
49.3
42.9
41.2
25.5
29.2
36.4
33


Reviews:
[{"review_id": "uxrkPUBvKU", "reviewer": "Reviewer_GraR", "summary": "This paper addresses the limitations of video LLMs in temporal reasoning, which involves tracking objects or events in a sequence of video frames. The authors discover and point out the deficiencies stemming from the limitations of LLM processing. Inspired by this, the authors present a text-based framework which can be used to finetune the LLM part of the video LLM, which can result in improved temporal understanding from text-only data. This approach enables smaller models to outperform larger, video-trained models on benchmarks such as TempCompass and Video-MME.", "questions": "- Are the temporal-oriented questions generated from a different distribution from the answer choice distribution of the video benchmarks? For example, are the orders in which the objects appear in the temporal-oriented questions not in the benchmark datasets? Does this still improve the performance when the training data is manipulated to be strictly out-of-distribution to the benchmark data?\n- What is the performance of the fine-tuned video LLMs on the existing video benchmarks, when they are only provided the text part of the question? \n- If there is an analysis of the number of data points provided and the performance gain, it could provide a better insights to the validity of this approach.\n\nThe first two questions were critical in my rating decision.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Interesting approach to tackle temporal understanding of Video LLMs, that the LLM fails to understand the temporal behavior of the text prompts. \n- Utilizing only text data proposes an efficient and scalable framework, although the gains are questionable for larger models.\n- The performance gains are quite significant", "weaknesses": "- The biggest limitation of this work lies in the scope of the temporal concepts, but not simply because they were not covered in the paper. For example, consider rotation, direction, counting the number of a certain event, and relative velocity of an object relative to another object. There are some temporal concepts that cannot be represented in discrete sentences - for example, how would you describe the number of rotations of a diver in the form of text? How would you describe that one person is running faster than the other one without giving out the information explicitly?\n- The above concern also brings up the question of whether the success of this approach is largely due to how video LLMs are tested on current video benchmarks: given a few discrete frames, the models are asked to answer a question. For example, a good video LLM would solve problems in \"Order\" when given a frame of book cover, die, school bus and then split-screen (example in figure 4). If the vision encoder encodes this information but the output quality is bottlenecked by the LLM, this method would improve the model. However, it is unclear whether this method will apply to questions that require virtually the entire video frames to be answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcEPl1v51n", "reviewer": "Reviewer_UiVz", "summary": "This paper tried to use text-only reordering style data to fine-tune MLLM to enhance the temporal ordering performance.", "questions": "See comments above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Show that visual features contain necessary information for order understanding task.\n2. Writing is clear.", "weaknesses": "1. Why choosing LongVA? LongVA is only trained on images and text, therefore, evaluating and improving LongVA but evaluating on video understanding is not proper. Because everything is zero-shot manner. Instead, choosing backbones like LLaVA-onevision, phi-3.5-vision, or Qwen2-VL is proper. Those models have videos as the training data. If authors still show that there is clear improvement with their text-only ordering data, then the claim can be supported. \n2. Probing visual features is unfair. For LLM (decoder only), authors did not fine-tune. For visual encoder features, authors fine-tuned. Since the task diversity is limited, therefore, overfitting to this fixed task is not hard. This also explains that a LSTM can overfit this task. The interesting thing should be: if you use LSTM after LLM features, can LSTM also overfits? \n3. The empirical part of this paper demonstrate one thing: using text-only ordering data can improved relatively small LLM backbones. A more foundational question is: since large LLM can already solve the ordering task pretty well, is authors’ approach just a temporary  solution? Can authors’ approach improves Qwen-2-VL-72B?\n4. Evaluation limited. Evaluations on benchmarks like NextQA, ActivityNet, etc is still necessary. This is because ordering task is naturally just a small subset of temporal reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4aXXK0fJ0", "reviewer": "Reviewer_hvnd", "summary": "The paper attempts to address current limitations in temporal reasoning of Video Large Language Models (Video LLMs). Current models still often struggle with understanding temporal sequences, despite advances in video comprehension. Through a detailed analysis, the authors empirically demonstrate that the bottleneck primarily lies in the LLM's capability to understand temporal concepts. This is in contrast to a multitude of concurrent work that often aim to improve the temporal encoding or adaptor functions. To address this limitation, this paper also introduces the Textual Temporal Reasoning Transfer (T3) approach, that generates temporal reasoning tasks solely in textual format. The experiments reported in the paper demonstrates the effectiveness of the approach where it obtains performance gains in multiple video question answering benchmarks, without requiring additional video and language annotations for training.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) In terms of significance, this paper addresses the problem of video and language reasoning in multimodal Large Language Models (MLLMs) from a new perspective. It attributes the limitation of existing (MLLMs) to the LLM component instead of the video or adaptor functions. This is a very interesting perspective since it proposes an approach that improves the LM's ability to understand temporal concepts through text. The proposed Textual Temporal Reasoning Transfer (T3) approach creatively uses text-only datasets to improve temporal reasoning without requiring any video-language annotations.\n\n2) The quality of the paper is relatively high, especially with regards to the empirical analysis of the bottleneck in video and language reasoning. The experiments conducted in the analysis are well motivated. The finding that the performance of existing MLLMs is constrained by their LLM decoders instead of the visual encoding function is insightful.\n\n3) In terms of clarity, the paper is well-written and motivated. The model figures are informative and especially helpful in helping the reader to understand the differences between questions that focus on different temporal reasoning abilities in videos.", "weaknesses": "1) The text-based tasks used in T3 are largely devised based on templates and may not always be similar to the language concepts that are contained in real-world videos. For example, such templated sentences may lack the subtle temporal cues that naturally occur in descriptions of complex events. This might affect the model's ability to generalize to different downstream tasks.\n\n2) In this work, the authors only address four temporal aspects: order, attribute change, temporal referring, and grounding. However, this excludes some other common aspects of temporal reasoning such as understanding causal relationships and durations. Including more training samples that require more complex forms of reasoning could improve robustness of the trained models across a broader spectrum of temporal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "krq08lCbSM", "reviewer": "Reviewer_SHtr", "summary": "The paper studies the video's large language models, particularly its temporal reasoning ability. It first designs probing experiments using synthesized videos and analyzes the bottleneck in temporal understanding: primarily on LLMs but not the vision encoder.  To mitigate the issue, the authors propose a simple text data augmentation method to leverage the image-text datasets to generate diverse\ntemporal reasoning tasks and enhance the video LLMs training. The experiments based on LongVA-7B show its competitive performances across different datasets.", "questions": "Besides the questions I listed before, there are two additional questions for the authors:\n\n1. How to explain the strong temporal reasoning ability of GPT-4o? Is it only because 4o has a very strong temporal reasoning ability from the language side, not related to its vision encoder or alignment? \n\n2. Is there any way you can conduct the probing experiments on real datasets, not the synthesized ones?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well structured and written, making it easy to understand its key points. \n\n2. The enhanced model, built on LongVA-7B, has shown strong performances, surpassing large-scale models such as InternVLChat-V1.5-20B and VILA1.5-40B.", "weaknesses": "1. The claims of the paper do not fully convince me of the paper made. I agree that most of the vision encoders used today do not help much with temporal reasoning, even though they are trained with millions of videos. However, this can not be claimed as \"the temporal reasoning most of rely on the temporal reasoning ability from LLMs. There might be reasons that the temporal reasoning also replies on the model alignment ability between the text and the images/videos. The authors should design extensive prompting experiments on more models and datasets to support their strong claims. \n\n2. The synthesized tasks are mainly based on GPT-4o. And for me, how to design the synthesized tasks seems very arbitrary and empirical. Are there any particular reasons you designed these tasks and how can you ensure its generalization on more domain/datastes? The authors should have more deep analysis and experiments on those two issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVpfe9YWUD", "reviewer": "Reviewer_c8nY", "summary": "This paper addresses the limitations in Vision Language Models (VLMs) for temporal reasoning tasks in video understanding. Through detailed probing experiments, the authors identify the primary bottleneck as the LLM component rather than the visual encoder. To address this, they introduce an approach called T3, which fine-tunes only the LLM component on text-only data, enhancing performance on several temporal reasoning benchmarks.", "questions": "1. Could you report results on larger LLMs (e.g., Qwen2-72B)? Although the performance gain is noted to be limited, these results could be informative for future research.\n2. Could you provide examples of captions generated by GPT-4o as mentioned in L-133? It would help clarify whether specific attribute information is inadequately encoded in the caption, potentially leading to near-random results in LLM responses. (You partial addressed this in L216-L217, but the performance achieved by larger model is still 20% percent lower than the performance achieved by LSTM, which makes it not convincing).\n3. Minor issue: It appears LLaVA-ReCap-558K is derived from LCS-558K (a subset of the LAION/CC/SBU dataset) rather than BLIP558K (as stated in lines 861-862) [1].\n4. I question whether the performance improvement is due to enhanced instruction-following ability (e.g., the ability to \"Answer the option only\") rather than true temporal reasoning. Could you report the portion of mis-formatted or invalid responses across all experiments to clarify this?\n\n[1] Section on Open Source Release: https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors provide a well-defined motivation for their work, verifying through experiments that the LLM, rather than the VLM, is the limiting factor in video understanding tasks, and proposed solutions.\n2. The authors introduce an innovative fine-tuning technique that only tunes the LLM within a VLM, bypassing the need for visual data. This approach yields significant performance improvements across several video understanding benchmarks.\n3. The dataset is generated in a fully automated manner, adding to the scalability of their approach.\n4. Extensive experiments and in-depth analysis are provided.", "weaknesses": "1. As mentioned in L-538, the approach shows minimal improvement when applied to larger LLMs, which could restrict its broader applicability.\n2. Several areas of the paper would benefit from further clarification (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)