Seeing the Arrow of Time in Large Multimodal Models
Authors: Zihui Xue
Mi Luo
Kristen Grauman
Venue: NeurIPS 2025 poster ()
Keywords: arrow of time
large multimodal models
video understanding
videoQA
Primary area: applications
TL;DR: Finding that today's LMMs poorly grasp the arrow of time in video, we propose ArrowRL to enhance their temporal perception and AoTBench for rigorous evaluation.

Abstract: The Arrow of Time (AoT)—time's irreversible flow shaping physical events—is fundamental to video comprehension, yet remains a significant challenge for modern large multimodal models (LMMs). Current LMMs struggle to perceive and utilize temporal directionality in video when responding to language queries, obstructing deeper temporal understanding. We tackle this deficiency by first providing a critical analysis of existing benchmarks and models. We then introduce ArrowRL, a reinforcement learning (RL)-based training strategy with an innovative reverse reward that instills AoT awareness by encouraging divergent video interpretations between forward and reversed visual frames. For rigorous evaluation, we additionally develop AoTBench, a new multi-faceted benchmark probing temporally challenging questions. Experiments show ArrowRL greatly advances temporal perception: it not only achieves substantial improvements on our challenging AoTBench but also demonstrably boosts performance on standard video question answering (VQA) benchmarks (with peak accuracy gains reaching over 20% and 10% respectively). This validates ArrowRL's effectiveness and highlights the critical need for dedicated AoT understanding in LMMs.

Seeing the Arrow of Time in Large Multimodal Models
Zihui Xue Luo
Kristen Grauman
The University of Texas at Austin
Abstract
The Arrow Time (AoT)-time's irreversible flow shaping physical events-is fundamental to video comprehension, yet remains significant challenge for mod- ern large multimodal models (LMMs). Current LMMs struggle to perceive and utilize temporal directionality in video when responding to language queries, obstructing deeper temporal understanding. We tackle this deficiency by first providing critical analysis of existing benchmarks and models. We then intro- duce ArrowRL, reinforcement learning (RL)-based training strategy with an innovative reverse reward that instills AoT awareness by encouraging divergent video interpretations between forward and reversed visual frames. For rigorous evaluation, we additionally develop AoT'Bench, new multi-faceted benchmark probing temporally challenging questions. Experiments show ArrowRL greatly advances temporal perception: it not only achieves substantial improvements on our challenging AoTBench but also boosts performance on standard video ques- tion answering (VQA) benchmarks (with peak accuracy gains reaching 20% and 10% respectively). This validates ArrowRL's effectiveness and highlights the critical need for dedicated AoT understanding in LMMs.!
Introduction
Our world unfolds with distinct rhythm, governed by time's relentless forward march. We watch cream swirl into coffee, smoke disperse, or a glass shatter-common events whose reversal would feel instantly unnatural, or physically impossible (Fig. (a)). This Arrow of Time (AoT)) concept is rooted in physical laws and shapes the causal structure of reality [31]. We humans navigate the world with an innate grasp the AoT, effortlessly perceiving the flow of events and decoding temporal directionality for visual narratives. Crucially, this sensitivity goes beyond identifying physical irreversibility; fundamentally involves understanding the sequential progression of events and interpreting the semantic meaning embedded within that progression. Equipping intelligent how events unfold over time is essential for
systems with this broader AoT achieving genuine video understanding.
Early research video representation leamiing has leveraged the inherent temporal dimension of video as a form of self-supervision. designing a variety of interesting pretext tasks such as forward/reverse sequence classification (an early form of AoT) 66, 23), video order prediction [48, 17, 30, 32] and enforcing temporal alignment [19. 75]. Throughout these developments, temporal understanding treated as an intrinsic, vision-only problem based on physical cues.
Today the landscape of video understanding is rapidly shifting with the rise of large multimodal models (LMMs) [39, 8), which link video perception with the generative and interactive capabilities of large language models (LLMs) via textual interfaces. This evolution motivates us to rethink AoT from a language-aware perspective. A video's semantic meaning is often intertwined with the AoT: consider Fig. 1 (b), where the forward and reversed videos yield opposite interpretations conveyed through language. While many efforts aim to improve the general temporal understanding of these
Project webpage: https://vision cs utexas. wttp/proocct///eeooo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Which video segment played in reverse?
LMM ArrowRL (ours)
LMM
(b) What is happening in the video?
Base LMM: gas stove burner ignites and produces blue fiame.
temporally- insensitive
Base LMM: The gas burner ignited and producing steady blue flame.
Base LMM ArrowRL (ours]: The video shows gas stove burner being turned off, with the blue flame gradually extinguishing.
ArrowRL (ours]: A gas stove ignitee and produces blue
AoT-aware
Figure I: Arrow of Time (AoT) perception challenges, demonstrated by a strong representative
base LMM (Qwen2.5-VL-7BB [5]). (a) Basic visual directionality (forward vs. reverse), trivial for humans, often confounds these models. (b) Deeper temporal insensitivity is also observed, where LMMs often generate the same description (e.g.. "ignite") for events with opposite semantics based on temporal direction. We propose ArrowRL to instill AoT awareness for LMM temporal perception.
powerful LMMs [85, 20, 77, 89, 38], we posit that AoT sensitivity is a fundamental yet overlooked component required for deeper understanding.
Consequently, aim to instill this core temporal sensitivity, empowering LMMs to leverage directional information when generating language responses as appropriate. This capability is essential for advancing LMMs past superficial pattern recognition, it would ground their internal world model [28, 27, 47] in the fundamental reality of temporal progression and causality, towards human-level reasoning.
Naturally, this leads us to ask: how capable are current LMMs in perceiving the AoT? Perhaps surprisingly. despite rapid advancements elsewhere, our findings indicate critical gap. First, profound temporal insensitivity is evident even in state-of-the-art LMMs. They fail on basic AoT- related tasks, such as distinguishing forward from backward video playback (Fig. I (a)), and ignore crucial semantic differences between forward and reverse sequences, erroneously providing identical descriptions (Fig. (b)).² Second, broadening our investigation, on multiple standard video dneestion-asseeree
identify alarming failure: (VQA) benchmarks [70, 21, 51, 46, 9, 37, 36, 68], shuffling reversing video frames often causes only slight even no performance degradation for leading representative LMM (see Fig. and details in Supp. B.2).
The observed performance reflects problematic interplay between intrinsic model capabilities and current benchmarks. First, on the modeling side, the observed failures clearly indicate a fundamental lack of temporal directionality understanding; this perceptual capability for grasping causal- temporal dynamics remains underdeveloped in current LMMs. Second, on the benchmarking side, existing evaluations fail to adequately probe AoT sensitivity. While the single-frame bias issue [7, 33, 89, 15, 11, 69] (where one static frame suffices for answers) is widely studied, we identify a distinct critical limitation: many multi-frama frame questions still lack dependence order, allowing models to succeed even with shuffled or reversed video. We present a systematic study across eight existing VQA benchmarks offer insights into this AoT insensitivity at both the benchmark and model levels, addressing a gap in prior evaluation studies.
temporal
Building these insights, work aims empower LMMs to "see" the AoT in video. We first propose ArrowRL, novel reinforcement learning (RL) algorithm based on Group Relative Policy Optimization (GRPO) [57]. The core idea is unique reverse reward that promotes divergence between the model's forward and backward video interpretations, fostering AoT sensitivity for temporally demanding questions. In addition, address benchmark inadequacies, secondary
"This is not general failure to process reversed videos -iitr--iistiitttion (OOD) inputs. See Supp. B.1 for control experiment where LMMs excel on temporally cemporally-insensitivee insensitive task regardless of video direction.

---

MCQ Ace. (%)
75-
Forward Shuffle Reverse
50
XaX0m AO
345-22 MA
LD DEME
EquSchema
LongVideBench
MYBench
NEXT-QA
Perteptionfisst
TomporatBencc (S)
Tompoontteench (L)
1TTATECS
VideoMME.
Figure 2: Missing the AoT: Multiple choice question (MCQ) accuracy of strong representative
LMM ILLaVA-OV-BB [34]) standard VQA benchmarks [70, 21, 51, 46, 9, 37, 36, 68] when processing forward, shuffled, and reversed video sequences. St short, L: long. The small or negligible performance drop across conditions highlights low temporal sensitivity, stemming from deficiencies in LMM capabilities and benchmark question design.
contribution, develop AoTBench, benchmark comprising three distinct tasks for rigorous evaluation of LMMs' AoT perception capabilities.
Our experiments demonstrate the efficacy of ArrowRL. Across three distinct base LMMs, it con- sistently boosts task performance on AoTBBench, and, importantly, generalizes strongly to multiple standard VQA benchmarks [44, 80, 15] (e.g., +65.9% relative gain [80]). These results underscore that improving temporal sensitivity in LMMs translates to broader performance gains, revealing the fundamental role of AoT awareness in achieving genuine temporal perception.
2 Related Work
"Time" in Video Time has long been recognized as valuable self-supervised signal in video [52, 66, 23, 48, 17, 32, 30, 19, 75]. Early works capitalize on this property within the visual domain, leveraging the arrow of time (AoT) by using forward/reverse sequence classification as pretraining task, which benefits downstream action recognition [52, 66, 23]. Relatedly, random shuffling has been explored for action analysis [48, 53, 56]. Current multimodal vision foundation models [22], such as CLIP [54] and VideoCLIP [72], can benefit action understanding, driven by text-video contrastive alignment objectives. A few works [3, 65, 18] employ temporal reversal as source of negative examples for aligning video and text encoders, yet the discussion remains specific to dual-encoder architectures and only addresses basic temporal concepts with before/after relations [3] or action antonyms [65, 18]. Critically, while the field has advanced to LMMs capable of free- form language-based interaction with video content [39, 8]. we find that these models exhibit surprising insensitivity to fundamental temporal directionality. This suggests their internal world models [28, 27, 47] may fail to incorporate the basics of real-world event progression and causality. While recent work investigates AoT in the context of text-only LLMs [50], and concurrent efforts explore it for physical reasoning [2], how to instill AoT awareness within LMMs remains a critical but largely unexplored problem.
Large Multimodal Models (LMM) LLM capabilities have been extended to the visual domain through the development of LMMs [39, 8), which integrate vision encoder with powerful LLM, enabling them to perceive and reason about visual content. While early LMMs focus on images [42, 41, 87, 61], research on video-LMMs [12, 85, 89, 38, 34, 63, 5, 62, 43, 73, 14, 74] is rapidly progressing, including the development of high-quality video datasets [12, 85] and exploration of the architecture design space [89, 38, 43]. Our work is orthogonal to these data and architecture innovations. We investigate how a core understanding of the AoT can be instilled directly through training, offering a complementary path towards enhanced temporal perception in LMMs.
Benchmarks with Temporal Sensitivity Evaluating the true temporal understanding of LMMs challenging, even with many benchmarks available for video question answering [70, 21, 51, 46, 9, 44, 37, 36] and captioning [10, 62). Recent studies [7, 33, 89, 11] reveal how questions on many popular video benchmarks can be answered correctly with minimal temporal context (single/no frame) due to static video content, non-temporal question design, or language queries that provide unintended hints. However, merely requiring multiple frames is insufficient for probing deep temporal understanding. Our work moves beyond the "single-frane frame bias" to address the overlooked issue that many multi-frame questions still lack sensitivity to temporal directionality. Inspired by preliminary experiments that explore frame shuffling on limited datasets [15, 69], our work provides, to
3

---

Model Temporal Sensitivity
Benchmark Temporal Seasitivity
40-
Avp. Btd uto Auc. Deup
60
Forward
Reverse
Owen2 57L-73
Vilco
MCD
38
0.2-
20
Video
BPLLA
PLLaNA 78
famler BB
Lo Wa-e----- 7B
Oven2.5Vii 7-
VITSTECS
Temporadennc (3)
NBKFO
Perspitta Teat
Video
Haegroad (1)
Vangraand
Trapi TeepCampau
TVbaach
Figure 3: Temporal Sensitivity Analysis. (Left) Model Sensitivity: comparing MCQ accuracy on TVBench [15] for various LMMs [36, 43, 73, 62, 34, 85, 63, 5], on forward vs. reverse video sequences. LLaVA-OV-7B [34], LLaVA- LLaVA-Video-72 7B [85], and Qwen2.5-VL.
[5] demonstrate highest accuracy and AoT sensitivity. (Right) Benchmark Sensitivity: comparing the proposed temporal divergence score (TDS) for various VQA benchmarks [37, 9, 70, 51, 21, 80, 44, 15], along with relative accuracy drop (mean + std.) calculated using forward reverse videos. Benchmarks with higher scores (Vinoground [80], TempCompass [44], T'Bench [15]) are identified as temporally sensitive and better suited for evaluating temporal perception. S: Short, V: Video, T: Text.
knowledge, the first systematic and rigorous evaluation of AoT sensitivity across eight diverse VQA benchmarks. By comparing fine-grained metrics with forward, reversed, and shuffled video, we offer benchmark- and model-level insights currently lacking in the field.
Reinforcement Learning to Enhance LMMs Post-training using reinforcement learning (RL) is increasingly employed to refine LLMs, aligning model responses with human preferences and enhancing complex reasoning abilities [49, 88, 55, 26, 57]. A family of successful algorithms, including PPO [88]. DPO [55], online DPO [26], and GRPO [57], is now being actively extended to LMMs, although applications have mainly focused on image-LMMs [59, 78, 81, 84), with video comprehension being a newer frontier. While these efforts have concentrated on mitigating visual
hallucination [83, 64, 79] or (concurrently) enabling structured reasoning [20, 13, 45] like Chain- of-Thought [67], our work investigates a distinct dimension: enhancing the model's fundamental temporal awareness, by leveraging AoT signals naturally embedded in videos.
Approach
Our preliminary experiments (Fig. 2) reveal striking insensitivity to temporal directionality, sug- gesting deficiencies in both the benchmark design and/or LMMs' intrinsic ability to perceive the AoT. We systematically disentangle these intertwined factors. by first examining temporal sensitivity for existing benchmarks (Sec. 3.1), and then detailing our proposed approach to bestow AoT perception (Sec. 3.2) along with AoTBench, new benchmark designed to evaluate this capability (Sec. 3.3).
3.1 Are Today's Benchmarks Time-Sensitive? Revisiting Current LMM Evaluation
To systematically study the inherent AoT sensitivity of existing benchmarks without costly human evaluation, we employ an automated strategy using LMMs to probe benchmark properties. To select suitable "evaluator" L.Mss, we assess pool of candidate models under 10B parameters (for compu-
tational feasibility) VideoChat2 [36], ST-LLM [43], PLLAVA-7B [73], Tarsier-7B [62], Qwen2-
VL-7B [63], LLaVA-OV-7B [34], LLaVA VA-Vieoo-B [85], 1wwn2.5-VL-7B [5]-on TVBench [15]. TVBench is selected as its design explicitly requires higher degree of temporal understanding [15]. Comparing accuracy for forward vs. reversed video on TVBench (Fig. 3 (left)) reveals leading models that exhibit both high overall accuracy and a significant difference (delta). Consequently, we select
LLaVA-OV-7B [34], LLaVA- LLaVA-Video-77 Video-7B [85], and Sowen2.5-VL-77 [5] for subsequent analysis.
We then evaluate the three selected LMMs across eight popular VQA benchmarks (VITATECS [37], TemporalBench [9], NEXT--QA [70], PerceptionTest [51], VideoMME [21]. Vinoground [80]. Tem- PCompass [44] and TVBench [15]). To quantify benchmark and sample-level AoT sensitivity (facilitating later construction of AoTTBncch c.f., Sec. 3.3), we propose the temporal divergence score
NNot that our analysis focuses on MCQ tasks for their popularity and clear evaluation structure, as opposed open-ended QA or captioning tasks that require additional LLM evaluator, introducing uncertainty.

---

Low-TDS Sample
High-TDS Samples
Q. which hand does the person
Q. What the white hair
Q. What happening pink flowers?
conceal a small object drawn sign?
after picking the girl up?
Left
Bring girl closer to the tree
Turning from withered to full blossom
Right
Follow her instructions to sit
Turning from full blossom to withered
Figure 4: Illustrative low vs. high TDS VQA examples, sourced from PerceptionTest [51], NExT- QA [70] and TempCompass [44]. Samples with high TDS necessitate AoT reasoning, whereas the low-TDS sample can be solved without understanding video temporal progression.
(TDS). For evaluator LMM f. consider MCQ sample comprised of video us, language query 1 and ground truth answer yi from K, options. Denote the reversed video as o. The MCQ setup instructs the model to respond only with the option's letter. The first-token probability distribution therefore a direct measure of the model's confidence over the K, choices [6, 29, 40J. We compute the probability distribution for both forward and reversed video: p: = TrrrstTToenProbb(c v. v,,4) 44) and P; FirssTTokenPPnen TokenProb(fF (f, 0; 4t), where Ps: Px E RRKK,TT .To ensure our analysis focuses on instances gen- uinely requiring arrow of time (AoT) reasoning, we first filter out outlier samples where performance on the reversed video exceeds that of the forward one, as these cases typically indicate ambiguous or irrelevant temporal cues. The TDS for any remaining sample is defined using KL divergence: TDS; Dri. [ps [..]. A high TDS signifies that the model's prediction is highly sensitive to the video's temporal direction-canl causing disagreement between forward and reverse responses responses-therefinn therefore suggesting the sample requires AoT understanding for correct interpretation and answer.
Compared with prior efforts that examine accuracy delta (comparing full video vs. single-frame performance) [7, 33, 89, 15], which only reflects lop-prediction flips, TDS is inherently more fine- grained by comparing the output probability distributions from forward vs. reverse video inputs via caaetrrrncecattuiinnggn
KL divergence capturing shifts in confidence or the ranking of alternatives. This enables robust quantification of AoT sensitivity and allows for effective sorting of samples by AoT sensitivity.
Fig. 3 (right) presents the benchmark-level AoT sensitivity analysis, comparing TDS values. See Fig. 4 for low and high-TDS examples. We also report the coarser relative accuracy drop (averaged across evaluator LMMs) for reference. Our analysis reveals varying levels of AoT sensitivity across benchmarks: TVBench [155] Vinoground [80]. and TempCompass [44] emerge as sensitive whereas benchmarks like VITATECSS [37], TemporalBench [9], and NEXT-QA [70] show lower sensitivity to video playback direction. These benchmark-level insights, derived from TDS, can aid future benchmark interpretation, and they directly influence our AoTBench design (Sec. 3.3).
3.2 ArrowRL: Learning the Arrow of Time
We now discuss our approach to equipping given LMM mo with AoT understanding for enhanced temporal perception. Our core premise is that videos are inherently dynamic, and LMMs should cap- ture this dynamic progression when responding. Specifically, given a question q = (v. I) (composed of vidco and language query !) and a target response o", drawn from the underlying data distribution P(Q, O+) of standard instruction tuning dataset, the goal is for Ty to produce responses that align well with o". To foster broad AoT sensitivity, our training utilizes a diverse suite of tasks-MCQ, open-ended QA, and video captioning (see Sec. for the specific datasets used).
Given that modern LMMs are pretranee from extensive data, we follow advances in LLMs [49] and employ RL a targeted post-training strategy. GRPO [57] is a leading RL algorithm that refines model outputs by comparing multiple responses generated for one prompt, adjusting probabilities based relative scores within that specific group. In this way it provides finer control compared to using absolute rewards or response pairs and is thus well-suited for our goal of teaching AoT sensitivity. We adapt GRPO with novel reward structure focused on temporal directionality, and term our approach ArrowRL. Fig. 5 provides an overview of the ArrowRL training process.
Given question q. let for S denote the set of G candidate responses generated by ma. We design reward signal to rank these responses within the GRPO framework, integrating two complementary objectives: target fidelity and reverse reward.
5

---

Reward Computation
Video Reversal
4
A
Sim
Policy Model
O2
7
F2
A2
T
1 Sim
...
Sim [0
0g
fg
AG
Ref Model
update
Figure 5: Overview of our proposed ArrowRL framework. Given input video and language prompt I, the policy LMM me generates a group of candidate responses {or);-1- The reward calculation for response o, combines a fidelity reward (r; fd), encouraging similarity between o, and larget o², and reverse reward (r;")), enforeing temporal directional sensitivity between o; and response ò generated from the reversed video ö. These rewards {ri] are then normalized to obtain advantages {Ai] which drive the GRPO optimization relative to the reference policy Tref-
Target Fidelity Reward First, we define fidelity reward encouraging alignment between each can- didate response o, and the target response o", fid Similarity o²). The function Similarity(:: returns a scalar between and and is calculated via accuracy matching (1.0 o, o", else 0.0) for MCQ tasks, or via an LLM-based similarity score for open-ended QA and captioning (see Supp. B.3 for the specific prompts and the LLM judge employed).
Reverse Reward Second, central to ArrowRL is the reverse reward. Rewarding only fidelity to the target response o* (via r. (d)) can be insufficient for capturing directional nuance- particularry if ot lacks temporal progression details. Therefore, we utilize the reversed video (denoted by û) as a natural directional contrast signal, which often distinguishes itself from the forward version, to enforce AoT sensitivity. Intuitively, an AoT-sensitive model should yield different responses when observing the same video in different sequential order (though not without exceptions, which we handle below). Specifically, we maximize the dissimilarity between forward candidate responses o and the reverse response o, i.e., the response produced by my when given ö. The reverse reward is thus defined as: r, rev 1 Similarity(o,, o). This reward design discourages generic static responses and compels attention to the dynamic, directional details conveyed by AoT in forward v. Note that in this process, the reversed video (7) is utilized solely signal for computing the reverse reward (r, re)e; the training objective always pertains to generating correct and temporally-awan responses for the forward video (u), not improving responses conditioned the reversed video itself.
Fig. 6 illustrates this mechanism. conditioned response ō will receive low r. rex For instance (left example). both o2 and ō describe "cooking beef" (where ø's description is contextually wrong for the forward video), thus 02 is penalized via lower reward. In the right example, given that the reversed video yields ô of "picking up", candidate 02 stating "placing...on shows larger dissimilarity from than the more generic candidate o stating "arranging". Thus, 02 receives a higher reward as it better reflects the forward video's is distinct temporal dynamics.
candidate response o that incorrectly mirrors the reverse-
The two reward components work in tandem: the reverse reward compels the model to perceive AoT by enforcing divergence between forward and reverse responses, while the fidelity reward maintains relevance to the target response, preventing trivial outputs that merely differ from the reverse case. This complementary design encourages meaningful, temporally-awarn for ArrowRL on response i is defined as: T; r, fid + a,T;" ,or where a; balances the two rewards.
responses. The final reward
natural concern arises regarding the applicability of the reverse reward when its underlying premise semannic distinctiveness between forward and reverse video-is is not met (e.g. for static queries, non-dynamic or cyelic videos). We address this through training data selection (c.f., Sec. 4), adopting samples exhibiting high temporality, and also through dynamic weighting scheme for the reverse reward's contribution a; During training, we measure the similarity between the reverse response and the target one o". If Similarity o²) > 7 (where is threshold hyperparameter),

---

Q After seasoning the beef, does the chef immediately start chopping vegetables?
Q. Describe the video.
[Reverse response The showe person picking up piece cheese from wooden board.
[Reverse response No, chef does immedistely start chopping vegetables. Instead, he continues to cook the beef in the pan.
[Candidate response 0,] The video diepleys women's hands arranging various types cheese on a serving board.
[Candidate response o,] No, the chef doesn't immediately start chopping vegetables. After seasoning the beef, we see the chef preparing the pan with oil and duck fat. - 0.45
r rt 0.6
[Candidate response 0y] The video depicts female hand reaching out and placing a piece of yellow cheese on a wooden table, which already has several other cheeses, ss well as breed and berries.
[Candidate response 0j] the chef starts cooking the pan instead chopping vegetables.
12 t 0.15
1 m 0.8
Figure 6: Illustrating the reverse reward (r; re), designed to favor forward responses o; that dissimilar to the response generated from the reversed video (0), thus enforcing AoT sensitivity.
suggesting the sample is not highly AoT-sensiiive, we disable the reverse reward for that sample by setting a, = Otherwise, a takes a default hyperparameter value a.
With this reward in hand, we proceed to the optimization step. For each question and the corre- sponding group of generated responses {o:}e G we compute the group reward {re). and then obtain advantage A,, as the normalized reward, i.e. A,, = =; [-mmen(r) The policy model is
sid(rr
optimized by maximizing the following GRPO objective:
lod1
To(oi, q: (i,t)))) Ai, Te (or, q. O(.e)
TARFO (0) P(1.0r)--((000))()(00000 (Ol))
min
G
10:1
To(o y, ci,)), e, 1 +
}
clip
Ai,
3l...
[TTo (tret
(1)
(oi q, 0c,c())
where and are hyperparameters controlling clipping and KI. regularization respectively, Ty the current policy being optimized, Thi is the policy from the previous iteration used for importance sampling and मe the reference model (set as the initial model checkpoint) used for regularization.
3.3
AeTch:: Addressing Temporal Evaluation Gaps in Video Understanding
Our preceding analysis (Sec. 3.1) and ArrowRL (Sec. 3.2) highlight the need for focused evaluation benchmarks. To this end, we propose AoTBench, the first dedicated benchmark to assess temporal direction sensitivity- core component of robust video perception throouh three distinct elements:
Sequence Direction Classification. The task is to classify video playback as forward or reverse based visual cues (Fig. 1 (a)), capability with applications like video forensics, yet remains unexplored for LMMs. Following early vision work [66, 23], we adopt videos from Reverse- Film [66] and UCF101 [58], from which we manually select and verify total of 613 videos demonstrating strong temporal properties suitable for this evaluation.
Directional Caption Matching. This task probes the model's ability connect video dynamics with corresponding textual descriptions. We target semantically directional video sequences where forward and reverse playback correspond to distinct textual descriptions (Fig. (b)). Leveraging the RTime benchmark [18], which conveniently provides 2,000 such high-temporality videos alongside human captions, we formulate two complementary MCQ tasks: (1) video-to-text (V2T): choosing the correct caption (forward vs. reverse) for given video, and (2) text-to-video (T2V); choosing the correct video sequence (forward vs. reverse) for a given caption.
AoT-sensitive VQA. We curate temporally sensitive VQA samples from existing benchmarks. Using the per-sample TDS computed earlier, we select the top 200 high-TDS questions from each source benchmark, after removing easy samples unanimously answered correctly by all evaluator LMMs. This yields a subset of 1,800 VQA samples that necessitate robust AoT understanding and challenge current models' temporal perception capabilities. We illustrate the effectiveness of using TDS to isolate temporally challenging VQA samples in Fig. 4.
7

---

Table 1: Benchmark results comparing ArrowRL-enhaneed LMMs with their base models, along- side leading ppen-source models of comparable scale and larger proprietary models (gray row). Cells marked with indicate results reported in literature; others are reproduced using official code. AoT'Bench consists of three distinct tasks: (1) sequence direction classification (direc. cls.) on ReverseFilm (RFilm) and UCF101 (UCF), (2) directional caption matching (cap. match), and (3) AoT-sensitive VQA (AoT-VQA). ArrowRL consistenly improves models' temporal perception ability across all three different base LMMs, leading to great performance gains on both AoTBench and existing video benchmarks.
AoTBench
Existing Temporal Benchmarks
Direc.
Cls.
Cap.
Match
A-T-
Temp
TV
Vinoground
Model
RFilm
UCF
T2V
V2T
VQA
Comp.
Bench
Text
Video
Group
Random Chance
50.0
50.0
50.0
50.0
39.3
44.3
33.3
25.0
25.0
16.7
GPT-4o [1]
52.8
54.0
56.5
69.5
67.8
74.8*
55.2*
54.0"
38.2"
24.6'
Gemini-1.5-Pro [60]
51.4
52.8
60.4
58.9
57.3
52.8*
35.8"
22.6"
10.2*
Aria [35]
50.0
51.6
56.4
57.5
55.9
73.6*
51.0"
MiniCPM-V 6-2.6 [76]
50.0
51.2
54.8
61.3
46.8
69.1"
32.6*
29.2'
11.2*
IntemL.M-XC-2.5 [82]
50.0
51.6
52.0
53.6
45.8
67.1
51.6*
28.8*
27.8"
9.6*
LLaVA-Video-7B [85]
50.0
51.6
57.2
63.1
46.7
71.4
53.2
37.4
28.6
13.6
Video-R1-7B [20]
50.0
51.6
56.3
62.5
46.7
73.1
53.7
36.8
28.2
11.8
LLaVA-OV-7B [34]
50.0
51.6
56.3
62.4
46.2
69.6
48.9
42.2
29.0
14.8
+ ArrowLL
54.2
57.4
57.9
66.1
53.2
70.7
49.9
43.4
31.2
17.6
(Gain)
(+4.2)
(+5.8)
(+1.6)
(+3.7)
(+7.0)
(+1.1)
(+1.0)
(+1.2)
(+2.2)
(+2.8)
Qwen2-VI-78 [63]
50.0
51.6
56.3
62.3
44.3
72.3
48.6
40.0
31,6
14.4
+ ArrowRL
69.1
72.6
57.1
68.8
51.1
74.8
51.5
46.6
33.6
20.0
(Gain)
(+19.1)
(+21.0)
(+0.8)
(+6.5)
(+6.8)
(+2.5)
(42.9)
(+6.6)
(+2.0)
(+5.6)
Qwen2.5-VL-7BB [5]
50.0
51.6
53.4
66.6
49.6
73.8
54.7
46.2
31.4
16.4
+ ArrowLL
51.4
54.8
55.6
69.6
58.8
75.5
56.2
48.8
42.4
27.2
(Gain)
(+1.4)
(+3.2)
(+2.2)
(+3.0)
(+9.2)
(+1.7)
(+1.5)
(+2.6)
(+11.0)
(+10.8)
4 Experiments
Post-training Data Our ArrowRL training data comprises comprehensive suite of tasks, with data selected or adapted for each to emphasize scenarios requiring AoT awareness: (1) MCQ tasks: we format sequence direction classification (Fig. 1 (a)) as MCQ, using 1.1K training videos selected from UCF101 [58] (following prior use of this dataset [23]): (2) Open-ended QA: we curate high- temporality subset of LLaVA- Vide--idoo1111 178K [85], filtering based on the perplexity difference between forward and reverse video to retain samples with great temporal sensitivity, totaling 11.8K samples; (3) Video Captioning: we employ the training set [18] of RTime, which provides high-temporality videos alongside distinct human captions for their forward and reverse versions, comprising 11.7K samples. See Supp. B.3 for data examples and prompts used.
ArrowRL RL Implementation Our framework presents a straightforward and highly efficient approach by enabling direct application to pretrained LMMs without an intermediate supervised fine-tuning (SFT) step; we find ArrowRL alone is sufficient to instill AOT awareness. We demonstrate its broad applicability across three leading base LMMs: LLaVA-OV-7B [34], Qwen2- VL-7B [63], and Qwen2.5-VLL-7B [5]. For efficiency and memory, we limit the maximum number of video frames to 16 during training. Hyperparameter a is set as 0.25 and 7 is set as 0.75. The response group size G' is set as Training consists of 2000 RI. steps on NVIDIA GH200 GPUs
Evaluation Setup Our evaluation is designed to assess LMM temporal perception capabilities, using MCQ accuracy the metric across multiple benchmarks. This includes our proposed AoTBench (Sec. 3.3), as well as three existing video benchmarks emphasizing temporal properties, as identified in Sec. 3.1: TempCompass [44], TVBench [15], and Vinoground [80]. We evaluate ArrowRL by comparing enhanced models to their respective base LMMs. Additionally, to situate these results within the broader landscape, we also report a diverse suite of leading LMMs as baselines. These include general-purpost models Aria [35] and MiniCPM-V-2. V-2.6 [76], video-focused models
8

---

Table 2: Results on general video benchmarks (base vs. Arrow-RL enhanced LLaVA-OV-7B). ArrowRL's specialization on temporal awareness does not degrade performance on these general video understanding tasks.
Model
VideoMME (S)
NEXT-QA
TemporalBench (S)
VITATECS
Perception Test
LLaVA-OV-7B [34]
67.78
77.11
60.57
84.66
57.48
ArrowRL.
68.11
78.11
61.17
85.37
57.64
Table 3: Ablation results of ArrowRL. We report average performance across all columns of AoTBench tasks in Ta- ble 1. ArrowRL greatly outperforms the SFT baseline trained on the same data, demonstrating the effectiveness of our RL approach. In addition, using our curated high-temporality post-training data provides performance gain, validating our data selection strategy.
Aolianid Anc. 190
Arnowftl.
ArrowL
"5
65
60
60
55
0.0
1.0
0.25
0.50
0.75
1.50
vvallo
Violus
Figure 7: Hyperparameter analy-
Model
Training data
Acc. (%)
sis of ArrowRL on AoTBench, using Qwen2.5-VL-7B as the base LMM. We identify an optimal setting at a
Qwen2.5-VIL-7B [5]
56.2
+FT
ArrowRL. post-train
57.4
+ArrowRL
LLLVA--Vido- 178K captions
57.7
0.25 (reverse reward weight, left) and
ArrowRL
RTime captions
60.4
= 0.75 (dynamic weighting thresh- old, right), which effectively balances the fidelity and reverse reward.
ArrowRL
ArrowRL post-train set
61.4
ELLVA--ided [85], InternLM-XC-22.5 [82] (strong in fine-grained video understanding), and Video- RI [20] (emphasizing video reasoning), along with two major proprietary (and much larger and more extensively trained) models GPT-4o [1] and Gemini-1.5-Pro [60].
As shown in Table 1, ArrowRL effectively enhances models' temporal perception capabilities across diverse settings. Results on AoTBench (left table) reveal critical deficiencies in current LMM AoT perception. Visually-focused tasks like sequence direction classification prove especially challenging: nearly all open-source models exhibit universal failure mode: their responses are identical regardless of video input, resulting consistent chance-level performance (50.0% on ReverseFilm and 51.6% on UCF101). Leading proprietary models (GPT-4o, Gemini-1.5-Prro while slightly better, still perform far below satisfactory levels. This is in stark contrast to the proprictary models' absolute accuracy other mainstream VQA tasks, where accuracy hovers around 70-85% Our finding and the proposed AoTBench illuminate visual temporal bottlenecks. Importantly, this improved AoT awareness also translates gains existing, unfiltered temporal VQA benchmarks (right table), such as +10.8 group score (+65.9% relative gain) for Qwen2.5-VL. VL-7B on Vinoground. These results underscore the promise of enhancing AoT perception for overall temporal perception. See Supp. C.1 for a follow-up discussion of the relative strengths of different base models and how they translate ArroRL''s largest margins.
Main Results
The most important outcome is ArrowRL's improvements over each base model, and, following that, its advantage over the open-source baselines- all of which are similar in scale at 10B parameters or less. The significantly larger proprictary models trained on much larger datasets (gray rows) are a useful high-level reference, but do not constitute direct apples-to-apples comparison.
Results on General Video Benchmarks To ensure that specializing in AoT does not degrade general video understanding, we evaluate LenhweL-enhanced LLLVVAOV-7B five benchmarks previously identified as less temporally sensitive (see Fig. 3 right). As shown in Table 2, ArrowRL. RL. preserves or even slightly improves performance on these general benchmarks where AoT is not required, indicating it enhances temporal perception without sacrificing general video understanding.
Fig. presents our hyperparameter analysis, confirming two key design choices. First, the reverse reward is essential: the left plot shows that removing it (a = 0) degrades performance to below that of the base LMM. Second, the dynamic weighting mechanism is beneficial: the right plot shows that disabling it (7 1) yields less improvement than our dynamic approach, which can
Ablations

---

Q. What is the order of the letters on the table at the end? Qwen2.5-VL-7BB storm
LLAVA-OV-7B: hand unwrapping
LLAVA-OV-7B ArrowRL: hand removingalumindee aluminum foll to reveal...
Qwen2.5-VL-7B. ArrowRL: tmrso
Q. Which caption best describes this video?
LLAVA-OV-7B: video showcases close-up view of baked pastry, flaky and rich filling, and the of unwrapping it reveal its contents.
Qwen2-VL-7B: toddler plays around the grass field before he picks water bottle and drinks.
The video features close- up view a pastry, showcasing layers and filling, by hand interacting with foil wrapping.
LLAVA-OV-7B ArrowRL:
Qwen2-VL--2 + ArrowRL: a toddler picks up a water bottle and drinks before plays around the grass field.
Figure 8: Qualitative results comparing base LMMs vs. Arrow-RL enhanced models. Arrow RL correctly answers AoT-sensitive VQA questions (left) and generates distinct, temporally accurate captions (right), while the base model fails to capture temporal progression in these examples. More examples and failure cases in Supp. C.2.
zero out the reward for nos-sve-sen samples. We further investigate different inference frame settings and find ArrowRL generalizes robustly despite its fixed-framm training. See Supp. C.I for details.
Qualitative Results Fig. 8 provides qualitative examples, comparing base LMMs against our ArrowRI. enhancement. The VQA examples (left) demonstrate ArrowRL. enabling base models to correctly answer temporally challenging questions. Similarly, when prompted to describe forward and reverse sequences (right), the base model provides the "unwrapping" description for both, failing to connect frames dynamically to understand the temporal progression and possibly influenced by language bias (favoring the more common "unwrapping"). Conversely, ArrowRL demonstrates clear temporal awareness by producing distinct and correct captions reflecting the temporal direction. See Supp. C.2 for more qualitatives, including failure cases.
Limitations and Future Work While AoTBench grounds temporal directionality in language, we acknowledge that describing irreversible events remains inherently difficult natural language. Future work could explore how learned representations encode time beyond language [4], and probe whether models develop specialized activations for time-reversed, physically implausible inputs [16]. For ArrowRI, important future directions include extending our methods to long videos (e.g., keyframe selection) and exploring integration with temporal reasoning tasks that necessitate Chain-of- Thinnoh-Thoouht reasoning. See Supp. D for a comprehensive discussion of limitations.
5
Conclusion
Our work underscores the crucial yet unexplored role of AoT understanding within LMMs. Towards this end, we introduce ArrowRL, an GRPO-based reinforcement learning algorithm with novel reverse reward design to enhance temporal perception, and develop AoTBench, a comprehensive benchmark targeting temporal sensitivity. Great performance gains on both AoTBench and existing VQA benchmarks validate our approach and highlight the crucial role of AoT in advancing temporal understanding. We hope ArrowRL offers a promising path towards LMMs with true temporal understanding capabilities, and that AoTBench serves valuable tool for future research.
Acknowledgements This research was supported in part by the UT Austin IFML NSF AI Institute, the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC). We would like to thank the anonymous reviewers for their thoughtful and constructive feedback, which provided valuable guidance in refining this work.
with compute on the Vista GPU Cluster through
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. Cosmos-reasoni From physical common sense to embodied reasoning. arXiv preprint arXiv:2503. 2025.
[3] Piyush Bagad, Makarand Tapaswi, and Cees GM Snoek. Test of time: Instilling video-language models with sense of time. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2503-2516, 2023.
[4] Piyush Bagad and Andrew Zisserman. Chirality in action: Time-aware video representation learning by latent straightening. arXiv preprint arXiv:2509.08502, 2025.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:250.113923, 2025.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901 2020.
[7] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in languge understanding. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2917-2927, 2022.. 2022.
[8] Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: survey. arXiv preprint arXiv:2402. 12451, 2024.
[9] Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, al. Temporalhench: Benchmarking fine-graines temporal understanding for multimodal video models. arviv preprint arXiv:2410. J0818, 2024.
[10] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 2011.
[11] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- vision-language models? arXiv preprint 20X 2024.
[12] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37: 37:19472- 19495, 2024.
[13] Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, and Xihui Liu, Exploring the effect of reinforcement learning on video understanding: Insights from bed-benchh-rl. arXiv preprint arXiv:2503.24376 2025.
[14] Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song. Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open- access data and models for detailed visual understanding. arXiv preprint arXiv:2504. 13180, 2025.
[15] Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki Asano. Tvbench: Redesigning video-language evaluation. arXiv preprint arXiv:2410.0775. 2024.

---

[16] Gustavo Deco, Yonatan Sanz Perl, Laura de la Fuente, Jacobo D Sitt, BT Thomas Yeo, Enzo Tagliazucchi, and Morten Kringelbach. The arrow of time of brain signals in cognition: Potential intriguing role of parts of the default mode network. Network Neuroscience, 7(3):666- 998, 2023.
[17] Michael Dorkenwald, Fanyi Xiao, Biagio Brattoli, Joseph Tighe, and Davide Modolo. Sevrl: Shuffled contrastive video representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4132-4141, 2022.
[18] Yang Du, Yuqi Liu, and Qin Jin. Reversed in time: A novel temporal-emphasized benchmark for cross-modal video-text retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages
2024.
[19] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cyclicconnistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1801-1810, 2019.
[20] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang. Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning milms. arXiv preprini arXiv:2503.217777 2025.
[21] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint 2055::2405.21075. 2024.
[22] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision- language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4): 1):16333122 2022.
[23] Amir Ghodrati, Efstratios Gavves, and Cees GM Snoek. Video time: Properties, encoders and evaluation. arXiv preprint TeXXiv:807.00900 2018.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint 20Xiv:2407:21783, 2024.
[25] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
[26] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 2024.
[27] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018.
[28] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 2023.
[29] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[30] Longlong Jing, Xiaodong Yang. Jingen Liu, and Yingli Tian. Self-supervised spatiotemporal feature learning via video rotation prediction. arXiv preprint arxiv:1811.1138) 2018.
[31] David Layzer. The arrow of time. Scientific American, 233(6):56-69, 1975.
[32] TisinYYing Lee, Jia-Bin Huang, Maneesh Singh, and Hing-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. Proceedings of the IEEE international conference on computer vision, pages 667-676, 2017.
12

---

[33] Jie Lei, Tamara L Berg, and Mohit Bansal, Revealing single frame bias for video-and aarto--nd--aagggan language learning. XrXiv preprint arXiv:2206.05422, 2022.
[34] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 03336,, 2024.
[35] Dongxu Li. Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, et al. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024.
[36] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195 22200,, 2024.
[37] Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. In European Conference on Computer Vision, pages 331-348. Springer, 2024.
[38] Yun Li, Zhe Liu, Yajing Kong, Guangrui Li, Jiyuan Zhang, Chao Bian, Feng Liu, Lina Yao, and Zhenbang Sun. Exploring the role of explicit temporal modeling in multimodal large language models for video understanding. arXiv preprint arXiv:2501 1578666601..6866 2025.
[39] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405-409, 2024.
[40] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint ar arXiv:2109.07958 2021.
[41] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[43] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[44] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.00476, 2024.
[45] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. When thinking drifts: Evidential grounding for robust video reasoning. arXiv preprint 2rXiv225 2025.
[46] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances Neural Information Processing Systems, 36:46212 11442222-6244, 2023.
[47] Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. arXiv preprint arXiv:2308. /0901, 2023.
[48] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 527-544. Springer, 2016.
[49] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
13

---

[50] Vassilis Papadopoulos, Jérémie Wenger, and Clément Hongler. Arrows of time for large language models. arXiv preprint arXiv:2401.17500 2024.
[51] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36447744466,4 2023.
[52] Lyndsey C Pickup, Zheng Pan, Donglai Wei, YiChang Shih, Changshui Zhang, Andrew Zisser- man, Bernhard Scholkopf, and William T Freeman. Seeing the arrow of time. In Pa Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2035-2043 2014.
[53] Will Price and Dima Damen. Retro-actions: Learning's close' time-reversing open' videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages (-0, 2019.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning,
pages 8748-8766 8763. PmLR, 2021.
[55] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53777 2023.
[56] Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 535-544, 2021.
[57] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033300 30000000000000 2024.
[58] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint /2Xiv:/212.0444 2012.
[59] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
[60] Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
[61] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37:87310-88 17-5666, 2024.
[62] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 2024.
[63] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lasion-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[64] Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406. 16338, 2024.
[65] Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching action knowledge in video-language foundation models. Advances in Neural Information Processing Systems, 36:20729-2001 2023.
14

---

[66] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8052-8060, 2018
[67] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 45:24824-24837, 2022.
[68] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828-28888 28857, 2024.
[69] Junbin Xiao, Nanxin Huang, Hangyu Qin, Dongyang Li, Yicong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, al. Videoqa in the of Ilms: An empirical study. International Journal of Computer Vision, pages 1-24, 2025.
[70] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 9777-9786, 2021.
[71] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprini arXiv:2309.1 1489, 2023.
[72] Hu Xu, Gargi Ghosh, Po- Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109. 140844 2021.
[73] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
[74] Zihui Xue, Joungbin An, Xitong Yang, and Kristen Grauman. Progress-aware video frame captioning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13639- 13650, 2025.
[75] Zihui Sherry Xue and Kristen Grauman. Learning fine-grained view-invariant representa- tions from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36:53688- 044:5388--3700, 2023.
[76] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.00 20040000.0000 2024.
[77] En Yu. Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Jingyu Wang, et al. Unhackable temporal rewarding for scalable video mllms. arXiv preprint arXiv:2502. /2081,
[78] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807-13816, 2024.
[79] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Na Zhao, Zhiyu Tan, Hao Li, and Jingjing Chen. Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint 2409.1697, 2024.
[80] Jianrui Zhang, Mu Cai, and Yong Jae Lee. Vinoground: Scrutinizing Imms over dense temporal reasoning with short videos. arXiv preprint arXi::2410.02763, 2024.
[81] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint 2503. 2025.
15

---

[82] Pan Zhang, Xiaoyi Dong, Yuhang Zang. Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang. Linke Ouyang, al. Internlm-xcome seeemlm-xoompssrr2... A versatile large vision language model supporting long-contextual input and output. arXiv preprint urXiv:2407.03320, 2024.
[83] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint urXi:2240.001258, 2024.
[84] Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang. Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprini arXiv:2410.16. 16/98, 2024.
[85] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.0277 2024.
[86] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
[87] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing llnguaal understanding with advanced large language models. arXiv preprins arXiv:2304.100 10592, 2023.
[88] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: SEKiv:1900.08999 0999908899, 2019.
[89] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Fstruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding large multimodal models. arXiv preprint a41iv:2412. 10360, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly describe our main claims such as the findings, contributions, and results in the abstract and introduction. They match our idea, methodology, and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a detailed discussion of limitations in the Supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Please Sec. 4 and Supplementary for full implementation details to repro- duce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are committed to reproducibility and will open-source the code, data and model upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
ceaaiiissssoooiiiiiiiiiooioooioi oeeiionooooooooo oolicy)) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ wccppbllc///aeeeccceeeceecceelee http//uudee//eeeee CCtpss//ooSSubimeecc cessonnoolcyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please Sec. 4 and Supplementary for full experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see Fig. 3 for error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Sec. and Supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /wwurrrrrs..../ cc hwwppaaic/ttiistciii.ieeoooopeee
Answer: [Yes]
Justification:
We strictly conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Supp. for societal impact discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We propose training algorithm (ArrowRL) and an evaluation benchmark (AoTBench) derived from standard academic datasets. The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please Sec. 4 and Supplementary.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http://ies...//eeeeeeeeee httpe//ris..e/eeoeeeeeeeeeeeeeee c for what should or should be described.
23

---

Contents
Supplementary Video
24
Method Details
24
B.1 Control Task to Disentangle OOD Effects
24
B.2 Temporal Sensitivity Analysis
25
B.3 ArrowRL
25
B.4 AoTBench
27
C Additional Results
27
C.I Quantitative Results
27
C.2 Qualitative Results
29
Limitations
29
Societal Impacts
31
A
Supplementary Video
We invite readers to view the supplementary video available at httts: projects/SeeAoT for visual demonstration of our work's overview and additional qualitative examples. Understanding how video content differs between forward and reverse playback-a key aspect our study-is is most effectively conveyed through video. Therefore, the supplementary video will offer readers more intuitive grasp of how ArrowRL successfully enhances temporal sensitivity in LMM responses and how AoTBench effectively probes this crucial capability.
utexas edu/
B
Method Details
B.1
Control Task Disentangle OOD Effects
To verify that LMMs' poor performance AoT tasks (observed in Fig. 1 of the main paper) is not simply due to reversed videos being treated as out-of-distribution (OOD) inputs, we design control experiment inspired by [3]. Using videos and action labels from UCF101 [58], formulate a binary-choice question with the prompt, "What action is being performed in this video?" The correct action provided one choice, and the incorrect option is randomly drawn from the of all other action labels. We then evaluate several LMMs on both the original (forward) and time-reveent reverreerssed versions of these videos.
Table We conduct binary action classification on UCF101 as a control task, to disentangle AoT insensitivity from out-of-distribution (OOD) effects. All LMMs achieve near-perfect accuracy regardless video playback direction, suggesting that failures on AoT tasks are not due to general inability to process reversed videos.
Binary Acc. (%)
Model
forward
reverse
GPT-4o [1]
100.0
100.0
Gemini-1.5-Prr [60]
98.2
99.4
LLaVA-Ov-7B [34]
99.2
99.4
Qwen2.5-VL-7B [5]
99.4
99.4
24

---

The results in Table 4 show that all models perform at near pear-ctrfect accuracy on this control task, irrespective of video direction. This contrasts sharply with their near-random performance on our AoT--sensiivv sequence direction classification task (Table 1, UCF column). This analysis provides strong evidence that the failure of existing LMMs on AoT tasks is due their inability to perceive temporal directionality, not a general failure to process reversed videos as OOD inputs.
B.2 Temporal Sensitivity Analysis
The temporal sensitivity analysis, visualized in Fig. 2 of the main paper, is conducted using LLaVA-OV-7B [34] across several evaluation benchmarks: EgoSchema [46], Long VideoBench [68], MVBench [36], NEXT-QA [70]. PerceptionTest [14], TemporalBench [9], VITATECS [37], and VideoMME [21]. To ensure consistent and fair evaluation, we utilize Imms-eval" and a standardized setting of 16 imput frames across three conditions: forward, reversed and shuffled video. We specifi- cally select deterministic MCQ tasks to obviate the need for potentially unreliable third-party LLM
evaluators and mitigate evaluation ambiguity.
Our subsequent analyses, including the development of AoTBench, focus on short videos. This approach allows for a targeted investigation of AoT awareness, separating it from complexities specific to long video processing, especially since current LMMs already struggle with shorter temporal sequences. Table 5 presents the forward, reversed, and shuffled frame performance (i.e., MCQ accuracy) for three selected LMMs (LLaVA-OV-7B [34], --V--oo----B [85], and Qwen2.5- VL-7B [5]) on several VQA benchmarks: VITATECS [37]. TemporalBench [9], NEXT-QA [70], PerceptionTest [51], VideoMME [21], Vinoground [80], TempCompass [44] and TVBench [15]. These results, which supplement the TDS-based benchmark sensitivity analysis in Fig. 3 of the main paper, further illustrates the great variance in temporal order sensitivity across benchmarks: some, such as Vinoground, TempCompass, and TVBench, demonstrate stronger ability probe this, whereas others, like VITATECS, show extreme insensitivity.
Table 5: Impact of video frame order manipulation (forward, reversed, shuffled) on MCQ Accuracy (%) for three LMMs across various VQA benchmarks. S: short, V: video, T: text.
LLaVA-OV-7B
LLaVA-Video- Vidco-7B
Qwen2.5-VL. VL--BB
Benchmark
forward
reverse shuffled forward reverse shuffled
forward
reverse shuffled
VITATECS
85.27
83.00
84.98
87.74
85.87
87.28
82.95
81.33
81.54
TemporalBench (S)
61.92
57.53
59.04
62.85
58.43
59.57
68.36
66.24
67.23
NeXT-QA
78.29
77.31
77.85
81.97
80.27
80.80
81.47
79.80
79.80
Perception Test
57.15
55.72
56.52
67.63
65.07
65.57
68.81
64.86
65.89
VideoMME (S)
70.89
69.22
68.89
76.33
72.89
73.67
72.33
71.00
69.56
Vinoground (V)
58.00
56.90
52.30
56.90
54.20
49.90
56.00
53.40
48.60
Vinoground (T)
68.20
35.80
53.80
65,90
36.80
51.80
61.10
38.70
49.50
TempCompass
69.78
53.34
62.16
72.09
$5.29
63.24
73.88
$7.15
65.21
TVBench
49.11
33.86
40.51
53.19
34.73
42.77
54.69
33.66
42.34
B.3 ArrowRL
Post-training Data To enhance AoT understanding, our training data incorporates three core tasks. For MCQ-based sequence direction classification, we use selected videos from UCF101 [58]. For video captioning, we leverage the RTime dataset [18] with a varied set of 16 prompts. For open-ended QA, we employ original questions from -.LAV--eeT--17 178K [85]. The prompts used for MCQ and captioning tasks are detailed below. For the MCQ- based sequence direction classification task, we follow [80] to concatenate the forward video and its reversed version separated by a 2-second black frame- into a single video input, as LMMs typically process one video stream a time.
htttpp::/githuu..com///// cwt.:/o..coo/o/ee/eee Lab haabs//mmm-aam eval
25

---

Training Prompt (MCQ-based Sequence Direction Classification)
This video contains two segments showing the same action, separated by 2-second black frame. One segment is played forwards, and the other is played in reverse. Which video segment is played in reverse?
A. The first segment (before the black frame)
B. The second segment (after the black frame)
Answer with the option's 's letter from the given choices directly.
Training Prompt II (Video Captioning)
Describe the following video in detail.
Provide detailed description of the given video.
Give an elaborate explanation of the video you see.
Share comprehensive rundown of the presented video.
Offer thorough analysis of the video.
Explain the various aspects of the video before you. Clarify the contents of the displayed video with great detail.
Characterize the video using a well-detailed description.
Break down the elements of the video in detailed manner.
Walk through the important details of the video.
Portray the video with a rich, descriptive narrative.
Narrate the contents of the video with precision.
Analyze the video in comprehensive and detailed manner.
Illustrate the video through descriptive explanation.
Examine the video closely and share its details.
Write an exhaustive depiction of the given video.
Q Why does the person into the closet?
[Candidate response o,] The steps into the closet likely to fetch item package. This supported by the fact that the
[Target response o'] The person steps the closet appearing search for something, specifically retrieve brown cardboard box.
looking around inside the box, which suggests was the sequence what they were searching for.
xf - 0.7
= 1.0
inside
presence
fd 0.85 -.7
[Candidate response 0z] The person steps into closet to retrieve box from the shelf.
[Reverse response 0] The person steps into the closet to place the box inside. This suggests they are organizing or storing the box, possibly to keep out of the way or to make spece in the room.
[Candidate response ox] The person steps
likelly
to retrieve the item they are carrying, which appears to
fid 0.7
cardboard box. seems that they are either placing the box in the closet, retrieving something from it, or organizing items. The action suggests a purposeful movement to fetch or store the box.
sim
Figure 9: An illustration of the reward calculation process for ArrowRL, using one VQA example from LLL responses. While the fidelity reward ensures candidate responses align with the target o', the reverse reward r, rev cultivates AoT sensitivity by using dissimilarity from the reverse response as its signal. Consequently, temporally correct and sensitive responses that diverge from ò (like o1) favored, while those that misunderstand AoT (like 0g, highlighted red) penalized via lower reward.
[85]. An auxiliary LLM is employed compute similarity scores between
Reward Calculation Fig. 9 demonstrates ArrowRL's reward calculation using VQA example that necessitates causal-temporal reasoning. Here, the fidelity reward (rfi)) alone might be insufficient;
26

---

for instance, candidates and 03 both exhibit high similarity to the target o". The reverse reward distinguishes them by leveraging the semantic difference between forward and reverse video plays. response ò to the reversed video (e.g., describing "organizing or storing the box") is used as negative reference. The reverse reward, rt = like if they incorrectly align with the reverse response ő (as indicated by red highlighting), thereby favoring temporally aware responses like 0₃ that accurately reflect the forward video's AoT.
Similarity(o;; o), then penalizes forward-video responses
Implementation As discussed Sec. 3.2 of the main paper, Similarity(::) return a similarity score between 0 and I and is implemented follows. For MCQ tasks, it uses deterministic value checking (1.0 for correct match, 0.0 otherwise); For open-ended QA and captioning, we employ 1--aa-..--0B--nmee
L.M judge, which is prompted to output semantic similarity score within the [o, I] range. using the prompts below. The previously defined language query 4, target response o", candidate response of and reverse response ō are referenced here.
[24]
LLM-based Similarity Calculation
1Open-ended QA prompt]
Please compare the following two answers for the question below and rate their similarity on scale of 0 to 1.
Question: [
Answer 1: 0;
Answer 2: ō
Output only single numeric value between 0 and 1 (no additional text explanation).
[Captioning Prompt]
Compare the following video caption with the ground truth caption and rate their similarity scale to
Generated caption: 0;
Ground truth caption: o*
Output only single numeric value between 0 and 1 (no additional text explanation).
During training, candidate responses o, are generated with a temperature 1.0 to encourage ex- ploration. The reverse-conditione response is generated deterministically (temperature set 0), ArrowRL training for each model involves 2000 RL steps over approximately 3 days on NVIDIA GH200 GPUs
To ensure a fair comparison, inference settings for base LMMs and their ArrowRLeenhanced are identical, differing only by model checkpoint. Our default input configuration is 16 frames for ILLVAAOV-7B, and FPS (with a maximum of 16 frames) for Qwen2-VLL-7B and Qwen2.5-VIL-7B. Benchmark-speciffe adjustments include processing up to 32 frames (sampled at FPS for Qwen models) for TVBench (due video length) and reporting Vinoground at 4FPS for Qwen models to align with [80] (further frame rate analysis in Fig. 11). For TempCompass, adhering to [82], we report only on its deterministic subtasks (multi-choice QA, yes/no QA, caption matching; 5536 samples).
B.4 AoTBench
To evaluate AoT perception in LMMs, we construct AoTBench. The dataset composition is detailed in Table with illustrative examples in Fig. 10. For the first and second T2V task, concatenate the forward, a two-second black video and reversed video segments into a single input, and use the same prompt as Training Prompt above, but with shuffled MCQ options test model generalization.
Furthermore, Table 7 quantifies the increased AoT sensitivity of our selected VQA subset, showing significantly higher TDS values compared to the original benchmark.
C
Additional Results
C.1
Quantitative Results
Further Results Analysis From Table the main paper, we see that A to leverage inherent base model strengths for improvements. For instance, the visually adept DWB72-VLL7B
27

---

Table 6: AoTBench Dataset Breakdown. The benchmark comprises three distinct tasks, derived from a diverse suite of video sources, designed to assess AoT awareness of LMMs.
Task
Vidco Source
# VQA
ReverseFilm [66]
144
Sequence Direction Classification
UCFI01 [58]
500
Directional Caption Matching (V2T)
RTime [18]
1,992
Directional Caption Matching (T2V)
RTime [18]
1,992
VITATECS [37], TemporalBench [9], NExT-QA [70],
AoT-sensitive VQA
PerceptionTest [51], VideoMME [21], Vinoground [80],
1,800
TempCompass [44]. TVBench [15]
L
AoTBench
[Sequence Directionality Classification] video segments presented: plays Which video segment is played
[Directional Caption Matching T2V] video segments are presented: one plays forward, the other reverse. Which video segment matches the caption "sameone taking personal care items out of a clear plastic bag"?
[Directional Caption Matching - V2T] Which caption best describes the video? A bulb is installed to the white round lamp shade by person and lighted A person removes the luminous bulb from the white round lamp shade with his hands.
[AoT-sensitive VQA] change occurring the house model?
being constructed
being dismantled being renovated
Visual overview of the three core task components in AoTBench, designed to evaluate different facets of AoT understanding in LMMs.
Figure 10:
(evidenced by its base performance on T2V caption matching) sees +21.0% gain with ArrowRL on the visually-focused sequence direction classification task. Meanwhile, Owen2.5-VL-7B, which exhibits greater language proficiency (reflected in its base V2T caption matching scores), achieves its largest gains (49.2%) on the more language-centric AoT-VQA task after ArrowRL enhancement.
Frame Rate Analysis Fig. 11 presents our inference frame analysis on Vinoground, comparing ArrowRL-enhanced Qwen2.5-VL-77 against its base model across different frame settings (1-4 FPS). Crucially, although ArrowRL training utilizes a fixed frames per input (for efficiency), the performance gains provided by ArrowRI remain consistent across varying temporal granularities at inference, showcasing its generalization beyond the specific training setup, and giving evidence that the base models overlook temporal detail even with higher framerates.
28

---

Table 7: Comparing Temporal Divergence Score (TDS) averaged for all samples vs. top 200 selected TDgh-TDS& samples across nine existing VQA benchmarks. The selection process yields 1,800- sample subset with substantially increased average TDS, specifically designed to challenge LMM temporal perception.
TemporalBench (S)
PerceptionTest
VidcoMME (S)
Coterouudd (V)
Vinoground (T)
TtempCompar
VITATECS
NEXT-QA
T'VBench
All
0.039
0.062
0.073
0.083
0.113
0.212
0.408
0.461
0.549
Selected
0.738
1.258
1.757
3.185
0.690
0.512
1.287
4.617
3.504
80
Qwen2.5-VL-1
60
+ALrowRL
Score
40
38.4
36.4
40.843.
38.8
42.0
33.0
42.6
32.5
26.0
23.2
27.2
20
15.0
8.4
Video
Text
Group
Video
Text
Group
Video
Text
Group
Video
Text
Group
1FPS
2FPS
3FPS
4FPS
Figure 11: Impact inference frame rate analysis on Vinoground. The ArrowRL's performance gains remain consistent across different frame settings (1-4FPS), showcasing its generalizability.
C.2 Qualitative Results
the main paper, Fig. 12 presents additional qualitative examples that reinforce the effectiveness of ArrowLL. These comparisons highlight how base LMMs often overlook temporal progression or direction, often relying on static cues or language biases. Conversely, ArrhweLeenhanced models exhibit improved AoT sensitivity, leading to more accurate and temporally coherent responses across these challenging scenarios.
More Qualitative Results Supplementing Fig.
Failure Cases Fig. 13 presents one failure case on AoTBench, where neither the base LMM nor the AnowRL-enhanced version answers correctly. The failure stems from the uniform sampling of 16 frames (6 visualized here) not capturing the key visual moments where man a dark suit holds ring-a crucial cue for inferring he is about to get married. Such failures suggest potential avenues for future improvement, like advanced keyframe selection methods over uniform sampling, or incorporation of auxiliary modalities such as audio, which in this case contains helpful cues.
D Limitations
Our construction of AoTBench relies using selected LMMs evaluators, meaning its sensitivity inherently dependent on the initial AoT perception capabilities of these models. Potentially chal- lenging samples might be missed if current evaluator models universally fail to exhibit sensitivity (i.e., yield low TDS) despite underlying temporal relevance. Nevertheless, to provide quantitative check of AoTBench, manually verify small subset and find our TDS- TDS-based selection of tempo- rally challenging examples aligns with human judgment 44 out of 50 times, suggesting reasonable concordance where models do possess baseline sensitivity.
Additionally, the reward calculation for ArrowRI. utilizes an auxiliary LLM to generate similarity scores; this dependence LLM judge, while common [86, 25, 71], can introduce degree of

---

Q. Why does the dog walk toward the lady after the lady reaches her hand toward it?
Qwen2-VL-7B: to catch the food
Qwen2-VL-7B ArrowRL: the lady hug it
Q. Which caption best describes this video?
Qwen2.5-VL B----BB- woman speaking while she handling an air filter with both hands. Using her right hand to hold place filter device, left hand pulls the filter the outer shield of the filter from the table.
place the device. hands pick up
(wen2.5-VL-77 ArrowRL: A woman is speaking while she handling air filter with both hands. Using her right hand to hold in place the filter device, the left hand pushes the filter into its place in the device. Both hands then pick up the outer shield of the filter from the table.
Q. Provide one-sentence description of the video, focusing solely on the primary actions taking place. LLaVA-OV-7B: video captures serene and dynamic movement of clouds against tranquil sky, creating peaceful and picturesque
LLaVA-OV-7B ArrowRL: The video features series of images capturing the movement of clouds the sky, with the clouds transitioning from lighter to darker shades as the light diminishes.
Figure 12: Additional qualitative examples comparing base LMMs with their ArrowRL-enhanced counterparts. ArrowRL enables models succeed AoT-sensitive VQA and produce temporally coherent captions, while base LMMs often struggle with understanding temporal progression.
According to the video, who is about get married?
A. The man in the dark suit.
Qwen2.5-VL-7B: It is unclear
Qwen2.5-VL-7B + ArrowRL (ours): is unclear.
Figure 13: A failure case from AoTBench. Uniformly sampling 16 frames (6 visualized) fails to capture the critical visual moments (i.c., the man in the dark suit holding wedding ring).
uncertainty or potential bias into the reward signal. Nevertheless, we found this approach viable for our purposes, as the LLM performs a relatively easy, straightforward text-to-text similarity assessment. We empirically find that (also illustrated in Fig. 6 of the main paper and Fig. 9) the resulting similarity scores, which underpin our reward calculation, are consistent and appear reasonable.
Furthermore, our current investigation focuses on short videos, scope chosen due to the significant temporal challenges already evident these scenarios for leading LMMs. We view this work as initial step towards improving AoT understanding. Future directions include extending these methods long videos (potentially incorporating keyframe selection) and exploring integration with temporal reasoning tasks that necessitate explicit reasoning traces (c.g. Chain-of- Chain-of-Thought)
30

---

E
Societal Impacts
This work focuses on improving fundamental aspect temporal perception (AoT sensitivity) in LMMs. Positive impacts stem from creating more reliable and rational AI systems. Better AoT perception can lead LMMs with more accurate internal world models, improving their utility in tasks requiring understanding of processes, procedures, or event timelines. This could benefit applications like education, assistive technology and robotics. The primary risks are associated with the general capabilities of advanced LMMs rather than AoT sensitivity specifically. Any improvement could potentially be misused if integrated into systems for generating disinformation invasive surveillance, though method doesn't directly enable these.
31


Reviews:
[{"review_id": "ZntAAygwAS", "reviewer": "Reviewer_tJ9G", "summary": "This paper focuses on enhancing the temporal directionality (Arrow of Time) perception of MLMMs. By analyzing the limitations of existing benchmarks and models, the authors find that even SOTA MLMMs exhibit a clear insensitivity to temporal order, they often fail to distinguish between forward and reversed video playback, and may even generate identical descriptions for semantically opposite videos. To address this issue, the authors propose a GRPO-based reinforcement fine-tuning method that encourages the model to produce different interpretations for forward and reversed video frames, thereby cultivating AoT sensitivity. Additionally, they design a comprehensive benchmark to evaluate models’ AoT perception capabilities and introduce the Temporal Divergence Score (TDS) to quantify temporal sensitivity at both the benchmark and sample levels.", "questions": "1.\tForcing the model to learn from both forward and reversed videos may lead to the generation of physically implausible descriptions. It is recommended to include an additional set of experiments to analyze and illustrate this potential issue.\n2.\tIt would be helpful to clarify in the paper how the questions in AoTBench were constructed, whether they were directly extracted from existing datasets or re-annotated for forward and reversed videos.\n3.\tIf possible, providing a more detailed analysis of how different values of 𝛼 and  𝛾 affect training performance would make the paper more comprehensive and informative.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\t\t\n1.\tThe reverse reward mechanism in ArrowRL is an innovative design that effectively leverages the natural contrastive signal of video temporal directionality. By employing reinforcement learning, it enhances the model’s sensitivity to the Arrow of Time.\n2.\tIn addition to proposing a novel method, the paper also develops a dedicated benchmark (AoTBench) and conducts a systematic analysis of several existing benchmarks, providing extensive experimental data.\n\nWeaknesses:\n1.\tThe Reverse Reward mechanism may be overly simplistic and brute-force in nature. By forcing the model to distinguish reversed videos, it risks encouraging learning of superficial temporal cues rather than a true understanding of event dynamics. For instance, it may fail to differentiate between physically irreversible actions (e.g., glass shattering) and reversible ones (e.g., opening/closing a door). Specifically, while the method is practical and effective for reversible actions, it might lead the model to generate physically implausible descriptions for irreversible actions, as the reward function does not explicitly model physical constraints. Moreover, for cyclic or repetitive events, the Reverse Reward could have adverse effects.\n2.\tWere the questions in AoTBench directly extracted from existing benchmarks, or were the forward and reversed versions manually re-annotated based on videos from those datasets? If it’s the former, how is temporal sensitivity ensured? Wouldn’t that essentially mean the new benchmark is just a collection of previously difficult samples, rather than a dedicated test of temporal directionality?\n3.\tThe paper only discusses the impact of extreme values of 𝛼 and 𝛾 (i.e., 0 and 1) on training performance, but lacks analysis of intermediate values (e.g., 0.5), which could provide more insight into the trade-offs and sensitivity of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer tJ9G for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n**1. Reverse Reward Mechanism**\n\n> Forcing the model to learn from both forward and reversed videos may lead to the generation of physically implausible descriptions. It is recommended to include an additional set of experiments to analyze and illustrate this potential issue.\n\nWe agree that applying a reverse reward indiscriminately would be problematic, and emphasize that ArrowRL was designed specifically with this nuance in mind. As noted in *Ln 207-214*, ArrowRL incorporates two key mechanisms—high-temporality data curation and dynamic weighting—to ensure that the model learns a true understanding of event dynamics rather than superficial cues.\n\n**High-temporality Data Curation**. First, at the data level, a portion of our training data is sourced from RTime [14], a benchmark of high-temporality videos where the reversed version is both physically plausible and semantically distinct from the forward version (*Ln 251-253*). By training on these vetted (manually verified to demonstrate high temporality) videos, ArrowRL learns the core patterns of reversible, meaningful actions without being confused by inapplicable cases.\nAs suggested, we ran an additional experiment to isolate the effect of our data selection strategy. We train a model only for the captioning task, where the high-quality RTime data was replaced by uncurated videos from LLaVA-Video-178K [73]. As the table shows, the model trained on curated, high-temporality data performs better, underscoring the importance of our data selection strategy, and (we think) consistent with the reviewer’s expectation.\n\n| Model                          | AoTBench Acc. (%) |\n|-----------------------------|:-----------------:|\n| Base LMM                      |       56.2        |\n| w/ ArrowRL (RTime Cap.)            |       **60.4**        |\n| w/ ArrowRL (LLaVA-Video-178K Cap.) |       57.7        |\n\n**Dynamic reward weighting**. Next, to ensure scalability beyond curated data like RTime, we also train on the large, uncurated LLaVA-Video-178K dataset for open-ended QA (*Ln 248-250*), where irreversible events are likely to appear. Our dynamic reward weighting is designed to handle precisely these cases. Consider the glass shattering example the reviewer mentioned: since the reversed action is physically impossible, a model will naturally struggle to describe it. In fact, as we see across many examples, LMMs tend to default to descriptions that are semantically very close to the forward ground truth in such cases. Our mechanism detects this by calculating a high Similarity$(\\tilde o, o^*)$ that surpasses a threshold 𝛾. Therefore, if the video is not meaningfully reversible, ArrowRL will \"turn off\" the reverse reward for that specific example by setting its weight to zero. This ensures our model is not forced into the task of creating a plausible description for an implausible event. The importance of our dynamic weighting is already ablated in *Table 2* (row +ArrowRL (𝛾=1)), which shows that removing this dynamic weighting hurts performance.\n\nWe believe this two-part strategy ensures that ArrowRL is a nuanced and targeted tool, not a brute-force mechanism, and will clarify this further in the final version of the paper.\n\n---\n\n**2. Clarification of AoTBench Construction**\n\n> It would be helpful to clarify in the paper how the questions in AoTBench were constructed, whether they were directly extracted from existing datasets or re-annotated for forward and reversed videos.\n\nWe designed AoTBench to be a dedicated test of LMM temporal sensitivity, rather than a generic collection of difficult samples. As detailed in *Sec. 3.3* and *Supp. Sec. B.3*, AoTBench is a multi-faceted benchmark where each component is designed to probe this specific capability. \n\nThe first two tasks, sequence direction classification & directional caption matching, are inherently temporal. For these, we use videos from existing datasets and design questions that require the model to classify a video's playback direction or match it to a corresponding forward/reverse caption (*Table 4* & *Fig. 9*). These tasks cannot be solved without understanding temporal directionality.\n\nFor the third task, AoT-sensitive VQA, we compose data samples from 8 existing video benchmarks. Rather than pulling generically \"hard\" questions from existing benchmarks, we develop a more principled approach. We introduce the Temporal Divergence Score (TDS), a metric that precisely quantifies how critical the video's AoT is to arriving at the correct answer (*Sec. 3.1*). TDS allows us to analyze samples from different benchmarks and select only those that require true temporal perception. This principled filtering guarantees our VQA subset is a true test of temporal perception, not just a collection of unrelated hard cases, as illustrated by the examples in *Fig. 6 & 8*.  \n\n---\n\n**3. Different values of α & 𝛾**\n\n> If possible, providing a more detailed analysis of how different values of 𝛼 and 𝛾 affect training performance would make the paper more comprehensive and informative.\n\nWe appreciate the suggestion and have conducted additional experiments on the hyperparameters 𝛼 (reverse reward weight) and 𝛾 (dynamic weighting threshold), building upon the ablations already in *Table 2*. \n\n**Analysis of Reverse Reward Weight (𝛼)**. Our main paper reports results for our chosen value of 𝛼 = 0.5 and includes an ablation of 𝛼 = 0 (disabling the reverse reward). As requested, we have now evaluated a fuller range of values. The results below show the performance across different weights.\n \n| Setting        | AoTBench Acc. (%) |\n|----------------|:-------------------:|\n| Base LMM       | 56.2              |\n| 𝛼 = 0          | 55.6              |\n| 𝛼 = 0.25       | **61.3**              |\n| 𝛼 = 0.5        | 60.7              |\n| 𝛼 = 0.75       | 61.1              |\n| 𝛼 = 1.0        | 54.0              |\n\nThese results confirm that the reverse reward is a critical component of our method. Performance is poor at the extremes (𝛼 = 0 and 𝛼 = 1.0), but strong and stable across the range of [0.25, 0.75]. This detailed analysis identifies 𝛼 = 0.25 as the optimal weight, and we will update our model and paper accordingly.\n\n**Analysis of Dynamic Weighting Threshold (𝛾)**. Our main paper reports results for our chosen value of 𝛾 = 0.5 and ablates the case with no thresholding (𝛾 = 1). We have now tested additional intermediate values as suggested. Note that 𝛾 = 0 is an invalid setting as it would filter out all training data. The extended analysis is below.\n\n| Setting        | AoTBench Acc. (%) |\n|----------------|:-------------------:|\n| Base LMM       | 56.2              |\n| 𝛾 = 1       | 58.3             |\n| 𝛾 = 0.75        | 61.1              |\n| 𝛾 = 0.5       | 60.7              |\n| 𝛾 = 0.25        | **61.3**              |\n\nThis analysis confirms our dynamic weighting is a critical component, as removing it (𝛾 = 1) greatly degrades performance. The method is also robust, with stable and strong results across the [0.25, 0.75] range. This detailed analysis identifies 𝛾 = 0.25 as the optimal weight. We thank the reviewer for this valuable suggestion and will update our model and paper accordingly."}, {"review_id": "VhBU4jfjFg", "reviewer": "Reviewer_mxwr", "summary": "This paper addresses the temporal directionality in LMMs for video understanding. The authors show that existing LMMs often fail to distinguish between forward and reversed videos, leading to semantic errors in tasks such as question answering or captioning. To tackle this, they propose ArrowRL, a reinforcement learning approach using a novel reverse reward that encourages models to produce different responses for forward versus reversed inputs. They also introduce AoTBench, a benchmark specifically designed to evaluate AoT sensitivity with tasks like sequence direction classification, directional caption matching, and AoT-sensitive VQA. Extensive experiments show that ArrowRL significantly improves AoT perception across multiple open-source LMMs and outperforms baselines on both AoTBench and existing VQA datasets.", "questions": "Why, in Fig. 3 (left), are the forward and reverse results similar? How should this be explained?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper demonstrates that AoT sensitivity is a fundamental yet neglected requirement for video LMMs.\n\n(2) ArrowRL introduces a novel reverse reward in a GRPO framework to explicitly teach temporal directionality, which is an original and well-motivated idea.\n\n(3) The authors show consistent improvements across three base LMMs, multiple AoTBench tasks, and existing video QA benchmarks, demonstrating strong generalization.\n\n(4) The paper contributes a new, carefully curated benchmark with multiple sub-tasks targeting AoT sensitivity, filling a clear evaluation gap in the field.\n\nWeaknesses:\n\n(1) Most of the multiple-choice questions do not require temporal reasoning and therefore do not clearly demonstrate the need for time awareness. As a result, it is not convincing to use MCP as examples to illustrate the motivation for AoT.\n\n(2) The formulation is confusing, e.g., what does P denote in line 167?\n\n(3) Applying GRPO (or other RL methods) may not be necessary. Augmenting the pre-training (or SFT) datasets with reversed videos could potentially address this problem.\n\n(4) Why is maximizing the dissimilarity between forward and backward responses essential? Enforcing them to be dissimilar does not guarantee that the reversed response will be correct.\n\n(5) The evaluation is not comprehensive. It lacks comparison with other popular video benchmarks (e.g., MVBench, VideoMME).", "was_revised": true, "final_justification": "After reading the feedbacks from authors, I was convinced that the proposed technique to enhance the model’s reverse-time understanding capability is helpful. However, the overall effectiveness is marginal. I would like to increase the score to 3.", "rebuttal": "We thank reviewer mxwr for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. On the Use of MCQ for Evaluating Temporal Awareness**\n\n> Most of the multiple-choice questions do not require temporal reasoning and therefore do not clearly demonstrate the need for time awareness.\n\nThank you for highlighting this critical point, which in fact perfectly states the central motivation for our work. We agree entirely that most existing video MCQ samples do not adequately test for temporal awareness. This widespread issue is exactly what inspired us to develop a more rigorous approach. In *Sec. 3.1 & 3.3*, we detail our diagnostic analysis of existing benchmarks, propose the Temporal Divergence Score (TDS) metric, which sifts through these inadequate benchmarks and algorithmically identifies the set of questions that are truly sensitive to the Arrow of Time. \n\nFor the evaluation itself, as noted in *Supp. Ln 344-346*, we chose the MCQ format because it is the standard for objective and reproducible LMM evaluation. In contrast, open-ended QA and captioning are notoriously difficult to evaluate and often require LLM-based or human judges, which can introduce noise and reduce comparability [12].\n\nTherefore, the combination of our targeted TDS filtering with a standard, objective evaluation format provides what we are confident is a robust and convincing test for the Arrow of Time. If there are particular aspects about our evaluation that the reviewer finds unconvincing, we would be glad to elaborate further. \n\n---\n\n**2. Clarification on Notation $P$**\n\n> what does P denote in line 167?\n\nWe follow the convention used in the original GRPO paper [47], where $P$ is used to denote the underlying data distribution of a dataset. In our context, $P$ refers to the distribution from which we draw our training samples, each consisting of a question $q$ and response $o$ (*Ln 165-166*). We will revise the sentence to improve clarity. \n\n--- \n\n**3. On the Necessity of GRPO**\n\n> Applying GRPO (or other RL methods) may not be necessary. Augmenting the pre-training (or SFT) datasets with reversed videos could potentially address this problem.\n\nWe appreciate the comment and agree that SFT is a critical comparison point. In fact, we included this exact experiment in our paper to test the necessity of our RL-based approach.\nAs shown in *Table 2*, we trained a model using SFT on the exact same instruction tuning datasets designed for ArrowRL. While this approach improves upon the base LMM (57.4% vs. 56.2%), confirming the value of our training data, ArrowRL demonstrates a significant advantage, achieving 60.7%. This result indicates that RL is necessary to unlock the full potential of the training signal.\n\n--- \n\n**4. Clarification on ArrowRL reward**\n\n> Why is maximizing the dissimilarity between forward and backward responses essential? Enforcing them to be dissimilar does not guarantee that the reversed response will be correct.\n\nWe fully agree with the reviewer that maximizing the dissimilarity between forward and backward responses would not guarantee correctness. Our methodology was designed with this principle in mind. As detailed in *Sec. 3.2*, ArrowRL’s reward is composed of two parts: (1) a fidelity reward ensures the forward response is correct by rewarding similarity to the ground truth, (2) a reverse reward encourages this correct response to be temporally sensitive by rewarding dissimilarity from reverse response. It is important to clarify that our objective is to generate correct and temporally-aware responses for forward videos, not to ensure the correctness of responses for reversed videos. The reversed video is used solely as a training signal (*Ln 202-214*). \n\nThe final reward is a weighted sum of both components. *Fig. 4* and *Supp. Fig. 8* illustrate these two reward signals, and *Table 2* (along with our response #3 to Reviewer tJ9G) analyzes the effect of varying the weight between them. We thank the reviewer again and are happy to discuss further if there are additional concerns.\n\n---\n\n**5. Clarification on Evaluation**\n> The evaluation is not comprehensive. It lacks comparison with other popular video benchmarks (e.g., MVBench, VideoMME).\n\nWe appreciate this comment and would like to clarify our principled approach to benchmark selection, which draws from 8 popular existing benchmarks, and then present additional results as requested.\n\n**Principled Benchmark Selection**. As noted in *Ln 46-54*, *Fig. 2* and *Sec. 3.1*, a core finding of our work is that many existing video benchmarks are not well-suited to evaluate temporal sensitivity. To address this, we introduced the Temporal Divergence Score (TDS) to quantify each benchmark's reliance on the Arrow of Time. Our analysis of 8 existing benchmarks (*Fig. 3 right*) revealed that TVBench, TempCompass, and Vinoground are the most temporally sensitive benchmarks. We notably selected TVBench over the older MVBench, as TVBench was specifically designed to address the temporal insensitivity of its predecessor while using much of the same underlying video data. We therefore focused our main evaluation (*Table 1*) on these benchmarks to provide the clearest measure of improvement on temporal awareness (*Ln 157-161, 264-265*). \n\n**Additional Evaluation on General Benchmarks**. To address the reviewer's question about performance on other popular benchmarks that don’t require temporal reasoning (like VideoMME), we have conducted the requested experiments. The question is whether our method harms performance on tasks that do not require AoT reasoning. The results below show that it does not; in fact, we see neutral to slightly positive results across the board.\n\n\n| Benchmark         | Base LMM | w/ ArrowRL |\n|-------------------|:------------------------:|:------------:|\n| VideoMME (S)      | 67.78                  | 68.11      |\n| NExT-QA           | 77.11                  | 78.11      |\n| TemporalBench (S) | 60.57                  | 61.17      |\n| VITATECS          | 84.66                  | 85.37      |\n| PerceptionTest    | 57.48                  | 57.64      |\n\nThese results demonstrate that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in *Table 1* with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (*Ln 280-283*). We thank the reviewer for this suggestion and will add this analysis to the final paper.\n\n--- \n\n**6. Clarification on Fig. 3**\n\n> Why, in Fig. 3 (left), are the forward and reverse results similar? How should this be explained?\n\nThank you for your question. Your observation that the forward and reverse results are similar for many of the models shown is entirely correct, and this is precisely the finding that the figure is designed to illustrate.\n\nOur *Fig. 3* provides the temporal sensitivity analysis. As noted in *Ln 128-136, 155-161*, it shows that most existing LMMs are largely insensitive to temporal direction (left) and that current benchmarks fail to meaningfully evaluate AoT perception (right). In other words, even when provided with semantically distinct forward and reverse videos, these models often generate nearly identical responses—demonstrating a lack of temporal awareness, likely caused by an over-reliance on language priors, where the model's textual knowledge overrides the visual evidence of temporal direction (*Fig. 1(b), Ln 46–64*).\n\nThis observed limitation directly motivates our work: we propose ArrowRL to improve temporal sensitivity in LMMs, and AoTBench as a focused benchmark to evaluate this capability. We thank the reviewer again and are happy to discuss further if there are additional concerns."}, {"review_id": "j5lguXq5dY", "reviewer": "Reviewer_6fDT", "summary": "This paper identifies the problem that existing Video LLMs are insufficiently sensitive to the temporal directionality of events in videos. It designs ArrowRL to enhance the perception of temporal directionality in Video LLMs through a reverse reward mechanism, and develops AoTBench to evaluate this capability. Experiments show that the method significantly improves model performance on time-related tasks.", "questions": "* How did the authors construct response pairs where the difference between reverse responses and forward responses can reflect temporal directionality?\n* Does this method still show improvements on general temporal benchmarks like VideoMME?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The problem identified by the authors is mentioned in previous Video Benchmarks (e.g., Tempcompass), but no related work has attempted to solve it. This paper fills this gap, demonstrating significant research significance.\n* The authors' method is rationally designed: leveraging the intuition that \"forward-time sampled outputs should differ substantially from reverse-time ground truth outputs\", they use an LLM to judge the similarity between the two as a reward to supervise Video LLMs in enhancing their temporal direction perception.\n* The authors provide AoTBench to systematically quantify Video LLMs' understanding of temporal order, effectively drawing the community's attention to this issue.\n* The experimental results are solid, showing notable improvements for SOTA models on temporal benchmarks (e.g., Qwen2.5VL achieves a 1.7% improvement on Tempcompass).\n\n### Weaknesses\n\n* Whether the difference between Reverse Response and Forward Response can reflect temporal directionality is crucial to the method's effectiveness, yet the authors lack description on how to construct reverse-forward response pairs that meet this requirement.\n* The method proposed in the paper mainly improves benchmark performance for tasks explicitly related to temporal directionality (e.g., temporal direction classification, forward-reverse temporal caption matching), but it remains unclear whether it can enhance the general ability to understand temporal information in videos.", "was_revised": true, "final_justification": "The authors have comprehensively addressed my questions:\n\n1. Regarding the question of whether \"the difference between reverse responses and forward responses can reflect temporal directionality,\" the authors have resolved it through \"video source selection\" and \"dynamically weighting forward/reverse rewards.\" Overall, this is reasonable.\n2. The training scheme introduced by the authors can slightly improve the performance on General Video Benchmark.\n\nTherefore, I retain my score as positive.", "rebuttal": "We thank reviewer 6fDT for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. Clarification Regarding ArrowRL Response Generation**\n\n> How did the authors construct response pairs where the difference between reverse responses and forward responses can reflect temporal directionality?\n\nWe appreciate this insightful question, and fully agree that ensuring the difference between forward and reverse responses reflects true temporal directionality is crucial.\n\nFirst, we clarify that our method does not construct or use pre-defined \"forward-reverse response pairs.\" Instead, as part of the GRPO framework, all responses are generated on-the-fly during training: the policy model generates a group of forward responses to the original video, and a reverse response. These responses are then used to compute the fidelity and reverse rewards in real time.\n\nThis dynamic approach means the central challenge is ensuring the reverse reward is applied intelligently, which is crucial for the method's effectiveness. We achieve this with two techniques:\n\n**High-temporality Data Curation**. For captioning tasks, we leverage the curated RTime dataset [14], which consists of videos specifically selected for having meaningful and distinct forward vs. reverse actions (*Ln 251-253*). This provides a clean, high-quality signal for learning. \n\n**Dynamic Reward Weighting**. For larger, less curated datasets, we employ a dynamic weighting scheme (*Ln 207–214*) that detects when the reversed response is too similar to the forward ground truth and automatically disables the reverse reward for that sample.\nThe effectiveness of these strategies is supported by our experiments.  The ablation in *Table 2* (+ArrowRL (γ=1)) already demonstrates that removing the dynamic weighting hurts performance. Furthermore, to isolate the effect of data selection, we ran a new experiment focused only on the captioning task, replacing the high-quality RTime data with uncurated videos from LLaVA-Video-178K. The results below (same as shown for Reviewer tJ9G) underscore the importance of our data curation strategy.\n\n| Model                          | AoTBench Acc. (%) |\n|-----------------------------|:-----------------:|\n| Base LMM                      |       56.2        |\n| w/ ArrowRL (RTime Cap.)            |       **60.4**        |\n| w/ ArrowRL (LLaVA-Video-178K Cap.) |       57.7        |\n\nTogether, these strategies ensure our reverse reward is applied precisely and only to cases where it can encourage a meaningful understanding of temporal directionality.\n\n---\n\n**2. General Video Benchmark Performance**\n\n> Does this method still show improvements on general temporal benchmarks like VideoMME?\n\nWe thank the reviewer for this important question regarding ArrowRL’s performance beyond temporally-focused benchmarks. We reiterate that ArrowRL is designed as a targeted enhancement for an LMM's temporal sensitivity. As our analysis in *Sec. 3.1* shows, many existing video benchmarks do not actually require this capability to be solved (e.g., they can be answered with single, shuffled or reversed frames). For this reason, our main results in *Table 1* focus on the benchmarks we identified as temporally sensitive, where ArrowRL shows significant boosts.\n\nThe reviewer raises a valuable follow-up question: whether our specialized training harms performance on easier tasks where AoT is not required. To answer this, we have conducted experiments on the benchmarks that our own analysis identified as less sensitive (*Fig. 3 right*), including VideoMME as suggested by the reviewer.\n\n| Benchmark         | Base LMM | w/ ArrowRL |\n|-------------------|:------------------------:|:------------:|\n| VideoMME (S)      | 67.78                  | 68.11      |\n| NExT-QA           | 77.11                  | 78.11      |\n| TemporalBench (S) | 60.57                  | 61.17      |\n| VITATECS          | 84.66                  | 85.37      |\n| PerceptionTest    | 57.48                  | 57.64      |\n\nAs shown, ArrowRL preserves or even slightly improves performance across these general video benchmarks that do not demand temporal awareness. This new analysis (same as shown for Reviewer mxwr) confirms that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in *Table 1* with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (*Ln 280-283*). We thank the reviewer for this suggestion and will add this analysis to the final paper."}, {"review_id": "TQ5FbW8LTJ", "reviewer": "Reviewer_GqEN", "summary": "**Problem**: In this work, the authors highlight the problem of modern large multimodal models (LMMs) not using the \"arrow of time\" while perceiving videos. \n\n**Key contributions**: First, it is shown that existing LMM benchmarks (e.g., TemporalBench) and LMM models (e.g., Qwen2-VL) are not sufficiently sensitive to the arrow of time, i.e., shuffling or reversing the frame order does not substantially affect the accuracy. Second, as a redressal, an RL based post-training strategy is proposed to instil this time-sensitivity into an existing LMM (e.g., Qwen2-VL). Two reward signals are used: target fidelity (to encourage the correct answer) and reverse reward (to force different answers if the video is presented in a reverse order). Third, a new benchmark, AoTBench is proposed with three sub-tasks: (i) direction classification: whether a video if played forward/reverse, (ii) caption matching: matching forward/reversed videos to the right language description, (iii) VQA samples from existing benchmarks that are more likely to be sensitive to the arrow of time.\n\n**Main outcome:** It is experimentally shown that the proposed strategy not only improves performance on the proposed AoTBench, but also improves performance on existing video benchmarks like TV-Bench and Vinoground.", "questions": "**Questions**\n\nSome of these are already mentioned in the weaknesses section. I have elaborated a bit on them here.\n\n1. Why should an existing video LLM model understand what one means by \"forward or reverse direction of the video\"? I do agree that checking understanding of arrow of time for modern language-based video models is essential. \n2. Describing direction of time in language is tricky for irreversible videos. How would one describe a reversed video of \"smoke diffusing\"? Although I know this is not the main task, but it is worth asking how the model's perception changes when encountering a reversed video (e.g., are certain activations activated more when this happens?)\n3. Does the added post-training affect \"spatial\" performance at all? For example, on non temporal benchmarks, does the model suffer at all or retain its performance?\n\n**Suggestions**\n\n1. Reversed videos may be out of distribution for existing models. Perhaps, is there a control task (like in [1]) where the question is insensitive to the arrow of time (e.g., describe all objects appearing in the video) and the model does well with both forward and reverse videos? This can establish that low performance of existing models need not be because the video is OOD.\n2. A justification for the choice of measure (KLD between output distributions of **first token**) of arrow of time consistency would make the paper stronger. Likewise, it would be helpful to clarify L145 on filtering out samples.\n\n[1] Test of Time: Instilling Video-Language Models with a Sense of Time. Bagad et al. CVPR 2023.", "limitations": "While the Supp includes failure cases, it may be worth adding a couple of lines in the main text on what kinds of samples the model fails on and if there is a pattern to it. That could also guide future work.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Elegant adaptation of GRPO for time-sensitive video understanding. While the problem of arrow of time is not new, I think this paper presents a thorough method to tackle it in context of modern video-LLMs. The \"reverse reward\" for distinctive answers for forward/reverse videos is a sensible choice.\n2. The weaknesses shown in existing video models/benchmarks should guide the video community towards better models/benchmarks. For example, the insight that LLAVA-OV-7B, Qwen2.5-VL-7B are more time-sensitive models (on TVBench) and Vinoground/TVBench/TempCompass are more time-sensitive benchmarks is very useful.\n3. The design of new AoTBench and the subsequent experiments are very thorough and well thought-out. The proposed ArrowRL shows clear benefits on arrow of time tasks while also improving performance on existing benchmarks. \n4. The paper is written very well; the flow is very smooth and logical.\n\n\n**Weaknesses**\n1. Grounding arrow of time for irreversible videos in language is probably ill-posed. In the first AoTBench task, the authors propose a VQA-like task where the model is tasked with choosing which of the two videos is played in forward/reverse directions.\n\t1. For irreversible videos (e.g., where reverse video violates a physical law), the task is purely visual. It is awkward to describe a waterfall going upwards in language.\n\t2. Why should an existing video LLM model understand what one means by \"forward or reverse direction of the video\"? In this work, the train set contains such questions. So the resulting model knows what \"forward\" means. But what about existing models which may not have \"forward\"/\"reverse\" in their train set?\n\t3. Reversed videos may be out of distribution for existing models. Could the low performance be attributed to that instead of just not understanding arrow of time?\n2. The metric designed to measure if the LMM understands arrow of time is slightly unconvincing. As described in L140-L145, the probability of the first token over the MCQ options is considered. A justification for this is lacking. Also, \"We first filter out instances where the average reverse performance exceeds its forward performance.\" -- this is confusing.", "was_revised": true, "final_justification": "I have retained my original positive rating of the paper. \n\nFrom the initial review, all my concerns were addressed by the rebuttal. Particularly, we had a fruitful discussion on the aspect of grounding physics (arrow of time in particular) in language. \n\nI also went through all the other reviews and the corresponding rebuttal responses. Reviewer `mxwr` raises a fair question on whether encouraging  dissimilarity between forward and backward responses is essential. In my view, the fact that the proposed RL post-training improves arrow of time reasoning while maintaining performance on standard benchmarks is a major positive. One can see arrow of time as a fundamental desirable property which may not be applicable to everyday videos, but can be fundamental to understanding intuitive physics, for example. So instilling arrow of time while doing well on standard benchmarks is a good outcome in my view. Overall, I think the authors addressed all questions quite comprehensively.", "rebuttal": "We thank reviewer GqEN for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. On Grounding Arrow of Time in Language**\n\n> Why should an existing video LLM model understand what one means by \"forward or reverse direction of the video\"?\n\n> Describing direction of time in language is tricky for irreversible videos. How would one describe a reversed video of \"smoke diffusing\"?\n\nWe thank the reviewer for these insightful comments. They touch on the core challenge of evaluating a concept as fundamental as the Arrow of Time within a language-based framework. We address the key points below.\n\n**Describing Arrow of Time in Language**. We agree that describing a physically irreversible event in reverse (like a waterfall going up) is linguistically awkward. Our evaluation is designed accordingly to ensure that it never requires a model to generate a free-form description of a physically implausible reversed video:\n- The quantitative sequence direction classification task in AoTBench is a classification, not a captioning task. LMMs are required to map its visual understanding of sequence directionality to a simple linguistic label (\"forward\" or \"reverse\"), testing a fundamental aspect of multimodal reasoning.\n- For qualitative examples that do involve description (e.g., *Fig. 1, Fig. 7, right*), we use \"reversible\" videos where both directions are physically plausible. We will add clarification in the paper. \n\nTo your specific question about describing reversed physically implausible events such as \"smoke diffusing\", our observation is that current LMMs, due to lack of temporal directionality understanding, fail to describe the implausible dynamics. They tend to focus on static objects and often repeat the forward-action verb (e.g., still saying \"smoke diffusing\"), confirming their temporal insensitivity.\n\nIn all, we believe the ability to recognize that a video violates the physical laws of nature and connect that visual intuition to a symbolic linguistic concept is a fundamental capability. For LMMs to progress, they need to be able to ground simple linguistic concepts like \"forward\" and \"reverse\" in their visual understanding—a task humans perform effortlessly. Our classification task provides a crucial testbed for this capability.\n\n**On Evaluating Base LMMs**. About whether an existing LMM should be expected to understand this task without explicit training, we clarify that our goal in testing base LMMs is not to claim they should succeed, but rather to use the task as a diagnostic tool. Their near-random performance, as shown in *Table 1*, is a key finding of our paper. It provides a critical baseline that demonstrates a clear capability gap in current models and powerfully motivates the need for targeted training methods like ArrowRL.\n\nFinally, we agree with the reviewer's insightful point that understanding when a visual phenomenon is difficult to describe is a key challenge for more advanced LMMs. While our work focuses on the foundational skill of recognizing directionality, we believe exploring the limits of linguistic description for complex physical events is an important direction for future research. We welcome any further suggestions from the reviewer on improving this task format and will revise the paper accordingly to reflect this discussion.\n\n---\n\n**2. Control Task to Disentangle OOD Effects**\n\n> Is there a control task (like in [1]) where the question is insensitive to the arrow of time (e.g., describe all objects appearing in the video) and the model does well with both forward and reverse videos? This can establish that low performance of existing models need not be because the video is OOD.\n\nWe appreciate the suggestion of adding a control task, which is well-motivated by prior work ([1] Bagad et al., cited in our paper as [2]) on disentangling temporal effects. Accordingly, we design action recognition as the control task, using videos from the UCF101 dataset and their action label. For each video, we formulate a binary-choice question with the prompt, \"What action is being performed in this video?\" The correct action was provided as one choice, and the incorrect option was randomly drawn from the set of all other action labels. We then evaluated several LMMs on both the original (forward) and time-reversed versions of these videos. \n\n| LMM               | Forward Acc. (%) | Reverse Acc. (%) |\n|-------------------|:------------------:|:-------------------:|\n| GPT-4o            | 100.0            | 100.0             |\n| Gemini-1.5-Pro    | 98.2             | 99.4              |\n| LLaVA-OV-7B       | 99.2             | 99.4              |\n| Qwen2.5-VL-7B     | 99.4             | 99.4              |\n\nThe results above show that all models perform at near-perfect accuracy on this control task, irrespective of video direction. This contrasts sharply with their near-random performance on our AoT-sensitive sequence direction classification task (*Table 1*, UCF column). This analysis provides strong evidence that the failure of existing LMMs on AoT tasks is due to their inability to perceive temporal directionality, not a general failure to process reversed videos as OOD inputs. We appreciate the reviewer for prompting this important analysis and will incorporate it into our paper. \n\n[1] Test of Time: Instilling Video-Language Models with a Sense of Time. Bagad et al. CVPR 2023.\n\n---\n\n**3. Justification for Using the First Token Probability and Clarification on Ln 145**\n\n> A justification for the choice of measure (KLD between output distributions of first token) of arrow of time consistency would make the paper stronger. Likewise, it would be helpful to clarify L145 on filtering out samples.\n\n**First Token Probability**. We use the first-token probability to estimate the model’s likelihood of selecting each option in the MCQ format. Since our prompt instructs the model to “Answer with the option’s letter from the given choices directly,” the probability of the first token (e.g., A, B) serves as a reliable proxy for the model’s chosen answer. This approach is standard and widely adopted by the LLM community [1–3]. Our novelty lies in adapting this standard evaluation practice for temporal analysis by computing the KL Divergence between the distributions generated from the forward vs. the reversed video.\n\n**Ln145 Clarification**. In some rare cases, we observe that our candidate LMMs perform better on reversed videos than on the original forward ones. These are typically outlier cases where temporal direction is either ambiguous or irrelevant. We exclude these samples from further analysis to ensure that our evaluation focuses on instances where AoT reasoning is actually required.\nWe appreciate the reviewer’s suggestion and will update the manuscript to clarify these design choices.\n\n[1] Brown et al., Language Models are Few-Shot Learners, NeurIPS 20\n\n[2] Hendrycks et al., Massive Multitask Language Understanding, ICLR 21\n\n[3] Lin et al., TruthfulQA: Measuring How Models Mimic Human Falsehoods, ACL 22\n\n---\n\n**4. General Video Benchmark Performance**\n \n> On non temporal benchmarks, does the model suffer at all or retain its performance?\n\nWe thank the reviewer for this important question regarding ArrowRL’s performance beyond temporally-focused benchmarks. We reiterate that ArrowRL is designed as a targeted enhancement for an LMM's temporal sensitivity. As our analysis in *Sec. 3.1* shows, many existing video benchmarks do not actually require this capability to be solved (e.g., they can be answered with single, shuffled or reversed frames). For this reason, our main results in *Table 1* focus on the benchmarks we identified as temporally sensitive, where ArrowRL shows significant boosts.\n\nThe reviewer raises a valuable follow-up question: whether our specialized training harms performance on easier tasks where AoT is not required. To answer this, we have conducted experiments on the benchmarks that our own analysis identified as less sensitive (*Fig. 3 right*).\n\n| Benchmark         | Base LMM | w/ ArrowRL |\n|-------------------|:------------------------:|:------------:|\n| VideoMME (S)      | 67.78                  | 68.11      |\n| NExT-QA           | 77.11                  | 78.11      |\n| TemporalBench (S) | 60.57                  | 61.17      |\n| VITATECS          | 84.66                  | 85.37      |\n| PerceptionTest    | 57.48                  | 57.64      |\n\nAs shown, ArrowRL preserves or even slightly improves performance across these general video benchmarks that do not demand temporal awareness. This new analysis (same as shown for Reviewer mxwr and 6fDT) confirms that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in *Table 1* with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (*Ln 280-283*). We thank the reviewer for this suggestion and will add this analysis to the final paper.\n\n---\n\n**4. On Including Analysis of Failure Cases in Main Text**\n\n> While the Supp includes failure cases, it may be worth adding a couple of lines in the main text on what kinds of samples the model fails on and if there is a pattern to it. That could also guide future work.\n\nThank you for your comment. We will move a concise summary of our failure case analysis (detailed in *Supp. Sec. D*) into main paper as suggested."}]

Author rebuttal:

Meta-review: This submission addresses the lack of temporal directionality (Arrow of Time) sensitivity in large multimodal models. The proposed ArrowRL method combines a reverse reward with curated data and dynamic weighting, and AoTBench provides a principled benchmark for evaluating AoT sensitivity.

The paper received mixed reviews. Reviewer GqEN recommended acceptance, highlighting methodological soundness and the broader value of AoTBench. Reviewer 6fDT gave a borderline accept, finding the rebuttal convincing on issues of forward/reverse response construction and generalization. Reviewer mxwr remained skeptical, concerning that the improvements may be marginal and potentially achievable without RL; although their score increased after rebuttal, it remained borderline negative. Reviewer tJ9G raised concerns about the reward mechanism, benchmark construction, and hyperparameter sensitivity, but did not re-engage in the discussion phase. After examining the rebuttal, the AC believes these concerns are properly addressed by the authors’ additional experiments.

Overall, the authors and most pf the reviewers engaged actively and constructively during the discussion. The added hyperparameter analyses, control experiments showing that poor AoT performance is not simply due to OOD effects, and expanded evaluations on broader benchmarks strengthen the case for the method. The rebuttal also clarified AoTBench’s principled design through the Temporal Divergence Score. While some conceptual issues remain (e.g., handling irreversible events in language, or the necessity of RL compared to simpler methods), these are acknowledged and framed as promising directions for future work.

Given these contributions and discussions, and despite lingering doubts about the magnitude of impact, the AC recommends acceptance. The paper is technically solid, makes a timely contribution, and provides resources that are likely to spur further research into temporal reasoning for LMMs.
Decision: Accept (poster)