Improve Temporal Reasoning in Multimodal Large Language Models via Video Contrastive Decoding
Authors: Daiqing Qi
Dongliang Guo
Hanzhang Yuan
Handong Zhao
Mengxuan Hu
Lehan Yang
Sheng Li
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Video Large Language Model
Primary area: applications

Abstract: A major distinction between video and image understanding is that the former requires reasoning over time.
Existing Video Large Language Models (VLLMs) demonstrate promising performance in general video understanding, such as brief captioning or object recognition within individual frames. However, they often struggle with temporal reasoning such as understanding continuous actions or tracking object transformations over time—which typically demands the integration of multiple frames in a temporally coherent manner.
We first explore and explain such failures in Video LLMs from the perspective of \textit{language and ``image'' priors.}
While existing research has attempted to enhance the temporal understanding of VLLMs through various training strategies, the demand for expensive computational resources and training data often presents significant barriers.
To this end, we further propose a simple yet novel idea for improving temporal reasoning in videos at no additional training cost.
Specifically, to better capture the temporal structure across multiple frames—the key to effective temporal reasoning—we distort the temporal consistency in key frames \textit{during the decoding phase}. Such corruption induces time-insensitive wrong responses from the model, which are then contrastively avoided when generating the final correct output. In this way, the model is encouraged to perform more temporally coherent reasoning.
Our method yields consistent improvements across both temporal-specific and general video understanding benchmarks, demonstrating its effectiveness and generalizability.

Improve Temporal Reasoning in Multimodal Large Language Models via Video Contrastive Decoding
Daiqing Qi!
Dongliang Guo!
Hanzhang Yuan
Handong Zhao?
Mengxuan Hu!
Lehan Yang!
Sheng Li
University of Virginia
2 Adobe Research
Abstract
A major distinction between video and image understanding is that the former requires reasoning over time. Existing Video Large Language Models (VLLMs) demonstrate promising performance in general video understanding, such as brief captioning object recognition within individual frames. However, they often struggle with temporal reasoning such as understanding continuous actions or track- ing object transformations of multiple frames in a temporally coherent manner. We first explore and explain such failures in Video LLMs from the perspective of language and "image" priors. While existing research has attempted to enhance the temporal understanding of VLLMs through various training strategies, the demand for expensive computa- tional resources and training data often presents significant barriers. To this end, we further propose simple yet novel idea for improving temporal reasoning videos at no additional training cost. Specifically, to better capture the temporal structure across multiple frames the key to effective temporal reasoning-vy we distort the temporal consistency in key frames during the decoding phase. Such corruption induces time-insensitive wrong responses from the model, which are then contrastively avoided when generating the final correct output. In this way, the model is encouraged to perform more temporally coherent reasoning. Our method yields consistent improvements across both temporal -eneeri--ppecfficc and general video understanding benchmarks, demonstrating its effectiveness and generalizability.
typically demands the integration
Introduction
Benefiting from the significant advancements in Large Language Models in recent years, Video LLMs 14 34 35 16 30 have also experienced rapid development, exhibiting strong capabilities in general video understanding. A key distinction between video understanding and image understanding is that the former requires models to comprehend not only individual input frames but also the temporal relationships among them. Consequently, temporal perception is crucial for Video LLMs. However, recent studies [1] 21 have shown that even for simple and straightforward temporal reasoning questions that humans can easily answer, Video LLMs, such as LLaVA- Video [35] Video- LLaVA [14] and VILA [66] 22 often make mistakes that are clearly inconsistent with the ground truth. Recent studies 28
29 have investigated the limitations of temporal reasoning Vidco LLMs from model-centrin perspective. Given that typical Video-LLM is composed of a vision encoder and an LLM backbone, a common analytical approach to disentangle and examine the contributions of each component to temporal reasoning. While some works [28] g 29 attribute the temporal reasoning deficiencies of Video LLMs ineffective video embeddings and accordingly focus on improving temporal information aggregation, others [1] leave aside the vision modules and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ieguts whst hoppon to
rgut Wher www.
nite rage?
Cinduuiiiy semppenrig pienoe by place Graducly watten diopppaatng
& O
Ellpadd
(Whona)
Ludea 15 1381 (Ohongg
Owenn 7B Instruch
LLaVA V deo Owen2 Oura
Input: le dircction in the cun modding
What recttionn sunmining WV? A-fuling
Fulling
S,thang
Stoyn gin one place
Owand VL-TD Tedeetsmens since the c0s2e loing mismore kodk de setting
LLaKA Video Owend: (Wrrngl
Cheerd-
thannsing.
Figure 1: Example of how language and "image" priors affect Multimodal I Large I Language Models or Video Large Language Models in image or video understanding. (I), (3) show the negative effect of language priors and (2) reveals the how "image" priors from static visual content mislead the Video LLM's understanding of the video in (4).
embeddings, highlighting that the LLM itself exhibits weak sensitivity to the order of long textual sequences, as well as long video embeddings.
Existing works primarily focus on explaining the limitations of temporal perception from the perspec- tive of the model itself. Consequently, the improvements derived from such analyses often require modifications to the model, such as refining video encoding [28] g 29] or improving the language counterpart [. However, these strategies typically involve modeling training (e.g.. instruction tuning), which is computational- and data- -ata-intensive. Moreover, due to the increasing diversity of video LLM architectures nowadays, observations or findings from one certain architecture may not always generalize to others.
Towards this end, instead of digging into model architecture, we analyze model behavior through its responses to investigate under what circumstances video L.MSs are more prone to making mistakes. We find that when video LLMs fail to understand temporal information in videos, they exhibit a behavior similar to what multimodal LLMs exhibit when hallucinate-namell being influenced by language priors. More interestingly, due to the temporal dimension inherent video inputs (compared to static image inputs), observe that video LLMs also suffer from a negative influence of an "image" prior. For instance, the static visual features in individual frames weinhs-which do not earry temporal information, however, can mislead the LLM's 's perception of temporal dynamics. To avoid model-dependent modifications, we do not directly mitigating the negative effects at the model level. Instead, take the opposite approach: induce erroneous responses, and then use them as contrasting objectives in Contrastive Decoding [12 This allows us explicitly steer the decoding process away from such failure modes and toward generating the correct answers.
amplify these influencing factors to intentionally
2 Uncovering Language and "Image" Prior in Temporal Reasoning
What Undermines Temporal Perception in Video LLMs? In contrast to existing works, we study the problem of limited temporal reasoning in Video-LLMs from different perspective: instead of focusing on the model itself, we explore the conditions under which Video LLMs are more likely to struggle with temporal understanding. Although different models exhibit varying results on the same benchmark, is observed that they tend to make mistakes more frequently on a specific subset of questions than on others. By analyzing the characteristics of these failure cases, key factors that undermine temporal reasoning can be uncovered.
Language prior typically refers to the prior knowledge in LLMs during (textual) pre-training or instruction-tuning such as commonsense knowledge and reasoning in the context of multimodal understanding [174] 4. However, it can sometimes hinder the model from grounding its answers in visual content, over-relying on the prior knowledge.
Fig. (1) (1) (top left) presents a typical example in image understanding with MILMss. When asked about the shape of the watermelon in the image, L.L.VA [19] chooses the incorrect option 'C', which aligns with the common sense that a watermelon is ellipsoidal. This strong language prior biases the

---

model toward selecting "C" instead of grounding to the visual input. When posing the same question without the image to its LLM backbone, Vicuna-13B.. 13B-v1.5. the LLM responses with the same answer, revealing the influence of the LLM language prior on the MILM's behavior. Similar observations are also mentioned recent works 4 on ML..M hallucinations image understanding. We are naturally curious whether the language prior has similar influence on Video LLMs? If so, given the uniqueness of video inputs- such as temporally ordered multi-frame sequences- are there any distinct behaviors that Video LLMs may exhibit compared to MLLM with image inputs?
Language Prior in Video LLMss We conducted experiment to investigate whether language priors affect the temporal reasoning in Video LLMs. For this study, we selected two representative open-source Video LLMs- LL ViAA-iddo-7B--een (35) and Video-LLaVA VA [ test models, and used TempCompass 21 as the benchmark, which includes a variety of tasks closely related to temporal understanding, such as event ordering and attribute change. First, we performed standard inference with each Video LLM on TempCompass with both the video and textual input are provided. Then, we randomly sampled 200 questions where Video LLMs made mistakes, and conducted blind evaluation on them, where the video input was removed and only the textual prompt was given to the LLM. The results from this blind evaluation can roughly reflect the influence of the language prior on inducing mistakes in Video L.MSs.
Results show that, among the incorrect predictions made by LLaVA, 46.7% and 38.9% respectively matched the answers produced by blind LLM, significantly higher than random chance. In these cases, the Video LLMs failed to ground their responses in videos and instead relied on language priors, leading to incorrect answers.
and Video-
Fig. (3)) provides illustrative example. When the video is not provided, the blind LLM (Qwen2- 7B- 1n-sustuctt selects option C, which corresponds to the answer with a higher prior probability that aligns with commonsense expectations. When both the video and the question are provided the Video LLM, the model still chooses option C. This indicates that, despite access to visual input, the model remains influenced by the language prior and overlooks the temporal information present in the video. However, we observed in the experiments that in number cases, even when the blind LLM predicted the correct answer, the Video LLM surprisingly made an entirely different and incorrect choice contradicting both the language prior and the information presented in the video.
Image Prior in Video LLMs. Fig. D (4) illustrates an interesting case. When the image is not provided, the blind LLM makes the right choice, which can be viewed as a positive influence of the language prior. However, the Video LLM still chooses the incorrect answer, "falling", a bias in this case introduced by the visual content of the video itself. As shown in Fig. | (2), when we extract a along with
single frame from the middle of the video and input it into an MLLM (Qwen2-VL- the question, the model perceives the scene as sunset and chooses "falling" accordingly. This visual bias ultimately leads the Video LLM to overlook the temporal progression of the sun rising and to make the wrong prediction.
Briefly, for image understanding, the input consists of both textual and visual information, and language priors can influence how MLLMs interpret visual content, In video understanding, the model must understand textual, visual, and temporal information simultaneously. In this setting, beyond language priors, visual bias-specificallly, biases introduced by static frames, which we term as "image" priors, can also negatively impact temporal perception. As demonstrated in Fig. (2), certain frames (e.g., one that resembles a sunset) may dilute the model's ability to accurately perceive temporal dynamics.
3 Video Temporal Distortion
Our analysis demonstrates that language and "image" priors can impair the temporal perception of Video LLMs. However, the underlying mechanisms can be more complex than they appear, potentially influenced by factors from pre- traiirain instruction tuning strategies, to model architecture. Instead of intervening in model design, such as the architecture or training strategies, which is often compute- and data-intensive, we seek a post-hoc correction approach. Given that priors can impair the temporal reasoning of Video L.Mss, is it possible to develop a model-agnostic, plug-in method that uses the model's own temporally insensitive errors as contrastive signals, explicitly guiding the model away from such biases and increasing the probability of generating correct predictions?
3

---

Contrastive Decoding (CD) [12] offers promising solution. CD is decoding approach that aims to find text which maximizes the gap between the pre-oabbbilitiss of good and bad LLM response. It helps us better avoid selecting tokens with high probability in the bad response, since higher probability in the bad response means smaller gap with its corresponding probability in the good response. Intuitively, if we can induce temporally insensitive responses from Video LLMs, they can be explicitly avoided by contrastive decoding. Therefore, the key challenge that follows is to guide Video LLMs consistently generate such bad responses, so that they can be used contrasting objectives during the final decoding phase.
3.1 Contrastive Decoding in Video Large Language Models
Decoding in LLMs. Considering a Video LLM parametrized by 0. It takes as input text query x and vidco context V, and generate a relevant response y the text query. The response y is sampled auto-regressively from the probability distribution conditioned on the query and the video context V, formulated as:
Po V,x,y<t) x exp (y: V V,x,y<t).
where yr denotes the token time step I, and y<rr represents generated tokens up (1 1).
Contrastive Decoding in Video LLMs. Specifically, given a text query x and a video input V, the model generates two output distributions: distorted video imput V", which is derived by applying pre-defined distortion (c.g., adding noise to visual features as the simplest case) to V. Then, new contrastive probability distribution is computed by leveraging the differences between two original distributions. The new contrastive distribution Pvia is formulated as:
conditioned on the original V and the other on the
(y V, V x) softmax [(1 + (o)oggia v.x) o V x)],
(2)
where larger a indicate a stronger amplification of the differences (a 0 reduces to regular decoding). The process is shown in Fig. 3 Wrong option "B" is assigned possibilities in the original normal distribution logit, (y V, x) due to language or image priors. However, temporal information is removed in the distorted video input. which further amplifies the bias, option "B" receives significantly higher scores in logit, (y V V',x). Finally, the score of "B" is notably reduced petd after Eq. 3 is applied, leading to correct answer "A". Following Li et al. [12), we also apply an adaptive plausibility constraint on Pvtá. where CD applied only to high-probability tokens whose probabilities exceed fraction 3 E [0, 1] of the maximum token probability. Details are discussed in Appendix A.
3.2 Video Temporal Distortion
What Makes an Effective Temporal Distortion. Distortion strategies play key role in elevating the probabilities of bad responses in logit, (y V', x)) while diminishing those of good responses. To consistently generate bad responses as contrasting objectives, we need remove temporal clues in the video input while maintaining confounding priors that induces incorrect answers neglecting temporal information. Such balance is critical. For example, the case shown in Fig 4)), masking the entire video unexpectedly leads to the correct answer. This is undesirable, because if the correct answer is used as a contrasting objective, its possibility reduced with Eq. 3 What we need is an carefully distorted input, like in Fig 2), which lacks temporal clues and misleads the model.
Intuitive solutions include: ① adding noise visual fea- tures 2 randomly shuffling frame sequences O randomly dropping frames. Fig. 2 shows results of LLa VA-Video- 7B (35) on EventHallusion (32), where the videos depict continuous actions that require strong temporal percep- tion. We adopt three different strategies when applying the distortions for contrastive decoding: randomly apply distortion on 1. all frames, 2. first half frames and 3. last half frames. While the overall results improve, they are notably sensitive to different sampling preferences toward the beginning or the end of the video. It implies that designing adaptive distortion strategies with greater stability and adaptability is crucial. Towards this end, we propose Video Temporal Distortion strategy that adaptively distorts frames.
Overa Rist Hall
Lest Hal
Varilla Model
Figure 2: Distortion results.

---

diraction thalitit gaing?
Fallng down (Ground ug
Winging forward
Homa ning ninghares
teyFores
lightty
opont desrEu a6.e7
Tempora tonn
(L+ allggry (--0 r-c--loett Phag
Anghtty 1V)x)
Tomporal Distartion Unit
Figure 3: Overview. The original video embeddings V are first forwarded the LLM with text query x to obtain logit& (y V, x), where intermediate attention maps in LLM layers are retrieved and fed to Temporal Distortion Unit guide the distortion of original video embeddings. Then distorted video V' is imput to LLM to obtain ly V x). Final token generated from pvtd in Eq.
Temporal Distortion Criteria. We argue that, an ideal distortion should satisfy the following criteria: (1) it removes temporal information. (2) it includes priors that are likely mislead the model, such as salient static visual features. (3) The distortion applied to the video should be adaptive towards different video inputs than random ensure stability. Our solution is illustrated in Fig. Given original video embeddings V, and text query x, we obtain distorted video embeddings V with our Temporal Distortion Unit guided by attention maps from intermediate LLM layers adaptively distort important frames with rich temporal information, while keeping less relevant frames that can potentially provide misleading context information or priors.
Framework. Fig. 3 (Right) shows the distortion pipeline. Given a set of K original video frame embeddings j-j== { and corresponding attention maps (A,), from L intermediate LLM layers, We first compute the importance of each image token. To mitigate the bias relying solely on the attention maps from the final layer, compute token importance at each layer and aggregate it with momentum. Specifically, given attention map A; at layer I, token importance matrix is computed as:
S: Σ
A (1:)))
[-1], A E (h.m.)
where h, denote the value of attention heads and unmasked input tokens so far, respectively. Consequently, the importance of an image token v, indexed at is SyDi] from layer 1. To obtain the final token importance score, we apply momentum-bases accumulation over all layers. Specifically, we iteratively update the aggregated importance map SJ as follows:
S W. Š + (1 wm)) St.
where um [0, 1) is the momentum coefficient that controls the contribution of previous layers. This update emphasizes recent layer information while retaining long-range contributions from earlier layers. Then, the frame importance is computed as the sum of the importance scores of all image tokens within the frame.
Key Frame Fusion. First, we select the top-wsar (Frame Distortion Ratio) most important frames and remove the their temporal information by substituting each of them with the results of mean pooling over the selected set. In this way, temporal clues are removed while coarse- carse-grained image context is retained, which can induce biased temporal insensitive inaccurate response. We further add small amount of Gaussian noise with weight Wfpr (Frame Perturbation Ratio) to the pooled embeddings. Compared directly dropping selected frames, mean pooling retains more confounding visual context, and is more robust than shuffling, yielding better results in practice consistently.
5

---

Siding Window dkg =0
Figure 4: Disrupt moving content in remaining frames within a sliding window. Left: Marked dynamic blocks. Right: Fusion results of marked dynamics.
Distorting Key Tokens in Remaining Frames. Although the remaining frames are less important compared to the top-Widr most important ones, their large quantity still preserves certain degree of temporal information. If frame-level fusion is applied to these frames in the same manner as it is to key frames, can lead excessive information loss across the video and damage the misleading image priors, making the model unable to consistently produce the most probable mistakes. As result, the contrasting objective becomes overly random and loses its guiding effect in contrastive decoding. Therefore, we design more fine-grained distortion strategy specifically for this subset of frames. First, for each remaining frame, we mask the top-Wsdr (Token Distortion Ratio) most important image tokens, thereby masking regions that potentially contain temporal cues.
Disrupting Moving Content in Remaining Frames. In the previous step, we performed fine-grained token-level masking on each of the remaining frames based on token importance. However, since attention maps sometimes do not accurately reflect the actual importance of each token, we instead leverage visual similarity between frames in this step to "blur" the moving content within the frames (if any), thereby removing potential temporal changes while preserving the overall image context.
Fig 4 visualizes this step. A sliding window (non-overlapping) of size Wws is adopted to process the entire video sequence, where we independently process the frames within window each time. A token is considered dynamic if its corresponding visual content changes significantly across the window. However, due the high spatial resolution and token granularity of popular vision encoders like CLIP (27), direct per-token comparisons are neither robust nor efficient. To address it, we first downsample each frame from size [H, W) (_H.., W by applying mean pooling over "Way whs
non-overlapping patches, as illustrated in Fig. Each resulting region is termed as Block.
Downsampling. Note that in the downsampling stage, i.e., when we downsample each frame from size (H, W) (un)) wi.) ), do not really reduce token numbers. As illustrated in Fig. 4 (left), actually replace all image tokens within one "downsampled" region with the mean of the tokens is the mean of the four included. For example, the value of each of the four token within B (0,0)
tokens.
For given frame and block at position (i,j), we denote it B (1,)) We define its similarity score as the average cosine similarity with all blocks at the same position in the other frames within the window:
sim B ((.))
cos
B B (3,5) ((,)))
10w
Blocks with lower similarity scores are considered more dynamic, as they indicate greater temporal change across frames. With similarity scores for all blocks, we select the top-Wetr (Content Fusion Ratio) blocks with the lowest similarity scores across the entire window and label them as Dynamic Blocks. For each block (.,) if it is not dynamic, values. If it is dynamic and the same position across other frames in the window also contains dynamic blocks, we apply mean pooling across those positions to overwrite B(,)) as follows.
retain its original (pre-downsampled) token
B ((,) PiSs EVED B
where D(s.)) is the set of frames within the window where block (i. is marked as dynamic. Otherwise remains unchanged. This process disrupts temporal cues by smoothing moving content while preserving static visual context details. Finally, the concatenation of the distorted key frames and the remaining frames constitutes the temporally distorted video representation V which participates in contrastive decoding defined in Eq. Technical details are available in Appendix B.

---

Video-
Video-
LLaMA-
ShareGPT4-
LLaVA-
Metries
VILA
PLLaVA
vVC
1SID
+TCD
Ours
ChatGPT
Chat2
VID
Video
Video
Match Rate
100.0
100.0
100.0
83.13
100.0
89.4
100.0
100.0
100.0
100.0
100.0
Entire
53.5
45.6
14.9
16.7
30.7
11.4
52.1
53.4
53.3
53.6
55.3
Interleave
68.9
79.8
58.6
98.9
93.7
65.4
66.8
75.6
Misleuding
83.3
81.4
21.6
22.6
43.1
6.8
82.5
82.8
82.9
83.3
84.2
Overall
65.0
65.5
47.2
37.9
66.0
49.1
63.5
66.5
66.4
67.2
72.1
Table 1: EventHallusion evaluation results. Our method notably outperforms baselines on all tasks. Best results are shown in bold.
4 Experiments
Benchmarks. We first evaluate models on cumporally-orinteer benchmarks, including TempCom- pass (21) and EventHallussin 332 The former targets temporal reasoning, while the latter assesses understanding of continuous actions. We then perform a comprehensive evaluation on general video
understanding benchmarks, including VideoMME [5] and MLVU (36).
Models. We choose representative and widely-used video LLMs, Video-LLaVA [5] and LLaVA- Mileo-7B-wwen2 [35), backbone. Based them, we apply our temporal distortion mechanism with contrastive decoding. In our experiments, we compare our method with popular Video LLMs as well as alternative contrastive decoding strategies, including VCD
[9] SID 7 and TCD 32)
Configuration. By default, we adopt frm for Video-LLLVA 15 and 32 frm for LLaVA- --Veood---- Qwen2 [35. We run all inferences on NVIDIA A6000 GPUs and A 100 GPUs. Detailed experiment configuration and hyperparameter settings are available Appendix
4.1 Results
Through the experiments, we fist aim to explore the following questions: (1) Can our video temporal distortion effectively induce poor responses, thereby serving as contrasting signal to improve temporal reasoning via contrastive decoding? (2) If so, can enhanced temporal perception also benefit general video understanding tasks? (3) Does strengthening temporal perception introduce a potential conflict with spatial understanding capabilities?
Across all four tasks and five Temporal Reasoning. Tab. shows results on TempCompass 21
temporally-oriented categories, our method consistently improves model performance when applied Video-LLaVA and LLaVA- LLaVA-Video-7B-O Video--B-----en.2 Compared to existing contrastive decoding methods such as VCD, our approach also demonstrate clear advantages. Similarly, on Tab. I our method yields notably better results all tasks over baselines.
Holistic Video Understanding. Our method shows clear advantages on temporally orienttd bench- marks, which primarily feature vidcos depicting short-term, continuous changes of an event object. In contrast, comprehensive video understanding benchmarks involve much longer vidcos and broader reasoning tasks across multiple aspects of video content. We evaluate LLaVA- ILaVA-Videe-77B 7B-Qwen2, the more powerful Video LMM, on Video-MME [5] and MLVU [36] for holistic video understand- ing. Results are shown Tab. 3 and Tab. 4 Our method also demonstrates clear advantages baselines, especially on cemporally-oriente
tasks, such as Temporal Perception and Reasoning in Video-MME ( and Action Count (AC) and Action Order (AO) in MLVU (36].
4.2 Analysis
Ablations. We also explore the contribution of each component in our method, as well as how hyperparameters affect model performance. Tab. (aa..5demm demonstrates the effecttivnness of each module in Temporal Distortion Unit, and the importance of sampling key frames with attention guidance. In Fig. 5 show how each hyperparameter in Contrastive Decoding (CD) and Temporal Distortion Unit (TDU) influences model performance on TempCompass with V--V-VVVo-7-BB Bween22 [35].
From Fig. 5)
we observe that moderate level of distortion is crucial for effective contrastive decoding. the ablation study of parameters most closely related to the degree of distortion such Frame Distortion Ratio, Token Distortion Ratio, Content Fusion Ratio, and again Frame Distortion
7

---

Model
Intern-
LLaVA (0
Long
VILAA
16
VID-
.VC
SID
.TCD
40ars
LLaVA
+VC
+SID
+TCD
4Ours
OneVision
Video
Multh Choice QA
Action
93.8
96.5
92.3
92.9
76.0
76.9
77.0
77.2
78.4
+24
95.6
96.4
96.4
96.3
96.7
+..
Direction
43.9
40.6
36.7
33.7
35.2
35.8
35.8
35.9
36.7
40.3
41.3
41.6
41.6
43.6
Speed
51.1
45.4
43.2
44.1
35.6
37.0
37.1
37.4
38.6
50.5
49.2
49.6
49.9
51.7 +12
Event Order
67.2
69.5
54.3
50.0
37.7
39.0
39.1
39.4
40.4
(27
71.2
71.8
71.2
69.9
72.5 (13
Autr: Chunge
59.9
56.9
52.4
60.0
40.9
42.0
42.2
42.1
43.8
71.5
72.5
72.7
72.8
75.3
Average
65.5
64.8
56.1
56.4
45.5
46.5
46.6
46.8
48.1
12
65.8
66.1
66.2
66.1
68.0
Action
84.8
86.0
86.2
84.8
74.3
75.2
75.4
75.4
76.1
87.7
88.4
88.7
88.9
90.4
VosNo Q4
Direction
53.2
55.3
50.4
52.2
51.8
52.5
52.4
52.7
53.8
54,3
54,9
54.6
55.1
56.7
Speed
57.4
53.1
54.3
50.2
51.2
51.4
53.3
13.1
58.1
58.9
59.8
59.4
(40
Event Orler
70.7
76.2
61.8
61.2
49.2
49.2
49.3
49.2
49.7
+0.5
67.2
66.4
66.5
66.5
67.4 +02
Attr. Change
63.0
59.1
54.5
61.3
51.1
52.0
51.9
52.1
53.2
63.6
63.7
63.8
63.7
65.8
|Average
68.2
69.7
62.1
63.6
56.3
57.1
57.1
57.2
58.3
66.8
67.4
67.5
67.4
69.4
Action
96.6
96.0
94.6
95.0
87.9
88.0
88.0
88.2
89.1
96.0
96.2
96.4
96.0
96.6
Matching
Direction
$9.9
56.9
54.4
58.7
53.8
53.8
53.9
54,
55.2
59.3
59.6
59.7
59.9
60.9
Speed
61.9
53.3
58.4
58.9
59.9
+1.5
61.6
107
Event Order
84.0
81.3
64.3
66.0
59.0
61.3
61.9
62.2
65.0
71.7
75.4
75.8
76.0
80.7
Attr. Change
77.1
73.8
62,5
65.3
58.3
59.8
59.7
59.4
63.5
71.5
73.9
74.0
74.2
77.1
|Average
77.1
73.8
65.7
68.9
63.3
64.1
64.3
64.3
66.4 -1.1
71.8
73.1
73.2
73.2
75.2
Action
84.6
79.3
75.8
74.7
50.8
51.4
51.5
51.7
53.3
85,5
86.1
85.9
85.7
87.9
Generation
Direction
38.8
30.7
35.3
36.2
28.7
29.2
29.0
29.3
30.3 116
39.8
40.4
40.6
40.7
41.6
Speed
25.3
32.2
31.7
23.2
24.3
+13
32.0
33.1
+5.1
Event Order
60.8
56.8
35,3
46.7
38.2
38.3
38.5
38.6
39.9
61.7
62.1
61.9
62.2
63.1
Attr: Change
60.2
57.4
45.8
47.1
33.6
33.9
34.1
33.9
35.6
V20
61.3
62.2
62,3
61.9
63.5
11.2
AAeeaaee
52.1
47.6
44.7
47.1
34.8
35.3
35.4
35.4
36.9
55.8
56.5
56.4
56.5
57.8
Arg. Categury
Direction
Action
53,2
84.8
86.0
55.3
44.2
86.4
85.9
45.3
42.4
71.4
42.8
72.8
42,7
72.9
43.0
73.1
44.0
74.2
+1.6
123
91,2
48.4
91.7
49.1
91.8
49.1
49.2
91.7
50.7 +3.3
93.0
Speed
57.4
45.8
47.7
41.9
42.8
+M4
50.5
50.7
50.9
Event Order
70.7
76.2
53.0
55.6
45.7
46.9
47.2
47.3
48.8
68.0
68.9
68.8
68.7
70.9
Attr: Change
63.0
59.1
53.3
58.0
45.7
46.9
46.9
46.8
49.0
67.0
68.1
68.2
68.1
70.4 134
Overall
66.0
64.2
56.9
58.8
49.8
50.4
50.5
50.6
52.1
.23
65.0
65.7
65.8
65.7
67.5
Table 2: Video temporal understanding evaluation on TempCompass. Best results are shown in bold.
Method
Temporal Perception
Temporal Reasoning
Short
Medium
Long
Overall
Video V.
15
45.3
38.0
36.2
39.9
LeaVA-NeXT Video--deo---BBB
18
40.0
29.4
48.9
42.0
35.6
42.1
Jama--3ViLLA155.0 1-BB 16
50.9
41.2
56.1
42.1
39,6
45.9
VILAI15-40B h+
60.0
40.7
72.0
61.2
53.8
62.3
IntemVL. Chat 21B
45.5
33.3
60.2
46.4
45.6
50.7
VongVA-7B 7B 33
58.2
37.3
61.1
50.4
46.2
$2.6
L.VVA-Video-7--Qeenn
35
54.3
73.5
51.1
59.8
-VD
55.3
73.9
55.2
50.8
60.0
SSID
75.3
55.4
74.0
55.1
50.6
59.9
TTD
32
75.9
55.8
73.6
55.1
50.9
59.8
OOurs
84.1
57.8
75.3
57.2
52.2
61.6
Table 3: Video-MME evaluation results. Our method enhances LLaVA-Video accuracy across various video durations, even outperforming VILA1.5-40B in temporal reasoning. Best results are shown in bold.
Ratio we find that setting the values too low results in limited improvements, while excessively high values, i.c., severe distortion, lead to relative decline in performance. This aligns with our carlier analysis: overly severe distortion tends to randomize the model's s responses, thereby undermining its role negative response to guide the generation in contrastive decoding. Only appropriately calibrated distortion can effectively induce negative responses, thereby enhancing performance via contrastive decoding.
Is There a Trade-off Between Temporal and Spatial Understanding? Intuitively, when humans watch a video and focus temporal changes in the scene, their attention to static details tends to decline. Interestingly, we observe a similar phenomenon Video L.Mss. On the MLVU benchmark, while our performance improves notably temporally related tasks such as AC and AO, it drops
8

---

Model
AC
ER
Needle QA
AO
Plot QA
AR
TR
Overall
Video CharGPT-7B
31.1
42.0
40.3
25.1
29.9
24.0
26.9
31.3
Video-LLaVA-IB
35.9
45.2
53.2
20.1
48.4
57.0
71.6
47.3
MA-LMM-7B
24.3
38.9
43.1
25.1
35.8
35.5
51.9
36.4
Llama-3-VILAI
0.0
24.7
32.4
27.0
VILAI.5-40 [6
11.7
35.8
38.3
56.4
84.7
46.2
InterVL. Chat V1,5-20B
13.3
24.5
40.0
14.3
42.0
51.3
80.2
37.9
LongVA 7B
25.2
48.6
70.4
41.7
68.1
58.5
82.2
56.4
Muna--iiee--B--weeee
35
41.8
68.5
76.3
57.9
75.1
67.8
84,5
67.4
VCD
42.8
68.5
77.0
60.1
75.1
65.1
82.8
67.3
SID
42.9
68.5
77.2
60.1
75.1
65.2
82.7
67.4
TCD 34
42.3
68.6
76.7
75.2
65.0
83.0
67.3
Ours
44.1
68.8
78.5
75.8
65.7
83.8
Table MLVU evaluation results. Our method achieves the best overall performance, with notable gains in temporal-relater aspects. TR: Topic Reasoning, AR: Anomaly Recognition, ER: Ego Reasoning, AO: Action Order, AC: Action Count. denotes temporal-related dimensions. Best results are in bold.
68.5
68.5
67.5
63.0
68.0
66.0
67.0
65.5
67.0
0.4
10
67.0
0.2
0.4
0.6
66.0
0.4
0.8
65.2
68.2
68.2
68.2
67.4
67.4
67.4
67.4
67.0
67.0
15
0.2
0.3
0.4
0.5
0.4
0.6
0.8
1.0
Window Size
Content Rusion Ratio
Block Size
Frame Perturbation Ratio
Figure 5: Sensitivity to Hyperparameter Settings TempCompass.
AR and TR. We further conduct an experiment TempCompass, where we progressively increase the Token Distortion Ratio and the Frame Distortion Ratio, respectively. As shown in the line plot, the accuracy of the sub-task "Attribute Change" keeps increasing, indicating improvement in temporal perception. However, the overall accuracy (represented by the bar chart) begins to decline when the distortion ratio is approximately 0.6, suggesting that other sub-tasks negatively affected.
Method
Tompuss
EventHallasion
67.5
Atlcation Guidancc
63.7
67.1
Frame Fusion
66.1
66.3
Key Distortion
Moving Content Discuption
66.9
69.7
Table 5: Ablation Study Results on Temporal Benchmarks: TempCompass and EventHallusion.
0.1
0.3
Distortion Ratio
0.6
0.9
Figure 6: Influence of Distortion
5 Related Work
are evolving rapidly, advancing image-text dialogue through fine-tuning pre-trained Large Language Models (LLM) with image features from additional visual encoders. Video LLMs extend MLLMs from image to video understanding by involving encoded video frames during training, such as Video- [34 To include stronger temporal information video representations, recent works 21 design additional temporal-aware encoders customize their own training data with time-stamp annotations. Similarly, Lict al. [1] also improve
Video Large Language Models. Multimodal Large Language Models (MLLM) 20 25 26
LLaVA 14), VILA-series 222 16 and -LLVV--ee---e

---

temporal reasoning with additional data but sourced from text. In contrast to existing works, our method enhances temporal reasoning in videos by incorporating a new temporal-awe aware decoding strategy to avoid time-insensitive responses at no additional training cost.
Temporal Understanding in Video LLMs. Temporal understanding is fundamental in Video LLMs. While existing popular benchmarks such as Video-MME [5] and MLVU (36] focus on the general evaluation of Video LLMs across diverse video categories and durations, TempCompass [2]] specifically focuses on the temporal reasoning ability of Video LLMs with variety of temporally focused tasks. EventHallusion 32 investigates Video LLMs' capability to understand continuous events. Our method not only achieves notable improvements on temporal understanding tasks, but also demonstrates promising results on general video benchmarks.
Contrastive Decoding. Contrastive Decoding (CD) 12 is a search-based LLM decoding approach that improves text generation by explicitly avoiding poor responses during decoding. Recent works [ 7 explore its application reducing hallucination MILMs by deliberately introducing distorted image inputs to elicit poor responses, which are then used to guide the model away from such errors. TCD [32] inherits similar idea to avoid video event hallucination by randomly dropping frames. Different from existing works, our work particularly focuses on improving temporal reasoning in Video LLMs. Extended discussions are available in Appendix E.
6 Limitations
When performing video distortion, our Temporal Distortion Unit relies solely on signals from the model itself--peccfficllll the attention maps extracted from the intermediate LLM layers-as guidance to estimate the importance of each visual token and each video frame. Compared to treating all frames equally and applying uniform random sampling, our approach represents a significant improvement. However, it is still not perfect. Attention maps do not always accurately reflect the true importance of each visual token, and relying on them often yields only coarse-grain grained results. To more precisely assess the importance of visual representations, future work may explore more guidance.
accurate and robust methods beyond
Moreover, our current study is limited to Video LLMs, with distortion applied only to the visual representations. In practice, many videos come with accompanying subtitles, and models often take both video and subtitle inputs. An interesting future direction would to distort both modali- ties applying not only visual distortion but also vidco-aware distortion subtitles. This would be challenging and different from the purely text-based distortion strategies employed in existing works on contrastive decoding for LLMs.
Conclusion
In this work, we investigated the challenges of temporal reasoning in Video Large Language Models (Video LLMs) and identified two key factors contributing to their failures-language prior and image prior. Building on these insights, we proposed video contrastive decoding with temporal distortion, simple yet effective method that enhances temporal coherence without requiring additional training or computational overhead. By intentionally introducing temporal distortions in key frames and contrastively optimizing against such failures, our method encourages models to maintain temporal consistency and avoid approach significantly improves both temporal-speeific and general video understanding benchmarks, showing strong effectiveness, generalizability, and scalability for improving temporal reasoning in multimodal large language models.
predictions. Extensive experiments demonstrate that our
Acknowledgment
The work is supported in part by the U.S. Office of Naval Research Award under Grant Number 4---164-4- the National Science Foundation under Grants IIS-2316300 and CNS-2330215, the National Institutes of Health (NTH) under Grant R01EB293388, and gifts from Adobe Research.
10

---

References
[1] Yukang Chen, Fuzhao Xue, Dacheng Li. Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang. Haotian Tang. Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin. Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos, 2024.
[2] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internll: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[3] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprini arXiv:2312.142388 2023.
[4] Alessandro Favero, Luca Zancato, Matthew Trager. Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. arXiv:2403.144 14003.
[5] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint 1xXix:2405.21075 2024.
[6] Bo He, Hengduo Li. Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Megeee-augmeeted large multimodal model for long-term video understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13504- 13514, 2024.
[7] Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision-language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. Published as a conference paper at ICLR 2025.
[8] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xianchun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint 1rXiv:2311.08046, 2023.
[9] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[10] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang. Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 033266 2024.
[11] Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang. Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
[12] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). Toronto, Canada, 2023. Main conference long paper.
[13] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors. Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 19944, Miami, Florida, USA. November 2024, Association for Computational Linguisties.
[14] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: learning united visual representation by alignment before projection. arXiv prepriat arXiv:2311.10122 2023.
[15] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint 1..iv:2311.10122 2023.
[16] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre- pre-training for visual language models. 2023. Preprint.
11

---

[17] Zhiqiu Lin, Xinyue Chen. Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Revisiting the role of language priors in vision-language models. In Proceedings of the 41xt International Conference on Machine Learning (ICML), 2024. arXiv:2306.01879
[18] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. 2024. Preprint.
[19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. 2023.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[21] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempeompass: Do vidco Ilms really understand videos? arXiv preprint arXiv:2403.00476 2024.
[22] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang. Yuming Lou, Shang Yang. Haocheng Xi, Shiyi Cao. Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang. Yukang Chen, Cheng- Yu Hsieh, De-An Huang. An-Chieh Cheng. Vishwesh Nath, Jinyi Hu. Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang. Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, and Yao Lu. Nvila: Efficient frontier visual language models, 2024.
[23] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt Towards detailed video understanding via large vision and language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). 2024.
[24] Ming Nie, Dan Ding. Chunwei Wang. Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang. Slowfocus: En- hancing fine-grained temporal understanding in video Ilm. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024. https: /ttp:bub htcphhub.commeeacooaemom..oeaeee com/ fudan- z
[25] Daiqing Qi, Handong Zhao, Jing Shi, Simon Jenni, Yifei Fan, Franck Dernnncourt Scott Cohen, and Sheng Li. The photographer's eye: Teaching multimodal large language models to and critique like photographers. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 24807-24816, June 2025.
[26] Daiging Qi, Handong Zhao, Zijun Wei, and Sheng Tag- grounddddd visual instruction tuning with retrieval augmentation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2008-2026, 2024.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[28] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint wwXiv:2312.0205.1 2023.
[29] Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanich Deilamsalchy, Bryan A. Plummer. Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13581-13591, 2024.
[30] Yolo Yunlong Tang, Jing Bi, Pinxin Liu, Zhenyu Pan, Zhangyun Tan, Qianxiang Shen. Jiani Liu, Hang Hua, Junjia Guo, Yunzhong Xiao, Chao Huang, Zhiyuan Wang, Susan Liang, Xinyi Liu, Yizhi Song, Yuhc Nie, Jia-Xing Zhong, Bozheng Li, Daiqing Qi, Ziyun Zeng, Ali Vosoughi, Luchuan Song, Zeliang Zhang, Daiki Shimada, Han Liu, Jicbo Luo, and Chenliang Xu. Video-Imm post-training; A deep dive into vidco reasoning with large multimodal models, 2025.
[31] Hang Zhang. Xin Li. and Lidong Bing. Video-llama: An instruction -uned audio-visual language model for video understanding. arXiv preprint WiXiv:2306.02855 2023.
[32] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnos- ing event hallucinations video llms. Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI). Association for the Advancement of Artificial Intelligence, 2025. To appear.
[33] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chumyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[34] Yuanhan Zhang, Bo Li. haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024.
12

---

[35] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[36] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, et al. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:4406.04D6 0426444444444 2024.
13

---

A Adaptive Plausibility Constraint
Contrastive Decoding in Video Large Language Models. Given a text query x and video input V, the model generates two output distributions: one conditioned on the original V and the other on the distorted video input V which is derived by applying pre-defined distortion (e.g., adding noise to visual features the simplest case) to V. Then, new contrastive probability distribution computed by leveraging the differences between two original distributions. The new contrastive distribution Peid is formulated as:
Pyta V, v
x) =softmax [(1 + (y) V.x) ologit, (y V V',x],
(3)
where larger a indicate a stronger amplification of the differences (a = 0 reduces to regular decoding).
Adaptive Plausibility Constraint. Eq. rewards texts favored by the response with original video inputs and penalizes texts favored by the response with distorted video inputs. However, the response with distorted video inputs is not always mistaken. Although video inputs are distorted, they may still preserve useful information, which can lead to correct answers. Therefore, penalizing all texts from response with distorted video inputs indiscriminately would penalize these correct answers, and conversely reward implausible answers. To tackle this issue, we follow Li et al. [12 to introduce the plausibility constraint.
Adaptive plausibility constraint contingent upon the confidence level associated with the output distribution with original video inputs:
Vhead (yend(y<))) (y<1) {yr V poly: V.x,y<t) 2 max (o V V.x.y<))
(4)
(y: V, V x) = 0, if y: & (y
(5)
where V is the output vocabulary of LVLMs and a hyperparameter in [0, for controlling the truncation of the next token distribution. Larger /3 indicates more aggressive truncation, keeping only high-probability tokens.
Combining the video contrastive decoding and the adaptive plausibility constraint, we obtain the full formulation:
y: softmax [(1 + a)) logit (y: V V,x,y<t)
lyg V x,,,,,)))
(6)
subject to yı E Veead (y<t).
B Technical Details of Video Temporal Distortion
B.1 Disrupting Moving Content in Remaining Frames
Bkddin Window
Figure 7: Disrupt moving content in remaining frames within a sliding window. dynamic blocks. Right: Fusion results of marked dynamics.
Left: Marked
Downsampling, Note that in the downsampling stage, i.e., when we downsample each frame from size (H. W) w W.), we do not really reduce token numbers. As illustrated in Fig. 7 (left), we actually replace all image tokens within one "downsampled" region with the mean the tokens included. For example, the value of each of the four token within B is the mean of the four (0.0)
tokens.
14

---

C Experiments
C.1 Experimental Configuration
In TempCompass 21,,
the following hyperparameters::
0.2, Widr 0.2,
Wtdr 0.4, Was 8, Wefr 0.3, Whs 3, Wipr = 0.5, Wmomentum = 0.8. In EventHallusion [32]. the following hyperparameters:
B 0.2, Widr 0.5, Wedr 0.5, Wws
Wefr = 0.3, Whs = 3, Wfpr 0.8, Wmomentum 0.8. In Video-MME [5) and MLVU [36]. we use
the following hyperparameters: a -,, 0.2, Widr
0.6, wadr
0.8, Wws 8, weft 0.3,
Wbe 3, Wipr = 0.8, Wmomentum = 0.5.
69.0
68.0
67.0
68.0
87.0
67.0
0.2
68.0
0.4
68.2
68.2
68.2
67.0
67.0
67.0
67.8
67.4
67.4
67.4
5
10
15
0.2
0.3
0.4
3
0.4
0.6
0.8
Window Size
Content Fusion Ratio
Block Size
Frome Perturbation Ratio
Figure 8: Sensitivity to Hyperparameter Settings TempCompass 21
C.2 Analysis
From Fig. S. we observe that a moderate level of distortion is crucial for effective contrastive decoding. In the ablation study of parameters most closely related the degree of distortion-
as Frame Distortion Ratio, Token Distortion Ratio, Content Fusion Ratio, and again Frame Distortion Ratio- find that setting the values too low results in limited improvements, while excessively high values, ie., severe distortion, lead to a relative decline in performance. This aligns with our earlier analysis: overly severe distortion tends randomize the model's responses, thereby undermining its role as negative response to guide the generation in contrastive decoding. Only appropriately calibrated distortion effectively induce negative responses, thereby enhancing performance via contrastive decoding.
D Extended Discussion
We are among the first to explore video temporal understanding from the perspective of language and image priors, and to enhance it using contrastive decoding with video temporal distortion.
We are among the first to explore video temporal understanding from the perspective of language and image priors, and to enhance it using contrastive decoding with video temporal distortion.
Recent works 9 have applied contrastive decoding to mitigate hallucinations image understand- ing with MILMS. For example, VCD [9] introduces random noise distort the original image, while SID (7) prunes important tokens based on attention guidance. TCD 32 alleviates event hallucination videos by randomly dropping frames.
SID adopts a similar strategy to estimate token importance and removes the most important tokens this is conceptually similar to the second step of video temporal distortion. However, there are notable differences in how attention maps are utilized and how the pruning is applied. Specifically. SID attention maps from the k-th layer to assess token importance and then prunes the most important tokens starting from the (I: + 1)-th layer.
In contrast, approach aggregates attention maps from all layers, from shallow to deep, compute a more accurate importance score. Furthermore, while SID performs pruning from intermediate layers, input the distorted video representations directly the first layer, ensuring that the dropped
15

---

information is effectively masked from the very beginning. This design allows our method to better mask information that should be dropped.
E
Efficiency
Due to the nature of Contrastive Decoding (which requires two forward passes), CD-based methods (e.g., VCD and ours) are inevitably slower than the vanilla model generation speed.
In practical scenarios, it's difficult to optimize performance, time efficiency, and memory efficiency all once. Different applications prioritize different aspects. CD-based methods are less tfeeefffiieet,, but they offer better performance without significant increase in memory usage. This makes them well-suited for applications where content quality critical, such as education and medical assistance, where accuracy really matters and errors come at a high cost.
Time-Effier Efficient Implementation. In practice, can optimize the code to reduce the time by nearly half with no impact on the results, making as efficient as the vanilla Video LLM.
The implementation is simple: parallelize the two forward passes during next-token generation. Previously, is implemented in the def sample( () function as follows:
outputs self forward(...) outputs_cd = self forward(...)
inference with raw video inference with distorted video
calculate final logits with outputs and outputs_cd:
Now, with just a few extra lines, we use torch, cuda. Stream() to run both forward passes in parallel:
stream_maie maim torch. cuda. Stream() . stream_ cd torch. cuda. Stream() outputs holder 0
outputs_cd_holcee
0
submit main inference
with torch. cuda. www.setstteemmmmaiiiiii
outputs_holders ['main'] self forward(...)
submit contrastive inference
with torch. cuda. wtreatttttttstttasssmmmam
outputs_cd_hollee ['cd]] foraaddr
cuda. synchroniccch)e
outputs = cetputs_holdders ["mmin']
outputs_cd
cd]]
calculate final logits with outputs and outputs_cd:
With this implementation. our method can be as fast as the vanilla Video LLM with the same average GPU memory usage. Note that the peak GPU memory usage will be larger.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: they are in the abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix
17

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: no thcoretical result
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: included in appendix
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA].
Justification: publish when accepted
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guudeeee http://oeeubiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: in paper and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA].
Justification: doesnt apply
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: in paper and appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wurpp..coo.oooo.ccccaaaa
Answer: [Yes]
Justification: we followed it
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: yes
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: no new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing
22

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not705 involve LLMs as any important, original, or non-standard components.700 706
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp::://///i///// aiperrips.oo cc/ //onfferencee/22255555 MLMMLLL for what should or should be described.
23


Reviews:
[{"review_id": "2ewfhsAI6n", "reviewer": "Reviewer_7wGw", "summary": "This work attempts to improve the temporal understanding and reasoning abilities of modern Video Large Language Models (VLLMs). The authors first improve understanding of the ways in which Video LLMs fail by looking at the language and image priors which can dominate VLLM QA abilities due to the large amount of text and image data that is seen in pretraining. In order to fix these issues, the authors propose a modification tot he decoding phase of VLLMs (thus no additional training) to improve the temporal understanding capabilities of these models. Specifically, the authors propose a method to distort temporal consistency of key frames during decoding. These distortions produce time-insensitive incorrect model responses which can then be avoided in generating the final output via a contrastive decoding technique. The authors show this method improves temporal-specific and video understanding benchmark performance across many evaluations.", "questions": "See questions / concerns above in the Weaknesses section", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. This work is relatively comprehensive in it’s understanding of how to distort videos in the best way to create effective negative examples during contrastive decoding. Unlike standard methods like just adding noise, random shuffling or dropping, they come up with a novel way to identify distortions that remove temporal information but still maintain salient spatial features of important frames. This is to my knowledge quite novel. \n2. The writing is clear and the main ideas are well communicated\n3. The authors evaluate on a reasonable set of benchmarks and use multiple video models to show the results generalize across at least 2. \n4. The results compared to other contrastive decoding methods look to be significant. \n5. The atuhors show that even on MLVU, on average the gains on temporal understanding outweigh slight losses in spatial understanding \n\nWeaknesses:\n1. There are quite a lot of hyperparameters here all shown in Figure 5. While the performance is somewhat stable across these, there are clear “best” choices and I don’t know with this many hyper params, how easy it would be for this method to generalize truly to all kinds of video LLMs or benchmark videos. \n2. The authors could test on more benchmarks that are known to test failures in temporal perception like PerceptionTest and others. This woudl just strengthen the generality of the result. \n3. From a practical perspective, the authors don’t address the extra inference compute needed to do this distortion generation and contrastive decoding at test-time. How does this practically impact the deployment of such a method for models as we scale the size of the LLMs. \n4. There is no commentary I can see (unless I missed it) on statistical significance of the results. In particular how can we know that the gains from this method are statistically significant compared with the other contrastive decoding methods. For example, in the case where the distortion is just dropping random frames, could you run this multiple times to get a standard devation to compare?", "was_revised": true, "final_justification": "I maintain my score of 5 (accept) as I believe this work is a valuable for improving our understanding of video understanding in VLMs and specifically demonstrates how to construct contrastive decoding examples for targeting temporal understanding in a relatively general way. I feel aligned with the authors that the other reviewer concerns are not sufficient grounds to reject the paper. Specifically, the other main concerns: increased inference time compute, and lack of novelty can be addressed by understanding this work in the context of the field of contrastive decoding and test-time compute for VLMs. It should not be held against the authors that contrastive decoding adds inference compute (this is a feature of all contrastive decoding methods and they do address this and show that their method strongly outperforms generic contrastive decoding). In addition, reviewers felt this work does not have strong novelty because it reuses visual contrastive decoding (which has already been introduced). However, like the authors say, the key aspect of contrastive decoding comes down to how you design the contrastive examples, and they clearly show that the prior generic visual contrastive decoding approaches to not give any benefits in temporal understanding tasks. On the other hand, their careful study of how to construct these examples gives insights into the features that VLMs use for video understanding and how we might bias them to focus on visual temporal features in QA tasks. \n\nOverall, I think the work is well presented, has decently strong results and introduces a simple but effective method for improving the temporal understanding in VLMs without extra training. I feel this is enough to warrant acceptance.", "rebuttal": "Thank you for your review and suggestions!\n\n## Weakness 1\n\nThanks for the valuable advice! Different distortion ratios do have an impact on model performance. With slight adjustments based on the task, the model can achieve best results. Intuitively, different videos or tasks have varying levels of information density, so using different distortion ratios allows better adaptation to specific scenarios. However, significant per-task tuning is not necessary to achieve improvements. As long as the ratio is not set to an extreme (i.e., very close to 0 or 1), simply choosing a moderate value around 0.5 already leads to consistent gains across different tasks. We show such setting (Ratio=0.5) on LLaVA-Video-Qwen2-7B on different tasks in Tab. 1.\n\nFor other parameters such as window size, block size, alpha, and beta, we use the **same** values across **all tasks** and **models** without any per-task or model tuning.\n\n|                | Vanilla Model | Best Setting | Ratio=0.5 |\n| -------------- | ------------- | ------------ | --------- |\n| EventHallusion | 63.5          | 72.1         | 71.3      |\n| TempCompass    | 65.0          | 67.5         | 66.7      |\n| MVBench        | 58.5          | 59.7         | 59.2      |\n| Video-MME      | 59.8          | 61.6         | 61.1      |\n\n(Tab. 1. Ratio Setting)\n\n\n\n## Weakness 2\n\nThank you for the suggestion! To strengthen the generalizability of our results, we evaluated on a broader set of benchmarks (EgoSchema, LVBench, MVBench, and a multiple-choice subset of PerceptionTest-Valid), and included more recent models from the Qwen2.5-VL family and models at different sizes. As shown in Tab. 2, our method consistently outperforms baselines, demonstrating its generalizability and scalability.\n\n| Model                 |          | Video-MME | MLVU     | EgoSchema | LVBench  | MVBench  |\n| --------------------- | -------- | --------- | -------- | --------- | -------- | -------- |\n| LLaVA-Video-7B-Qwen2  | Vanilla  | 59.8      | 67.4     | 57.3      | 58.2     | 58.5     |\n|                       | VCD      | 60.0      | 67.3     | 57.3      | 58.3     | 58.7     |\n|                       | **Ours**     | **61.6**  | **68.5** | **58.6**  | **59.1** | **59.7** |\n| LLaVA-Video-72B-Qwen2 | Vanilla  | 67.5      | 71.2     | 65.6      | 61.9     | 64.1     |\n|                       | VCD      | 67.6      | 72.2     | 65.7      | 61.9     | 64.3     |\n|                       | **Ours**     | **68.8**  | **73.1** | **66.9**  | **62.7** | **65.2** |\n| Qwen2.5-VL-7B         | Vanilla  | 65.1      | 70.2     | 65.0      | 45.3     | 69.6     |\n|                       | VCD      | 65.3      | 70.3     | 65.2      | 45.3     | 69.7     |\n|                       | **Ours** | **66.7**  | **71.2** | **66.5**  | **46.1** | **70.6** |\n| Qwen2.5-VL-72B        | Vanilla  | 73.3      | 74.6     | 76.2      | 47.3     | 70.4     |\n|                       | VCD      | 73.3      | 74.6     | 76.3      | 47.2     | 70.6     |\n|                       | **Ours** | **74.5**  | **75.5** | **77.2**  | **48.0** | **71.3** |\n\n(Tab. 2. More evaluation with different benchmarks and models)\n\nDifferent from common recent Video LLM benchmarks such as EgoSchema, LVBench, and MVBench, PerceptionTest includes a broader set of tasks, including tracking and localization, which some Video LLMs cannot directly handle. To maintain consistency with prior benchmarks, we selected the multiple-choice question subset of PerceptionTest for evaluation. Results from Tab. 3 show our model also demonstrates clear advantages.\n\n|                             | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 + VCD | LLaVA-Video-7B-Qwen2 + Ours |\n| --------------------------- | -------------------- | -------------------------- | --------------------------- |\n| PerceptionTest MCQ (Valid ) | 60.18                | 60.57                      | **62.33**                   |\n\n(Tab. 3. Evaluation on PerceptionTest)\n\n\n\n## Weakness 3\n\nThanks for the meaningful advice! Due to the nature of Contrastive Decoding (which requires two forward passes), CD-based methods (e.g., VCD and ours) are inevitably slower than the vanilla model in generation speed.\n\nIn practical scenarios, it's difficult to optimize performance, time efficiency, and memory efficiency all at once. Different applications prioritize different aspects. CD-based methods are less time-efficient, but they offer better performance without a significant increase in **memory usage**. This makes them well-suited for applications where content quality is critical, such as education and medical assistance, where accuracy really matters and errors come at a high cost.\n\nFrom Tab. 4, we observed that the time and memory efficiency of our TDU+CD is similar to VCD. CD based models (ours and VCD) are inevitably slower than the vanilla model due to CD'2 nature, but the additional memory usage is efficent, especially when model size grows.\n\n|                                      | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 |\n| ------------------------------------ | -------------------- | -------------------- | -------------------- | --------------------- | --------------------- | --------------------- |\n| **Metric**                           | Vanilla              | + VCD                | + Ours               | Vanilla               | + VCD                 | + Ours                |\n| Avg Inference Latency per sample (s) | 1.026                | 1.670                | 1.675                | 5.315                 | 9.162                 | 9.165                 |\n| Avg Tokens per sec (tokens/s)        | 3.52                 | 2.107                | 2.101                | 0.679                 | 0.394                 | 0.394                 |\n| Avg GPU Memory Usage (MiB)           | 41418                | 43986                | 43988                | 193261                | 199590                | 199596                |\n| Total GPU Hours (h)                  | 1.144                | 1.862                | 1.867                | 6.034                 | 10.223                | 10.227                |\n\n(Tab. 4. Inference Statistics on A100 80GB on TempCompass)\n\nFor CD, its time efficiency might be improved through techniques such as leveraging the KV cache in some way, token pruning, or early stopping in the first forward pass, etc. This presents an interesting future research direction. However, our focus is not on optimizing the speed of CD algorithms, but rather on designing CD-based methods that effectively enhance temporal reasoning. Our guiding principle is to ensure that **our approach does not show inference performance degradation compared to common CD based methods like VCD**, rather than aiming to match the efficiency of the vanilla model. It is a separate and challenging reseach area different from our goal of temporal resoning.\n\n\n\n## Weakness 4\n\nThank you for advice. The results we reported in the paper are the average over three runs. \n\nHere we included results with standard deviations for reference:\n\n| Method                             | TempCompass | EventHallusion |\n| ---------------------------------- | ----------- | -------------- |\n| **Ours** (Full Method)             | 67.5 ± 0.21 | 72.1 ± 0.45    |\n| w/o **Attention Guidance**         | 65.7 ± 0.34 | 67.1 ± 0.71    |\n| w/o  **Frame Fusion**              | 66.1 ± 0.39 | 66.3 ± 0.73    |\n| w/o  **Token Distortion**          | 66.5 ± 0.41 | 68.3 ± 0.75    |\n| w/o  **Moving Content Disruption** | 66.9 ± 0.37 | 69.7 ± 0.69    |\n| VCD                                | 65.7 ± 0.49 | 66.5 ± 0.77    |\n| TCD                                | 65.7 ± 0.45 | 67.2 ± 0.79    |\n\n(Tab. 5. Results with standard deviations)\n\nWe sincerely thank you again for your review. Your comments are very helpful. We hope our responses can address your concerns. If you have any further questions or unresolved issues, we are glad to have further discussions!"}, {"review_id": "TEUERwiHOc", "reviewer": "Reviewer_mNF7", "summary": "This paper proposes a decoding method to improve temporal reasoning in Video LLMs by leveraging contrastive decoding with video temporal distortion. The core idea is to deliberately distort temporal consistency in key frames during decoding, which induces temporally insensitive wrong responses (e.g., based on text/image priors) from the model that can then be contrastively avoided during generation, guiding the model toward temporally grounded outputs. The method is applied to existing Video LLMs (e.g., LLaVA-Video-7B-Qwen2, Video-LLaVA), demonstrating consistent improvements on temporally-focused benchmarks (TempCompass, EventHallusion) and general video understanding benchmarks (Video-MME, MLVU), outperforming other contrastive decoding methods.", "questions": "Please check the questions in the weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a simple and effective approach for improving temporal reasoning in Video LLMs without retraining, using contrastive decoding with targeted temporal distortion.\n\n2. The proposed approach consistently improves performance on multiple temporal reasoning and general video understanding benchmarks, outperforming other contrastive decoding baselines.\n\nWeaknesses:\n\n1. While the identified “language prior” and “image prior” sound intuitively reasonable, the paper lacks empirical evidence to quantify how much these biases actually affect the model’s temporal reasoning performance. Providing concrete measurements would strengthen the motivation for the proposed approach.\n\n2. Building on the previous point, while the paper proposes targeted temporal distortion methods motivated by these prior biases, the methods appear very heuristic and may heavily depend on the specific data distributions of the evaluated datasets. This raises concerns about the scalability and generalizability of the approach across different domains and video types.\n\n3. Therefore, I would like to see the model’s performance on additional commonly used video understanding benchmarks, such as LVBench, EgoSchema, and MVBench, to better assess the generalization capability of the proposed method.", "was_revised": true, "final_justification": "After reading the authors' responses, most of my concerns are addressed. I decided to increase my score.", "rebuttal": "Thank you for your review and suggestions!\n\n## Weakness 1\n\nThanks for the valuable advice! In fact, we provide a simple quantitative study in Sec. 2 (L91) to illustrate the impact of language priors. In this experiment, we analyze a randomly sampled subset of questions that Video-LLaVA and LLaVA-Video-7B-Qwen2 answered incorrectly on TempCompass, and measure how many of these were also answered incorrectly in a blind evaluation (where only the text questions are given without the video). This experiment highlights the correlation between language priors and temporal reasoning errors. We copied the results in Tab. 1.\n\nResults show that, among the incorrect predictions made by LLaVA-Video-7B-Qwen2 and Video-LLaVA, 46.7% and 38.9% respectively matched the answers produced by blind LLM,  significantly higher than random chance. It shows the correlation between language prior and temporal reasoning mistakes. In these cases, Video LLMs failed to ground their responses in videos and instead relied on language priors, leading to incorrect answers.\n\n|             | Video-LLaVA | LLaVA-Video-7B-Qwen2 |\n| ----------- | ----------- | -------------------- |\n| TempCompass | 46.7        | 38.9                 |\n\n(Tab. 1. Study on language prior)\n\nFor image priors, we conducted a similar experiment. The difference is that we analyzed how many of the questions that Video-LLaVA and LLaVA-Video-7B-Qwen2 got wrong on TempCompass were also answered incorrectly in an image-only evaluation (where the model is given only the text questions and a randomly selected frame from the video, without access to the full video). This experiment aims to reveal the correlation between image priors and temporal reasoning errors.\n\nSimilar to the results of the language prior experiment, the matching rates of 48.3% and 40.4% are significantly higher than random chance, indicating a strong correlation between image priors and temporal mistakes.\n\n|             | Video-LLaVA | LLaVA-Video-7B-Qwen2 |\n| ----------- | ----------- | -------------------- |\n| TempCompass | 48.3        | 40.4                 |\n\n(Tab. 2. Study on image prior)\n\nTogether, these two experiments demonstrate a clear connection between language and image priors and temporal reasoning errors. This is an interesting research direction, and constructing a new dedicated benchmark may be required to perform a comprehensive and systematic analysis on it. However, our focus is not entirely on this aspect. Instead, we also aim to motivate from these observations and analyses to propose our methodology that can improve video temporal understanding.\n\n\n\n## Weakness 2\n\nThanks for the insightful question.\n\n> ... the methods appear very heuristic and may heavily depend on the specific data distributions of the evaluated datasets ...  This raises concerns about the scalability and generalizability of the approach across different domains and video types.\n\nIn fact, generalizability and scalability are strengths of our approach. In essence, Contrastive Decoding (CD) works by generating bad answers and guiding the model to steer away from them to produce better results. In our case, we create bad answers by disrupting temporal information—specifically, by weakening temporal cues and emphasizing textual and static visual information in videos. This inherently ensures generalizability and scalability, as temporal information is one of the most fundamental and pervasive elements in video content. By disturbing it, we can consistently impair the model’s understanding across different types of videos and produce bad answers. \n\nSimilarly, we added a study on the correlation between priors and reasoning mistakes on the MCQ subset of PerceptionTest that require the understanding of memory, abstract patterns, physics, and semantics etc. On PerceptionTest, we also calculated the matching rate (both language and image) that is proposed *weakness 1*. We can still see a clear correlation between language and image priors and mistakes.\n\n| PerceptionTest                 | Video-LLaVA | LLaVA-Video-7B-Qwen2 |\n| ------------------------------ | ----------- | -------------------- |\n| Matching Rate (language prior) | 43.1        | 37.5                 |\n| Matching Rate (image prior)    | 42.6        | 36.4                 |\n\n(Tab. 3. Matching Rate Results on PerceptionTest)\n\n|                            | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 + VCD | LLaVA-Video-7B-Qwen2 + Ours |\n| -------------------------- | -------------------- | -------------------------- | --------------------------- |\n| PerceptionTest MCQ (Valid) | 60.18                | 60.57                      | **62.33**                   |\n\n(Tab. 4. Evaluation on PerceptionTest)\n\nFollowing your suggestion, we also performed evaluation on more video understanding benchmarks.\n\n## Weakness 3\n\nTab. 1, 2 and 3 quantitatively demonstrate the correlation between priors and temporal reasoning errors. In addition, following your suggestion, we incorporated commonly used benchmarks such as **LVBench**, **EgoSchema**, and **MVBench**. We also evaluated our method using larger 72B models and the most recent state-of-the-art image/video LLMs, Qwen2.5-VL. Our method consistently maintains a performance advantage on Tab. 5.\n\n| Model                 |          | Video-MME | MLVU     | EgoSchema | LVBench  | MVBench  |\n| --------------------- | -------- | --------- | -------- | --------- | -------- | -------- |\n| LLaVA-Video-7B-Qwen2  | Vanilla  | 59.8      | 67.4     | 57.3      | 58.2     | 58.5     |\n|                       | VCD      | 60.0      | 67.3     | 57.3      | 58.3     | 58.7     |\n|                       | Ours     | **61.6**  | **68.5** | **58.6**  | **59.1** | **59.7** |\n| LLaVA-Video-72B-Qwen2 | Vanilla  | 67.5      | 71.2     | 65.6      | 61.9     | 64.1     |\n|                       | VCD      | 67.6      | 72.2     | 65.7      | 61.9     | 64.3     |\n|                       | Ours     | **68.8**  | **73.1** | **66.9**  | **62.7** | **65.2** |\n| Qwen2.5-VL-7B         | Vanilla  | 65.1      | 70.2     | 65.0      | 45.3     | 69.6     |\n|                       | VCD      | 65.3      | 70.3     | 65.2      | 45.3     | 69.7     |\n|                       | **Ours** | **66.7**  | **71.2** | **66.5**  | **46.1** | **70.6** |\n| Qwen2.5-VL-72B        | Vanilla  | 73.3      | 74.6     | 76.2      | 47.3     | 70.4     |\n|                       | VCD      | 73.3      | 74.6     | 76.3      | 47.2     | 70.6     |\n|                       | **Ours** | **74.5**  | **75.5** | **77.2**  | **48.0** | **71.3** |\n\n(Tab. 5. More evaluation with different benchmarks and models)\n\nWe sincerely thank you again for your review. Your advice are very helpful to us. We hope our responses can address your concerns. If you have any further questions or unresolved issues, we look forward to have further discussions!"}, {"review_id": "RS9D4koCEM", "reviewer": "Reviewer_H8xN", "summary": "This paper proposes a training-free method to improve temporal reasoning in Video Large Language Models (VLLMs). By intentionally distorting the temporal order of key frames during decoding and contrastively avoiding the resulting incorrect outputs, the model is guided toward more temporally coherent predictions.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is architecture-agnostic and can be applied to multiple video prediction models, showing improvements across diverse datasets and backbones.\n2. The proposed method yields consistent improvements across both temporal-specific and general video understanding benchmarks.\n\nWeaknesses:\n1. **Inference Overhead Not Addressed**:\nAlthough the method is training-free, it introduces additional computational cost during inference. A quantitative comparison of inference time with other contrastive decoding methods would better justify its efficiency.\n\n2. **Missing Failure Case Analysis**:\nThe paper does not present any failure cases or qualitative error analysis. Including such examples would help clarify the method’s limitations and guide future improvements.\n\n3. **Limited Baseline Comparisons**:\nThe experimental evaluation lacks comparisons with more recent MLLM baselines, such as InternVL3. Including stronger and more up-to-date models would provide a more comprehensive assessment of the method’s effectiveness.", "was_revised": true, "final_justification": "After reading the authors' responses, most of my concerns are addressed. I decided to maintain my positive score.", "rebuttal": "Thank you for your review and suggestions!\n\n## Weakness 1\n\nThanks for the meaningful advice! Due to the nature of Contrastive Decoding (which requires two forward passes), CD-based methods (e.g., VCD and ours) are inevitably slower than the vanilla model in generation speed.\n\nIn practical scenarios, it's difficult to optimize performance, time efficiency, and memory efficiency all at once. Different applications prioritize different aspects. CD-based methods are less time-efficient, but they offer better performance without a significant increase in memory usage. This makes them well-suited for applications where content quality is critical, such as education and medical assistance, where accuracy really matters and errors come at a high cost.\n\nFrom Tab. 1, we observed that the time and memory efficiency of our TDU+CD is similar to VCD. CD based models (ours and VCD) are inevitably slower than the vanilla model due to CD's nature, but the additional memory usage is efficent, especially when model size grows.\n\n|                                      | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 |\n| ------------------------------------ | -------------------- | -------------------- | -------------------- | --------------------- | --------------------- | --------------------- |\n| **Metric**                           | Vanilla              | + VCD                | + Ours               | Vanilla               | + VCD                 | + Ours                |\n| Avg Inference Latency per sample (s) | 1.026                | 1.670                | 1.675                | 5.315                 | 9.162                 | 9.165                 |\n| Avg Tokens per sec (tokens/s)        | 3.52                 | 2.107                | 2.101                | 0.679                 | 0.394                 | 0.394                 |\n| Avg GPU Memory Usage (MiB)           | 41418                | 43986                | 43988                | 193261                | 199590                | 199596                |\n| Total GPU Hours (h)                  | 1.144                | 1.862                | 1.867                | 6.034                 | 10.223                | 10.227                |\n\n(Tab. 1. Inference Statistics on A100 80GB on TempCompass)\n\nFor CD, its time efficiency might be improved through techniques such as leveraging the KV cache in some way, token pruning, or early stopping in the first forward pass, etc. This presents an interesting future research direction. However, our focus is not on optimizing the speed of CD algorithms, but rather on designing CD-based methods that effectively enhance temporal reasoning. Our guiding principle is to ensure that **our approach does not show inference performance degradation compared to common CD based methods like VCD**, rather than aiming to match the efficiency of the vanilla model. It is a separate and challenging reseach area different from our goal of temporal resoning.\n\n\n\n## Weakness 2\n\nThanks for the suggestions! In Appendix D (Limitations), we have discussed the possible weaknesses and failures of our method, as well as possible future directions. \n\nOur method tends to show relatively degraded performance when the input video is highly complex or extremely long. A key underlying reason is that attention maps do not always accurately capture the true importance of each frame and visual token, especially when the input videos are intricate as mentioned above. In such cases, their reliability further deteriorates, and relying on them often leads to only coarse-grained results. \n\nWe will include more examples in the final version to better illustrate the analysis!\n\n\n\n## Weakness 3\n\nThanks for your advice! We added results of: InternVL3 (8B, 38B), QwenVL2.5 (7B, 72B), and LLaVA-Video-72B-Qwen2 for evaluation. Tab. 2 shows our method also improves recent MLLMs such as InternVL3 and Qwen2.5-VL, and can scale to larger models (38B, 72B) on different benchmarks.\n\n| Model                 |      |            |      | Video-MME | MLVU     | LVBench  | MVBench  |\n| --------------------- | ---- | ---------- | ---- | --------- | -------- | -------- | -------- |\n| LLaVA-Video-7B-Qwen2  |      | Vanilla    |      | 59.8      | 67.4     | 58.2     | 58.5     |\n| LLaVA-Video-7B-Qwen2  |      | + VCD      |      | 60.0      | 67.3     | 58.3     | 58.7     |\n| LLaVA-Video-7B-Qwen2  |      | **+ Ours** |      | **61.6**  | **68.5** | **59.1** | **59.7** |\n| LLaVA-Video-72B-Qwen2 |      | Vanilla    |      | 67.5      | 71.2     | 61.9     | 64.1     |\n| LLaVA-Video-72B-Qwen2 |      | + VCD      |      | 67.6      | 72.2     | 61.9     | 64.3     |\n| LLaVA-Video-72B-Qwen2 |      | **+ Ours** |      | **68.8**  | **73.1** | **62.7** | **65.2** |\n| Qwen2.5-VL-7B         |      | Vanilla    |      | 65.1      | 70.2     | 45.3     | 69.6     |\n| Qwen2.5-VL-7B         |      | + VCD      |      | 65.3      | 70.3     | 45.3     | 69.7     |\n| Qwen2.5-VL-7B         |      | **+ Ours** |      | **66.7**  | **71.2** | **46.1** | **70.6** |\n| Qwen2.5-VL-72B        |      | Vanilla    |      | 73.3      | 74.6     | 47.3     | 70.4     |\n| Qwen2.5-VL-72B        |      | + VCD      |      | 73.3      | 74.6     | 47.2     | 70.6     |\n| Qwen2.5-VL-72B        |      | **+ Ours** |      | **74.5**  | **75.5** | **48.0** | **71.3** |\n| InternVL3-8B          |      | Vanilla    |      | 66.3      | 71.4     | 58.8     | 75.4     |\n| InternVL3-8B          |      | + VCD      |      | 66.4      | 71.4     | 58.8     | 75.5     |\n| InternVL3-8B          |      | **+ Ours** |      | **68.1**  | **72.5** | **59.6** | **76.7** |\n| InternVL3-38B         |      | Vanilla    |      | 72.7      | 77.8     | 67.3     | 76.9     |\n| InternVL3-38B         |      | + VCD      |      | 72.7      | 77.7     | 67.2     | 77.0     |\n| InternVL3-38B         |      | **+ Ours** |      | **73.8**  | **78.7** | **68.1** | **78.1** |\n\n(Tab. 2. More evaluation with different benchmarks and models)\n\nWe sincerely thank you again for your review. Your comments are very helpful. We hope our responses can address your concerns. If you have any further questions or unresolved issues, we are glad to have further discussions!"}, {"review_id": "aKOo0e3pF7", "reviewer": "Reviewer_SAQe", "summary": "This paper proposes a plug-in, model-agnostic method to enhance the temporal reasoning capabilities of Video Large Language Models at inference time. Building on Contrastive Decoding, the authors introduce a Temporal Distortion Unit that adaptively corrupts key frames or tokens, guided by intermediate LLM attention map, to generate a “bad” video embedding. By contrasting the original logits with those from the distorted input, the decoding process down-weights temporally insensitive errors and boosts correct predictions \n. The approach is evaluated on temporally focused benchmarks (TempCompass, EventHallusion) and broad video benchmarks (Video-MME, MLVU), demonstrating consistent gains over baseline Video-LLaVA and LLaVA-Video-7B-Qwen2 models .", "questions": "1. See weakness section. \n\n2. Can you provide inference latency and GPU usage metrics to quantify the runtime overhead of TDU + CD compared to vanilla decoding?\n\n3.  Fig. 5 shows performance fluctuates as distortion ratios (wfdr, wtdr, wcfr, etc.) and α vary. Does the proposed method requires need significant per-task tuning?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. Model-agnostic post-hoc correction requires no additional training, making it easy to apply to any Video LLM.\n\n2. Outperforms prior contrastive methods (VCD, SID, TCD) by 1–3 pts on temporal tasks (e.g., Action Count, Action Order in MLVU) \n\n3. Component-level analysis (attention guidance, key-frame fusion, token distortion, content disruption) shows each module’s contribution, and Fig. 5 illustrates sensitivity to hyperparameters\n\n**Weakness**\n\n1. The method largely follows the existing visual CD framework; the only substantive innovation is the specific design of the TDU. Beyond repurposing CD for video, there is limited new insight.\n\n2. Experiments are restricted to two 7 B Video LLMs (Video-LLaVA and LLaVA-Video-7B-Qwen2). Those models are relatively old. No evaluation on other sota Video LLMs such as internvl3 or qwenvl 2.5\n\n3. I am not sure if this method will scale to larger models such as 30B. There should not be high barrier from trying larger models (30B) nowadays given the inference-only nature of the proposed method and the advancement of highly optimized inference libraries (sglang & vllm)\n\n4. Contrastive decoding with TDU requires at least two forward passes per token and additional distortion computation. The paper reports accuracy gains but omits any runtime, memory, or GPU-hour comparisons; a 2–3 % boost may not justify the extra cost in real-time settings. For example, a larger models (14B) often yield ≥ 2 % improvements out-of-the-box compared to a 7B model.", "was_revised": true, "final_justification": "As written in my response to the authors rebuttal.  I find this paper has significant limitations and little potential impact. I would like to keep my rating.", "rebuttal": "Thank you for your review and suggestions!\n\n## Weakness 1\n\nOur substantive innovation is not only the TDU, but also our unique analysis of temporal understanding with CD. Specifically:\n\n1. An investigation into the distinct factors, language and image priors, that affect temporal reasoning in videos **(Sec. 2)**; and\n2. Three critical conditions when applying CD to effectively counteract biases introduced by these priors **(Sec. 3.2, L155–L182)**.\n\nSubstantially, most recent existing (visual or textual) CD approaches share the same principle: *distort inputs in a way to induce bad outputs as negative targets*. The novelty and interesting part is not the principle itself but lies in analyzing what factors are critical for reasoning in different tasks **(Sec. 2)**, and how to manipulate them effectively **(Sec. 3.2)**.\n\n**Our analyses on the unique aspects of temporal understanding with CD have not been covered by existing visual CD works.** It is the unique observations and analyses that led us to propose TDU, which consistently outperforms existing visual CD methods on temporal reasoning. Therefore, our work is not just a simple repurposing CD for video, but a targeted study of the unique challenges in temporal understanding and how to correctly and effectively address them with TDU.\n\n\n\n## Weakness 2, Weakness 3\n\nThanks for your advice! We added results of: InternVL3 (8B, 38B), QwenVL2.5 (7B, 72B), and LLaVA-Video-72B-Qwen2, with the same setting as LLaVA-Video-Qwen2 for evaluation. Tab. 1 shows our method also improves recent MLLMs such as InternVL3 and Qwen2.5-VL, and can scale to larger models (38B, 72B) on different benchmarks.\n\n| Model                 |      |            |      | Video-MME | MLVU     | LVBench  | MVBench  |\n| --------------------- | ---- | ---------- | ---- | --------- | -------- | -------- | -------- |\n| LLaVA-Video-7B-Qwen2  |      | Vanilla    |      | 59.8      | 67.4     | 58.2     | 58.5     |\n| LLaVA-Video-7B-Qwen2  |      | + VCD      |      | 60.0      | 67.3     | 58.3     | 58.7     |\n| LLaVA-Video-7B-Qwen2  |      | **+ Ours** |      | **61.6**  | **68.5** | **59.1** | **59.7** |\n| LLaVA-Video-72B-Qwen2 |      | Vanilla    |      | 67.5      | 71.2     | 61.9     | 64.1     |\n| LLaVA-Video-72B-Qwen2 |      | + VCD      |      | 67.6      | 72.2     | 61.9     | 64.3     |\n| LLaVA-Video-72B-Qwen2 |      | **+ Ours** |      | **68.8**  | **73.1** | **62.7** | **65.2** |\n| Qwen2.5-VL-7B         |      | Vanilla    |      | 65.1      | 70.2     | 45.3     | 69.6     |\n| Qwen2.5-VL-7B         |      | + VCD      |      | 65.3      | 70.3     | 45.3     | 69.7     |\n| Qwen2.5-VL-7B         |      | **+ Ours** |      | **66.7**  | **71.2** | **46.1** | **70.6** |\n| Qwen2.5-VL-72B        |      | Vanilla    |      | 73.3      | 74.6     | 47.3     | 70.4     |\n| Qwen2.5-VL-72B        |      | + VCD      |      | 73.3      | 74.6     | 47.2     | 70.6     |\n| Qwen2.5-VL-72B        |      | **+ Ours** |      | **74.5**  | **75.5** | **48.0** | **71.3** |\n| InternVL3-8B          |      | Vanilla    |      | 66.3      | 71.4     | 58.8     | 75.4     |\n| InternVL3-8B          |      | + VCD      |      | 66.4      | 71.4     | 58.8     | 75.5     |\n| InternVL3-8B          |      | **+ Ours** |      | **68.1**  | **72.5** | **59.6** | **76.7** |\n| InternVL3-38B         |      | Vanilla    |      | 72.7      | 77.8     | 67.3     | 76.9     |\n| InternVL3-38B         |      | + VCD      |      | 72.7      | 77.7     | 67.2     | 77.0     |\n| InternVL3-38B         |      | **+ Ours** |      | **73.8**  | **78.7** | **68.1** | **78.1** |\n\n(Tab. 1. More evaluation with different benchmarks and Video LLMs)\n\n\n\n## Weakness 4, Question 2\n\nThanks for the insightful question. Due to the nature of Contrastive Decoding (which requires two forward passes), CD-based methods (e.g., VCD and ours) are inevitably slower than the vanilla model in generation speed. \n\nIn practical scenarios, it's difficult to optimize performance, time efficiency, and memory efficiency all at once. Different applications prioritize different aspects. CD-based methods are less time-efficient, but they offer better performance without a significant increase in **memory usage**. This makes them well-suited for applications where content quality is critical, such as education and medical assistance, where accuracy really matters and errors come at a high cost.\n\nFrom Tab. 2, we observed that the time and memory efficiency of our TDU+CD is similar to VCD. CD based models (ours and VCD) are inevitably slower than the vanilla model due to CD'2 nature, but the additional memory usage is efficent, especially when model size grows.\n\n|                                      | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-7B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 | LLaVA-Video-72B-Qwen2 |\n| ------------------------------------ | -------------------- | -------------------- | -------------------- | --------------------- | --------------------- | --------------------- |\n| **Metric**                           | Vanilla              | + VCD                | + Ours               | Vanilla               | + VCD                 | + Ours                |\n| Avg Inference Latency per sample (s) | 1.026                | 1.670                | 1.675                | 5.315                 | 9.162                 | 9.165                 |\n| Avg Tokens per sec (tokens/s)        | 3.52                 | 2.107                | 2.101                | 0.679                 | 0.394                 | 0.394                 |\n| Avg GPU Memory Usage (MiB)           | 41418                | 43986                | 43988                | 193261                | 199590                | 199596                |\n| Total GPU Hours (h)                  | 1.144                | 1.862                | 1.867                | 6.034                 | 10.223                | 10.227                |\n\n(Tab. 2. Inference Statistics on A100 80GB on TempCompass)\n\nFor CD, its time efficiency might be improved through techniques such as leveraging the KV cache in some way, token pruning, or early stopping in the first forward pass, etc. This presents an interesting future research direction. However, our focus is not on optimizing the speed of CD algorithms, but rather on designing CD-based methods that effectively enhance temporal reasoning. Our guiding principle is to ensure that **our approach does not show inference performance degradation compared to common CD based methods like VCD**, rather than aiming to match the efficiency of the vanilla model. It is a separate and challenging reseach area different from our goal of temporal resoning.\n\n> a 2–3 % boost may not justify the extra cost in real-time settings ... a larger models (14B) often yield ≥ 2 % improvements out-of-the-box compared to a 7B model.\n\nThanks for the valuable opinion. Compared to 7B/8B models with TDU+CD, Tab. 2 shows that larger vanilla models (e.g., 38B and 72B) indeed exhibit significant better results. However, it is noted that when these larger models are combined with our TDU+CD approach, **their performance can be further improved**. Our goal is not to make smaller models outperform much larger ones, which is almost impossible in this task given the substantial initial performance gap, but rather to **enhance the capabilities of models at different scale**. Our method achieves this effectively.\n\nFor example, LLaVA-Video-72B-Qwen2 and Qwen2.5-VL-72B are already the largest models in their respective series, and there are no larger variants available within those model families. Yet, even at this scale, applying TDU+CD still brings improvements. Furthermore, in many practical scenarios where GPU resources are limited, increasing model size is not a viable option for improving accuracy. In such cases, TDU+CD offers a practical and efficient alternative for boosting performance without notably scaling up the model size.\n\n\n\n## Question 3\n\nDifferent distortion ratios do have an impact on model performance. With slight adjustments based on the task, the model can achieve best results. Intuitively, different videos or tasks have varying levels of information density, so using different distortion ratios allows better adaptation to specific scenarios. However, significant per-task tuning is not necessary to achieve improvements. As long as the ratio is not set to an extreme (i.e., very close to 0 or 1), simply choosing a moderate value around 0.5 already leads to consistent gains across different tasks. We show such setting (Ratio=0.5) on LLaVA-Video-Qwen2-7B on different tasks in Tab. 3.\n\nFor other parameters such as window size, block size, alpha, and beta, we use the **same** values across all tasks without any per-task tuning.\n\n|                | Vanilla Model | Best Setting | Ratio=0.5 |\n| -------------- | ------------- | ------------ | --------- |\n| EventHallusion | 63.5          | 72.1         | 71.3      |\n| TempCompass    | 65.0          | 67.5         | 66.7      |\n| MVBench        | 58.5          | 59.7         | 59.2      |\n| Video-MME      | 59.8          | 61.6         | 61.1      |\n\n(Tab. 3. Different ratio settings)\n\nWe sincerely thank you again for your review. Your comments are very helpful. We hope our responses can address your concerns. If you have any further questions or unresolved issues, we are glad to have further discussions!"}]

Author rebuttal:

Meta-review: This paper investigates common failure cases in temporal reasoning with pre-trained video LLMs and proposes a simple solution combining targeted temporal distortion and contrastive decoding. Initial reviews were mixed, but after the rebuttal and discussion all reviewers recommended acceptance. Reviewers highlighted the technical innovation and strong empirical results, though two concerns were raised: (1) novelty over prior contrastive decoding methods, and (2) inference-time overhead. On the first, most reviewers agreed the work offers sufficient innovation despite building on contrastive decoding. On the second, the authors provided additional results addressing inference overhead during the discussion phase. After considering the paper, reviews, and discussion, the AC concurs with the reviewers’ post-rebuttal consensus and recommends acceptance. The authors are encouraged to incorporate the rebuttal clarifications into the camera-ready version.
Decision: Accept (poster)