Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency
Authors: Jiafeng Liang, Shixin Jiang, Xuan Dong, Ning Wang, Zheng Chu, Hui Su, Jinlan Fu, Ming Liu, See-Kiong Ng, Bing Qin
Accepted at: acl25

Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency

Jiafeng Liang1*, Shixin Jiang1*, Xuan Dong1, Ning Wang, Zheng Chu1, Hui Su4,
Jinlan Fu3†, Ming Liu1,2‡, See-Kiong Ng3, Bing Qin1,2
1Harbin Institute of Technology, Harbin, China
2Peng Cheng Laboratory, Shenzhen, China
3National University of Singapore, Singapore
4Meituan Inc., Shanghai, China
{jfliang,sxjiang,mliu}@ir.hit.edu.cn, jinlanjonna@gmail.com

Abstract

Large Multimodal Models (LMMs) have recently demonstrated impressive performance on general video comprehension benchmarks. Nevertheless, for broader applications, the robustness of their temporal analysis capability needs to be thoroughly investigated yet predominantly ignored. Motivated by this, we propose a novel temporal robustness benchmark (**TemRobBench**), which introduces temporal inconsistency perturbations separately at the visual and textual modalities to assess the robustness of models. We evaluate 16 mainstream LMMs and find that they exhibit over-reliance on prior knowledge and textual context in adversarial environments, while ignoring the actual temporal dynamics in the video. To mitigate this issue, we design panoramic direct preference optimization (**PanoDPO**), which encourages LMMs to incorporate both visual and linguistic feature preferences simultaneously. Experimental results show that PanoDPO can effectively enhance the model’s robustness and reliability in temporal analysis.

1 Introduction

Large Multimodal Models (LMMs) (Wang et al., 2024b; OpenGVLab, 2024; Li et al., 2024c; Ye et al., 2024) can effortlessly understand videos with the support of temporal analysis capability. Recent research (Ren et al., 2025) further highlights this capability, proving that capturing visual changes alone can substantially enhance knowledge acquisition without the need for text labels. As a vital sensor for perception and learning, exploring its robustness is crucial yet largely overlooked.

Inspired by this, we conduct a preliminary exploration of mainstream LMMs and observe that they exhibit two different aspects of shortcut phenomena triggered by temporal inconsistency anomalies. First, when the temporal information in the video conflicts with common sense, the model primarily relies on prior knowledge to generate responses, which we refer to as **Intrinsic Temporal Shortcut** (shown in the Fig. 1 (b)). Second, the model exhibits a pronounced inclination to the textual context when discrepancies occur between the video and the accompanying text prompt, termed **Extrinsic Temporal Shortcut** (shown in the Fig. 1 (c)). More importantly, we notice that these phenomena are prevalent in mainstream LMMs and arise with high frequency. As illustrated in Fig. 2 (a), more than 59% of responses are taken shortcuts, indicating a flaw in the robustness of their tempo-

![An example of the Intrinsic Temporal Shortcut (b) and Extrinsic Temporal Shortcut (c). The model tends to excessively rely on prior knowledge or textual context when temporal inconsistencies arise between video content and common sense or text prompt.](page_1012_370_388_388.png)

Figure 1: An example of the Intrinsic Temporal Shortcut (b) and Extrinsic Temporal Shortcut (c). The model tends to excessively rely on prior knowledge or textual context when temporal inconsistencies arise between video content and common sense or text prompt.
Figure 2: (a) Response distribution when asking the question within temporal inconsistencies. The majority of errors stem from shortcuts. (b) The model discriminative ability on the correct and shortcut answer is represented by the difference in log-likelihoods.

ral analysis. Thus, a comprehensive benchmark to investigate these issues is necessary. However, the existing robustness benchmarks (Yi et al., 2021; Zeng et al., 2024; Li et al., 2024e) exhibit two major drawbacks that make it difficult to support our study: (i) Lack of consideration for temporal dimension: They only focus on adding feature perturbations to frames, such as Gaussian noise, which cannot reflect the model’s temporal dynamic robustness. (ii) Lack of consideration for textual context: They solely assess the model’s robustness to visual content, overlooking the text, which is insufficient for handling complex multimodal scenarios.

To address these limitations, we introduce TemRobBench, a novel temporal robustness benchmark, which incorporates four levels of temporal inconsistency perturbations across both visual and textual modalities. Additionally, we design temporal questions and corresponding options to check if a specific answer is flipped to shortcuts due to perturbations, totally collecting 1,686 QA pairs from 562 videos. Through extensive evaluations of 16 LMMs, we find that they generally exhibit weak temporal robustness, especially when perturbed by visual modalities, which leads to over-reliance on prior knowledge. Further observations indicate that correct answers generated by LMMs are not entirely reliable, as they often randomly guess when confronted with perturbations rather than referring to the video content. To mitigate these issues, we propose a panoramic direct preference optimization (PanoDPO) method, which introduces additional video- and question-conditioned preference pairs by incorporating adversarial information and employs multimodal guidance during preference learning, encouraging LMMs to focus on both visual and linguistic features simultaneously. Fig. 2 (b) shows the shifts of likelihood difference between correct and shortcut answers after aligning the model with conditioned preference data via PanoDPO, indicating that our method helps the model discriminate shortcuts, thereby enhancing its robustness.

Our main contributions are summarized as:

• We identify the robustness weaknesses of current LMMs, which frequently take shortcuts based on prior knowledge and textual context against temporal inconsistency anomalies.

• We introduce TemRobBench and conduct extensive investigations into temporal robustness of various LMMs to provide detailed insights.

• We propose a panoramic optimization method, PanoDPO, which effectively enhances the model’s robustness in temporal analysis.

2 TemRobBench

2.1 Benchmark Design Principal

We present TemRobBench, a novel benchmark designed to evaluate the temporal robustness of Large Multimodal Models (LMMs) against temporal inconsistency. TemRobBench focuses on investigating the degree to which: LMMs genuinely perceive temporal information in videos, without being influenced by intrinsic and extrinsic priors to take shortcuts. Specifically, intrinsic and extrinsic shortcuts refer to LMMs exhibiting over-reliance on inherent knowledge and textual context while neglecting the actual video content. To achieve this, we design adversarial perturbations on both the visual and textual modalities, and create samples with varying levels of inconsistency severity.

2.2 Inconsistency Perturbation Construction

2.2.1 Intrinsic Temporal Shortcut

To investigate whether the model relies on inherent temporal knowledge to take shortcuts, we design inconsistency perturbation to the video. Specially, we first apply shuffled editing to the original clips in the video, each representing an event. The edited video typically discords with common sense. As shown in Fig. 3, we swap the sequence of event “Strip the insulation” and action “Arrange the separated wire”, which rarely occurs in reality. The perturbations are grouped into two categories: light disorder and severe disorder. Light disorder means
Figure 3: Overview of the TemROBBENCH. The benchmark emphasizes evaluating the model’s robustness against temporal inconsistency, especially take intrinsic shortcuts (over-reliance on prior knowledge) and extrinsic shortcuts (over-reliance on textual context). We construct inconsistencies with knowledge and textual context by shuffling video clips and event descriptions, and design corresponding shortcut answers to verify the evidence of the response.

<table>
  <tr>
    <th>Category</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>Video Sources</td>
    <td>562</td>
  </tr>
  <tr>
    <td>- Events / per Video</td>
    <td>6.4</td>
  </tr>
  <tr>
    <td>- Maximum Duration</td>
    <td>149.9s</td>
  </tr>
  <tr>
    <td>- Minimum Duration</td>
    <td>20.8s</td>
  </tr>
  <tr>
    <td>- Average Duration</td>
    <td>106.7s</td>
  </tr>
  <tr>
    <td>Questions / per Video</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Total Samples</td>
    <td>1686</td>
  </tr>
</table>

Figure 4: Comprehensive statistics from different perspectives (left) and detailed inconsistency perturbation classes (right) in the TemROBBENCH.

swapping adjacent events once, while severe disorder involves multiple random swaps of different events. Then, we design a unified question to ask the model about the correct sequence of events in the video, with four options: One correct option that matches the edited sequence to test whether the model accurately captures the temporal information, one shortcut option that matches the unedited clip sequence to evaluate whether the model over-rely on prior knowledge, and two incorrect options to test whether the model makes temporal errors.

2.2.2 Extrinsic Temporal Shortcut
To examine if the model depends on textual context to shortcut its temporal analysis, we introduce perturbations into the text prompt. Specially, we provide shuffled event descriptions that are inconsistent with the video to models, and ask them to determine the actual order of two events. Examples of the shuffle edited text can be seen in Fig. 3. The perturbations are grouped into two categories: absolute disorder and relative disorder. In absolute disorder, we first select an adjacent event pair \( \{x_p, x_q\} \) from list \( E \), and shuffle the remaining ones. Then, we reverse the order and randomly insert them into \( E \), formulated as \( E^a = \{..., x_q, x_p, ...\} \). In relative disorder, an irrelevant event \( x_k \) is inserted in between target event pair to create perturbation, expressed as \( E^r = \{..., x_p, x_k, x_q, ...\} \). Similarly, we design four options: one correct option matches the video, one shortcut option matches the sequence of event captions, and two incorrect options.

2.3 Benchmark Statics
Typically, mainstream LMMs adopt caption (Heilbron et al., 2015) and VideoQA (Xiao et al., 2021) datasets as finetuning data. To minimize potential data leakage issues that could hinder the zero-shot evaluation, we use the action detection dataset COIN (Tang et al., 2019) as our data source, which contains high-quality raw manual annotations. As shown in Fig. 4, we collect 562 videos and automatically construct 1,686 multi-choice QA pairs adapted from raw annotations for TemROBBENCH, each video includes two different types of inconsistency perturbations. Moreover, to minimize the
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Frame</th>
    <th colspan="4">Intrinsic Temporal Shortcut</th>
    <th colspan="4">Extrinsic Temporal Shortcut</th>
  </tr>
  <tr>
    <th colspan="2">Clean</th>
    <th colspan="2">Adversarial</th>
    <th colspan="2">Clean</th>
    <th colspan="2">Adversarial</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Acc↑</th>
    <th>Acc↑</th>
    <th>FR↓</th>
    <th>WFR ↓</th>
    <th>Acc↑</th>
    <th>Acc↑</th>
    <th>FR↓</th>
    <th>WFR ↓</th>
  </tr>
  <tr>
    <td colspan="10"><i>7B LLM</i></td>
  </tr>
  <tr>
    <td>VideoChat2 (Li et al., 2024f)</td>
    <td>16</td>
    <td>33.4</td>
    <td>26.5 <span style="color:red;">-6.9</span></td>
    <td><b>28.4</b></td>
    <td><b>74.1</b></td>
    <td>22.6</td>
    <td>14.5 <span style="color:red;">-8.1</span></td>
    <td><b>18.5</b></td>
    <td>90.6</td>
  </tr>
  <tr>
    <td>VidcolLLaVA (Lin et al., 2024a)</td>
    <td>8</td>
    <td>31.5</td>
    <td>25.5 <span style="color:red;">-6.0</span></td>
    <td>62.4</td>
    <td>85.3</td>
    <td>22.9</td>
    <td>9.8 <span style="color:red;">-13.1</span></td>
    <td>19.4</td>
    <td>85.7</td>
  </tr>
  <tr>
    <td>LLaVA-Hound (Zhang et al., 2024)</td>
    <td>32</td>
    <td>35.3</td>
    <td>26.5 <span style="color:red;">-8.8</span></td>
    <td>60.3</td>
    <td>84.9</td>
    <td>17.8</td>
    <td>10.0 <span style="color:red;">-7.8</span></td>
    <td>42.6</td>
    <td>77.7</td>
  </tr>
  <tr>
    <td>ShareGPT4Video (Chen et al., 2024)</td>
    <td>16</td>
    <td>30.1</td>
    <td>24.0 <span style="color:red;">-6.1</span></td>
    <td>65.9</td>
    <td>87.0</td>
    <td>77.1</td>
    <td>22.8 <span style="color:red;">-54.3</span></td>
    <td>58.2</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td>InternVideo2 (Wang et al., 2024c)</td>
    <td>8</td>
    <td>35.3</td>
    <td>25.8 <span style="color:red;">-9.5</span></td>
    <td><b>57.2</b></td>
    <td>89.8</td>
    <td>52.8</td>
    <td>31.8 <span style="color:red;">-21.0</span></td>
    <td>28.8</td>
    <td>53.5</td>
  </tr>
  <tr>
    <td>VILA1.5 (Lin et al., 2024b)</td>
    <td>16</td>
    <td>24.0</td>
    <td>20.8 <span style="color:red;">-3.2</span></td>
    <td>70.2</td>
    <td>89.9</td>
    <td>66.8</td>
    <td>13.8 <span style="color:red;">-53.0</span></td>
    <td>56.7</td>
    <td>82.3</td>
  </tr>
  <tr>
    <td>VideoLLaMA2 (Cheng et al., 2024)</td>
    <td>32</td>
    <td>38.8</td>
    <td>29.3 <span style="color:red;">-8.5</span></td>
    <td>61.6</td>
    <td>85.0</td>
    <td>62.1</td>
    <td>28.3 <span style="color:red;">-33.8</span></td>
    <td>40.2</td>
    <td>62.8</td>
  </tr>
  <tr>
    <td>PLLaVA (Xu et al., 2024)</td>
    <td>32</td>
    <td>39.0</td>
    <td>27.2 <span style="color:red;">-11.8</span></td>
    <td>63.8</td>
    <td>88.8</td>
    <td>37.4</td>
    <td>14.5 <span style="color:red;">-22.9</span></td>
    <td>21.3</td>
    <td>78.7</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3 (Ye et al., 2024)</td>
    <td>32</td>
    <td>55.5</td>
    <td>26.4 <span style="color:red;">-29.1</span></td>
    <td>75.0</td>
    <td>87.5</td>
    <td><b>86.9</b></td>
    <td>33.1 <span style="color:red;">-53.8</span></td>
    <td>50.2</td>
    <td>63.1</td>
  </tr>
  <tr>
    <td>InternVL-2.5 (OpenGVLab, 2024)</td>
    <td>32</td>
    <td>54.9</td>
    <td>22.8 <span style="color:red;">-32.1</span></td>
    <td>81.8</td>
    <td>91.3</td>
    <td>81.8</td>
    <td>48.4 <span style="color:red;">-33.4</span></td>
    <td>33.4</td>
    <td>41.6</td>
  </tr>
  <tr>
    <td>Qwen2-VL (Wang et al., 2024b)</td>
    <td>32</td>
    <td>54.9</td>
    <td>24.6 <span style="color:red;">-30.3</span></td>
    <td>83.7</td>
    <td>93.7</td>
    <td>76.6</td>
    <td>38.8 <span style="color:red;">-37.8</span></td>
    <td>33.0</td>
    <td>51.9</td>
  </tr>
  <tr>
    <td>LLaVA-OV (Li et al., 2024e)</td>
    <td>32</td>
    <td>48.0</td>
    <td>29.2 <span style="color:red;">-18.8</span></td>
    <td>72.2</td>
    <td>86.2</td>
    <td>81.8</td>
    <td>21.2 <span style="color:red;">-60.6</span></td>
    <td>60.3</td>
    <td>75.4</td>
  </tr>
  <tr>
    <td colspan="10"><i>13B LLM</i></td>
  </tr>
  <tr>
    <td>VILA1.5 (Lin et al., 2024b)</td>
    <td>16</td>
    <td>38.4</td>
    <td>28.5 <span style="color:red;">-9.9</span></td>
    <td>69.7</td>
    <td>87.5</td>
    <td>59.8</td>
    <td>14.3 <span style="color:red;">-45.5</span></td>
    <td>57.1</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>PLLaVA (Xu et al., 2024)</td>
    <td>32</td>
    <td>42.0</td>
    <td><b>31.2</b> <span style="color:red;">-10.8</span></td>
    <td>62.5</td>
    <td>87.9</td>
    <td>44.9</td>
    <td>21.2 <span style="color:red;">-23.7</span></td>
    <td>19.0</td>
    <td>70.3</td>
  </tr>
  <tr>
    <td>InternVL-2.5 (OpenGVLab, 2024)</td>
    <td>32</td>
    <td>60.7</td>
    <td>24.7 <span style="color:red;">-36.0</span></td>
    <td>85.5</td>
    <td>90.4</td>
    <td>82.7</td>
    <td>52.4 <span style="color:red;">-30.3</span></td>
    <td>32.2</td>
    <td>37.6</td>
  </tr>
  <tr>
    <td colspan="10"><i>Closed-Source</i></td>
  </tr>
  <tr>
    <td>GPT-4o (OpenAI, 2023)</td>
    <td>32</td>
    <td><b>67.1</b></td>
    <td>16.8 <span style="color:red;">-50.3</span></td>
    <td>82.3</td>
    <td>91.6</td>
    <td><b>83.6</b></td>
    <td><b>57.9</b> <span style="color:red;">-25.7</span></td>
    <td>19.0</td>
    <td><b>34.1</b></td>
  </tr>
</table>

Table 1: Experiment results for LMMs answering the same questions under two different settings: Clean and Adversarial. Red numbers represent the accuracy drop caused by temporal inconsistencies, compared to the model under Clean setting. The values in bold and underlined represent the best and the second-best results, respectively.

evaluation bias due to the constrain window size of LMMs (unable to handle too much frames), we select videos with short duration (average 106.7s).

3 Evaluations

To deeply investigate the temporal perception robustness of LMMs, we set up two different question answering scenarios for comparative analysis. The first scenario is that the model answers the question with the unperturbed data, i.e., raw video and event descriptions, which we refer to as Clean setting. The second is that the model responses to the same question with inconsistency perturbation data from our TemRobBench, termed Adversarial setting. Both settings are multiple-choice QA format, and we instruct the model to select the correct option.

3.1 Evaluation Metrics

In this part, we introduce our evaluation metrics.

Accuracy (Acc): Due to the setup of the multiple-choice QA, we evaluate the correctness for the \( i \)-th sample by checking if the correct answer is matched the generated response. For the clarity of subsequent statements, we formalize this as:

\[
\text{Score}_i = 1, \text{ if } y_i \text{ in } \hat{y}_i \text{ else } 0,
\]

where \( y_i \) and \( \hat{y}_i \) denote the correct answer and the selected answer, respectively. Naturally, the overall accuracy can be formalized as:

\[
\text{Acc}(Y, \hat{Y}) = \frac{\sum_{i=1}^N \text{Score}_i(y_i, \hat{y}_i)}{N},
\]

where \( \text{Acc}(Y, \hat{Y}) \) represents the model’s accuracy score over the entire setting, \( Y = \{y_1, y_2, ..., y_N\} \) and \( \hat{Y} = \{\hat{y}_1, \hat{y}_2, ..., \hat{y}_N\} \).

Flip Rate (FR): To systematically estimate the degree of LMMs take shortcuts, we adopt FR (Zhong et al., 2024) to evaluate how many of the model’s original correct responses are misled by perturbations and change to match with our curated shortcut answers:

\[
\text{FR} = \frac{\sum_{i \in D^+} \text{Score}_i(y_i^-, \hat{y}_i^-)}{\sum_{i=1}^n \text{Score}_i(y_i^+, \hat{y}_i^+)},
\]

\[
D^+ = \{ i \mid \text{Score}(y_i^+, \hat{y}_i^+) = 1 \},
\]

where \( \hat{y}_i^+ \) and \( \hat{y}_i^- \) represent selected answers under Clean and Adversarial settings, respectively. \( y_i^+ \) and \( y_i^- \) are correct answers, \( y_i^- \) denotes shortcut answers in Adversarial.

Weak Flip Rate (WFR): In addition, we use a more general metric WFR, which calculate how
Figure 5: Accuracy (left) and flip rate (right) of two different aspects (i.e., blue bars and red bars represent intrinsic and extrinsic temporal shortcuts, respectively) for each perturbation classes. The stripe pattern denotes performance drop due to the temporal inconsistencies.

many mistakes (includes shortcuts and substantive errors) the model makes in perturbation scenarios:

\[
\mathrm{WFR} = \frac{\sum_{i \in D^+}(1 - \mathrm{Score}_i(y_i^{-}, \hat{y}_i^{-}))}{\sum_{i=1}^{n}\mathrm{Score}_i(y_i^{+}, \hat{y}_i^{+})}.
\]

3.2 Models

We investigate the temporal robustness in the following 16 mainstream LMMs: VideoChat2 (Li et al., 2024f), VideoLLaVA (Lin et al., 2024a), LLaVA-Hound (Zhang et al., 2024), ShareGPT4Video (Chen et al., 2024), Intern-Video2 (Wang et al., 2024c), VILA1.5 (Lin et al., 2024b), VideoLLaMA2 (Cheng et al., 2024), PLLaVA (Xu et al., 2024), mPLUG-Owl3 (Ye et al., 2024), InternVL-2.5 (OpenGVLab, 2024), Qwen2-VL (Wang et al., 2024b), LLaVA-OV (Li et al., 2024c), and GPT-4o (OpenAI, 2023). Detailed comparisons are provided in App. A.

3.3 Result and Analysis

*LMMs typically exhibit weak temporal robustness.* As shown in Tab. 1, we can observe that most LMMs exhibit significant performance degradation under temporal inconsistencies (See the accuracy of *Adversarial*) compared to consistent scenarios (See the accuracy of *Clean*). For current advanced open-source LMMs such as Qwen2-VL (Wang et al., 2024b) and InternVL-2.5 (OpenGVLab, 2024), despite their strong video understanding capability, they still suffer an over 50% performance drop. Further observation of the high flip rate (FR) reveals that the model’s responses are easily misled by temporal inconsistencies. Moreover, a higher weak flip rate (WFR) indicates that models incur comprehension deviation under adversarial settings. By comparing the performance of the same LMMs with larger sizes, such as 7B and 13B of PLLaVA (Xu et al., 2024), although there is a certain improvement in accuracy, the nearly constant FR indicates that the shortcuts have not been effectively alleviated.

**LMMs are more vulnerable to visual perturbations, relying on prior knowledge.** Comparing the FR results between two different perturbations (See the FR of intrinsic and extrinsic temporal shortcut), we find that LMMs are more susceptible to visual perturbation and take intrinsic shortcuts, i.e., over-reliance on prior knowledge. We consider this may due to the current LMMs are built on powerful Large Language Models (LLMs), which are better at determining whether the text modality contains misleading content. However, when perturbation occurs in visual modality, they become confused and tend to respond based on thought inertia.

*More temporal inconsistency leads to more extrinsic shortcuts but fewer intrinsic shortcuts.* We further present the accuracy and flip rate (FR) of detailed categories of perturbation in the Fig. 5. For text perturbation, absolute disorder (more conflict to the normal video) is more likely to cause the model to take external temporal shortcuts, indicating the model places more trust in textual context when faced with uncertainty. In contrast, for video perturbation, we observe a nearly 100% FR on the light disorder (more similar to the normal video). We consider this could attributed to the “over-confidence”, where the model quickly

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">ITS</th>
    <th colspan="2">ETS</th>
  </tr>
  <tr>
    <th>Frame</th>
    <th>Acc*</th>
    <th>T-Acc*</th>
    <th>Acc*</th>
    <th>T-Acc*</th>
  </tr>
  <tr>
    <td>7B LLM</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VideoChat2 (Li et al., 2024f)</td>
    <td>16</td>
    <td>26.5</td>
    <td>3.3 / 47.5%</td>
    <td>14.5</td>
    <td>4.1 / 71.8%</td>
  </tr>
  <tr>
    <td>VideoLLaVA (Lin et al., 2024a)</td>
    <td>8</td>
    <td>25.5</td>
    <td>6.7 / 71.7%</td>
    <td>9.8</td>
    <td>1.9 / 49.0%</td>
  </tr>
  <tr>
    <td>LLaVA-Hound (Zhang et al., 2024)</td>
    <td>32</td>
    <td>26.5</td>
    <td>0.7 / 97.3%</td>
    <td>10.0</td>
    <td>1.4 / 86.0%</td>
  </tr>
  <tr>
    <td>ShareGPT4Video (Chen et al., 2024)</td>
    <td>16</td>
    <td>24.0</td>
    <td>0.6 / 97.3%</td>
    <td>22.8</td>
    <td>13.7 / 39.9%</td>
  </tr>
  <tr>
    <td>InternVideo2 (Wang et al., 2024c)</td>
    <td>8</td>
    <td>25.8</td>
    <td>0.7 / 97.3%</td>
    <td>31.8</td>
    <td>14.5 / 54.4%</td>
  </tr>
  <tr>
    <td>VILA1.5 (Lin et al., 2024b)</td>
    <td>16</td>
    <td>20.6</td>
    <td>1.0 / 98.9%</td>
    <td>13.8</td>
    <td>1.6 / 86.2%</td>
  </tr>
  <tr>
    <td>VideoLLaMA2 (Cheng et al., 2024)</td>
    <td>32</td>
    <td>29.3</td>
    <td>16.2 / 47.0%</td>
    <td>28.3</td>
    <td>11.7 / 57.8%</td>
  </tr>
  <tr>
    <td>PLLaVA (Xu et al., 2024)</td>
    <td>32</td>
    <td>27.2</td>
    <td>1.6 / 94.1%</td>
    <td>14.5</td>
    <td>2.5 / 82.8%</td>
  </tr>
  <tr>
    <td>mPLUG-Owl3 (Ye et al., 2024)</td>
    <td>32</td>
    <td>26.4</td>
    <td>2.5 / 83.8%</td>
    <td>33.1</td>
    <td>2.5 / 72.7%</td>
  </tr>
  <tr>
    <td>InternVL-2.5 (OpenGVLab, 2024)</td>
    <td>32</td>
    <td>22.8</td>
    <td>2.0 / 95.8%</td>
    <td>48.4</td>
    <td>34.0 / 29.8%</td>
  </tr>
  <tr>
    <td>Qwen2-VL (Wang et al., 2024b)</td>
    <td>32</td>
    <td>24.8</td>
    <td>1.6 / 98.8%</td>
    <td>38.3</td>
    <td>19.7 / 79.8%</td>
  </tr>
  <tr>
    <td>LLaVA-OV (Li et al., 2024c)</td>
    <td>32</td>
    <td>29.2</td>
    <td>8.8 / 89.9%</td>
    <td>21.2</td>
    <td>11.7 / 44.8%</td>
  </tr>
  <tr>
    <td>13B LLM</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VILA1.5 (Lin et al., 2024b)</td>
    <td>16</td>
    <td>28.5</td>
    <td>10.1 / 64.6%</td>
    <td>14.3</td>
    <td>7.6 / 46.9%</td>
  </tr>
  <tr>
    <td>PLLaVA (Xu et al., 2024)</td>
    <td>32</td>
    <td>31.2</td>
    <td>12.7 / 59.7%</td>
    <td>21.2</td>
    <td>2.1 / 90.1%</td>
  </tr>
  <tr>
    <td>InternVL-2.5 (OpenGVLab, 2024)</td>
    <td>32</td>
    <td>24.7</td>
    <td>2.4 / 94.8%</td>
    <td>52.4</td>
    <td>48.1 / 4.2%</td>
  </tr>
  <tr>
    <td>Closed-Source</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o (OpenAI, 2023)</td>
    <td>32</td>
    <td>16.8</td>
    <td>16.7 / 46.6%</td>
    <td>57.9</td>
    <td>53.6 / 7.4%</td>
  </tr>
</table>

Table 2: The accuracy (Acc) and true accuracy (T-Acc) under *Adversarial* settings. T-Acc denote the voting result where the model answers correctly in three or more times out of four shuffling options rounds for each sample. **Red numbers** represent the ratio of unreliable parts. ITS and ETS denotes intrinsic and extrinsic temporal shortcut, respectively.
Figure 6: Overview of the PanoDPO. Vanilla DPO (a) expects LMMs to learn response preferences only. PanodPO (b) integrates additional video and question preference learning objectives to encourage models to focus the response interactions with both the video and question. Moreover, we construct the visual- and text-conditioned preference data (c) for PanoDPO learning.

glances at the video and assumes it aligns with typical patterns, directly based on prior knowledge.

The accuracy remains unreliable under inconsistency perturbations. Although LMMs flip the answer due to shortcuts or misunderstandings when perturbed, we find that the remaining correct parts are still not entirely reliable, as the model might randomly guess. To minimize this bias and further investigate the model’s true temporal robustness, we design a control setting following the (Li et al., 2024f) and (Hu et al., 2024). Specifically, we shuffle the order of the options and place the correct one in different positions in Adversarial, conducting four evaluation rounds. If the model answers correctly three or more times, we consider it actually perceiving the temporal information, which is defined as true accuracy (T-Acc). Surprisingly, despite performing well with high accuracy, some LMMs (e.g., VILA1.5 (Xu et al., 2024)) almost entirely rely on gambly guess when faced with interference. This phenomenon further suggests that LMMs exhibit weak temporal robustness when handling temporal interference.

4 PanoDPO

From the perspective of intrinsic and extrinsic temporal shortcut phenomena, LMMs tend to respond based on prior knowledge or textual context when they conflict with video content, neglecting the visual information. To mitigate this issue, we propose panoramic direct preference optimization (PanoDPO), which encourages LMMs to simultaneously focus on both visual and linguistic features.

4.1 Direct Preference Optimization

Direct Preference Optimization (DPO) (Rafailov et al., 2023) is a method that originates from RLHF (Ouyang et al., 2022), designed to encourage Large Language Models (LLMs) to generate responses that better align with human preferences without relying on explicit reward modeling or reinforcement learning. Specifically, given an input \( x \), we optimize the response \( y \) of the model \( \pi \) and constrain it to adhere to normal language patterns by KL divergence:

\[
\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{X}, y \sim \pi_\theta} \left\{ r(x, y) - \beta \mathrm{KL}\left[ \pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x) \right] \right\},
\]

where \( r \) and \( \pi_{\text{ref}} \) denotes reward function and reference model. DPO formulates the reward as follows:

\[
r(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + Z(x),
\]

where \( Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r(x, y)/\beta) \) is the partition function. Given the corresponding preferred (chosen) answers \( y_w \) and non-preferred (rejected) answers \( y_t \), DPO seeks to maximize the difference between their rewards. Thus, the objective can be derived based on the Bradley-Terry model (Bradley and Terry, 1952):

\[
\mathcal{L}_{\text{DPO}} = -\log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_t|x)}{\pi_{\text{ref}}(y_t|x)} \right).
\]

Naturally, as shown in Fig. 6 (a), the DPO objective in multimodal scenarios can be formulated as:

\[
\mathcal{L}_{\text{DPO}_m} = -\log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x,v)}{\pi_{\text{ref}}(y_w|x,v)} - \beta \log \frac{\pi_\theta(y_t|x,v)}{\pi_{\text{ref}}(y_t|x,v)} \right),
\]

where \( v \) is the visual modality input.
4.2 Panoramic Preference Optimization
To mitigate the issue of overlooking visual information in perturbed environments and enhance robustness, we propose the panoramic preference optimization approach, which integrates optimization modules for both the video and question components based on Vanilla DPO. As shown in Fig. 6 (b), given a pair of tuples \((v_w, x_w, y_w)\) and \((v_l, x_w, y_w)\), where \(v_w\) is the chosen video, and \(v_l\) is the rejected one constructed by disruptive visual information. Subsequently, visual-condition optimization DPO_v can be established, where video is the sole variable:
\[
\mathcal{L}_{\text{DPO}_v} = -\log \sigma \left( \beta \log \frac{\pi_{\theta}(y_w|x_w, v_w)}{\pi_{\theta}(y_w|x_w, v_l)} - \beta \log \frac{\pi_{\theta}(y_l|x_w, v_w)}{\pi_{\theta}(y_l|x_w, v_l)} \right).
\]
Similar to the DPO_v, the text-conditioned DPO_t includes tuples pairs \((v_w, x_w, y_w)\) and \((v_w, (x_w + c), y_w)\) with the question as the only variable, and its optimization objective can be formulated as:
\[
\mathcal{L}_{\text{DPO}_t} = -\log \sigma \left( \beta \log \frac{\pi_{\theta}(y_w|x_w+c, v_w)}{\pi_{\theta}(y_w|x_w+c, v_l)} - \beta \log \frac{\pi_{\theta}(y_l|x_w+c, v_w)}{\pi_{\theta}(y_l|x_w+c, v_l)} \right),
\]
where \(v_w\) is the chosen question, and \(c\) denotes the perturbative content introduced into the question. Ultimately, we perform panoramic optimization by combining vanilla DPO, DPO_v, and DPO_t:
\[
\mathcal{L}_{\text{PanoDPO}} = \mathcal{L}_{\text{DPO}_m} + \mathcal{L}_{\text{DPO}_v} + \mathcal{L}_{\text{DPO}_t}.
\]

4.3 Preference Data Construction
To acquire visual- and text-conditioned preference data for DPO_v and DPO_t, we expand the existing dataset ShareGPTVideo-DPO (Zhang et al., 2024), which contains videos, questions and preference pairs of response. As shown in Fig. 6 (c), to obtain rejected videos, we perform video editing using three methods to construct destructive visual content: shuffling the video frames, randomly cropping the frames, and replacing certain frames with blank, respectively. Furthermore, to acquire rejected questions, we introduce perturbations into the original questions. Specifically, we first generate captions for the videos, and then use GPT-4o (OpenAI, 2023) to construct contextually adaptive perturbation text based on the caption, question, and correct answer. Note that these perturbations appear to be plausible but misleading, as they are actually contradictory to the video content. More details of preference data are provided in App. B.

5 Experiments

5.1 Experimental Setups
Baseline Methods. We compare our PanoDPO against the following three baselines: SFT refers to the fine-tuned model without any preference optimization. Prompt is utilized to instruct the model to focus on the given video without overly relying on prior knowledge or textual context. Vanilla DPO (Rafailov et al., 2023) is designed to fine-tune LMMs to learn response preferences based on the video and question. We evaluate the effectiveness of the baseline methods and our PanoDPO on two LMMs: LLaVA-OV-7B (Li et al., 2024c) and LLaVA-Hound-7B (Zhang et al., 2024).

Evaluation Metrics. According to the observations in Sec. 3.3, the temporal perception ability reflected by accuracy (Acc) is not entirely reliable. Therefore, we adopt true accuracy (T-Acc) as the metric, which excludes the potentially random guess aspects of Acc. Additionally, we use flip rate (FR) to assess the model’s temporal robustness.

Implementation Details. All models are fine-tuned using LoRA for 3 epochs with a batch size of 64. We use the learning rate of 1e-5, a cosine scheduler, and warm-up ratio of 0.1. The preference optimization coefficient \( \beta \) is set to 0.1.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Frame</th>
    <th colspan="2">ITS</th>
    <th colspan="2">ETS</th>
  </tr>
  <tr>
    <th>T-Acc↑</th>
    <th>FR↓</th>
    <th>T-Acc↑</th>
    <th>FR↓</th>
  </tr>
  <tr>
    <td>LLaVA-OV-7B</td>
    <td></td>
    <td>8.8</td>
    <td>72.2</td>
    <td>11.7</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td>w/SFT</td>
    <td>32</td>
    <td>8.9</td>
    <td>71.8</td>
    <td>11.3</td>
    <td>59.5</td>
  </tr>
  <tr>
    <td>w/Prompt</td>
    <td>32</td>
    <td>9.6</td>
    <td>69.4</td>
    <td>12.2</td>
    <td>58.8</td>
  </tr>
  <tr>
    <td>w/Vanilla DPO</td>
    <td>32</td>
    <td>16.6</td>
    <td>55.5</td>
    <td>15.3</td>
    <td>50.3</td>
  </tr>
  <tr>
    <td>w/PanoDPO (ours)</td>
    <td>32</td>
    <td>8.9</td>
    <td>56.4</td>
    <td>5.3</td>
    <td>18.4</td>
  </tr>
  <tr>
    <td>LLaVA-Hound-7B</td>
    <td></td>
    <td>0.7</td>
    <td>60.3</td>
    <td>1.4</td>
    <td>42.6</td>
  </tr>
  <tr>
    <td>w/SFT</td>
    <td>32</td>
    <td>0.6</td>
    <td>61.0</td>
    <td>1.4</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>w/Prompt</td>
    <td>32</td>
    <td>0.9</td>
    <td>61.2</td>
    <td>1.7</td>
    <td>41.8</td>
  </tr>
  <tr>
    <td>w/Vanilla DPO</td>
    <td>32</td>
    <td>8.9</td>
    <td>56.4</td>
    <td>5.3</td>
    <td>18.4</td>
  </tr>
  <tr>
    <td>w/PanoDPO (ours)</td>
    <td>32</td>
    <td>8.9</td>
    <td>56.4</td>
    <td>5.3</td>
    <td>18.4</td>
  </tr>
</table>

Table 3: Comparison of our PanoDPO to other baseline methods on two backbone LMMs under adversarial setting. ITS and ETS denotes intrinsic and extrinsic temporal shortcut, respectively.

The experimental results are shown in the Table Tab. 3. We find that the prompt-based method performs almost identically to the SFT method, indicating that the temporal shortcut is an inherent issue that is difficult to alleviate through instructions. Vanilla DPO (Rafailov et al., 2023) provides a certain relief, but the effect is still inconspicuous, as this method lacks targeted optimization strategies for visual and textual conditions. In contrast, our proposed PanoDPO mitigates both intrinsic and extrinsic shortcut phenomena through panoramic optimization preference optimization, significantly
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">ITS</th>
    <th colspan="2">ETS</th>
  </tr>
  <tr>
    <th>Frame</th>
    <th>T-Acc</th>
    <th>FR</th>
    <th>T-Acc</th>
    <th>FR</th>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/ PanoDPO (ours)</td>
    <td>32</td>
    <td><b>16.6</b></td>
    <td><b>55.5</b></td>
    <td><b>15.3</b></td>
    <td><b>50.3</b></td>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/o DPO<sub>v</sub></td>
    <td>32</td>
    <td>12.4</td>
    <td>61.0</td>
    <td>15.2</td>
    <td>52.3</td>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/o DPO<sub>t</sub></td>
    <td>32</td>
    <td>16.3</td>
    <td>58.2</td>
    <td>12.9</td>
    <td>58.0</td>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/ DPO<sub>v</sub>, crop</td>
    <td>32</td>
    <td>13.7</td>
    <td>59.7</td>
    <td>14.8</td>
    <td>52.2</td>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/ DPO<sub>v</sub>, replace</td>
    <td>32</td>
    <td>14.3</td>
    <td>60.4</td>
    <td>15.1</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>LLaVA-OV-7B<br>w/ DPO<sub>v</sub>, shuffle</td>
    <td>32</td>
    <td><b>16.6</b></td>
    <td><b>55.5</b></td>
    <td><b>15.3</b></td>
    <td><b>50.3</b></td>
  </tr>
</table>

Table 4: Ablation on different condition modules in PanoDPO and rejected video construction strategies.

enhancing its temporal perception robustness. In Fig. 7, we show the shifts of average likelihood difference between correct and shortcut answer in each inference batch under different optimization methods. The results demonstrate that our PanodPO better helps the model to distinguish shortcuts (i.e., larger shifts reflects stronger discrimination), effectively enhancing the robustness.

5.3 Analysis

Impact of the different optimization conditions. To investigate the effectiveness of the video- and question-condition modules of PanoDPO, we conduct an ablation on DPO<sub>v</sub> and DPO<sub>t</sub> separately. As shown in Tab. 4, when DPO<sub>v</sub> or DPO<sub>t</sub> are removed, the model suffer significantly performance decreases, indicting that both conditions play a targeted role in improving the robustness.

Impact of the different construction strategies. In DPO<sub>v</sub>, we try three different rejected video construction strategies: shuffling the video frame order, cropping random regions, and replacing random frames with blank spaces. To investigate the effects of them, we conduct an ablation by training with different rejected data separately. The results in Tab. 4 show that shuffle achieves the best performance, demonstrating that temporal dynamic is an important factor for videos. Disrupting temporal sequence can effectively destroy video information.

General capability analysis. PanoDPO is proposed to enhance temporal robustness of LMMs. To further verify and analyze its capability in general video understanding, we evaluate it on three mainstream video understanding benchmarks, including VideoMME (Fu et al., 2024), LongVideoBench (Wu et al., 2024a), and ActivitynetQA (Yu et al., 2019). We compare LLaVA-OV+DPO with LLaVA-OV+PanoDPO, and the results are shown in the Tab. 5. The results demonstrate that our proposed PanoDPO remains effective in maintaining general capabilities.

![The discriminative ability of the backbone model LLaVA-OV for correct and shortcut answers with different optimization strategies, represented by difference in log-likelihoods.](page_1012_180_388_388.png)

Figure 7: The discriminative ability of the backbone model LLaVA-OV for correct and shortcut answers with different optimization strategies, represented by difference in log-likelihoods.

<table>
  <tr>
    <th>Model</th>
    <th>VideoMME</th>
    <th>LongVideoBench</th>
    <th>ActivityNet-QA</th>
  </tr>
  <tr>
    <td>LLaVA-OV-7B</td>
    <td>58.2</td>
    <td>55.6</td>
    <td>56.6</td>
  </tr>
  <tr>
    <td>w/ Vanilla DPO</td>
    <td>58.8</td>
    <td>56.4</td>
    <td>57.6</td>
  </tr>
  <tr>
    <td>w/ PanoDPO (Ours)</td>
    <td><b>60.9</b></td>
    <td><b>58.9</b></td>
    <td><b>59.8</b></td>
  </tr>
</table>

Table 5: The general capability evaluation results on three general video understanding benchmarks.

6 Related Work

Large Multimodal Models. Large Multimodal Models (LMMs) (OpenGVLab, 2024; Wang et al., 2024b; Ye et al., 2024; Li et al., 2024c) have seen impressive developments in recent years. They are primarily built upon large language models (LLMs), which extend temporal dynamics by leveraging strong linguistic capabilities. Despite, they demonstrate impressive performance on general video understanding, for broader applications, the robustness of their temporal capabilities needs to be thoroughly investigated yet largely overlooked.

Temporal Robustness Benchmark. Currently, there are numerous methods and benchmarks related to robustness focused on the image domain (Qiu et al., 2024; Li et al., 2024a; Zhao et al., 2023; Zhou et al., 2023; Lee et al., 2024; Wu et al., 2024b; Li et al., 2024b). However, research related to videos remains insufficient. Existing works such as (Li et al., 2024e; Zeng et al., 2024; Yi et al., 2021; Schiappa et al., 2022) primarily focus on applying feature perturbations to individual frames while neglecting the unique temporal characteristics of videos. Furthermore, they mainly investigate the model’s robustness to visual and overlook the text, which is inadequate for multimodal scenarios. A work similar to ours is TempCompass (Liu et al., 2024b), which innovatively introduces temporal adversarial data through video editing. However,
due to its simplistic adversarial approach, the challenges it presents are relatively limited. Additionally, TempCompass is difficult to effectively diagnose the source of errors. In contrast, we propose TemRobBench that systematically investigates multimodal robustness in LMMs against temporal inconsistency.

Direct Preference Optimization. Direct preference optimization (DPO) (Rafailov et al., 2023), which focuses on directly optimizing large language models (LLMs) to align human preferences has gained significant traction in the context of RLHF (Ouyang et al., 2022). Previous works (Xie et al., 2024; Wang et al., 2024a) primarily emphasize constructing image contrast data to optimize visual preferences. Recently, some works such as (Zhang et al., 2024) and (Liu et al., 2024a) transfer DPO to video tasks. However, they target only response optimization, which is limited to multimodal scenarios. In contrast, we propose PanoDPO, which performs panoramic optimization on both the video, question and response, encouraging the model to simultaneously prioritize visual information and linguistic features.

7 Conclusion

In this paper, we identify the robustness weaknesses of intrinsic and extrinsic shortcuts in LMMs against temporal inconsistency, where the model over-rely on prior knowledge and textural context to response, neglecting the actual video content. To systematically investigate these issue, we carefully design TemRobBench, which includes diverse temporal inconsistency settings. The extensive evaluations demonstrate that the temporal robustness of LMMs is generally fragile, despite their strong performance on understanding general videos. Additionally, we propose a preference optimization method PanoDPO, which effectively enhance robustness of LMMs in temporal analysis and alleviates the shortcut phenomenon.

Limitations

Although we construct a comprehensive benchmark and propose a methodology to investigate and mitigate the shortcut phenomena caused by the weak temporal robustness of LMMs, our work still has limitations. Firstly, the temporal inconsistency scenarios in our dataset are relatively simplistic. We focus on classifying them based on varying degrees of inconsistency, as expanding to more complex and diverse scenarios would increase the difficulty and demand considerable effort. Secondly, our experiments are conducted on models of 7B and 13B sizes, and we evaluate our proposed PanoDPO on a few selected models. This is due to computational limitations.

Acknowledgements

The research in this article is supported by the National Science Foundation of China (U22B2059, 62276083). This research is also supported by A*STAR, CISCO Systems (USA) Pte. Ltd and National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award I21001E0002).

References

Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345.

Cong Chen, Mingyu Liu, Chenchen Jing, Yizhou Zhou, Fengyun Rao, Hao Chen, Bo Zhang, and Chunhua Shen. 2025. Perturbollava: Reducing multimodal hallucinations with perturbative visual training. In ICLR.

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. 2024. Sharegpt4video: Improving video understanding and generation with better captions. CoRR, abs/2406.04325.

Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-lmms. CoRR, abs/2406.07476.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Chaoyou Fu, Yuhai Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. Video-nme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. CoRR, abs/2405.21075.

Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 961–970. IEEE Computer Society.

Bo Hu, Meng Zhang, Chenfei Xie, Yuanhe Tian, Yan Song, and Zhendong Mao. 2024. RESEMO: A benchmark chinese dataset for studying responsive emotion from social media content. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 16375–16387. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. CoRR, abs/2310.06825.

Kang-il Lee, Minbeom Kim, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2024. Vlind-bench: Measuring language priors in large vision-language models. CoRR, abs/2406.08702.

Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024a. Naturalbench: Evaluating vision-language models on natural adversarial samples. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024b. Naturalbench: Evaluating vision-language models on natural adversarial samples. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024c. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024d. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. CoRR, abs/2407.07895.

Jinmin Li, Kuofeng Gao, Yang Bai, Jingyun Zhang, Shutao Xia, and Yisen Wang. 2024e. Fmm-attack: A flow-based multi-modal adversarial attack on video-based llms. CoRR, abs/2403.13507.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. 2024f. Mybench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 22195–22206. IEEE.

Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. 2023. Unmasked teacher: Towards training-efficient video foundation models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 19891–19903. IEEE.

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024a. Video-llava: Learning unified visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 5971–5984. Association for Computational Linguistics.

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024b. VILA: on pre-training for visual language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26679–26689. IEEE.

Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. 2024a. Videodpo: Omni-preference alignment for video diffusion generation. CoRR, abs/2412.14167.

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024b. Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 8731–8772. Association for Computational Linguistics.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

OpenGVLab. 2024. Internvl2.5: Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. 2024. Benchmarking robustness of multimodal image-text models under distribution shift. Preprint, arXiv:2212.08044.

Rafael Raffailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. 2025. Videoworld: Exploring knowledge learning from unlabeled videos. arXiv preprint arXiv:2501.09781.

Madeline Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S. Rawat, and Vibhav Vineet. 2022. Robustness analysis of video-language models against visual and language perturbations. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019. COIN: A large-scale dataset for comprehensive instructional video analysis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1207–1216. Computer Vision Foundation / IEEE.

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhaq Chen. 2024a. mdpo: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8078–8088. Association for Computational Linguistics.

Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. 2023. Videomae V2: scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 14549–14560. IEEE.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. CoRR, abs/2409.12191.

Yi Wang, Kunchang Li, Xinzhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilian Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. 2024c. Intervideo2: Scaling foundation models for multimodal video understanding. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXXV, volume 15143 of Lecture Notes in Computer Science, pages 396–416. Springer.

Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024a. Longvideobench: A benchmark for long-context interleaved video-language understanding. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Shujin Wu, Yi Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. 2024b. MACAROON: training vision-language models to be your engaged partners. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 7715–7731. Association for Computational Linguistics.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 9777–9786. Computer Vision Foundation / IEEE.

Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. 2024. V-DPO: mitigating hallucination in large vision language models via vision-guided direct preference optimization. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 13258–13273. Association for Computational Linguistics.

Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, and Jiashi Feng. 2024. Plava : Parameter-free llava extension from images to videos for video dense captioning. CoRR, abs/2404.16994.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guangting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni,
Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. CoRR, abs/2407.10671.

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2024. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. CoRR, abs/2408.04840.

Chenyu Yi, Siyuan Yang, Haoliang Li, Yap-Peng Tan, and Alex C. Kot. 2021. Benchmarking the robustness of spatial-temporal models against corruptions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I. NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueteng Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 9127–9134. AAAI Press.

Runhao Zeng, Xiaoyong Chen, Jiaming Liang, Huisi Wu, Guangzhong Cao, and Yong Guo. 2024. Benchmarking the robustness of temporal action detection models against temporal corruptions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 18263–18274. IEEE.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941–11952. IEEE.

Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. 2024. Direct preference optimization of video large multimodal models from language model reward. CoRR, abs/2404.01258.

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. 2023. On evaluating adversarial robustness of large vision-language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 11991–12011. Association for Computational Linguistics.

Kankan Zhou, Eason Lai, Wei Bin Au Yeong, Kyriakos Mouratidis, and Jing Jiang. 2023. ROME: evaluating pre-trained vision-language models on reasoning beyond visual common sense. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 10185–10197. Association for Computational Linguistics.

Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Caivan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
A  Detailed descriptions of LMMs

Videochat2 (Li et al., 2024f) is a model built on visual encoder UMT (Li et al., 2023) and LLM Vicuna-v0 (Chiang et al., 2023), trained through a three-stage progressive training process.

VideoLLaVA (Lin et al., 2024a) is a model constructed upon the foundation of Language-Bind (Zhu et al., 2024), which pre-aligns images and videos, and Vicuna-v1.5 (Chiang et al., 2023), undergoing a two-phase training regimen that blends both image and video data.

LLaVA-Hound extends the pipeline of Video-LLaVA and introduces additional video DPO (Rafailov et al., 2023) training.

ShareGPT4Video leverages GPT4v (OpenAI, 2023) to generate dense and precise video captions for training, building upon the foundation of LLaVA-NEXT (Li et al., 2024d).

InternVideo2 is constructed by expanding the visual encoder (Wang et al., 2023) and integrating Mistral (Jiang et al., 2023), while continuing the training process established by VideoChat2.

VILA1.5 (Lin et al., 2024b) is built upon SigLIP (Zhai et al., 2023) and Vicuna-1.5, utilizing large-scale interleaved image-text data for pre-training to enhance alignment efficacy.

VideoLLaMA2 (Cheng et al., 2024) is built upon SigLIP and Mistral-7B-Instruct, employing 3D convolutions to build an alignment layer that circumvents information loss due to token compression.

PLLaVA (Xu et al., 2024) introduces a pooling strategy on the basis of LLaVA-NEXT, circumventing the bias of learned high-norm visual features that arise from utilizing image-language models.

mPLUG-ow3 (Ye et al., 2024) integrates several hyper attention transformer blocks within the transformer blocks of Qwen2 (Yang et al., 2024) to facilitate the fusion of multimodal information, thereby preventing the loss of visual information during the front-end processing of the language model.

InternVL2.5 (OpenGVLab, 2024) is built upon the foundation of InternViT and InternLM, with enhancements in data quality and training strategy optimization to bolster model performance.

Qwen2-VL (Wang et al., 2024b) is constructed upon Qwen2 and ViT (Dosovitskiy et al., 2021), incorporating naive dynamic resolution and M-RoPE strategies to effectively integrate information across different modalities, enabling the comprehension of very long videos.

LLaVA-OV (Li et al., 2024c) is built upon Qwen2 and SigLIP, leveraging a pooled anyres strategy to achieve superior performance across single-image, multi-image, and video scenarios.

GPT-4o (OpenAI, 2023) builds upon the GPT4v, further enhancing its multimodal and multilingual abilities, enabling it to comprehend various modalities including images, videos, and audio.

B  More Details in Preference Data Construction

All preference data for training PanoDPO is extended from ShareGPTVideo-DPO (Zhang et al., 2024), which contains 17K samples, including videos, questions, chosen and rejected responses.

B.1  Visual-conditioned Preference Data

We explore three different methods to construct the rejected videos. For the shuffle, we generate random numbers to rearrange the indices. For the crop, we randomly remove 20% of the area in each frame to reduce the available visual information. For the replace, we randomly change 50% of the video frames with blank during the training.

B.2  Textual-conditioned Preference Data

We first generate captions for each video, and then generate perturbation texts based on the captions, questions, and correct answers. As shown in the prompt in Fig. 8, we have carefully defined the rules inspire by (Chen et al., 2025) for generating perturbation content to ensure their high quality.

C  Introduction to the COIN Dataset

The COIN dataset (Tang et al., 2019) consists of videos related to 180 different tasks, which are all collected from YouTube. The average length of a video is 2.36 minutes. Each video is labelled with 3.91 step segments, where each segment lasts 14.91 seconds on average and corresponding to a manually annotated event description.

D  More Implementation Details

During DPO training, we freeze the vision encoder and only optimize the LoRA parameters of the LLM and the parameters of projector. Our training codes are based on the HuggingFace TRL. Additionally, we use A100 80GB GPUs for training and apply full shared data parallel (FSDP) and gradient checkpointing to save GPU memory. Each DPO training takes approximately 10 hours.
Prompt for Generating Perturbation Content

[User]
-Goal-:
Your mission is to subtly and strategically mislead the multimodal model by crafting responses that seem plausible, confident, and logical, yet are fundamentally incorrect. Your objective is to avoid directly pointing out any errors or contradictions in the video. Instead, guide the model into making its own incorrect assumptions by using persuasive language-based reasoning that conflicts with the visual content. This isn’t about trapping the model—it’s about leading it astray through sophisticated, indirect misdirection.

-Steps-:
1. Analyze the (Video content) and the (Question), noting key elements from the video and the correct answer. However, refrain from directly confronting any discrepancies in the video.
2. Create subtle perturbations that align with widely accepted world knowledge or logical reasoning, but interpret the video content in a way that leads to a conflicting or misleading conclusion. The following principles will help guide your responses:
   • Never directly state that something in the video is incorrect. Instead, offer an interpretation that, while plausible, subtly deviates from the visual truth. The model must be misled into arriving at an incorrect conclusion on its own.
   • Use established knowledge to give weight to your misdirection. Frame your reasoning confidently and authoritatively to make it harder for the model to question.
   • Lead with implication and speculation, encouraging the model to rely on language-based patterns and general knowledge rather than visual context.
3. Ensure your perturbations meet these criteria:
   • They must subtly reinterpret or mislead without directly pointing to errors in the video. Avoid any blatant contradictions or confrontations with the video content.
   • The perturbations should sound natural and rational, pulling the model into an incorrect response that feels justified through reasoning, not forced by direct contradictions.
   • Leverage a tone of authority and confidence to subtly strengthen the incorrect interpretation, making the model less likely to double-check against the visual input.
4. Refining your perturbation:
   • Have you avoided stating that something is wrong in the video? Subtlety is key. You should be leading the model, not trapping it.
   • Is the perturbation focused on language-based reasoning and assumptions, subtly bypassing the need for visual validation?
   • Does the response sound logical, confident, and authoritative while leading to a fundamentally incorrect conclusion?
5. Once you’ve crafted the perturbation, ensure it is as detailed and complex as possible, introducing multiple layers of misdirection and logical reasoning to produce an elaborate and convincing, yet fundamentally incorrect conclusion.
6. Avoid making the conclusion too lengthy. Output the (Perturbation): .

-Real Data-:
(Video content): '[Caption]'
(Question): '[Question]'
(Answer): '[Correct answer]'
output:

...... Several Examples ......

Reviewer paper summaries: This paper introduces TEMROBBENCH, a novel benchmark for evaluating the temporal robustness of Large Multimodal Models (LMMs) when faced with temporal inconsistencies in visual and textual modalities. The authors observe that current LMMs exhibit an intrinsic temporal shortcut where models rely on prior knowledge when video content conflicts with common sense and an extrinsic temporal shortcut where models favor textual context over video evidence when discrepancies exist. The paper evaluates 16 LMMs, finding significant robustness issues, and proposes a method to mitigate these problems.

The paper proposes TemRobBench, a benchmark for evaluating the robustness of multimodal LLMs against temporal inconsistency. The TemRobBench is carefully designed to reveal the intrinsic and extrinsic temporal bias of multimodal LLMs, and extensive experimental results on the benchmark show that existing multimodal LLMs typically exhibit weak temporal robustness. Moreover, the authors build preference data and a variant of DPO named PanoDPO to enhance the robustness of temporal inconsistency of multimodal LLMs, which lifts their performance on the TemRobBench test data.
Summary of strengths: 1. The paper identifies an important issue in multimodal models - their tendency to rely on shortcuts when facing temporal inconsistencies.

2. The evaluation across 16 different LMMs helps establish that this is a general problem worth investigating.

3. The figures are well-crafted.

1. **Writing**: The draft is quite easy to follow and well-organized.
2. **Data resource contribution**: The proposed  TemRobBench benchmark is meticulously designed for assessing the intrinsic and extrinsic temporal bias of multimodal LLMs, which would be useful for future research concerning the temporal understanding capability of multimodal LLMs.
3. **Extensive experimental evaluation**: Comprehensive evaluation results show that current multimodal LLMs typically exhibit weak temporal robustness, and the proposed temporal alignment post-training method named PanoDPO improves their performance on the proposed benchmark.
Summary of weaknesses: 1. The "intrinsic" vs. "extrinsic" shortcut framework lacks theoretical grounding in cognitive science or machine learning principles about multimodal integration.

2. The paper fails to analyze how different temporal modeling architectures across the 16 LMMs affect their susceptibility to shortcuts. 

3. The paper claims that "panoramic optimization" encourages models to focus on visual and linguistic features simultaneously but provides no attention visualization or feature analysis to verify this claim.

4. The authors fail to provide an anonymous link to TEMROBBENCH, making it hard to assess data quality or verify their claims regarding temporal perturbations.

5. The control experiments are insufficient. The authors attribute performance drops to "shortcuts" without properly isolating variables.

6. No error analysis is provided to differentiate between different types of temporal reasoning failures (e.g., order confusion vs. missing temporal transitions).

7. The paper doesn't adequately analyze how model scale affects robustness to temporal inconsistencies.

1. **Lack of discussion on prior works on the temporal understanding ability of multimodal LLMs**: Prior studies like TempCompass [1] have comprehensively studied the temporal perception ability of Video LLMs. I sincerely encourage the authors to point out the key contributions and new insight provided by the proposed TemRobBench benchmark compared to prior studies like TempCompass.

2. **The effectiveness of the PanoDPO method is in doubt**. First, even after the PanoDPO post-training, the T-acc of LLaVa models on the test data are still below 20%, which suggests that the post-training hardly improves their temporal perception. Second, it is unknown whether the PanoDPO training harms model performance in the general domain, as only the results on the proposed TemRobBench benchmark are shown.

[1] Liu, Yuanxin, et al. "TempCompass: Do Video LLMs Really Understand Videos?." Findings of the Association for Computational Linguistics ACL 2024. 2024.
Comments, suggestions and typos: 1. The paper posits that models "rely on prior knowledge" but provides no causal evidence for this claim.

Please see the "Summary Of Weaknesses".
Overall assessment scores: 3.5

3