Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation
Authors: Qiji Zhou, YiFan Gong, Guangsheng Bao, Hongjie Qiu, Jinqiang Li, Xiangrong Zhu, Huajian Zhang, Yue Zhang
Accepted at: acl25

Reasoning is All You Need for Video Generalization: A Counterfactual Benchmark with Sub-question Evaluation

Qiji Zhou* 1, YiFan Gong* 2, Guangsheng Bao1, Hongjie Qiu2, Jinqiang Li2 Xiangrong Zhu2, Huajian Zhang1, Yue Zhang† 1
1School of Engineering, Westlake University
2College of Computer Science and Technology, Hangzhou Dianzi University
{zhouqiji, baoguangsheng, zhanghuajian, zhangyue}@westlake.edu.cn
{gongyifan, qiuHongjie, lijinQiang, zhuxiangrong}@hdu.edu.cn

Abstract

Counterfactual reasoning is crucial for robust video understanding but remains underexplored in existing multimodal benchmarks. In this paper, we introduce COVER (COunterfactual VidEo Reasoning), a multidimensional multimodal benchmark that systematically evaluates MLLMs across the abstract-concrete and perception-cognition dimensions. Beyond prior multimodal benchmarks, COVER decomposes complex queries into structured sub-questions, enabling fine-grained reasoning analysis. Experiments on commercial and open-source models reveal a strong correlation between sub-question accuracy and counterfactual reasoning performance, highlighting the role of structured inference in video understanding. Furthermore, our results suggest a key insight: enhancing the reasoning capability of models is essential for improving the robustness of video understanding. COVER establishes a new standard for assessing MLLMs’ logical reasoning abilities in dynamic environments. Our work is available at https://github.com/gongyifan-hash/COVER-Benchmark.

1 Introduction

In recent years, the rapid advancement of large language models (LLMs) has spurred growing interest in multimodal large language models (MLLMs) (Hurst et al., 2024; Anthropic, 2024; Chen et al., 2024; Zhang et al., 2024a, 2025; Wang et al., 2024; Wu et al., 2024b). Various early benchmarks have been proposed to assess multimodal understanding ability of MLLMs, particularly in static images (Fu et al., 2023; Hudson and Manning, 2019; Liu et al., 2024; Yu et al., 2024). More recently, benchmarks involving complex images and dynamic videos have emerged to evaluate MLLM’s capabilities in temporal reasoning, spatio-temporal recognition, and object detection (Fu et al., 2024; Li et al., 2024b, 2023). Despite these advances, these benchmarks often overlook counterfactual reasoning, which is a critical component for evaluating inference in complex and realistic environments. As a result, they fall short of providing a comprehensive assessment of MLLMs’ reasoning capabilities.

Counterfactual reasoning, which posits hypothetical alternatives to observed realities, is pivotal for advanced video inference and is closely tied to out-of-distribution generalization (Yang et al., 2023; Bao et al., 2025). Previous work has attempted to construct counterfactual queries for images and videos (Li et al., 2024d,e,c; Patel et al., 2022; Wu et al., 2023). Most existing multimodal counterfactual benchmarks tend to focus on assessing subtask-specific robustness of reasoning ability (Li et al., 2024e; Wu et al., 2024c, 2023). However, they do not assess the underlying factors that contribute to the robustness of these reasoning capabilities. Such benchmarks often lack a systematic progression from abstract to concrete dimensions and from low-level perception to high-level cognition, limiting their ability to comprehensively capture multimodal reasoning processes in MLLMs. Furthermore, these benchmarks rarely investigate how robust video understanding interacts with stepwise reasoning in dynamic environments, leaving a gap in our assessment of advanced inference skills.

To bridge this gap, we propose COVER, a counterfactual video reasoning benchmark driven by a multidimensional abstraction level evaluation mechanism. Unlike existing multimodal counterfactual benchmarks, which often focus on multitask-oriented questions, COVER systematically classifies tasks into four quadrants. We define specific tasks for each quadrant to evaluate MLLMs’ diverse reasoning capabilities in complex video scenarios. Beyond merely posing counterfactual questions, COVER introduces a sub-question reasoning mechanism derived from necessary con-
Figure 1: An example from the COVER benchmark. The ground-truth answers are highlighted in green. All data—including original questions, counterfactual questions, sub-questions, and videos—have been manually verified as part of COVER. The diagram in the upper right corner illustrates the division of each COVER task into four quadrants.

ditions, enabling a deeper evaluation of performance across MLLMs. This approach allows us to establish a connection between the accuracy of intermediate steps and the overall robustness of counterfactual reasoning. As shown in Figure 1, when asked to determine whether a boy completes a series of actions in a specified order, COVER decomposes the problem into multiple steps, each representing a necessary condition. For instance, sub-question \( Q1 \) may inquire about which action occurs first in the reversed video, while sub-question \( Q2 \) targets the final action. This structured approach not only helps evaluate how a model adapts to event-sequence changes but also reveals its strengths and weaknesses in extracting and synthesizing critical information under counterfactual assumptions. By encompassing a broad range of abstraction levels, COVER stands as the most comprehensive dataset of its kind, paving the way for more rigorous and holistic evaluations of MLLMs’ dynamic and counterfactual reasoning capabilities.

Building on the COVER benchmark, we conducted a series of systematic experiments using both open-source and commercial closed-source models of varying scales. Our results indicate a strong positive correlation between the models’ sub-question accuracy and performance in counterfactual reasoning and robust video understanding. The findings underscore the tight linkage between sophisticated inference capabilities and high-level video comprehension. Furthermore, we examine how automatically generated versus human-guided sub-question decomposition (chain-of-thought, CoT (Wei et al., 2022)) influences complex reasoning and identifies the key factors impacting inference accuracy in MLLMs. Through these experiments, COVER offers valuable insights into how structured reasoning can enhance the robustness of video understanding by constructing a sub-question–based counterfactual video QA benchmark across multiple levels of abstraction and thoroughly evaluating mainstream MLLMs’ logical reasoning abilities.

2 Related Work

Multimodal Large Language Models and Their Evaluation. Recent advances in MLLMs have greatly improved their capacity to understand and reason over diverse modalities, such as images, text, and videos. To evaluate these models, benchmarks targeting static image comprehension have emerged, including MM-Vet (Yu et al., 2024), MME (Fu et al., 2023), MMBench (Liu et al., 2024), and GQA (Hudson and Manning, 2019). These primarily assess visual recognition and spatial reasoning. Extending beyond static content, video-centric benchmarks like Video-MME (Fu et al., 2024), MvBench (Li et al., 2024b), and
<table>
  <tr>
    <th>Benchmark</th>
    <th>Video</th>
    <th>Q&A</th>
    <th>Qs Source</th>
    <th>CF</th>
    <th>SQP</th>
    <th>PCD</th>
    <th>ACD</th>
  </tr>
  <tr>
    <td>CoFCA (Wu et al., 2024a)</td>
    <td>✗</td>
    <td>✓</td>
    <td>H&amp;A</td>
    <td>✓</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CFMM (Li et al., 2024e)</td>
    <td>✗</td>
    <td>✓</td>
    <td>H</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Video-MIME (Fu et al., 2024)</td>
    <td>✓</td>
    <td>✓</td>
    <td>H</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CRIPP-VQA (Patel et al., 2022)</td>
    <td>✓</td>
    <td>✓</td>
    <td>H</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>VITATECS (Li et al., 2024c)</td>
    <td>✓</td>
    <td>✗</td>
    <td>H&amp;A</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>COVER (ours)</td>
    <td>✓</td>
    <td>✓</td>
    <td>H&amp;A</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparison with existing benchmarks. Video: whether the benchmark involves video data; Q&A: whether it follows a question-and-answer format; Qs source: H indicates human annotation, A indicates automatic annotation; CF: whether counterfactual questions are included; PCD: whether the benchmark is categorized by the model’s perceptual and cognitive demands; ACD: whether tasks are divided based on object abstraction (abstract vs. concrete).

SEED-Bench (Li et al., 2023) focus on temporal dynamics and contextual reasoning. Together, these benchmarks reflect the growing demand for evaluating multimodal understanding in both static and dynamic environments.

Chain-of-Thought and Counterfactual Reasoning in MLLMs. Chain-of-Thought (CoT) reasoning (Wei et al., 2022) enhances logical inference by breaking down complex tasks into intermediate steps. Multimodal adaptations (Zhang et al., 2024b; Zheng et al., 2023) extend this strategy across modalities, showing gains in structured reasoning. Counterfactual reasoning, which examines hypothetical changes and their consequences, has also gained traction. Prior work explores this in text (Wu et al., 2024c,a), visual QA (Li et al., 2024e), and hybrid settings. ACQUIRED (Wu et al., 2023) proposes a taxonomy of counterfactual types, while AuroraCap (Chai et al., 2024) and CoFCA (Wu et al., 2024a) assess models’ sub-task decomposition and multi-step reasoning. These approaches collectively underscore the importance of structured, causal reasoning in complex multimodal tasks.

Multimodal Generalization and Video Counterfactual Benchmarks. Although several benchmarks target video-based counterfactual understanding—such as CRIPP-VQA for physical properties, VITATECS for captioning, and ACQUIRED for scenario taxonomy (Li et al., 2024c; Patel et al., 2022)—they remain narrow in scope. Most fail to capture the breadth of reasoning demands in real-world counterfactual scenarios.

To address this, COVER introduces a fine-grained framework for evaluating counterfactual video reasoning via sub-question decomposition. It explicitly distinguishes between abstract vs. concrete object attributes and perceptual vs. cognitive reasoning demands. As summarized in Table 1, COVER broadens the evaluation spectrum, enabling a more nuanced and comprehensive assessment of multimodal counterfactual reasoning than prior efforts.

3 The COVER Benchmark

![Overview of the 13 tasks in COVER. Numbers on the outer edge of the rose chart indicate the total number of question pairs for each task, while inner labels denote the corresponding dimension: A&C (Abstract Cognition), C&C (Concrete Cognition), A&P (Abstract Perception), and C&P (Concrete Perception).](page_1092_1047_368_368.png)

Figure 2: Overview of the 13 tasks in COVER. Numbers on the outer edge of the rose chart indicate the total number of question pairs for each task, while inner labels denote the corresponding dimension: A&C (Abstract Cognition), C&C (Concrete Cognition), A&P (Abstract Perception), and C&P (Concrete Perception).

This section provides a comprehensive overview of the construction of COVER. We introduce our data partitioning framework designed to evaluate MLLM reasoning ability across four complementary dimensions. Next, we describe the data curation process, which domain experts have rigorously validated to ensure the high quality and reliability of the benchmark.

Our benchmark includes approximately 2,800
videos, which are paired with around 12,000 to 13,000 individual QA instances. As stated in L- Figure 6, the enhanced version of our dataset consists of about 2.9k question pairs, with each pair comprising at least three individual QA items: one original question, one counterfactual question, and at least one sub-question (often multiple).

3.1 Benchmark Definition
As illustrated in Figure 2, we categorize the 13 benchmark tasks into four quadrants based on the abstract-concrete and perceptual-cognitive dimensions. Abstract-Perception: (1) Emotion: Understanding and recognizing emotional states. Concrete-Perception: (2) Counting: Quantity recognition and calculation. (3) Color: Perceiving object colors. (4) Direction: Sensing motion trends. (5) Size: Identifying object dimensions. (6) Shape: Perceiving object shapes. (7) Material: Recognizing object materials. (8) Location: Detecting object positions. Concrete-Cognition: (9) Action Recognition: Identifying specific actions. (10) Object Recognition: Recognizing specific objects. Abstract-Cognition: (11) Action Prediction: Forecasting future actions. (12) Procedure Understanding: Comprehending sequential processes and logic. (13) Social Relation: Understanding social relationships.

Division of Abstract and Concrete Scenes. This distinction reflects a functional differentiation within cognitive representation systems. Neuroscientific studies (Katja Wiemer-Hastings and Xu, 2005) suggest that concrete concepts rely heavily on multi-modal perceptual simulations (e.g., object shape, material), while abstract concepts are primarily represented through language-mediated symbolic operations. Abstract tasks often require integrating non-perceptual information, such as contextual encoding for emotion recognition or constructing temporal causal models for action prediction.

Division of Perception and Cognition. Perception involves the initial reception of external stimuli through sensory organs, converting them into neural signals that provide raw environmental data. Cognition, built upon perception, refers to the further processing of these signals, encompassing higher-level mental functions such as memory, attention, language comprehension, problem-solving, and reasoning. This distinction underscores different stages of information processing, with perception forming the foundation upon which cognitive functions are built.

3.2 Data Construction

![Bar chart showing distribution of quadrants](page_1012_180_377_246.png)

(a)

![Bar chart showing distribution of tasks](page_1012_420_377_246.png)

(b)

Figure 3: (a) Distribution of question pairs across the four quadrants. (b) Distribution of question pairs across the 13 tasks.

To construct COVER, we carefully selected a diverse range of open-source and research-available video sources, including Sigurdsson et al. (2016); Yi et al. (2020); Xie et al. (2024); Tan et al. (2020); Shahroudy et al. (2016); Pâtrăucean et al. (2023); Zhang et al. (2023); Gao et al. (2017); Jang et al. (2017); Wang et al. (2019); Krantz et al. (2020). These sources encompass various real-world scenarios, ranging from daily activity recognition to complex scene understanding. As shown in Appendix Figure 6, we collected 146 videos and designed 150 aspect-specific QA pairs, each of which underwent dual-team review for validation. To ensure balanced coverage across the four quadrants, we expanded the seed data using GPT-generated instances (720–760 per quadrant) to mitigate any potential biases. The detailed statistical findings are comprehensively presented in Figure 3. The frame count of videos in COVER ranges from 16 to 1739, with an average of 294.34 frames. We finally constructed 2,923 high-quality counterfactual question-answer pairs. Each question-answer pair consists of an original question, which presents no hypothetical context, and a counterfactual question, which introduces situational assumptions and sub-
<table>
  <tr>
    <th></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>70.26</td>
    <td>45.93</td>
    <td>56.94</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>67.32</td>
    <td>51.47</td>
    <td>55.94</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>63.60</td>
    <td>38.04</td>
    <td>49.40</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Pro</td>
    <td>74.82</td>
    <td>49.64</td>
    <td>63.76</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash</td>
    <td>73.90</td>
    <td>48.75</td>
    <td>62.52</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td><b>77.18</b></td>
    <td>46.90</td>
    <td>62.92</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td>76.74</td>
    <td><b>59.46</b></td>
    <td><b>67.23</b></td>
  </tr>
  <tr>
    <td>LlaVA-Video-72B</td>
    <td>64.35</td>
    <td>56.04</td>
    <td>61.54</td>
  </tr>
  <tr>
    <td>InternVL2.5-26B</td>
    <td>75.40</td>
    <td>51.08</td>
    <td>62.65</td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td>74.31</td>
    <td>57.75</td>
    <td>61.65</td>
  </tr>
  <tr>
    <td>VideoLLaMA3-8B</td>
    <td>73.04</td>
    <td>51.25</td>
    <td>60.09</td>
  </tr>
  <tr>
    <td>LlaVa-OV-7B</td>
    <td>62.74</td>
    <td>51.80</td>
    <td>56.42</td>
  </tr>
  <tr>
    <td>LlaVA-Video-7B</td>
    <td>60.52</td>
    <td>51.93</td>
    <td>55.11</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>71.83</td>
    <td>46.90</td>
    <td>58.40</td>
  </tr>
  <tr>
    <td>VILA-U-7B</td>
    <td>60.01</td>
    <td>38.42</td>
    <td>47.32</td>
  </tr>
  <tr>
    <td>VILA1.5-7B</td>
    <td>60.25</td>
    <td>57.34</td>
    <td>53.18</td>
  </tr>
</table>

Table 2: General assessment results of COVER. ori<sub>acc</sub>, c<sub>acc</sub>, and sub<sub>acc</sub> denote the accuracies of the original, counterfactual, and sub-questions, respectively.

questions that enable granular reasoning analysis.

Eight annotators further validated the dataset and checked logical consistency to ensure the reasoning relied solely on the video content. Additionally, three experts cross-validate the dataset (see Appendix Table 9) to confirm the structural balance.

4 Experiments

In this section, we systematically evaluate MLLMs of varying scales on the COVER benchmark to foster transparent and reproducible research. Our evaluation spans four key dimensions: cognition, perception, abstraction, and concreteness. It encompasses diverse reasoning sub-tasks, including counterfactual reasoning, direct inference, and sub-question-guided reasoning. We compare both open-source and proprietary models across different parameter scales to analyze their relative strengths and limitations. We begin by detailing the experimental setup.

4.1 Settings

To ensure a thorough evaluation, we selected a diverse set of representative MLLMs, including commercial closed-source models such as GPT-4o (Hurst et al., 2024), Claude (Anthropic, 2024), and Gemini (Reid et al., 2024), as well as leading open-source models such as InternVL2.5 (Chen et al., 2024), LLAVA-Video (Zhang et al., 2024a), LLaVA-OV (Li et al., 2024a), Qwen2-VL (Wang et al., 2024), VideoLLaMA3 (Zhang et al., 2025), vila-u (Wu et al., 2024b), and VILA (Lin et al., 2024). These models span a wide range of parameter scales and design paradigms, offering a comprehensive view of the current landscape in multimodal learning.

We evaluate model performance on video understanding using three metrics: ori<sub>acc</sub> (original question accuracy), c<sub>acc</sub> (counterfactual question accuracy), and sub<sub>acc</sub> (sub-question accuracy), with scores averaged over at least three runs. All models are tested under identical conditions, using a consistent frame extraction strategy that samples 16 frames per video segment. The impact of alternative sampling strategies is discussed in Chapter 5.

4.2 Main Results

As shown in Table 2, Gemini-2.0-Flash (ori<sub>acc</sub> 77.18%) and InternVL2.5-78B (ori<sub>acc</sub> 76.74%) rank as the top two models, demonstrating their strong foundational video understanding. The lower scores of VILA-U-7B (ori<sub>acc</sub> 60.01%) and LLaVA-Video-7B (ori<sub>acc</sub> 60.52%) highlight the limitations of smaller models in processing long sequences. InternVL2.5-78B (c<sub>acc</sub> 59.46%) shows significant dominance in handling conditional reasoning and complex contexts. Notably, counterfactual questions cause sharp accuracy drops compared to the original questions: GPT-4o (-24.33%) and Gemini-1.5-Pro (-25.18%), indicating that most models struggle with counterfactual reasoning.

Most models exhibit higher sub<sub>acc</sub> than c<sub>acc</sub> (e.g., Claude-3.5-Sonnet 49.40% vs. 38.04%, LLaVA-Video-72B 61.54% vs. 56.04%). This suggests better stability in localized reasoning tasks than in holistic tasks, where error accumulation impacts performance. In the Appendix, we provide detailed case demonstrations in Figure 8.

Closed-source Model Performance. As shown in Table 3, Gemini 1.5 Pro demonstrates strong dominance in both concrete cognition (ori<sub>acc</sub> 82.14%) and abstract perception tasks (ori<sub>acc</sub> 75.48%). Gemini 2.0 Flash excels in abstract perception (ori<sub>acc</sub> 75.90%) and concrete perception tasks (ori<sub>acc</sub> 74.22%), showcasing strong capabilities in handling high-complexity perceptual tasks.

Open-source Model Performance. As shown in Table 3, InternVL2.5-78B leads in abstract cognition (ori<sub>acc</sub> 72.88%) and concrete perception tasks (c<sub>acc</sub> 58.25%), reflecting a deep understanding of abstract concepts and complex logic. Lightweight models like Qwen2-VL-7B perform well in concrete cognition (ori<sub>acc</sub> 82.14%) but face limitations in abstract tasks (ori<sub>acc</sub> 65.96% in A&C) due to
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">A&C (%)</th>
    <th colspan="3">C&C (%)</th>
    <th colspan="3">C&P (%)</th>
    <th colspan="3">A&P (%)</th>
  </tr>
  <tr>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>face</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>face</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>face</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>face</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>71.05</td>
    <td>41.81</td>
    <td>41.70</td>
    <td>74.87</td>
    <td>43.65</td>
    <td>68.36</td>
    <td>69.95</td>
    <td>42.62</td>
    <td>50.52</td>
    <td>65.01</td>
    <td>55.65</td>
    <td>63.97</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>62.29</td>
    <td>52.40</td>
    <td>42.97</td>
    <td>76.32</td>
    <td>54.37</td>
    <td>66.49</td>
    <td>64.62</td>
    <td>40.85</td>
    <td>44.78</td>
    <td>65.56</td>
    <td><b>58.26</b></td>
    <td>65.96</td>
  </tr>
  <tr>
    <td>Claude-3.5-sonnet</td>
    <td>56.92</td>
    <td>37.01</td>
    <td>35.55</td>
    <td>70.11</td>
    <td>42.33</td>
    <td>61.77</td>
    <td>60.03</td>
    <td>32.88</td>
    <td>40.08</td>
    <td>66.94</td>
    <td>39.81</td>
    <td>56.81</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Pro</td>
    <td>69.49</td>
    <td>44.49</td>
    <td>53.36</td>
    <td>82.14</td>
    <td>51.98</td>
    <td>72.78</td>
    <td>71.76</td>
    <td>43.93</td>
    <td>56.81</td>
    <td>75.48</td>
    <td>57.99</td>
    <td>69.54</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>70.48</td>
    <td>45.34</td>
    <td>52.23</td>
    <td>82.01</td>
    <td>49.34</td>
    <td>71.51</td>
    <td>70.67</td>
    <td>42.02</td>
    <td>51.90</td>
    <td>72.04</td>
    <td><b>58.26</b></td>
    <td><b>71.36</b></td>
  </tr>
  <tr>
    <td>Gemini 2.0 Flash</td>
    <td><b>74.29</b></td>
    <td>44.36</td>
    <td>51.38</td>
    <td><b>83.99</b></td>
    <td>47.75</td>
    <td><b>72.84</b></td>
    <td>74.22</td>
    <td>38.74</td>
    <td>58.26</td>
    <td><b>75.90</b></td>
    <td>57.71</td>
    <td>66.84</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td>72.88</td>
    <td><b>59.60</b></td>
    <td><b>57.67</b></td>
    <td>80.95</td>
    <td>63.62</td>
    <td><b>75.62</b></td>
    <td>75.99</td>
    <td><b>58.25</b></td>
    <td><b>63.65</b></td>
    <td><b>76.86</b></td>
    <td>56.20</td>
    <td>70.07</td>
  </tr>
  <tr>
    <td>LLaVA-Video-72B</td>
    <td>53.11</td>
    <td>54.94</td>
    <td>53.14</td>
    <td>65.34</td>
    <td>60.45</td>
    <td>67.03</td>
    <td>67.94</td>
    <td>52.39</td>
    <td>53.49</td>
    <td>70.66</td>
    <td>56.20</td>
    <td>70.01</td>
  </tr>
  <tr>
    <td>InternVL2.5-26B</td>
    <td>71.05</td>
    <td>47.74</td>
    <td>50.53</td>
    <td>80.95</td>
    <td>58.99</td>
    <td>72.17</td>
    <td><b>76.13</b></td>
    <td>47.20</td>
    <td><b>60.12</b></td>
    <td>73.14</td>
    <td>50.00</td>
    <td>65.61</td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td>69.77</td>
    <td><b>58.62</b></td>
    <td>49.96</td>
    <td>80.95</td>
    <td><b>64.55</b></td>
    <td>71.02</td>
    <td>73.94</td>
    <td>55.80</td>
    <td>54.66</td>
    <td>72.18</td>
    <td>51.79</td>
    <td>68.19</td>
  </tr>
  <tr>
    <td>VideoLLama3-8B</td>
    <td>68.08</td>
    <td>45.62</td>
    <td>49.68</td>
    <td>81.35</td>
    <td>54.89</td>
    <td>68.36</td>
    <td>72.99</td>
    <td>50.75</td>
    <td>51.62</td>
    <td>69.28</td>
    <td>53.44</td>
    <td>67.90</td>
  </tr>
  <tr>
    <td>LLaVA-ov-7B</td>
    <td>54.66</td>
    <td>51.69</td>
    <td>47.49</td>
    <td>62.96</td>
    <td>53.04</td>
    <td>61.77</td>
    <td>64.53</td>
    <td>49.66</td>
    <td>49.48</td>
    <td>68.60</td>
    <td>52.75</td>
    <td>64.73</td>
  </tr>
  <tr>
    <td>LLaVA-Video-7B</td>
    <td>50.14</td>
    <td>55.23</td>
    <td>44.52</td>
    <td>61.64</td>
    <td>50.53</td>
    <td>60.50</td>
    <td>63.57</td>
    <td>52.52</td>
    <td>49.97</td>
    <td>66.39</td>
    <td>49.59</td>
    <td>63.03</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>65.96</td>
    <td>49.15</td>
    <td>48.41</td>
    <td>82.14</td>
    <td>43.39</td>
    <td>67.03</td>
    <td>71.21</td>
    <td>45.57</td>
    <td>50.52</td>
    <td>67.49</td>
    <td>49.72</td>
    <td>65.02</td>
  </tr>
  <tr>
    <td>VILA-U-7B</td>
    <td>58.19</td>
    <td>39.83</td>
    <td>38.87</td>
    <td>63.10</td>
    <td>41.93</td>
    <td>54.51</td>
    <td>59.07</td>
    <td>37.93</td>
    <td>37.94</td>
    <td>59.50</td>
    <td>33.88</td>
    <td>55.34</td>
  </tr>
  <tr>
    <td>VILA1.5-7B</td>
    <td>54.80</td>
    <td>55.93</td>
    <td>39.29</td>
    <td>66.93</td>
    <td>62.30</td>
    <td>63.52</td>
    <td>55.25</td>
    <td><b>58.53</b></td>
    <td>44.64</td>
    <td>63.64</td>
    <td>52.34</td>
    <td>61.91</td>
  </tr>
</table>

Table 3: Performance of MLLMs on COVER, based on our quadrant formulation (A&C, C&C, C&P, A&P), measured by original, counterfactual, and sub-question accuracy.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Without CoT</th>
    <th colspan="2">With CoT</th>
    <th colspan="2">Guide-CoT</th>
  </tr>
  <tr>
    <th>c<sub>face</sub></th>
    <th>c<sub>face</sub></th>
    <th>c<sub>face</sub></th>
    <th>c<sub>face</sub></th>
    <th>c<sub>face</sub></th>
    <th>c<sub>face</sub></th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>51.47</td>
    <td>58.62</td>
    <td>57.93</td>
    <td>68.07</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td><b>59.46</b></td>
    <td><b>60.42</b></td>
    <td><b>58.33</b></td>
    <td><b>68.29</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVA-Video-72B</td>
    <td>56.04</td>
    <td>56.24</td>
    <td>53.51</td>
    <td>63.12</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td><b>57.75</b></td>
    <td>57.06</td>
    <td>52.41</td>
    <td>57.75</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>VideoLLama3-8B</td>
    <td>51.25</td>
    <td>52.82</td>
    <td>53.06</td>
    <td>52.79</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVA-Video-7B</td>
    <td>51.93</td>
    <td>51.42</td>
    <td>51.39</td>
    <td>54.12</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>46.90</td>
    <td>50.36</td>
    <td>45.71</td>
    <td>50.88</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Comparison between CoT and Guide-CoT performance across MLLMs on the COVER benchmark.

their smaller parameter size, revealing distinct capabilities across model types. Commercial models, such as the Gemini series, maintain strong performance in concrete cognition and abstract perception tasks but generally fall behind open-source models in counterfactual reasoning. Most models struggle with counterfactual reasoning, with only InternVL2.5-78B and VILA1.5-7B showing some task-specific advantages, highlighting the need for targeted optimization in conditional hypothesis modeling.

4.3 Sub-question Guideline

We propose Guide-CoT to study the influence of different reasoning paths on model performance through human-annotated sub-problems. We design comparative experiments between CoT and Guide-CoT to analyze how automatically generated sub-questions from CoT versus manually annotated sub-questions affect model reasoning capabilities.

Comparing the Without CoT and CoT approaches based on Table 4, we find that the \( c_{face} \) of most models under CoT significantly exceeds the Without CoT baseline, such as Qwen2-VL-7B (+3.46%) and GPT-4o-mini (+7.15%), which indicates that CoT enhances reasoning processes, particularly in more complex tasks.

However, examining Guide-CoT results reveals that manually designed sub-questions may not always lead to substantial improvement over automatically generated ones, as seen with GPT-4o-mini’s \( c_{face} \) of 57.93% under Guide-CoT, slightly lower than the 58.62% under CoT. This does not imply the ineffectiveness of manual sub-questions but suggests that model behaviors may not always align with human-designed reasoning paths, potentially due to task complexity or the nature of the sub-questions themselves. We hypothesize that manually provided sub-questions could introduce extraneous patterns or "pseudo-features" that are not directly relevant to the reasoning task, leading to a subtle reduction in performance.

The \( c_{face} \) column in Guide-CoT indicates sub-questions that include standard answers. For
<table>
  <tr>
    <th rowspan="2">Frames</th>
    <th colspan="3">InternVL2.5-1B</th>
    <th colspan="3">InternVL2.5-2B</th>
    <th colspan="3">InternVL2.5-4B</th>
    <th colspan="3">InternVL2.5-8B</th>
  </tr>
  <tr>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>facc</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>facc</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>facc</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>facc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>2</td>
    <td>66.16</td>
    <td>35.61</td>
    <td>55.27</td>
    <td>65.31</td>
    <td>44.20</td>
    <td>54.99</td>
    <td>72.56</td>
    <td>48.31</td>
    <td>60.88</td>
    <td>71.26</td>
    <td>58.50</td>
    <td>60.07</td>
  </tr>
  <tr>
    <td>4</td>
    <td>68.32</td>
    <td>34.72</td>
    <td>55.52</td>
    <td>68.83</td>
    <td>42.11</td>
    <td>58.84</td>
    <td>74.41</td>
    <td>46.49</td>
    <td>61.79</td>
    <td>73.35</td>
    <td>58.47</td>
    <td>60.96</td>
  </tr>
  <tr>
    <td>8</td>
    <td>68.94</td>
    <td>35.10</td>
    <td>55.11</td>
    <td>68.22</td>
    <td>41.43</td>
    <td>55.75</td>
    <td>75.03</td>
    <td>45.60</td>
    <td>61.79</td>
    <td>74.14</td>
    <td>57.06</td>
    <td>61.60</td>
  </tr>
  <tr>
    <td>16</td>
    <td>69.76</td>
    <td>35.89</td>
    <td>55.19</td>
    <td>70.07</td>
    <td>40.68</td>
    <td>55.49</td>
    <td>75.61</td>
    <td>45.23</td>
    <td>61.63</td>
    <td>74.31</td>
    <td>57.75</td>
    <td>61.65</td>
  </tr>
  <tr>
    <td>32</td>
    <td>69.04</td>
    <td>36.50</td>
    <td>55.04</td>
    <td>70.13</td>
    <td>39.69</td>
    <td>55.48</td>
    <td>75.54</td>
    <td>45.09</td>
    <td>60.96</td>
    <td>74.10</td>
    <td>57.03</td>
    <td>61.42</td>
  </tr>
  <tr>
    <td>64</td>
    <td>68.18</td>
    <td>37.39</td>
    <td>54.80</td>
    <td>68.90</td>
    <td>40.06</td>
    <td>55.44</td>
    <td>74.41</td>
    <td>46.56</td>
    <td>60.70</td>
    <td>74.20</td>
    <td>58.09</td>
    <td>61.30</td>
  </tr>
</table>

Table 5: Performance of MLLMs on COVER using different frame sampling strategies. The frame selection follows standard practices in video QA benchmarks, where the number of input frames is set to min(video length, predefined sampling count).

InternVL2.5-78B, c<sub>fwithans</sub> reaches 68.29%, reflecting an 8.63% improvement over the no-CoT baseline, in contrast to CoT’s modest gain of only 0.96% (from 59.46% to 60.42%). This suggests that providing complete answers substantially enhances reasoning accuracy, particularly in complex or multi-step tasks. Standard-answer sub-questions enable the model to better integrate information and verify intermediate reasoning steps, resulting in improved consistency and overall performance. Detailed case studies are presented in Appendix Figure 9 to further illustrate these findings and analyze the interplay between reasoning paths and task complexity.

The results from our experiments strongly support the notion that reasoning plays a pivotal role in model robustness and generalization. Our study extends these insights by demonstrating that multimodal models, especially in the context of video tasks, rely heavily on robust reasoning capabilities for effective generalization. The significant performance improvements observed with counterfactual reasoning and sub-question decomposition highlight that models’ ability to handle complex, conditional, and dynamic contexts is crucial for their robustness, a finding not fully explored in prior research.

5 Analysis

In this chapter, we begin by analyzing the impact of video frame sampling rates on MLLMs’ video understanding and reasoning abilities. We then proceed with an in-depth examination of MLLMs’ robustness and logical reasoning performance.

5.1 Ablation Study of Video Frames

As shown in Table 5, as the parameter size of LLMs increases, there is a rising trend in ori<sub>acc</sub>, c<sub>facc</sub>, and sub<sub>acc</sub>. For instance, with 16 frames, the InternVL2.5-1B model achieves ori<sub>acc</sub>, c<sub>facc</sub>, and sub<sub>acc</sub> of 69.76%, 35.89%, and 55.19% respectively. The InternVL2.5-2B scores 70.07%, 40.62%, and 55.49%, while the InternVL2.5-4B reaches 75.61%, 45.23%, and 61.68%, indicating that larger LLMs have enhanced capabilities in handling complex problems. Under the same vision tower settings, ori<sub>acc</sub> shows a clear upward trend as the number of frames increases. For example, the InternVL2.5-8B’s ori<sub>acc</sub> rises from 71.26% at 2 frames to 74.20% at 64 frames. However, c<sub>facc</sub> tends to decrease with more frames. The InternVL2.5-2B’s c<sub>facc</sub> drops from 44.20% at 2 frames to 40.06% at 64 frames. Models with more parameters generally perform better in ori<sub>acc</sub>, c<sub>facc</sub>, and sub<sub>acc</sub>, highlighting the significant role of LLMs in multimodal reasoning. Additionally, increasing visual information (by raising the frame count) can enhance ori<sub>acc</sub>, but excessive visual information, especially in complex or counterfactual reasoning scenarios, may impair the model’s reasoning ability, leading to a decline in c<sub>facc</sub>.

5.2 Robustness and Logical Reasoning in MLLMs

The ability of MLLMs to answer original questions serves as a key indicator of their overall understanding capabilities, while performance on sub-questions reveals single-step reasoning proficiency. A notable observation is the strong Pearson correlation between ori<sub>acc</sub> and sub<sub>acc</sub> reaches 0.836, indicating a strong connection between model understanding and reasoning capabilities. Furthermore, as shown in Figure 5, the correlation between sub<sub>acc</sub> and c<sub>facc</sub> is 0.608. These moderately strong correlations indicate that a model’s ability to comprehend original questions plays a fundamental role in enabling effective step-by-step reasoning. Similarly, the correlation between ori<sub>acc</sub> and sub<sub>acc</sub> suggests that models with a higher understanding
Figure 4: Heatmaps of task performance for Gemini-1.5-pro and InternVL2.5-78B, using hollow circles to depict task distributions across the four quadrants. The top three panels show results for Gemini-1.5-pro, and the bottom three for InternVL2.5-78B. Left: Accuracy on original questions. Middle: Performance on counterfactual questions. Right: Accuracy on sub-questions. A gradient color bar—from azure (low accuracy) to crimson (high accuracy)—is placed along the right margin of each heatmap to indicate performance levels.

<table>
  <tr>
    <th>Model</th>
    <th>P(cf_right|sub_right)</th>
    <th>P(cf_wrong|sub_right)</th>
    <th>P(cf_right|sub_wrong)</th>
    <th>P(cf_wrong|sub_wrong)</th>
  </tr>
  <tr>
    <td>gemini-1.5-pro</td>
    <td>56.54</td>
    <td><b>43.45</b></td>
    <td>44.99</td>
    <td><b>55.01</b></td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>59.49</td>
    <td>40.51</td>
    <td>47.65</td>
    <td>52.35</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td>62.90</td>
    <td>37.10</td>
    <td><b>56.67</b></td>
    <td>43.34</td>
  </tr>
  <tr>
    <td>LlaVA-Video-72B</td>
    <td><b>63.28</b></td>
    <td>36.72</td>
    <td>51.60</td>
    <td>48.40</td>
  </tr>
</table>

Table 6: Conditional probabilities of counterfactual accuracy given sub-question outcomes. P(cf_right | sub_right) and P(cf_wrong | sub_right) denote the likelihood of answering the counterfactual question correctly or incorrectly when the sub-question is correct; similarly, P(cf_right | sub_wrong) and P(cf_wrong | sub_wrong) apply when the sub-question is incorrect.

capability tend to perform better when solving decomposed sub-questions, reinforcing the notion that comprehension and reasoning are interdependent. However, the moderate correlation between sub_acc and cf_acc suggests that counterfactual reasoning involves additional complexities, making it a more challenging task than single-step reasoning.

As illustrated in Table 6, We observed that across multiple models, the probability \( P(cf_{right}|sub_{right}) \) was significantly higher than \( P(cf_{right}|sub_{wrong}) \), clearly indicating that the correctness of sub-questions is a strong predictor of overall counterfactual performance.

Analysis of the heat maps in Figure 4 reveals different performance patterns in the quadrants, highlighting the interaction between comprehension, step-by-step reasoning, and counterfactual inference. In abstract reasoning tasks such as social inference and procedural understanding, the drop from sub_acc to ori_acc is minimal, and the transition to cf_acc remains stable. This suggests that models can effectively leverage sub-question reasoning and maintain performance even under counterfactual assumptions. In contrast, the concrete perception quadrant—involving tasks like object recognition and motion understanding—shows a sharper decline from sub_acc to ori_acc, and further to cf_acc. This indicates that perception-heavy tasks pose greater challenges, as models struggle to decompose complex sensory input into reasoning steps
Figure 5: Scatter plot showing correlations among \( ori_{acc} \), \( sub_{acc} \), and \( cf_{acc} \) across models. The red line represents the linear function fitted between \( ori_{acc} \) and \( sub_{acc} \), while the purple line represents the linear function fitted between \( cf_{acc} \) and \( sub_{acc} \).

required for counterfactual understanding.

Overall, our findings indicate that counterfactual reasoning is inherently more challenging than single-step reasoning, especially in perception-intensive tasks where models must infer causality beyond pattern recognition. In contrast, the relatively stable gap between \( sub_{acc} \) and \( cf_{acc} \) in abstract-cognitive tasks suggests that models can better leverage conceptual knowledge. Enhancing counterfactual reasoning in perception-heavy scenarios remains a key challenge, likely requiring improved causal inference and reasoning mechanisms.

5.3 The Effects of Model Scale

We conduct systematic analyses to characterize performance gaps across original, counterfactual, and sub-question accuracies. Our goal is to mitigate these gaps by examining factors such as model scale, training alignment, and reasoning strategies. As shown in Table 7, with similar visual backbones, increasing language model size significantly reduces the performance gap—particularly between sub-question and counterfactual accuracy. Specifically, the absolute difference between \( ori_{acc} \) (70.07%) and \( cf_{acc} \) (40.68%) is 29.39% for the 2B model, increases slightly to 30.38% for the 4B model, and then drops substantially to 16.56% for the 8B model. Similarly, the gap between \( cf_{acc} \) and \( sub_{acc} \) grows from 14.81% (2B) to 16.40% (4B), before narrowing sharply to 3.90% (8B).

<table>
  <tr>
    <th>Model</th>
    <th>ori<sub>acc</sub></th>
    <th>cf<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>InternVL2.5-2B</td>
    <td>70.07</td>
    <td>40.68</td>
    <td>55.49</td>
  </tr>
  <tr>
    <td>InternVL2.5-4B</td>
    <td>75.61</td>
    <td>45.23</td>
    <td>61.63</td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td>74.31</td>
    <td>57.75</td>
    <td>61.65</td>
  </tr>
</table>

Table 7: Variations in three accuracy metrics across different model sizes.

6 Conclusion

We introduce COVER, a comprehensive benchmark for counterfactual video reasoning that evaluates MLLMs across four dimensions: abstract-concrete and perception-cognition. By decomposing complex queries into structured sub-questions, COVER enables fine-grained analysis and reveals a strong correlation between sub-question accuracy and overall reasoning performance. Our results highlight the need for improved reasoning abilities in dynamic video tasks, and position COVER as a new standard for evaluating multimodal logical reasoning.
Acknowledgments

We would like to thank the anonymous reviewers for their valuable feedback. We thank Junshu Pan, Panzhong Lu, Fang Guo, Zijie Yang, Pai Liu, and other global collaborators for their valuable discussions and help. This work is funded by the National Natural Science Foundation of China Key Program (Grant No. 62336006), the Pioneer and “Leading Goose” R&D Program of Zhejiang (Grant No. 2022SDXHDX0003), and the Research Program (Grant No. WU2023C020) of the Research Center for Industries of the Future, Westlake University.

Limitations

COVER offers a novel benchmark for counterfactual video reasoning, but some limitations exist. First, while it focuses on video reasoning, its applicability to other multimodal tasks, such as image or text reasoning, remains unexplored. Second, COVER relies on sub-question decomposition, and automated methods may not always match human-designed questions, especially in complex scenarios. Finally, while we demonstrate COVER’s effectiveness on various models, further validation across different model architectures and real-world tasks is needed to assess its generalizability.

Ethical Considerations

COVER is designed with ethical considerations in mind, aiming to enhance counterfactual reasoning in video understanding while ensuring fairness, transparency, and responsible AI development. We acknowledge the ongoing challenges in bias mitigation, fairness, and environmental sustainability and encourage the broader research community to collaborate in addressing these concerns. By establishing COVER as an open and structured evaluation benchmark, we aim to promote robust and ethical AI advancements in multimodal reasoning.

We ensured that the human annotators were compensated with fair remuneration, which exceeded the local minimum wage standards, reflecting the value of their work. Furthermore, we took steps to ensure that the annotation process did not pose any risks to their physical or mental well-being. The tasks were designed to be manageable, and we provided adequate support to ensure a safe and respectful working environment.

In this study, AI was used solely for data augmentation and grammar/typo correction, with no involvement in generative or creative tasks. We carefully considered potential risks to ensure AI usage did not compromise the originality or transparency of the research.

References

AI Anthropic. 2024. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card.

Guangsheng Bao, Hongbo Zhang, Cuxiang Wang, Linyi Yang, and Yue Zhang. 2025. How likely do LLMs with CoT mimic human reasoning? In Proceedings of the 31st International Conference on Computational Linguistics, pages 7831–7850, Abu Dhabi, UAE. Association for Computational Linguistics.

Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Saining Xie, and Christopher D. Manning. 2024. Auroracap: Efficient, performant video detailed captioning and a new benchmark. CoRR, abs/2410.03051.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jingguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR, abs/2412.05271.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhengyu Qiu, Wei Lin, Jinrui Yang, Xiaow Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394.

Chaoyou Fu, Yuhai Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renhui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiaow Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. CoRR, abs/2405.21075.

Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. TALL: temporal activity localization via language query. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5277–5285. IEEE Computer Society.

Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700–6709. Computer Vision Foundation / IEEE.

Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Alexander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Alison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. 2024. Gpt-4o system card. CoRR, abs/2410.21276.

Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: toward spatio-temporal reasoning in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1359–1367. IEEE Computer Society.

Katja Katja Wiemer-Hastings and Xu Xu. 2005. Content differences for abstract and concrete concepts. Cognitive science, 29(5):719–736.

Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII, volume 12373 of Lecture Notes in Computer Science, pages 104–120. Springer.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. CoRR, abs/2307.16125.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. 2024b. Mybench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 22195–22206. IEEE.

Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. 2024c. VITATECS: A diagnostic dataset for temporal concept understanding of video-language models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXX, volume 15128 of Lecture Notes in Computer Science, pages 331–348. Springer.

Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, and Yu-Gang Jiang. 2024d. Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models. CoRR, abs/2404.12966.

Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, Na Zhao, and Yu-Gang Jiang. 2024e. Look before you decide: Prompting active deduction of mlms for assumptive reasoning. Preprint, arXiv:2404.12966.

Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. VILA: on pre-training for visual language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26679–26689. IEEE.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. Mmbench: Is your multi-modal model an all-around player? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VI, volume 15064 of Lecture Notes in Computer Science, pages 216–233. Springer.

Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. 2022. CRIPP-VQA: counterfactual reasoning about implicit physical properties via video question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9856–9870. Association for Computational Linguistics.

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continent, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon
Osindero, Dima Damen, Andrew Zisserman, and João Carreira. 2023. Perception test: A diagnostic benchmark for multimodal video models. Preprint, arXiv:2305.13786.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soriciut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankhesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530.

Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. 2016. NTU RGB+D: A large scale dataset for 3d human activity analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 1010–1019. IEEE Computer Society.

Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, volume 9905 of Lecture Notes in Computer Science, pages 510–526. Springer.

Ganchao Tan, Daqing Liu, Meng Wang, and Zheng-Jun Zha. 2020. Learning to discretely compose reasoning module networks for video captioning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 745–752. ijcai.org.

Qwen Team. 2024. Qvq: To see the world with wisdom.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. CoRR, abs/2409.12191.

Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4580–4590. IEEE.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichtler, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. 2024a. Cofca: A step-wise counterfactual multi-hop qa benchmark. Preprint, arXiv:2402.11924.

Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischelbed, and Nanyun Peng. 2023. ACQUIRED: A dataset for answering counterfactual questions in real-life videos. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11753–11770, Singapore. Association for Computational Linguistics.

Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. 2024b. VILA-U: a unified foundation model integrating visual understanding and generation. CoRR, abs/2409.04429.

Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najuong Kim, Jacob Andreas, and Yoon Kim. 2024c. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1819–1862, Mexico City, Mexico. Association for Computational Linguistics.

Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. 2024. [inline-graphic not available: see fulltext] funqa: Towards surprising video comprehension. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part I, volume 15059 of Lecture Notes in Computer Science, pages 39–57. Springer.

Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanneng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. GLUE-X: Evaluating natural language understanding models from an out-of-distribution generalization perspective. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12731–12750, Toronto, Canada. Association for Computational Linguistics.

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jianjun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2020. Cleverer: Collision events for video representation and reasoning. Preprint, arXiv:1910.01442.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan
Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. CoRR, abs/2501.13106.

Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. 2023. Movqa: A benchmark of versatile question-answering for long-form movie understanding. Preprint, arXiv:2312.04817.

Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024a. Video instruction tuning with synthetic data. CoRR, abs/2410.02713.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2024b. Multimodal chain-of-thought reasoning in language models. Preprint, arXiv:2302.00923.

Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibe Yang. 2023. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

A Appendix

A.1 Data Construction Details

In this section, we present additional details on COVER construction, including information about the task splitting scores, annotation agreements, data augmentation prompts and process flow.

We invited three expert annotators to independently score each benchmark task based on our two-dimensional quadrant framework (abstract vs. concrete and perception vs. cognition). Their scoring results in Table 8 demonstrates the strictness, consistency, and logical coherence of our task categorization, effectively preventing overlaps and ambiguity.

The annotators were recruited to evaluate COVER across multiple dimensions, with the resultant assessments systematically compiled in Table 9, ensuring comprehensive evaluation coverage. The methodological workflow for data augmentation is schematically outlined in Figure 6.

The schematic framework outlined in Figure 7 delineates the methodology employed for contextual data augmentation, leveraging the generative capabilities of GPT-4o(Hurst et al., 2024) to construct domain-specific instructional prompts.

A.2 Additional Results

In this section, we present additional experiments on COVER. The comprehensive evaluation framework delineated in Table 14 presents granular performance metrics across 13 meticulously defined tasks.

GPT-4o exhibited notable vulnerability in the Procedure Understanding task. While it attained a respectable raw accuracy of 78.17%, its counterfactual accuracy plummeted to 28.97%, representing a precipitous decline of 49.2%. This substantial drop suggests that the performance of GPT-4o in understanding procedures may be overly reliant on surface-level features. Counterfactual perturbations, such as changes in conditions, can severely disrupt its reasoning capabilities, thereby highlighting a robustness limitation of the model when handling complex tasks.

Figure 5 (a) depicts the relationship between ori\(_{acc}\) and sub\(_{acc}\) across different models, with a purple regression line characterizing the functional correlation between mean ori\(_{acc}\) and mean sub\(_{acc}\). Figure 5 (b) demonstrates the association between cf\(_{acc}\) and sub\(_{acc}\) across different models, with a red regression line characterizing the functional correlation between mean cf\(_{acc}\) and mean sub\(_{acc}\). The bivariate correlation analysis delineated in Figure 5 demonstrates statistically significant covariation patterns (r = 0.836) between semantic comprehension and multi-step reasoning capabilities in MLLMs.

We conducted an additional ablation study to examine whether the observed trend where excessive visual information impairs reasoning accuracy holds consistently across both short and long videos. Our results are summarized in Table 10, 11. We observed a clear pattern across both short and long videos: model accuracy typically peaks within a moderate frame range (8–32 frames) and subsequently declines at the maximum setting (64 frames). This decline is particularly pronounced in tasks involving the original questions (ori) and sub questions (sub), suggesting that an excessive amount of visual input can indeed negatively impact model performance, regardless of video length.
<table>
  <tr>
    <th>Task</th>
    <th>A_x</th>
    <th>A_y</th>
    <th>B_x</th>
    <th>B_y</th>
    <th>C_x</th>
    <th>C_y</th>
    <th>Avg_x</th>
    <th>Avg_y</th>
  </tr>
  <tr>
    <td>Counting</td>
    <td>-3.2</td>
    <td>-3.4</td>
    <td>-3.1</td>
    <td>-3.6</td>
    <td>-3.3</td>
    <td>-3.7</td>
    <td>-3.2</td>
    <td>-3.57</td>
  </tr>
  <tr>
    <td>Color</td>
    <td>-4.1</td>
    <td>-4.4</td>
    <td>-4.4</td>
    <td>-4.2</td>
    <td>-4.2</td>
    <td>-4.3</td>
    <td>-4.23</td>
    <td>-4.3</td>
  </tr>
  <tr>
    <td>Material</td>
    <td>-3.8</td>
    <td>-3.3</td>
    <td>-3.9</td>
    <td>-3.2</td>
    <td>-4.0</td>
    <td>-3.4</td>
    <td>-3.9</td>
    <td>-3.3</td>
  </tr>
  <tr>
    <td>Size</td>
    <td>-2.4</td>
    <td>-2.5</td>
    <td>-2.6</td>
    <td>-2.3</td>
    <td>-2.2</td>
    <td>-2.4</td>
    <td>-2.4</td>
    <td>-2.4</td>
  </tr>
  <tr>
    <td>Shape</td>
    <td>-3.3</td>
    <td>-3.2</td>
    <td>-3.5</td>
    <td>-3.2</td>
    <td>-3.8</td>
    <td>-4.0</td>
    <td>-3.53</td>
    <td>-3.47</td>
  </tr>
  <tr>
    <td>Emotion</td>
    <td>-2.4</td>
    <td>4.0</td>
    <td>-2.5</td>
    <td>3.5</td>
    <td>-2.4</td>
    <td>3.1</td>
    <td>-2.43</td>
    <td>3.53</td>
  </tr>
  <tr>
    <td>Location</td>
    <td>-1.7</td>
    <td>-1.4</td>
    <td>-2.0</td>
    <td>-1.6</td>
    <td>-1.3</td>
    <td>-1.7</td>
    <td>-1.67</td>
    <td>-1.57</td>
  </tr>
  <tr>
    <td>Direction</td>
    <td>-2.1</td>
    <td>-1.7</td>
    <td>-2.5</td>
    <td>-1.5</td>
    <td>-2.6</td>
    <td>-1.8</td>
    <td>-2.4</td>
    <td>-1.67</td>
  </tr>
  <tr>
    <td>Object Recognition</td>
    <td>3.0</td>
    <td>-3.0</td>
    <td>2.4</td>
    <td>-2.0</td>
    <td>1.2</td>
    <td>-2.3</td>
    <td>2.2</td>
    <td>-2.43</td>
  </tr>
  <tr>
    <td>Action Recognition</td>
    <td>2.5</td>
    <td>-3.1</td>
    <td>2.3</td>
    <td>-3.0</td>
    <td>2.1</td>
    <td>-3.5</td>
    <td>2.3</td>
    <td>-3.2</td>
  </tr>
  <tr>
    <td>Action Prediction</td>
    <td>3.9</td>
    <td>2.4</td>
    <td>3.8</td>
    <td>2.5</td>
    <td>3.2</td>
    <td>2.2</td>
    <td>3.63</td>
    <td>2.37</td>
  </tr>
  <tr>
    <td>Procedure Understanding</td>
    <td>3.0</td>
    <td>3.5</td>
    <td>3.6</td>
    <td>3.2</td>
    <td>2.2</td>
    <td>3.3</td>
    <td>2.93</td>
    <td>3.33</td>
  </tr>
  <tr>
    <td>Social Relation</td>
    <td>3.4</td>
    <td>4.3</td>
    <td>3.0</td>
    <td>4.4</td>
    <td>3.1</td>
    <td>4.1</td>
    <td>3.17</td>
    <td>4.27</td>
  </tr>
</table>

Table 8: Annotator scoring table. Annotators A, B, and C provide ratings along two axes: the perceptual–cognitive dimension (x-axis, from −5 to 5, where higher values indicate more cognitive tasks) and the concrete–abstract dimension (y-axis, from −5 to 5, where higher values indicate more abstract tasks).

![Flowchart depicting the data augmentation pipeline.](page_370_682_1002_627.png)

Figure 6: Flowchart depicting the data augmentation pipeline.

Additionally, we evaluated test-time reasoning strategies on manually curated seed data using long-chain reasoning models in Table 12. Notably, models such as InternVL2.5-78B-CoT show significant improvement in bridging the cf–sub–ori gap, further supporting that reasoning-guided prompting (e.g., CoT) helps align sub-level and cf-level accuracy. These observations suggest a promising direction: larger and better-aligned models, when combined with explicit reasoning strategies, are more capable of maintaining coherence across perception, decomposition, and abstract reasoning tasks.
<table>
  <tr>
    <th colspan="2">Enlarge Prompt</th>
  </tr>
  <tr>
    <td colspan="2">
      System: You are a professional video Q&A data annotator. Your task is to generate question-answer pairs based on video frames while adhering to strict formatting rules and avoiding information leakage.
    </td>
  </tr>
  <tr>
    <td colspan="2">
      User: Below is your task description and examples provided.
    </td>
  </tr>
  <tr>
    <td colspan="2">
      Required Q&A Types<br>
      original_qa: A question focused on [aspect] with multiple-choice options (ABCD).<br>
      counterfactual_qa: Identical to original_qa but adds one carefully constructed if-clause.<br>
      sub_qas: At least two sub-questions addressing intermediate reasoning steps for counterfactual_qa, also with multiple-choice options.
    </td>
  </tr>
  <tr>
    <td colspan="2">
      Rules<br>
      All questions must strictly follow {aspect_rules[aspect]}.<br>
      No information leakage: Questions must not reveal answers without analyzing the video.<br>
      Choices: Must have semantic distinctions; avoid overlapping or ambiguous options.<br>
      Format consistency: original_qa and counterfactual_qa share identical choices; sub_qas inherit the if-clause.
    </td>
  </tr>
  <tr>
    <td>Expected Output Format</td>
    <td>
      original: "qs": "...",<br>
      counterfactual: "qs": "if <clause>, ...?",<br>
      sub: "qs": "if <clause>, ..."
    </td>
  </tr>
  <tr>
    <td>Notes:</td>
    <td>
      Use double quotes for JSON keys/values.<br>
      Ensure the if-clause in counterfactual_qa and sub_qas is logically compatible with the question body.
    </td>
  </tr>
  <tr>
    <td>In-Context Examples</td>
    <td>
      original: "qs": "...",<br>
      counterfactual: "qs": "if <clause>, ...?"<br>
      sub: "qs": "if <clause>, ..."
    </td>
  </tr>
  <tr>
    <td>Action Instructions</td>
    <td>Analyze Video Frames: Extract key visual elements relevant to (aspect).<br>
      Generate Q&A Triplets: Follow the format and rules strictly.<br>
      Validate: Ensure no information leakage and logical consistency.
    </td>
  </tr>
  <tr>
    <td>Final Output:</td>
    <td><Your generated JSON here></td>
  </tr>
</table>

Figure 7: Methodological framework for data augmentation using GPT-4o.

<table>
  <tr>
    <th>Aspect</th>
    <th>A</th>
    <th>B</th>
    <th>C</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Data Quality</td>
    <td>4</td>
    <td>4</td>
    <td>5</td>
    <td>4.3</td>
  </tr>
  <tr>
    <td>Data Diversity</td>
    <td>5</td>
    <td>4</td>
    <td>5</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>Relevance</td>
    <td>4</td>
    <td>5</td>
    <td>4</td>
    <td>4.3</td>
  </tr>
  <tr>
    <td>Annotation Quality</td>
    <td>4</td>
    <td>5</td>
    <td>5</td>
    <td>4.7</td>
  </tr>
  <tr>
    <td>Dataset Usability</td>
    <td>4</td>
    <td>4</td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Innovation</td>
    <td>5</td>
    <td>5</td>
    <td>4</td>
    <td>4.7</td>
  </tr>
</table>

Table 9: Cross-annotator validation on COVER. The table summarizes quality scores assigned by three annotators. A, B, and C denote randomly assigned codes for the assessment data, and Average indicates the mean score across all entries.

<table>
  <tr>
    <th rowspan="2">Frame</th>
    <th colspan="3">InternVL2.5-4B</th>
    <th colspan="3">InternVL2.5-8B</th>
  </tr>
  <tr>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>2</td>
    <td>69.09</td>
    <td>45.94</td>
    <td>60.33</td>
    <td>68.72</td>
    <td>56.53</td>
    <td>61.14</td>
  </tr>
  <tr>
    <td>4</td>
    <td>70.81</td>
    <td>46.18</td>
    <td>60.91</td>
    <td>68.97</td>
    <td>56.90</td>
    <td>60.68</td>
  </tr>
  <tr>
    <td>8</td>
    <td>71.31</td>
    <td>43.97</td>
    <td>60.62</td>
    <td>69.83</td>
    <td>56.28</td>
    <td>61.14</td>
  </tr>
  <tr>
    <td>16</td>
    <td>70.81</td>
    <td>44.83</td>
    <td>59.86</td>
    <td>70.07</td>
    <td>56.40</td>
    <td>61.20</td>
  </tr>
  <tr>
    <td>32</td>
    <td>70.69</td>
    <td>43.84</td>
    <td>59.57</td>
    <td>69.21</td>
    <td>56.90</td>
    <td>61.26</td>
  </tr>
  <tr>
    <td>64</td>
    <td>69.95</td>
    <td>46.55</td>
    <td>59.63</td>
    <td>69.09</td>
    <td>57.27</td>
    <td>60.62</td>
  </tr>
</table>

Table 10: Performance of MLLMs with different numbers of sampled frames for short videos (1–64 frames).

<table>
  <tr>
    <th rowspan="2">Frame</th>
    <th colspan="3">InternVL2.5-4B</th>
    <th colspan="3">InternVL2.5-8B</th>
  </tr>
  <tr>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
    <th>ori<sub>acc</sub></th>
    <th>c<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>2</td>
    <td>73.90</td>
    <td>49.22</td>
    <td>61.09</td>
    <td>72.24</td>
    <td>59.26</td>
    <td>59.67</td>
  </tr>
  <tr>
    <td>4</td>
    <td>75.79</td>
    <td>46.61</td>
    <td>62.13</td>
    <td>75.04</td>
    <td>59.07</td>
    <td>61.07</td>
  </tr>
  <tr>
    <td>8</td>
    <td>76.46</td>
    <td>46.23</td>
    <td>62.24</td>
    <td>75.79</td>
    <td>57.37</td>
    <td>61.78</td>
  </tr>
  <tr>
    <td>16</td>
    <td>77.45</td>
    <td>45.38</td>
    <td>62.31</td>
    <td>75.94</td>
    <td>58.27</td>
    <td>61.82</td>
  </tr>
  <tr>
    <td>32</td>
    <td>77.40</td>
    <td>45.57</td>
    <td>61.49</td>
    <td>75.98</td>
    <td>57.08</td>
    <td>61.49</td>
  </tr>
  <tr>
    <td>64</td>
    <td>76.13</td>
    <td>46.57</td>
    <td>61.11</td>
    <td>76.17</td>
    <td>58.41</td>
    <td>61.55</td>
  </tr>
</table>

Table 11: Effect of different frame sampling strategies on MLLM performance for long videos (64–2000 frames).

A.3 Sample Reaults on Test Time Long Reasoning Models

As illustrated in Figure 10, the reasoning model QVQ-72B-Preview (Team, 2024), equipped with a built-in Chain-of-Thought (CoT) mechanism, exhibits human-aligned reasoning patterns. Its cognitive process integrates detailed scenario descriptions, systematic elimination of implausible options (e.g., excluding candidates A/B/C), and rigorous conclusion verification. In contrast, InternVL2.5-78B employs a CoT mechanism that presents an-
<table>
  <tr>
    <th>Model</th>
    <th>ori_acc</th>
    <th>cf_acc</th>
    <th>sub_acc</th>
  </tr>
  <tr>
    <td>QVQ-72B-Preview</td>
    <td>69.33</td>
    <td>59.33</td>
    <td>58.76</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B-CoT</td>
    <td>70.00</td>
    <td>71.33</td>
    <td>70.80</td>
  </tr>
</table>

Table 12: Variation in accuracy across different test-time reasoning strategies.

<table>
  <tr>
    <th></th>
    <th>ori<sub>acc</sub></th>
    <th>cf<sub>acc</sub></th>
    <th>sub<sub>acc</sub></th>
  </tr>
  <tr>
    <td>QVQ-72B-Preview</td>
    <td>69.33</td>
    <td>59.33</td>
    <td>58.76</td>
  </tr>
  <tr>
    <td>Claude-3.7-sonnet</td>
    <td>46.00</td>
    <td>59.33</td>
    <td>46.72</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td>70.00</td>
    <td>71.33</td>
    <td>70.80</td>
  </tr>
  <tr>
    <td>VILA1.5-13B</td>
    <td>65.33</td>
    <td>44.67</td>
    <td>53.65</td>
  </tr>
</table>

Table 13: Performance of different chain-of-thought (CoT) reasoning architectures on a manually annotated dataset of 150 samples. QVQ and Claude-3.5-Sonnet represent dedicated reasoning models, while the others apply CoT-based augmentation.

swers in a bullet-point format without explanatory justification, reflecting weaker anthropomorphic reasoning characteristics.

However, the \( cf_{acc} \) discrepancy in Table 13 (QVQ-72B-Preview: 59.33% < InternVL2.5-78B: 71.33%) suggests that contemporary reasoning models may rely more on memorization than on structured reasoning. InternVL2.5-78B’s concise response paradigm appears to leverage rapid pattern recognition and information retrieval, leading to superior accuracy. While QVQ-72B-Preview’s elaborate reasoning workflow better approximates human cognition, potential redundancies or logical inconsistencies may reduce answer precision.

Table 13 further indicates that InternVL2.5-78B achieves a substantial lead in the \( sub_{acc} \) metric (70.80%), significantly outperforming QVQ-72B-Preview (58.76%) and Claude-3.7-sonnet (46.72%). This performance hierarchy remains consistent across models when evaluated on the \( ori_{acc} \) metric: InternVL2.5-78B (70.00%) > QVQ-72B-Preview (69.33%) > Claude-3.7-sonnet (46.00%). Empirical evidence suggests a statistically significant positive correlation between reasoning capability (\( sub_{acc} \)) and comprehension ability (\( ori_{acc} \)). In addition, under the CoT paradigm, reasoning capability demonstrates stronger generalization, exhibiting a positive correlation with performance on human-annotated essential logical sub-problems, thereby reinforcing the intrinsic relationship between logical reasoning and generalizability.

Moreover, the reasoning processes of models such as QVQ frequently generate sub-problem content that aligns with human-annotated data, which to some extent suggests that the inferential patterns of test-time long-reasoning models demonstrate closer correspondence with human cognitive intuition. For instance, in the Figure 11 the analytical content regarding the opening and closing scenes of videos (highlighted in blue font) exhibits precise alignment with the manually curated sub-problems in the upper-right annotation (specifically addressing inquiries about video commencement and conclusion scenarios), thereby empirically validating this cognitive congruence.

A.4 Examples of Sub-question Guidelines

Figure 8 illustrates how sub-question errors propagate to counterfactual question failures. In Figure 9, we observe that subtle errors in the reasoning process lead to reasoning failures, highlighting the model’s sensitivity to the integrity of its reasoning steps.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Type</th>
    <th colspan="17">Task</th>
  </tr>
  <tr>
    <th>Action Prediction</th>
    <th>Procedure Understanding</th>
    <th>Social Relation</th>
    <th>Action Recognition</th>
    <th>Object Recognition</th>
    <th>Color</th>
    <th>Counting</th>
    <th>Direction</th>
    <th>Location</th>
    <th>Material</th>
    <th>Shape</th>
    <th>Size</th>
    <th>Emotion</th>
  </tr>
  <tr>
    <td rowspan="2">GPT-4o</td>
    <td>ori_acc</td>
    <td>65.20</td><td>78.17</td><td>69.00</td><td>74.87</td><td>74.87</td><td>92.23</td><td>75.25</td><td>50.88</td><td>70.59</td><td>79.12</td><td>72.00</td><td>52.88</td><td>65.01</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>41.41</td><td>28.97</td><td>56.33</td><td>44.65</td><td>42.67</td><td>37.86</td><td>40.59</td><td>33.33</td><td>42.86</td><td>59.34</td><td>58.00</td><td>29.81</td><td>55.65</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>51.85</td><td>22.82</td><td>52.43</td><td>69.54</td><td>67.09</td><td>51.94</td><td>47.52</td><td>56.90</td><td>48.96</td><td>58.08</td><td>55.56</td><td>36.08</td><td>63.97</td>
  </tr>
  <tr>
    <td rowspan="4">GPT-4o-mini</td>
    <td>ori_acc</td>
    <td>50.22</td><td>72.22</td><td>63.32</td><td>78.61</td><td>74.08</td><td>84.47</td><td>70.30</td><td>52.63</td><td>57.98</td><td>71.43</td><td>62.00</td><td>56.73</td><td>65.56</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>44.05</td><td>51.19</td><td>62.01</td><td>56.42</td><td>52.36</td><td>26.21</td><td>39.60</td><td>36.84</td><td>59.66</td><td>52.75</td><td>53.00</td><td>17.31</td><td>58.26</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>53.16</td><td>19.44</td><td>58.85</td><td>68.03</td><td>64.82</td><td>38.35</td><td>38.12</td><td>53.88</td><td>47.72</td><td>53.89</td><td>53.44</td><td>28.85</td><td>65.96</td>
  </tr>
  <tr>
    <td>ori_acc</td>
    <td>43.61</td><td>63.10</td><td>63.32</td><td>66.31</td><td>73.82</td><td>79.61</td><td>68.32</td><td>48.25</td><td>52.10</td><td>63.74</td><td>66.34</td><td>45.19</td><td>66.94</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>ori_acc</td>
    <td>39.21</td><td>33.33</td><td>38.86</td><td>43.85</td><td>40.84</td><td>36.89</td><td>19.80</td><td>35.96</td><td>40.34</td><td>37.36</td><td>40.59</td><td>18.27</td><td>39.81</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>46.19</td><td>15.87</td><td>46.68</td><td>62.54</td><td>60.93</td><td>79.81</td><td>24.26</td><td>48.82</td><td>48.55</td><td>46.11</td><td>37.70</td><td>34.13</td><td>38.61</td>
  </tr>
  <tr>
    <td rowspan="4">Gemini-1.5-Pro</td>
    <td>ori_acc</td>
    <td>54.65</td><td>80.95</td><td>71.62</td><td>83.42</td><td>80.89</td><td>84.17</td><td>73.27</td><td>61.40</td><td>67.47</td><td>31.52</td><td>67.83</td><td>59.62</td><td>75.48</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>46.70</td><td>29.76</td><td>51.82</td><td>45.45</td><td>58.38</td><td>46.60</td><td>37.62</td><td>35.96</td><td>42.86</td><td>54.95</td><td>55.45</td><td>36.54</td><td>57.99</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>57.52</td><td>19.68</td><td>64.38</td><td>28.95</td><td>32.48</td><td>41.06</td><td>49.56</td><td>48.39</td><td>55.20</td><td>49.30</td><td>48.44</td><td>40.53</td><td>47.84</td>
  </tr>
  <tr>
    <td>ori_acc</td>
    <td>53.55</td><td>85.32</td><td>70.74</td><td>82.62</td><td>81.41</td><td>62.52</td><td>21.80</td><td>57.02</td><td>70.59</td><td>79.12</td><td>69.31</td><td>68.27</td><td>72.04</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash</td>
    <td>ori_acc</td>
    <td>45.81</td><td>34.92</td><td>56.33</td><td>49.20</td><td>49.48</td><td>41.75</td><td>37.62</td><td>33.33</td><td>41.18</td><td>64.84</td><td>53.47</td><td>25.96</td><td>58.26</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>61.87</td><td>32.94</td><td>63.94</td><td>73.28</td><td>69.60</td><td>46.60</td><td>43.07</td><td>62.93</td><td>54.77</td><td>62.87</td><td>55.50</td><td>37.98</td><td>71.36</td>
  </tr>
  <tr>
    <td>ori_acc</td>
    <td>50.35</td><td>86.90</td><td>74.24</td><td>87.97</td><td>80.10</td><td>90.29</td><td>69.31</td><td>64.04</td><td>78.99</td><td>81.32</td><td>70.30</td><td>66.35</td><td>75.90</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>ori_acc</td>
    <td>42.29</td><td>36.51</td><td>51.79</td><td>44.12</td><td>51.31</td><td>20.30</td><td>39.60</td><td>31.58</td><td>37.82</td><td>57.14</td><td>56.44</td><td>31.73</td><td>57.71</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>60.78</td><td>35.52</td><td>59.51</td><td>73.16</td><td>72.49</td><td>69.90</td><td>59.41</td><td>65.95</td><td>53.53</td><td>58.08</td><td>58.64</td><td>42.31</td><td>66.84</td>
  </tr>
  <tr>
    <td>InternVL2.5-78B</td>
    <td>ori_acc</td>
    <td>67.84</td><td>75.00</td><td>75.55</td><td>79.68</td><td>82.20</td><td>94.17</td><td>82.18</td><td>52.63</td><td>76.47</td><td>76.92</td><td>83.17</td><td>69.23</td><td>78.66</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>43.61</td><td>76.19</td><td>57.21</td><td>65.51</td><td>61.78</td><td>87.38</td><td>37.62</td><td>47.37</td><td>75.63</td><td>61.54</td><td>57.43</td><td>39.43</td><td>56.20</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>62.09</td><td>44.64</td><td>67.70</td><td>79.60</td><td>62.28</td><td>79.13</td><td>69.90</td><td>66.58</td><td>58.09</td><td>62.58</td><td>59.96</td><td>40.54</td><td>70.26</td>
  </tr>
  <tr>
    <td>LLaVA-Video-72B</td>
    <td>ori_acc</td>
    <td>43.17</td><td>50.79</td><td>65.50</td><td>67.07</td><td>69.90</td><td>84.54</td><td>69.31</td><td>51.75</td><td>73.11</td><td>74.73</td><td>61.39</td><td>61.54</td><td>76.06</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>44.93</td><td>59.92</td><td>59.39</td><td>63.10</td><td>57.85</td><td>62.14</td><td>42.57</td><td>47.37</td><td>66.39</td><td>53.85</td><td>51.49</td><td>41.35</td><td>56.20</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>59.26</td><td>32.94</td><td>69.47</td><td>67.56</td><td>66.46</td><td>63.99</td><td>52.97</td><td>61.21</td><td>45.23</td><td>55.69</td><td>53.40</td><td>43.27</td><td>70.01</td>
  </tr>
  <tr>
    <td>InternVL2.5-26B</td>
    <td>ori_acc</td>
    <td>57.27</td><td>78.58</td><td>76.42</td><td>82.35</td><td>79.58</td><td>91.26</td><td>74.26</td><td>62.88</td><td>85.71</td><td>74.73</td><td>78.62</td><td>66.35</td><td>73.14</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>47.14</td><td>45.24</td><td>51.09</td><td>60.43</td><td>57.59</td><td>29.83</td><td>25.74</td><td>45.61</td><td>60.70</td><td>49.11</td><td>25.00</td><td>25.00</td><td>49.00</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>59.91</td><td>61.68</td><td>62.39</td><td>71.88</td><td>73.23</td><td>82.67</td><td>62.81</td><td>58.49</td><td>67.98</td><td>61.78</td><td>56.90</td><td>43.64</td><td>58.57</td>
  </tr>
  <tr>
    <td>InternVL2.5-8B</td>
    <td>ori_acc</td>
    <td>35.61</td><td>75.00</td><td>78.17</td><td>81.88</td><td>80.64</td><td>90.59</td><td>70.30</td><td>63.16</td><td>78.99</td><td>74.63</td><td>67.58</td><td>66.87</td><td>72.18</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>48.42</td><td>76.19</td><td>47.87</td><td>71.39</td><td>57.85</td><td>84.47</td><td>38.25</td><td>39.15</td><td>56.49</td><td>43.57</td><td>33.54</td><td>37.55</td><td>31.79</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>59.97</td><td>29.37</td><td>66.88</td><td>89.89</td><td>72.24</td><td>52.43</td><td>52.97</td><td>60.34</td><td>53.53</td><td>56.49</td><td>51.44</td><td>48.19</td><td>58.19</td>
  </tr>
  <tr>
    <td>VidECLLama-3-8B</td>
    <td>ori_acc</td>
    <td>8.02</td><td>61.75</td><td>86.56</td><td>65.08</td><td>64.04</td><td>60.68</td><td>37.01</td><td>58.86</td><td>52.72</td><td>37.50</td><td>46.92</td><td>52.22</td><td>76.92</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>35.68</td><td>46.29</td><td>59.08</td><td>58.47</td><td>60.62</td><td>42.57</td><td>64.11</td><td>54.72</td><td>32.58</td><td>58.24</td><td>48.51</td><td>33.58</td><td>59.95</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>49.45</td><td>33.93</td><td>67.48</td><td>67.91</td><td>68.84</td><td>57.97</td><td>39.60</td><td>58.62</td><td>49.79</td><td>53.89</td><td>54.45</td><td>47.12</td><td>67.90</td>
  </tr>
  <tr>
    <td>LLaVA-ov-7B</td>
    <td>oriacc</td>
    <td>48.90</td><td>48.81</td><td>66.81</td><td>60.43</td><td>65.45</td><td>86.41</td><td>63.37</td><td>44.74</td><td>63.03</td><td>72.33</td><td>61.39</td><td>63.46</td><td>68.60</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>43.61</td><td>64.29</td><td>45.85</td><td>55.35</td><td>50.79</td><td>59.22</td><td>42.57</td><td>45.61</td><td>52.94</td><td>60.44</td><td>57.43</td><td>30.77</td><td>52.75</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>50.11</td><td>30.36</td><td>63.94</td><td>62.78</td><td>60.68</td><td>54.85</td><td>42.08</td><td>53.45</td><td>45.64</td><td>50.80</td><td>48.69</td><td>50.96</td><td>64.73</td>
  </tr>
  <tr>
    <td>LLaVA-Video-7B</td>
    <td>ori_acc</td>
    <td>50.66</td><td>35.71</td><td>65.50</td><td>64.15</td><td>67.02</td><td>83.50</td><td>67.33</td><td>41.23</td><td>58.82</td><td>74.72</td><td>64.36</td><td>59.62</td><td>66.39</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>44.93</td><td>73.02</td><td>45.85</td><td>59.09</td><td>42.15</td><td>73.79</td><td>42.57</td><td>48.25</td><td>60.80</td><td>54.30</td><td>47.53</td><td>25.58</td><td>49.99</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>48.85</td><td>29.66</td><td>63.60</td><td>62.19</td><td>58.67</td><td>55.34</td><td>41.09</td><td>56.03</td><td>43.15</td><td>54.19</td><td>52.88</td><td>48.08</td><td>61.03</td>
  </tr>
  <tr>
    <td>Qwen2-VL-TMB</td>
    <td>ori_acc</td>
    <td>34.40</td><td>84.12</td><td>67.52</td><td>54.22</td><td>80.12</td><td>85.35</td><td>70.29</td><td>57.89</td><td>77.34</td><td>74.73</td><td>65.34</td><td>69.23</td><td>67.47</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>42.71</td><td>58.73</td><td>45.14</td><td>38.02</td><td>67.85</td><td>61.39</td><td>13.78</td><td>35.78</td><td>23.00</td><td>43.77</td><td>55.01</td><td>35.73</td><td>49.34</td>
  </tr>
  <tr>
    <td>sub/rec</td>
    <td>53.37</td><td>10.16</td><td>45.77</td><td>65.53</td><td>66.73</td><td>12.30</td><td>24.08</td><td>54.08</td><td>51.87</td><td>52.69</td><td>31.58</td><td>31.44</td><td>65.02</td>
  </tr>
  <tr>
    <td>ViLA-UB-TMB</td>
    <td>ori_acc</td>
    <td>45.37</td><td>73.02</td><td>54.59</td><td>66.31</td><td>59.95</td><td>31.85</td><td>41.39</td><td>54.09</td><td>61.75</td><td>47.25</td><td>47.12</td><td>50.90</td><td>39.50</td>
  </tr>
  <tr>
    <td>C Earth</td>
    <td>cas neg</td>
    <td>51.47</td><td>42.36</td><td>46.35</td><td>33.14</td><td>35.93</td><td>45.64</td><td>23.95</td><td>43.96</td><td>35.71</td><td>42.19</td><td>43.21</td><td>38.68</td><td>43.35</td>
  </tr>
  <tr>
    <td>VILA1.5-7B</td>
    <td>oriacc</td>
    <td>52.85</td><td>50.40</td><td>66.96</td><td>15.04</td><td>46.05</td><td>24.77</td><td>-21.53</td><td>43.26</td><td>45.48</td><td>42.39</td><td>48.31</td><td>43.81</td><td>62.71</td>
  </tr>
  <tr>
    <td>c/rec</td>
    <td>28.44</td><td>24.80</td><td>59.96</td><td>61.73</td><td>65.45</td><td>48.06</td><td>31.19</td><td>50.86</td><td>42.74</td><td>47.31</td><td>45.55</td><td>46.63</td><td>51.91</td>
  </tr>
</table>

Table 14: Overall performance of MLLMs on 13 tasks in COVER, including original accuracy, counterfactual accuracy, and sub-question accuracy.
Figure 8: Example from COVER, showing a video accompanied by three related questions. The video is divided into four key action frames (left), with dashed lines indicating reasoning steps. Single-step prediction errors are marked with red crosses on the right, while sub-questions that do not support counterfactual reasoning are marked with red crosses on the left.

Figure 9: An example from COVER. The top section shows the video input and corresponding counterfactual questions. The middle section presents three reasoning processes—CoT, Guide-CoT, and Standard—where correct steps are marked with green checkmarks. In the analysis, correct reasoning paths are shown in green text, while incorrect ones are highlighted in red. The bottom section displays the final model predictions, with green checkmarks indicating correct answers and red crosses denoting errors.
Figure 10: An example from the 150 seed samples. The top section shows the video input and corresponding counterfactual questions. The middle section compares two reasoning frameworks: the test-time long reasoning model QVQ and InternVL2.5-78B with CoT, with green marks indicating validated response components. The bottom section displays final model predictions, where green checkmarks indicate correct answers.

Figure 11: An example from the 150 seed samples. The top section presents the video input and corresponding counterfactual questions. The middle section compares QVQ and InternVL2.5-78B with CoT, using a dual-color annotation scheme: blue indicates conceptual alignment with manual sub-problem annotations, and green highlights validated response components. The bottom section shows the final model predictions, with green checkmarks indicating correct answers.

Reviewer paper summaries: This paper introduces COVER (COunterfactual VidEo Reasoning), a video QA benchmark designed to test MLLMs on counterfactual reasoning tasks. It systematically evaluates models along two axes: abstract vs. concrete concepts and perception vs. cognition tasks. The benchmark includes a set of 2923 video-based question pairs, each including an original question about the factual video content and a related counterfactual question that introduces a hypothetical variation, plus a set of structured sub-questions that break down the reasoning steps required to answer the counterfactual query. The authors evaluate several state-of-the-art MLLMs and report that models’ accuracy on sub-questions correlates with their performance on the overall counterfactual questions.
Summary of strengths: The benchmark spans multiple reasoning dimensions (abstract/concrete, perception/cognition). The sub-question evaluation framework is well-motivated. Explicitly breaking each complex counterfactual query into intermediate questions allows fine-grained analysis of model reasoning.
Summary of weaknesses: 1. Despite various spans, the abstract/concrete and perception/cognition categorization for example, lacks clear criteria, causing potential confusion or overlaps between tasks.

2. The evaluation setup conflates video comprehension (perception) with logical reasoning, potentially misrepresenting models' true reasoning capabilities.

3. The total of 146 unique videos and ~2.9k QA pairs is relatively small compared to other video QA benchmarks. It’s also focused on a particular style of questioning (counterfactuals with necessary conditions) which, albeit broad in covered topics, might not encompass all forms of reasoning a model may need for “dynamic real-world video tasks.” In terms of novelty, the idea of sub-question decomposition for analysis is not entirely unprecedented. Thus, the contribution is more in the careful engineering of this benchmark than in a fundamentally new evaluation concept.
Comments, suggestions and typos: See Weakness.
Overall assessment scores: 2.5
Metareview score: 3
Metareview: This paper introduces COVER (COunterfactual VidEo Reasoning), a new benchmark for evaluating counterfactual reasoning in video understanding tasks for MLLMs. The benchmark contains 2,923 video-based counterfactual QA pairs. The query is decomposed into structured sub-questions, which break down the reasoning steps required to answer the counterfactual query. The authors evaluate a number of state-of-the-art MLLMs and report a strong correlation between model performance on sub-questions and overall counterfactual reasoning accuracy.

Additionally, the authors should be praised for providing a timely rebuttal response to a last-minute emergency review with a short turnaround.
Summary of reasons to publish: After going through the paper, review, and rebuttal, AC agrees with most reviewer comments and finds compelling reasons to publish this paper:
* The paper clearly outlines its motivation and offers a valuable addition to multimodal counterfactual benchmarks (4QHw, YaNA)
* The evaluation of various MLLMs is thorough and provides valuable insights for the field (4QHw, YaNA)
* The benchmark covers multiple reasoning dimensions, and the sub-question framework allows for fine-grained analysis of model reasoning (kagD)