Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities
Authors: Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, Weidong Cai
Accepted at: acl25

Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities

Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang,
Ao Ma, Jieting Long, Weidong Cai
School of Computer Science, The University of Sydney
{zhli0736, hwan9147, czha5168, aoma0081, jlon5443}@uni.sydney.edu.au
{dongnan.liu, tom.cai}@sydney.edu.au

Abstract

Multimodal Large Language Models (MLLMs) have showcased exceptional Chain-of-Thought (CoT) reasoning ability in complex textual inference tasks including causal reasoning. However, will these causalities remain straightforward when crucial hints hide in visual details? If not, what factors might influence cross-modal generalization? Whether we can effectively enhance their capacity for robust causal inference across both text and vision? Motivated by these, we introduce MuCR - a novel Multimodal Causal Reasoning benchmark that leverages synthetic siamese images and text pairs to challenge MLLMs. Additionally, we develop tailored metrics from multiple perspectives, including image-level match, phrase-level understanding, and sentence-level explanation, to comprehensively assess MLLMs’ comprehension abilities. Our experiments reveal that current MLLMs fall short in multimodal causal reasoning compared to their performance in purely textual settings. Additionally, we find that identifying visual cues across images is key to effective cross-modal generalization. Finally, we propose a VC-CoT strategy that better highlights visual cues, and our results confirm its efficacy in enhancing multimodal causal reasoning. The project is available at: https://github.com/Zhiyuan-Li-John/MuCR

1 Introduction

Causal reasoning is the process of identifying the relationship between a cause and its effect, which is regarded as a fundamental capability of artificial intelligence (Liu et al., 2024c). Recent advancements in CoT reasoning capabilities of MLLMs (OpenAI, 2024b; Guo et al., 2025) have driven significant progress in complex analytical tasks, including causal reasoning within the textual modality (Jin et al., 2023; Bagheri et al., 2024; Ashwani et al., 2024). These developments involve enabling MLLMs to generate coherent explanations (Kiciman et al., 2023), providing multi-step chain-of-thought (CoT) (Bao et al., 2024), or even analyzing complex causal relationships that typically demand expert-level topological structure knowledge (Vashishtha et al., 2023). Despite these advancements, existing linguistic benchmarks (Singh et al., 2021; Du et al., 2022; Jin et al., 2023) are beginning to fall short in assessing the more advanced visual capabilities of the latest MLLMs such as GPT-01 (OpenAI, 2024b), Deepseek-R1 (Guo et al., 2025), Gemini-1.5 (DeepMind, 2024), and Claude-3.5 (ClaudeAI, 2024b), not to mention facilitating cross-modal comparison and analysis (as shown in Figure 1).

Following this, we propose three key questions: Can MLLMs achieve the same level of causal reasoning comprehension as they do in textual modal-

![An example from MuCR challenges MLLMs with weather-related causality across two modalities.](page_1012_186_377_377.png)

Figure 1: An example from MuCR challenges MLLMs with weather-related causality across two modalities.
Figure 2: (a) Comparison of our MuCR and related datasets on reasoning tasks. (b) Detailed illustration of our dataset structure and corresponding cross-modal generalization exploration.

ity? If not, what factors might influence cross-modal generalization? How can we enhance their capacity for robust causal inference? We find that most existing benchmarks fail to address such comparisons or support further exploration in this area. Especially, as shown in Figure 2 (a), we identify two major drawbacks in previous benchmarks: Absence of visual modality: Linguistic causal reasoning benchmarks (Singh et al., 2021; Li et al., 2021; Du et al., 2022; Frohberg and Binder, 2022; Jin et al., 2023, 2024) fail to assess visual comprehension ability of MLLMs. Incomplete of cross-modal analysis: Most causal reasoning VQA tasks (Zellers et al., 2019a; Girdhar and Ramanan, 2020; Zhang et al., 2021; Hessel et al., 2022) neglect cross-modal comparison. Recently, some benchmarks (Bitton-Guetta et al., 2024; Fu et al., 2024) have begun exploring this domain. For instance, Blink (Fu et al., 2024) examines cross-modal comparisons and conducts basic generalization analyses involving factors like shape and size. As illustrated in Figure 2(b), our proposed MuCR comprehensively evaluates causal reasoning at the image, phrase, and sentence levels and offers a multi-faceted analysis of cross-modal generalization that encompasses both visual form factors and semantic elements. Moreover, we propose a novel VeCoT strategy to further enhance cross-modal generalization by improving visual cue perception.

We evaluate current state-of-the-art (SOTA) MLLMs on our MuCR benchmark. Experiment results indicate that all models fall short of human performance, particularly in multimodal settings. Moreover, they exhibit a pronounced cross-modal gap when discerning causal links across modalities. In addition, we conduct in-depth generalization analysis and demonstrate that visual semantic factors, especially the ability to identify visual cues across siamese images, play a pivotal role.

Our contributions are summarized as follows:

• We identify the limitations of current causal reasoning benchmarks, including failing to evaluate the advanced visual capabilities of the latest MLLMs and offering incomplete cross-modal analyses.

• We propose the MuCR benchmark, which can comprehensively evaluate MLLMs’ causal reasoning ability across two modalities.

• Our extensive experiments with SOTA MLLMs reveal interesting insights and suggest potential directions for future research.

2 Related Work

2.1 Causal Reasoning

The ability to perform causal reasoning is widely considered a core feature of artificial intelligence. With the development of Large Language Models (LLMs), they have exhibited increasingly robust capabilities in causal reasoning tasks. Previous benchmarks, such as Com2sense (Singh et al., 2021) and CausalBank (Li et al., 2021), are becoming insufficient for evaluating linguistic abilities. To address this, Romanou et al. (2023) introduced the CRAB benchmark, which requires LLMs to capture explicit causal relationships in real-world scenarios. However, these benchmarks focus solely on the text modality, leaving the crucial question of multimodal reasoning unaddressed. Hessel et al. (2022) introduced Sherlock to challenge MLLMs in identifying visual clues scattered throughout a scene and making reasoning inferences combined with commonsense and life experience. More recently, Guetta et al. (2024) and Fu et al. (2024) presented complex visual reasoning challenges to further explore MLLMs’ capabilities. Although these benchmarks have considered the visual modality, they still fail to comprehensively analyze cross-modal generalization capacity. In this paper, we make an early attempt to extensively explore multimodal
Figure 3: The overview of our MuCR benchmark construction process. It follows synthesis in four core levels: generating core caption pairs, producing contextual description pairs, creating siamese images, and generating human annotations.

causal reasoning tasks across modalities.

2.2 LLMs’ Generalization

The field of LLMs generalization has gained significant traction in recent years, with numerous tasks proposed to evaluate models’ ability to handle previously unseen contexts and domains. Existing tasks can be broadly divided into compositional, cross-task, cross-lingual, cross-domain, and robustness-based categories. Compositional tasks, such as CFQ (Keysers et al., 2020) and COGS (Kim and Linzen, 2020), test whether models can systematically combine smaller linguistic units to form novel expressions. Cross-task generalization often involves multi-task learning setups, such as De-caNLP (McCann et al., 2018) and BIG-Bench (Srivastava et al., 2022), where models must adapt to tasks with minimal guidance. Cross-lingual benchmarks, like XNLI (Conneau et al., 2018) and XTREME (Hu et al., 2020), measure performance across languages, while cross-domain tasks emphasize shifting between specialized fields (Li et al., 2023; Zhou et al., 2024). Meanwhile, robustness-oriented evaluations such as HellaSwag (Zellers et al., 2019b) and adversarial GLUE (Wang et al., 2021) assess how well models withstand noisy, ambiguous, or adversarial inputs. In this paper, we shift our focus to the generalization in multimodal causal reasoning tasks, conducting a concise but comprehensive analysis of the factors that hinder cross-modal generalization and exploring strategies to enhance it for robust causal reasoning.

3 The MuCR Dataset

In this section, we detail the construction of the MuCR dataset. Figure 3 illustrates the systematic workflow of our multimodal cause-and-effect benchmark generation including: generating core caption pairs, producing contextual description pairs, creating siamese images, and generating human annotations (see Appendix A.2 for further examples and details).

3.1 Dataset Creation

Generating Core Caption Pairs. The MuCR benchmark is designed to assess MLLMs’ ability to perform causal inference across modalities. To achieve this, we begin by generating core caption pairs that clearly illustrate cause-and-effect relationships. In order to minimize individual bias, we employ twelve volunteers and group each two as a team: one processes and refines the captions based on initial ideas and iterative feedback, while the other reviews them and offers suggestions for improvement (see Appendix A.3 for an explanation of why we structure the generation process this way, as well as illustrative examples). Through these steps, we create 4,000 cause-and-effect caption pairs.

Producing Contextual Description Pairs. While core caption pairs effectively depict the cause-and-effect relationship, they often lack contextual details such as appearance, clothing, and environmental context that serve as crucial visual cues for high-quality cause-and-effect image
Figure 4: (a) Examples from our MuCR dataset featuring different categories and styles. The “Mixture” category represents two or more tags involved in the causality. (b) Category distribution overview showing the proportions of human, animal, character, plant, and mixture categories. (c) Style distribution overview illustrating the proportions of comic, photographic, and black–white styles.

synthesis. To address this issue, we leverage the linguistic capabilities of LLMs to enhance core caption pairs by enriching contextual details. By maintaining these elements consistently across images, our approach not only effectively depicts causality at a semantic level but also improves visual coherence (see Appendix A.4 for further explanation).

Create Siamese Images. We employ diffusion models with contextual descriptions as prompts to generate cause-and-effect image pairs. Specifically, we utilize DALL-E (Ramesh et al., 2021), DeepAI (DeepAI, 2024), Stability-AI (Stability AI, 2023), and Flux1 (FLUXAI, 2024) for image synthesis, aiming to minimize model bias and enhance the diversity of the generated images. We also incorporate three styles (photograph, comic, and black-white) when creating these images. Specifically, each sentence yields 10 images per style, resulting in 20 images for every cause-and-effect pair in one style (a total of 240k images). Then, volunteers manually select the two representations that best capture the semantic causality and maintain visual consistency. This process produces 12k cause-and-effect image pairs spanning various categories (humans, animals, plants, characters, and mixtures) and three styles (photograph, comic, and black-white). Figure 4 illustrates examples from our MuCR benchmark, showcasing multiple categories and styles alongside an overview of their distribution (see Appendix A.5 for more high-quality samples).

Generate Human Annotation. We require volunteers to create text annotations for each cause-and-effect image pair. As shown in Figure 3, it consists of a phrase-level list (cue phrases) and sentence-level description (cause-and-effect explanations). The cue phrases comprise a list of four options, each being a word or phrase. Among these, only one phrase correctly explains or is highly relevant to the causality, while the other three are striking elements in the images but do not serve as proper cues. The sentence-level annotation is designed to verify whether the MLLMs truly understand multimodal causality and can select reasonable explanations. To achieve this, we require volunteers to structure the explanation by first describing the content of the cause, followed by the content of the effect, and concluding with the causal link connecting between them.

3.2 Evaluation Metrics

Image-level Metric. The image-level metric is call cause-to-effect (C2E) score. It is designed to assess whether the MLLMs can identify cue links and make the correct choice from four potential effects according to the given cause. Given the cause in the form \( G^*(c) \) (* can either be \( G^{text} \) representing text-based form or \( G^{multi} \) representing multimodal-based form), the model is required to select the optimal choice among four potential effects \( \{G^*(e)^{(i)}\}_{i=1}^4 \). The C2E score can be computed as follows:

\[
S_I^* = F(Q_I, G^*(c), \{G^*(e)^{(i)}\}_{i=1}^4),
\]
\[
f_I(S_I^*) = \begin{cases}
1, & S_I^* = S_I^{*'} \\
0, & \text{otherwise}
\end{cases}
\]

where \( S_I^* \) represents the MLLMs’ prediction. \( F \) represents MLLM. \( Q_I \) represents corresponding question for Image-level. \( f_I \) represents the function to calculate the C2E score. \( S_I^{*'} \) represents the correct answer.

Phrase-level Metric. The phrase-level metric is called CP score (Cue Phrase), which tests MLLMs’ capability to distinguish the correct cue from a list of fraudulent phrases according to the cause and effect. Given the cause-and-effect pairs
\{ \mathcal{G}^*(c), \mathcal{G}^*(e) \}, the model is required to select the optimal choice among four potential cue phrases \( \{ T_P^{(i)} \}_{i=1}^4 \). The **CP** score can be computed as follows:

\[
S_P^* = F(Q_P, \mathcal{G}^*(c), \mathcal{G}^*(e), \{ T_P^{(i)} \}_{i=1}^4) \tag{3}
\]
\[
f_P(S_P^*) = \begin{cases}
1, & S_P^* = S_P' \\
0, & \text{otherwise}
\end{cases} \tag{4}
\]

where \( S_P^* \) represents the MLLMs’ prediction. \( F \) represents MLLM. \( Q_P \) represents corresponding question for Phrase-level. \( f_P \) represents the function to calculate the CP score. \( S_P' \) represents the correct answer.

**Sentence-level Metric.** Our final metric is designed to evaluate MLLMs’ ability to identify the correct explanation according to the cause and effect. The sentence-level metric is called the explanation (**EXP**) score. Specifically, we collect four candidate explanations that share similar causalities but differ in their cues. Only one explanation accurately captures the causal relationship and matches the detailed cues, while the other three do not. Given the condition \( \{ \mathcal{G}^*(c), \mathcal{G}^*(e) \} \) with the corresponding question \( Q_S \), the model is required to select the optimal choice among four potential explanations \( \{ T_E^{(i)} \}_{i=1}^4 \). The **EXP** score is then computed as follows:

\[
S_S^* = F(Q_S, \mathcal{G}^*(c), \mathcal{G}^*(e), \{ T_S^{(i)} \}_{i=1}^4) \tag{5}
\]
\[
f_S(S_S^*) = \begin{cases}
1, & S_S^* = S_S' \\
0, & \text{otherwise}
\end{cases} \tag{6}
\]

where \( S_S^* \) represents the MLLMs’ prediction. \( F \) represents MLLM. \( f_S \) represents the function to calculate the **EXP** score. \( S_S' \) represents the correct answer.

4 Experiments

4.1 Experimental Setup

We evaluated several popular MLLMs on our MuCR benchmark, including GPT-01 (OpenAI, 2024b), GPT-4o (OpenAI, 2024a), Claude-3.5 (ClaudeAI, 2024b), Claude-3.0 (ClaudeAI, 2024a), Gemini-2.0 (DeepMind, 2025), Gemini-1.5 (DeepMind, 2024), Qwen2.5-VL (Yang et al., 2024), and Llama3.2-Vision (Meta, 2024). For the currently popular models, DeepSeek-R1 (Guo et al., 2025) and DeepSeek-V3 (Liu et al., 2024a), we did not fully evaluate their performance since their image readers currently only support extracting text from images without additional functionality (see Appendix B.1 for a comparison of their text-based performance). Additionally, we also considered some lightweight open-source models, including LLaVA-NeXT (Li et al., 2024), OpenFlamingo-v2 (Awadalla et al., 2023), LLaVA-v1.6 (Liu et al., 2024b), MiniGPT4-v2 (Zhu et al., 2023), and InstructBLIP (Dai et al., 2023). Since some models only accept a single image input, we provided all of them with a composite image composed of multiple smaller images, as shown in Figure 9 (a). Finally, we established a human performance baseline on the MuCR benchmark using crowd workers for comparison.

4.2 Experimental Results

Figure 5 presents the main results of popular MLLMs and human performance on the MuCR benchmark, leading to the following observations:
(1) **All models on MuCR lag behind human performance in both settings.** Among these models, GPT-01 (OpenAI, 2024b) achieves the highest scores, with 94% on C2E score, 75% on CP score, and 93% on EXP score in the text condition, while 87% on C2E, 62% on CP, and 78% on EXP in the multimodal condition. Nevertheless, these results still fall short of human performance, suggesting substantial room for improvement. (2) **All models exhibit a significant cross-modal performance gap.** All models show a noticeable drop in performance when handling multimodal causal inference, whereas humans do not. This discrepancy indicates potential factors restricting cross-modal generalization in MLLMs, likely stemming from the visual component, given that these models already demonstrate robust causal reasoning in text-based cases.

Figure 6 presents the multimodal performance of various lightweight open-source models, revealing that they lag significantly behind GPT-01. Among these, LLaVA-Next achieves the best results, with 29% on C2E, 17% on CP, and 21% on EXP, which are only around the random selection baseline of 25%. Compared to models like Llama3.2-Vision and Qwen2.5-VL, there is still considerable room for improvement for these models.

5 Cross-modal Generalization Analysis and Enhancement

In this section, we examine the factors that may affect cross-modal generalization. Building on previous findings that attribute these gaps primarily to
Figure 5: Main experimental results of several popular MLLMs on our MuCR benchmark. “Human” performance is represented by the average accuracy of ten attempts by volunteers.

Figure 6: Experimental results of lightweight open-source models on the multimodal-based form. For detailed numbers see Table 6. Best viewed by zooming in.

the visual component, we focus on two main categories: visual format factors and visual semantic factors.

• Visual Format Factors. These involve cases that share the same underlying semantics but differ in how they are visually presented, such as variations in picture style or the form of the visual input.

• Visual Semantic Factors. These involve cases with consistent visual formats but slight semantic differences, such as contextual variations in image details or the inclusion of additional text hints, resulting in richer semantic content.

In addition to investigating these cross-modal generalization factors, we also explore potential enhancement strategies based on our findings.

5.1 Visual Format Factors

Picture Style. We investigate how different picture styles may affect causal reasoning. Figure 7 shows an example of the same cause-and-effect

Figure 7: An example of cause and effect showing in three picture styles with the same semantic meanings.

Figure 8: The C2E score of different models tested on three different picture styles.
scenario presented in three styles. As indicated by the results in Figure 8, MLLMs perform similarly when presented with photographs and comic images, but with a slight drop for black-white images. Overall, altering the picture style while keeping the same semantic content has only a minimal effect on MLLMs’ performance (see Appendix C.1 for detailed comparison).

Form of Visual Input. We also explore whether the structure of visual inputs affects the final output. Figure 9 illustrates the three types of visual input forms we examined. Table 1 presents the performance of three models on MuCR using these different formats. It indicates that all models get marked performance improvements. Our case analysis suggests that, compared to Form-3, Forms-1 and Form-2 restrict MLLMs’ ability to perceive certain details that could serve as crucial visual cues for enhancing multimodal causal reasoning (see Appendix C.2 for case studies).

<table>
  <tr>
    <th>Visual Input</th>
    <th>Style</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <td colspan="5">GPT-o1 (<a href="https://openai.com/">OpenAI, 2024b</a>)</td>
  </tr>
  <tr>
    <td>Form-1</td>
    <td></td>
    <td>87.50</td>
    <td>62.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <td>Form-2</td>
    <td>Mixture</td>
    <td>84.25</td>
    <td>60.50</td>
    <td>79.00</td>
  </tr>
  <tr>
    <td>Form-3</td>
    <td></td>
    <td>89.00</td>
    <td>67.50</td>
    <td>86.25</td>
  </tr>
  <tr>
    <td colspan="5">Claude-3.5 (<a href="https://claude.ai/">ClaudeAI, 2024b</a>)</td>
  </tr>
  <tr>
    <td>Form-1</td>
    <td></td>
    <td>83.50</td>
    <td>59.75</td>
    <td>77.50</td>
  </tr>
  <tr>
    <td>Form-2</td>
    <td>Mixture</td>
    <td>53.50</td>
    <td>36.00</td>
    <td>68.50</td>
  </tr>
  <tr>
    <td>Form-3</td>
    <td></td>
    <td>85.00</td>
    <td>66.75</td>
    <td>82.25</td>
  </tr>
  <tr>
    <td colspan="5">Gemini-1.5 (<a href="https://deepmind.com/">DeepMind, 2024</a>)</td>
  </tr>
  <tr>
    <td>Form-1</td>
    <td></td>
    <td>66.50</td>
    <td>58.50</td>
    <td>70.50</td>
  </tr>
  <tr>
    <td>Form-2</td>
    <td>Mixture</td>
    <td>69.50</td>
    <td>57.25</td>
    <td>63.00</td>
  </tr>
  <tr>
    <td>Form-3</td>
    <td></td>
    <td>83.50</td>
    <td>65.25</td>
    <td>84.00</td>
  </tr>
</table>

Table 1: The performance of different visual input forms on our MuCR benchmark. The mixture means we test on mixture picture style.

Figure 9: The illustration of three different visual input forms we examined.

![Three different visual input forms illustrated](page_120_120_600_400.png)

Figure 10: Two image pairs illustrate the same cause-and-effect relationship but exhibit different contextual correlations.

![Two image pairs illustrating cause and effect](page_1040_120_600_400.png)

Figure 11: Using human selection as the standard, the models exhibit varying levels of selection accuracy.

![Bar chart showing selection accuracy for four models](page_1040_600_600_400.png)

5.2 Visual Semantic Factors

Contextual Variation. In addition to examining visual format factors, we also explore whether visual semantics influence MLLMs’ final output. As shown in Figure 1, MLLMs, particularly GPT-o1, can identify visual cues such as action, appearance, and environment, and integrate these details into their causal inference process. Additionally, the case study in the above paragraph also confirms that visual cues are essential for accurate multimodal causal inference. To further investigate, we assess whether the ability to identify visual cues correlates with multimodal causal reasoning performance. For this purpose, we use manually selected siamese image pairs that best capture semantic causality and maintain visual consistency, along with some pairs that exhibit minor contextual variations (see Figure 10). Our challenge is as follows: given a human-selected cause image, the models must identify the corresponding effect image from random 3 samples and 1 correct one. Figure 11 shows that among the four models tested, GPT-o1
Figure 12: Does adding text hints compensate for shortcomings in visual cue perception?

<table>
  <tr>
    <th>Add Hints</th>
    <th>Style</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <td colspan="5">GPT-01 (<a href="https://openai.com/blog/gpt-4/">OpenAI, 2024b</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>87.50</td>
    <td>62.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>91.25</td>
    <td>69.50</td>
    <td>88.50</td>
  </tr>
  <tr>
    <td colspan="5">GPT-4o (<a href="https://openai.com/blog/gpt-4/">OpenAI, 2024a</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>81.25</td>
    <td>57.25</td>
    <td>72.50</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>89.00</td>
    <td>66.50</td>
    <td>87.50</td>
  </tr>
  <tr>
    <td colspan="5">Claude-3.5 (<a href="https://claude.ai/">ClaudeAI, 2024b</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>83.50</td>
    <td>59.75</td>
    <td>77.50</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>87.50</td>
    <td>68.50</td>
    <td>86.00</td>
  </tr>
  <tr>
    <td colspan="5">Claude-3.0 (<a href="https://claude.ai/">ClaudeAI, 2024a</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>58.00</td>
    <td>50.25</td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>73.00</td>
    <td>59.50</td>
    <td>77.00</td>
  </tr>
</table>

Table 2: The impact of adding text hints on different models.

excels at identifying visual cues, while Claude-3.0 performs the worst, with GPT-4.0 and Claude-3.5 falling in between (see Appendix C.3 for case studies). This finding confirms a positive correlation between an MLLM’s ability to identify visual cues, distinguish contextual variations, and its overall multimodal causal reasoning performance.

Text Hints. Since we verified a positive correlation between multimodal causal reasoning and visual cue perception, the next question is whether text hints can compensate for shortcomings in visual cue perception. To explore this, we use the contextual descriptions generated during dataset creation as dense captions, as they provide detailed raw information while preserving correct semantic meanings. Table 2 shows that adding text hints significantly improves MLLMs’ performance, suggesting that enhancing visual cue identification is a promising avenue for improving cross-modal generalization.

5.3 Generalization Enhancement

Based on our above analysis, the most crucial factor affecting MLLMs’ cross-modal generalization is the ability to identify visual cues. In response, we propose VcCoT, a method designed to enhance visual cue identification for causal inference. Inspired by MMCoT (Zhang et al., 2023) and CCot (Mitra et al., 2024), our approach first converts images into dense captions, then extracts visual details categorized as Character and Background. Finally, these cues guide the MLLMs’ reasoning process, as illustrated in Figure 13. Table 3 demonstrates that VcCoT achieves superior performance than others. We also show some qualitative results in Appendix C.4.

<table>
  <tr>
    <th>Strategy</th>
    <th>Style</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <td colspan="5">GPT-01 (<a href="https://openai.com/blog/gpt-4/">OpenAI, 2024b</a>)</td>
  </tr>
  <tr>
    <td>Direct</td>
    <td></td>
    <td>87.50</td>
    <td>62.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td></td>
    <td>86.25</td>
    <td>61.50</td>
    <td>76.00</td>
  </tr>
  <tr>
    <td>CCoT</td>
    <td>Mixture</td>
    <td>88.00</td>
    <td>64.00</td>
    <td>79.50</td>
  </tr>
  <tr>
    <td>MMCoT</td>
    <td></td>
    <td>84.25</td>
    <td>60.5</td>
    <td>86.50</td>
  </tr>
  <tr>
    <td>VcCoT</td>
    <td></td>
    <td>89.75</td>
    <td>66.5</td>
    <td>83.00</td>
  </tr>
</table>

Table 3: The performance of different CoT strategies on MuCR benchmark.

Figure 13: The structure of our VcCoT. Best viewed by zooming in.

6 Conclusion

In this paper, we introduce MuCR, a novel multi-modal causal reasoning benchmark that challenges MLLMs to discern causal links across different modalities by leveraging synthetic siamese images and text pairs. We also propose comprehensive metrics to assess MLLMs’ understanding from multiple perspectives, including image-level alignment, phrase comprehension, and sentence-level explanation. Our experimental results reveal that current MLLMs exhibit a cross-modal gap in causal reasoning compared to their strong performance in purely textual settings. In-depth analysis highlights that effective visual cue identification is key to enhancing generalization, as MLLMs often struggle with implicit causal dependencies hidden in visual details. In response, we propose VcCoT, a method
designed to improve visual cue identification for causal inference, with experimental results demonstrating its effectiveness.

7 Limitation

Although our research provides a comprehensive analysis of the potential factors affecting generalization from visual components, it has two notable limitations. First, as noted by Wang et al. (2024a), cross-linguistic variations can influence performance and may require transfer learning. Figure 14 presents a simple comparison of transferring the question language from English to Chinese using the C2E score, indicating that cross-linguistic factors affect the final output of the models. However, due to human resource constraints, we did not extend this study to the CP and EXP scores, as these metrics require human reannotation of cue phrases and sentence explanations.

![A comparison of different models on the C2E score with cross-linguistic setting.](page_352_682_495_312.png)

Figure 14: A comparison of different models on the C2E score with cross-linguistic setting.

<table>
  <tr>
    <th>Fine-tune</th>
    <th>Style</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <td colspan="5">LLaVA-v1.6 (<a href="https://arxiv.org/abs/2402.16048">Liu et al., 2024b</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>23.50</td>
    <td>11.00</td>
    <td>16.50</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>20.00</td>
    <td>13.75</td>
    <td>15.25</td>
  </tr>
  <tr>
    <td colspan="5">MiniGPT4-v2 (<a href="https://arxiv.org/abs/2307.14974">Zhu et al., 2023</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>17.75</td>
    <td>11.50</td>
    <td>15.25</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>19.00</td>
    <td>13.50</td>
    <td>16.00</td>
  </tr>
  <tr>
    <td colspan="5">InstructBLIP (<a href="https://arxiv.org/abs/2301.09019">Dai et al., 2023</a>)</td>
  </tr>
  <tr>
    <td>Before</td>
    <td>Mixture</td>
    <td>12.25</td>
    <td>6.50</td>
    <td>9.50</td>
  </tr>
  <tr>
    <td>After</td>
    <td>Mixture</td>
    <td>7.50</td>
    <td>3.25</td>
    <td>4.75</td>
  </tr>
</table>

Table 4: The impact of direct fine-tuning on different models.

Additionally, we explored fine-tuning a few lightweight open-source models. As shown in Table 4, direct fine-tuning with the correct choices did not improve and in some cases even decreased the performance of these models. Our observations indicate that these models fail to capture the causal links between cause-and-effect images through fine-tuning. Notably, InstructBLIP even lost its ability to caption images accurately, exhibiting severe hallucinations. Due to limited resources, we did not investigate whether reinforcement learning (Guo et al., 2025) or alternative strategies (Niklas et al., 2025) could further address the generalization problem on larger models such as Qwen2.5-VL (Yang et al., 2024) or LLama3.2-Vision (Meta, 2024).

References

Swagata Ashwani, Kshiteesh Hegde, Nishith Reddy Mannuru, Mayank Jindal, Dushyant Singh Sengar, Krishna Chaitanya Rao Kathala, Dishant Banga, Vinija Jain, and Aman Chadha. 2024. Cause and effect: Can large language models truly understand causality? In Proceedings of the AAAI Symposium Series, pages 2–9.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Itsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390.

Abdolmahdi Bagheri, Matin Alinejad, Kevin Bello, and Alireza Akhondi-Asl. 2024. C2p: Featuring large language models with causal reasoning. arXiv preprint arXiv:2407.18069.

Guangsheng Bao, Hongbo Zhang, Linyi Yang, Cunxiang Wang, and Yue Zhang. 2024. LMs with chain-of-thought are non-causal reasoners. arXiv preprint arXiv:2402.16048.

Nitzan Bitton-Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. 2024. Visual riddles: a commonsense and world knowledge challenge for large vision and language models. arXiv preprint arXiv:2407.19474.

Jingyuan Chen, Fuchen Long, Jie An, Zhaofan Qiu, Ting Yao, Jiebo Luo, and Tao Mei. 2025. Ouroboros-diffusion: Exploring consistent content generation in tuning-free long video diffusion. arXiv preprint arXiv:2501.09019.

ClaudeAI. 2024a. Claude 3: Anthropic’s large language model. https://www.anthropic.com/clause.

ClaudeAI. 2024b. Claude 3.5 sonnet. https://www.anthropic.com/news/clause-3-5-sonnet.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of
the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

DeepAI. 2024. Deepai: Image generation. https://deepai.org/machine-learning-model/text2img.

DeepMind. 2024. Gemini 1.5 models. https://deepmind.google/technologies/gemini/.

DeepMind. 2025. Gemini 2.0 models. https://deepmind.google/technologies/gemini/.

Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-care: a new dataset for exploring explainable causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 432–446. Association for Computational Linguistics.

FLUXAI. 2024. Introducing flux.l tools. https://blackforestlabs.ai/flux-l-tools/.

Jörg Frohberg and Frank Binder. 2022. CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2126–2140, Marseille, France, European Language Resources Association.

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024. Blink: Multi-modal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer.

Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2024. Lcm-lookahead for encoder-based text-to-image personalization. In European Conference on Computer Vision, pages 322–340. Springer.

Rohit Girdhar and Deva Ramanan. 2020. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR.

Nitzan Bitton Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton, Idan Szpektor, Amir Globerson, and Yuval Elovici. 2024. Visual riddles: a commonsense and world knowledge challenge for large vision and language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Jack Hessel, Jena D Hwang, Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. 2022. The abduction of sherlock holmes: A dataset for visual abductive reasoning. In European Conference on Computer Vision, pages 558–575. Springer.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR.

Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, LYU Zhiheng, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, et al. 2023. Cladder: Assessing causal reasoning in language models. In Thirty-seventh conference on neural information processing systems.

Zhijing Jin, Jiariu Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. 2024. Can large language models infer causation from correlation? In ICLR 2024.

Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Łukasz Stafiniak, Tibor Tihon, et al. 2020. Measuring compositional generalization: A comprehensive method on realistic data. In 8th International Conference on Learning Representations, ICLR 2020.

Emre Cicimcan, Robert Ness, Amit Sharma, and Chen-hao Tan. 2023. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050.

Najoung Kim and Tal Linzen. 2020. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online. Association for Computational Linguistics.

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895.

Jianling Li, Meishan Zhang, Peiming Guo, Min Zhang, and Yue Zhang. 2023. LLM-enhanced self-training for cross-domain constituency parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8174–8185, Singapore. Association for Computational Linguistics.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. 2014. Deepreid: Deep filter pairing neural network for person re-identification. In CVPR.

Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2021. Guided generation of cause and effect. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.

Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tianrui Guan, Haoliang Wang, Tong Yu, et al. 2024c. Large language models and causal inference in collaboration: A comprehensive survey. arXiv preprint arXiv:2403.09606.

Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. 2024. Videostudio: Generating consistent-content and multi-scene videos. In European Conference on Computer Vision.

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.

Meta. 2024. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-version-edge-mobile-devices/.

Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. 2024. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14431.

Muennighoff Niklas, Yang Zitong, Shi Weijia, Li Xiang Lisa, Fei-Fei Li, Hajishirzi Hannaneh, Zettle-moyer Luke, Liang Percy, Candès Emmanuel, and Hashimoto Tatsunori. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.

OpenAI. 2024a. Hello gpt-4o. https://openai.com/index /hello-gpt-4o/.

OpenAI. 2024b. Introducing openai o1-preview. https://openai.com/index/introducing-openai-o1-preview.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Dall-e: Creating images from text. https://www.openai.com/blog/dall-e.

Angelika Romanou, Syrielle Montariol, Debitj Paul, Leo Laugier, Karl Aberer, and Antoine Bosselut. 2023. Crab: Assessing the strength of causal relationships between real-world events. arXiv preprint arXiv:2311.04284.

Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-Lin Wu, Xuezhe Ma, and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021., volume ACL/IJCNLP 2021 of Findings of ACL, pages 883–898. Association for Computational Linguistics.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.

Stability AI. 2023. Stability ai: Image generation. https://stability.ai.

Aniket Vashishtha, Abbavaram Gowtham Reddy, Abhinav Kumar, Saketh Bachu, Vineeth N Balasubramanian, and Amit Sharma. 2023. Causal inference using llm-guided discovery. arXiv preprint arXiv:2310.15117.

Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840.

Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiaron Xu, and Fandong Meng. 2024a. Cross-lingual knowledge editing in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11676–11686, Bangkok, Thailand. Association for Computational Linguistics.

Qixun Wang, Xu Bai, Haofan Wang, ZeKui Qin, and Anthony Chen. 2024b. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. 2021. Acre: Abstract causal reasoning beyond covariation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 10643–10653.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multi-modal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923.

Xiaomao Zhou, Qingmin Jia, Yujiiao Hu, Renchao Xie, Tao Huang, and F Richard Yu. 2024. Geng: An llm-based generic time series data generation approach for edge intelligence via cross-domain collaboration. In IEEE INFOCOM 2024-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pages 1–6. IEEE.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.
Appendix

A  The MuCR Dataset

A.1 Task Formulation

As shown in Figure 2 (b), our dataset \( \mathcal{D} := \{(Q, G^*(A), \{B^{(i)}\}_{i=1}^{4})^{(k)}\}_{k=1}^{N} \) consisting of \( N \) triples, each contains a question \( Q \), an input \( G^*(A) \) (where * represents the input form), and four potential choices \( \{B^{(i)}\}_{i=1}^{4} \). The MLLMs are required to according to the question \( Q \) and an input \( G^*(A) \) to select the correct answer from four potential choices \( \{B^{(i)}\}_{i=1}^{4} \). The goal of this benchmark is to determine whether the input form (*) affects the MLLMs’ prediction accuracy. To this end, the biggest challenge is defined as follows:

\[
G^{text}(A) \overset{\text{semantic}}{\approx} G^{multi}(A),
\]

where \( \overset{\text{semantic}}{\approx} \) means \( G^*(A) \) retains identical or closely aligned semantic meaning across different modalities. To address this, we propose a novel transfer strategy that harnesses the linguistic capabilities of LLMs alongside the image generation abilities of diffusion models, effectively preserving semantic content while altering the input form.

A.2 Overall Structure

Section 3 only illustrates the simplified process of our MuCR benchmark generation. Here, we delve into more details about the generation process and the corresponding prompts. Figure 15 showcases the detailed generation process of a weather-related causal case in our MuCR dataset. Our process begins with generating core caption pairs, each consisting of one caption describing the cause and the other stating the effect. We then leverage the language capabilities of LLMs to entail these paired captions into contextually relevant descriptions, enhancing the consistency of sentences to facilitate the creation of cause-and-effect image pairs. Then, we employ diffusion models to generate numerous Siamese images based on these descriptions. Finally, we annotate cue phrases and causality explanations for each pair.

A.3 Generating Core Caption Pairs

Our MuCR benchmark begins with the creation of core caption pairs, where one caption outlines the cause and the other describes the effect. These pairs maintain semantic causality and serve two roles. First, they function as textual causal inference cases to challenge MLLMs’ textual reasoning ability. Second, they guide the subsequent synthesis of Siamese images. As shown in Figure 16, we employ a structured refinement loop that transforms initial brainstorming ideas into precise caption pairs, clearly depicting the cause-and-effect relationships. This process is guided by the principle: “Whether the expression is concrete and can be effectively represented through visual means”. Here, we discuss the rationale behind this rule and explain why volunteers are instructed to create core caption pairs in accordance with it.

Figure 17 compares the initial spark and core caption pairs in image synthesis. The comparison reveals that the initial spark often contains semantically ambiguous elements, leading to visual gaps in the generated images. For instance, the phrase “the baker left the cake in the oven” might result in an image depicting only a cake in the oven, as the diffusion model may struggle to interpret or visually represent the action “left”. Another issue is subject conflict. For example, the phrase “the food became inedible” might simply produce an image of unappealing food on a plate. However, within a cause-and-effect scenario, a human would easily infer that “food” refers specifically to the “cake.” In contrast, our core caption pairs resolve these ambiguities by translating them into more concrete actions, such as replacing “careless” with “played his phone.” This refinement significantly improves the quality of the generated images and the semantic causality between the pairs.

We ask the volunteers to design four paired captions as a group, each sharing similar causalities but containing different visual cues. These groups are intended to explore the capability of distinguishing

![A detailed example of generating our MuCR dataset. Best viewed by zooming in.](page_1012_180_393_312.png)

Figure 15: A detailed example of generating our MuCR dataset. Best viewed by zooming in.
<table>
  <tr>
    <th>Cause</th>
    <th>Effect</th>
    <th>Category</th>
  </tr>
  <tr>
    <td>The man drove his car at an excessive speed.</td>
    <td>The man got a speeding ticket.</td>
    <td>Person</td>
  </tr>
  <tr>
    <td>The woman drove her car at an excessive speed.</td>
    <td>The woman was pulled over by the policeman.</td>
    <td>Person</td>
  </tr>
  <tr>
    <td>The old man drove his car at an excessive speed.</td>
    <td>The old man was pulled over by the policeman.</td>
    <td>Person</td>
  </tr>
  <tr>
    <td>The old woman drove her car at an excessive speed.</td>
    <td>The old woman got a speeding ticket.</td>
    <td>Person</td>
  </tr>
  <tr>
    <td>The shark is attacking the fish.</td>
    <td>The fish got wounded on its back.</td>
    <td>Animal</td>
  </tr>
  <tr>
    <td>The shark is attacking the seal.</td>
    <td>The seal got wounded on its back.</td>
    <td>Animal</td>
  </tr>
  <tr>
    <td>The shark is attacking the manta ray.</td>
    <td>The manta ray got wounded and bleeding.</td>
    <td>Animal</td>
  </tr>
  <tr>
    <td>The shark is attacking the penguin.</td>
    <td>The penguin got wounded and bleeding.</td>
    <td>Animal</td>
  </tr>
  <tr>
    <td>The chrysanthemum is blooming.</td>
    <td>The chrysanthemum attracting bees to collect nectar.</td>
    <td>Plant</td>
  </tr>
  <tr>
    <td>The tulip is blooming.</td>
    <td>The tulip attracting bees to collect nectar.</td>
    <td>Plant</td>
  </tr>
  <tr>
    <td>The rose is blooming.</td>
    <td>The rose attracting bees to collect nectar.</td>
    <td>Plant</td>
  </tr>
  <tr>
    <td>The jasmine is blooming.</td>
    <td>The jasmine attracting bees to collect nectar.</td>
    <td>Plant</td>
  </tr>
  <tr>
    <td>The rabbit worked hard.</td>
    <td>The rabbit earn much money.</td>
    <td>Character</td>
  </tr>
  <tr>
    <td>The monkey worked hard.</td>
    <td>The monkey earn much money.</td>
    <td>Character</td>
  </tr>
  <tr>
    <td>The bear worked hard.</td>
    <td>The bear earn much money.</td>
    <td>Character</td>
  </tr>
  <tr>
    <td>The fox worked hard.</td>
    <td>The fox earn much money.</td>
    <td>Character</td>
  </tr>
  <tr>
    <td>The gardener planted a tree.</td>
    <td>The tree grew tall.</td>
    <td>Mixture</td>
  </tr>
  <tr>
    <td>The farmer planted seeds.</td>
    <td>The seeds sprouted into crops.</td>
    <td>Mixture</td>
  </tr>
  <tr>
    <td>The child planted flowers.</td>
    <td>The flowers bloomed in the garden.</td>
    <td>Mixture</td>
  </tr>
  <tr>
    <td>The woman planted herbs.</td>
    <td>The herbs grew in the pot.</td>
    <td>Mixture</td>
  </tr>
</table>

Table 5: Case studies for the paired caption generation process.

![The process of generating paired captions through refinement loops, ensuring the final captions are precise and can be effectively represented through visual means.](page_184_670_627_312.png)

Figure 16: The process of generating paired captions through refinement loops, ensuring the final captions are precise and can be effectively represented through visual means.

![A comparison of directly using initial spark and our core caption pairs to generate cause-and-effect images through the diffusion model.](page_900_670_420_312.png)

Figure 17: A comparison of directly using initial spark and our core caption pairs to generate cause-and-effect images through the diffusion model.

similar causalities occurring in different subjects across various scenarios. Furthermore, to maintain the diversity of our dataset, we include a portion of non-human cases. While many causality scenarios feature humans as subjects, we also incorporate cases involving animals, plants, comic characters, and their interactions. Table 5 shows generated paired-caption examples (i.e., four captions sharing similar causalities but involving different visual cues are paired as a group) for various scenarios (i.e., cases involving humans, animals, plants, comic characters, and mixtures). Abstract expressions are concretized during the paired-caption generation process according to the causality. For instance, the scenario “driving at excessive speed” is rephrased in terms of its potential outcomes, such as “getting a speeding ticket” or “being pulled over by a police officer”. Similarly, the concept of “blooming” is illustrated through its possible consequence, “attracting bees to gather nectar”. This process leverages causal reasoning to ground abstract ideas in real-world outcomes, thereby enhancing the intelligibility and reproducibility of the generated captions.
(a) The drawback of missing context

Caption 1:
"The child was drawing a picture in the room."

Requirement
Select two images that best represent the cause-and-effect captions?

Caption 2:
"The completed picture was colorful."

Consideration
Age: ✓
Hair: ✓
Clothes: ✗
Background: ✓

(b) The advantage incorporating context

Description 1:
"natural photograph, a child, yellow shirt, blonde hair, draw a picture..."

Requirement
Do the two images illustrate the core causality?

Description 2:
"natural photograph, a child, yellow shirt, blonde hair, the completed picture..."

Consideration
Age: ✓
Hair: ✓
Clothes: ✓
Background: ✓

Figure 18: An example of core captions vs contextual descriptions in cause-and-effect image synthesis.

A.4 Producing Contextual Description Pairs

The absence of crucial visual cues could introduce randomness in image creation, which may lead to inconsistencies and potentially undermine the perceived causality between the siamese images. Recent research on content consistency (Chen et al., 2025; Long et al., 2024) has become popular in long video generation by maintaining coherent content across frames. For image content consistency, Figure 18 highlights the drawbacks of missing context and the advantages of incorporating context. As shown in Figure 18 (a), although the two columns of images accurately represent the core caption, mismatched clothing disrupts the sense of causality, making it difficult to form coherent pairs. In contrast, the example in Figure 18 (b) demonstrates that incorporating contextual information and transforming core captions into contextual descriptions effectively resolves this issue and reduces randomness in image synthesis. To achieve this, we leverage the linguistic capabilities of LLMs to enhance core caption pairs by enriching contextual details such as appearance, clothing, environment, and atmosphere. Additionally, we introduce subtle changes, such as variations in facial expressions, within the contextual description pairs to reflect the passage of time. These detailed variations emphasize the impact of causality over time, making the connection between siamese images more natural and coherent.

ID Preserving Technique

Style: + ID: → Output:

Our Prompt-Driven Technique

Cause:
"A natural photograph of a beachgoer, a woman with long blonde hair, wearing a red bikini and lying on a large yellow towel on a sandy beach. She appears relaxed and content, her eyes closed, enjoying the moment despite the foreboding dark clouds gathering in the sky. The expansive ocean and the horizon under the dark, brooding clouds enhance the image's contrast."

Effect:
A natural photograph of the beachgoer, with long blonde hair, now drenched and sitting on the wet yellow towel as rain pours down heavily. Her red bikini clings to her body, and her face shows a mix of surprise and discomfort as she shields her eyes from the rain. The beach setting remains consistent, with dark, turbulent skies above.

Output:

Figure 19: A comparison of identity-preserving technique and our prompt-driven technique on image synthesis.

We also compare identity-preserving techniques with our prompt-guidance method (Figure 19). Traditional identity-preserving image synthesis methods (e.g., LCM (Gal et al., 2024) and IP-Adapter (Ye et al., 2023)) focus on image personalization by retaining identity details through a region encoder during the generation process (Wang et al., 2024b). However, this approach leads to two major issues. First, most existing identity-preserving techniques rely heavily on guided images, which limits their capacity for semantically-driven image generation and requires finding a suitable ID image for each causal scenario. Second, as the name suggests, identity-preserving methods focus primarily on maintaining facial identity (appearance) but struggle to incorporate cause-and-effect relationships across images. In contrast, our causal-and-effect image synthesis approach leverages the linguistic capabilities of large language models (LLMs) to integrate a richer spectrum of contextual information. It not only preserves human facial identity (appearance), but also accounts for additional details (e.g., clothing, environment, and overall atmosphere). This ensures that images remain coherent even when modifications are introduced through causal reasoning. The producing contextual description pairs prompt is organised as: "Task Overview: You need to convert causal-relevant captions into detailed image descriptions for generat-
ing images. Ensure the following: (A) Consistency in Characters: Use one sentence to describe the person’s appearance in each image description. Make sure two sentences in the two descriptions share the same information without using words like same or similar. (B) Face expression in Characters: If the descriptions contain the person’s appearance, please directly add a sentence following to describe the person’s facial expression, matching the scene. (C) Activities or Behaviors: Cause descriptions should exclusively detail the causal activities or behaviors, while effect descriptions should exclusively detail the resultant activities or behaviors. (D) Consistency in Scenes: If the scene remains unchanged between cause and effect, ensure the background description is identical in both descriptions. (E) Clear Causal Link: (1) Enhance Cause: Provide concrete details about what led to the effect. (2) Improve Effect: Ensure the effect is both visually and logically linked to the cause, using relatable or observable descriptions."

A.5 Siamese Images and Annotations

In this section, we show some high-quality examples as follows:

In the plant category, as shown in Figure 20, take the jasmine flower pair: the cause image shows a blooming jasmine flower, while the effect image features a group of bees swarming around it. For this pair, we select “bloom” as the positive cue phrase and “bee”, “flower”, and “sunshine” as the negative ones, aligning with the visual information. The annotation emphasizes the connection between the flower’s blooming and the attraction of bees.

In the character category, as shown in Figure 21, consider the cat pair: the first image shows a cat lifting weights at the gym, while the second image depicts the cat gaining strength and muscle. For this, “fitness” is used as the positive cue phrase, with “gym”, “muscle”, and “dumbbells” as the negative ones, matching the visual content. The annotation focuses on the connection between consistent workouts and muscle gains.

In the mixture category, as shown in Figure 22, take the female planting pair: the cause image shows a woman planting seedlings in a garden, while the effect image displays the same woman smiling and holding a large pot of flourishing plants. Here, “plant” is the positive cue phrase, and “grow”, “green”, and “land” are the negative ones, aligning with the visual information. The annotation emphasizes the relationship between her nurturing care and the plant’s growth, along with her pride.

B Experiments

In this section, we delve into extended experiments and provide supplementary details that were not included in the main paper for the sake of clarity and brevity.

B.1 Experimental Results

As discussed in Section 4.1, we did not include the currently popular models DeepSeek-R1 (Guo et al., 2025) and DeepSeek-V3 (Liu et al., 2024a) in the main paper. Here, we provide a brief comparison of their text-based performance against GPT-01 (OpenAI, 2024b). Table 7 shows that DeepSeek-R1 achieves results comparable to GPT-01 in the text domain, while DeepSeek-V3 performs slightly less effectively.

In addition, we provide a detailed breakdown of each model’s performance on our MuCR benchmark. Table 6 presents these results. We observe that all popular MLLMs significantly outperform random chance, whereas most lightweight open-source models perform below the random baseline of 25%. This indicates that the latter group lacks robust causal reasoning capabilities.

C Cross-modal Generalization Analysis and Enhancement

C.1 Picture Style

Here, we present a detailed case analysis comparing the influence of picture style on Claude-3.5’s predictions, as illustrated in Figure 23.

In the black-white images, Image 1 shows a warthog bending down to drink water, placing it in a vulnerable position. The cause is clear—the warthog’s need to drink compels it to lower its head, thus reducing its awareness of potential threats. Among the follow-up images, Image 5 best represents the effect: it shows a crocodile emerging from the water, poised to attack a drinking animal, maintaining consistent compositional elements such as the animal at the water’s edge and the predator’s emergence. While Images 2, 3, and 4 depict similar scenarios with different animals, Image 5 most directly mirrors the cause-and-effect relationship suggested by Image 1. However, the analysis in this style tends to lack detail in some of the incorrect answers, which could potentially influence the model’s predictive accuracy in nuanced cases.
Explanation

"Cause: the flower is in full bloom. Effect: the flower attracts numerous bees. The causal link between them is that the flower's blooming attracts bees for pollination."

Explanation

"Cause: the red tulips are shown in full bloom. Effect: the tulips attract numerous bees. The causal link between them is that the blooming tulips draw the bees."

Explanation

"Cause: the red rose is blooming in the garden. Effect: the red rose is attracting bees that are gathering around it. The causal link between them is that the rose's blooming has attracted the bees."

Explanation

"Cause: the jasmine flower is in full bloom. Effect: the bees are seen gathering around the blooming jasmine flowers. The causal link between them is that the flower's blooming has attracted the bees."

Figure 20: Example 1 - Plant
Explanation
"Cause: the rabbit is lifting dumbbells in a gym. Effect: the rabbit gaining strength. The causal link between them is that the rabbit's workout in the gym leads to it gaining strength and showing off its muscles."

Explanation
"Cause: the cat is lifting weights at the gym. Effect: the cat gains strength and muscle. The causal link between them is that the cat's consistent weightlifting and workout routine at the gym resulted in gaining muscle strength."

Explanation
"Cause: the dog is at the gym, preparing for a workout. Effect: the dog is shown with well-developed muscles. The causal link between them is that the dog's consistent workout routine at the gym resulted in significant muscle growth and increased strength."

Explanation
"Cause: the eagle is lifting a dumbbell at the gym. Effect: the eagle is flexing its muscles. The causal link between them is that the eagle's consistent weightlifting and workout routine at the gym resulted in gaining significant muscle strength and mass."

Figure 21: Example 2 - Character
Explanation

"Cause: the man is planting a tree in a garden. Effect: the tree is fully grown with vibrant pink flowers. The causal link between them is that the man's efforts in planting and nurturing the tree led to its growth and blossoming."

Explanation

"Cause: the farmer planting seeds in a field. Effect: the crops fill the lush, green field with rows of growth. The causal link between them is that the farmer's planting efforts resulted in the successful growth of the crops."

Explanation

"Cause: the young child carefully planting flowers in a garden. Effect: this child standing proudly among a vibrant array of blooming flowers. The causal link between them is that the child's planting efforts resulted in the beautiful, blossoming garden."

Explanation

"Cause: the woman carefully planting seedlings in a garden. Effect: this woman smiling and holding a large pot with flourishing plants. The causal link between them is that the woman's diligent planting and care for the seedlings led to their healthy growth, resulting in the flourishing plants she holds with pride."
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Text-based Form</th>
    <th colspan="3">Multimodal-based Form</th>
  </tr>
  <tr>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <th colspan="7">Popular MLLMs</th>
  </tr>
  <tr>
    <td>GPT-01 (OpenAI, 2024b)</td>
    <td>94.00</td>
    <td>75.50</td>
    <td>93.00</td>
    <td>87.50</td>
    <td>62.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <td>GPT-4o (OpenAI, 2024a)</td>
    <td>92.75</td>
    <td>71.75</td>
    <td>91.50</td>
    <td>81.25</td>
    <td>57.25</td>
    <td>72.50</td>
  </tr>
  <tr>
    <td>Claude-3.5 (ClaudeAI, 2024b)</td>
    <td>92.50</td>
    <td>77.00</td>
    <td>92.75</td>
    <td>83.50</td>
    <td>59.75</td>
    <td>77.5</td>
  </tr>
  <tr>
    <td>Claude-3.0 (ClaudeAI, 2024a)</td>
    <td>88.25</td>
    <td>66.75</td>
    <td>82.00</td>
    <td>58.00</td>
    <td>50.25</td>
    <td>57.00</td>
  </tr>
  <tr>
    <td>Gemini-2.0 (DeepMind, 2025)</td>
    <td>93.00</td>
    <td>76.00</td>
    <td>90.50</td>
    <td>75.50</td>
    <td>60.75</td>
    <td>70.25</td>
  </tr>
  <tr>
    <td>Gemini-1.5 (DeepMind, 2024)</td>
    <td>89.00</td>
    <td>73.25</td>
    <td>91.50</td>
    <td>66.50</td>
    <td>58.50</td>
    <td>70.75</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL (Yang et al., 2024)</td>
    <td>89.00</td>
    <td>66.00</td>
    <td>90.00</td>
    <td>77.00</td>
    <td>54.50</td>
    <td>72.00</td>
  </tr>
  <tr>
    <td>Llama3.2-Vision (Meta, 2024)</td>
    <td>83.50</td>
    <td>62.50</td>
    <td>86.00</td>
    <td>54.00</td>
    <td>48.25</td>
    <td>53.25</td>
  </tr>
  <tr>
    <th colspan="7">Lightweight Open-source Models</th>
  </tr>
  <tr>
    <td>LLaVA-NeXT (Li et al., 2024)</td>
    <td>54.50</td>
    <td>37.50</td>
    <td>48.00</td>
    <td>29.00</td>
    <td>17.00</td>
    <td>21.00</td>
  </tr>
  <tr>
    <td>OpenFlamingo-v2 (Awadalla et al., 2023)</td>
    <td>23.00</td>
    <td>16.00</td>
    <td>17.25</td>
    <td>20.00</td>
    <td>9.75</td>
    <td>18.00</td>
  </tr>
  <tr>
    <td>LLaVA-v1.6 (Liu et al., 2024b)</td>
    <td>25.25</td>
    <td>17.25</td>
    <td>18.00</td>
    <td>23.50</td>
    <td>11.00</td>
    <td>16.50</td>
  </tr>
  <tr>
    <td>MiniGPT-4v2 (Zhu et al., 2023)</td>
    <td>13.50</td>
    <td>18.50</td>
    <td>16.75</td>
    <td>17.75</td>
    <td>11.50</td>
    <td>15.25</td>
  </tr>
  <tr>
    <td>InstructBLIP (Dai et al., 2023)</td>
    <td>14.50</td>
    <td>10.00</td>
    <td>8.50</td>
    <td>12.25</td>
    <td>6.50</td>
    <td>9.50</td>
  </tr>
  <tr>
    <th>Human</th>
    <th>96.75</th>
    <th>91.00</th>
    <th>98.50</th>
    <th>95.50</th>
    <th>89.50</th>
    <th>98.50</th>
  </tr>
</table>

Table 6: Main experimental results of different models on our MuCR benchmark.

<table>
  <tr>
    <th>Input Form</th>
    <th>Style</th>
    <th>C2E</th>
    <th>CP</th>
    <th>EXP</th>
  </tr>
  <tr>
    <th colspan="5">GPT-01 (OpenAI, 2024b)</th>
  </tr>
  <tr>
    <td>Text</td>
    <td>\</td>
    <td>94.00</td>
    <td>75.50</td>
    <td>93.00</td>
  </tr>
  <tr>
    <td>Image</td>
    <td>Mixture</td>
    <td>87.50</td>
    <td>62.00</td>
    <td>78.00</td>
  </tr>
  <tr>
    <th colspan="5">DeepSeek-R1 (Guo et al., 2025)</th>
  </tr>
  <tr>
    <td>Text</td>
    <td>\</td>
    <td>96.00</td>
    <td>73.50</td>
    <td>95.00</td>
  </tr>
  <tr>
    <th colspan="5">DeepSeek-V3 (Li et al., 2014)</th>
  </tr>
  <tr>
    <td>Text</td>
    <td>\</td>
    <td>91.50</td>
    <td>72.25</td>
    <td>92.00</td>
  </tr>
</table>

Table 7: The performance comparison between GPT-01 and DeepSeek models in the text domain on MuCR.

In contrast, the comic style analysis also begins with Image 1, where a warthog is depicted looking down at ripples in the water, seemingly unaware of any lurking danger. The potential effects are illustrated across multiple images: Image 2 shows a wildebeest encountering a crocodile, Image 3 depicts a zebra facing a crocodile, Image 4 features a gazelle or antelope in a similar scenario, and Image 5 shows another warthog confronting a crocodile. Here, Image 5 stands out as the best representation of the effect because it features the same animal as in the cause image in a comparable setting, now facing the implied threat signaled by the ripples. The consistent composition and environmental context reinforce the direct cause-and-effect relationship.

The comic style analysis provides a richer context and more detailed narrative for the causal relationship, whereas the black-white analysis, although accurate in identifying the correct image, offers less detailed reasoning for some incorrect options.

C.2 Form of Visual Input

Our case analysis demonstrates that, compared to Form-3, Form-1 and Form-2 impose limitations on MLLMs’ ability to recognize and leverage critical visual cues necessary for multimodal causal reasoning. As shown in Figure 24, Form-3 provides GPT-4o with direct visual information, enabling it to successfully identify essential details, such as the continuity in a person’s appearance across cause-and-effect images. This was evident in GPT-4o’s output, where it correctly determined that the woman in the cause image, overwhelmed by paperwork, was the same individual in the effect image, now engaged in a serious discussion about work. This recognition of visual consistency is crucial for establishing causal relationships. However, when using Form-1, GPT-4o was unable to incorporate this specific visual cue and instead selected a different effect image (a generic team meeting) based on a more abstract textual interpretation rather than a direct visual correlation.

The key issue with Form-1 and Form-2 is that they rely on structured textual descriptions that predefine categories of reasoning, which may inadvertently filter out implicit but important visual details. These formats encourage MLLMs to focus on generalized textual patterns rather than independently deriving causal relationships from visual features like facial expressions, body language, and scene continuity. By contrast, Form-3 allows GPT-4o to analyze raw visual inputs more freely, enhanc-
ing its ability to establish causal links based on direct visual observation rather than abstracted textual hints. This distinction highlights the potential shortcomings of rigid textual input structures in multimodal causal reasoning tasks. While textual guidance can be helpful, it may also constrain the model’s reasoning process, making it less sensitive to nuanced visual cues. Ensuring that MLLMs receive input formats that preserve rich visual information is therefore essential for improving their ability to perform causal inference in multimodal settings.

C.3 Contextual Variation
Visual cues are crucial for accurate multimodal causal inference because they provide a consistent framework for linking cause and effect. Taking Figure 25 as an example:

• Consistency: Shared elements like the hiking setting, outfit, and subject positioning help the model recognize that the images belong to the same sequence.

• Clear Transitions: Changes in lighting, perspective, and mood signal the progression from cause (a clear, well-lit forest) to effect (a foggy, atmospheric scene), reinforcing the narrative flow.

• Disambiguation: Detailed cues identify Image 2 as the best continuation among similar options, ensuring the causal relationship is accurately maintained.

The analysis shows that visual cues—ranging from consistent environmental context and subject details to nuanced transitions in lighting, perspective, and mood—are crucial for establishing a clear and coherent narrative. These cues allow the model to accurately determine the causal links between images, ensuring that the inferred relationships are both logical and contextually grounded. Without such detailed visual information, the model would face challenges in differentiating between similar scenarios, potentially leading to inaccurate or incomplete causal inferences.

C.4 Qualitative Results of VcCoT
To prove the efficiency of our VcCoT, we provide some qualitative results, as shown in Figure 26.
Question:
Based on the cause image, which of the below four images best represents the effect?
Effects:
(1): (3):
(2): (4):

Case:

Claude-3.5

Among the follow-up images, Image 5 best represents the effect that would logically follow from the cause shown in Image 1.

In Image 1, we see a warthog bending down to drink water, placing itself in a vulnerable position. The cause here is the warthog's need to drink, forcing it to lower its head and reduce its awareness of potential threats.

Image 5 shows what appears to be the natural consequence - a crocodile emerging from the water to potentially attack the animal (in this case a wildebeest) that has come to drink. This maintains the same compositional elements (animal at water's edge, crocodile emerging) and represents the predator-prey relationship that would naturally follow from the vulnerable drinking position shown in Image 1.

While Images 2, 3, and 4 show similar scenes with different animals, Image 5 most directly mirrors the cause-and-effect relationship implied by the first image, showing the moment of potential conflict between the drinking animal and the opportunistic predator.

Question:
Based on the cause image, which of the below four images best represents the effect?
Effects:
(1): (3):
(2): (4):

Case:

Claude-3.5

Looking at the cause and effect relationship in these images:
Image 1 shows a warthog looking down at ripples in the water, seemingly unaware of any danger.
Looking at potential effects in the following images:

• Image 2 shows a wildebeest encountering a crocodile
• Image 3 shows a zebra encountering a crocodile
• Image 4 shows a gazelle/antelope encountering a crocodile
• Image 5 shows another warthog encountering a crocodile

Image 5 best represents the effect as it shows the same type of animal (warthog) from Image 1 in a similar environment, but now facing the revealed danger (crocodile) that was hinted at by the ripples in the first image. The composition and setting are also very similar to Image 1, making it the most direct cause-and-effect relationship among the options presented.
Question:
Based on the cause image, which of the below four images best represents the effect?

Effects:

(1):   (3):
(2):   (4):

Case:

GPT-40

Visual Input Form-1

From the provided image, the cause is represented by a woman in an office setting appearing overwhelmed with paperwork. This suggests a scenario of being overworked, stressed, or under pressure at work.

Now, let's analyze the possible effects depicted in the four images:

1. Image 1: A woman having a discussion with a colleague, possibly about work-related matters.
2. Image 2: A group meeting with individuals in conversation, possibly discussing workload or solutions.
3. Image 3: Two people talking, one appearing to provide guidance or support.
4. Image 4: Two people holding a certificate, which could indicate recognition, achievement, or resolution.

Best Choice: Image 2

Reason: If the cause is an overwhelmed employee, a likely effect is a team meeting or discussion to address workload issues, making Image 2 the most logical representation of the effect.

Visual Input Form-3

Let's analyze the best causal match.

• Cause: A woman looking exhausted at her desk, overwhelmed by paperwork.
• Effect 1: The same woman (with same appearance) in a serious discussion with another woman, possibly about work.
• Effect 2: A tense discussion or argument between men in a work setting.
• Effect 3: A positive business handshake.
• Effect 4: A certificate being awarded, symbolizing success.

Best Answer:

Effect 1 seems to be the best match because it directly follows from the overwhelmed state in the cause image. It suggests she is now addressing work-related issues, possibly discussing a problem that arose due to the workload.

Figure 24: Case study for visual input form influence.
Question:
Based on the cause image, which of the below four images best represents the effect while maintaining visual consistency?

Effects:

(1):        (3):
(2):        (4):

Case:

Claude-3.5

Looking at the visual sequence, Image 2 best represents the effect while maintaining visual consistency with Image 1. Here's why

1. Image 2 maintains the same:
   • Hiking setting
   • Outfit (gray beanie, same jacket, blue backpack)
   • Subject positioning (clear view of the backpack)
   • Professional photo quality and composition

2. It creates a clear cause-and-effect relationship:
   • Image 1 shows the hiker starting their journey in a clear, well-lit forest
   • Image 2 shows them continuing into foggy, atmospheric conditions, suggesting progression deeper into the trail

3. Visual Transition:
   • The lighting changes naturally from clear to misty
   • The perspective shifts from frontal to profile view, creating a narrative flow
   • The mood transitions from preparation to contemplation

While Images 3, 4, and 5 are similar, they either show different clothing (Image 4 with an orange jacket), different compositions, or don't flow as naturally from the initial image. Image 2 provides the most seamless visual continuation from Image 1 while maintaining all key visual elements.

Figure 25: Case study for Contextual Variation.
Question:
Based on the cause image, which of the below four images best represents the effect?

Effects:

(1):        (3):
(2):        (4):

Case:

GPT-o1

VcCoT

Consider the following visual hint

Character:
1. chef
2. white hat
3. white coat
4. wear glass
5. watering plants

Background:
1. garden

Results

Based on the given visual hints and character consistency, the best effect image should maintain the following elements from the cause image:

• Character: A chef wearing a white hat, white coat, and glasses, who was previously watering plants.
• Background: A garden setting.

Analysis of the Images:

1. The second image (chef picking herbs in the garden) closely aligns with the cause image. It maintains:
   o The same character (chef).
   o Consistency in attire (white hat, white coat, and glasses).
   o The same garden environment.
   o A logical cause-and-effect sequence: after watering plants, the chef is now picking herbs, likely for cooking.

2. Other images do not align as well:
   o The third and fourth images depict scientists or researchers in lab coats, working in a controlled indoor environment, which does not match the chef’s attire or setting.
   o The fifth image shows elderly gardeners in an outdoor setting, but they are not chefs and do not maintain the white uniform or glasses.

Conclusion:

The second image (chef picking herbs in the garden) best represents the effect, as it maintains character consistency and follows a logical cause-and-effect progression.

Figure 26: Qualitative results for VcCoT.

Reviewer paper summaries: This paper proposed MuCR, a Multimodal Causal Reasoning benchmark that leverages synthetic siamese images and text pairs to evaluates MLLMs' causal reasoning at the image, phrase, and sentence levels. MuCR has 12K cause-and-effect image pairs spanning various categories (humans, animals, plants, characters, and mixtures) and three styles (photograph, comic, and black-white). This paper evaluated 8 popular MLLMs and 5 lightweight open-source models and conducted multi-faceted analysis of visual form factors and semantic elements. This paper also propose a VcCoT strategy to further enhance visual cue perception, improving the performance of MLLMs on MuCR.

The authors propose MuCR, a benchmark that evaluates the causal reasoning ability of LVLMs at the image, phrase, and sentence levels. The benchmark uses synthetic images-pairs. The authors also propose metrics that assess the models comprehension ability. Finally, they introduce VcCoT a method that highlights visual cues in an image, thereby improving models multimodal causal reasoning. The paper emphasizes the causal aspect within the language-vision domain and evaluate several sota models.
Summary of strengths: 1. This paper identify the limitations of current causal reasoning benchmarks, including failing to evaluate the advanced visual capabilities of the latest MLLMs and offering incomplete analyses of different modalities.
2. This paper propose the MuCR benchmark, which can comprehensively evaluate MLLMs’ causal reasoning ability on two modalities (text and image) from three views (image, phrase, and sentence levels).
3. This paper conduct aboundant experiments, examining the factors that may affect MLLMs' causal reasoning, including visual format factors and visual semantic. The results of these comparative experiments reveal interesting insights and and suggest potential directions for future research. In addition, this paper explore potential enhancement strategies based on their findings, which has a certain value and enlightening significance.

- The objective of the paper, "assessing causal relations in LVLM" is really interesting.

- Part of the dataset creation is done synthetically through LLM/Diffusion models and makes it more scalable than dataset only relying on real images/text

- Extensive evaluations were performed on multiple sota LVLMs.

- Interesting experiments that explore the impact of the form of the visual input, the visual format, and the visual semantic
Summary of weaknesses: The MuCR dataset proposed in this paper has potential flaws and aspects that are confusing to me.

1. Issues with Question Content Design
The design of the question content has certain shortcomings. How is the cross-modal capability evaluated? The paper repeatedly emphasizes that this dataset can assess the cross-modal generalization ability of MLLMs. However, based on my observation of the dataset's question format and examples provided in the paper and supplementary materials, the questions in the dataset either require causal reasoning based on textual information or causal reasoning based on visual information. In the image-based mode, the textual information only appears in fixed and uniform questions, and the clues for causal reasoning are not present in the text but rely solely on visual cues extracted from the images. It seems that there are no questions in the dataset that require combining textual and visual information for causal reasoning.

2. Issues with Question Format Design
The design of the question format has certain limitations. Can multiple-choice questions truly evaluate the model's reasoning ability? The paper adopts a 4-choice-1-answer format across all levels, and the three metrics essentially measure the accuracy of these multiple-choice questions. With this setup, random guessing alone can achieve a performance of 25\%, making it difficult to accurately assess the true reasoning capabilities of different models, especially given the advanced capabilities of models today.

3. Issues with Option Settings
The design of the question options has certain flaws. Does the presence of the same subject in images or text reduce the causal reasoning task to an image-matching task? I observed that, in the data examples provided in the paper, many options mix different subjects. Among the result images, 2 to 3 have different subjects from the cause image, while 1 to 2 share the same subject. In this setup, could the model simply select the correct option based on matching the same person or animal, rather than reasoning causally? How can we distinguish whether the dataset truly evaluates reasoning ability based on the existing experimental results? This remains unclear to me.

4. Additional Concerns Regarding Code and Data
Furthermore, I have additional concerns regarding the supplementary code and data. The data section provides a jsonl file with annotation information and 34 sample images. The code section only includes scripts for calculating scores, combining images, and conducting style analysis, but lacks the critical code for evaluation methods. As a result, I cannot access the complete QA pairs, nor can I generate the dataset for evaluation using the provided code. This makes it impossible to conduct evaluations of MLLMs in line with the paper or to verify the main experimental results, significantly hindering the reproducibility of this work.

- Dataset creation

		○How is the level of challenge instructed for the annotators (=how do you make sure the cause and effect pairs are challenging enough?)

		○ In general most of the examples shared could be answered by only a semantic/contextual aspect similarity. I see them as more about determining whether it’s the same context, person, and place rather than identifying the "correct" causal relationships.  Figure 23: here the answer only relies on the fact that the context is not preserved, not really on the causal aspect of the relations. If we would have seen the same woman in another situation, that would have been more challenging. Figure 24: can be 1 or 2 - also the hat does not look the same from case to any of the effects. Also, all the scenes of effects show the same concept, only the context changes. Theoretically, all could be effects of the cause- context put aside. Figure 26: I don’t see how/why image 1 is an effect of the case. They show 2 different persons (one with a bear and the other without) which happen to be chefs but the scene shows the same concept: someone taking care of plants - not sure if I see any causal relations here.

		○ Figure 17 is a bit confusing to me of how someone could understand that "because" the baker forgot to set a time playing on her phone, the cake burned. It could have been that the baker was setting a timer on the phone. I don’t see from the image why one is the cause of the other.

		○ A cause and its effect are not interchangeable; a cause cannot turn into an effect. How do the authors take care of this constraint when generating the samples? Figure 20, 25, 26 show cause and effect pictures that could be interchangeable

		○ How do you make sure that the contextual descriptions are diverse enough in the data? How do you instruct the LLM ? Could you share the prompts?  Are you using the exact same context for the cause and for the effect captions? I am not sure I understood how the integration of contextual description preserve human facial identity.

		○ How has the dataset been evaluated?  Was it evaluated? Human annotation is involved in generating cue phrases and explanations, but how do you make sure that the images are good (since they are synthetically generated) and coherent? How do you select the tuple from the 20 images generated at each sample?

	- Metrics

		○ From section 3.2, it is very hard to understand the metrics computed. 

			§ Image-level: what is Gtext here? What text are the authors referring to? The captions? The causal explanations? Cue phrases? what does Gmulti means here? Does the sample contain both image and text? And again, which text? How the 3 other potential effects are selected? If both options contain text, how can we be sure the model wont rely only on the text to answer? Examples from the appendix show also what we could call Gimage? Where we only have images as cause and effect (figure 24, 25, 26)

			§ Phrase-Level metric: this metric is unclear to me

			§ Sentence-level metric: this metric is also unclear to me 

	- VcCoT

		○ I don’t see the novelty in this approach
Comments, suggestions and typos: 1. Suggestions for Section 2.2
Many of the works cited in Section 2.2 are from the period of 2018--2022, and the focus is primarily on the generalization capabilities of LLMs. Given the research content of this paper, it is recommended to include more recent works from 2023 onwards, particularly those related to multimodal reasoning in MLLMs. This would help to elaborate on the progress of MLLMs in a wider range of multimodal reasoning tasks.

2. Suggestions for Section 3.2
The three metrics defined in Section 3.2 are essentially the accuracy rates of three types of questions. To make the definitions more accessible to readers, it might be beneficial to streamline the definitions or explicitly clarify that these metrics represent accuracy rates.

3. Suggestions for Section 4.1
In Section 4.1, the model referred to as "GPT-o1" would be more appropriately and formally denoted as "OpenAI o1" to align with standard naming conventions.

-  Image generation: it is not clear how was done the usage of the 3 diffusion models mentioned. Did the authors used them all? How the 10 images were produced? What changes in the setup to produce 10 images? Just seeds?

- Capital letter for "we" line 926

- Figure 23 should be turned 90deg, otherwise it is not readable

- I got a bit lost in all the figures. I would have liked a figure that shows the 3 metrics based on 3 samples with the corresponding pairs and their content (text? Image?)

- General comment: Lots of work seems to have been done (several and diverse experiments and analysis), but can also go against the paper since main concepts (like the metrics, how does the inputs to the model look like at the end?) are not properly explained (I assume due to page limit constraints)

- General question: I was also wondering why not using videos, identifying critical scenes that would show causal relations? Videos contain lots of information with context preserved in time...
Overall assessment scores: 3

1.5