Title: Revealing Multimodal Causality with Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Learning

Large Language Models

Causal Discovery
Abstract: Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data. The implementation code and data are available at https://github.com/JinLi-i/MLLM-CD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces **MLLM-CD**, a framework that leverages multimodal large language models (MLLMs) to perform causal discovery directly from unstructured, multimodal data. Key ideas include:\n\n1. **Contrastive Factor Discovery (CFD)** selects intra- and inter-modal contrastive pair samples and prompts an MLLM to propose candidate factors. The MLLM then consolidates and annotates the dataset into a structured dataset based on the proposed factor set.\n2. Structured data is fed into a causal discovery algorithm, such as FCI, to produce a causal graph \n3. **Multimodal Counterfactual Reasoning (MCR)** generates and validates new counterfactual samples (textual and visual) based on the predicted causal graph, using an MLLM to generate new image and text data. The new data is then added to the dataset for the next round of refinement.\n\nExperiments on a synthetic Multimodal Apple Gastronome (MAG) dataset and a real-world Lung Cancer dataset, evaluated across four MLLMs (GPT-4o, Gemini 2.0, LLaMA-4, Grok-2v), show that MLLM-CD outperforms prior methods in both factor identification and structure recovery.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- This work addresses the challenge of causal discovery in unstructured multimodal data, a topic that is both important and timely.\n- Experiments demonstrate strong performance in both factor identification and structure recovery.\n- This MCR module is well-justified both theoretically and empirically.\n- Writing is generally clear and well-structured.\n\nWeakness\n\n- My main concern is the CFD module. Introducing an extra MLLM (e.g., CLIP) adds complexity, and its necessity is not well justified:\n    - Ablation study (Sec. 4.1) is limited. CFD is compared only with random sampling. Other simple baselines should also be considered, such as:\n        - Random sampling from each category of the target variable;\n        - Pairing samples with the largest difference in the target variable;\n        - Ablate the Intra-modal and Inter-modal independently.\n    - Theoretical justification is misleading. Assumption D.2 claims that $V_{CFD}$ is at least as effective, and potentially superior, in terms of a combined measurement of F1-score. This assumption effectively leads to a circular reasoning fallacy, as it presupposes that the factor sets proposed by CFD is better. This assumption needs to be justify empirically.\n\nQUESTIONS:\nQuestions:\n\n- Have you experimented with other, simpler strategies for sampling data for factor proposals?\n- Could you provide evidence to support Assumption D.2?\n- Which CLIP checkpoint is being used?\n\nI will increase my score if more evidence supporting CFD is provided.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MLLM-CD, a framework for causal discovery from multimodal data using MLLMs. The approach consists of three key components: (1) Contrastive Factor Discovery (CFD), which identifies candidate causal variables by comparing semantically similar multimodal instances with different outcomes; (2) Causal Structure Discovery, which uses these factors to infer the causal graph; and (3) Multimodal Counterfactual Reasoning (MCR), which generates and evaluates counterfactual examples to refine the graph structure. The method is evaluated on a synthetic multimodal dataset (MAG) and a real-world lung cancer dataset, using standard metrics for node and edge accuracy, as well as structural Hamming distance. Results are compared against existing baselines in multimodal and causal discovery.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The overall formulation of the framework is well-structured and logically presented, with a clear separation of its core components. The integration of MLLMs into causal discovery is thoughtfully executed, where its use during counterfactual reasoning for refining causal graphs seems novel and well-motivated.\n\n2. The paper compares the proposed method against a range of baselines, including both traditional and recent MLLM-based approaches.\n\n## Weaknesses\n\n1. The paper evaluates its approach on only two datasets, especially with a relatively small number of causal factors for the Lung Cancer dataset. This could raise questions about the generalizability of the results, especially in more complex or large-scale real-world scenarios.\n\n2. All variables used in the experiments are binary, which simplifies causal discovery but limits the applicability of the method. It remains unclear whether the approach can handle continuous or ordinal variables, or how well it would perform in settings with more diverse data types.\n\nQUESTIONS:\nGiven my limited background in causal inference, I would welcome clarification on the following points:\n\n1. If the model generates incorrect or implausible counterfactuals, how might that affect the final causal graph? Could errors at the counterfactual stage mislead the entire structure discovery process?\n\n2. Even with the use of MLLMs, the model can only infer causes based on what is explicitly represented in the text and images. How does the method handle potential unobserved confounders or latent factors that are not captured in the input modalities?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of inferring the underlying causal structure from multimodal unstructured data. They do this by first using multimodal LLMs (MLLM) to extract the causal factors within and across different modalities. This is done via extracting representations for a sample within a modality and computing the cosine similarity within and across modalities. The samples with high misalignment (high cosine distance) are fed along with prompts as contrastive pairs to extract the factors causing these differences, which are then consolidated. A causal structure is then inferred from these factors via a causal discovery algorithm. To improve the causal discovery, counterfactual data is generated from the MLLM for uncertain causal relationships using the internal knowledge of the MLLM. These are checked for semantic plausibility and causal consistency and then appended to the original dataset. The method is tested on two synthetic datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Although the idea of extracting causal variables from unstructured data using LLMs is not new, using multimodal inputs is definitely an interesting idea.\n\n**Weaknesses:**\n- The counterfactual reasoning module is not rigorous enough and I have a few questions regarding the validity of this approach (see questions). It's not clear to me that this module actually provides the benefits that it claims to provide. A more rigorous testing of this can be done in simpler causal discovery settings where the variables are clearly defined. If it is tested rigorously in these settings, it would be reasonable to use it in the harder unstructured data setting. I'm not sure the semantic plausibility and causal consistency are rigorous enough checks on the validity of counterfactual samples. \n- The theory makes assumptions that make the conclusion of the theory obvious. In my opinion, the aim of the paper should be to show that these assumptions are true. This is lacking in the current work.\n    - Assumption D.2 states that the contrastive probing increases the F1 score of finding the true factors, which is then used to prove that contrastive probing improves structural accuracy. \n    - Similarly, assumption D.6 states that the MLLM can generate valid counterfactuals. If this were true, the MLLM would have internalised the correct structure and functional mechanism to determine the whole causal model. Why not just prompt the MLLM to return the correct model, or generate the correct counterfactual data and simply perform regression to find the causal structure.\n- Engagement with causal representation is missing, which is the task this paper is trying to solve. This area has conditions (e.g. interventions) under which the causal variables and structure is identifiable [1]. An obvious missing baseline is comparison against these methods in identifiable cases. This would show whether the method works when we know the conditions are there to identify the variables and structure. \n- The experiments are only carried out on two different toy datasets, for a method without formal guarantees more experiments with varying conditions would be useful. The different components were not tested thoroughly except for one ablation study. This raises questions like: Does the factor discovery reliably recover the correct factors? Does the counterfactual reasoning actually generate reasonable samples?\n\n[1] https://openreview.net/forum?id=lk2Qk5xjeu\n\nQUESTIONS:\n*Questions regarding counterfactual reasoning module:*\n- Does it actually make sense to augment the \"counterfactual\" sample back into the dataset? For example if the true causal relationship is $X \\to Y$, and the causal discovery algorithm returns $X - Y$ then an intervention on $Y$ (which is what you are doing by setting $V_a=v'_{ka}$ (L207)) would make $X$ and $Y$ independent. Putting this back into the sample doesn't actually help learn the relationship $X \\to Y$. In fact, it will make the causal discovery (which assumes the whole dataset has come from the same distribution) worse.\n- The semantic plausibility does not make sense to me. By definition, a counterfactual sample is from a different distribution and hence the embedding *should* be different from the observational sample. It seems non-trivial to me to quantify how different it should be. For $X \\to Y$, intervening on $X$ might not create too different of a sample, but for $X \\to Z_1 \\to Z_2 \\to \\cdots \\to Z_D \\to Y$ intervening on $X$ may create a drastically different sample. Furthermore, is the inequality in equation 4 meant to be less than $\\tau$?\n- For the causal consistency why isn't equation 6 an equality with $0$? A change should not affect the non-descendents at all. How does this algorithm handle inconsistencies with the learnt graph and the \"counterfactual sample\" generated by the LLM? \n- What if the MLLM does not have information about the query asked as it does not have enough information about a concept? Won't it still generate data however useless it may be?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}

comments: SUMMARY:
The paper claims to solve the problem of latent variable causal discovery using a multi-modal foundation modeling framework. It uses CLIP embeddings from multi-modal inputs using contrasting learning, and then performs FCI on these embeddings. It further refines this graph by generating counterfactual samples and reintegrating them into the training loop.

STRENGTHS AND WEAKNESSES:
Strengths: 
- The overall idea to use CLIP and causal graph on top of it innovative. 
- The empirical results look strong, although the causal graphs being used seems simpler than how general the authors claim that their algorithm is, in the paper. 

Weaknesses: 
- Lack of clarity: 
1. \Psi is used before it is defined. 
2. Counterfactual generation and how it is done is not entirely clear and could be better described in the text. 
3. It is not clear how FCI was optimized within an LLM framework, since it's a constraint-based algorithm. 
4. There are some concerns regarding the theory - please see questions below. 
5. The empirical results do not include ground truth graphs with latent variables (when in the space of factors), yet the authors claim that their method can successfully work in this setting.

QUESTIONS:
1. Why are the optimal graphs in proposition 3.2 different? 
2. In the proof of 3.2, how do authors justify making Assumption D.2? What does it mean to use a direct approach to obtain the features -- could the authors be more specific about these techniques? 
3. How do the authors solve FCI within a continuous optimization framework? Furthermore, they claim that their algorithm can solve the latent variable problem but the ground truth graph does not include any latent variables. Could the authors elaborate more on that?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-05