Revealing Multimodal Causality with Large Language Models
Authors: Jin Li
Shoujin Wang
Qi Zhang
Feng Liu
Tongliang Liu
Longbing Cao
Shui Yu
Fang Chen
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Large Language Models
Causal Discovery
Primary area: deep_learning
TL;DR: A new framework using multimodal large language models to enable causal discovery from multimodal unstructured data.

Abstract: Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data. The implementation code and data are available at https://github.com/JinLi-i/MLLM-CD.

Revealing Multimodal Causality with Large Language Models
Jin Li', Shoujin Wang': Qi Zhang?, Feng Liu Tongliang Liu', Longbing Cao², Shui Yu', Fang Chen
University of Technology Sydney TTnggii University
3 University of Melhourne UUniversity of Sydney 'Macquarie University
jin.li 1imli-40studde
(shoujin.wang, shui.yu, faiiiyyu,,nnneemmm (aan.ccennnaaii chtnceutt.edd..a wee.uu@uunmelluuaaa.
tongliang wtwpliinnn..liyyy. Li@esydeyyy.ycuuuu
Abstract
Uncovering cause-and-effect mechanisms from data fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains critical challenge. Even with the advent multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed MLLM M-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data. The implementation code and data are available at https; //github. com/ JinLi Miuii-iiMMan
Introduction
Causal discovery (CD), which aims to infer the underlying causal structures from the data, is a fundamental task across numerous real-world scenarios, including healthcare [1. finance (3. and machine perception 4 playing a vital role in advancing scientific inquiry and human cognition. As data generation and collection technologies [7-9 continue evolve, real-world scenarios increasingly characterized by multimodal and unstructured data, such as the combination of texts, images, and/or audio, making CD particularly crucial yet challenging in practice. Effectively tackling CD in such cases typically involves two key steps: (1) identifying potential causal variables (factor discovery) and their values from the raw unstructured data, and (2) inferring the causal relationships among these variables (structure discovery). Traditional CD methods [10, though successful on structure discovery [13], heavily rely on predefined causal variables structured data and lack the
"Corresponing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multimodal Unstructured Data
Contrastive Discovery
MLLM
Structured
Embeekinggs
Intramadaal Contracive
Probe
Antebation
Intra modal Ceuncastive
ytu male
Factor
Palient Info
Identiircation
Causal Structure Discuvery
Itenative Reiineeet
Aligumeni
Factors
Bh worea smarer
Medical Image
Deggsss
(Degrens
Seeking
Leiia
Capaa Dixarrayy Algosiis
- -
Multimodal Counterfactual Reasoning
snooker with
GaI
What if the factur Leston changed?
Gender
Sanking
Consirenccy
Oiagnock
Insoge
Counterfactual Samples
Counteefactual
World
Age
Linion
Generation
Knowledge
Gruod Graph
with Ambiguttics
Figure 1: Illustration of MLLM-CD in lung cancer diagnosis. It first employs contrastive factor discovery with MLLMs to identify potential causal variables and form structured data. Then, a CD algorithm is performed infer causal structures. To further reduce ambiguities, it leverages MLLM's world knowledge to generate multimodal counterfactual samples for iterative refinement.
inherent capability perform factor discovery. As such, they are ill-equipped handle unstructured data that lacks predefined variables, let alone the more complex multimodal unstructured data.
Identifying genuine causal variables from unstructured data is challenging and costly, it often requires extensive manual feature engineering and domain expertise [1). This makes it difficult to scale with the increasing volume and complexity of modern data. Recently, the advent of large language models (LLMs) [15]19] offers a promising opportunity to bridge this gap. With their remarkable capabilities in understanding context and processing natural language, LLMs can be leveraged to automate factor discovery from raw textual data [20 21], paving the way for the following structure discovery. Integrating L.LMs into CD pipelines holds significant potential to broaden the scope and applicability of conventional CD, yet this direction remains largely under-explored.
To the best of our knowledge, COAT 220 is the only existing work that explicitly targets CD from unstructured data using LLMs, marking a significant step forward in this direction. first extracts causal variables via LLMs and then infers the causal relationships with traditional CD methods. However, is limited to unimodal data, typically text. contrast, the rapid proliferation of multimedia content necessitates CD methods capable of effectively handling multimodal information 23 across various domains, such as medical diagnosis (24) where clinical notes, medical images, feasible adapt unimodal approaches like COAT to multimodal scenarios using multimodal LLMS (MLLMs) (88), findings suggest that such naive adaptation falls short. As shown in Figure 2
22
and lab results collectively inform causal understanding 27 While it may
the adapted COAT struggles with two critical limitations on multimodal unstructured data: (1) uncovers only a small of causal factors, and (2) the inferred causal edges remain undirected.
Such limitations stem from two key challenges (CHs) in- herent to multimodal CD which can not be addressed by simple adaptations. CH1. Probing Intra- and Inter- modal Interactions. Causal variables are often embed- ded in different modalities, and become identifiable only when interactions between them probed (e.g., the ob- served interaction "smaller apples (image) have lower tar- get scores (text)" helps reveal the variable of size). With- out guidance from such cross-modal interactions, existing methods tend identify only the most salient factors (e.g.. taste, aroma, and defects), while overlooking more implicit yet equally important ones (e.g.. nutrition). CH2. Handling Structural Ambiguities. Inferring accu-
ammi
FARTICAT
jiiiice
(b)
Figure 2: Results on MAG dataset'. (a) Ground truth. (b) Causal graph discov- ered by COAT. Visual and verbal causal variables are presented in blue and black.
Results are based on Gemini 2.0. Please see Section 4.1 and Appendix D.3 for detailed experimental settings.

---

rate causal structures from observational data alone is inherently challenging 29). as multiple causal structures can yield the same statistical dependencies [3], leading to undesired ambiguity. This issue is further amplified in multimodal settings since they typically involve more causal variables while the number of possible causal structures grows exponentially with the number of variables.
To overcome these pressing challenges, we introduce MLLM-CD, novel framework that integrates the strengths of MLLMs with conventional CD methods. As illustrated in Figure 1 MLLM-CD is devised with three key modules. First, a contrastive factor discovery module is introduced for multi- modal factor identification. It encourages the MLLM to explore intra- and modrl interactions through variations in semantically contrastive sample pairs, helping reveal implicit variables hidden in multimodal inputs. Second, causal structure discovery is employed to infer the causal structure among the identified factors. Third, a novel iterative multimodal counterfactual reasoning module is proposed to refine the inferred causal structures by reducing structural ambiguities. Specifically, by performing counterfactual reasoning on factors with uncertain relationships, it generates hypothetical yet causally consistent multimodal samples for iterative refinement. In this way, MILLMs offer counterfactual evidence beyond the observational data to mitigate structural ambiguities through their world knowledge and reasoning capabilities. Our main contributions are summarized as follows:
We propose MILM-CD, to the best of our knowledge, the first work for causal discovery in multimodal unstructured data with a novel framework that integrates MLLMs into CD pipelines. This significantly extends the scope of CD beyond structured and/or unimodal data settings for practical use.
We introduce an innovative contrastive factor discovery module that encourages MLLMs to capture the intra- and inter-modal interactions for accurate factor identification under the multimodal unstructured context.
We design novel iterative multimodal counterfactual reasoning module to iteratively refine the inferred causal structures by generating multimodal counterfactuals.
We establish the first benchmark work for multimodal unstructured causal discovery with synthetic and real-world datasets. Extensive experiments demonstrate the superior perfor- mance of MLLM-CD in both factor discovery and causal structure discovery.
2
Related Work
Causal Discovery [. 30 aims to uncover the graphical structures, e.g., directed acyclic graph (DAG), with a causal interpretation from observational data. Traditional causal discovery methods largely fall into two categories: constraint-based
311-44] and score-based approaches 37
Constraint-based methods, such as PC [31 and FCI [3] 32] J, learn the causal graph structure by performing a series of conditional independence tests on the data. They rely on assumptions like the causal Markov condition and faithfulness [13] to link conditional independence relations to
the graph structure. score-based methods, like GES [8 and NOTEARS (36). search for a graph structure that maximizes a defined scoring function (e-g.. Bayesian Information Criterion (39). While powerful under ideal conditions, these methods face challenges in real-world applications, where data is typically unstructured and multimodal. Our work builds upon these foundational statistical CD methods and addresses their inherent limitations in multimodal and unstructured contexts by integrating the multimodal understanding and reasoning capabilities of MLLMs.
LLM for Causal Discovery (40]47) has recently shown great potential in supporting conventional CD pipelines. One of the main approaches is to leverage the general and domain-specific knowledge of LLMSs refine the discovery results 58 58 from statistical CD methods. However, as post- refinement of conventional CD methods, these approaches also suffer from limitations in unstructured and multimodal data. Motivated by the advanced capabilities of LLMs in understanding natural language descriptions, another line of work focuses on using LLMs to identify causal factors (20 and structures 22 61 from unstructured data. For instance, Liu et al. 20 introduce a LLM-driven CD framework, which uses LLMS to identify the Markov blanket of the target variable from natural language descriptions. To test the effectiveness of LLMS in causal relationship discovery, Kiciman et al. (29) propose pairwise prompting method and find that LLMs can be a good candidate for identifying relationships between a pair of variables. Vashishtha et al. (60) further extend this idea to a more accurate and robust triplet-based prompting to resolve the issue of multiple cycles. Nevertheless,
these methods are not devised to handle the multimodal nature of data, thus limiting their effectiveness
and applicability. Moreover, integrating the contextual understanding and reasoning capabilities
3

---

of LLMs into statistical CD pipelines -withou compromising their statistical rigor remains a non-trividl challenge. These limitations motivate the development of our proposed approach.
Causal Representation Learning (CRL) [62 64 aims to extract high-level representations and
causal dependencies from low-level observational data, such as images videos. Much of the work in this area is related to disentangled representation learning [65H68], which tries to map distinct factors of variation in the data to separate latent variables. In addition observational data, using data generated under interventions or multiple environments has also been explored to learn more robust and causally meaningful representations 669 Z7. While CRL has shown promise in providing provable identifiability of latent causal variables under certain assumptions, it remains a developing field with open challenges [69 [72 73 in practical use. this paper, we aim identify human- interpretable causal variables and corresponding causal structures from multimodal unstructured data by leveraging the contextual understanding and reasoning capabilities of ML.LMs.
3 Multimodal Causal Discovery
3.1 Problem Definition
We consider a dataset D = X2, X2,...
where each sample X E X comprises observations from multiple modalities. Let X₁ (Xx1, X*2.-.. , Xgm). where Xkj represents the j-th modality (e.g., text, image, video) of the k-th sample. We assume there exists a set of true, unobserved potential factors (i.e., causal variables) V (VI, Vi2 ....... V& that, along with target variable Y, govern the data generation process. The true causal relationships among V U {Y} are represented by an unknown true causal directed acyclic graph (DAG) G" (V*U(Y); E*). To discover the true causal graph, two main tasks need to be accomplished: (1) identifying potential factors from multimodal unstructured data, and (2) inferring the causal structure among these factors as a causal graph.
.x,),
Factor Identification. This task involves identifying the relevant set of causal variables V from multimodal unstructured data D. Crucially, it also requires annotating the value Uky for each variable V3 E V corresponding to each sample X5 E D. This process essentially transforms the unstructured multimodal data D {[X11, ... Xlem j== into structured dataset D3 = {(Vk1. , Uka) F==1" Moving beyond solely unimodal factor discovery [20), this step must consider the multimodal nature of the data, and handle: (1) Intra-modal complexity to extract high-level semantic concepts within each modality (e.g.. identifying the lesion in a medical image); (2) Inter-modal dependencies to recognize how concepts span or relate across different modalities (e.g., linking the lesion in medical images to the examination findings in clinical notes).
Causal Structure Discovery. Given the identified variables V and their annotated values in Ds, the goal is to infer causal relationships among V and the target variable Y, forming a causal graph G (V U {Y), E) that best explains the statistical dependencies observed in Ds. However, even with the identified variables, this task remains challenging due to issues like Markov equivalence classes 113, where multiple DAGs encode the same conditional independencies.
Our proposed MLLM-CD framework aims to address the challenges in these tasks by leveraging the advanced understanding and reasoning capabilities of MLLMs. It consists of three key components: (1) a contrastive factor discovery (CFD) module, which probes intra- and inter-modal interactions using contrastive signals, facilitating accurate factor identification; (2) a causal structure discovery module to infer causal structures based on statistical dependencies; and (3) an iterative multimodal counterfactual reasoning (MCR) module to iteratively refine the CD and reduce ambiguities by incorporating both observational data and counterfactual samples generated from the world knowledge of MLLMs. The overall framework is illustrated in Figure |
3.2 Contrastive Factor Discovery
We propose a CFD module to identify potential causal factors (t) and annotate corresponding values in the f-th round. This module adopts MLLM, denoted as V. explore the intra- and mnter-modal interactions with contrastive signals, from which it can identify the underlying factors.
Semantic Representation:
We begin by obtaining semantically aligned multimodal representations for each sample. As a common practice 774, we utilize pre-trained multimodal foundation models, such as CLIP 75), known for their ability to extract consistent embeddings from heterogeneous modalities. By adopting appropriate foundation models 776)78) as fa(.), we can effectively extract semantic representations for the k-th sample in i-th modality ex fx())).

---

Intra-modal Contrastive Exploration: To uncover factors embedded within i-th modality, we select the top-K pairs of samples P: {Xx.,, [o=))ol==1 with the maximum distance d(a. b)
1 - -iim( sim, @b;), where (eai 0bx)
ebi/ [ ||| []) measures cosine similarity, and the Fuclidean norm. These samples are then presented to V with a prompt Pintra (see Appendix D.2
to analyze the underlying intra-modal interactions and identify the contributing factors.
Inter-modal Contrastive Exploration: To highlight the inter-modal interactions, we construct cross- modal contrastive pairs of samples with the maximum misalignment. We quantify this mismatch using& a combined score s(a. b) = (1 sim((Cas: Obj)) + (ly, yyl) that considers both semantic distance in subspace and the difference in their scores of the target variable Y. We present the top- K pairs Px with the highest mismatch scores the model, along with prompt Pinter (see Appendix D.2) that highlights the potential contradiction across modalities. This encourages reasoning about the expected correspondence across modalities and helps pinpoint specific attributes or concepts where this correspondence breaks down, thereby revealing factors governing inter-modal relationships.
The contrastive analyses generate multiple sets of candidate factors, which may contain redundancies or overlap with previously identified factors ((t-1), For deduplication, we prompt V with Pm to merge similar factors and produce final factor set V(O, Then, we use the model with prompt Pa to annotate corresponding values, forming a structured dataset D for causal structure discovery. The overall process of the CFD module is summarized as:
Factor Consolidation and Annotation:
V = ((((( (intt) (inx.einte)) (((-1), Pm), D == VD((),, p()).a). (I)
3.3
Causal Structure Discovery
With the identified factors V and their annotated values in D we adopt a statistical CD method (c.g., FCI PC (3I]) to infer the causal structure into a causal graph:
g()) = C(D)) v (Y)).
(2)
Given the potential presence of unobserved confounders in real-world scenarios, algorithms robust to latent variables, such as FCI algorithm [3T] or its variants (e.g., RFCI (79). are particularly relevant. In this work, we instantiate our framework with the standard FCI algorithm. However, it can be easily adapted with other algorithms [31 for different causal assumptions and data types (80].
While statistical methods provide principled foundation for structure learning, they rely solely on observational data and thus struggle to address ambiguities (29]. This motivates the need for integrating additional knowledge and reasoning capabilities to refine the discovered structure.
3.4 Multimodal Counterfactual Reasoning
The causal graph (()) inferred by statistical methods from the derived datasets D. often contains ambiguity (e-g., represented by undirected edges in partial ancestral graph) due to potential limitations of finite and noisy observational data. To alleviate this issue, we introduce the MCR module, which leverages the implicit world knowledge and reasoning capabilities of an MLLM V to refine the causal graph iteratively.
The core idea is to use the MLLM to explore "what if" scenarios regarding the uncertainty in (()), thereby generating counterfactuals to support genuine causal relationships. By simulating interventions on specific factors and predicting how other factors are affected and how the sample changes, we generate counterfactual samples that implicitly encode causal assumptions derived from MLLM's knowledge. Once validated for plausibility and consistency, they are combined with observational data to provide complementary evidence supporting statistical discovery.
Counterfactual Generation: We first identify the uncertain relationships in the inferred graph g(t), This typically includes factors connected with ambiguous endpoints, indicating uncertainty about the direction of causation or the possibility of latent confounders. These factors then become candidates for counterfactual intervention. For an uncertain factor Va. we prompt & with PMCR to
perform counterfactual reasoning based on selected samples p(t) = {P((*) pm (P(()) The model predicts the counterfactual values vag [1/1 Vhall for X, p(t) under the hypothetical scenario
5

---

V₂ the Based on the reasoning results, it generates the corresponding multimodal samples X For instance, revises the original text X5: minimally to reflect the changes in factor values, producing a counterfactual text X&e- For visual counterfactuals, it generates a brief description of the image modifications and leverages an image generation model + (e.g., Gemini 2.0) to produce X If no changes are needed, then X = Xki- Formally, this process is summarized as:
(X V/ 9/) =- (XX,, Vk, Va U/ku PMCR; ))
Since MLLMs may suffer from hallucination issues 81 we also need to validate the generated counterfactual samples for plausibility and consistency. We mainly apply the following two checks.
Semantic Plausibility: We ensure the generated counterfactual sample S (X2. v4 is semantically coherent and not drastically different from the original sample. This is approximated by measuring the similarity between the original and counterfactual embeddings:
sem (S))
II! m
1
Σ
m
(sim(eks, e(r)) 2 Tsem]
where [[]] is the indicator function, and Tsem is threshold.
Causal Consistency: This check verifies if the changes in factor values from VA to v'g are consistent with the causal structure implied by g(4), A key principle is that an intervention on Va should ideally not cause changes in its causal non non-descendants, assuming no confounding paths are activated in unexpected ways by the MLLM's reasoning. To assess this, we first identify the set of non- descendants of the intervened variable Va in g(t,, denoted as NonDesc(Va, g(9). Then, calculate the proportion of non-descendant nodes that exhibit a significant change. Let Aeks Wks fkyl for VJ NonDesc(VV, Va, G())), we have
Randep (Sk, Vo: Vi)) EVeeNoDDese gw)) "Arks 2
(5)
[NonDesc Vo. g(9)]
The counterfactual Sy is considered causally consistent if the proportion of changed non-descendants below a threshold Teausal, i.e.,
Irausal (Sk,, Va, g()) = [[BiddoppSs Va,g S Teausat]-
(6)
Note that, since the current graph (()) used for the check is not ground truth and may contain errors, non-zero threshold Teausal is necessary. This threshold serves trade-off between injecting MLLM's knowledge and maintaining consistency with a potentially improvable graph. The validated
samples D(G with Tsem A Teansal = 1 are included for the next round as D(++1) = D(0) U D()) In this way, the observational dataset is effectively augmented using the MLLM's knowledge and reasoning capabilities, facilitating the iterative refinement of causal factor and structure discovery. The overall process of MLLM-CD is summarized Algorithm (Appendix
Despite the promise of modern MLLMs in demonstrating high-level general knowledge and reasoning capabilities [15], is important acknowledge that they may still fall short in challenging scenarios requiring cross domain or domain-specific expertise. In such cases, we recommend applying stricter validation to filter out noisy counterfactuals and integrating with strategies like -uggmette generation (RAG) 82 83 or knowledge graph (KG) grounding 84 85 knowledge base and reasoning capabilities.
to enhance the model's s
Experiments
We evaluate both the causal factor and structure discovery performance of MLLM-CD on both synthetic and real-world multimodal datasets, based on state-of-the-art multimodal LLMs including GPT-4o [86), Gemini 2.0 877. I.LaMA 4 Maverick [88), and Grok-2v [89] Due to the space limit,
full comparison results and detailed analyses, including more ablation studies, parameter analysis,
time complexity and scalability analysis, and case studies, are provided in Appendix D
4.1 Experimental Setup
We construct two multimodal datasets for evaluation: (1) Multimodal Apple Gastronome (MAG) dataset, which is synthetic dataset with 200 samples with 9 high-level factors, and (2) Lung Cancer

---

Table I: Causal factor identification and structure discovery performance on the MAG dataset
Mathod
NP
NR
NF
AP1
f
AF
FSHD
META
0.15
0.52 0.06
0..07
0.72 + 0.05
0.37 = 0.06
0.49 0.01
21.33 4:
Pairwise
0.56 0.10
0.33
0.41
0.11
43.33 6.43
GPT-4o
Triplet
0.33 + 0.06
0.37 0.06
0.35 0.06
61.67 5.77
COAT
0.78
0.37 +
0.19
0.32
0.19
0:22
18.67
MLLM-CD
0.83 0.111
0.85 1: 0.08
0.84 1: 0.09
0.69 4: 0.05
0.41 =: 0.06
0.51 0.04
15.33 :: 2.31
META
0.71 0.07
0.63 0.06
0.67 0.07
0.69 0.08
0.41 0.13
0.51 0.11
18-67 + 2.31
Pairwise
0.49 4: 0.09
0.56 0.11
0.51 0.06
30.00 4 2.00
Gemini 2.0
Triplet
0.41 + 0.02
0.69 + 0.33
0.48 0.05
32.00 2.00
COAT
0.133
0.51
0.10
0.26
0.37 0.05
MLLM-CD
0.86 0.06
0.89 + 0.00
0.87 + 0.08
0.76 + 0.08
0.52 0.13
0.60 = 0.06
14.00 + + 3.46
META
0.06
0.45 0.05
0.26
0.07
21.67
Pairwisc
0.50 0.17
0.30 + 0.06
0.36 0.04
31.67 + 6.66
4
Triplet
0.66 4 0.32
0.30 0.06
0.38 0.04
34.33 1 7.77
COAT
1.00
0.41 + 0.06
0.58 0.07
0.89 +: 0.19
0.30 = 0.06
0:41 0.10
11.67 + 1.15
MLLM-CD
1.00 0.00
0.85 1: 0.06
0.92 0.04
0.62 à: 0.08
0.59 1: 0.06
0.60 0.04
13.33
0.58
META
0.11
0.44 0.00
0.49 0.04
0.76 0.00
0.33 + 0.00
0.46 0.00
20
3.06
Pairwise
0.35 + 0.02
0.33 + 0.00
0.34 0.01
28.33
11.37
Gmk-2v
Triplet
0.02
0.01
COAT
1.00 +.0000000
0.37 + 0.17
0.53 0.18
0.17 + 0.29
0.07 0.13
0.10 0.18
16:33 1.15
MLLM-CD
1.00
0.85 + 0.06
0.92 0.04
0,79 = 0.21
0.44 = 0.00
0.56 0.06
11,00
2.65
META
0.56 0.122
0.50 1: 0.10
0.52 1: 0.10
0.74 0.10
0.34 1: 0.09
0.46 0.07
21. 3.11
Pairwise
0.47 1 0.12
0.38 0.13
0.40 0.09
31.08 1: 8.76
Average
Triplet
0.20
0.40
0.39 0.07
15.00
COAT
0.91 0.91 0.14
0.39 4: 0.10
0.53 1: 0.11
0.53 =: 0.36
0,20 +: 0.13
0.29 0.19
16.
2.02
MLLM-CD
0.92 +: 0.10
0.86 +: 0.05
0.89 0.06
0.72 0.12
0.49 0.10
0.57
0.06
13.42 2.68
dataset, which is real-world dataset with 60 samples with 5 high-level factors. Please refer to Appendix D.3 for more details on the experimental settings and environment information.
Following the practice in work (20] we generate a multimodal version of the Apple Gastronome dataset with 200 samples, where visual factors (i.c., color, size, and defects) and 5 verbal factors (i.e., aroma, taste, jui juiciness, nutrition on, and recmd). Each sample represents an apple with a specific combination of these attributes. The target variable the overall score of the apple, which is influenced by the other factors as shown in the ground truth causal graph in Figure B (a). The visual factors (shown as blue nodes) are represented by images of apples, which are generated by Stable Diffusion 3.5 [90], while the verbal factors (shown as black nodes) are represented by a review text generated by Gemini 2.0 [87.
Multimodal Apple Gastronome:
We construct real-world multimodal dataset for lung cancer diagnosis. The samples are collected from the MedPix database {] We select 60 representative lung cancer cases (e.g., Non-Small Cell Lung Cancer (9I)) with 5 high-level factors. (1) verbal factors: gender and age from the demographic data; smoking from the history data; and (2) visual factors: lesion from the medical images (e.g., CT scans). The target variable is the diagnosis of lung cancer, which is affected by other factors in the way of the causal graph shown in Figure (a). More details on the dataset construction are provided in Appendix D.1
Lung Cancer:
Baselines: META 20 performs zero-shot factor and structure proposal given only contextual information to LLM; Pairwise (29) and Triplet [60] use LLM to infer causal relationships among pairwise or triplet variables: COAT [20] is the state-of-the-art LLM-driven unstructured CD method and MLLMs are used to adapt it to multimodal settings. The original implementation of COAT only focuses on the discovery of Markov Blanket. We extend it to discovering the entire causal graph for fair comparison. Note that while other LLM-driven CD methods exist, e.g., [4] 48/55) 57 92-)7), they only focus on structural CD and cannot be directly adapted unstructured multimodal data.
Metrics: Following the common practice in CD 30 we evaluate the performance of both factor identification and causal structure discovery with commonly used metries. For factor identification, we use: (1) Node Precision (NP), (2) Node Recall (NR), and (3) Node F1 -s-oe (NF). For causal structure discovery, we adopt (1) Adjacency Precision (AP), (2) Adjacency Recall (AR), and (3) Adjacency Fi-score (AF). To jointly evaluate both perspectives, we use an extended Structural
Pairwise and Triplet do not involve the step of factor identification, thus metrics related factor identification are not applicable. Instead, we use the factors discovered by META for their structure discovery.
https: //twpixx.dxx//////e/// nlm. nih, gov/ hoom
7

---

Hamming Distance [98] (ESHD), which incorporates additional penalization for mismatched nodes and edges. More details are offered in Appendix D.3
4.2 Analysis with Multimodal Apple Gastronome Dataset
The empirical results of causal discovery on the MAG dataset are shown in Table [ and Figure 3 The full comparison results are available in Ap- pendix MLLM-CD overall outperforms base- lines in most cases.
Factor Identification: MILMM-DD achieves the most accurate and comprehensive factor identifi- cation in terms of quantitative metrics (NP, NR, NF) and qualitative results (Figure (D). This is attributed the contrastive factor discovery, which effectively guides the MLLM to identify factors by exploring intra-modal and inter-modal interac- tions. Meanwhile, we observe that MLLM-CD can achieve satisfactory factor identification performance in the first iteration, indicating the efficiency of the CFD module. In contrast, COAT' fails to identify sufficient factors and has a lower recall rate.
Da Groand mu
(4) Togle:
CRAA
Figure 3: Discovered graphs on MAG dataset.
Structure Discovery: MLLM-CD achieves the highest adjacency evaluation scores (AP, AR, AF) in most cases and has the lowest ESHD score, indicating that the discovered causal graph is closer to the ground truth. The improvement is mainly due the multimodal counterfactual reasoning module, which refines the causal structure by leveraging the MLLM's knowledge and reasoning capabilities. COAT, on the other hand, can hardly determine certain causal relationships. Although META has lower factor identification performance, it successfully identifies several correct and certain causal relationships based on its general knowledge, showing the potential of causal reasoning in MLLMs.
4.3 Analysis with Lung Cancer Dataset
Apart from the synthetic dataset, we also evaluate the performance of MLLM-CD on the real-world Lung Cancer dataset. The results are shown in Table and Figure 4 Consistent with the results on the MAG dataset, MLLM-CD achieves the best quantitative performance in most cases, especially in terms of ESHD, showing the overall improve- ment of MLLM-CD over the baselines. The qual- itative results show that MLLM-CD accurately identifies the causal relationships between all the factors and the target variable. COAT achieves comparable performance but includes incorrect di- rections and noisy/missing factors. Since META, Pairwise, and Triplet solely rely on MLLM's knowledge, the generated causal graphs are certain but often overly complex with multiple redundant edges. The results indicate that MLL.MCD is capable of effectively identifying the causal factors and structures in real-world multimodal datasets, demonstrating its practical applicability.
COXT
IO ML
Figure 4: Discovered causal graphs Lung Cancer dataset.
4.4 Ablation Study
We conduct an ablation study to evaluate the effectiveness of each module in MLLM-CD. We compare the full MILM-CD with the following variants: (1) MLLM-CD w/o Contrastive Factor Discovery (w/o CFD): This variant does not perform contrastive factor discovery, and instead, only uses a simple method, similar to (20) for factor identification.
(2) MLLM-CD w/o Counterfactual Reasoning (w/o CR): This variant only uses the contrastive factor discovery module and the causal structure discovery module. (3) MLLM-CD w/o Both (w/o Both): This variant only uses simple factor identification prompt from 200] and then performs causal structure discovery without any multimodal reasoning. The performance of these variants with Gemini 2.0 on two datasets is shown in Table 3

---

NR Table 2: Causal factor identification and structure discovery performance on the Lung Cancer dataset."
Method
NP
NR
NF1
AP
AP1
FSHD
META
0.33 0.05
0.00 0.00
0.12 = 0.01
1,00 : 0.00
0.30
0.00
0.67 0.00
21.67 4.62
Pairwise
0.67 + 0.00
0.50
0.00
0.57 0:00
36.33 10.21
GPT-40
Triplet
0.07 1: 0.00
0
0.37 0:00
47.33 14.98
COAT
0.11
0.40 0.00
0
0.58
0.23
MLLM-CD
0.89 0.10
1.00 1: 0.00
0.94 0.05
0.92 1: 0.14
0.58
0.14
0.69
1 0.05
5.00 1:00::000 0.00
META
0.07
0.73 0.12
0.54 0.08
0.92 + 0.14
0.67 0.14
0.76+00.10
16.00 +:
Pairwise
0.56 1: 0.10
0.67 0.14
0.59
0.02
33:33 4 3.51
Gemini 2.0
Triplet
0.56 + 0.10
0.67 = 0.14
0.39
0.02
40.00 + 8.19
COAT
0.47 4:
0.56
1.00
0.33
0.15
2.08
MULM-CD
0.94 0.100
1.00 + 0.00
0.97 0.05
0.92 + 0.14
0.83 = 0.14
0.87 0.13
4.67 0.58
META
0.11
0.67 1:
0.50 0.04
0.42
Pairwisc
0.76 + 0.21
0.58 0.11
0.63
0.05
32.33
17:21
4 4
Triplet
0.61 1: 0,10
0.58 = 0.14
0.58
4 0.02
36.67
20.40
COAT
0.11
0.47 + 0.12
0.58 0.08
1.00 + 0.00
0.25 = 0.00
0:10 0.00
7.67
0.58
MLLM-CD
0.82 1: 0.02
0.93 1: 0.12
0.87 0.06
0.92 1: 0.14
0.67
0.14
0.10 0.76
0.58 5.67
META
0:29:0004
0.40 0.00
0.33 0.03
1.00 + 0.00
0.25
0.00
0.40
0.00
20.00 5.29
Pairwise
1.00 + 0.00
0.25
0.00
0.40 0.00
28. 28.00 + 8.80
Grok-2v
Triplet
1.00
0.25
8.00
COAT
0.56 0:21
0.47 + 0.23
0.51 + 0.22
0.67 + 0.58
0.17 = 0.14
0.27 0.23
9.67 1.53
MLLM-CD
0.112
0.80 + 0.00
0.83 = 0.05
1.00 + 0.00
0.25 0.00
0.40 0.00
6.00 1.00
META
0.36 0.08
0.60 +: 0.15
0.45
0.91 # 0.17
0.46 =
0.39 0.17
19:25
-05---4222 4:27
Pairwise
0.74 1 0,20
0,50
0.35 0.10
32.50 1 9.97
Average
Triplet
0.50
0.08
13.36
COAT
0.65 +: 0.23
0.45 1: 0.12
0.52 0.13
0.75 4: 0.45
0,21 = 0.14
0.32 1: 0.21
9.42
2.31
MLLMMCDD
0.88 0.09
0.93 +: 0.10
0.90 0.07
0.94 0.11
0.58 0.25
0.68 0.19
5.33
0.78
Please refer to Appendix for the full results. One can observe that excluding either the contrastive fac-
Table 3: Ablation Study of MLLM-CD on the MAG and Lung Cancer datasets.
tor discovery or the counterfactual reasoning module leads to significant drop in performance. In spe- cific, contrastive factor discovery shows more impact on the accuracy and completeness of factor identi- fication. The counterfactual reasoning module, on the other hand, shows more effectiveness in refining the causal structure, as we can see from the compar- ison of MLLM-CD w/o CR and the full MLLM-CD, Meanwhile, it contributes to larger improvements on the relatively smaller dataset (Lung Cancer) than the larger one (MAG). This indicates the importance of introducing plausible counterfactuals based on the MLLM's knowledge and reasoning capabilities to alleviate the data scarcity issue and possible data noise. By combining both modules, they jointly contribute to the improvement of causal factor and structure discovery, demonstrating the effectiveness of MLLM-CD.
Datoeet
Variant
NF 1
AF1
PSHD
w/o CPD
I. 0.07
0.07 4: 0.003
15,00 d: 0.000
wio CRR
0.81 + 0.00
0.52 + 0.06
15.07 + 5.51
wlo Both
0.54 1 0.08
0.01 1 0.08
16.53 1 2.08
MLLMECD
0.87
0.03
0.00 0.06
14.00 3.46
Wdo CFD
0.62 0.34
0.96 + 0.34
8.00
1.30
w/o CR
0.04 0.05
0.58 3: 0.04
5.35
1.53
Lung
Both
11.55
9.67
MLLMMCCD
0.18
0.58
In addition, further evaluate the effectiveness of different sampling strategics used in our contrastive factor discovery module. We compare the following variants: (1) Random Sampling (Random): This variant randomly samples data points from each cat- egory of the target variable.
Table 4: Ablation Study of the CFD module on the MAG and Lung Cancer datasets.
Dataset
Variant
NP1
NF
Randon
0.94
0.07
(2) Simple Pairwise Sampling (Simple Pair): This variant pairs samples with the largest difference in the target variable. (3) Intra-modal Contrastive Sampling (Intra-modal): This variant only considers intra-modal contrastive pairs within each modality. (4) Inter-modal Con-
Simple Pair
0.14
0.70
0.75 0.14
MAG
Intro-modal
0.88
0.13
0.76
0,82 0.12
(ncr-modal
0.57
2: 0.19
0.29 : 0.17
0.G3 2: 0.18
ML.1M-DD
0.96 + 0.05
0.89
0.00
0.87 + 0.03
Random
0.53 1: (1,0
0.00 0.30
0.02 4: 0.01
Simple Phir
0.50-+0.000
0.60 0.30
0.35 + 0.000
Lung
Intri modal
1.80.10..00
0.80 0.00
0.50 1 0.00
Inser-modal
0.53-1:0.12
0.12
MD
0.14
1.00 0.00
0.97
trastive Sampling (Inter-modal): This variant only
considers inter-modal contrastive pairs across differ-
ent modalities for sampling. The key quantitative results are shown in Table 4 and we have the
following observations: (1) Incorporating finer- aeer-gg signals by finding semantic- contrastive pairs within each modality, the intra-modal contrastive sampling further improves the completeness of factor discovery. However, it often overlooks the cross-modal interactions. (2) Inter-modal contrastive sampling acts as an ideal complement to intra-modal strategy, as it examines across modalities and can effectively identify factors missed by intra-modal. Using inter-modal only has lower quantitative results, because it is prompted emphasize factors tied to cross-mode interactions, complement

---

to intra mordal strategy, rather than aiming to discover all factors. (3) By combining both intra-modal and onter-modal strategies, MLLM-CD achieves the best overall performance in factor discovery. demonstrating the effectiveness of our contrastive factor discovery module.
Conclusion
This paper presents MLLM-CD, a novel framework for multimodal causal discovery that leverages the multimodal understanding and reasoning capabilities of MLLMs. MLLM-CD consists of three main components: (I) contrastive factor discovery module to identify accurate and comprehensive causal factors from multimodal unstructured data in a novel multimodal contrastive manner, (2) a causal structure discovery module to learn the causal relationships among the identified factors, and (3) a multimodal counterfactual reasoning module to refine the discovered results by generating and validating counterfactual samples. We conduct extensive experiments on both synthetic and real-world multimodal datasets, demonstrating the superior performance in causal factor and structure discovery and the practical applicability of MLLM-CD.
Limitations. While promising, our approach faces several limitations. A primary challenge is the scarcity of large-scale, diverse benchmark datasets for multimodal causal discovery. While the datasets used in this work are thoughtfully curated, they are constrained by small sample sizes, and their ground truth causal graphs depend on domain expert knowledge. Secondly, the range of modalities MLL.M-DD can effectively process is bound by the capabilities of the core MILLM, presenting challenges for integrating specialized modalities like sensor data or genomic sequences. Finally, the overall effectiveness of the proposed framework reasoning and generation capabilities of MLLMs, which may still exhibit hallucinations, reflect biases from their training data, or fail to capture subtle causal nuances, despite recent advancements in model alignment.
dependent on the sophisticated
Future Work. These limitations open several avenues for future research. crucial direction is the development of larger, more diverse, and rigorously annotated benchmark datasets, potentially using semi-automated annotation or simulation-barl based approaches, which will also allow for exploring the scalability of MLLM-CD. We will also focus on enhancing the framework to handle wider range of modalities by developing more sophisticated multimodal fusion and cross-modal reasoning mechanisms. To address the reliance on MILM capabilities, we plan to develop methods to rigorously validate and improve MLLM outputs within the causal discovery pipeline. This includes integrating with interventional analysis, designing uncertainty quantification for generations, and exploring techniques to mitigate biases and hallucinations specifically for causal reasoning.
Please refer to Appendix B for more discussions on limitations and future work.
Acknowledgments and Disclosure of Funding
We sincerely appreciate the reviewers' valuable feedback. This work is partially supported by Australia ARC LP220100453 and ARC DP240100955. TLL is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, 1P22010057,, LP220200949.
References
[1] Ruibo Tu, Kun Zhang, Bo C. Bertilson, Hedvig Kjellström, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. In Advances in Neural Information Processing Systems, pages 12773-12784 2019.
[2] Wei Xu, Gang Luo, Weiyu Meng, Xiaobing Zhai, Keli Zheng, Ji Wu, Yanrong Li, Abao Xing, Junrong Li, Zhifan Li, Ke Zheng, and Kefeng Li. Mragent: an llm-based automated agent for causal knowledge discovery disease via mendelian randomization. Briefings in Bioinformatics, 26(2), 2025.
[3] Chang Gong, Di Yao, Chuzhe Zhang, Wenbin Li, Jingping Bi, Lun Du, and Jin Wang. Causal discovery from temporal data. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5803-5804, ACM, 2023.
10

---

[4] David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, and Léon Bottou. Discovering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 58-66. IEFE, 2017.
[5] Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. In Advances in Neural Information Processing Systems, 2020.
[6] Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision-and- -annuuugg navigation via causal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 13139-13150. IEEE, 2024.
[7] Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In Advances in Neural Information Processing Systems, 2023.
[8] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g: Generating images in context with multimodal large language models. In The International Conference on Learning Representations. OpenReview.net, 2024.
[9] Ye Tian, Ling Yang, Haotian Yang. Yuan Gao, Yufan Deng, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di Zhang, and Bin Cui. Videotetris: Towards compositional text-to-video generation. In Advances in Neural Information P Processing Systems, 2024.
[10] Antti Hyttinen, Frederick Eberhardt, and Patrik O. Hoyer. Experiment selection for causal discovery. The Journal of Machine Learning Research, 14(1):304130711, 2013.
[11] Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi. and João Gama. Methods and tools for causal discovery and causal inference. WIREs Data Mining and Knowledge Discovery, 12(2), 2022.
[12] Yujia Zheng, Biwei Huang, Wei Chen, Joseph D. Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, and Kun Zhang. Causal-learn: Causal discovery in python. Journal of Machine Learning Research, 25:60:1-60ss 2024.
[13] Judea Pearl. Causality. Cambridge University Press, 2009.
[14] Matthew J. Vowels, Necati Cihan Camgöz, and Richard Bowden. D'ya like dags? A survey structure learning and causal discovery. ACM Computing Surveys, (1:)):82:182:3366 2023.
[15] Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. Openagi: When LLM meets domain experts. In Advances in Neural Information Processing Systems, 2023.
[16] Shengnan An, Zexiong Ma. Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your LLM fully utilize the context. In Advances in Neural Information Processing Systems, 2024.
[17] Hao Zhou, Chengming Hu, Ye Yuan, Yufei Cui, Yili Jin, Can Chen, Haolun Wu, Dun Yuan, Li Jiang, Di Wu, Xue Liu, Charlie Jianzhong Zhang, Xianbin Wang, and Jiangchuan Liu. Large language model (LLM) for telccommumications:: A comprehensive survey on principles, key techniques, and opportunities. CoRR, abs/2405. 10825, 2024.
[18] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu. Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. CoRR, abs/2411. 2024.
[19] Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J. Topol. Llm-based agentic systems in medieine and healthcare. Nature Machine Intelligence, 6(12):1418-1420) 2024.
[20] Chenxi Liu, Yongqiang Chen, Tongliang Liu, Mingming Gong, James Cheng, Bo Han, and Kun Zhang. Discovery of the hidden world with large language models. Advances Neural Information Processing Systems, 2024.

---

[21] Zhiheng Lyu, Zhijing Jin, Rada Mihalcea, Mrinmaya Sachan, and Bernhard Schölkopf. Can large language models distinguish cause from effect?? In The Conference on Uncertainty in Artificial Intelligence Workshop on Causal Representation Learning, 2022.
[22] Yun-Hao Yuan, Jin Li, Yun Li, Jipeng Qiang, Yi Zhu, Xiaobo Shen, and Jianping Gou. Learning canonical f-correlation projection for compact multíview representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19238-19247, 2022.
[23] Jin Li, Shoujin Wang, Qi Zhang, Shui Yu, and Fang Chen. Generating with fairness: A modality-diffusee counterfactual framework for incomplete multimodal recommendations. In Proceedings of the ACM on Web Conference, pages 2787-2798, 2025.
[24] Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. Improving the accuracy of medical diagnosis with causal machine learning. Nature Communications, 11(1):3923, 2020.
[25] James S. Dunean and Nicholas Ayache. Medical image analysis: Progress over two decades and the challenges ahead. IEEE Transuctions on Pultern Analysis and Machine Intelligence, 22(1):85-106, 2000.
[26] Chenxin Li, Xinyu Liu, Wuyang Li, Cheng Wang, Hengyu Liu, Yifan Liu, Zhen Chen, and Yixuan Yuan. U-KAN makes strong backbone for medical image segmentation and generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4652-4660 AAAI Press, 2025.
[27] Naveed Rabbani, Grace YE Kim, Carlos Suarez, and Jonathan Chen. Applications of machine learning in routine laboratory medicine: Current state and future directions. Clinical Biochemistry, 103:1-7, 2022.
[28] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal LLM. In Proceedings the International Conference on Machine Learning. OpenReview.net. 2024.
[29] Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. Transactions on Machine Learning Research, 2023.
[30] Alessio Zanga, Elif Ozkirimli, and Fabio Stella. A survey on causal discovery: Theory and practice. International Journal of Approximate Reasoning, 151:101-129, 2022.
[31] Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT Press, 2000.
[32] Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 172(16-17::1873-18 2008.
[33] Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. In Advances in Neural Information Processing Systems, 2020.
[34] Joris M. Mooij, Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. Journal of Machine Learning Research, 21:99:1-99:100 108, 2020.
[35] Karren D. Yang, Abigail Katoff, and Caroline Uhler. Characterizing and learning equivalence classes of causal dags under interventions. In Proceedings of the International Conference Machine Learning, volume 80, pages 5537-5546. PMLR, 2018.
[36] Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Dags with NO TEARS: continuous optimization for structure learning. In Advances in Neural Information Processing Systems, pages 9492-9503, 2018
[37] Chandler Squires, Yuhao Wang, and Caroline Uhler. Permutation- causal structure learning with unknown intervention targets. In Proceedings of the Conference on Uncertainty Artificial Intelligence, volume 124 of Proceedings of Machine Learning Research, pages 1039-1048. AUAI Press, 2020.
12

---

[38] Juan Ignacio Alonso-Barba, Luis delaOssa, José A. Gámez, and José M. Puerta. Scaling up the greedy equivalence search algorithm by constraining the search space of equivalence classes. International Journal of Approximate Reasoning, 54(4):429 451, 2013.
[39] Gideon Schwarz. Estimating the dimension of a model. The Annals of Statistics, pages 461-464, 1978.
[40] Guangya Wan, Yuqi Wu, Mengxuan Hu, Zhixuan Chu, and Sheng Li. Bridging causal discovery and large language models: comprehensive survey of integrative approaches and future directions. CoRR, abs/2402. 1068,, 2024.
[41] Taiyu Ban, Lyuzhou Chen, Derui Lyu, Xiangyu Wang, Qinrui Zhu, and Huanhuan Chen. Lim-driven causal discovery via harmonized prior. IEEE Transactions on Knowledge and Data Engineering, 37(4):1943- 2025.
[42] Tao Feng, Lizhen Qu, Niket Tandon, Zhuang Li, Xiaoxi Kang, and Gholamreza Haffari. From pre-training corpora to large language models: What factors influence LLM performance in causal discovery tasks? CoRR, abs/2407. 19638, 2024.
[43] Hao Duong Le, Xin Xia, and Zhang Chen. Multi-agent causal discovery using large language models. CoRR, abs/2407.15073, 2024.
[44] Yu Zhou, Xingyu Wu, Beicheng Huang, Jibin Wu, Liang Feng, and Kay Chen Tan. Causalbench: comprehensive benchmark for causal learning capability of Ilms. CoRR, abs/2404.06349, 2024.
[45] Xin Li, Zhuo Cai. Shoujin Wang. Kun Yu, and Fang Chen. A survey on enhancing causal reasoning ability of large language models. In Advances in Knowledge Discovery and Data Mining Pacific-Asia Conference Knowledge Discovery and Data Mining, volume 15873 of Lecture Notes in Computer Science, pages 399-416, 2025.
[46] Ivaxi Sheth, Sahar Abdelnabi, and Mario Fritz. Hypothesizing missing causal variables with Ilms. CoRR, 1ew.2449.02604 2024.
[47] Ivaxi Sheth, Bahare Fatemi, and Mario Fritz. Causalgraph2llm: Evaluating Ilms for causal queries. CoRR, abs/24 2024.
[48] Taiyu Ban, Lyuzhou Chen, Derui Lyu, Xiangyu Wang, and Huanhuan Chen, Causal structure learning supervised by large language model. CoRR, abs/2311 1669, 2023.
[49] Stephanie Long, Alexandre Piché, Valentina Zantedeschi, Tibor Schuster, and Alexandre Drouin. Causal discovery with language models as imperfect experts. CoRR, 15x/3377002990 2023.
[50] Ahmed Abdulaal, Adamos Hadjivasiliou, Nina Montaña Brown, Tiantian He, Ayodeji Ijishakin, Ivana Drobnjak, Daniel C. Castro, and Daniel C. Alexander. Causal modelling agents: Causal graph discovery through synergising metadata- and data-driven reasoning. In International Conference on Learning Representations. OpenReview.net, 2024.
[51] Yuzhe Zhang, Yipeng Zhang, Yidong Gan, Lina Yao, and Chen Wang. Causal graph discovery with retreeval-uugmented generation based large language models. CoRR, abs/2402.15301, 2024.
[52] Elahe Khatibi, Mahyar Abbasian, Zhonggi Yang, Iman Azimi, and Amir M. Rahmani. ALCM: autonomous lim-augmented causal discovery framework. CoRR, abs/2405.01745 01444,144104 2024.
[53] Masayuki Takayama, Tadahisa Okuda, Thong Pham, Tatsuyoshi Ikenoue, Shingo Fukuma, Shohei Shimizu, and Akiyoshi Sannai. Integrating large language models in causal discovery: A statistical causal approach. CoRR, abs/2402.01 0144, 2024.
[54] ChengAo Shen, Zhengzhang Chen, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, and Jingehao Ni. Exploring multi-modal integration with tool-augmented L.M agents for precise causal discovery. CoRR, abs/2412.13667 2024.
13

---

[55] Kai-Hendrik Cohrs, Gherardo Varando, Emiliano Diaz, Vasileios Sitokonstantinou, and Gus- tau Camps-Vall Valls. Large language models for constrained-basee causal discovery. CoRR. abs/2406.07378, 2024.
[56] Peiwen Li, Xin Wang, Zeyang Zhang, Yuan Meng, Fang Shen, Yue Li, Jialong Wang. Yang Li, and Wenwu Zhu. Realted: Temporal causal discovery from interventional data with large language model. In Proceedings of the ACM International Conference Information and Knowledge Management, pages 4669-4677. ACM, 2024.
[57] Xiangyu Wang, Taiyu Ban, Lyuzhou Chen, Derui Lyu, Qinrui Zhu, and Huanhuan Chen. Large-scale hierarchical causal discovery via weak prior knowledge. IEEE Transactions Knowledge and Data Engineering, 3765):265--771, 2025.
[58] Xiaoxuan Li, Yao Liu, Ruoyu Wang, and Lina Yao. Regularized multi-llms collaboration for enhanced score-based causal discovery. In Web Information Systems Engineering, volume 15439, pages 177-188. Springer, 2024.
[59] Moritz Willig, Matej Zecevic, Devendra Singh Dhami, and Kristian Kersting. Can foundation models talk causality? CoRR, abs/2206.10591 2022.
[60] Aniket Vashishtha, Abbavaram Gowtham Reddy, Abhinav Kumar, Saketh Bachu, Vineeth N. Balasubramanian, and Amit Sharma. Causal order: The key to leveraging imperfect experts in causal inference. In International Conference on Learning Representations. OpenReview.net, 2025.
[61] Junfei Liu, Shaotong Sun, and Fatemeh Nargesian. Causal dataset discovery with large language models. In Proceedings of the Workshop Human-Ing -aata----ttt---tt Data Analytics, pages 1-8, ACM, 2024.
[62] Francisco Silva, Hélder P. Oliveira, and Tània Pereira. Causal representation learning through higher-level information extraction. ACM Computing Surveys, 57(2):33:1-4533 -3:337, 2025.
[63] Zizhen Deng, Hu Tian, Xiaolong Zheng, and Daniel Dajun Zeng. Deep causal learning: Representation, discovery and inference. ACM Computing Surveys, 58(2), 2025.
[64] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 11165):612-634, 2021.
[65] Giambattista Parascandolo, Niki Kilbertus, Mateo
and Bernhard Schölkopf. Learning independent causal mechanisms. In Proceedings of the International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4033-4041, 2018
[66] Raphael Suter, Dorde Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disen- tangled causal mechanisms: Validating deep representations for interventional robustness. In Proceedings of the International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages (056-6065, 2019.
[67] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9593- 2021.
[68] Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric P. Xing, and Kun Zhang. Temporally disentangled representation learning under unknown nonstationarity. In Advances in Neural Information Processing Systems, 2023.
[69] Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based causal representation learning: Linear and general transformations. Journal of Machine Learning Research, 26(1 12):1-90, 2025.
[70] Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Identifying linearly-mixed causal representations from multi-node interventions. Causal Learning and Reasoning, volume 236 of Proceedings of Machine Learning Research, pages 843-867, 2024.
14

---

[71] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 372 2023.
[72] Dingling Yao, Caroline Muller, and Francesco Locatello. Marrying causal representation learning with dynamical systems for science. In Advances in Neural Information Processing Systems, 2024.
[73] Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. From causal to concept-based representation learning. In Advances in Neural Information Processing Systems, 2024.
[74] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: CLIP embeddings for embodied AI. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 14809-14818 IEEE, 2022.
[75] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference Machine Learning, volume 139, pages 8748-8763, 2021.
[76] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP learning audio concepts from natural language supervision. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1-5, 2023.
[77] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirk-Kikkat and Shlomo Dubnov. Large-scale contrastive aunuuag--audio pretraining with feature fusion and keyword- to-caption augmentation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1-5, 2023.
[78] Ziang Zhang. Zehan Wang. Luping Liu, Rongjie Huang, Xize Cheng, Zhenhui Ye, Wang Lin, Huadai Liu, Haifeng Huang, Yang Zhao, Tao Jin, Siqi Zheng, and Zhou Zhao. Extending multi- modal contrastive representations. In Advances Neural Information Processing Systems, 2024.
[79] Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas Richardson. Learning dinensiinni directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294 0241-21,,20 2012.
[80] Erdun Gao, Ignavier Ng, Mingming Gong, Li Shen, Wei Huang, Tongliang Liu, Kun Zhang, and Howard D. Bondell. Missdag: Causal discovery in the presence of missing data with continuous additive noise models. In Advances in Neural Information Processing Systems, 2022.
[81] Xuefeng Du, Chaowei Xiao, and Sharon Li. Haloscope: Harnessing unlabeled LLM gener- ations for hallucination detection. In Advances in Neural Information Processing Systems, 2024.
[82] Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on RAG meeting 1lms: Towards retrieval-un -ugmeette large language models. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491-6501, 2024.
[83] Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. A survey on RAG with llms. In Knowledge- Based and Intelligent Information & Engineering Systems: Proceedings of the 28th International Conference EES-2024, Seville, Spain, 11-13 September 2023, volume 246 of Procedia Computer Science, pages 3781-3790, 2024.
[84] Yu Wang, Nedim Lipka, Ryan A. Rossi, Alexa F. Siu, Ruiyi Zhang, and Tyler Derr. Knowl- edge graph prompting for multi-docummet question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19206-19214, 2024.
15

---

[85] Yutong Zhang, Lixing Chen, Shenghong Li, Nan Cao, Yang Shi, Jiaxin Ding, Zhe Qu, Pan Zhou, and Yang Bai. Way specialist: Closing loop between specialized LLM and evolving domain knowledge graph. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1996-2007, 2025.
[86] OpenAL. Hello, GPT-4o! hatps//oopnaai.com//ddd///elllle gpt-4o/, 2024. Accessed: 2025-05-15.
[87] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, and et al. Gemini: family of highly capable multimodal models. CoRR, abs/2312.111 1880,
[88] Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. -
2025. Accessed: 2025-05-15.
[89] xAI. Grok image generation release. Accessed: 2025-05-15.
2024.
[90] Stability AL Introducing stable diffusion 3.5. http:///sablilyaaaleeewswioooitd diffusion-3-5, 2024. Accessed: 2025-05-15.
[91] Cesare Gridelli, Antonio Rossi, David P Carbone, Juliana Guarize, Niki Karachaliou, Tony Mok, Francesco Petrella, Lorenzo Spaggiari, and Rafael Rosell. Non-small-cell lung cancer. Nature Reviews Disease Primers, 1(1):1-16, 2015.
[92] Taiyu Ban, Lyuzhou Chen, Xiangyu Wang. and Huanhuan Chen. From query tools to causal architects: Harnessing large language models for advanced causal discovery from data. CoRR, 0DW/2306.19022. 2023.
[93] Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. Efficient causal graph discovery using large language models. CoRR, abs/2402.01207, 2024.
[94] Victor-Alexandru Darvariu, Stephen Hailes, and Mirco Musolesi. Large language models are effective priors for causal graph discovery. CoRR, htth/2440.115511, 2024.
[95] Shiv Kampani. David Hidary, Constantijn van der Poel, Martin Ganahl, and Brenda Miao. Llm-initialized differentiable causal discovery. CoRR, abs/2410.21141,, 2024
2024.
[96] Khadija Zanna and Akane Sano. Fairness-driven I1m- based causal discovery with active learning and dynamic scoring. CoRR, abs/2503. 17569, 2025.
[97] Xinyue Wang, Kun Zhou, Wenyi Wu, Har Simrat Singh, Fang Nan, Songyao Jin, Aryan Philip, Saloni Patnaik, Hou Zhu, Shivam Singh, et al. Causal-copilot: An autonomous causal analysis agent. CoRR, abs/2504. 13263, 2025.
[98] Ioannis Tsamardinos, Laura F. Brown, and Constantin F. Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine Learning, 65(1):31-78, 2006.
[99] Antonio L Visbal, Brent A Williams, Francis C Nichols III, Randolph S Marks, James R Jett, Marie-Christinn Aubry, Eric S Edell, Jason A Wampfler, Julian R Molina, and Ping Yang. Gender differences in non-small-cell lung cancer survival: an analysis of 4,618 patients diagnosed between 1997 and 2002. The Annals of Thoracic Surgery, (8(1):209-211 2004.
[100] Jessica Donington and Yolonda L Colson. Sex and gender differences in non-small cell lung cancer. In Seminars in Thorucic and Cardiovascular Surgery, volume 23, pages 137-145. Elsevier, 2011.
[101] Brown, D Eraut, C Trask, and AG Davison. Age and the treatment of lung cancer. Thorax, 51(6):564-568 1996.
[102] Yahan Liang and Hua Cao. Large language model-driven semi-automated construction of named entity datasets in specific knowledge domain. The International Symposium on Computer and Information Processing Technology, pages 312-318. IEEE, 2024.
[103] Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. Lim-check: Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems, 2024.
16

---

[104] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han. Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, CoRR. abs/2404.18930, 2024.
[105] David Maxwell Chickering. Optimal structure identification with greedy search. Journal of Machine Learning Research, --507-554, 2002.
[106] Takashi Nicholas Maeda and Shohei Shimizu. Causal additive models with unobserved variables. In Proceedings of the Conference Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 97-106. AUAI Press, 2021.
[107] Lanling Xu, Jianxun Lian, Wayne Xin Zhao, Ming Gong, Linjun Shou, Daxin Jiang, Xing Xie, and Ji-Rong Wen. Negative sampling for contrastive representation learning: A review. CoRR, Abs/2206.00212, 2022.
[108] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. Proceedings of the International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[109] Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery with- out acyclicity constraints. In The International Conference Learning Representations. OpenReview.net, 2022.
[110] Xiu-Chuan Li and Tongliang Liu. Efficient and trustworthy causal discovery with latent variables and complex relations. In The International Conference on Learning Representations. 2025.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the end of the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 5 and Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
and Appendix for detailed experimental settings and see Abstract for the link of implementation code and data.
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
See Abstract for the link to the implementation code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuiu wwwti//ggui////eeeeceoocc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1 and Appendix D) for detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard deviations are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D for details of experimental environment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: http://...cppppcccccccccccoooeee
Answer: [Yes]
Justification: Authors follow the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No sensitive risky content is included in this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix for the license and terms of use of the datasets used this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsspipetattteeeeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
22

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Appendix D for the details of the constructed datasets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
23

---

Answer: [NA]
Justification: LLMs were used solely for minor editing and language refinement. They did not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy lattps://neuts cetpouripscccoooffeeeee cc/Conffreences/20255 for what should or should not be described.
24

---

Appendix of MLLM-CD
Contents
Table of Notations
26
Limitations and Future Opportunities
27
B.1
Data Availability, Scale and Quality
27
B.2 Modality Coverage
27
B.3 MLLM Capabilities and Assumptions
27
Algorithm
28
D
More Details about Experiments
29
D.I Details on Dataset Construction
29
D.2
Details on Prompts
31
D.3
Experimental Settings and Environment
................................
36
D.4
The Full Results on MAG Dataset
................................
37
D.5
The Full Results on Lung Cancer Dataset
40
D.6
The Full Results on Ablation Study
42
D.7
MLLM-CD with Difterent Causal Discovery Algorithms
42
D.8
Parameter Analysis
42
D.9 Discussion on the Time Complexity and Scalability
44
D.I0 Case Study
45
Broader Impacts
46
25

---

A
Table of Notations
Notation
Description
D
The raw multimodal unstractured dataset.
Ds
The structured multimodal dataset.
DS
The structured multimodal dataset at the t-th iteration.
(c)
The validated counterfactual samples at the t-th iteration.
X
The k-th multimodal sample in the dataset D.
X
The generated counterfactual multimodal sample of X
Xki
The i-th modality of the k-th multimodal sample X₁-
X
The generated counterfactual of the i-th modality of X
C
The statistical causal discovery algorithm.
V
The set of ground truth factors.
V
The set of discovered factors.
VG
The j-th discovered factor.
The j-th ground truth factor.
Ukj
The annotated value of the j-th factor in the k-th multimodal sample.
The counterfactual value of the j-th factor the k-th multimodal sample.
Vk
The factor values of the multimodal sample Xx-
V
The counterfactual factor values of the k-th multimodal sample.
V(t)
The set of discovered factors at the t-th iteration.
Y
The target variable.
yk
The value of the target variable of the k-th multimodal sample.
1/4
The counterfactual value of the target variable of the k-th multimodal sample.
g*
The ground truth causal graph.
G
The discovered causal graph.
E
The ground truth edges in g*
The discovered edges in G.
ek
The semantic representation of Xgs.
e/3
The semantic representation of the generated counterfactual X'e
m((-)
The foundation model used for extracting semantic representations of i-th modality.
The MILM for contrastive factor discovery and multimodal counterfactual reasoning. The image generation model for visual counterfactual generation.
+
P.
The set of intra-modal contrastive pairs in the i-th modality.
Px
The set of inter- modal contrastive pairs.
p(()
The union set of selected intra- and inter-modal contrastive pairs at the 1-th iteration.
Dintra
The prompt for intra-modal contrastive exploration.
Pinter
The prompt for inter-modal contrastive exploration.
Pm
The prompt for factor consolidation.
Pa
The prompt for factor annotation.
PMCR
The prompt for multimodal counterfactual reasoning.
Tsem
The threshold for semantic plausibility check.
Trausal
The threshold for causal consistency check.
26

---

B Limitations and Future Opportunities
While MLLM-CD demonstrates significant promise in revealing causality from multimodal unstruc- tured data, several limitations present avenues for future research and development.
B.1 Data Availability, Scale and Quality
Limitation: Multimodal unstructured causal discovery is still in its early stages, with limited availability of benchmark datasets. In particular, there is a notable lack of large-scale, diverse, and publicly accessible unstructured multimodal datasets tailored for causal discovery. While the datasets used in this work are thoughtfully curated, they are constrained by small sample sizes, and their ground truth causal graphs depend on domain expert knowledge [9] [0-[ON, which can be subjective, incomplete, and costly to scale.
Future Opportunity: Developing larger, more diverse, and rigorously annotated benchmark datasets for multimodal unstructured causal discovery. This could involve semi-automated annotation meth- ods [102]. leveraging simulations with known causal grounds, creating platforms for community- driven dataset creation and ground truth curation. Additionally, exploring the scalability of MI CD larger datasets is crucial direction for future research. Please refer to Section D.9 for detailed discussion on the time complexity and scalability of MLLM-CD.
B.2 Modality Coverage
Limitation: The range of modalities effectively processed by MLLM-CD is principally guided by the capabilities of the leveraged MLLMs. While current MLLMs support an expanding array of common modalities (e.g., text, image, audio), future advancements in MILMM architectures will be beneficial for seamlessly integrating a wider spectrum of specialized (e.g., sensor data, genomic sequences, tabular data embedded within reports) highly heterogeneous data types. This will enable MLLM-CD to tackle an even broader range of real-world problems.
Future Opportunity: Future work could explore techniques for more sophisticated multimodal fusion and cross-modal reasoning, facilitating the discovery of causal relations spanning more diverse modalities.
B.3 MILM Capabilities and Assumptions
Limitation: As we discussed the theoretical analysis, the effectiveness of MLLM-CD is affected by the sophisticated reasoning and generation capabiliiiess inherent in MLLMs. These could include faithful interpretation, consistent logical reasoning, accurate generation of causally plausible counter- factuals across modalities, and robust understanding of intra- and inter-modal interactions. Despite rapid advancements in alignment research [T03] 104], MLLMs may still hallucinate, exhibit biases present in their training data, fail to capture subtle causal nuances.
Future Opportunity: Developing methods rigorously validate, calibrate, and enhance MLLM out- puts within the causal discovery pipeline. This includes designing more robust prompting strategies, incorporating uncertainty quantification for MLLM-derived. factors and relationships, and exploring techniques to mitigate MLLM biases and hallucinations specifically in the context of causal reasoning. Investigating how to formally verify if MLLM reasoning aligns with established causal principles is crucial area.
27

---

C
Algorithm
Algorithm 1 The MLLM-CD Framework
Require:
Multimodal unstructured dataset D; MLLM V; Image generation model Φ; Statistical CD method C: Target Y; Maximum number of iterations T; Prompts Pintra, Pinter: Pm: Pa PMCRi Thresholds Teem Teausal: e; Number of contrastive pairs K.
Ensure: Final discovered causal factors V(2) Final discovered causal graph g(
1: Initialization: t + 1, D <- D. V <- (.
2: while not converged and < T do
3: // Contrastive Factor Discovery
4: Extract semantic representations {ess) for all samples X₁ E D(I) using fi(-).
5: For each modality = 1....
Select top-K K mntrllmodal contrastive pairs p(()) (xxss xbr)ol from D(t).
6:
Candidate factors from intra-modal contrastive exploration v(t) intra +- (PP (1) pm Pintra):
7:
Select top-K cross-modal contrastive pairs p from D(t),
8:
Candidate factors from inter-modal contrastive exploration (t)
inter <- ((P! -Pinter).
9:
Consolidate factors: v < (v(((t)) v(t) inter V(t-1), Pm).
10:
intra
Annotate values: D (D(tt) (
Pa).
12:
// Causal Structure Discovery
13:
Infer causal graph: g(+) C(D (Y)).
14:
// Multimodal Counterfactual Reasoning
Initialize set of validated counterfactuals D + 0.
15:
16:
Identify candidate factors connected with ambiguous endpoints in g(
Let the set of candidate samples for counterfactual reasoning be pDt) <- (P[ j U UP!
18:
for each sample X₃ in p(() do for each uncertain factor V. do
19:
20:
Generate counterfactual: (X,,, v1. 1/66 t (X Vk. Uk. Va = than PMCR; b). Extract embeddings e/ for X
21:
22:
Isem (8)) +
(sim((e))) eky)) 2 Teem
Ny,eNonDuuc [Am
23:
Rindop (Sk,) Va, G(1)
[NonDese (V. g(0))
24:
Teansal
S
Va, ((t) -- -[[Riddep (S Va, G(0) < Trausal).
25:
smm
(S A Teausal (S (S% Va) then
26:
(t)
D CF
U (X₂}.
> Store multimodal counterfactual samples
27:
end if
28:
end for
29:
end for
30:
D(t+1) D(t) (
CF
Augment multimodal dataset
31:
++t+1.
end while
33:
return V(1)gg())
28

---

D More Details about Experiments
D.1 Details on Dataset Construction
we construct multimodal version of the Apple Gastronome dataset [20]. We consider several key factors from different modalities to contribute to the target variable, i.e., the overall rating score of the apple by gastronomes. These factors include 3 visual features (i.e., color, size, and defects) and 5 verbal features (i.e., aroma, taste, juiciness, nutrition, and recmd). The ground truth definitions of these factors are as follows:
Multimodal Apple Gastronome (MAG): Following 20
color: The color of the apple, which can be bright red (positive), or greenish (negative).
size: The size of the apple, which can be large (positive), small (negative).
defects: The presence of defects on the apple's surface, which can be "free of defects" (positive), or "with noticeable defects" (negative).
aroma: The aroma of the apple, which can be strong (positive), musty/rotten (negative).
taste: The taste of the apple, which can be sweet (positive), or sour (negative).
The juiciness of the apple, which can be "abundant and refreshing moisture" (positive), or "dry and lacking moisture" (negative)."
juiciness:
nutrition: The nutritional value of the apple, which can be "highly nutritious with essential nutrients" (positive), "relatively low in nutritional value" (negative).
recmd: The market potential of the apple, which can "has significant market potential and deserves wider recognition" (positive), or "might not bring the expected returns and could even lead to losses" (negative).
We use the Apple Gastronome construction script from the COAT project - to randomly generate 200 samples, each representing a different combination of factor values. These samples constitute the ground truth structured data and are generated according to the causal relationships defined in 5
(a). We also show the faithful causal graph discovered by the FCI algorithm [31 in Figure (b), inferred directly from the ground truth structured data. Then, we use the Gemini 2.0 model [87 to generate the review text for each sample to present verbal features. The image modality is generated by Stable Diffusion 3.5 [90] to present visual features. The generated review text and images are combined to form the final multimodal unstructured dataset.
Examples of the MAG dataset are given Figure 6 The prompts used for generating review texts and images are shown in Prompts D. and D.2 This dataset will be open-sourced under CC-BY 4.0
and
Lung Cancer: This is real-world dataset collected from the MedPix database under Open Database License We select 60 representative lung cancer cases (e.g., Non-Small Cell Lung Cancer [91]. Each case involves four factors, i.e., gender, age, smoking, and lesion (image modality), which jointly contribute to the likelihood of lung cancer diagnosis, i.e., the target variable diagnosis. The definitions of these factors are as follows:
gender: The patient's gender, which is extracted from the demographic information and can be male (positive) or female (negative). Gender differences in lung cancer incidence have been observed [99 [00), with males historically showing higher rates, lower survival rates at one and five years, and significantly increased risk of mortality.
age: The patient's 's age, which is extracted from the demographic information and can be "> 60" (positive) or << 60" (negative). Lung cancer incidence increases with age, with most cases occurring in individuals aged 60 older [100] [01].
smoking: The patient's 's smoking history, which is extracted from the medical history and can be "smoker" (positive) or "non-smoker" (negative). Smoking remains the main risk factor for lung cancer 91
httpe://causaalcost.t github. 10/
https://medpiz.nn nlm. nih. wwa...oo/ooogg
29

---

defects
defects
color
size
aroma
taste
color
size
aroma
taste
nutrition
score
juiciness
nutrition
score
juiciness
recmd
recmd
(a) Ground truth
(b) FCI results
Figure 5: Ground truth and faithful (via FCI algorithm) causal graphs in the MAG dataset.
high profile, opple presents significant drrwhacks. Its musy oder, predomitantly sour tuste, dry testure render it commercially poor and potential financial losses.
This apple presents conceming profile limited mutrtioona value and off-gutinng musty aroms, exsceriated an overwfielmingly sour flaver. While as moisture content approcisble, these strongly suggeat likely inderperfomm finunciallyy and risks genenting negrivve feedback..
This apple presents aroma, thongh decidedly protile and somewhat dry texture may oppeal all palates, Despoi this, its distinct character and significami market potestial suggest deserves greater visibility and wäder appreciation.
Review
unviable,
Image
Score: -3 Figure 6: Examples of the MAG dataset.
Score: -1
Score: 1
lesion: The presence of a lesion in the lung, which is extracted from the medical imaging (e.g., CT scans) and can be "with lesion" (positive) "no clear lesion" (negative). Lesions often present the early visible signs of lung cancer, and imaging techniques such as X-ray, CT, and PET scans among the most widely used tools for its detection and diagnosis 91
These factors contribute to the target variable diagnosis in the way as shown in the ground truth causal graph in Figure (a). (a). The faithful causal graph discovered by the FCI algorithm [3]] is shown in Figure (b). These factors and the causal relationships are carefully curated based on evidence from established medical literature [9]] 99]]00]]
Examples of the Lung Cancer dataset are given in Figure
gender
age
smoking
lesion
gender
age
smoking
lesion
diagnosis
diagnosis
(a) Ground truth
(b) FCI results
Figure 7: Ground truth and faithful (via FCI algorithm) causal graphs in the Lung Cancer dataset.
30

---

Patient Info
your male hisoory of heoduche and unsteady gait.
47 ylu black male with 30 emoking histery. complains dyspnes exertion
oaggan presents emergency depert- ment with worsnning dyspnec on exertion. IIe lad mild dry cough with & weight loss month change sppatiic chills.
repostedd listary
Image
Diagnosis: 2
Diagnosis:
Diagnosis: 5
Figure 8: Examples of the Lung Cancer dataset.
D.2
Details on Prompts
In this section, we provide examples of prompts used in our experiments, including:
Generating review texts for the MAG dataset (Prompt D.I
Generating images for the MAG dataset (Prompt D.2
Intra-modal contrastive exploration (Prompt D.3
Inter-modal contrastive exploration (Prompt D.4
Factor consolidation (Prompt D.5)
Annotation (Prompt D.6
Counterfactual generation (Prompt D.7
Prompt D.1: Generating Review Texts for MAG Dataset
You picky gastronome on apples. You ready to **evaluate apples and write reviews** Your writing should be **clear, solid and convincing to suppliers and customers*
## Task
Please write short review about the evaluation results for given apple.
Evaluation Results: (apple feature)
Requirement:
Combine all of those evaluation results into more detailed review comment.
Single paragraph; No quotation marks; The review comments should be complete. Modern English.
No more than 60 words.
Only output the review comment content directly without any other format content.
Prompt D.2: Generating Images with Stable Diffussin for MAG Dataset
A [size] apple, that is (color), and [defects), white background, centred, fully in frame, well-lit, realistic photo.
Negative Prompt: cropped, partial view, of frame, low quality, low resolution.
31

---

Prompt D.3: An Example of Intra-modal Contrastive Exploration Prompt Pintra
# Intra-modal Contrastive Analysis
I'm showing you pairs of samples with significant differences.
## Pair
### Sample A (ID: (sample A ID)):
** *Score*#; {score A)
* (review A)
### Sample B (ID: (sample B ID)):
*c [score B}
* Reviee***; (review B)
Based on these contrastive pairs, analyze the underlying interactions among potential factors that lead to the observed differences.
# Task: Factor Identification
You are an expert food analyst specializing apple evaluation. Based on the contrasting pairs and interaction analysis:
Identify the key factors that differentiate these samples
2. Focus factors that be clearly observed. Each factor should focus on concrete aspect without overlap with ffactors identified in previous iterations) Create factors that have clear positive, neutral, and negative criteria
# Output Format
Part * Explain your analysis about the key differences between each sample pair.
** Part 2* List the discovered factors using this exact template:
#*Factor Name**
1: [Positive Criterion]
0: [Otherwise; not mentioned]
-1: [Negative Criterion]
32

---

Prompt D.4: An Example of Inter-modal Contrastive Exploration Prompt ps
# Inter-modal Mismatch Analysis
I'm showing you samples where the textual description (review) and visual appearance (image) of samples seem to contradict each other.
## Pair
### Text from Sample A (ID: (sample ID)):
**Score#:; {score A}
** Review** (review A)
### Image from Sample (ID: (sample B ID)):
* Score**: {score B)
Image*** (image B)
Notice the potential mismatch between what the review describes and what the image shows. Analyze the underlying interactions among potential factors that could lead to this mismatch.
Task: Identify Factors that Explain Text-Image Discrepancies
You expert food analyst specializing in apple evaluation. Based the contrastive pairs and interaction analysis:
1. Identify key factors where the textual reviews contradict what's visible in the paired images
2. Each factor should focus on one concrete aspect without overlap identified in previous iterations)
3. Create factors that can explain these discrepancies with clear positive, neutral, and negative criteria
# Output Format
*Part *** Explain your observations about the mismatches of each pair of textual and visual information.
Part 2** List factors that explain these discrepancies using this exact template:
**Factor **me**
1: [Positive Criterion]
0: [Otherwise; or not mentioned]
-1: [Negative Criterion]
33

---

Prompt D.5: An Example of Factor Consolidation Prompt Pm
# Factor Deduplication and Refinement
Below list of factors identified from different contrastive analyses:
Factor I: (factor /}
1: [Positive Criterion]
-0: [Neutral Criterion]
--1: [Negative Criterion]
Please:
Identify and merge similar factors; Each factor should be distinct and not overlap with others and be specific to one aspect; Avoid general factors like "overall quality"
Refine factor definitions for clarity and precision
Output consolidated list of the most important and non-overlapping factors
Only output the final results using this exact format for each factor:
**Factor Name**
1: [Positive Criterion]
0: [Otherwise; or not mentioned]
-1 [Negative Criterion]
Prompt D.6: An Example of Annotation Prompt Pa
# Factor Annotation
Please annotate the following samples based on these factors:
## Factor 1: [factor 1}
1: [Positive Criterion]
0: [Neutral Criterion]
-1: [Negative Criterion]
# Samples to Annotate
## Sample 1:
** Score**; {score }}
*Revie (review /)
** Image*** (image /)
# Task: For each sample, assign a value (-1, 0, or 1) to each factor based on the criteria above.
# Output Format
Please format your response strictly follows:
**Sample [ID]**
Factor ([factorl name]): [Value]
Factor ([factor2 namel): [Value]
Repeat for all samples.
34

---

Prompt D.7: An Example of Counterfactual Generation Prompt PMCR
# Refining Causal Factors and Relationships
I'm analyzing factors that affect sample scores based on reviews and images. need your help to refine my understanding of causal relationships.
## Data
### Sample 1:
Score**; {score }}
** Review (review I}
Image*** (image 1)
## Current Factors
Factor information!
## Aunotated Factors {Annttatdd factor values)
## Uncertain Causal Relationships
I've identified these causal factors, but there is uncertainty in the causal relationships of the following factors: (Uncertain factor information)
# Task: Counterfactual Reasoning
For each uncertain relationship above, please create a counterfactual scenario: "If factor X were different (i.e., value being reversed the factor mentioned and skip the sample the factor is not mentioned for this sample), how would other factors be affected?" Based on this assumption and your knowledge of apples, predict the values of other factors. Only create counterfactual scenarios that support the valid and reasonable causal relationships and directions. Specifically, there are two types of factors, verbal and visual factors. If the verbal factor is modified, revise the review text directly with minimum changes to reflect the new scenario. If the visual factor modified, state a short instruction the changes that need to be applied to the image, e.g.. "Change the color of the apple to red." If no visual changes, please state 'N/A'.
# Output Format
Please structure your response in the following format and respectively list the values of all the factors for each sample in each counterfactual scenario:
## Counterfactual Scenario 1: [Changed Factor Name]
* *aample **#;
Factor 1: [Value]
Factor 2: [Value]
Factor N: [Value]
Review: [Modified Review Text]
Image: [Image Modification Description] (If applicable, otherwise 'N/A")
** **Sample 2**;
## Counterfactual Scenario [Changed Factor Name]
35

---

D.3 Experimental Settings and Environment
D.3.1 Implementation Details
We detail the implementation of MLLM-CD as follows.
In the contrastive factor discovery module, we use the pretrained CLIP model 75 with the ViT- B/32 checkpoint from OpenAI's official release to extract textual and visual embeddings from the multimodal samples in the MAG and Lung Cancer datasets. For intra- and inter-modal contrastive exploration, we choose the top K pairs of samples with the prompts in Section D.2 for factor identification and annotation.
In the causal structure discovery module, adopt the FCI algorithm 31
infer the causal structure from the annotated factors. Additional discussion on different CD methods can be found in Section D.7
We use the FCI implementation from the causal-learn library [2]. available the website
In the multimodal counterfactual reasoning module, we set the threshold parameters as Tsem 0.7 and Tcausal 0.4 for consistency validation. is small constant set to 10 to highlight any changes
in the non-descendant nodes. Following [20), the maximum number of iterations is set to T = 3. Further analysis of parameter choices is discussed in Section D.8) The MLLMs used in experiments are accessed via API calls, and all experiments are conducted on a server with two Intel Xeon 6346
CPUs, 256GB RAM, and two NVIDIA A40 GPUs.
D.3.2 Baselines
We provide detailed list of the representative baselines used in experiments, including:
META a strategy from 20 that identifies both causal factors and causal relationships using only the knowledge encoded in MLLMs, guided by contextual information and task descriptions.
Pairwise [29] leverages the knowledge of MLLMs to identify potential causal relationships between the given pairs of factors. Note that, this method only focuses on predicting causal relationships based on given factors, and cannot directly uncover causal factors from unstructured data. In our experiments, we use the factors identified by META as input to this method.
Triplet [60] extends the Pairwise method by using triplet-based queries and robust aggre- gation strategy to improve causal relation identification. Similar to Pairwise, does not involve causal factor discovery, and we use the factors discovered by META as input.
COAT [20) is the most closely related work to ours and, to the best of our knowledge, the only method specifically designed for causal discovery from unstructured data. We use the official implementation of COAT and adapt it to discover the full causal graph using FCI algorithm (3I. Following the original setup 200 we set the number of iterations to T
D.3.3 Evaluation Metrics
Two major steps are involved in causal discovery from multimodal unstructured data, i.c.. factor identification and structure discovery. To validate the effectiveness of both steps, we adopt the following widely used metrics (30
1) Factor Identification: We assess the accuracy and comprehensiveness of identified causal factors using precision (NP), recall (NR), and Fi score (NF). Note that, the identified factor names may vary across different methods and models. For evaluation, we manually align semantically correct factors with the ground truth names ensure consistency.
2) Structure Discovery:
The quality of the discovered causal structure is evaluated using adjacency precision (AP), adjacency recall (AR), and their F1 score (AF).
3) Combination:
Different from conventional CD settings where all methods operate on the same predefined set of factors, methods designed for unstructured data may identify different sets of factors, leading to subgraphs that differ in coverage of the ground truth. For fair and comprehensive
36

---

Table 6: The full results for factor identification and structure discovery on the MAG dataset
MLLM
Method
NP
NR
NF1
AP
AR
AF
ESHD
META
0.08
0.52
0.48 0.07
0.72
0.37 0:06
0.49 0.04
24.33 3.51
GPT-4o
Triplet
5.77
COAT
0.78 0.19
0.37 = 0.13
0.19 1: 0.15
0.37 0.32
0.19 = 0.17
0.25 0.22
18.67 2.52
MLLMM-CD
0.83 +: 0.11
0.85 +: 0.06
0.84 0.09
0.69 4: 0.03
0.41 0.06
0.51 0.04
15.33 2.31
META
0.71 0.07
0.63 0.06
0.07 0.07
0.09 0.08
0.41 0.13
0.51 0.11
18.07 2.31
Pairwise
0.49 0.09
0.56 0.11
0.51 0.06
30.00 2.00
Gemini 2.0
Triplet
0.41 0.02
0.59 0.13
0.48 0.05
32.00 4: 2.00
COAT
0.133
0.09
0.20 0.06
MLLM-CD
0.03
0.13
META
0.51 0.06
0.41 + 0.06
0.45 1.45+:000
0.81 # 0.17
0.26 +: 0.06
0.39 0.07
21.67 + 0.58
Pairwise
0.50 0.17
0.30 0.06
0.36 = 0.04
34.67 + 6.66
LLaMA
Triplet
0.66 à: 0.32
0.30 4: 0.08
0.38 0.04
34.13 1. 7.77
COAT
1.00 0.00
0.41 4: 0.06
0.58 0.07
0.89 :: 0.19
0.30 +: 0.08
0.44 0.10
14.67 4: 1.15
MLLM-CD
1.00
0.85 + 0.06
0.92 0.04
0.62 0.08
0.59 0.06
0.80 0.01
13.33 0.58
META
0.11
0.44
0.75
3.06
Pairwise
0.02
0.33 0.00
0.34 0.01
28.33 11.37
Grok-2v
Triplet
0.32 0.02
0.33 0.00
0.33 = 0.01
30.33 12.34
COAT
1.00 0.00
0.37 +: 0.17
0.53 0.18
0.17 :: 0.29
0.07 +: 0.13
0.10 = 0.18
16:13 1.15
MLLM-CD
1.00
0.85 0.06
0.92 0.04
0.79 0.21
0.41 0.00
0.56 =: 0.06
11.00 2.65
META
0.56::0.11
0.50 0.10
0.52 0.10
0.74 0.10
0.34 4: 0.09
0.46 0.07
21.25 3.11
Pairwise
0.38
Average
Triplct
39.35
COAT
+:
+:
d:
MILM-CD
0.06
0.57
evaluation, we adapt the standard SHD metric [88 to our scenarios, and present the extended SHD) (ESHD) metric evaluate the overall performance of causal factor and structure discovery. Apart from the computation of standard SHD on the matched subgraph, it also accounts for missing and spurious factors, as well as the corresponding missing and spurious edges associated with those factors. A lower ESHD value indicates that the discovered graph more closely matches the ground truth, thereby reflecting better overall performance both factor and structure discovery.
D.4 The Full Results MAG Dataset
The full quantitative results for factor identification and structure discovery on the MAG dataset are shown in Table Here, we also illustrate the visual comparison of causal graphs discovered by different methods and MLLM backbones on the MAG dataset. The results are shown in Figures
and 12 Several key observations can be made:
1. META, Pairwise, and Triplet infer causal relationships directly from the inherent knowledge of MLLMs. While this leads to non-ambiguity, the resulting graphs tend to be overly complex, often containing numerous redundant edges (e.g.. Figure TO (b)-(d)).
2. COAT identifies only limited subset of causal factors and relationships. Moreover, the inferred relationships often remain uncertain.
MLLM-CD produces more comprehensive set of causal factors and relationships. Compared to COAT, the structural ambiguities are reduced, and the resulting graph aligns more closely with the ground-truth causal graph and the faithful graph shown Figure 5 (b).
'Pairwise and Triplet do not involve the step of factor identification, thus metrics related to factor identification applicable. Instead, the factors discovered META for their structure discovery.
37

---

defects
color
size
aroma
taste
aarorni
Laste
aciidit
nutrition
score
juiciness
color
Cuciness
Alvorr
uiciness
recmd
score
(a)
Ground truth
(b) META
(c) Pairwise
juiciness
nutrition
score
aroma
taste
sentiment
color
defects
recmd
Score
defects
(d) Triplet
(e) COAT
(1) MLLM-CD
Figure 9: Causal graphs discovered with Gemini 2.0 on the MAG dataset.
defocts
color
sizee
aroma
laste
score
rOcind
(a) Ground truth
(b) META
(c) Pairwise
shape
color
deflccs
nutrition
nutrition
amnotthmess
defocts
SCOrO
recmd
jucibess
tastc
(d) Triplet
(c) COAT
(f) MLLM-CD
Figure 10: Causal graphs discovered with GPT-4o on the MAG dataset.
38

---

defects
origin
color
size
aroma
taste
nutrition
score
juiciness
juiciness
taste
color
preparation
recmd
SCore
(a) Ground truth
(b) META
(c) Pairwise
eutribion
Lasto
SEUIT
consislency
texture
culor
laste
aroma
aroma
juiciness
roenn
(d) Triplet
(e) COAT
(I) MLLM-CD
Figure 11: Causal graphs discovered with LLaMA on the MAG dataset.
defects
color
size
aroma
taste
freshness
variety
JUCEKO
nutrition
Score
juiciness
taste
color
juiciness
aroma
reomd
(a) Ground truth
(b) META
(c) Pairwise
nutrition
juiciness
nutrition
aroma
aroma
juiciness
defects
taste
(d) Triplet
(c) COAT
( MLLM-CD
Figure 12: Causal graphs discovered with Grok-2v on the MAG dataset.
39

---

D.5 The Full Results on Lung Cancer Dataset
The detailed results of different methods using various MLLM backbones the Lung Cancer dataset are presented in Table 7 Consistent with the findings on the MAG dataset, MLLM-CD demonstrates superior performance both causal factor identification and causal structure discovery
most cases. and
Qualitative comparisons of discovered causal graphs are shown in Figures 13
14 15 and T6 Among
all methods, MLLM-CD consistently produces high-quality causal graphs that closely align with the ground truth and the faithful causal graphs illustrated in Figure (b), for example, see Figure T3 (f).
Table 7: Causal factor identification and structure discovery performance on the Lung Cancer dataset."
LLM
Method
NP
NR
NF1
AP
AP1
ESHD
META
0.05
0.42
0.50
0.67 0.00
31.67 +
Pairwise
0.67 1: 0.00
0.30 = 0.00
0.57 0.00
36.33 4: 10.21
GPT-4o
Triplet
0.67 1. 0.00
0.30 =: 0.00
0.57 0.00
47.33 14.98
COAT
0.47 0.118
0.40 0.00
0.42 0.07
0.33 0.58
0.08 = 0.14
0.13 0.23
11.67 + 3.06
MLLMMCD
0.89 1: 0.10
1.00 +: 0.00
0.94 0.05
0.92 à: 0.14
0.58 0.14
0.69 0.05
5.00 0.00
META
0.73 0.00
0.54 0.06
0.92 0.14
0.67
0.76
16.00
0.67
Gemini
Triplet
0.02
COAT
0.25
0.47 0.12
0.50
1.00 0.00
0.33 0.14
0:49
8.67 2.08
MLLM-CD
0.11
1.00 0.00
0.97 0.05
0.92 0.14
0.83 0.14
0.87 0.13
4.67 0.58
META
0.088
0.67 + 0.12
0.50 0.04
0.72 + 0.25
0.42 0.14
0.52 0.17
19:33 + 5.03
Pairwise
0.76 0.21
0.58 0.14
0.63 0.05
32.33 : 17:24
4
Triplet
0.61 1: 0.10
0.58 0.14
0.58 0.02
36.67 20.40
COAT
0.11
0.47
0.000
MLLM-CD
0.02
META
0.29 0.04
0.40 0.00
0.33 0.03
1.00 + 0.00
0.25 0.00
0.40 0.00
20.00 + 5.29
Pairwise
1.00 :: 0.00
0.25 = 0.00
0.40 0.00
28. 8.89
Grok-2v
Triplet
1.00 1: 0.00
0.25 0.00
0.40 0.00
31.00 1. 8.00
COAT
0.56 0.21
0.47 1: 0.23
0.51 0.22
0.67 1: 0.58
0.17 0.14
0.27 0.23
9.67 1.53
MLLM-CD
0.12
0.80 0.00
0.83 0.05
1.00 0.00
0.25 0.00
0.40 0.00
6.00 1.00
META
0.008
0.09
Pairwise
0.20
0.50
0.55 0.10
32.50
Average
Triplet
0.71 1: 0.19
0.50 A: 0.18
0.54 0.08
38.
13.36
COAT
0.65 1: 0.23
0.45 4: 0.12
0.52 0.13
0.75 4: 0.45
0.21 0.14
0.32 112:::00 0.21
9.42 2.31
MLLM-CD
0.88 0.09
0.93 0.10
0.90 0.07
0.94 0.11
0.58 0.25
0.68 0.19
5.33 0.78
risk
smoking
environment
lesion
age
cancer
gender
age
smoking
lesion
response
diagnosis
Diagnosis
(a) Ground truth
(b) META
(c) Pairwise
smoking
gender
lesion
ago
smoking
symptom
diagnosis
diagnosis
Iesion
(d) Triplet
(e) COAT
(1) MLLM-CD
Figure 13: Causal graphs discovered with Gemini 2.0 on the Lung Cancer dataset.
40

---

gender
smokingg
lesion
oxpeers
proteppetre
diagnosis
corroticttio
syrppters
Iniee
Inkery
(a) Ground truth
(b) META
diagnosis
dependence
(c) Pairwise
impact
lesion
gender
ago
symptom
lesion
Jiagnosis
smoking
symplom
(d) Triplet
(c) COAT
(1) MLLM-CD
Figure 14: Causal graphs discovered with GPT-4o on the Lung Cancer dataset.
gender
age
smoking
lesion
smaking
history
exposure
symptoms
exposure
diagnosis
lesson
smokking
history
(a) Ground truth
diagnosus
Josion
sympioons
(b) META
diagnosis
(c) Pairwise
diagnosis
lesion
gender
smoking
Age
smoking
symptoms
lesion
diagnosis
symptoms
degannds
(d) Triplet
(e) COAT
(1) MLLM-CD
Figure 15: Causal graphs discovered with LLaMA 4 on the Lung Cancer dataset.
gender
age
smoking
lesion
oppc
prstepoations
Howsyle
Nevvnk
prdiquotice
diagnosis
Frstiicon
(a) Ground truth
PEporeno
Traoyle
Book
diugncen
dagsoat
(b) META
(c) Pairwise
symptoms
lesion
smoking
diagmosis
gender
lesion
diagnosis
age
condition
(d) Triplet
(c) COAT
(1) MLLM-CD
Figure 16: Causal graphs discovered with Grok-2v on the Lung Cancer dataset.
41

---

D.6 The Full Results on Ablation Study
As complement to Section 4.4, we present additional ablation study results on the MAG and Lung Cancer datasets using two representative MLLMs, i.e., GPT-4o and Gemini 2.0. The detailed results are shown Table g and Table respectively.
Table 8: Ablation study of MLLM-CD on the MAG dataset.
LLM
Method
NP
NR
NFT
APT
AR
AF
ESHD
0.15
0.67
0.73
0.32
0.22 0.11
0.30 0.10
0.74
0.80
0.30
0.30
0.38 0.20
GPT-4o
w/o Both
0.79 i 0.26
0.48 0.32
0.58 0.31
0.30 0.26
0.15 - 0.13
0.20 = 0.17
16.33 2.08
MIJ M-CD
0.83 0.11
0.85 + 0.06
0.84 0.09
0.69 + 0.05
0.41 + 0.06
0.51 0.04
15.33 +: 2.31
w/o CFD
0.94 0.100
0.60 0.06
0.07
0.76 + 0.23
0.37 + 0.13
0.47 0.09
15.00 0.00
w/o CR
0.81 0.08
0.78 0.11
0,81 0.09
0.71 0.07
0.11 = 0.06
0.32 0.06
15.67 3.51
Gemini
Both
0.81 0.11
0.41 +: 0.06
0.54 0.08
1.00 0.00
0.26
0.41 0.08
16:33 4: 2.08
MLLM-CD
0.00
Table 9: Ablation study of MLLM-CD on the Lung Cancer dataset.
IIM
Method
N
NR
NF
AP
AR
AF
ESHD /
CFD
0.73
0.33 0.58
0.17
9.67
wlo
0.83 = 0.00
1.00 0.00
0.91 +
0.72 0.25
0.42 0.14
0.49 0.09
5.33
GPT-4o
wio Both
0.56 =: 0.10
0.53 1 0.31
0.52 1: 0.22
0.33 0.58
0.17 1: 0.29
0.22 0.38
9.87 4: 2.08
MLLM-CC
0.89 =: 0.10
1.00 0.00
0.94 1: 0.05
0.92 0.14
0.58 :: 0.14
0.69 0.05
5.00 4: 0.00
w/o CFD
0.65 = 0.09
0.60 1.00:000 0.00
0.62 + 0.04
1.00 0.00
0.33 + 0.14
0.36 0.34
8.00 1.00
CR
1.00
0.94
0.83 0.29
0.25 +
0.38
5.33 1.53
Gemini
Both
0.35
0.08
9.67
MLLM-CD
0.94 = 0.10
1.00 0.00
0.97 +: 0.05
0.92 0.14
0.83 0.14
0.87
4.67 + 0.58
D.7 MLLM-CD with Different Causal Discovery Algorithms
Since MLLM-CD is designed as general framework for causal discovery from multimodal un- structured data, it can implemented with various causal discovery algorithms. In this section, we evaluate the performance of MLLM-CD when combined with different algorithms and compare its effectiveness against the state-of-the-an unstructured causal discovery method, COAT. We select three representative causal discovery algorithms from different categories: (1) PC 31 (constraint-based) (2) GES [005] (score-based), and (3) CAM-UV (106) (functional causal model-based with latent variable handling). Using Gemini 2.0 as the MILM backbone, we evaluate the structure discovery performance on both the MAG and Lung Cancer datasets. The results, presented in Table TO) show that MLLM-CD achieves overall superior performance than COAT across various causal discovery algorithms, demonstrating its robustness and adaptability.
D.8 Parameter Analysis
In this section, we analyze the impact of different parameters on the performance of MLLM-CD. We mainly focus on the following parameters: (1) the number of iterations T; (2) the number of pairs K; (3) the semantic plausibility threshold Tsem: and (4) the causal consistency threshold Trausal-
Iteration Number T. Following (20). we set the maximum number of iterations to 3. As shown in Table IT the performance of MLLM-CD improves notably from to 2 iterations, while it remains relatively stable between 2 and iterations. In most cases, using iterations provides a good trade-off between performance and computational efficiency.
Figure 17 further illustrates the structural refinement achieved from iteration to iteration 2. Notably, the structure becomes more accurate and less ambiguous, yielding results that more closely align with the faithful causal graph identified by FCI, shown in Figure (b).
Pair Number K. We explore the impact of the number of contrastive pairs K on the performance of MLLM-CD. As shown in Figures 18 (a) and 19 (a), the performance of MLLM-CD improves
42

---

Table 10: Causal structure discovery results of COAT and MLLM-CD with different causal discovery algorithms on the MAG and Lung Cancer datasets.
Dataset
CD
Method
AP
AR 1
AF
ESHD +
COAT
1.00
0.22
0.36
19.00
PC
MLLM-CD
0.89
0.56
0.68
14.00
COAT
0.67
0.22
0.33
16.00
MAG
GES
MLLM-CD
0.75
0.33
0.46
16.00
COAT
1.00
0.11
0.20
16.00
CAM-UV
MLLM-CD
0.50
0.22
0.31
13.00
COAT
1.00
0.50
0.67
8.00
PC
MLLM-CD
0.75
0.75
0.75
2.00
COAT
1.00
0.25
0.40
11.00
Lung
GES
MLLM-CD
0.75
0.75
0.75
3.00
COAT
1.00
0.25
0.40
10.00
CAM-UV
MLLM-CD
0.60
0.75
0.67
3.00
Table 11: The performance of MLLM-CD with different iteration numbers on the MAG and Lung Cancer datasets.
Dataset
T
NP
NR
NF1
AP
AR f
AF
ESHI 4
0.84
0.78
0.81
0.74
0.41
0.52
15.67
MAG
2
0.86
0.89
0.87
0.76
0.52
0.60
14.00
3
0.89
0.89
0.89
0.80
0.44
0.57
13,00
1
0.89
1.00
0.94
0.83
0.25
0.38
5.33
Lung
2
0.94
1.00
0.97
0.92
0.83
0.87
4.67
3
0.94
1.00
0.97
1.00
0.75
0.86
4.00
with increasing at first, but diminishes slightly after reaching a certain point. This implies that using too few pairs may fail to capture sufficient contrastive information for effective factor discovery, while an excessive number of pairs could dilute the guidance signal and reduce model focus. Based findings, we choose K value in all experiments.
balanced and effective setting for both datasets and adopt this
Semantic Plausibility Threshold Tsem This parameter determines how many counterfactual samples are retained based on their semantic plausibility. A high threshold risks discarding many poten- tially useful samples, thereby limiting the benefits of counterfactual reasoning in causal discovery. Conversely, a low threshold may admit low-quality or hallucinated samples, introducing noise. As shown in Figures 18 (b) and 19 (b), MLLM-CD achieves optimal and stable performance when Taem 0.7 on both datasets. We therefore set this value in our experiments. addition, we observe that MLLM-CD is relatively insensitive to variations in Tsem indicating robustness to this parameter.
Causal Consistency Threshold Teausal- This parameter controls the stringency of the causal consis- tency check. As observed in Figures 18 (c) and (C)(), setting a low threshold results in overly strict filtering, which can reduce number of usable counterfactuals and hinder performance. other hand, high threshold may allow too many causally inconsistent samples, introducing noise. We find that Teausal 0.4 offers strong performance regarding factor identification and causal structure discovery across both datasets, and use this value in experiments.
43

---

diagnosis
lesion
age
smoking
gender
lesion
age
smoking
gender
diagnosis
(a) Iteration
(b) Iteration
Figure 17: Causal graphs discovered with Grok-2v on the Lung Cancer dataset.
ESHD
04
06
07
03
03
Tarm
Texnsal
(4)
(0)
Figure 18: The performance of MLLM-CD with different parameters of K, Teem and Teausal on the
MAG dataset. (The lower the ESHD, the better the performance.)
D.9 Discussion on the Time Complexity and Scalability
Assume there are samples, modalities, d factors, K contrastive pairs, and T iterations. Assume the cost of single MILLM query as Cy and an image generation query as CĄ- The number of samples grow in each iteration due to counterfactual augmentation; let N, be the number of samples in the t-th iteration. We first analyze the complexity one iteration
Contrastive Factor Discovery: In intra motall contrastive exploration, it requires O(m - N2) to find top-K K intra-modal contrastive pairs, and (
Cv) for the MLLM query. In moderl contrastive exploration, it requires O(N?) ) for pair selection and O(K Cy for the MLLM query. It also requires O(C₂) for factor consolidation and ONN, C.) for factor annotation. Therefore, the total complexity for this module is (m N? + K Cy + N₂ CC)).
Causal Structure Discovery: The complexity depends heavily the chosen causal discovery algorithm C, N₁. and d. We denote the complexity as O(CesD(N,, d)).
Multimodal Counterfactual Generation: It requires O (Cv + C+)) for counterfactual generation, where NCF, the number of generated counterfactual samples t-th iteration. For semantic plausibility validation and causal consistency validation. typically requires O(NeF, mm)
and O(Ne,, d). Thus, the total complexity for this module is O(NeF, (C* + C + d)).
Overall Complexity: Let Nmaz be the maximum Nr, we have the following rough upper bound for the time complexity of the overall MILM-CD:
O (mN7 max + Nmaz Cy CcSD (Nmax, d) NCE, (Cw + C. + d)))
(D.7)
Scalability: The scalability of MLLM-CD with respect to the dataset size is primarily challenged by two aspects:
1) The CFD module exhibits quadratic complexity N2 ally expensive for large datasets. However, this can be mitigated by using strategies such negative sampling [107 [088] for contrastive pair selection.
)) which can become computation-
2) The statistical causal discovery algorithm often involves polynomial complexity in N and exponential complexity in d. Learning causal structures over a large number of samples and factors remains open problem in the literature [109].
In the future, as we briefly discussed B. we will construct larger-scale multimodal unstructured datasets for causal discovery and improve the scalability and efficiency of MILM-CD by leveraging advanced sampling techniques [107 108 and more efficient causal discovery algorithms, such as
ENCO [19 and ETCD [100].
44

---

E3HD
ESSDD
7aem
Teumaal
(8)
(6)
Figure 19: The performance of MLLM-CD with different parameters of K, Tsem and Teausal on the Lung Cancer dataset. (The lower the ESHD, the better the performance.)
apple Scare: disupoiniingyy low nutritional value and an off put- ting musy aroma indicative of decap. predeminantly
Score:
Discovered Factors
prodile,
Despite concerning potertial storage need addressing, this apple boasts notably high nutritional profile and pleasing swortnrss, suggosting signift appeal that greater attention and distribution.
Aroma
Taste
Text Text
and dry any appeal,
Juiciness
suggesting is unlikely to meet consumer expectations and risks signifi- flnancial losses.
texture
Nutrition
Sentiment
Recmd
Score:
Image Image
Defects
This apple peesenlls with conceming profile: discemible musty aroma precedes distieclyy Couplad upplcssntlly dry
Color
Text Image
low nutritional value, this spreames is unilkely meet cusiomer expecta posing significant and poteutial
Defects
Figure 20: Examples of selected intra-modal and inter-modal contrastive pairs on the MAG dataset.
D.10 Case Study
We present a case study to illustrate how MILM-CD contributes to the multimodal unstructured causal discovery process.
Contrastive Factor Discovery: We first show some examples of intra-modal and inter-modal contrastive pairs selected by MLLM-CD on the MAG dataset in Figure 20 For intra-modal contrastive exploration, MLLM-CD effectively constructs contrastive pairs that highlight several key variations, such as the differences in nutritional profiles, taste, and the presence of defects. By analyzing the underlying intra-mode modal interactions, the model successfully identifies largely accurate and comprehensive set of factors. For inter-modal contrastive exploration, MLLM-CD selects mismatched pairs expose contradictions between modalities. This encourages the model to uncover more subtle interactions. Notably, it identifies the factor color, which was not
factors governing captured by the intra-modal analysis. This demonstrates the strength and necessity the CFD module in enhancing factor identification from multimodal unstructured data.
To demonstrate the effectiveness the MCR module in generating meaningful counterfactual samples, we present both textual and visual counterfactuals generated by MLLM-CD on the MAG and Lung Cancer datasets, as shown in Figures 2T and [2 respectively. The generated counterfactuals are both semantically plausible and causally consistent. For instance, Figure 2T when reasoning about counterfactual scenario "whut if nutritional profile were differrnn", the model generates a new review comment that aligns semantically and causally with the original context. More importantly, captures the latent causal relationship between the nutritional profile and market potential (as shown in the ground truth causal graph in Figure 5(a))
Multimodal Counterfactual Reasoning:
45

---

Original
Counterfactuals
This upple bousts remarkable nutri- tional profile, captivating pleasingly sweet yet refreshing flavor, complemented by its abundant mois- ture; its significant clearly warrants broader atrention and availability.
This apple presents concerning nutri- tional profile, captivating aromn, and plensingly sweet yet refreshing flavor, complemented by its abundani moisttre;; limited
Different Nutrition?
potential
Counterfactual Generation
potential suggests requires further improvements to gain broader atlention and availability
Having Defects?
Counterfactual Generation
Figure 21: Examples of counterfactual samples generated by MLLM-CD on the MAG dataset.
Original
Counterfactuals
54 y/o female with report abnormal chest radiograph another institution. A prior routine chest radiograph from four years earlier was reportedly normal.
yla male with report of abboormal chest radiograph at another institution. A prior routine chest radiograph from four carlier reportedly normal.
Different Gender?
Counterfactual Generation
Reduce Lesion?
Counterfactual Generation
Figure 22: Examples of counterfactual samples generated by MILM-CD the Lung Cancer dataset.
based on its knowledge and appropriately modifies the market potential response to changes in the nutritional profile. Similarly, in the visual domain, the counterfactual samples accurately reflect causal changes. In Figure 22] for instance, the generated image clearly shows reduction lesion area, reflecting the intended counterfactual condition. These results highlight the capability of the MCR module to produce high-quality, causally informative counterfactual samples that contribute meaningfully the causal discovery process.
We further illustrate the overall causal discovery process of MLLM-CD using an example from the MAG dataset, shown in Figure 23 Given multimodal, unstructured input, MLLM-CD first identifies the factors nutrition, color, defects, aroma, taste, jui juiciness. sentiment. recmd, along with the target variable score. For clarity, some of these factors are highlighted directly within the original sample. The model then annotates the values of these factors to construct the structured data input for causal structure discovery. Once initial causal graph is inferred, MILM-CD applies the MCR module for iterative refinement, enhancing the accuracy and redueing structural ambiguity.
Overall Causal Discovery Process:
E Broader Impacts
This work aims to advance causal discovery by extending it to multimodal unstructured data, leverag- ing the understanding and reasoning capabilities of multimodal large language models (MI.LMs). The goal is to enable broader applications and societal benefits, such accelerating scientific dis-
46

---

Multimodal L Lsstracturd Data
Factor Idencificalon & Annotation
Caual Siructure Bacnery
Thargaa
deleto
rbcsn mas dry luk Suc
rupor
Multinudal
Figure 23: An example of the causal discovery process MLLM-CD on the MAG dataset.
covery, enhancing multimodal decision-maie making, and enhancing medical diagnosis systems. This study does not involve any human subjects or raise new ethical concerns. Dataset usage adheres to public availability and anonymization principles. The proposed method is intended for beneficial applications and does not introduce specific risks related to harmful insights, privacy, security, legal compliance, or research integrity, beyond the general considerations for MLLM technologies.
47


Reviews:
[{"review_id": "CAw3YU2Bev", "reviewer": "Reviewer_p1ec", "summary": "The paper claims to solve the problem of latent variable causal discovery using a multi-modal foundation modeling framework. It uses CLIP embeddings from multi-modal inputs using contrasting learning, and then performs FCI on these embeddings. It further refines this graph by generating counterfactual samples and reintegrating them into the training loop.", "questions": "1. Why are the optimal graphs in proposition 3.2 different? \n2. In the proof of 3.2, how do authors justify making Assumption D.2? What does it mean to use a direct approach to obtain the features -- could the authors be more specific about these techniques? \n3. How do the authors solve FCI within a continuous optimization framework? Furthermore, they claim that their algorithm can solve the latent variable problem but the ground truth graph does not include any latent variables. Could the authors elaborate more on that?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n- The overall idea to use CLIP and causal graph on top of it innovative. \n- The empirical results look strong, although the causal graphs being used seems simpler than how general the authors claim that their algorithm is, in the paper. \n\nWeaknesses: \n- Lack of clarity: \n1. \\Psi is used before it is defined. \n2. Counterfactual generation and how it is done is not entirely clear and could be better described in the text. \n3. It is not clear how FCI was optimized within an LLM framework, since it's a constraint-based algorithm. \n4. There are some concerns regarding the theory - please see questions below. \n5. The empirical results do not include ground truth graphs with latent variables (when in the space of factors), yet the authors claim that their method can successfully work in this setting.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's constructive comments, which have helped us significantly improve the clarity of our paper.\n\n**W1.** $\\Psi$ is used before it is defined.\n\n**RW1.** We thank the reviewer for pointing this out. The symbol $\\Psi$ is first defined and used on Page 4, Line 155, where we state “This module guides the MLLM $\\Psi$ to explore the intra- and inter-modal interactions…”. Here, $\\Psi$ is defined as the MLLM for contrastive factor discovery and multimodal counterfactual reasoning. It is also defined in our Appendix A (Table of Notations, Page 2). To improve clarity, we will define it more explicitly and clearly in Line 155, as “This module adopts an MLLM, denoted as $\\Psi$, to explore the intra- and inter-modal interactions…”\n\n\n**W2.** Counterfactual generation and how it is done is not entirely clear and could be better described in the text.\n\n**RW2.** Thank you. The process of counterfactual generation is introduced in Section 3.4 (Page 5), with other details that can help comprehension, such as the organized algorithm in Appendix C (Page 5), and illustrated examples in Appendix E.10 (Figures 21&22, Page 26). Specifically, the process includes:\n\n1) **Identifying Uncertainty**: We select factors with ambiguous relationships (e.g., undirected edges) in the computed graph $G^{(t)}$.\n2) **Prompting for Reasoning**: We use the MLLM ($\\Psi$) with a “what-if” prompt to reason about the counterfactual scenario of an intervention on a candidate factor (e.g., In the given sample, if the factor “nutrition” were different, how would other factors be affected?)\n3) **Generating Samples**: By thinking of the counterfactual value of each factor, model $\\Psi$ generates the corresponding multimodal counterfactual samples. Textual data can be directly revised by model $\\Psi$ from the original text to reflect the changes in factor values, while visual data will be produced by an image generation model $\\Phi$ with a brief description of image modifications by model $\\Psi$. Some generated examples can be seen in Figures 21&22, Page 26.\n4) **Validation**: We validate these generated samples for semantic plausibility and causal consistency to filter out incorrect and implausible counterfactuals.\n\nThank you for your suggestions. We will revise Section 3.4 to make this step-by-step process more explicit.\n\n\n**W3.** It is not clear how FCI was optimized within an LLM framework, since it's a constraint-based algorithm.\n\n**RW3.** Thank you. There may be some misunderstanding. Actually, we do not optimize FCI within an LLM framework. Our method, MLLM-CD, is a three-stage process:\n\n1) **Factor Discovery (LLM-based)**: We use MLLMs to transform unstructured multimodal data into a structured dataset with causal factors and their values (as examples in Table 3.1, and an illustrative example in Figure 23, Page 27).\n2) **Causal Structure Discovery (Statistical)**: We then apply a **standard statistical** causal discovery algorithm like FCI to learn causal structures from the extracted structured data.\n3) **Counterfactual Reasoning (LLM-based)**: We use MLLM’s reasoning ability to generate plausible counterfactual samples for those factors with ambiguity to iteratively refine the discovery process.\n\nAs stated in Appendix E.3.1 (Page 16, Line 292), we use the FCI implementation from the causal-learn library. The LLM does not interact with the internal workings of FCI to maintain the **statistical rigor in causal discovery**. This hybrid paradigm is also used and verified effective in relevant work [1]. We will clarify this in the revised paper.\n\n**Table 3.1** Examples of extracted structured data.\n|nutrition|color|defects|aroma|taste|juciness|sentimen|recmd|score|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|-1|1|1|1|-1|1|1|1|3|\n|-1|0|-1|-1|-1|1|-1|-1|-5|\n|...|...|...|...|...|...|...|...|...|\n\n\n**W4**. There are some concerns regarding the theory - please see questions below.\n\n**RW4.** Thanks. We have provided the point-by-point responses in **RQ1** and **RQ2** to answer the questions regarding theories. Hope they address your concerns.\n\n\n**W5.** The empirical results do not include ground truth graphs with latent variables (when in the space of factors), yet the authors claim that their method can successfully work in this setting.\n\n**RW5.** Thank you for your comments. We did not claim addressing “latent variables” as a contribution of our work. We mean those algorithms robust to latent variables are relevant to addressing this setting with latent variables. \n\nWe only mentioned “latent variables” in Lines 183-184: “*Given the potential presence of unobserved confounders in real-world scenarios, algorithms robust to latent variables, such as FCI algorithm [29] or its variants (e.g., RFCI [64]), are particularly relevant.*” Here, “latent variables” refer to unobserved confounders, which make causal structure discovery more challenging. Since FCI is known to be more robust to confounders in real-world applications [2, 3], we choose it as the causal structure discovery algorithm in our framework. Thus, the ability to handle latent variables is a merit of the FCI algorithm itself and is beyond our contribution. We will clarify this in the paper and explore this complex setting in future work.\n\n\n**Q1.** Why are the optimal graphs in proposition 3.2 different?\n\n**RQ1.** The optimal graphs $G^\\*\\_{V\\_{CFD}}$ and $G^\\*\\_{V\\_{direct}}$ in proposition 3.2 differ from the notation $G^*$ (Line 236) because, given two subsets of factors discovered by two methods, the performance is also compared based on the corresponding optimal subgraphs induced by these factor subsets (as defined in Lines 242-243).\n\n\n**Q2.** In the proof of 3.2, how do authors justify making Assumption D.2? What does it mean to use a direct approach to obtain the features -- could the authors be more specific about these techniques?\n\n**RQ2.** The **\"direct approach\"**, as you may notice, is defined in Lines 74-75, Page 6. It refers to a non-contrastive baseline where an MLLM is directly prompted to identify causal factors from data samples, e.g., by asking \"*what are the high-level factors that contribute to the apple scores*\", similar to the baseline method **COAT [1]**. We will provide the direct prompt in Appendix E.2 for clarity.\n\n**Justification for Assumption D.2:** We would like to justify making Assumption D.2 from the following two aspects:\n\n1) **Mechanistic Design:** Our contrastive factor discovery (CFD) is superior to direct prompting as it offers critical guidance to uncover factors hidden in multimodal unstructured data. CFD forces the MLLM to focus on salient differences in cross-modal and within-modal pairs of samples, guiding it to identify genuine causal factors rather than superficial features. The direct approach, such as the one used in COAT, lacks this guidance and thus usually fails to discover comprehensive factors from multimodal data, as seen in Figure 2 (Page 2).\n2) **Empirical Evidence:** Our experimental results provide strong evidence for this assumption: **Tables 1&2** (Page 7, 9) show that MLLM-CD (using CFD) consistently achieves significantly higher **Node Recall (NR)** and **Node F1 (NF)** for factor discovery compared to the direct approach of COAT. A similar observation can also be found in the ablation study in Table 3 (Page 9), where the NF metric significantly decreases by 36% on the Lung dataset without CFD (comparing the rows of *w/o CFD* and *MLLM-CD*).\n\nAdditionally, to further enhance the empirical justification, we now provide a direct comparison between CFD and direct prompting on the performance of the factor proposal. Key results are in the following Table 2.1. As we can see, CFD consistently outperforms direct prompting in terms of correctness and completeness, and the significance test results (p<0.05) indicate that this large improvement is statistically meaningful. Thus, Assumption D.2 can be empirically justified.\n\n**Table 2.1** Comparison between CFD and direct prompting on the factor discovery performance.\n\n|      |         | NP↑         | NR↑         | NF↑         |\n| ---- | ------- | ---------- | ---------- | ---------- |\n| MAG  | Direct  | 0.6945     | 0.4074     | 0.5094     |\n|      | CFD     | **0.8796** | **0.8148** | **0.8453** |\n|      | p-value | 0.0129     | 0.0041     | 0.0023     |\n| Lung | Direct  | 0.6667     | 0.5333     | 0.5926     |\n|      | CFD     | **0.8889** | **1.0000** | **0.9394** |\n|      | p-value | 0.0471     | 0.0099     | 0.0165     |\n\n\n**Q3.** How do the authors solve FCI within a continuous optimization framework? Furthermore, they claim that their algorithm can solve the latent variable problem but the ground truth graph does not include any latent variables. Could the authors elaborate more on that?\n\n**RQ3.** Thank you. As we explained in **RW3**, we apply the standard FCI algorithm to the extracted factors and structured data to compute causal structures. The LLM does not interact with the internal workings of FCI to maintain the **statistical rigor in causal discovery**.\n\nIn addition, as clarified in **RW5**, solving latent variables is not a main focus or contribution of our work. The term “latent variables” appears only in Lines 183-184 to introduce an inherent strength of FCI, not to position it as the focus of this work. We will clarify this point in Section 3.3. Thanks to your comments, we will also consider exploring more complex and challenging scenarios involving latent variables in future work.\n\n\n**References:**\n\n[1] C Liu, et al. Discovery of the hidden world with large language models. NeurIPS, 2024.\n\n[2] P Spirtes, C N Glymour, R Scheines. Causation, prediction, and search. MIT Press, 2000.\n\n[3] J M Ogarrio, P Spirtes, J Ramsey. A hybrid causal search algorithm for latent variable models. Conference on Probabilistic Graphical Models. 2016."}, {"review_id": "L5ZInLEO1j", "reviewer": "Reviewer_yjmf", "summary": "The authors tackle the problem of inferring the underlying causal structure from multimodal unstructured data. They do this by first using multimodal LLMs (MLLM) to extract the causal factors within and across different modalities. This is done via extracting representations for a sample within a modality and computing the cosine similarity within and across modalities. The samples with high misalignment (high cosine distance) are fed along with prompts as contrastive pairs to extract the factors causing these differences, which are then consolidated. A causal structure is then inferred from these factors via a causal discovery algorithm. To improve the causal discovery, counterfactual data is generated from the MLLM for uncertain causal relationships using the internal knowledge of the MLLM. These are checked for semantic plausibility and causal consistency and then appended to the original dataset. The method is tested on two synthetic datasets.", "questions": "*Questions regarding counterfactual reasoning module:*\n- Does it actually make sense to augment the \"counterfactual\" sample back into the dataset? For example if the true causal relationship is $X \\to Y$, and the causal discovery algorithm returns $X - Y$ then an intervention on $Y$ (which is what you are doing by setting $V_a=v'_{ka}$ (L207)) would make $X$ and $Y$ independent. Putting this back into the sample doesn't actually help learn the relationship $X \\to Y$. In fact, it will make the causal discovery (which assumes the whole dataset has come from the same distribution) worse.\n- The semantic plausibility does not make sense to me. By definition, a counterfactual sample is from a different distribution and hence the embedding *should* be different from the observational sample. It seems non-trivial to me to quantify how different it should be. For $X \\to Y$, intervening on $X$ might not create too different of a sample, but for $X \\to Z_1 \\to Z_2 \\to \\cdots \\to Z_D \\to Y$ intervening on $X$ may create a drastically different sample. Furthermore, is the inequality in equation 4 meant to be less than $\\tau$?\n- For the causal consistency why isn't equation 6 an equality with $0$? A change should not affect the non-descendents at all. How does this algorithm handle inconsistencies with the learnt graph and the \"counterfactual sample\" generated by the LLM? \n- What if the MLLM does not have information about the query asked as it does not have enough information about a concept? Won't it still generate data however useless it may be?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Although the idea of extracting causal variables from unstructured data using LLMs is not new, using multimodal inputs is definitely an interesting idea.\n\n**Weaknesses:**\n- The counterfactual reasoning module is not rigorous enough and I have a few questions regarding the validity of this approach (see questions). It's not clear to me that this module actually provides the benefits that it claims to provide. A more rigorous testing of this can be done in simpler causal discovery settings where the variables are clearly defined. If it is tested rigorously in these settings, it would be reasonable to use it in the harder unstructured data setting. I'm not sure the semantic plausibility and causal consistency are rigorous enough checks on the validity of counterfactual samples. \n- The theory makes assumptions that make the conclusion of the theory obvious. In my opinion, the aim of the paper should be to show that these assumptions are true. This is lacking in the current work.\n    - Assumption D.2 states that the contrastive probing increases the F1 score of finding the true factors, which is then used to prove that contrastive probing improves structural accuracy. \n    - Similarly, assumption D.6 states that the MLLM can generate valid counterfactuals. If this were true, the MLLM would have internalised the correct structure and functional mechanism to determine the whole causal model. Why not just prompt the MLLM to return the correct model, or generate the correct counterfactual data and simply perform regression to find the causal structure.\n- Engagement with causal representation is missing, which is the task this paper is trying to solve. This area has conditions (e.g. interventions) under which the causal variables and structure is identifiable [1]. An obvious missing baseline is comparison against these methods in identifiable cases. This would show whether the method works when we know the conditions are there to identify the variables and structure. \n- The experiments are only carried out on two different toy datasets, for a method without formal guarantees more experiments with varying conditions would be useful. The different components were not tested thoroughly except for one ablation study. This raises questions like: Does the factor discovery reliably recover the correct factors? Does the counterfactual reasoning actually generate reasonable samples?\n\n[1] https://openreview.net/forum?id=lk2Qk5xjeu", "was_revised": true, "final_justification": "I still have reservations about this work. I believe the factor discovery with MLLMs is sound and interesting. I have some doubts about the theory and validity of the counterfactual samples: how they are generated (what exact distribution they are generated from), how they are combined with the observational data to feed into the CD algorithm, and what exactly the LLM does when it does not have domain knowledge.\n\nIf the authors are not performing interventions to compute “counterfactuals”, it will simply reproduce samples from the observational conditionals. Which should not help identify causal relations.", "rebuttal": "Thank you for the detailed and insightful feedback. We hope our response can sufficiently address your concerns.\n\n**W1** Questions regarding MCR and additional test settings.\n\n**RW1** Thank you. We have provided point-by-point responses in **RQ1-RQ3** to address your concerns about the MCR module and validation checks.\n\nFurthermore, we highlight that the validity of MCR has been extensively demonstrated through our experiments:\n\n1) We provide **direct, quantitative evidence** of MCR's benefits in ablation study (Table 3, Page 9). Removing MCR leads to a clear performance drop, e.g., AF drops from 0.87 to 0.38 on the Lung Cancer dataset.\n2) Case studies in Appendix E.10 (Figs 21-22, Page 26) provide **qualitative evidence** of causally meaningful and semantically plausible samples generated by MCR, that are crucial for refining the causal graph (Fig 17, Page 24).\n\nWe have tested MCR on **standard causal datasets** (Aisa, Child [2]) in Table 1.1. MCR consistently improves structural discovery in classic settings, supporting the rationale for applying it in our unstructured cases.\n\nTable 1.1 Causal discovery results.\n|||AP↑|AR↑|AF↑|SHD↓|\n|-|-|-|-|-|-|\n|Asia|FCI|0.77|0.50|0.58|9.67|\n||FCI+MCR|**0.89**|**0.58**|**0.67**|**8.33**|\n|Child|FCI| 0.91|0.68|0.78|24.0|\n||FCI+MCR|**0.93**|**0.74**|**0.82**|**22.0**|\n\n**W2.1** Justification of Assumption D.2\n\n**RW2.1** We justify this from two aspects:\n\n1) **Mechanistic Design:** CFD approach offers critical guidance beyond direct prompting by encouraging the MLLM to focus on salient differences in inter- and intra-modal sample pairs. This helps uncover genuine causal factors, rather than superficial features. In contrast, direct prompting usually fails to capture comprehensive factors, as in Fig 2.\n2) **Empirical Evidence:** Our experiments strongly support this assumption: **Tables 1&2** (Page 7, 9) show that MLLM-CD (using CFD) consistently achieves higher NR and NF than COAT (with direct prompting). Similarly, in the ablation study (Table 3, Page 9), removing CFD leads to a 36% drop in NF on the Lung dataset.\n\nTo further support this, we include a direct comparison between CFD and direct prompting in Table 2.1. CFD consistently outperforms direct prompting in both correctness and completeness, and a significance test (p<0.05) confirms the improvement is statistically meaningful.\n\nTable 2.1 Comparison between CFD and direct prompting.\n|||NP↑|NR↑|NF↑|\n|-|-|-|-|-|\n|MAG|Direct|0.6945|0.4074|0.5094|\n||CFD|**0.8796**|**0.8148**|**0.8453**|\n||p-value|0.0129|0.0041|0.0023|\n|Lung|Direct|0.6667|0.5333|0.5926|\n||CFD|**0.8889**|**1.0000**|**0.9394**|\n||p-value|0.0471|0.0099|0.0165|\n\n**W2.2** Justification of Assumption D.6.\n\n**RW2.2** We justify Assumption D.6 as follows:\n\n1) Assumption D.6 does **not** claim that MLLM provides a perfect causal model. Instead, it makes a weaker and more realistic claim: there exists a **non-zero probability** $p_{MCR}>0$ (Line 146, Page 7) of generating a validated counterfactual. Our method uses these insights while being robust to potential failures through validation checks.\n2) **Qualitative evidence** supports this: as shown in Figs 21&22 (Page 26), MLLM-generated counterfactuals often reflect correct causal dependencies. For example, lowering “nutrition” also affects “recmd”. Fig 17 (Page 24) further shows a significant improvement by MCR, confirming that $p_{MCR}>0$ is **a practical assumption**.\n3) **Quantitative support** comes from ablation studies (Tables 7&8). It directly measures the significant benefit of the MCR module (e.g., AF drops from 0.87 to 0.38 when MCR is removed on Lung dataset).\n4) While MLLMs can aid simple causal tasks [1], they still require guidance with observational data and validation checks to reduce hallucination and inconsistency, as evidenced by baseline results like META in Tables 1&2 (Pages 7&9). Thus, we cannot directly prompt it for the graph or data.\n\n**W3** Engagement with causal representation\n\n**RW3** The task setting in this work is quite different from those used in standard causal representation learning (CRL):\n\n1) Standard CRL methods usually rely on data from multiple environments (e.g., observational and interventional) as input. Our work addresses the more common case where only unstructured and observational data is available.\n2) CRL seeks identifiability by recovering a latent vector z mathematically equivalent to true causal variables, which is typically in **latent space and lacks labels** [3]. In contrast, MLLM-CD directly discovers human-interpretable causal factors (e.g., “color”, “lesion”) and then learning the causal graph over these extracted factors.\n\nThus, it is not directly comparable with CRL in our tasks. We will introduce representative CRL methods and discuss their connections in Section 2.\n\n**W4** Experiment scales and questions on modules\n\n**RW4** Discovering causality from multimodal unstructured data has significant applications in areas like scientific discovery and healthcare [4]. However, no public benchmark datasets exist for large-scale evaluation. \n\nOur work builds on the pioneering study [5]. Following their practices, we carefully constructed two new multimodal datasets, one synthetic (Multimodal Apple Gastronome) and one real-world (Lung Cancer), to validate our method. While these datasets are not large-scale, they are the first of their kind and represent a key step in benchmarking this new capability.\n\nWithin the context of these available benchmarks, we extensively evaluated our framework:\n\n1) **Factor Discovery:** Tables 1&2 (Pages 7, 9) show that MLLM-CD consistently outperforms baselines in NP, NR, and NF, confirming reliable recovery of causal factors.\n\n2) **Rationale of Counterfactuals:** The benefit of counterfactual reasoning is proven by the significant performance gain in ablation study (Tables 7&8, Page 22). Furthermore, case studies in Appendix E.10 (Figs 21-22, Page 26) offer qualitative evidence of how reasonable samples are generated.\n\n3) **Thorough Testing:** We provide a comprehensive analysis of the components, including:\n\n   * **Four MLLM backbones** (Tables 5&6, Pages 17, 20).\n   * **Different causal discovery algorithms** (Table 9, Page 23).\n   * **Parameter analysis** in Appendix E.8 (Page 22).\n   * Additional ablation studies on sampling strategies in **RW1.1** to reviewer ynPz.\n\nThis comprehensive evaluation highlights the robustness and effectiveness of our method.\n\nWe acknowledge the limitation of datasets (Appendix B.1) and are committed to contributing larger benchmarks in future work. We believe this will be a crucial step for the community, and we thank the reviewer for highlighting this direction.\n\n**Q1** Rationale of MCR\n\n**RQ1** Thanks. There may be some misunderstanding, as **MCR fundamentally differs from a statistical hard intervention** (e.g., do-calculus). We do not simply intervene on Y to make X and Y independent. Instead, MCR uses the MLLM’s reasoning and world knowledge to generate a plausible, causally coherent new sample.\n\nGiven $G^{(t)}$ and a counterfactual scenario where $V_a$ changes, the MLLM infers how other factors would adjust (Lines 197-198). For example, if we change Y, it evaluates whether X should also change, preserving causal coherence and reinforcing the true direction X->Y.\n\nThus, rather than harming discovery, we are **implicitly encoding the MLLM’s world knowledge into data**, creating samples that complement observational data and help resolve structural ambiguities.\n\n**Q2** Rationale of semantic plausibility check\n\n**RQ2** We clarify that:\n1) The motivation of this check is to **against hallucination**. While counterfactuals could differ from the original, their **semantic** embeddings are expected to remain similar as: 1. MLLM is prompted to perform a **minimal revision** (Line 208) to ensure high content overlap; 2. The core entity and context remain the same.\n2) Equation 4 is **correct**. $sim()$ is a similarity metric to filter out unrelated or nonsensical generations with low similarity. A \"less than\" inequality would do the opposite.\n\nThis ensures the generated sample is a *coherent variation*, not a random output.\n\n**Q3** Threshold for causal consistency check\n\n**RQ3** The main reason for a non-zero threshold is that the current graph G(t) used for the check **is not ground truth**.\n\n1) G(t), derived from observational data, may be partially incorrect. Thus, this threshold acts as a **trade-off** between injecting MLLM’s knowledge and maintaining consistency with a potentially improvable graph.\n2) Empirically, when observational data is limited or noisy, a strict zero-equality check can be overly punitive (Appendix E.8, Page 25). With the proper threshold, the cumulative effect of major consistent counterfactuals outweighs minor imperfect ones.\n\nWe will clarify this in the final version.\n\n**Q4** What if MLLM has limited information\n\n**RQ4** This is a more challenging case, requiring cross-domain or domain-specific knowledge, which is currently beyond the main focus of this work. However, it can be addressed with easy extensions:\n\n1) Set stricter validation checks to filter out noisy samples, as they are likely to violate causal constraints.\n2) Integrate with strategies like RAG or MoE [6] to enhance fidelity in domain-specific cases.\n\nWe will add a discussion of this potential extension to the paper.\n\n**References**\n\n[1] A Vashishtha, et al. Causal order: The key to leveraging imperfect experts in causal inference. ICLR, 2025\n\n[2] T Ban, et al. LLM-driven causal discovery via harmonized prior. IEEE TKDE, 2025\n\n[3] B Schölkopf, et al. Toward causal representation learning. IEEE, 2021\n\n[4] R Tu, et al. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. NeurIPS, 2019\n\n[5] C Liu, et al. Discovery of the hidden world with large language models. NeurIPS, 2024\n\n[6] C Shi, et al. Unchosen experts can contribute too: Unleashing moe models power by self-contrast. NeurIPS, 2024"}, {"review_id": "wtwuEYZF70", "reviewer": "Reviewer_EBFe", "summary": "This paper proposes MLLM-CD, a framework for causal discovery from multimodal data using MLLMs. The approach consists of three key components: (1) Contrastive Factor Discovery (CFD), which identifies candidate causal variables by comparing semantically similar multimodal instances with different outcomes; (2) Causal Structure Discovery, which uses these factors to infer the causal graph; and (3) Multimodal Counterfactual Reasoning (MCR), which generates and evaluates counterfactual examples to refine the graph structure. The method is evaluated on a synthetic multimodal dataset (MAG) and a real-world lung cancer dataset, using standard metrics for node and edge accuracy, as well as structural Hamming distance. Results are compared against existing baselines in multimodal and causal discovery.", "questions": "Given my limited background in causal inference, I would welcome clarification on the following points:\n\n1. If the model generates incorrect or implausible counterfactuals, how might that affect the final causal graph? Could errors at the counterfactual stage mislead the entire structure discovery process?\n\n2. Even with the use of MLLMs, the model can only infer causes based on what is explicitly represented in the text and images. How does the method handle potential unobserved confounders or latent factors that are not captured in the input modalities?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The overall formulation of the framework is well-structured and logically presented, with a clear separation of its core components. The integration of MLLMs into causal discovery is thoughtfully executed, where its use during counterfactual reasoning for refining causal graphs seems novel and well-motivated.\n\n2. The paper compares the proposed method against a range of baselines, including both traditional and recent MLLM-based approaches.\n\n## Weaknesses\n\n1. The paper evaluates its approach on only two datasets, especially with a relatively small number of causal factors for the Lung Cancer dataset. This could raise questions about the generalizability of the results, especially in more complex or large-scale real-world scenarios.\n\n2. All variables used in the experiments are binary, which simplifies causal discovery but limits the applicability of the method. It remains unclear whether the approach can handle continuous or ordinal variables, or how well it would perform in settings with more diverse data types.", "was_revised": true, "final_justification": "The integration of MLLMs into causal discovery is a timely and promising direction. The concerns I raised regarding the limited evaluation on small, binary-variable datasets and the absence of benchmarks were acknowledged in detail. While these limitations remain unresolved, I appreciate the authors’ justification and their commitment to future contributions in this area. Similarly, while the handling of unobserved confounders remains a broader challenge, the use of the FCI algorithm and contrastive discovery mechanisms provides a reasonable safeguard within scope. The additional explanation of the verification process for counterfactuals was helpful, though I believe further refinement in this area could strengthen the robustness of the method. Given my limited background in causal inference, I maintain a confidence score of 3, in case there are methodological limitations I may have overlooked.", "rebuttal": "We are grateful for the reviewer’s positive feedback and for the thoughtful questions, which help clarify important aspects of our method’s robustness and limitations.\n\n**W1.** The paper evaluates its approach on only two datasets, especially with a relatively small number of causal factors for the Lung Cancer dataset. This could raise questions about the generalizability of the results, especially in more complex or large-scale real-world scenarios.\n\n**RW1.** Thank you. Discovering causality from multimodal unstructured data has significant applications in areas like scientific discovery and healthcare [1]. However, there is currently no public benchmark that can be used for large-scale evaluation. \n\nOur work builds on the pioneering research [2]. Following their experimental practices, we have taken the next step by carefully constructing two new multimodal datasets, one synthetic and one real-world, to validate our proposed method. While these datasets are not large-scale, they are, to our knowledge, the first of their kind and represent a crucial step in benchmarking this new capability.\n\nTo be prepared for future large-scale scenarios, we have analyzed the **complexity and scalability** of our method in Section E.9 (Page 24). To handle large-scale datasets, the bottleneck is in contrastive sampling and causal discovery algorithms. As we discussed in Lines 435-437, possible solutions include leveraging efficient sampling techniques [3, 4] and causal discovery algorithms, such as ENCO [5] and ETCD [6].\n\nWe acknowledged the limitation of lacking large-scale benchmarks (Appendix B.1, Page 4). We are committed to pushing this important research direction forward and plan to contribute to larger, more diverse public benchmark dataset in future work. We believe this will be a crucial step for the community, and we thank the reviewer for highlighting its importance.\n\n\n**W2.** All variables used in the experiments are binary, which simplifies causal discovery but limits the applicability of the method. It remains unclear whether the approach can handle continuous or ordinal variables, or how well it would perform in settings with more diverse data types.\n\n**RW2.** Thank you for raising this concern. We used binary variables as a common and simplifying starting point [2, 7], which is practical for many real-world factors (e.g., presence/absence of a lesion, smoker/non-smoker). \n\nHowever, our framework is **not inherently limited to binary variables**. The factor annotation prompt (Appendix E.6, page 15) can be easily adapted to request continuous or categorical values. For example, the prompt can be changed to: \n\n>“*For each sample, assign a continuous score within [0, 1] to each factor based on the criteria above*”. \n\nThen, the causal discovery algorithm can be adjusted to accommodate the new data format. For instance, a constraint-based method would need a conditional independence test suitable for continuous or categorical data. We will add this clarification to Sections 3.2 and 3.3 to highlight the potential for extension.\n\n\n**Q1.** If the model generates incorrect or implausible counterfactuals, how might that affect the final causal graph? Could errors at the counterfactual stage mislead the entire structure discovery process?\n\n**RQ1.** Thank you. We have a crucial safeguard in place to prevent this. As detailed in Section 3.4 (Lines 212-219), every counterfactual sample generated by the MLLM undergoes two validation checks before it is added to our dataset:\n\n1) **Semantic Plausibility (Eq. 4):** We ensure the generated sample is semantically coherent and not a wild hallucination.\n2) **Causal Consistency (Eq. 6):** We check if the changes in the counterfactual are consistent with the current causal graph G(t). For instance, an intervention on a node should not affect its non-descendants.\n\nOnly counterfactuals that pass both checks are used for refinement. This filtering mechanism is designed specifically to prevent errors from misleading the structure discovery process. However, the threshold parameters need to be tuned empirically, as overly loose validation can introduce noise and harm discovery performance (Appendix E.8, Figures 18&19, Pages 24&25).\n\n\n**Q2.** Even with the use of MLLMs, the model can only infer causes based on what is explicitly represented in the text and images. How does the method handle potential unobserved confounders or latent factors that are not captured in the input modalities?\n\n**RQ2.** This is a fundamental and common challenge in causal discovery [8, 9], though it is currently beyond the main focus of this work. However, our framework alleviates this problem in two ways:\n\n1) **Factor Discovery:** Our Contrastive Factor Discovery (CFD) module is designed to uncover a more comprehensive set of causal factors from the data, reducing the chances of omitting important variables that could act as confounders.\n2) **Structure Discovery:** We use the FCI algorithm, which is designed to maintain robust performance in cases with unobserved confounders (Lines 183-184, Page 5). FCI outputs a graph that can explicitly represent uncertainty about latent confounding through bidirected edges. This is a standard and robust approach in the causal discovery literature for dealing with this problem.\n\nIn the future, we will explore more challenging and complex scenarios to develop mechanisms with MLLMs that explicitly identify potential confounders to help address this problem.\n\n\n**References:**\n\n[1] R Tu, et al. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. NeurIPS, 2019.\n\n[2] C Liu, et al. Discovery of the hidden world with large language models. NeurIPS, 2024.\n\n[3] L Xu, J Lian, et al. Negative sampling for contrastive representation learning: A review. CoRR, 2022.\n\n[4] T Chen, S Kornblith, et al. A simple framework for contrastive learning of visual representations. ICML, 2020.\n\n[5] P Lippe, T Cohen, E Gavves. Efficient neural causal discovery without acyclicity constraints. ICLR, 2022.\n\n[6] X Li, T Liu. Efficient and trustworthy causal discovery with latent variables and complex relations. ICLR, 2025.\n\n[7] A Chen, Q Zhou. Causal Discovery on Dependent Binary Data. AISTATS, 2025.\n\n[8] P Spirtes, C N Glymour, R Scheines. Causation, prediction, and search. MIT Press, 2000.\n\n[9] J M Ogarrio, P Spirtes, J Ramsey. A hybrid causal search algorithm for latent variable models. In the Conference on Probabilistic Graphical Models. 2016: 368-379."}, {"review_id": "RuRvFMYQ0y", "reviewer": "Reviewer_ynPz", "summary": "The paper introduces **MLLM-CD**, a framework that leverages multimodal large language models (MLLMs) to perform causal discovery directly from unstructured, multimodal data. Key ideas include:\n\n1. **Contrastive Factor Discovery (CFD)** selects intra- and inter-modal contrastive pair samples and prompts an MLLM to propose candidate factors. The MLLM then consolidates and annotates the dataset into a structured dataset based on the proposed factor set.\n2. Structured data is fed into a causal discovery algorithm, such as FCI, to produce a causal graph \n3. **Multimodal Counterfactual Reasoning (MCR)** generates and validates new counterfactual samples (textual and visual) based on the predicted causal graph, using an MLLM to generate new image and text data. The new data is then added to the dataset for the next round of refinement.\n\nExperiments on a synthetic Multimodal Apple Gastronome (MAG) dataset and a real-world Lung Cancer dataset, evaluated across four MLLMs (GPT-4o, Gemini 2.0, LLaMA-4, Grok-2v), show that MLLM-CD outperforms prior methods in both factor identification and structure recovery.", "questions": "Questions:\n\n- Have you experimented with other, simpler strategies for sampling data for factor proposals?\n- Could you provide evidence to support Assumption D.2?\n- Which CLIP checkpoint is being used?\n\nI will increase my score if more evidence supporting CFD is provided.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- This work addresses the challenge of causal discovery in unstructured multimodal data, a topic that is both important and timely.\n- Experiments demonstrate strong performance in both factor identification and structure recovery.\n- This MCR module is well-justified both theoretically and empirically.\n- Writing is generally clear and well-structured.\n\nWeakness\n\n- My main concern is the CFD module. Introducing an extra MLLM (e.g., CLIP) adds complexity, and its necessity is not well justified:\n    - Ablation study (Sec. 4.1) is limited. CFD is compared only with random sampling. Other simple baselines should also be considered, such as:\n        - Random sampling from each category of the target variable;\n        - Pairing samples with the largest difference in the target variable;\n        - Ablate the Intra-modal and Inter-modal independently.\n    - Theoretical justification is misleading. Assumption D.2 claims that $V_{CFD}$ is at least as effective, and potentially superior, in terms of a combined measurement of F1-score. This assumption effectively leads to a circular reasoning fallacy, as it presupposes that the factor sets proposed by CFD is better. This assumption needs to be justify empirically.", "was_revised": true, "final_justification": "The additional experiments have empirically addressed my concerns about the validity of the assumptions. Although I still find the theoretical section of the paper somewhat odd and unnecessary, the proposed method is interesting and demonstrates strong empirical performance. Therefore, I am willing to give this paper an accept rating.", "rebuttal": "We thank the reviewer for the valuable feedback and helpful suggestions for improving our paper.\n\n**W1.** My main concern is the CFD module. Introducing an extra MLLM (e.g., CLIP) adds complexity, and its necessity is not well justified:\n\n**W1.1.** Ablation study (Sec. 4.1) is limited. CFD is compared only with random sampling. Other simple baselines should also be considered, such as:\n\n* Random sampling from each category of the target variable;\n\n* Pairing samples with the largest difference in the target variable;\n\n* Ablate the Intra-modal and Inter-modal independently.\n\n**RW1.1.** Thank you for your constructive suggestions. We have conducted additional ablation studies to validate the effectiveness of different sampling strategies in terms of their impact on factor discovery (NP, NR, and NF) and overall discovery accuracy (ESHD). The key quantitative results are shown in Tables 1.1 and 1.2, and we have the following observations:\n\n* Compared with random sampling, pairing with the largest difference (simple pair) provides improved ESHD. \n* Incorporating finer-grained signals by finding semantic-level contrastive pairs within each modality, intra-modal further improves the completeness of factor discovery. However, it often overlooks the cross-modal interactions, leaving factors like “defects” or “size” undiscovered. \n* Inter-modal acts as an ideal complement to intra-modal, as it examines across modalities and can effectively identify factors missed by intra-modal. Using inter-modal only has lower quantitative results, as it is prompted to emphasize factors tied to cross-modal interactions, as a complement to intra-modal, rather than aiming to discover all factors. \n* By combining both strategies, our MLLM-CD achieves the best performance in terms of NR, NF, and ESHD. \n\nThe full quantitative results and some qualitative comparisons will be provided in the revised manuscript.\n\n**Table 1.1** Ablation study of different sampling strategies on the MAG dataset.\n\n|                | NP↑        | NR↑       | NF↑       | ESHD↓      |\n| -------------- | --------- | -------- | -------- | --------- |\n| Random         | **0.94** | 0.60     | 0.73     | 15.00     |\n| Simple Pair    | 0.79      | 0.70     | 0.75     | 13.67     |\n| Intra-modal | 0.88      | 0.78     | 0.82     | 13.67     |\n| Inter-modal | 0.67      | 0.59     | 0.63     | 15.33     |\n| MLLM-CD        | 0.89      | **0.89** | **0.89** | **13.33** |\n\n\n**Table 1.2** Ablation study of different sampling strategies on the Lung dataset.\n\n|                | NP↑       | NR↑       | NF↑      | ESHD↓     |\n| -------------- | -------- | -------- | -------- | -------- |\n| Random         | 0.65     | 0.60     | 0.62     | 8.00     |\n| Simple Pair    | 0.50     | 0.60     | 0.55     | 6.33     |\n| Intra-modal | 0.80     | 0.80     | 0.80     | 6.00     |\n| Inter-modal | 0.53     | 0.53     | 0.53     | 7.00     |\n| MLLM-CD        | **0.94** | **1.00** | **0.97** | **4.67** |\n\n\n**W1.2.** Theoretical justification is misleading. Assumption D.2 claims that V_{CFD} is at least as effective, and potentially superior, in terms of a combined measurement of F1-score. This assumption effectively leads to a circular reasoning fallacy, as it presupposes that the factor sets proposed by CFD is better. This assumption needs to be justify empirically.\n\n**RW1.2.** Thank you. Apart from existing empirical evidence in the ablation study (Table 3, Page 9), e.g., NF significantly decreases by 36% on the Lung dataset without CFD, we have provided additional empirical comparison between CFD and direct prompting to support this assumption. Key results are shown in Table 1.3. As we can see, CFD consistently outperforms direct prompting in terms of both correctness and completeness on two datasets, and the significance test results (p<0.05) indicate that this large improvement is statistically meaningful. This empirical evidence will be included in the final version of our paper.\n\n**Table 1.3** Comparison between CFD and direct prompting on the factor discovery performance. \n\n|      |         | NP↑        | NR↑         | NF↑         |\n| ---- | ------- | ---------- | ---------- | ---------- |\n| MAG  | Direct  | 0.6945     | 0.4074     | 0.5094     |\n|      | CFD     | **0.8796** | **0.8148** | **0.8453** |\n|      | p-value | 0.0129     | 0.0041     | 0.0023     |\n| Lung | Direct  | 0.6667     | 0.5333     | 0.5926     |\n|      | CFD     | **0.8889** | **1.0000** | **0.9394** |\n|      | p-value | 0.0471     | 0.0099     | 0.0165     |\n\n\n\n**Q1.** Have you experimented with other, simpler strategies for sampling data for factor proposals?\n\n**RQ1.** Thank you. We have now provided the comparison of different sampling strategies for factor proposal in **RW1.1**. In summary, the intra-modal strategy offers finer-grained contrastive guidance and leads to superior discovery performance compared to \"random\" and \"simple pair\" methods. The inter-modal strategy, as a complement, can effectively identify factors missed by the Intra-modal method. Combining both strategies achieves the best overall performance.\n\n\n**Q2.** Could you provide evidence to support Assumption D.2?\n\n**RQ2.** Thank you. We have now directly compared the performance of CFD and direct prompting for factor proposal in **RW1.2**. As shown, CFD consistently outperforms direct prompting with statistically meaningful improvements, providing additional empirical evidence in support of Assumption D.2.\n\n\n**Q3.** Which CLIP checkpoint is being used?\n\n**RQ3.** Many thanks. We used the ViT-B/32 checkpoint from OpenAI’s official CLIP release. We will add this detail to Section E.3.1 (Implementation Details, Page 16) in the final version of the paper."}]

Author rebuttal:

Meta-review: This paper presents MLLM-CD, a framework for causal discovery from unstructured multimodal data, combining Contrastive Factor Discovery, causal structure learning, and Multimodal Counterfactual Reasoning. Reviewers noted that the paper is well-motivated, technically coherent, and demonstrates strong empirical performance on synthetic and real-world datasets. The use of MLLMs for factor discovery and counterfactual generation is a novel and appealing aspect. While some concerns remain regarding theoretical assumptions, limited ablations, and dataset scale, these do not outweigh the paper’s strengths. Overall, the work makes a meaningful contribution to multimodal causal discovery.
Decision: Accept (poster)