Causal Concept Graph Models: Beyond Causal Opacity in Deep Learning
Authors: Gabriele Dominici
Pietro Barbiero
Mateo Espinosa Zarlenga
Alberto Termine
Martin Gjoreski
Giuseppe Marra
Marc Langheinrich
Venue: ICLR 2025 Poster (2025)
Keywords: Concept Based Models
Concept Bottleneck Models
Explainable AI
Interpretability
Primary area: interpretability and explainable AI
TL;DR: Causal Concept Graph Models are a class of interpretable models whose decision-making process is causally transparent by design.

Abstract: Causal opacity denotes the difficulty in understanding the "hidden" causal structure underlying the decisions of deep neural network (DNN) models. This leads to the inability to rely on and verify state-of-the-art DNN-based systems, especially in high-stakes scenarios. For this reason, circumventing causal opacity in DNNs represents a key open challenge at the intersection of deep learning, interpretability, and causality. This work addresses this gap by introducing Causal Concept Graph Models (Causal CGMs), a class of interpretable models whose decision-making process is causally transparent by design. Our experiments show that Causal CGMs can: (i) match the generalisation performance of causally opaque models, (ii) enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections but also the reliability of the explanations provided for specific instances, and (iii) support the analysis of interventional and counterfactual scenarios, thereby improving the model's causal interpretability and supporting the effective verification of its reliability and fairness.

Published as a conference paper at ICLR 2025
CAUSAL CONCEPT GRAPH MODELS: BEYOND CAUSAL OPACITY IN DEEP LEARNING
Gabriele Dominici
Pietro Barbiero"
Università della Svizzera italiana gabriele. domi nici @usi ch
IBM Research
pietro. hat.reobbbbieeroiibmm.
Mateo Espinosa Zarlenga University of Cambridge me4660cam. cam..cccm... ac. uk
Alberto Termine
IDSIA
a.berto.
Martin Gjoreski Università della Svizzera italiana partin.gjoreste kusiiiiiiiie.. ch
Giuseppe Marra
KU Leuven giuseppe.marrakkuue warra@kuleuven.
Mare Langheinrich Università della Svizzera italiana marc. www.aaaennnnchuuu... .ch
ABSTRACT
Causal opacity denotes the difficulty in understanding the "hidden" causal structure underlying the decisions of deep neural network (DNN) models. This leads the inability to rely on and verify state-of-the-att DNN-based systems, especially in high-stakes scenarios. For this reason, circumventing causal opacity in DNNs represents key open challenge at the intersection of deep learning, interpretability, and causality. This work addresses this gap by introducing Causal Concept Graph Models (Causal CGMs), class of interpretable models whose decision-maker process is causally transparent by design. Our experiments show that Causal CGMs can: (i) match the generalisation performance of causally opaque models, (ii) enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections but also the reliability of the explanations provided for specific instances, and (iii) support the analysis of interventional and counterfactual scenarios, thereby improving the model's causal interpretability and supporting the effective verification of its reliability and fairness.
INTRODUCTION
Causal opacity refers to the difficulty in understanding a model's decision-making mechanisms (Termine & Primiero 2024). Like causal discovery, causal opacity, can be assessed through Pearl's framework of causality (Pearl 1995), categorizing an agent's causal understanding based on its ability to answer three types of questions: observational queries ('what is the relationship between the feature æ and the model's output?"),, interventional queries ("what happens if fix the feature
a value k:?"), and counterfactual queries ("what would the model's prediction had been feature had taken value k'' instead of k:?"). Unlike causal discovery, causal opacity concerns the decision-makee makinn process of DNN model. Whether DNN causally opaque (or transparent) depends whether users can answer interventional and counterfactual questions that concern the model's s inferential behaviour. this regard, notice that (i) the problem causal opacity is distinct from the problem of causal discovery, which involves detecting causal mechanisms in the real world,
"Equal contribution
Work conducted while employed Università della Svizzera italiana.

---

Published as a conference paper at ICLR 2025
Target
Target
Target
High-Levo Lvvd Concepity
Black
Raw Features
Raw Peatures
Rawi Features
(a) Standard DNNS
(h) Standard CBMs
Causal CGMs
(a) Standard DI. models are black boxes in the sense that the causal structure of their mapping from raw input features (e.g., pixels of an image) to the target remains opaque. (b) In Concept Bottleneck Models (CBM), high-level human-interpretable concepts are first extracted through an encoder g and then used to predict the target. Although CBMs are semantically transparent, the causal structure of the model's inference assumes straightforward causal structure where concepts are causally independent and are all direct causes of the target. (c) In Causal Concept Graph Models (Causal CGMs), both the concepts' semantics and the inference's 's causal structure are transparent.
Figure 1:
rather than understanding the causal structure of model's internal reasoning; and (ii) causal opacity (and the related form of causal understanding) neither implies nor requires discovering the "true" causal structure of the data-generating process. For example, model might effectively predict heart attacks based on the rule heart_att &- smoking + lung_cancer, even though the actual biological mechanism that smoking causes lung cancer, not the reverse. Further details on the distinction between causal opacity and causal discovery are discussed in App. A
Deep Learning (DL) models (Fig. Tap are causally opaque when applied to unstructured data like images, as interventions on low-level features, such as pixels, are not meaningful (Rudin, 2019 Ghorbani et al. 2 and therefore do not enable us to easily answer any of these sorts of questions. For example, asking "what happens if I fix the intensity of the pixel x; to k?" does little to reveal whether a DNN exploiting protected attributes like "gender" or "race" when classifying an image of person as "doctor". Yet, understanding when these sorts of high-level attributes are misused to make predictions is crucial for the proper verification and vetting of models (Kusner et al., 2017). particularly in high-impact real-world tasks (c.g., medical admission recommendations).
To address this issue, Concept Bottleneck Models (CBMs) (Koh et al. 2020) have been designed to support interventions high-level interpretable features known as "concepts" (Kim al. 2018). These concepts, whose semantics are aligned with units of information experts would use to solve the same task, allow for more intuitive interventions and counterfactual analysis (Wachter al. 2017 Abid et al. 2021). This is because CBMs shift from reasoning based on raw features to explicitly operating on interpretable variables such as "age", "income". "gender", or "BMI".
Although CBMs are semantically transparent (Kim et al. 2018 Facchini & Termine 2021 we argue they still rely on two unrealistic assumptions (visualised in Fig. lb). First, they assume that concepts are causally independent, an assumption that breaks in most real-world tasks. For instance, CBMs might treat "smoking" and other health conditions as independent predictors of an insurance premium, ignoring how intervening on smoking affects health. Second, and more importantly, CBMs assume concepts are the direct causes of model predictions. These assumptions oversimplify and constrain the model's inference structure, preventing the model from using more complex and effective concept dependencies (Beckers) 2022). As a result, the problem of causal opacity still represents a key open challenge at the intersection of DL, causality, and XAI.
Contributions In response to this challenge, we introduce Causal Concept Graph Models (Causal CGMs, see Figure Tc.. Causal CGMs new concept-basse architecture whose decision-makin process is causally transparent by design, avoiding the unrealistic assumption that concepts must be causally independent and direct causes of class prediction. The results of our experiments show that Causal CGMs can: (i) match the generalisation performance of state-of-the-art causally-opaque models, (ii) enable soman--n-the-loog corrections of mispredicted intermediate reasoning steps,
Herr by semantic transparent mean model that focuses on features that are meaningful and under- standable by human experts the model's application domain.
2

---

Published as a conference paper at ICLR 2025
boosting both downstream accuracy and explanation correctness, and (iii) support the analysis of both interventional and counterfactual questions, improving causal interpretability and enabling effective verification of reliability and fairness. The code related to this paper is publicly available
BACKGROUND
Causal Models causal model (CM) mathematical representation of the rules or laws that explain how different variables of given target-system causally influence each other. In computer science and statistics, Structural Causal Models (SCM) proposed pearr (2009) are a key framework for causal modelling, Formally, an SCM M is triplet (2, Fo)) where: U is set of exogenous variables representing latent factors with a causal influence on the modelled target-system; V is set of endogenous variables representing observable and measurable variables; Fo is set of functions (or structural equations) parametrised by 0 describing the causal mechanisms, which determine the values of each endogenous variable vs by computing the conditional probability p(u:] (u; pa(n;); f), where uy E U/ and pa(o:) are, respectively, the set of exogenous and endogenous variables causally affecting 22 E V. Every SCM corresponds a directed acyclic graph (DAG) whose nodes represent variables and edges represent direct causal connections. Hard interventions in SCMs are modelled through the do-operator (Pearl 1995 2009), which modifies the model's structure by fixing an endogenous variable to value R. breaking its original causal dependencies. Formally, applying do(v; = x) to a model M results in a new model Me,=x which is identical to M except that the equation for is replaced with constant value removing all arrows into in the DAG.
Concept-based models Concept-based models (Kim et al. 2018 Chen et al. 2020 Yeh et al. 2020 Kim et al. 2023 Barbiero et al., 2023) Oikarinen et al. 2023) are interpretable architectures that explain predictions using high-level information units (i.e.. "concepts"), Most follow the Concept Bottleneck Model (CBM) (Koh et al. 2020) approach. Given sample's raw features E X C Rd (e.g., an image's 's pixels), a set of r concepts & E CCC. 13" (e.g.. "red", "round"), and a set of I class labels y E Y S {0, 13' (e.g., labels "apple" "tomato"), a CBM estimates the conditional distribution 11, ply pa(y,) = CI....,,, 0 f II, p(c, x; x:0g) where p(c: x; 8g) is a set of independent Bernoulli distributions. This formulation relies on two key CBMs' assumptions: i) concepts directly cause class predictions, and ii) concepts are conditionally independent of one another. Usually, concept-based models represent a concept C; using its predicted probability E 10, 1]. However, this may degrade task accuracy when concepts are incomplete (Mahinpei et al., 2021). To address this, Concept Embedding Models (CEMs) (Espinosa Zarlenga et al. 2022) use high-dimensional embeddings è IR3 to represent concepts alongside their truth degrees E [0, 1].
CAUSAL CONCEPT GRAPH MODELS
This work addresses causal opacity, the challenge of uncovering a DL model's hidden causal structure. Research question: To overcome this, we aim to design DL models where each prediction can be traced back to a chain of semantically meaningful causes. Challenged assumptions: In pursuing this goal, challenge two key assumptions of Concept Bottleneck Models (CBMs): i) humans know a priori which interpretable variables (concepts) are direct causes of a class label (and datasets embed this prior knowledge), and ii) all interpretable variables used for explanations (i.e., concepts) are conditionally independent. Solution: We introduce Causal Concept Graph Models (Causal CGMs), a class of models making class predictions through cause-effect chains of interpretable variables. We first introduce Causal CGMs, then show how their decision-making process is causally transparent and conclude by describing their learning process.
Problem:
3.1 BLUEPRINT
Dropping the assumptions of CBMs concepts are independent and direct causes of class labels requires a model able to capture all possible dependencies between interpretable variables
https:/ /i thub. com/ www.gaabiiieee domi ni /Causalallecc CGM
This naive formulation leads to a cyclic probabilistic model, making computation intractable due to the difficulty of handling cycles. Causal Concept Graph Models address this by ensuring efficient learning and producing simple explanations by design (see Sec. 3.3
3

---

Published as a conference paper at ICLR 2025
(i.e., any classification label). To model such dependencies, Concept Graph Models duplicate the layer of interpretable variables V by introducing an additional layer of identical copies v Using this additional layer, Causal CGMs make predictions of each v E V using as possible inputs all v/ E V for all # i. This inference structure enables CGMs to trace each prediction in y back to a set of interpretable variables in V. As a result, the set of copies V in Causal CGM play the role of "explaining variables", akin to a CBM's concepts, while the set of variables V play the role of "explained variables", akin to a CBM's tasks.
Definition 3.1 (Causal Concept Graph Model). Given an observed feature vector x, a set of & E N latent factors u, U each associated with a pair of identical high-level interpretable variables v; E V and of E V with 22 = v, Vi E (I,...,,) a Causal Concept Graph Model [ = EN, with
nodes N = {x) UuuVUV,
edges & = [xx uz) [(d,, 0,) u V", Uy E V, V,i
u, E EU} (us, E k vi) l (1,... u
. and mechanisms Fo s.
represents the joint conditional distribution:
endogenous predictor
copies predictor exogenous encoder
k
p(v1.... Uk, 01. .... U//5 11..... Us x; 0)
p(e:
payr u; 0,)
p(u') (.;0,) plu, x; 0z)
1f1
(I)
3.2 CAUSAL TRANSPARENCY IN CONCEPT GRAPH MODELS
Notice how the distribution p(u; pay: u;; 0y) can be associated with a structural causal model McGar (fu), V" [ex (fH) with causal mechanism fs where uú E li is an exogenous variable representing latent, uninterpretable information (e.g., noise), V; E is an endogenous variable representing interpretable, symbolic information, and fa U x V
+ V is a function describing the causal mechanism that determines the value of v/ given its parents pay (w;). Such dependencies captured by a graph G (V U V", (1r), Uy) o/ V. E V. o * u. }), representing all direct causal dependencies between endogenous variables. By modelling such dependencies, and considering that by Def. 3.1 the variables v; and of are identical copies of the same interpretable variable, Causal CGMs allow any endogenous variable v; to be expressed in terms of its parents in V. denoted as (u;), rather than in terms of its parents in V i.e. pap (U;), as formalised in the following theorem (proof in App.
Theorem 3.2. Given a Causal Concept Graph Model r = (N. EoF.. let the set of endogenous root nodes be defined roots(g') = {of E V #(V,, v.) Eg. and the set of children of root nodes as: ch(roots(
(v; V 1 vx) E Eg, roots((")) Then, for all nodes in the set V \ (roots(G') U ch( roots(g"))) i.e., for all nodes that are neither root nodes nor children of root nodes, the following holds: p(u; pap un 0,) = p(v. pay((s), un
The above theorem shows that CGM's bipartite graph g' between nodes V and V can efficiently represent arbitrary long cause-effect chains among interpretable variables (example in Fig. 2). While during training, CGMs exploit the efficient bipartite representation, at test time CGMs can be unfolded following the graph as described in Theorem 3.2 (Sec. describes how to learn sparse and acyelic dependencies to make the test time unfolding more efficient and interpretable; we provide more
"A detailed analysis of this transformation, including the "unfolding" of the original cyclic model, is provided in App. B and neural parametrization in App. D
4

---

Published as a conference paper at ICLR 2025
(out (+)) (hiyoutsgg))) (ss)
Figure 2: p(u U/2 0y) is equivalent to p(v1 | Ube (1;0y) as they both represent the same query on the same conditional probability table.
details in App. H.6 This property makes CGMs causally transparent by design, as they can make class predictions through cause-effect chains of endogenous variables, allowing humans trace each prediction back to its underlying causes within an interpretable parental subgraph. Notice how-unlike CBMs Causal CGMs do not require to know in advance cause (concept)- (task) relationships, nor require that all causes (concepts) are independent from each other, as cause-effect dependencies among interpretable variables are not given, but learnt during training.
Remark 3.3. Causal CGMs are designed to address the problem of causal opacity, not causal discovery. Causal opacity- the problem of understanding model's decision- nor requires causal discovery- generating mechanism. Indeed, the causal structure of a model's decision-making process does necessarily mirror the causal structure of the world (Boge 2022). We report in appendices extended discussions on CGM's relations with causal discovery (App.) and causal representation learning (App. K. and on CGM's identifiability (App.
process- neither implies problem of identifying the rules governing the world or the data-
3.3 TRAINING CAUSAL CGMS
Learning sparse, directed, and acyelic structures In order to construct simple explanations, we can make Causal CGMs' graphs sparse and acyclic. To this end, we parametrise the bipartite graph G with an adjacency matrix of learnable weights M E R & We initialise the weights myj E M based on the conditional entropy between labels and uy to account for asymmetric concept dependencies (for other initialisation strategies, see App. [F.. These weights are then fine-tuned through an end-to- end learning process within the Causal CGM framework following common causal priors, where causal graphs are assumed to be sparse, directed, and acyclic (forming Directed Acyclic Graph or DAG). We introduce a parameter y IR to eliminate less significant dependencies and loss function, as described by Yang et al. (2020), to enforce the sparsity and acyclicity of the causal graph, ensuring that the adjacency matrix A effectively represents a DAG:
(initialisation)
aij" =
=b115 - Eveto Σ Ece(o,)) p(u, = b, U, = c) logp((; = b 1y =
(2)
(sparsity) A = M 1
(3)
(acyclicity) (2(A)= Tr
(4)
where & the number of endogenous, an indicator function, and , > 0 scaling hyperparameter.
Optimisation problem We can now state the general learning objective for Causal CGMs. Given X' C X (i.e. the imput) and (2) set of annotations for each exogenous variable v E V C V (i.e. the labels), we wish to find functions Ç. f, together with the adjacency matrix A, that maximise the log-likelihood of e, v'. while observing x (or equivalently {(x)):
(1) set of entities represented by their feature vectors
endogenous cupies' prediction
endogenous variables' prediction
graph prioes
L E Eu, "upp""p(( log p(e' w)] + Buw-:polddo(/"=v)) log log do(u' = v), u)] + (2
where X1 hyperparameters balancing optimisation objectives (see related ablation studies in App. H.3). Notice that we use the do-operator replacing 6 with labels UY to reduce leakage and provide better gradients to the endogenous predictor This enables Causal CGM to be aware of do-operations during training, making it effective in responding to do-interventions once deployed.
5

---

Published as a conference paper at ICLR 2025
dolns
effect path
=
dole)
do(es 0)
(a) Causal graph.
(b) Ground- toound-truuth intervention.
(c) do intervention.
(d) Blocking.
Figure 3: (a) A 5-variable causal graph. (b) A ground-truti intervention fixes the enror of the prediction tig to the ground-truth label Dg. (c) A do-intervention sets the value of the second variable constant i.e., 02 0. The intervention impacts V2 v2's effects i.e., 13,5, but does not alter 's causes i.e,, 11- This operation can override ground-truth interventions. (d) A do-intervention on 1i blocks the causal effects of U2 on Ug. As result, intervening on U2 cannot alter Us anymore.
3.4 NIGH-DIMENSIONAL ENDOGENOUS REPRESENTATION
As observed by Mahinpei et al. (221)) using scalar representations for concepts (corresponding to endogenous copies in our context) can significantly degrade predictive performance in realistic settings. To address this issue, and inspired by Espinosa Zarlenga et al. (2022). Causal CGMs employ high-dimensional representations for endogenous copies V For each endogenous copy, Causal CGMs learn a mixture of two embeddings with explicit semantics, representing the variable's ' state. This design enables the model to construct evidence both in favour of and against variable's 's state and supports simple concept interventions, it allows switching between the two embedding states during interventions. Specifically, we represent the context of each variable with two embeddings: 1_ = u = {(x), c E R Each embedding carries specific semantics: cj represents the active state of the variable, while cj represents the inactive state. Once these semantic embeddings are computed, the final endogenous embedding vj for vy is constructed as mixture of oj and oj weighted by the endogenous state:
(5) This formulation serves two primary purposes: (i) it forces the model to rely exclusively cf when the j-th endogenous variable is active (ty- 1) and on C when inactive, thereby creating two distinct and semantically meaningful latent spaces; (ii) it enables straightforward intervention strategy, where one can switch between embedding states to correct a mispredicted endogenous variable.
e/Jc +1-
3.5 CAUSAL REASONING AND VERIFICATION WITH CAUSAL CGM
CGMs support two kinds of interventions: ground-truth interventions, as in CBMs, and do- interventions, in causal models. In what follows, we first highlight how CGM's design makes ground-truth interventions more effective and then we describe how do-interventions can be applied in Causal CGMs to verify properties of the model's 's decision- makinn process.
Ground-truth interventions Causal CGMs support " "ground- "ground-truth interventions" (see Figure 3b) Ground-truth interventions are of the core motivations behind CBMs (Koh et al. 2020). Through ground-truth interventions, concept bottleneck models allow experts to improve a CBM's task perfor- mance by rectifying mispredicted concepts at test time, thus significantly improving task performance within a human-in-the-loww setting. In Causal CGMs, however, ground-truth interventions have potential impact on all endogenous variables descendant of an intervened node, which may include not only nodes corresponding to downstream tasks but also nodes corresponding to a CBM's interme- diate concepts. This enables a single concept ground-true -ruth intervention to potentially improve the prediction of intermediate concepts as well as downstream tasks.
Causal reasoning and verification: do-interventions, counterfactuals, and blocking Causal CGMs can answer interventional and counterfactual queries related to the model's decision-making process using the do-operator on the unfolded SCM. Do-interventions enable manipulation of Causal CGM's decision-makinn process by changing the value of a specific endogenous variable and observing how affects other variables' distributions (see Figure Bc). In Causal CGMs, the effect of the do-intervention is analysed through the interventional distribution, denoted as p( (1 (u k),, pa(,;)) which describes the distribution of the outcome variable e; after the intervention do(u, = x) has been performed. In particular, in Causal CGMs, the do-operation fixes the value of the

---

Published as a conference paper at ICLR 2025
intervened variable v, to fixed constant E {0,1) } and removes all causal dependencies from parent variables by zeroing all values of the i-th column of the adjacency matrix:
do H) Uj == K. E {0. [1,s] = 0, (implies that: pa(ey) = 0)
(6)
Causal CGMs also enable to answer counterfactual queries such as "What would the value of the i-th variable have been, had the j-th variable been K, given that we observed e and v, (extended motivation in App Answering these queries involves three steps (Pearl 1995): 1) Abduction: Infer a realisation of exogenous variables that is consistent with the observed v and My in the actual causal model. 2) Action: Modify the architecture of the Causal CGM M into May=x by replacing the structural equation for ; with s, to simulate the intervention. 3) Prediction: Compute the value of U; in the modified model Mx. representing the counterfactual outcome:
(abduction)
új p(u, x; 0c)
do(u; = k) = 1 =
(action)
(8)
(t-j) =
(prediction)
6 = p (1s), ue 0,) Vi $
This formalism allows us to not only estimate the effects of hypothetical interventions but also to explore the implications of alternative scenarios on individual outcomes, providing powerful tool for analysing the model's 's decision decision-making based on interpretable causal structures.
Model verification and blocking Causal analysis enables the verification of properties of Causal CGMs before deployment. For instance, using only the learnt causal graph, it is possible to prove that an endogenous variable V2 is independent of the variable ey by verifying that vj is not among the ancestors of vg. Another form of formal verification, which we call "blocking" employs the do-intervention (see Figure 3d)) Blocking allows one to formally verify the independence of pair of variables given a sequence of do-operations. Given a pair of variables vy and v, such that vy is an ancestor of Us, perform a blocking verification follows: 1) Block: perform do-intervention all child nodes of v3. 2) Verify: perform a do-operation on v itself and observe the impact on U, We can easily verify that the first step makes f/ and e; completely independent by observing that the do-operation on U/ no longer alters the distribution of uj-
4 EXPERIMENTS
Our experiments aim answer the following questions:
Concept-based performance and interpretability: Can Causal CGMs match the generali- sation performance of equivalent black-box models and existing CBMs? Can Causal CGMs enable more effective ground-truth interventions w.r.t. existing CBMs?
Causal Interpretability: Are Causal CGMs causally interpretable? Can Causal CGMs effectively block the causal effect of two causally related endogenous variables?
To answer these questions, we use four datasets: (i) Checkmark, a synthetic dataset composed of four endogenous variables; (ii) dSprites, where endogenous variables correspond to object types together with their position, colour, and shape; (iii) CelebA, a facial recognition dataset where endogenous variables represent facial attributes: (iv) CIFAR1O, an animal classification dataset where the endogenous variables are extracted automatically followingJOikarinen Oikarinen et al. (2023). Using these datasets, we compare the proposed approach with black box baseline and state-of-the--a concept-based architectures: Concept Bottleneck Models (CBM) (Koh et al. 2020),, and Concept Embedding Models (CEM) (Espinosa Zarlenga et al., 2022). We also compare a version of the proposed method (Causal CGM) where the causal graph is learnt end-to-end w.r.t. with version (Causal CGM+CD) where the causal graph is extracted from ground-trun truth labels using a causal discovery algorithm (Lam al. 2022) and injected into the model fixing the matrix A. Due the prohibitive computational complexity of the causal discovery method, this was not feasible for the CIFAR-10 10 dataset We provide further details our experimental setup and baselines in App. G
7

---

Published as a conference paper at ICLR 2025
A comprehensive set of experiments is detailed in App. where the experiments presented in this section for a subset of the datasets are extended to include all datasets. We also report an analysis of Causal CGMs' scalability and computational complexity in App. H.S5
We remark that, following the CBM literature, the goal of Causal CGMs is not to increase performance but interpretability. Therefore, the experiments aim to show that our Causal CGMs classification performance competitive against existing black-box models and state-of-the-an concept-based approaches (e.g., within one percentage point) while enabling better visibility and manipulation of its reasoning process.
4.1
KEY FINDINGS
Causal CGMs match the performance of causally opaque models. (Table I Causal CGMs demonstrate robust generalisation across all datasets, yielding a predictive performance close that of black-box architectures with an equivalent capacity. Causal CGMs using a pre-trained causal graph (Causal CGM+CD) tend to have slightly better label accuracy with respect to Causal CGMs where the causal graph is learned end-to-end (Causal CGM). Causal CGMs's low variance suggests consistent robustness with respect to weight initialisations over multiple training runs. Thanks to concept embeddings, concept incompleteness settings do not have a strong impact on Causal CGMs' performance (results in App. H.4).
Table 1: Label accuracy (f) is computed on all endogenous variables (concepts and task).
CHECKMARK
DSPRITES
CELEBA
CIPARTÜ
SEMANTIC TRANSPARENCY CAUSAL TRANSPARENCY
Black box
90. 1521..5 50
59 D3:nos
79.35.0.14
B4 SG
CBM
911.3120.5
99.55eaar
79.00)s0.38
17.17
CEM
88.08.1.a mm
99.48L007
19.17...36
92.0410.5
Causul CGM+CD
89.47:00.0
99. 19.00s
78.42:0.42
N/A
CGM
88.24_1.30
90.44
78.23.0045
93:321am
Ground-truth interventions on Causal CGMs improve both concept and task accuracy as opposed to CBMs (Figures 4 In Causal CGM, the causal graph induces natural strategy for ground-truth interventions. Indeed, the causal graph narrows down the set of variables to intervene upon: for any given node, we can just fix mispredicted labels of the node's 's ancestors intervening on other nodes will not have any impact. This property significantly decreases the required number of interventions to achieve desired outcome (e.g., to increase downstream task accuracy), as shown in App. H Another advantage of Causal CGM consists in the hierarchical nature of inference that allows ground-trute interventions impact all endogenous variables descendant of an intervened node. In particular, ground-truth interventions may affect not only nodes corresponding to downstream tasks (as CBM and CEM), but also nodes corresponding to a CBM's intermediate concepts. We experimentally verify this property and its impact by calculating for nodes that were not intervened upon (including both concepts and tasks) the change in accuracy before and after ground-trun interventions were applied on their ancestors. Our results (Figure 4 and App- H.T) show that Causal CGM improves nodes accuracy by ~ 15 percentage points after only 7 ground-truth interventions on CelebA. CBM and CEM, instead, achieve similar performance only after intervening all
Celeba
Detta Concope Accurscy! (96)
Celeba
Deta Accuracy (86
10.
1.0
4
32
9 10
Concept Groups Interveneed
Concept Groupz Interven ed
CBM
Causal CGM
CBM
Cousal CGM
CEM
Causal CGM
CEM
Causal CGM
Figure 4: Impact of ground-truth interventions non-intervened nodes (t). Intervention on Causal CGM improves both concept and task accuracy.
Figure 5: Impact of ground-truth interventions concept nodes (1). Intervention CBM and CEM do not improve concept accuracy unlike Causal CGM.
8

---

Published as a conference paper at ICLR 2025
concepts as their architecture assumes all concepts to be mutually independent. Focusing solely on concept-level analysis, Causal CGM enhances concept accuracy by up to ~ 25 percentage points, compared to CBM and CEM. where concept intervention does not impact the accuracy of other concepts (Figure Causal CGM's advantage increases with the number of concepts and connections, as single intervention can impact higher number of nodes in the causal graph.
Causal CGMs' endogenous predictors are causally interpretable (Figures
oa 6bb In Causal
CGM the decision-making process is causally interpretable and can be analysed by visualising the learnt causal graph and structural equations as in a structural causal model, as shown in Figure ba for CelebA and in Figure 6bb for Cifar10 (more results in App. H.2). The first image shows how Causal CGM exploited known biases in CelebA to infer facial attributes. For instance, Causal CGM predicts the attribute "wearing lipstick" directly from the attribute "attractive" and indirectly from attributes such "smiling" or "high cheek", all attributes that are known to be strongly correlated with each other in CelebA (Ramaswamy et al., [02T]] Wang & Russakovsky. 2023). Similarly, in the CIFAR-10 10 dataset, where concepts are automatically extracted, the model learns meaningful relationships between concepts (Figure 6b), such as the presence a "port" implying the presence of "dock", the connection between "beak" and "bird". We can also quantify the strength of the causal dependency between two nodes by computing the probability of necessity and sufficiency (PNS) (Pearl) (2022). In the figure, we represent the PNS w.r.t. the leaf node by colouring each node with a different shade of orange (more PNS of the other datasets in App. Db. This shows how, for Causal CGM, the attribute "heavy makeup" has the strongest impact on the leaf node. This high degree of causal transparency allows users to interpret Causal CGM's inference and can be eventually exploited to identify potential biases, thereby supporting the assessment of the model's counterfactual fairness (an example of generated counterfactuals in Table
As result, users can intervene directly on the causal structure of the decision- decision-making process and remove biases using do do-interventions, as shown in the next paragraph.
2
Wearing Lipetick
Us f 14
1.0
Biid
Attractive
0o f Us
PNS
Beak
Dock
Heavy Makoup
Pa 6- Atg
0.0
1/2
Smiling
High Cheek
Port
22 Mg
6; Ma
(a) CelebA
(b) Cifar10
Figure 6: Portion of the learnt causal graph and structural equations in CelebA (on the left) and Cifarl0 (on the right). A node's colour in the causal graph is proportional to the probability of necessity and sufficiency w.r.t. to the child nodes.
Table 2: Examples of counterfactuals generated on CelebA obtained via do-interventions (intervened variables are marked in red, changed variables are underlined).
Endogenous variables' activations
Smiling, Not Attractive, Mouth Slig. High Cheek, Not Wearing Li, Heavy Make,
Initial Predicted State
Not Wavy Hair, Not Big Lips, Not Oval Face. Not Makeup, Not Fem Model
Smiling, Attractive, Mouth Slig. High Chock, Wearing Li. Heavy Make,
What if would wear lipstick?
Not Wavy Hair, Not Big Lips. Not Oval Face, Not Makcup, Not Fem Model Smuling, Attractivg, Mouth Slig, High Cheek, Wearing Li, Heavy Make, Not Wavy Hair, Big Lips. Not Oval Face, Makeup, Fem Midel
What I would wear lipstick and also makeup?
Causal CGMs can make two causally-related variables causally independent by blocking all paths between these variables (Table(3 The causal transparency of the proposed approach allows users to modify the model's decision-making process (e.g., to de-bias the model's inference) by using do-interventions. In particular, we can make two causally-related variables and j causally independent by blocking all paths between the cause and the effect j. We experimentally verify this property and its impact by computing the Residual Concept Causal Effeect i.c., the ratio between the Concept Causal Effect (CaCE) (Goyal et al. (2020) obtained after and before blocking. The optimal value of this metric is zero, corresponding to perfect causal independence between and j (i.e., the

---

Published as a conference paper at ICLR 2025
optimal value for a de-biasing operation). The experimental procedure is reported in App. H.7 The results show that, in Causal CGMs, blocking a variable in the causal graph always yields a perfect Residual Concept Causal Effeet of zero across all datasets. In contrast, applying the same procedure in CBMs leads only to a negligible reduction in the average causal effect to 3 percentage points. CEMs not only fail to reduce the causal effect to zero but, in some cases, even increase the causal influence. These results underscore how Causal CGMs transparency enable users to manipulate the model's secision-making process achieve desired outcomes, as opposed existing CBMs.
Table 3: Residual Concept Causal Effect (Q) between consslly--elatte variables having blocked all paths between the two variables with do-interventions the causal graph. The optimal value is zero corresponding to perfect causal independence. (see App. H.7 for further clarification) Values above 100% mean that the causal effect increased instead of decreasing.
CHECKMARK
DSPRITES
CELEBA
CIFAR IO
Black box CBM
NIA
N/A
N/A
N/A
97.99+6.64
100.00=a.70
97.84
100.00=0.00
CEM
102.58.12.95
100.0014.62
106.0010.50
100.0010.00
Causal CGM+CD
0.00+0.00
0.00
0.0020.00
N/A
Causal CGM
0.0020.00
0.0020.00
0.0020.00
0.00=0.00
DISCUSSION
Related works Causal CGMs present substantial advantages compared to the state of the art. Com- pared to most causal feature-atributiion methods (e.g.. (Chattopadhyay al. 2019)). Causal CGMs focus on high-level human- intrapprrntrpetbabec concepts. Causal CGMs differ from existing CBMs in their approach intervention and causal relationships. Unlike vanilla CBMs, where concepts direct, independent causes of the target, Causal CGMs incorporate richer causal structures. This enables human-in-thee
corrections to mispredicted reasoning steps, improving both downstream accuracy and explanation fidelity. Causal CGMs are also more flexible than existing relational CBMs (Barbiero et al. 2024) where concept relations assumed to be provided a priori. The closest methods to Causal CGMs are post-hoc causal concept-based explainability techniques, like DiConStruct (de Oliveira Moreira et al., 2024) and conceptual counterfactual explanations (Abid) et al. 2021). These use surrogate causal models to emulate black box's 's predictions. However, as Rudin (2019) notes, matching predictive behaviour does not guarantee structural similarity in decision-making, making surrogate models unreliable explanatory proxies. Causally interpretable by-design architectures, such as Causal CGMs, do not suffer from this issue.
Limitations and future works Our method's limitations stem from those of CBMs and causal reasoning. The quality of learnt causal graphs depends on dataset quality and annotations; missing or noisy labels may lead to suboptimal graphs. Like generalised PGMs, Causal CGMs unfold easily when the final graph is acyelic. Cyclic variables remain inferable but require special unfolding techniques, which future work may explore. Finally, Causal CGM's learnt graph represents the model's 's inference, not necessarily the data-generating process, making it suitable for verification and control but not for understanding the dataset's 's distribution.
6 CONCLUSION
Causal opacity represents a key open challenge at the intersection of deep learning, interpretability. and causality. Causal CGMs address this challenge by employing an architecture which makes the decisign-making process causally transparent by design. This makes Causal CGMs reliable and verifiable compared to both usual DL architectures and standard (non-causal) CBMs. The results of our experiments show that Causal CGMs support the analysis of interventional and counterfactual ecenarios-therele improving the model's causal interpretability and supporting the effective verification of its reliability and faimess-aan and enable mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections, but also accuracy of the explanation provided for specific instance. As result. advancing this research line could significantly improve the reliability and verifiability of concept- based deep learning models, thus supporting their deployment in real-world applications.
corrections to
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
GD acknowledges support from the European Union's 's Horizon Europe project SmarCHANGE (No. 101080965). PB acknowledges support from Swiss National Science Foundation projects TRUST-ME and IMAGINE (No. TMPFP2 2222_222266... MEZ acknowledges support from the Gates Cambridge Trust via Gates Cambridge Scholarship. AT acknowledges the support by the Hasler Foundation grant Malescamo (No. 22050), and the Horizon Europe grant Automotif (No. 101147693). MG acknowledges support from Swiss National Science Foundation projects XAI-PAC (No. PZOOP2 216405). GM acknowledges support from the KU Leuven Research Fund (STG/22/021, CELSA/24/008) and from the Flemish Government under the "Onderzoeksprogramma. Artificiële Intelligentie (AI) Vlaanderen" programme. This work was also supported by the EU Framework Program for Research and Innovation Horizon under the Grant Agreement No 101073307 (MSCA-DN LeMuR).
(No. 205121L_21444
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully explaining model mistakes using conceptual counterfactuals. arXiv preprint arXhv:2106.122 12723,, 2021.
Christel Baier, Clemens Dubslaff, Holger Hermanns, and Nikolai Käfer. On the foundations cycles in bayesian networks. In Principles of Systems Design: Essays Dedicated to Thomas A. Henzinger on the Occasion of His 60th Birthday, pp. 343-363. Springer, 2022.
Pierre Baldi. Deep learning in science. Cambridge University Press, 2021.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-basse based logic explanations of neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6046-6054, 2022.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Charlotte Magister, Alberto Tonda, Pietro Lió, Frederic Precioso, Mateja Jamnik, and Giuseppe Marra. Inter- pretable neural-symbolic concept reasoning. In International Conference Muchine Learning, pp. 1801-1825. PMIR, 2023.
Pietro Barbiero, Francesco Giannini, Gabriele Ciravegna, Michelangelo Diligenti, and Giuseppe Marra. Relational concept bottleneck models. Advances Neural Information Processing Systems 37 (NeurIPS 2024), 2024.
Sander Beckers. Causal explanations and xai. In Conference on causal learning and reasoning, pp. 90-109. PMLR, 2022.
Florian J Boge. Two dimensions of opacity and the deep learning predicament. Minds and Machines, 32(1))43-75, 2022.
Aditya Chattopadhyay, Piyushi Manupriya, Anirban Sarkar, and Vineeth N Balasubramanian. Neural network attributions: A causal perspective. In International Conference on Machine Learning, pp. 981-990. PMLR, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782,
2020. 2020.
Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, and Stefano Melacci. Logic explained networks. Artificial Intelligence, 3 314: 103820 2023.
Ricardo Miguel de Oliveira Moreira, Jacopo Bono, Mário Cardoso, Pedro Saleiro, Mário AT Figueiredo, and Pedro Bizarro. Diconstruct: Causal concept- based explanations through black-box distillation. In Causal Learning and Reasoning, pp. 740-768. PMLR, 2024.
David Debot, Pietro Barbiero, Francesco Giannini, Gabriele Ciravegna, Michelangelo Diligenti, and Giuseppe Marra. Interpretable poucept-based memory reasoning. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024.
11

---

Published as a conference paper at ICLR 2025
Virginia Dignum. Responsible artificial intelligence: how develop and use Al in responsible way, volume 1. Springer, 2019.
Mateo Espinosa Zarlenga, Pictro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini. Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. Advances in Neural Information Processing Systems, 35, 2022.
Alessandro Facchini and Alberto Termine. Towards taxonomy for the opacity of ai systems. In Conference on philosophy and theory of artificial intelligence, pp. 73-89. Springer, 2021.
Amirata Ghorbani, Abubakar Abid, and James Zou, Interpretation of neural networks is fragile. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 3681-3688, 2019.
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace), 2020.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9()): 90-95, 2007. doi 10. 109/MCSE.2007.55
Jean Kaddour, Aengus Lynch, Qi Liu, Matt Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. arXiv preprint arXiv:2206. 15475, 2022.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMIR, 2018.
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01577 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https www.eeaatttcccccoo..oorrrrorerre
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Wai- Yin Lam, Bryan Andrews, and Joseph Ramsey. Greedy relaxations of the sparsest permutation algorithm. In James Cussens and Kun Zhang (eds.), Proceedings the Thihth-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 Proceedings of Machine Learning Research,
pp. 1052-1080 1062. PMLR, 01-05 Aug 2022. URL https:://proceeeee
ml ppress///
htm
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 133/4, 2021.
Ričards Marcinkevičs, Patricia Reis Wolfertstetter, Ugne Klimiene, Kieran Chin-Cheong, Alyssia Paschke, Julia Zerres, Markus Denzinger, David Niederberger, Sven Wellmann, Ece Ozkan, et al. Interpretable and intervenable ultrasonography- -based machine learning models for pediatric appendicitis. Medical Image Analysis, 91:103042, 2024.
2017. Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerehner. dsprites: Disentanglement
testing sprites dataset. http://ithabbcoommeeemmmiiiiisss
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wär Weng. Label-free feeee- concept bottleneck models. arXiv preprint arXiv:2304.06/29 2023.
12

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http::/papprss..errrs.....c
9015 pytorch an iee-immpratiin style high perforrrfoe deep learning Library.
pdf
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4))669-688, 1995.
Judea Pearl. Causality. Cambridge university press, 2009.
Judea Pearl. The seven tools of causal inference, with reflections machine learning. Communica- tions of the ACM, 62(3):54-60, 2019.
Judea Pearl. Probabilities of causation: three counterfactual interpretations and their identification. In Probabilistic and Causal Inference: The Works of Judea Pearl, pp. 317-372. 2022.
Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion. O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learming Research, 22:28252883, 2011.
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de-biasing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9301-9310, 2021.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22) 215, 2019.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models, 2023.
Alberto Termine and Giuseppe Primiero. Causality problems in machine learning systems. In Federica Russo and Phyllis Illari (eds.), Routledge Handbook of Causality and Causal Methods. Routledge, 2024.
Jin Tian and Judea Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1):287-313, 2000.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL. & Tech., 31:841, 2017.
Angelina Wang and Olga Russakovsky. Overcoming bias in pretrained models by manipulating the finetuning dataset. arXiv preprint arXiv:2303.06167 2023.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697 2020.
Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations deep neural networks. Advances in Neural Information Processing Systems, 33:20554-20565, 2020.
13

---

Published as a conference paper at ICLR 2025
A CAUSAL EXPLAINABILITY CAUSAL OPACITY, AND CAUSAL DISCOVERY
Deep Learning (DL) models have a pervasive impact many areas of contemporary research and society (Baldi 202T). Despite this success, there is growing concern about the widespread real-world application of DL, particularly in sensitive domains (Dignum 2019). This is partly due to lack of causal explainabillity of these models, which undermine their robustness, fairness and generalisability in out-of-distribution context (Schölkopf al., 2021). Causal explainability, in particular, is multi- faced issue, which includes a variety of different, albeit related, problems (Termine& & Primiero! 2024).
One such problem is that of causal discovery and concerns the possibility of using a model detect and understand the causal mechanisms of the data generating process (Pearl & Mackenzie 2018
Pearl 2019 Schölkopf et al. 2021 Kaddour et al., 2022). , An equally important but distinet problem is that of causal opacity, which denotes the difficulty of users to grasp and understand the "hidden" causal structure that underlines the predictions delivered by given model (see Figure Ta). This
problem can be better understood in the light of Pearl's 's framework of causality (Pearl 2009 2019), which relates causal understanding to the ability of an agent to answer what-if -f type of questions. In particular, Pearl identifies three different kinds of what-if questions organised in hierarchy of levels. At the bottom level we have observational questions, which concern the actual state of affairs one can observe in data. At the intermediate level we find interventional questions, which concern the effects of intervening on the actual state of affairs and fix the value of some variable to a pre-defined one. Finally, the top level encompasses counterfactual questions, which concern an hypothetical state of affairs that could have been occured but it is not the actual one. The three levels provide a measure of the causal understanding that agent has of a given target-system. Agents that answer only observational questions do not possess any causal understanding of the target system; agents with an intermediate level of causal understanding can answer interventional questions, whereas a complete causal understanding is required for counterfactual questions.
In this paper, we propose to involve Pearl's hierarchy to define and measure the causal opac- ity/transparency of DNN model based on the ability of its users to answer interventional and counterfactual questions concerning the structure of its inferential behaviour (e.g.. "what happens I fix the feature age to a value greater than 50?" or "what the model's prediction would have been if the feature age had taken a value greater than 50 instead of lower then or equals to 50?"). In this regard, complete causal transparency require that users answer both interventional and counterfactual questions concerning the model's inferential behaviour, whereas if only observational questions can be addressed (i.e., questions regarding the input-output associations of the model), then the model results completely causally opaque: this is notably the case of the majority of DNNs considered in contemporary AI research.
Notice that causal opacity is fundamentally distinet from the problem of causal discovery, which world and not that of model's inference. To clarify this point, consider a simple rules-based model as the one described in Fig. 7
concerns the causal structure of the
Risk Boaii__barb__tttte attack// high) Agol 50) v Income( S 10000000
Risk hes__hhatt_tttee attack( low) Agol 50) Income( > 100.000)
Figure 7: Rules- based Model M2
This model is causally transparent as users can easily compute interventional and counterfactual questions concerning its inferential behaviour (e.g., "what if the age of the patient would had been 2 50 instead of < 50"?). However, causal transparency does not imply that the causal structure of the model's 's inference (Fig Ba) resemble the causal structure of the world (Fig. 8b). which remains a very distinct animal.
To ensure that a model is causally transparent and matches the causal structure of the world, indeed, a suitable combination of causal transparency and causal discovery techniques is usually required. However, causal transparency per-sé is a fundamental requirement and sufficient for variety of tasks, such as establishing the model's fairness (especially within the popular framework of counterfactual fairness (Kusner et al.] 2017)). Consider the model in Fig. [ which is slightly modified version of the model depicted in Fig. ? The causal transparency of the model makes it easy for users to verify the un-fairness of its inferential behaviour (stemming from the fact that the model uses the variable
14

---

Published as a conference paper at ICLR 2025
Age
Income
Age
Iscome
Risk Hearth Attak
Risk of Hearth Attack
(a)
(b)
Figure A graphical representation of the causal structure of M1 's inferential behaviour (a) as opposed to the real causal structure of the world (b). As M1 uses the variable Income predictor of the variable Risk of Hearth Attack, a causal edge is present between the two variables in a. This edge not present in
direct causal connection exists between the two variables in the real world.
Gender
Risk Risk_hearthh hearthe jnttnck( - high) +- Age( 2 50) Incomel S (00.000)
Risk heurth, attack( low) Age( < 50) Income( 100.000)
Income
Incomot high) Gender male)
Income( Low) Gendent -male)
Risk Hearth Attack
(a)
(b)
Figure 9:
(a) Slightly modified version of Mi, and (b) graphical representation of the causal
structure of its inferential bchaviour.
gender to predict the variable income), and implement suitable corrections (e.g. remove the causal link between gender and income).
B DERIVATION OF CAUSAL CGMs
In the absence of causal priors, Causal CGMs need to learn causal dependencies between high-level features that lead to the model becoming more effective in solving its designated task. As result, we can formally describe Causal CGMs using a generalised probabilistic graphical model (PGM) that extends traditional PGM by allowing cycles:
Definition B.1 (Generalised Causal Concept Graph Model). Given an observed input feature x, set of E N latent factors u, E ld each associated with a high level interpretable variable U; E V, a Generalised Causal Concept Graph Model is the generalised probabilistic graphical model (PGM)
G = (W, with nodes N = {x} UU U V and edges & = { [xr, u:) u; E {} U {(us, 0;)
{1, (1.-.. .k] U [(0,, 15) | 0; 2g E V, * Uz} which represents the joint conditional distribution p(u,u x):
(10)
The cyclical nature of Causal CGMs comes from the necessity to model all possible dependencies among variables vy Notice that the model is uniquely identified by the set of all conditional probability distributions corresponding to the arrows in the graph. Unfortunately, in generalised PGMs. the model does not easily factorise in terms of such distributions due to the cycles. To deal with cycles while maintaining the independencies induced by the graph structure, we can use an unfolding semantics for cyclical PGMs (Baier et al.) (022). This semantics is based on the choice of a "cutset" i.e., specific set nodes Q C N in the PGM such that every cycle in the PGM contains at least one node in Q. Intuitively, by unfolding the nodes in the cutset, all cycles are broken leaving us with standard acyclical PGM. The consistency between the semantics of the original cyclical PGM and the unfolded acyclical PGM is only valid in the limit of infinite unfolding (Baier et al. 2022). However, when computing the likelihood of an observed complete set of variables U; E V. modelling one single unfolding (i.e., a single transition) suffices for learning the conditional probability distributions
15

---

Published as a conference paper at ICLR 2025
among the variables in V. as all the variables become conditionally independent each other. As a result, we can define Dissected Causal CGM as the one-step unfolding of the Causal CGM in Definition[B. [B.]
Definition B.2 (Dissected Causal CGM). Given Causal CGM. let V V be the cutset. Then, the dissected Causal CGM g = (W V", Evr) is an acyclic PGM obtained by extending the generalised PGM by (i) adding copy of all cutset nodes V {u; U; E V}. (ii) adding new set of edges directed from parents of cutset's nodes to the generated copies V i.e., Ey = {(a, 6) (a, b) E E, E V'JU ((a,b) (a, 6) E E, & V"J, and (iii) defining an initial probability distribution for the new copies p(u" [[)] given the latent variables. The resulting PGM, factorised as p(u, v', a) p(u v', u)p(v' u)p(u x). is:
(II)
C PROOF OF THEOREM 3.2
Proof. Let us consider the graph G (V U roots(g'), {(v wy) u, vj E V, (0), E Eg (v/, uy)lo? E roo))(9")). We will first prove that this graph is isomorphic to G' for all nodes V and then that the conditional probabilities are equivalent.
The graph g' is defined with nodes V U V", where V represents the endogenous variables and ) represents copies of the same high-level variables. Hence, the of nodes is the same for both graphs by definition. In G', the edge set Eg consists of edges (of, 2z) between nodes in V and V. Specifically, each of E is connected to vj V, where of # Ug. 9", the edge set Eg- consists of:
edges (wi, U/) between nodes V, which are present if (u,, vj) E Eg',
edges (u/, U3) where of E roots(G"), representing connections between the root nodes and their corresponding child nodes.
which represent one-to-one correspondence between the edges in Egr and Eg.. Thus, the two graphs are isomorphic.
Now, we prove that for all nodes ii E V \ (roots(g")) U (ch(roots((")), the conditional probabilities are equivalent: p(u; pay (u;), uz; 05) = p(u; (us), ueg 0y). In G', the conditional distribution of a node et V depends on its parents pay
pay((e)), (vt), which are nodes in V", as well as the latent factors uy Since V = V by Def. 3.1 and the graphs G' and G are isomorphic for all V nodes, the parent set pay (w;) in V" corresponds directly to the parent set pay( in V. up to the bijection between the two sets. Under full observability (the training conditions of CGMs), conditional queries of the form p(u) all but v) are just inspections of the conditional probability table. As a result, p(us (v;), us; 0f) and p(u; (o;), u,; 0y) correspond to repeating the same query on the same conditional probability table. Therefore, for all nodes o; V (roots(G") ch(roots(G"))). we
can write: p(v; (v)) u; 0y) =puu, pay (vx), u4 0,;
D NEURAL PARAMETRIZATION OF CONCEPT GRAPH MODELS
We can interpret the factors of dissected Causal CGM as follows: p(u, x; fx) is the exogenous encoder, i.e., deterministic distribution that is parametrised by neural network Ç X + U. In CBMs, this function represents the input encoder. In causal CGMs we use the exogenous encoder to predict the value of exogenous variables as we consider that the observed input (e.g., an image's pixels) holds most the contextual variability required to infer exogenous variables. For instance, an image provides information about background, lighting conditions, and object locations. All this information is used to anchor the endogenous predictor to a specific context and to correctly infer the values of the exogenous variables and the endogenous roots of the causal graph. The exogenous
x;
16

---

Published as a conference paper at ICLR 2025
encoder generates exogenous variables u, E U mapping raw input features (e.g., image's pixels) to latent embeddings ú; E R2, E N. In practice, this process mirrors the generation of context vectors in Concept Embedding Models to avoid information bottlenecks which may negatively affect the model's accuracy (Espinosa Zarlenga et al. (022). First, the encoder 1b X + H maps raw features a latent code h H. Then, a pair of neural networks (0z , map the latent code into two different embeddings whose concatenation (o] (b), 07 (h)]² corresponds to the exogenous variable Ui of the i-th concept:
(exogenous variables) 0J = ((x) (0x (x)())) o7 (x)())
(12)
In contrast, p(v', Uyj 0.) is the copies predictor. This is the product of k: independent Bernoulli distributions whose logits are parameterised by a neural network 8 : U + V. In CBMs, the composition of the exogenous encoder and the concept predictor is often called concept encoder g (os. In causal methods. this function represents a (supervised) causal feasufe learner (Kaddour et al. 2022). The copies predictor generates endogenous copies v V from latent embeddings ú, This is obtained using a neural network classifier U + V as a scoring function as in (Espinosa Zarlenga et al. 2
(endogenous copies) o = + (W,ú, b.)
(13)
Finally, p(u; pay: (ay)(e:), uy; 0y) is the endogenous predictor. This distribution is the product of k independent Bernoulli distributions whose logits are parameterised by neural network f V U -
V. The input to this function pay'(e)) (u:) (representing direct causal dependencies) is weighted by a learnable adjacency matrix M S IR. & where each learnable weight a.y models the strength of the dependency of v; from its parents v/y. In CBMs this function is called task predictor (Koh al., 2020p and intuitively represents the analogous of the structural equations that model causal mechanisms in SCMs (Kaddour et al., 2022). The endogenous predictor generates the endogenous variables i, E V by considering exogenous variables ú, and copies i with j * i. First, the function w: V x U R generates endogenous embeddings V using exogenous variables ú and copies iy. following (Espinosa Zarlenga et al., 2022). Then, all endogenous embeddings are weighted by the strength of the dependency azj and aggregated using a deepset-like neural network fr : :R R k + [0, 1] which maps endogenous embeddings endogenous predictions:
(endogenous embeddings)
V = w(i, úz) = o (v(x)) + (1- (1-b)07 (x(x))
(14)
(endogenous variables)
0 Sa (la, V)bE(....))
(15)
In order to learn explicit structural equations, existing logic- based aggregation methods can be used
from the concept literature (Barbiero et al. 2022 2023 Debot et al. 2024). App. F describes in more
detail their adaptation in Causal CGM.
E UNFOLDING CGMS
Notice how learning a DAG together with Definition B.2 to unfold a Causal CGM's endogenous predictor applying directed message passing on the associated structural causal model M, ensuring that the values of endogenous variables are derived solely from the nodes that are their ancestors on the causal graph (see Figure(TO) As a first step, compute the exogenous variables for all nodes. We then predict the values of endogenous variables in root nodes in the learned DAG from their corresponding exogenous variables. Following this, can generate the endogenous embeddings for root nodes and aggregate endogenous embeddings to compute the value of endogenous variables of each child node. We repeat this process until all leaf nodes of the graph are reached. We can obtain this by replacing in Eq. 14 the endogenous copies 6/ with the parents of the endogenous variable u;:
(unfolding) 0; fa ((arj((i), (;))), Vi.j E
(16)
Note that this causal unrolling guarantees two key properties (see Figure 3a and Fig. 10
(I)
modifying the value of a cause (parent node) will impact the effect (child node) in our model, (2) conversely, intervening upon an effect does not alter the cause. This is because, in our model, information flows sequentially following the graph, mirroring the fundamental nature of causal effects. Consequently, this layer not only facilitates the computation of task predictions but also enables the exploration of causal relationships through do-interventions and counterfactual analysis.
17

---

Published as a conference paper at ICLR 2025
Тop
Top
White
Guitt
White
White
fw
RedHeart
Input
Square
Square
Figure 10: Unfolded Causal CGM's endogenous predictor.
In App. H.6 we explain why 1-step unfolding of the causal graph during training is equivalent to any k-step (F >
and empirically show related results.
In rare cases, some edges with very small weights may still make the adjacency matrix A cyclic. At inference time, address this by iteratively removing the edges with the smallest weights from the cycles until the graph becomes DAG.
F ARCHITECTURE
Initialisation of adjacency matrix Causal CGM provides versatile initialisation options for the adjacency matrix A, tailored to specific scenarios. In certain instances, weights can be derived from domain expertise or provided along with training data and labels. Without such information, weights can be directly inferred from training labels through causal structural learning algorithms (Kaddour et al. (022) as a preliminary step. These approaches guide the model towards a predetermined deciston-making pathway. Alternativel,, weights can be learnt concurrently during the Causal CGM training phase, as outlined in Section 3.3 These weights may be initialised either randomly or based on the conditional entropy between labels, providing better starting point. Additionally, hybrid approach is feasible, where certain elements in A are fixed while others remain trainable. For instance, causal structural learning algorithm might yield a Partial Ancestral Graph (PAG) with undirected edges, allowing for the definition of directed edges and learning the direction for others to avoid cycle formation.
Causal mechanisms In Causal CGM, the function fi corresponds to causal mechanism in SCM. Such mechanisms are typically formalised via structural equations. For instance, linear models a common choice for label predictors in Concept Bottleneck Models (Koh et al. 2020) where endogenous embeddings are aggregated using a permutation invariant aggregator function + (such as the clement- wise maximum, or sum):
i =
W
a, + b,
(17)
JE(L....kk
However, other options are also available to increase the expressiveness and interpretability of the decision- decision-making process, such Deep Concept Reasoning (Barbiero et al. 2023) class predictors, which build based formulae to obtain class label predictions using endogenous embeddings:
V A
(xx)
A XEX (E(I.....k)
V
(Pis(ai)
(18)
XE Xpam (E........)
where l, denotes the literal of relevant 6/ representing the variable's 's sign or "polarity" in the logic rule (i.e., either Uy or m.,). For example, given three variables U1, U2, Uss Ug. DCR can predict the endogenous variable U2 using the rule 22 f- (ez A -us) V (-01 A Avs) which highlights the underlying
causal mechanism linking U1 V3, U4 to D2 (notice how DCR can also learn to remove irrelevant variables such v.).
18

---

Published as a conference paper at ICLR 2025
Compositional generalisation The training procedure of Causal CGMs is highly parallelizable and modular as only direct connections need to be trained together (e.g.. a + and + c), while the model takes care of distant connections in an indirect way. For instance, the connection - b can be obtained as a composition b c. As a result, it is trivial for a Causal CGM to make the causal graph grow even at test time (see Figure III. This can be done by composing two different graphs obtained by independent training procedures, encoders, datasets, or data types. We note that this is not possible in standard CBMs, which need to re-train the task predictor from scratch whenever new concepts or tasks are added to the mix. Moreover, this modularity enables form of out-of--istribun as it creates new distant connections between variables that were never part of the same training procedure (e.g., a and e in the previous example).
two different independent training procedures for a - b and
compositional generalisation
training
cansal graph
composed
causal graph
module #1
Prettim Coloe
Foeition Color
Preition
Pusition
Ödtor
Shape
Objjct
Objject
module #2 Caker
Culur
Color
Shape
Object
Shape
Shaps
Gioject
Figure 11: Compositional generalisation in Causal CGMs: two different Causal CGMs architectures are trained independently and then composed only at test time, thus creating a larger graph and allowing out-of distt--idittiitti causal inference.
G EXPERIMENTAL SETUP
G.1 DATASETS
In our experiments we use three different datasets:
Checkmark The dataset consists of tabular data with three features, each ranging from -1 to 1 (denoted as a, b, and c). The target variable d can be either 0 or 1. Each feature is annotated with a concept that indicates whether is positive or negative. The dataset also incorporates causal relationships among the features. For example, feature c is defined as the inverse of feature The target d is set to 1 when both features a and b are positive. This data set is used to test our hypothesis in straightforward and controlled setting.
dSprites (Matthey et al. 2017) - The dataset comprises images featuring one of three objects (square, heart) in various positions and sizes. The defined concepts include: (I) object shape (square or heart), (2) object size (small or large), (3) vertical position (top or bottom), (4) horizontal position (left right), (5) object colour (red or blue). Based on these, causal relationships and a binary classification task are established: if the object is a heart on the right side, it is large; if heart is at the top of the image, it is red; the label is positive if the object is both red and large.
CelebA (Liu al. 2015) The CelebA dataset features celebrity images annotated with various attributes. inc including lipstick presence, gender, facial shape, and hair type. Gender is used as the classification label. This dataset chosen for the presence of correlations and biases, such as the association between wearing lipstick and being identified as female.
CIFAR-10 (Krizhessky) 2009)- - The CIFAR-IO dataset consists of 60,000 colour images in 10 different classes, such as cats, dogs, and cars, with pre-defined concepts. To extract concepts, we follow the Label-Free Concept approach described in Oikarinen et al. (0023) ) where the concept set is extracted using auxiliary Vision Language Models (VLS), which are then used to establish causal relationships and perform classification tasks. This dataset is used to test our model's ability handle complex visual features in setting where concepts are not obtained through human annotations.
G.2 BASELINES
We evaluated approach against three established baselines:
19

---

Published as a conference paper at ICLR 2025
This model employs single predictor that processes the input simultaneussly predict the task label and relevant concepts. It lacks interpretability and does not differentiate between the importance of task labels and concepts.
Black Box:
Concept Bottleneck Model (CBM) (Koh et al. 2020): This model first uses concept predictor to infer concepts from the initial input, followed by task label prediction based on these concepts. It is designed to be interpretable and treats concepts as significant informational to predict the task label.
Concept Embedding Model (CEM) (Espinosa Zarlenga et al., 2022): Comprising context encoders, one for each concept, this model predicts each concept based on its respective context before predicting the final task label. It treats concepts and task labels in the same way as CBM.
In our experiments, all baselines are trained using joint training technique commonly used in CBMs (Koh al. (020). In joint training. the model is trained end-to-end with the task predictor directly taking the concept encoder's outputs as input, enabling simultaneous optimization of both concept and task predictions.
G.3 EXPERIMENTS
In our experiments, we evaluate our approach by examining four key dimensions: (i) performance accuracy, (ii) influence of ground-trute truth interventions, (iii) identification of causal structures, and (iv) blocking for the influence of one variable on another.
To evaluate the first dimension, we conducted a comparative analysis of our approach (using both a learned and a predefined graph) against Black Box, CBM and CEM. This was to determine graph-based inference would decrease model performance. For this assessment, we calculated the model's ' accuracy in predicting all concepts and the task. Typically, these metrics are calculated independently; however, in our study, we treated tasks and concepts equivalently, considering them collectively as labels.
In the second aspect, we evaluate our approach by comparing against CBM and CEM terms of response to ground-truth interventions. Enhancing the impact of interventions in the Concept-Bast Based Model is crucial for improving the role of humans the loop. our experiments, we initially perturbed the inputs to reduce label prediction accuracy, following methodologies established in prior research (Espinosa Zarlenga et al., (022). Subsequently, we implemented interventions on the most inaccurately predicted concepts CEM and CBM. This intervention strategy is considered highly effective, as noted in (Shin et al., 2023). For the Causal CGM, interventions began with concepts that have higher number of descendant nodes in the model's graph, aiming to maximise the intervention's 's effectiveness. To assess this dimension, we measured the change in accuracy for non-intervened labels before and after the interventions on concepts (Delta Label Accuracy).
In the third aspect, we visualise the DAG utilised during the inference stage by Casual CEM and derive the corresponding logic equations. We generate Sum of Product logic rules from a table that lists all possible combinations of input concept values alongside the most frequent prediction for each combination derived from the training set, similarly to what done by (Ciravegna al. 2023). It is crucial to note that while these logic rules are general for the model decision -aaionn process, exogenous information may alter predictions for particular instances.
In the final dimension of our analysis, we compare Causal CGM, which operates on a specified graph, against CBM and CEM in terms of their efficacy in mitigating variable's s influence on the task. Specifically, we perturb the prediction of a concept and then, following the graph structure utilised by Causal CGM, we intervene with the ground truth label on all descendant nodes (blocking). Consequently, in Causal CGM, all links between the altered concepts and the labels are deleted, a feat unachievable in CBM and CEM. To assess this characteristic, we calculated the Residual Concept Causal Effeect, ratio of the Concept Causal Effect (Goyal al. 2020) post- and pre-ap pre-applicttion of the blocking techniques. Ideally, this ratio should be zero, indicating that after blocking, the altered node's 's value no longer influences the outcome of the task.
20

---

Published as a conference paper at ICLR 2025
G.4 IMPLEMENTATION DETAILS
Additional details To maximise the efficacy of interventions in Causal CGM, the second term of the loss be regularised to maximise the average Causal Concept Effect (CaCE) (Goyal al. 2020
as follows:
LC.CB
1 2 72
Σ
lp(e In(oy/do(r,, =) -p(usldo(us, 0)l
Here, represents randomly chosen index for each sample, which is used to select one of the concepts following [Espinosa Zarlenga et al. (2022). This regularisation can be weighted using an hyperparameter, X3- Moreover, for all the experiments where the graph is learnt end-to-end, we initialise the learnable adjacency matrix with the conditional entropy between each pair of values, extracted from the training set.
Hyperparameters All baseline and proposed models were trained for varying epochs across different datasets: 500 for Checkmark, 200 for dSprites, 30 for CelebA and 25 CIFARIO. The optimal epoch for each was determined based on label accuracy on the validation set. A uniform learning rate of 0.01 was applied across all models and datasets. For the CBM and CEM models, both concept and task losses were equally weighted at 1. This weighting scheme was also applied to the loss terms for endogenous copies' prediction, endogenous variables' prediction (A1), and graph priors (A2). The weight assigned to the loss terms in our models to maximise CaCE is 0.05. Additionally, was treated as a learnable parameter, initialised at 0.1, and 3 was set to 1. All experiments were conducted using five different seeds (1, 2, 3, 3,4,55 5).
Code, licenses and hardware For our experiments, we implement all baselines and methods in Python 3.9 and relied upon open-source libraries such as PyTorch 2.0 (Paszke et al. 2019) (BSD
license), PytorchL.ightming v2. 1.2 (Apache Licence 2.0), Sklearn 1.2 (Pedregosa et al. 2011 (BSD license). In addition, shown in this paper. Two datasets we used are freely available on the web with licenses: dSprites (Apache 2.0) and CelebA, which is released for non-commercial use research purposes only. We also introduce the Checkmark dataset and we described it in this section. We will publicly release the code with all the details used to reproduce all the experiments under an MIT license. All the experiments except the CIFARI 10 ones were performed on a device equipped with an M3 Max and 36GB of RAM, without the use of a GPU. The CIFARTO 10 experimentswere conducted on workstation equipped with an NVIDIA RTX A6000 GPU, two AMD EPYC 7513 32-Core processors, and 512 GB of RAM.Approximatell 60 hours of computational time were utilised from the start the project, whereas reproducing the experiments detailed here requires only 8 hours.
used Matplotlib (Hunter 2007) 3.7 (BSD license) to produce the plots
H ADDITIONAL RESULTS
In this section, we include all the experiments shown in Section 4 for the four datasets in more detail and an ablation study on the value of Ag.
H.1 GROUND-TRUTH INTERVENTIONS
Figure 12 illustrates the performance comparison among Causal CGM (using both the provided and learnt graphs), CBM, and CEM regarding the effects of interventions. Delta Label Accuracy, which quantifies the change in label accuracy before and after interventions on a growing number of concepts, is calculated solely for the concepts not directly intervened upon. In particular, Causal CGM demonstrates superior performance when interventions involve fewer concepts. This superior performance is attributed to the propagation of intervention effects through all descendant nodes Causal CGM, unlike CBM and CEM, where the impact confined the final task without adjustments to other concepts. The most significant performance gain, approximately 15 percentage points, is observed in CelebA after seven interventions. This effect is particularly pronounced in scenarios with multiple concepts, such as in 10 and CelebA. In contrast, for simpler tasks with fewer concepts, like Checkmark, the benefits of our approach are less pronounced. The gap observed in CIFAR- 10 between CBM and Causal CGM after all interventions is partly due to evaluation setup: when perturbing the input prior to intervention, we occasionally generate out-of- distribution (OOD) concept embeddings, which only affected the CIFAR10 experiment. This reduces
21

---

Published as a conference paper at ICLR 2025
the effectiveness of interventions for embedding- models, while scalar-based interventions in CBMs remain unaffected. Despite this challenge, our model still achieved approximately 15% higher accuracy overall, with fewer required interventions. Furthermore, when focusing solely on the effects of interventions on the task label, the causal graph utilised by Causal CGM allows us to identify beforehand the specific subset of concepts influencing the task prediction. This prefidentification significantly decreases the required number of interventions to achieve the desired outcome. Figure 13 demonstrates that Causal CGM attains comparable improvements in task performance but after interventions on only three or four concepts, in contrast to the ten and eleven concepts required by CEM and CBM, respectively. The elevated standard error observed in Causal CGM with the learnt graph is attributed to the variability of the graph structure, which significantly influences the outcomes of interventions.
H.2 CAUSAL STRUCTURES
Figure [4] iillustrates the adjacency matrices corresponding the DAGs used by Causal CGM for inference in the first three datasets. On the other hand, T present the logic rules derived from the adjacency matrices depicted in the aforementioned figure. Notably, in the Checkmark dataset, both configurations successfully identified the ground truth graph and the correct logic rules. In the case of dSprites, the DAG identified through causal structural learning (GRaSP (Lam et al. 2022) accurately discovers the causal graph and associated logic rules. Although the end-to-end model accurately identifies the correct relationships between concepts and tasks, it proposes alternative methods for concept prediction. It is important to note that even though the model did not identify the correct causal graph, the model was still capable of performing causal inference with the existing graph. In the CelebA data set, where there is ground truth for either the graph or logic rules, the findings by GRaSP and the end-to-end model appear plausible and reveal biases inherent in the dataset, such as the strong correlation between makeup and gender potential causal links like smiling and a slightly open mouth. This scenario underscores the benefits of employing Causal CGM, particularly in demonstrating how specific concepts used to predict other concepts and tasks.
H.3 ABLATION STUDY
In Tables 7 8 and 9 we present the outcomes of varying the hyperparameter λ3. which weights the loss term designed to enhance the CaCE effect. The results indicate that optimising this loss term contributes to improved CaCE scores, thereby augmenting the efficacy of the interventions. Nonetheless, excessively high values of λ may lead to diminished model performance, as it tends to prioritise boosting the CaCE score at the expense of accurate predictions.
Table 4: Logic rules extracted for the Checkmark dataset from Causal CGM+CD with given DAG and from Causal CGM with a learnt DAG. A term which refers to an exogenous variable is omitted for simplicity.
METHOD
CHECKMARK
a + 60 b f E
Causal CGM
t~ b d - a A
a k CO
x E
Causal CGM+CD
c t~ b d + a/C
22

---

Published as a conference paper at ICLR 2025
Checkmark
dSprites
50
20
Delta Lable Accuracy (%)
40
Delta Lable Accuracy
15
30
20
5
10
2
0
2
5
Concept Groups Intervened
Concept Groups Intervened
CBM
Causal CGM+CD
CBM
Causal CGM+CD
CEM
Causal CGM
CEM
Causal CGM
(a) Checkmark
(b) dSprites
Cifar10
Celeba
Delta Lable Accuracy (%)
Delta Label Accuracy (%)
80
10
60
40
5
20
20
40
60
80
100
120
140
Concept Groups Intervened
Concept Groups Intervened
CBM
Causal CGM CD
CBM CEM
Causal CGM
CEM Causal CGM
(c) CelebA
(d) CIFARIO
Figure 12: Impact of ground-truth interventions on concepts across four datasets. This figure
illustrates the variations in accuracy for non-intervened labels, comparing performance before and after interventions on specific nodes.
CelebA
20
Delta Task Accuracy (%)
15
10
1
9 10 Concept Groups Intervened
4
7
CBM
Causal CGM +CD
CEM
Causal CGM
Figure 13: Impact on the task accuracy of ground-truth interventions performed on CelebA concepts. Causal CGM+CD has received in input causal graph, discovered with causal structural learning algorithm (GRaSP (Lam et al. 2022)), while Causal CGM learns it end-to-end.
23

---

Published as a conference paper at ICLR 2025
Checkmark DAG
Checkmark DAG
1.0
0.8
0.8
0.4
0.4
0.2
B
(a) Causal CGM with learnt graph
(b) Causal CGM with given graph
dSprites DAG
dSprites DAG
1.0
Shape
Shape
0.8
Size
0.8
PosY
PosY
Posk
PosX
0.4
0.4
Color
Color
0.2
Label
Shape Size
PosY
PosX
Color Label
Shape Size
PosY
PosX
Color Label
(c) Causal CGM with learnt graph
(d) Causal CGM with given graph
CelebA DAG
CelebA DAG
Smiling
1.0
Smiling
1.0
Attractive
Attractive
Mouth_St Sluh_SSig-
0.8
Mouth_Sii Siig
0.8
High_ Cheek
High_ Cheek
Wearing_u Li
Wearing_L Li
0.6
Heavy_ Make
Beavy_Make
Male
Male
Wavy_Hair
Wavy_Har Hair
0.4
Lips
Big
Oval Face
0.2
Oval Face
0.2
Makeup
Makeup
Fem Model
Model
0.0
0.0
Smiling
attractive
Mouth Slig
High Chesk
Wearing U
Heavy, Make
Male
Wavy Hair-
Big Liss
Oval Face
Makeup
Model
Smiling
Attractive
Mouth Slig
High. Cheek
Heavy, Make
Male
Wavy. Hair-
Big. Lips-
Face
Makeup
Fem, Model.
Wearing
(e) Causal CGM with learnt graph
Causal CGM with given graph
Figure 14: Adjaciency matrices representing the DAG used by Causal CGM during inference on the three datasets. On the left side, the matrices represent the DAG learnt end-to-end by the model, while on the right the DAG discovered with GRaSP (Lam et al.) 2022). It provides a PAG starting from the training data.
24

---

Published as a conference paper at ICLR 2025
Table 5: Logic rules extracted for the Dsprites dataset from Causal CGM+CD with a given DAG and from Causal CGM with learnt DAG. A term which refers to an exogenous variable is omitted for simplicity.
METHOD
DSPRITES
Shape <- Size
Size
PosY E2
Causal CGM
PosX + 63
Color + Shape
Label + Size A Color
Shape <-
Size <- Shape A PosX
PosY 62
Causal CGM+CD
PosX <3
Color Shape PosY
Label Size A Color
Table 6: Logic rules extracted for the Celeba dataset from Causal CGM+CD with a given DAG and from Causal CGM with learnt DAG. A term which refers to exogenous variable is omitted for simplicity.
METHOD
CELEBA
Smiling (S) 6o
Attractive (A) Heavy Make
Mouth Mough_SSig (MS)
High_Check Check (HC) f- 63
Wearing (WI.) Attractive
Heavy_ Make (HM) Smiling High_ Cheek
Causal CGM
Male f-n- Wearing LiA Heavy Make
Wavy Hair (WH) (HC A WLA Male) V (HC A WLA OF)
Big Big_ Lips (BI.) Smiling High_ CheekA,
Oval_
Oval Face (OF) +- False
Makeup (M) False
(MA V Male) v (M HCA WH) (WLA HC)
Smiling Mouth_ Slig
Attractive Wearing
Mouth_SSig
High. Cheek Smiling
Wearing +4
Henvy_Met Make +- (Attractive A Wearing Li) v (Wearing Li Oval_Face)
Causal CGM+CD
Male Wearing LiÄ Heavy_ Make
Wavy_ Hair Makeup (Attractive Wearing Male)
Big Big_Lips +- Makeup
Oval_Face Smiling Attractive A Wearing LiA, ~ Lips
Makeup False
Model +- Makeup
25

---

Published as a conference paper at ICLR 2025
Table 7: Ablation study regarding A3 on the Checkmark dataset.
Ag
Label Accuracy
Average CaCH
min CaCB
max CaCE
CaCH
CaCE block
Black Box
90.15 0.14
CBM
90.31 = 0.55
1.09 = 0.80
0.00 0.00
9.45 + 1.86
28.36 5.58
27.79 4 5.61
CEM
89.09 1 1.98
1.75 1.66
0.00 0.00
4.45 3.78
11.39 + 12.76
11.89 12.95
Causal CGM
0.05
88.21 1.30
2..4
0.00
41.37
5.58
16.15 11.25
0.00
Causal CGM CE
0.05
89.43 0.93
13.15 2.311
0.00
39.53
+ 3.62
20.99
0.00 0.00
Causal CGM
0.2
85.88 3.31
31.61
0.00
37.55
16.32
15.36
0.00
Causal CGM CF
0.2
85.09 + 2.78
16.39 5.00
0.00
39.69 + 10.97
31.15
19.48
0.00 + 0.00
Causal CGM
0
87, 96 + 1.66
6,98 + 2.95
0.00
18. 8.29
7.39 + 4.10
0.00 = 0.00
Causal
87.04
2.72
0.00 0.00
33.15 9.97
14.29
0.00 0.00
Table 8: Ablation study regarding X on the dSprites dataset.
Label Accuracy
Average CaCE
CaCE
CsCB
CaCB
CaCE block
BlackBox
98.53 + 0.15
CBM
99.55 0.07
2.77 0.30
0.00 + 0.00
8.51 + 1.03
0.64 + 0,70
0.64 -: 0.70
CEM
99.48 0.07
0.46
0.29
+ 0.00
2.15 1.67
2.43 1.62
2/43 4.62
CausalCEM
0.05
9.14
17.53 + 3.26
0.00 + 0.00
44.47
10.16
34.13
19.46
0.00 + 0.00
Causal CGM + CE
0.05
12.85 1: 0.59
0.00 + 0.00
27.72
2.2::3.00 3.66
28.95
1: 13.50
0.00 : 0.00
Causal CGM
0.2
98 33.80 1.24
3.39
+ 0.00
37.59
14.20
17.52
13.41
0,00 - 0.00
Causal CGM CE
0.2
99.30 0.13
0.51
0.00 + 0.00
34.01
6.99
16.84
6.79
0.00
Causal CGM
99.58 0.12
6.89 1.55
0.00 0.00
18.51
+ 5.14
12.11 13.01
0.00 0.00
Causal CGM + CE
99.51 0.05
5.67
1.21
0.00 1 0.00
12.4 1.82
15.15 4.10
0.00 - 0.00
H.4 CONCEPT INCOMPLETENESS AND GENERALISATION WITH A DIFFERENT NUMBER OF CONCEPTS
Concept incompleteness does not have strong impact on Causal CGM. As shown in the experiments with CLIP on CIFAR, we can automatically extract new concepts the provided ones are insufficient. However, to quantify the impact of incompleteness on classification performances, we report an ablation study on CBM benchmark (Marcinkevičs et al.) 2024) specifically designed to assess concept incompleteness. This benchmark is synthetic tabular dataset where we can control to what degree concepts explain the variance in the downstream task (see (Marcinkevićs et al.) 2024) for details). In Figure [1 we observe that, as the concept incompleteness increases from O. where all relevant concepts are available, to = 0.9, where only 10% of relevant concepts are available, the performance of CBMs drop, while that of black boxes, CEMs, and Causal CGMs do not thanks to the use of concept embeddings.
H.5 SCALABILITY AND COMPUTATIONAL COMPLEXITY
Causal CGMs pays small cost (at most quadratic in the number of class labels) to address causal transparency. In terms of the number of parameters, the endogenous predictor scales linearly with the number of concepts, in standard CBMs. Regarding time complexity, the endogenous predictor scales quadratically (both train and inference) with the number of concepts. This is because, during training, model all concept dependencies, and, during testing, inference is performed on a DAG (a triangular adjacency matrix in the worst case). This is small cost in exchange for interpretability (and not unique our work). To better communicate our method's scalability, we report the number of parameters and training and inference times as we increase the number of class labels on the same CBM benchmark used to assess incompleteness (Marcinkevičs et al. 2024). In Figure I6 we observe
Table 9: Ablation study regarding A3 on the CelebA dataset.
Aa
Label Accuracy
Average CaCE
min CaCB
max CaCB
CaCE
CaCE block
Black Box
90.15 + 1.30
CBM
79.00 1 0.18
0.54 1 0.03
0.00 = 0.00
1.67 0.15
5.58 + 2.36
5.46 :: 2.13
CEM
79.17 + 0.26
0.27 + 0.12
0.00 = 0.00
1.07 + 0.56
1.00 + 0.45
1.06 + 0.50
Causal CGM
0.05
78.23
0.45
2.17 + 1.44
0.00 = 0.00
8.18 + 4.38
0.04 + 0.07
0.00 1000+00 0.00
Causal CGM CE
0.05
78.42 0.42
0.42
5.48 1.84000 0.34
0.00 0.00
24.17 + 1.32
1.24
100000.00 0.00
Causal CGM
0.2
77.49 0.37
1.70 0.98
0.00
8.95
0.00
0.00
Causal CGM CE
0.2
78.08 + 0.39
6.15 1 0.31
0.00 = 0.00
29.57 + 1.03
0.82 + 0.46
0.00
Causal CGM
0
77.42
1.09
2.85 0.6:100 0.44
0.00 = 0.00
13.06 2.65
0.00
+ 0.00
0.00
Causal CGM CE
78.31 + 0.36
4.64 + 0.13
0.00 = 0.00
18.31 2.61
1.25
0.08
0.00 0.00
26

---

Published as a conference paper at ICLR 2025
10 concepts
30 concepts
concepts
100 concepts
0:1
0.5
0.1
0.5
0:1
0.5
0.9
0.5
0.9
Aesompleteress
InCorpplitonss
Incomplesooness
Black
Cazaal CGM
Figure 15: Label accuracy for different numbers of concepts with increasing concept incompleteness.
Total Porameters
Encoder Parameters
of Fagareeos
Tisk Prodictoor Parameters
Train Time
Inference Time
700000
Ine (pccods)
300)
Thme (buod nertte ot
0.06
000000
2000
0L04
Namber
230000
1000
0.00
10
100
10
100
10
60
100
Number Corcopts
Numrber of Corcopts
Narbbr of Concopts
Narbbrr of Concopts
Namba Concagts
Black Box
Coasal CGM
CBM
CEM
Figure 16: Number of parameters, train and inference time.
that the impact on the number of parameters is negligible as it is mostly due to the encoder. Here the encoder is a relatively small network, but it could be ballion-parameterr in asioon-lLanguage Model (VLM).. Training and inference time are slightly higher than baselines, but training includes a large encoder, say VLM, the relative impact of the endogenous predictor would be significantly reduced (we did not have time to train a VLM from scratch).
H.6 K-UNFOLDING ABLATION
In Causal CBM, 1-step unfolding of the causal graph during training is equivalent to any k-step (k > 1) unfolding as any step corresponds (theoretically and empirically) to repeating the same query on the same conditional probability table (CPT). Causal CGM learns the conditional probabilities of the variables given its parents (i.e. the CPT of the Bayesian network). Under full observability, conditional queries of the form p(u) all but v) are just inspections of the CPT and do not require any reasoning on the cyclical model (which alleviates us from setting a complex k-step semantics to unroll the cyclical model). The key observation is that the cyclical model served the purpose of templating multiple (any) acyclical graphs jointly without making assumptions on concept relations. During learning. the probabilities of the CPT (parameterised by NNs) are also subject an acyclicity objective. After learning, the resulting graph is acyclic thus inference is performed via standard message passing along the directed acyelic graph. To show this experimentally, we report the results on our CBM benchmark using a different number of unfolding steps and considering different levels of concept completeness in Table TO As expected, the results show that increasing the number of steps does not impact Causal CGM's classification performances.
Incompleteness ratio
0.0
0.1
0.5
0.9
Unfolding K=1
85.00 0.14
83.70 1.100
84.49 + 0.57
85.51 + 0.16
Unfolding K=2
83.78 - 0.81
83.72 + 0.85
85.26 + 0.20
84.96 + 0.14
Unfolding K=3
85.00 = -0.4
83.10 0.98
84.30 0.47
85.32 + 0.05
Table 10: Causal CGM's accuracy with i: u1n2.3)- and higher concept incompleteness.
27

---

Published as a conference paper at ICLR 2025
H.7 DEBIASING VIA DO-INTERVENTIONS
We can measure the effectiveness of blocking in debiasing Causal CGMs using the following procedure:
Selection of Variables i and j: We randomly select pair of concepts, i (parent) and j (child), connected in the graph discovered by the CD algorithm CausalCGM.
Do-Intervention: do-intervention is applied to the child node ()) cut off its dependence on the parent node (i). In our model, this intervention ensures that all subsequent nodes in the graph become independent of the parent node, as information flow is blocked by design.
Mcasuring the Impact: After the intervention, we modify the value of the parent node (i) and measure its impact on the final label (leaf node) with i as an ancestor. If no effect is observed, it indicates successful debiasing, as expected by our model's design. Cutting the edge breaks the information flow through the graph-from from root to leaf pree-eteeenngggg the parent node from influencing subsequent nodes.
Comparison with Baselines: Baselines have no graph constraints, therefore the do- intervention modifies the value of concept but does not alter the decision-making process. As a result, the parent node's influence cannot be entirely removed. This contrasts our model's ability to block the information flow fully.
The results of this experiment are reported in Table 3 using the Residual Concept Causal Effect (L) between causally-related variables having blocked all paths between the two variables with do-interventions
the causal graph. The optimal value is zero corresponding to perfect causal independence. To explain why the optimal value is zero, consider the following example. Consider graph A + B - C. For this graph, the variable A is cause of the variable C. As a result, expect the average causal effect of A on C to be higher than O. Our experiment aims to show that intervening B makes A and C' causally independent (ef. with L323-331). To show this, we apply a do-intervention in B, removing the edge A + B and generating two disconnected graphs: A and B + C. Now, we expect that the residual average causal effect of A on C' is 0 because there is no longer a connection between A and C. As result, 0 is the optimal desirable outcome.
I ANSWERING COUNTERFACTUAL QUERIES WITH CAUSAL CONCEPT GRAPH MODELS
In our experiments, we use Causal CGMs to compute Pearl's Probability of Sufficiency and Necessity (PNS), which is prototypical example of counterfactual query (Tian & Pearl 2000). In general, the computation of counterfactual queries requires three key ingredients (Pearl (009): (1) a causal graph, (ii) structural rules predicting an endogenous variable given its parents' values, and (iil) information about the exogenous variables distribution, which for us encoded the embeddings. In causal CGM, all these three ingredients are available, allowing the computation of both identifiable and unidentifiable causal queries. For the latter, as is the case of PNS, the result is given in terms of an interval exactly as it happens in causal inference with structural causal models (with probabilistic quantification over exogenous variables). Figure 17 illustrates the upper boundaries of the PNS tables obtained using Causal CGM across the four datasets. Additionally, Table provides an example of counterfactuals generated on the CelebA dataset, where we first intervene on the lipstick concept, followed by an intervention the makeup concept.
Table 11: Examples of counterfactuals generated on CelebA obtained via do-interventions (intervened variables are marked in red, changed variables are underlined).
Endogenous variables' activations
Smuling. Not Attractive, Mouth Slig. High Check, Not Wearing Li, Heavy Make,
Initial Predicted State
Not Wavy Hair, Not Big Lips, Not Oval Face, Not Makeup, Not Fem Model
What if I would wear lipatick?
Smiling, Attractive, Mouth Slig. High Check, Wearing Li, Heavy Make,
Not Wavy Hair, Not Lips, Not Oval Face, Not Makeup, Not Fem Model
Smiling, Attrctivee Mouth Slig. High Chock. Wearing Li. Heavy Make.
What if would wear lipstick and also makeup?
Not Wavy Hair, Big Lips. Not Oval Face, Makeup, Fem Model
28

---

Published as a conference paper at ICLR 2025
PNS Checkmark
644
5.2
0.
-64
0.5
Padk
Cake
Shop
NoT
Fast
6.
(a) Checkmark PNS
(b) dSprites PNS
PNS CalebA
PNS GIAA-10
Nouth S15
28
Weatng U
16
Mil
44
3 Lso
12
fem,
Ny
WWw. ANTT
Bg Lig
Dat Sice
Yalopp
Tenuhde
5.0
Harry
(c) CelebA PNS
(d) CIFARIO PNS
Figure 17: The upper boundary of the PNS computed on the Causal CGM models on the four different datasets. The PNS computed between each couple of nodes that are directly or indirectly connected in the graph.
29

---

Published as a conference paper at ICLR 2025
J CAUSAL CONCEPT GRAPH MODELS ADDRESS "CAUSAL OPACITY", NOT "CAUSAL DISCOVERY"
Our work is based fundamental yet subtle distinction between causal opacity/transparency and causal discovery that is poorly understood in the MI. literature. Although related, the two problems are different in nature: Causal opacity/transpareney refers the problem of understanding and manipulating the causal structure of a model's decision-making process, and it thus is a specific instance of what is usually referred to as the opacity or black-box problem. Causal discovery, on the other hand, is not concerned with how the model is but rather with how the world (Boge) (022). i.e. it concerns determining the causal structure of the generating mechanism beyond data. This distinction applies also to the notion of causal understanding. Indeed, we can distinguish between two forms of causal understanding depending on whether the object of onc's understanding is the model's meisiinnmmking process or the the related form of causal understanding) neither implics nor requires causal discovery. The causal structure of the model's decision- making process does not necessarily mirror the causal structure of the world. Of course, one could argue that a good model should always resemble the world's ss causal structure, but this is not always true or even wanted (e.g.. if the world is biased, we may want to construct an unbiased model that deviates from the data-generating process of its training data).
mechanism. Notice that causal opacity (and
K RELATIONS WITH CAUSAL REPRESENTATION LEARNING
If we assume a strict definition of causal representation learning (CLR) as any method that aims at learning high-level representations of the causal- genteattnn mechanisms of data (Schölkopf et al. 2021). then Causal CGMs are not an example of CRL. Causal CGMs aims not to discover causal mechanisms beyond data but to build models whose decision-making processes follow a causal structure that users can visualize, interpret, and manipulate. This requires the model to use high-level variables (concepts) that are meaningful for a user (as opposed to, say, an image's pixels) and to clearly display the causal relationships between these variables (e.g., via a casual graph).
Regardless, there are similarities between Causal CGMs and CRL, Both rely on Pearl's formalism of SCMs and use high-level interpretable variables and causal graphs to model causal dependencies. In the case of Causal CGMs, however, the learned causal structure may not necessarily mirror the causal structure of the data-generating mechanism, as it is required for CRL models.
L COMPARING DIFFERENT DECISION MAKING PROCESSES WITH CAUSAL CONCEPT GRAPH MODELS
Causal CGMs are not identifiable in the sense that we can obtain different causal graphs by training the model over the same dataset. This is similar to obtaining different parameter values and embeddings when training standard deep-learning black boxes on the same data. Similarly, the training of causal CGMs can result in different causal decision- decision-making structures for the same task. The difference is that in the case of the black box, we cannot compare the different models because they are (causally) opaque. In the case of Causal CGMs, we can inspect the multiple models obtained and verify them against desirable properties (e.g.. fairness), hence adopting the best one. This distinction impacts key properties, including identifiability. To further clarify this point, consider the following example. In the world, there is a single ground truth causal structure (e.g., "smoking causes lung cancer"). Guaranteeing identifiability is crucial here because it ensures that we can reliably recover the ground truth causal graph. In contrast, causal opacity deals with DL. models, which are systems that can implement a variety of decision- making mechanisms (c.g., a valid decision mechanism for DL model is: "the model predicts that an individual has lung cancer and this causes the model to predict that the individual has a smoking habit"). These mechanisms may differ even when trained on the same data, as DL models can learn different representations to predict the same outputs. As result, there is no single "ground truth" " causal structure to identify multiify-ultiple valid causal graphs exist, each reflecting different way the model processes information. This inherent variability makes identifiability irrelevant in the context of causal opacity.
30


Reviews:
[{"review_id": "OQ8awPOMK3", "reviewer": "Reviewer_ytSc", "summary": "This paper presents Causal Concept Graph Models (Causal CGMs), a concept-based architecture whose decision-making process is causally transparent by design. It avoids the unrealistic assumption that concepts must be causally independent and direct causes of the class prediction. The paper aims to design Deep Learning  models where each prediction can be traced back to a chain of\nsemantically meaningful causes.", "questions": "- > Concept Graph Models duplicate the layer of interpretable variables $V$ by introducing an additional layer of identical copies $V^{′}$ . Using this additional layer, Causal CGMs make predictions of each $v_{i} \\in V$ using as possible inputs all $v_{j}' \\in V^{′}$ for all $j \\neq i$\n\nHow do you guarantee the causal graph will be acyclic?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-  The paper is well written.\n-  The idea of introducing causal transparency in concept-based models is interesting.\n-  The experiments are sound.", "weaknesses": "-  > *Causal opacity refers to the difficulty in understanding the mechanisms behind a model’s decision-making process.* \n\n  The paper frequently references the concept of causal opacity, but it is too vaguely defined to be fully understood.\n\n- How the model performs abduction to answer counterfactual queries is unclear.\n\n- Minor: The description of Figure 3 is inconsistent with the picture. \n\n- It seems the proposed model sacrifices accuracy to improve interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sOBhO1fPtb", "reviewer": "Reviewer_eiBf", "summary": "In their work, the authors construct causal concept bottleneck models (CCBMs), which are interpretable models that aim to reveal the underlying decision process of deep learning models. The models are built with the assumptions that all relevant human-aligned concepts are known a priori and that these concepts are direct causes of the class prediction. The authors propose measure the predictive power of concepts towards each other other to figure out variable parents. The authors leverage the assumption, that distribution predictions between features predicted by noise, and features predicted from the prior noise-predicted values are invariant. The recovered structure is then used to extract logic rules and trace-back decision.\n\nThe approach is evaluated on smaller image datasets, including CelebA and CIFAR 10, showing performance gains over other non-causal methods.\n\n\n[Disclaimer: This is an emergency review]", "questions": "My questions mainly concern the weaknesses mentioned above. I would kindly like to ask the authors the following questions:\n\n1) How does method guarantee identifiability and non-confoundedness of concept predictions. Could the authors elaborate on the implications of theorem 3.2 towards this question and/or highlight possible limitations that might arise?\n2) Regarding point 3: Could the further explain how distributions of Theorem 3.2 can be equal, given, that $v'_j$ are inferred from exogenous variables only?\n3) How, does the presented approach compare to other causal deep-learning methods? Could the authors present further insights or explain, why other methods might not be comparable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The authors extend prior approaches by dropping the restriction of causally independent concepts and trying to approximate the underlying true graphical causal relation. This is an important extension, as most real-world settings, feature complex interactions between concepts and allows to inspect the consequences of interventions on the modeled process.\n\nBy observing predictive power among variables, the authors are able to (partially) recover the individual parents of the variables. If trained correctly, CCGM are therefore deep models, which are interpretable by design and possibly allow for human-in-the-loop adjustments.\n\nThe presented results improve over prior CBM and CEM, while lacking only slightly behind 'causally opaque' black-box models. The authors show good predictive performance under interventions, and are able to extract some underlying logic rules from the learned models.", "weaknesses": "The weaknesses concern extend to which the presented method 'is actually causal'. While existing works in the field of causality are mainly concerned with providing identifiability results, the presented method mainly leverages predictive power to identify causal relations between variables, which might not coincide with learning the correct causal structure and lead the model to learn spurious associations.\n\n1) Measuring the predictive performance of features towards each other does guarantee to recovery the underlying causal structure. Generally, additional assumptions such as interventions or sufficient variability need to be assumed. In that regard, theorem 3.2 does not state any of these assumptions and is provided without proof.\n\n2) In that regard the paper lacks a comparison and detailed discussion to existing methods in the field of causal representation learning [1], such as differences to other existing deep causal approaches ([2-4]), and -with regard to the prior point- guarantees on the identifiability of (latent) concepts [5-7].\n3) Variables $v'_i$ are predicted just from their exogenous noise variables $u_i$. Assuming that, variables are indeed embedded into a causal structure, and assuming that at least for some $v_i$ the parent set is non-empty, it can not be that $p(v'_i\\mid pa_{V'}(v_i),u_i) = p(v_i \\mid u_i)$.\n4) The experimental section compares to only non-causal methods (CEM and CBM) on their predictive performance (accuracy). Given the above discussion, it is unclear whether the models have actually learned the correspond process. No analysis or discussion towards identifiability and possible avoidance of confounding is presented. As a result a rather strong drop-off in performance between CCGM+CD (recovering the graph from ground-truth labels) and CCGM is observed. In particular, when the causal graph is recovered from data, the presented causal CGM+CD approach might by very similar to Neural Causal Models (NCM) [4].\n\n\n\n**Minor**\n\n* Table 1 is hard to read. I suggest breaking \"semantic transparency\" and \"causal transparency\" into two lines to make the texts bigger.\n\n\n[1] Schölkopf, Bernhard, et al. \"Toward causal representation learning.\" *Proceedings of the IEEE* 109.5 (2021): 612-634.  \n[2] Kladny, Klaus-Rudolf, et al. \"Deep backtracking counterfactuals for causally compliant explanations.\" *arXiv preprint arXiv:2310.07665* (2023).  \n[3] Lippe, Phillip, et al. \"Biscuit: Causal representation learning from binary interactions.\" *Uncertainty in Artificial Intelligence*. PMLR, 2023.  \n[4] Xia, Kevin, Yushu Pan, and Elias Bareinboim. \"Neural causal models for counterfactual identification and estimation.\" *arXiv preprint arXiv:2210.00035* (2022).  \n[5] Brehmer, Johann, et al. \"Weakly supervised causal representation learning.\" *Advances in Neural Information Processing Systems* 35 (2022): 38319-38331.  \n[6] Wendong, Liang, et al. \"Causal component analysis.\" *Advances in Neural Information Processing Systems* 36 (2024).  \n[7] von Kügelgen, Julius, et al. \"Nonparametric identifiability of causal representations from unknown interventions.\" *Advances in Neural Information Processing Systems* 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NcOONeJ87C", "reviewer": "Reviewer_hQg6", "summary": "The authors propose a new class of interpretable models whose decision-making process is causally transparent by design. The method represents relations between high-level concepts, associated with each data point, as a DAG allowing for interventional and counterfactual analysis of predictions and human improvements during the inference process.\n\nThe authors demonstrate that their approach matches the generalization performance of SoTA models. Additional experimental results showcase how explicit causal representation can be efficiently used to increase interpretability and improve performance with human interventions.", "questions": "1. I do not understand the experiment described in the last paragraph of the experimental section (lines 463-476).  Could the authors please provide an example? How are variables i and j selected for each dataset? Are the results in Table 3. averaged over all possible pairs? How do the authors decide which variables to intervene on (i.e. what structure is used as Ground Truth)? Why is 0 a desirable outcome?  - the authors write that the aim of this procedure is debiasing, to my understanding 0 RCCE means no change.\n2. Table 3 has two columns titled “CELEBA”", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is cleanly written. \n2. The approach is well-placed in the literature. \n3. The experimental section is extensive.\n4. The method description is detailed and well-structured.", "weaknesses": "1. One bit of experimental evaluation is unclear to me (see Questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JJWrGu1uHZ", "reviewer": "Reviewer_LQqh", "summary": "Over the past few years, there has been a lot of interest in designing models that are interpretable by construction. Concept Bottleneck Models are one such approach; they first predict an intermediate set of human-specified concepts and then use the concepts to predict the target. These intermediate concepts allow humans to understand the dynamics of model predictions through interventions. One key drawback of such an approach is that they assume each concept independently influences the predictions, while in practice, there is always a complex relationship between the concepts and how they influence the target variable. The paper addresses the above issue by introducing the Causal Graph Concept Graph Model, which is transparent by design and captures the complex dynamics of a model's decision processes.", "questions": "Refer to Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "A serious shortcoming of CBMs is that they assume that each human-specified concept independently influences the model's predictions. The authors address this issue by proposing a Causal Graph Concept model that captures the intricacies of the dependencies between the concepts & the target variable and among them.\n\nAs the proposed model is causally transparent by design, it allows a human to understand the inner workings of the model by enabling them to ask questions, such as:\n- What is the relationship between feature x & label y?\n- What happens to the prediction if I set feature x to k?\n- What if feature x was k' instead of k?\nThe proposed model's performance matches or is within the ballpark of the opaque state-of-the-art models on the chosen set of benchmarks.\n\nTheoretically, it also allows humans to intervene and correct erroneous intermediate reasoning steps.", "weaknesses": "- The paper isn't written well, and some sections are hard to follow.\n- One of the key weaknesses lies in the experiments.\n     - Why only choose simpler benchmarks, like CelebA & CIFAR10, and benchmarks with semantically richer tasks like AwA2, CUB, etc.?\n     - Understanding what the authors mean by joint training is a bit hard.\n          - Are the concept encoders also trained along with the process of learning CGMs? \n          - What formulation of CBMs do the authors build on? (independent bottleneck, joint bottleneck, or sequential bottleneck?)\n          - What is the architecture of the bottleneck layer? Does it influence the quality of CGMs learnt?\n     - One contribution is that CGMs allow human-in-the-loop corrections of erroneous intermediate reasoning steps. I don't see any experiments to support this. Did you evaluate whether the learnt causal graphs are sound and valuable? Did you do a user study to assess their utility to users? \n     - Do you have any intuition about why there is a drop in Figures 4 and 5 for a certain number of intervened concepts?\n     - Column 4 is Table 3 is CIFAR10?\n     - Line 345 says CIFAR10 is an animal classification dataset. Did you use the entire dataset or a subset for evaluation?\n- Another issue with CBMs is that they assume that human-specified concepts are readily available, which rarely happens in practice. Much work has been done in the literature to relax this assumption. I wonder why the authors build on the vanilla version of CBMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNgWATGfJ4", "reviewer": "Reviewer_JdcH", "summary": "The authors introduce a generalization of *Concept Bottleneck Models* (CBM), called *Causal Concept Graph Models* (causal CGM). In contrast to existing CBMs, causal CGM allows for modeling causal relationships between latent variables. In contrast to causal representation learning, causal CGM aims at addressing causal opacity (interpretability) of deep learning models, rather than identifying specific variables and/or mechanisms. The claimed properties of Causal CGM are experimentally assessed on various data sets and compared with a black-box deep learning baseline, a non-causal CBM baseline and Concept Embedding Models.", "questions": "line 88: (Minor comment) The authors write the causal CGM is related to XAI. Maybe it is more about interpretability than it is about explainability? To me, XAI methods aim at explaining black-box models. However, causal CGM is rather about constructing the model such that it is interpretable by construction.\n\nline 91: As already explained in weakness 1, I am not convinced that causal CGM is a new architecture. \n\nline 105: I would challenge that structural causal models are *the* standard framework for causal modeling. Rubin causal models are also quite common. I would suggest rephrasing this. For example: *\"Structural causal models are a widely used framework for causal modeling, alongside other approaches like Rubin causal models.\"*\n\nline 111: *\"which determine the values of each endogenous variable $v_i \\in V$ by computing the conditional probability ...\"* This sentence sounds contradictory to me. According to the SCM definition that I am familiar with, all endogenous variables can be expressed as a deterministic function of all exogenous variables (reduced form expression). Thus, this conditional probability should be just a Dirac measure? I would appreciate if the authors could clarify this.\n\nline 115: I believe this should read *hard interventions*. There also exist other types of interventions where variables are not fixed to a value.\n\nline 121: I am not too familiar with the literature on *concept-based models* (CBM). However, I do not understand how the term *concept* differs from *high-level feature* and how *concept-based modeling* is distinct from *representation learning* (see weakness 2).\n\nline 199: (Minor comment) Doesn't theorem 3.2 follow from d-separation? I could imagine that the proof could be greatly simplified with such a graphical argument.\n\nline 245: How does the training differ from the method proposed by [1] (see weakness 1)? It would be great if the authors could highlight this in detail.\n\nline 257: *maximise the log-likelihood of v, v′* How can the likelihood of unobserved variables be maximized? I suppose you maximize a variational lower bound of $\\text{log} \\\\; p(x)$?\n\nline 481: *Causal CGMs focus on high-level human interpretable concepts* As mentioned in weakness 3, this seems to be at odds with the original motivation of the work.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "1. The authors present an interesting perspective on causal representation learning, where the learned abstract variables and mechanisms are used to obtain insights into a trained model (interpretability) rather than aiming at generalizing to unseen domains. The latter is the main motivation of the related work that I am familiar with (e.g., [1, 2, 3]).\n\n2. The manuscript is particularly well-written and the plots are great, too. The authors did an excellent job at presenting their work well.\n\n3. There are plenty of experiments that (somewhat surprisingly) show that causal CGMs perform similarly to black box models that do not enforce a causal bottleneck.\n\n[1] von Kügelgen, Julius, et al. \"Nonparametric identifiability of causal representations from unknown interventions.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Lippe, Phillip, et al. \"Biscuit: Causal representation learning from binary interactions.\" Uncertainty in Artificial Intelligence. PMLR, 2023.\n\n[3] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.", "weaknesses": "1. In terms of the architecture and training of the proposed method, I am concerned about novelty. The method may seem novel within the CBM literature, where only independent latent variables have been considered so far. However, the proposed method seems to be (almost) identical to [1], which is branded as *causal variational autoencoder* rather than *concept bottleneck model*. Apart from the name and problem motivation, I cannot identify any differences. It would be great if the authors could highlight the differences, if they exist (I did not see them).\n\n2. While the authors start with an interesting perspective on learning latent causal variables (see strength 1), it is unfortunate that this idea is not taken very far. Specifically, I would be interested in questions such as how such a causal graph could be efficiently inferred from any black-box model post training (just an example of the kind of questions that came up to me). Instead, causal CGM seems to be an existing causal representation learning method (see weakness 1), but without identifiability guarantee. I would advise the authors to develop the paper more closely along this original idea (strength 1), as I believe there is a lot of unused potential.\n\n3. The *human-in-the-loop corrections* aspect seems contradictory to the motivation of the work: Causal CGM is introduced from the point of interpretability rather than identifiability of variables and mechanisms. Human-in-the-loop corrections indicate that specific variables (known the human) ought to be identified, which lies at odds with the causal opacity motivation. Therefore, I would suggest moving this aspect entirely to the appendix.\n\n[1] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)