Causal Concept Graph Models: Beyond Causal Opacity in Deep Learning

## Abstract

Abstract Causal opacity denotes the difficulty in understanding the “hidden” causal structure underlying the decisions of deep neural network (DNN) models.
This leads to the inability to rely on and verify state-of-the-art DNN-based systems, especially in high-stakes scenarios.
For this reason, circumventing causal opacity in DNNs represents a key open challenge at the intersection of deep learning, interpretability, and causality. This work addresses this gap by introducing Causal Concept Graph Models (Causal CGMs), a class of interpretable models whose decision-making process is causally transparent by design. Our experiments show that Causal CGMs can: (i) match the generalisation performance of causally opaque models, (ii) enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections but also the reliability of the explanations provided for specific instances, and (iii) support the analysis of interventional and counterfactual scenarios, thereby improving the model’s causal interpretability and supporting the effective verification of its reliability and fairness.

## 1 Introduction

Causal opacity refers to the difficulty in understanding the mechanisms behind a model’s decision-making process  (Termine & Primiero, ).
Intuitively, this type of opacity can be assessed through Pearl’s framework of causality (Pearl, ). Within this framework, we categorise an agent’s causal understanding based on its ability to answer three types of questions: *observational* queries (“what is the relationship between the feature $x$ and the model’s output?”), *interventional* queries (“what happens if I fix the feature $x$ to a value $k$?”), and *counterfactual* queries (“what would the model’s prediction had been if feature $x$ had taken value $k^{\prime}$ instead of $k$?”).
In our case, the target of causal understanding is the decision-making process of a DNN model. That is, whether and to what extent a DNN model is causally opaque (respectively, transparent) depends on the possibility of its users to answer interventional and counterfactual questions that concern the model’s inferential behaviour. In this regard, notice that the problem of causal opacity is distinct from the problem of *causal discovery*, which involves detecting causal mechanisms in the real world, rather than understanding the causal structure of a model’s internal reasoning. For example, a model might effectively predict heart attacks based on the rule $\texttt{heart}_\texttt{attack}\leftarrow\texttt{smoking}\leftarrow\texttt{
lung}_\texttt{cancer}$, even though the actual biological mechanism is that smoking causes lung cancer, not the reverse. Further details on the distinction between causal opacity and causal discovery are discussed in App. A.

Deep Learning (DL) models (Fig. 1(a)) are *causally opaque* when applied to unstructured data like images, as interventions on low-level features, such as pixels, are not meaningful (Rudin, ; Ghorbani et al., ) and therefore do not enable us to easily answer any of these sorts of questions. For example, asking “what happens if I fix the intensity of the pixel $x_{i}$ to $k$?” does little to reveal whether a DNN is exploiting protected attributes like “gender” or “race” when classifying an image of a person as a “doctor”.
Yet, understanding when these sorts of high-level attributes are misused to make predictions is crucial for the proper verification and vetting of models (Kusner et al., ), particularly in high-impact real-world tasks (e.g., medical admission recommendations).

To address this issue, Concept Bottleneck Models (CBMs) (Koh et al., ) have been designed to support interventions on high-level interpretable features known as “concepts” (Kim et al., ). These concepts, whose semantics are aligned with units of information experts would use to solve the same task, allow for more intuitive interventions and counterfactual analysis (Wachter et al., ; Abid et al., ). This is because CBMs shift from reasoning based on raw features to explicitly operating on interpretable variables such as “age”, “income”, “gender”, or “BMI”.

Although CBMs have been shown to be *semantically transparent* (Kim et al., ; Facchini & Termine, )(^1^11Here by *semantic transparent* we mean a model that focuses on features that are meaningful and understandable by human experts in the model’s application domain.), in this work we argue that these models still rely on two unrealistic assumptions (visualised in Fig. 1(b)). First, they assume that concepts are causally independent, an assumption that breaks in most real-world tasks. For instance, CBMs might treat “smoking” and other health conditions as independent predictors of an insurance premium, ignoring how *intervening on* smoking habits might have a significant impact on health conditions. Second, and perhaps more importantly, CBMs operate with the underlying assumption that concepts are the direct causes of the model’s predictions.
These assumptions oversimplify and constrain the model’s inference structure, preventing the model from using more complex and effective concept dependencies (Beckers, ). As a result, the problem of *causal opacity* still represents a key open challenge at the intersection of DL, causality, and XAI.

#### Contributions

In response to this challenge, in this paper we introduce *Causal Concept Graph Models* (Causal CGMs, see Figure 1(c)). Causal CGMs are a new a concept-based architecture whose decision-making process is *causally transparent by design*, avoiding the unrealistic assumption that concepts must be causally independent and direct causes of the class prediction.
The results of our experiments show that Causal CGMs can: (i) match the generalisation performance of state-of-the-art causally-opaque models, (ii) enable human-in-the-loop corrections of mispredicted intermediate reasoning steps,
boosting not just downstream accuracy after corrections but also the accuracy of the explanation provided for a specific instance, and (iii) support the analysis of both interventional and counterfactual questions, thereby improving the model’s causal interpretability and supporting the effective verification of its reliability and fairness. Finally we provide the code of the paper(^2^22[https://github.com/gabriele-dominici/causal-cgm](https://github.com/gabriele-dominici/causal-cgm)).

Figure: (a) Standard DNNs

## 2 Background

#### Causal Models

A causal model (CM) is a mathematical representation of the mechanisms (rules, laws) that explain how different variables of a given target-system causally influence each other.
In computer science and statistics, the standard framework for causal modelling is that of *Structural Causal Models* (SCM) proposed by Pearl ().
Formally, an SCM $\mathcal{M}$ is a triplet $(\mathcal{U},\mathcal{V},\mathcal{F}_{\theta})$ where: $\mathcal{U}$ is a set of exogenous variables representing latent factors with a causal influence on the modelled target-system; $\mathcal{V}$ is a set of endogenous variables representing observable and measurable variables;
$\mathcal{F}_{\theta}$ is a set of functions (or *structural equations*) parametrized by $\theta$ describing the causal mechanisms, which determine the values of each endogenous variable $v_{i}\in\mathcal{V}$ by computing the conditional probability $p(v_{i}|u_{i},\text{pa}(v_{i});\theta)$, where $u_{i}\in\mathcal{U}$ and $\text{pa}(v_{i})$ are, respectively, the set of exogenous and endogenous variables causally affecting $v_{i}\in\mathcal{V}$. Every SCM can be associated with a directed acyclic graph (DAG) whose nodes represent
variables and edges represent direct causal connections.
Interventions can be modelled in SCMs through the *do*-operator (Pearl, ; )
. This operator induces a modification on the model’s structure by changing the value of an endogenous variable which is fixed to a value $\kappa\in\mathbb{R}$ by external means, effectively breaking its original causal dependencies within the model. Formally, applying $do(v_{i}=\kappa)$ to a model $\mathcal{M}$ results in a new model $\mathcal{M}_{v_{i}=\kappa}$ which is identical to $\mathcal{M}$ except that the equation for $v_{i}$ is replaced with a constant value $\kappa$, removing all arrows into $v_{i}$ in the model annexed DAG.

#### Concept-based models

Concept-based models (Kim et al., ; Chen et al., ; Yeh et al., ; Kim et al., ; Barbiero et al., ; Oikarinen et al., ) are interpretable architectures that explain their predictions using high-level units of information (i.e., “concepts”). Most of these approaches can be formulated as a Concept Bottleneck Model (CBM) (Koh et al., ). Given a sample’s raw features $\mathbf{x}\in X\subseteq\mathbb{R}^{d}$ (e.g., an image’s pixels), a set of $r$ concepts $c_{i}\in C\subseteq\{0,1\}^{r}$ (e.g., “red”, “round”), and a set of $l$ class labels $y_{j}\in Y\subseteq\{0,1\}^{l}$ (e.g., labels “apple” or “tomato”), a CBM estimates the conditional distribution $\prod_{j}p(y_{j}\mid\text{pa}(y_{j})=c_{1},\dots,c_{r};\theta_{f})\prod_{i}p(c
_{i}\mid\mathbf{x};\theta_{g})$ where $p(c_{i}\mid\mathbf{x};\theta_{g})$ is a set of independent Bernoulli distributions. This formulation emphasises two key CBMs’ assumptions: i) *concepts are direct causes of class predictions*, and ii) *concepts are conditionally independent of one another*.
Usually, concept-based models represent a concept $c_{i}$ using its predicted probability $\hat{c}_{i}\in[0,1]$. This representation, however, might significantly degrade task accuracy when the provided concepts are incomplete (Mahinpei et al., ). To overcome this issue, Concept Embedding Models (CEMs) (Espinosa Zarlenga et al., ) use high-dimensional embeddings $\mathbf{\hat{c}}_{i}\in\mathbb{R}^{z}$ to represent concepts alongside their truth degrees $\hat{c}_{i}\in[0,1]$.

## 3 Causal Concept Graph Models

Problem: This work addresses causal opacity denotes the difficulty in understanding the hidden causal structure behind a DL model’s inference and decision-making process. Research question: To overcome this, we aim to design DL models where each prediction can be traced back to a chain of semantically meaningful causes. Challenged assumptions: In pursuing this goal, we challenge two key assumptions of Concept Bottleneck Models (CBMs): i) humans know a priori which interpretable variables (i.e., concepts) are direct causes of a given class label (and the dataset embeds this prior knowledge), and ii) all interpretable variables used for explanations (i.e., concepts) are conditionally independent. Solution: We introduce Causal Concept Graph Models (Causal CGMs), a class of models making class
predictions through cause-effect chains of interpretable variables. In what follows, we first introduce Causal CGMs and then proceed to show how their decision-making process is causally transparent. Finally, we conclude this section by describing their learning process.

### 3.1 Blueprint

Dropping the assumptions of CBMs—that concepts are independent and direct causes of class labels—requires a model able to capture all possible dependencies(^3^33This naive formulation would lead to a cyclic probabilistic model, which becomes computationally intractable due to the difficulty in handling cycles in probabilistic models. Causal Concept Graph Models tackle this issue by ensuring the learning process is efficient and produces simple explanations by design (as described in Sec. 3.3).) between interpretable variables (i.e., any classification label). To model such dependencies, Concept Graph Models duplicate the layer of interpretable variables $\mathcal{V}$ by introducing an additional layer of identical copies $\mathcal{V}^{\prime}$(^4^44A detailed analysis of this transformation, including the “unfolding” of the original cyclic model, is provided in App. B and a neural parametrization in App. D.).
Using this additional layer, Causal CGMs make predictions of each $v_{i}\in\mathcal{V}$ using as possible inputs all $v_{j}^{\prime}\in\mathcal{V}^{\prime}$ for all $j\neq i$. This inference structure enables CGMs to trace each prediction in $\mathcal{V}$ back to a set of interpretable variables in $\mathcal{V}^{\prime}$. As a result, the set of copies $\mathcal{V}^{\prime}$ in a Causal CGM play the role of “explaining variables”, akin to a CBM’s concepts, while the set of variables $\mathcal{V}$ play the role of “explained variables”, akin to a CBM’s tasks.

###### Definition 3.1 (Causal Concept Graph Model) .

Given an observed feature vector $\mathbf{x}$, a set of $k\in\mathbb{N}$ latent factors $u_{i}\in\mathcal{U}$ each associated with a pair of identical high-level interpretable variables $v_{i}\in\mathcal{V}$ and $v_{i}^{\prime}\in\mathcal{V}^{\prime}$ with $v_{i}=v_{i}^{\prime},\ \forall i\in\{1,\dots,k\}$, a *Causal Concept Graph Model* $\Gamma=(\mathcal{N},\mathcal{E},\mathcal{F}_{\theta})$ with

- •
nodes $\mathcal{N}=\{x\}\cup\mathcal{U}\cup\mathcal{V}^{\prime}\cup\mathcal{V}$,
- •
edges $\mathcal{E}=\big{\{}(\mathbf{x},u_{i})\mid u_{i}\in\mathcal{U}\big{\}}\cup\big
{\{}(u_{i},v_{i}^{\prime})\mid i\in\{1,\cdots,k\}\big{\}}\cup\big{\{}(u_{i},v_
{i})\mid i\in\{1,\cdots,k\}\big{\}}\cup\{(v_{i}^{\prime},v_{j})\mid v_{i}^{
\prime}\in\mathcal{V}^{\prime},v_{j}\in\mathcal{V},i\neq j\}$,
- •
and mechanisms $\mathcal{F}_{\theta}=\{\zeta,s,f\}$,

represents the joint conditional distribution:

$$ ${p(v_{1}\dots,v_{k},v_{1}^{\prime},\dots,v_{k}^{\prime},u_{1},\dots,u_{k}\mid x ;\theta)=\prod_{i=1}^{k}\overbrace{p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}} (v_{i}),u_{i};\theta_{f})}^{\text{endogenous predictor}}\prod_{j\neq i} \overbrace{p(v_{j}^{\prime}\mid u_{j};\theta_{s})}^{\text{copies predictor}} \overbrace{p(u_{j}\mid\mathbf{x};\theta_{\zeta})}^{\text{exogenous encoder}}}$ $$

$$ $x$ $u_{1}$ $u_{2}$ $\dots$ $u_{k}$ $v_{1}^{\prime}$ $v_{2}^{\prime}$ $\dots$ $v_{k}^{\prime}$ $v_{1}$ $v_{2}$ $\dots$ $v_{k}$ (1) $$

###### Definition 3.1 (Causal Concept Graph Model) .

### 3.2 Causal transparency in Concept Graph Models

Notice how the distribution $p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})$ can be associated with a structural causal model $\mathcal{M}_{CGM}=(\{u_{i}\},\mathcal{V}^{\prime}\cup\{v_{i}\},\{f_{i}\})$ with causal mechanism $f_{i}$
where $u_{i}\in\mathcal{U}$ is an exogenous variable representing latent, uninterpretable information (e.g., noise), $v_{i}\in\mathcal{V}$ is an endogenous variable representing interpretable, symbolic information, and $f_{i}:U\times V\to V$ is a function describing the causal mechanism that determines the value of $v_{i}$ given its parents $\text{pa}_{\mathcal{V}^{\prime}}(v_{i})$. Such dependencies are captured by a graph $\mathcal{G}^{\prime}=(\mathcal{V}\cup\mathcal{V}^{\prime},\{(v_{i}^{\prime},v_
{j})\mid v_{i}^{\prime}\in\mathcal{V}^{\prime},v_{j}\in\mathcal{V},v_{i}^{
\prime}\neq v_{j}\})$, representing all direct causal dependencies between endogenous variables. By modelling such dependencies, and considering that by Def. 3.1 the variables $v_{i}$ and $v_{i}^{\prime}$ are identical copies of the same interpretable variable, Causal CGMs allow any endogenous variable $v_{i}$ to be expressed in terms of its parents in $\mathcal{V}$, denoted as $\text{pa}_{\mathcal{V}}(v_{i})$, rather than in terms of its parents in $\mathcal{V}^{\prime}$, i.e., $\text{pa}_{\mathcal{V}^{\prime}}(v_{i})$, as formalised in the following theorem (proof in App. C).

###### Theorem 3.2 .

Given a Causal Concept Graph Model $\Gamma=(\mathcal{N},\mathcal{E},\mathcal{F}_{\theta})$, let the set of endogenous root nodes be defined as:
$\text{roots}(\mathcal{G}^{\prime})=\{v_{i}^{\prime}\in\mathcal{V}^{\prime}\mid
\nexists(v_{j}^{\prime},v_{i})\in\mathcal{E}_{\mathcal{G}^{\prime}}\}$,
and the set of children of root nodes as:
$\text{ch}(\text{roots}(\mathcal{G}^{\prime}))=\{v_{i}\in\mathcal{V}\mid\exists
(v_{j}^{\prime},v_{i})\in\mathcal{E}_{\mathcal{G}^{\prime}},v_{j}^{\prime}\in
\text{roots}(\mathcal{G}^{\prime})\}$.
Then, for all nodes in the set $\mathcal{V}\setminus\{\text{roots}(\mathcal{G}^{\prime})\cup\text{ch}(\text{
roots}(\mathcal{G}^{\prime}))\}$, i.e., for all nodes that are neither root nodes nor children of root nodes, the following holds:

$$ $p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})=p(v_{i} \mid\text{pa}_{\mathcal{V}}(v_{i}),u_{i};\theta_{f})$ $$

Figure: Figure 2: $p(v_{1}\mid v_{k}^{\prime},u_{1};\theta_{f})$ is equivalent to $p(v_{1}\mid v_{k},u_{1};\theta_{f})$ as they both represent the same query on the same conditional probability table.

The above theorem shows that CGM’s bipartite graph $\mathcal{G}^{\prime}$ between nodes $\mathcal{V}^{\prime}$ and $\mathcal{V}$ can efficiently represent arbitrary long cause-effect chains among interpretable variables (example in Fig. 2). While during training CGMs exploit the efficient bipartite representation, at test time CGMs can be unfolded following the graph as described in Theorem 3.2 (Sec. 3.3 describes how to learn sparse and acyclic dependencies to make the test time unfolding more efficient and interpretable; we provide more details in App. H.6).
This property makes CGMs causally transparent by design, as they can make class predictions through cause-effect chains of endogenous variables, allowing humans to trace each prediction back to its underlying causes within an interpretable parental subgraph.
Notice how—unlike CBMs—Causal CGMs do not require to know in advance cause (concept)-effect (task) relationships, nor require that all causes (concepts) are independent from each other, as cause-effect dependencies among interpretable variables are not given, but learnt during training.

###### Remark 3.3 .

Causal CGMs are designed to address the problem of causal opacity, not causal discovery.
Causal opacity—the problem of understanding a model’s decision-making process— neither implies nor requires causal discovery—the problem of identifying the rules governing the world or the data-generating mechanism. Indeed, the causal structure of a model’s decision-making process does not necessarily mirror the causal structure of the world (Boge, ). We report in appendices extended discussions on CGM’s relations with causal discovery (App. J) and causal representation learning (App. K), and on CGM’s identifiability (App. L).

###### Theorem 3.2 .

###### Remark 3.3 .

### 3.3 Training Causal CGMs

#### Learning sparse, directed, and acyclic structures

In order to construct simple explanations, we can make Causal CGMs’ graphs sparse and acyclic. To this end, we parametrize the bipartite graph $\mathcal{G}^{\prime}$ with an adjacency matrix of learnable weights $M\in\mathbb{R}^{k\times k}$. We initialise the weights $m_{ij}\in M$ based on the conditional entropy between labels $v_{i}$ and $v_{j}$ in order to account for asymmetric concept dependencies (for other initialisation strategies, see App. F). These weights are then fine-tuned through an end-to-end learning process within the Causal CGM framework following common causal priors, where causal graphs are assumed to be sparse, directed, and acyclic (forming a Directed Acyclic Graph or DAG). We introduce a parameter $\gamma\in\mathbb{R}$ to eliminate less significant dependencies and a loss function, as described by Yang et al. (), to enforce the sparsity and acyclicity of the causal graph, ensuring that the adjacency matrix $A$ effectively represents a DAG:

$$ (initialisation) $\displaystyle\quad a_{ij}=-\sum\nolimits_{b\in\{0,1\}}\sum\nolimits_{c\in\{0,1 \}}p(v_{i}=b,v_{j}=c)\log p(v_{i}=b\mid v_{j}=c)$ (2) (sparsity) $\displaystyle\quad A=M\cdot\mathbbm{1}_{M\geq\gamma}$ (3) (acyclicity) $\displaystyle\quad\mathcal{L}_{2}(A)=\text{Tr}\left(\Big{(}I+\frac{\beta}{k}A \cdot A\Big{)}^{k}\right)-k$ (4) $$

where $k$ is the number of endogenous, $\mathbbm{1}$ an indicator function, and $\beta>0$ a scaling hyperparameter.

#### Optimisation problem

We can now state the general learning objective for Causal CGMs.
Given (1) a set of entities represented by their feature vectors $\mathbf{x}\in\mathcal{X}\subseteq X$ (i.e. the input) and (2) a set of annotations for each exogenous variable $v\in\mathcal{V}\subseteq V$ (i.e. the labels),
we wish to find functions $\zeta$, $s$, $f$, together with the adjacency matrix $A$, that maximise the log-likelihood of $v,v^{\prime}$, while observing $x$ (or equivalently $u=\zeta(x)$):

$$ $\mathcal{L}=\overbrace{\mathbb{E}_{u,v^{\prime}\sim p(u,v^{\prime})}[-\log p(v ^{\prime}\mid u)]}^{\text{endogenous copies' prediction}}+\lambda_{1} \overbrace{\mathbb{E}_{v\sim p(v|do(v^{\prime}=v),u)}[-\log p(v\mid do(v^{ \prime}=v),u)]}^{\text{endogenous variables' prediction}}+\lambda_{2} \overbrace{\mathcal{L}_{2}(A)}^{\text{graph priors}}$ $$

where $\lambda_{1,2}$ are hyperparameters balancing optimisation objectives (related ablation studies can be found in App. H.3).
Notice that we use the $do$-operator replacing $\hat{v}_{j}^{\prime}$ with labels $v_{j}$ to minimise leakage and provide better gradients to the endogenous predictor $f$. This enables Causal CGMs to be aware of $do$-operations during training, thus making the model effective in responding to $do$-interventions once deployed.

### 3.4 Causal reasoning and verification with Causal CGM

CGMs support two kinds of interventions: ground-truth interventions, as in CBMs, and do-interventions, as in causal models. In what follows, we first highlight how CGM’s design makes ground-truth interventions more effective and then we describe how do-interventions can be applied in Causal CGMs to verify properties of the model’s decision-making process.

#### Ground-truth interventions

Causal CGMs support “ground-truth interventions” (see Figure 3(b)). Ground-truth interventions are one of the core motivations behind CBMs (Koh et al., ).
Through ground-truth interventions, concept bottleneck models allow experts to improve a CBM’s task performance by rectifying mispredicted concepts at test time, thus significantly improving task performance within a human-in-the-loop setting. In Causal CGMs, however, ground-truth interventions have a potential impact on all endogenous variables descendant of an intervened node, which may include not only nodes corresponding to downstream tasks but also nodes corresponding to a CBM’s intermediate concepts. This enables a single concept ground-truth intervention to potentially improve the prediction of intermediate concepts as well as downstream tasks.

#### Causal reasoning and verification: do-interventions, counterfactuals, and blocking

Causal CGMs can answer interventional and counterfactual queries related to the model’s decision-making process using the do-operator on the unfolded SCM.

Figure: (a) Causal graph.

Do-interventions enable manipulation of a Causal CGM’s decision-making process by changing the value of a specific endogenous variable and observing how it affects other variables’ distributions (see Figure 3(c)).
In Causal CGMs, the effect of the $do$-intervention is analysed through the interventional distribution, denoted as $p(v_{i}|do(v_{j}=\kappa),\text{pa}(v_{i}))$, which describes the distribution of the outcome variable $v_{i}$ after the intervention $do(v_{j}=\kappa)$ has been performed. In particular, in Causal CGMs, the do-operation fixes the value of the intervened variable $v_{i}$ to a fixed constant $\kappa\in\{0,1\}$ and removes all causal dependencies from parent variables by zeroing all values of the $i$-th column of the adjacency matrix:

$$ $\displaystyle\text{do}(v_{j}=\kappa):=\begin{cases}v_{j}:=\kappa,\quad\kappa \in\{0,1\}\\ a_{[:,j]}=0,\quad(\text{implies that: }\text{pa}(v_{j})=\emptyset)\end{cases}$ (5) $$

Causal CGMs also enable to answer counterfactual queries such as “What would the value of the $i$-th variable have been, had the $j$-th variable been $\kappa$, given that we observed $v_{i}$ and $v_{j}$?” (extended motivation in App I). Answering these queries involves three steps (Pearl, ): 1) Abduction: Infer a realisation of exogenous variables that is consistent with the observed $v_{i}$ and $v_{j}$ in the actual causal model. 2) Action: Modify the architecture of the Causal CGM $\mathcal{M}$ into $\mathcal{M}_{v_{j}=\kappa}$ by replacing the structural equation for $v_{j}$ with $\kappa$, to simulate the intervention. 3) Prediction: Compute the value of $v_{i}$ in the modified model $\mathcal{M}_{\kappa}$, representing the counterfactual outcome:

$$ (abduction) $\displaystyle\quad\hat{u}_{j}=p(u_{j}\mid\mathbf{x};\theta_{\zeta})$ (6) (action) $\displaystyle\quad\text{do}(v_{j}=\kappa):=\begin{cases}v_{j}:=\kappa\\ a_{[:,j]}=0\end{cases}$ (7) (prediction) $\displaystyle\quad\hat{v}_{i}=p(\hat{v_{i}}\mid\text{pa}_{\mathcal{V}}(v_{i}), u_{i};\theta_{f})\ \forall i\neq j$ (8) $$

This formalism allows us to not only estimate the effects of hypothetical interventions but also to explore the implications of alternative scenarios on individual outcomes, providing a powerful tool for analysing the model’s decision-making based on interpretable causal structures.

Model verification and blocking
Causal analysis enables the verification of properties of Causal CGMs before deployment. For instance, using only the learnt causal graph, it is possible to prove that an endogenous variable $v_{i}$ is independent of the variable $v_{j}$ by verifying that $v_{j}$ is not among the ancestors of $v_{i}$. Another form of formal verification, which we call “blocking”, employs the $do$-intervention (see Figure 3(d)). Blocking allows one to formally verify the independence of a pair of variables given a sequence of $do$-operations. Given a pair of variables $v_{j}$ and $v_{i}$ such that $v_{j}$ is an ancestor of $v_{i}$, we perform a blocking verification as follows: 1) Block: perform a do-intervention on all child nodes of $v_{j}$, 2) Verify: perform a do-operation on $v_{j}$ itself and observe the impact on $v_{i}$. We can easily verify that the first step makes $v_{j}$ and $v_{i}$ completely independent by observing that the do-operation on $v_{j}$ no longer alters the distribution of $v_{j}$.

## 4 Experiments

Our experiments aim to answer the following questions:

- •
Concept-based performance and interpretability:
Can Causal CGMs match the generalisation performance of equivalent black-box models and existing CBMs? Can Causal CGMs enable more effective ground-truth interventions w.r.t. existing CBMs?
- •
Causal Interpretability: Are Causal CGMs causally interpretable?
Can Causal CGMs effectively block the causal effect of two causally related endogenous variables?

To answer these questions, we use four datasets: (i) Checkmark, a synthetic dataset composed of four endogenous variables; (ii) dSprites, where endogenous variables correspond to object types together with their position, colour, and shape; (iii) CelebA, a facial recognition dataset where endogenous variables represent facial attributes; (iv) CIFAR10, an animal classification dataset where the endogenous variables are extracted automatically following Oikarinen et al. (). Using these datasets, we compare the proposed approach with a black box baseline and state-of-the-art concept-based architectures: Concept Bottleneck Models (CBM) (Koh et al., ), and Concept Embedding Models (CEM) (Espinosa Zarlenga et al., ).
We also compare a version of the proposed method (Causal CGM) where the causal graph is learnt end-to-end w.r.t. with a version (Causal CGM+CD) where the causal graph is extracted from ground-truth labels using a causal discovery algorithm (Lam et al., ) and injected into the model fixing the matrix $A$. Due to the prohibitive computational complexity of the causal discovery method, this was not feasible for the CIFAR-10 dataset We provide further details on our experimental setup and baselines in App. G.

A comprehensive set of experiments is detailed in App. H, where the experiments presented in this section for a subset of the datasets are extended to include all datasets. We also report an analysis of Causal CGMs’ scalability and computational complexity in App. H.5.

We remark that, following the CBM literature, the goal of Causal CGMs is not to increase performance but interpretability. Therefore, the experiments aim to show that our Causal CGMs’ classification performance is competitive against existing black-box models and state-of-the-art concept-based approaches (e.g., within one percentage point) while enabling better visibility and manipulation of its reasoning process.

### 4.1 Key findings

#### Causal CGMs match the performance of
causally opaque models. (Table 1 )

Causal CGMs demonstrate robust generalisation across all datasets, yielding a predictive performance close to that of black-box architectures with an equivalent capacity.
Causal CGMs using a pre-trained causal graph (Causal CGM+CD) tend to have slightly better label accuracy with respect to Causal CGMs where the causal graph is learned end-to-end (Causal CGM).
Causal CGMs’s low variance suggests a consistent robustness with respect to weight initialisations over multiple training runs. Thanks to concept embeddings, concept incompleteness settings do not have a strong impact on Causal CGMs’ performance (results in App. H.4).

**Table 1: Label accuracy ($\uparrow$) is computed on all endogenous variables (concepts and task).**
|  | checkmark | dsprites | celeba | cifar10 | semantic transparency | causal transparency |
| --- | --- | --- | --- | --- | --- | --- |
| Black box | $90.15{\scriptstyle\pm 1.30}$ | $99.53{\scriptstyle\pm 0.05}$ | $79.55{\scriptstyle\pm 0.14}$ | $94.85{\scriptstyle\pm 0.03}$ | ✗ | ✗ |
| CBM | $90.34{\scriptstyle\pm 0.55}$ | $99.55{\scriptstyle\pm 0.07}$ | $79.00{\scriptstyle\pm 0.18}$ | $92.17{\scriptstyle\pm 0.11}$ | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ | ✗ |
| CEM | $89.09{\scriptstyle\pm 1.98}$ | $99.48{\scriptstyle\pm 0.07}$ | $79.17{\scriptstyle\pm 0.26}$ | $92.04{\scriptstyle\pm 0.06}$ | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ | ✗ |
| Causal CGM+CD | $89.43{\scriptstyle\pm 0.93}$ | $99.40{\scriptstyle\pm 0.15}$ | $78.42{\scriptstyle\pm 0.42}$ | N/A | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ |
| Causal CGM | $88.24{\scriptstyle\pm 1.30}$ | $99.44{\scriptstyle\pm 0.11}$ | $78.23{\scriptstyle\pm 0.45}$ | $93.32{\scriptstyle\pm 0.03}$ | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ | ${\color[rgb]{0,0.390625,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.390625,0}\bm{\checkmark}}$ |

#### Ground-truth interventions on Causal CGMs improve both concept and task accuracy as opposed to CBMs (Figures 5 , 5 )

In Causal CGM, the causal graph induces a natural strategy for ground-truth interventions. Indeed, the causal graph narrows down the set of variables to intervene upon: for any given node, we can just fix mispredicted labels of the node’s ancestors as intervening on other nodes will not have any impact. This property significantly decreases the required number of interventions to achieve a desired outcome (e.g., to increase a downstream task accuracy), as shown in App. H. Another advantage of Causal CGM consists in the hierarchical nature of inference which allows ground-truth interventions to impact all endogenous variables descendant of an intervened node. In particular, ground-truth interventions may affect not only nodes corresponding to downstream tasks (as in CBM and CEM), but also nodes corresponding to a CBM’s intermediate concepts.
We experimentally verify this property and its impact by calculating—for nodes that were not intervened upon (including both concepts and tasks)—the change in accuracy before and after ground-truth interventions were applied on their ancestors.
Our results (Figure 5 and App. H.1) show that Causal CGM improves nodes accuracy by $\sim 15$ percentage points after only 7 ground-truth interventions on CelebA. CBM and CEM, instead, achieve a similar performance only after intervening on all concepts as their architecture assumes all concepts to be mutually independent. Focusing solely on concept-level analysis, Causal CGM enhances concept accuracy by up to $\sim 25$ percentage points, compared to CBM and CEM, where a concept intervention does not impact the accuracy of other concepts (Figure 5). Causal CGM’s advantage increases with the number of concepts and connections, as a single intervention can impact a higher number of nodes in the causal graph.

Figure: Figure 4: Impact of ground-truth interventions on non-intervened nodes ($\uparrow$). Intervention on Causal CGM improves both concept and task accuracy.
Refer to caption: x1.png

#### Causal CGMs’ endogenous predictors are causally interpretable (Figures 6(a) , 6(b) )

In Causal CGM the decision-making process is causally interpretable and can be analysed by visualising the learnt causal graph and structural equations as in a structural causal model, as shown in Figure 6(a) for CelebA and in Figure  6(b) for Cifar10 (more results in App. H.2). The first image shows how Causal CGM exploited known biases in CelebA to infer facial attributes. For instance, Causal CGM predicts the attribute “wearing lipstick” directly from the attribute “attractive” and indirectly from attributes such as “smiling” or “high cheek”, all attributes that are known to be strongly correlated with each other in CelebA (Ramaswamy et al., ; Wang & Russakovsky, ). Similarly, in the CIFAR-10 dataset, where concepts are automatically extracted, the model learns meaningful relationships between concepts (Figure 6(b)), such as the presence of a “port” implying the presence of a “dock”, or the connection between a “beak” and a “bird”. We can also quantify the strength of the causal dependency between two nodes by computing the probability of necessity and sufficiency (PNS) (Pearl, ). In the figure, we represent the PNS w.r.t. the leaf node by colouring each node with a different shade of orange (more PNS of the other datasets in App. I). This shows how, for Causal CGM, the attribute “heavy makeup” has the strongest impact on the leaf node. This high degree of causal transparency allows users to interpret Causal CGM’s inference and can be eventually exploited to identify potential biases, thereby supporting the assessment of the model’s counterfactual fairness (an example of generated counterfactuals in Table  2).
As a result, users can intervene directly on the causal structure of the decision-making process and remove biases using do-interventions, as shown in the next paragraph.

Figure: (a) CelebA
Refer to caption: x3.png

**Table 2: Examples of counterfactuals generated on CelebA obtained via do-interventions (intervened variables are marked in red, changed variables are underlined).**
|  | Endogenous variables’ activations |
| --- | --- |
| Initial Predicted State | Smiling, Not Attractive, Mouth Slig, High Cheek, Not Wearing Li, Not Heavy Make, |
| Not Wavy Hair, Not Big Lips, Not Oval Face, Not Makeup, Not Fem Model |  |
| What if I would wear lipstick? | Smiling, Attractive, Mouth Slig, High Cheek, Wearing Li, Heavy Make, |
| Not Wavy Hair, Not Big Lips, Not Oval Face, Not Makeup, Not Fem Model |  |
| What if I would wear lipstick and also makeup? | Smiling, Attractive, Mouth Slig, High Cheek, Wearing Li, Heavy Make, |
| Not Wavy Hair, Big Lips, Not Oval Face, Makeup, Fem Model |  |

#### Causal CGMs can make two causally-related variables causally independent by blocking all paths between these variables (Table 3 )

The causal transparency of the proposed approach allows users to modify the model’s decision-making process (e.g., to de-bias the model’s inference) by using do-interventions. In particular, we can make two causally-related variables $i$ and $j$ causally independent by blocking all paths between the cause $i$ and the effect $j$. We experimentally verify this property and its impact by computing the Residual Concept Causal Effect i.e., the ratio between the Concept Causal Effect (CaCE) (Goyal et al., ) obtained after and before blocking. The optimal value of this metric is zero, corresponding to perfect causal independence between $i$ and $j$ (i.e., the optimal value for a de-biasing operation).

The results show that, in Causal CGMs, blocking a variable in the causal graph always yields a perfect Residual Concept Causal Effect of zero across all datasets.
In contrast, applying the same procedure in CBMs leads only to a negligible reduction in the average causal effect to $3$ percentage points. CEMs not only fail to reduce the causal effect to zero but, in some cases, even increase the causal influence. These results underscore how Causal CGMs transparency enable users to manipulate the model’s decision-making process to achieve desired outcomes, as opposed to existing CBMs.

**Table 3: Residual Concept Causal Effect ($\downarrow$) between causally-related variables having blocked all paths between the two variables with do-interventions on the causal graph. The optimal value is zero corresponding to perfect causal independence. Values above $100\%$ mean that the causal effect increased instead of decreasing.**
|  | checkmark | dsprites | celeba | celeba |
| --- | --- | --- | --- | --- |
| Black box | N/A | N/A | N/A | N/A |
| CBM | $97.99{\scriptstyle\pm 5.64}$ | $100.00{\scriptstyle\pm 0.70}$ | $97.84{\scriptstyle\pm 2.13}$ | $100.00{\scriptstyle\pm 0.00}$ |
| CEM | $102.58{\scriptstyle\pm 12.95}$ | $100.00{\scriptstyle\pm 4.62}$ | $106.00{\scriptstyle\pm 0.50}$ | $100.00{\scriptstyle\pm 0.00}$ |
| Causal CGM+CD | $0.00{\scriptstyle\pm 0.00}$ | $0.00{\scriptstyle\pm 0.00}$ | $0.00{\scriptstyle\pm 0.00}$ | N/A |
| Causal CGM | $0.00{\scriptstyle\pm 0.00}$ | $0.00{\scriptstyle\pm 0.00}$ | $0.00{\scriptstyle\pm 0.00}$ | $0.00{\scriptstyle\pm 0.00}$ |

## 5 Discussion

#### Related works

Causal CGMs present substantial advantages compared to the state of the art.
Compared to most causal feature-attribution methods (e.g., (Chattopadhyay et al., )), Causal CGMs focus on high-level human interpretable concepts.
Causal CGMs differ from existing CBMs in their approach to intervention and causal relationships. The causal structure of the inferences that Causal CGMs deliver does not subsume weak forms of causal dependencies as in existing CBMs, where concepts are all direct causes of the target and causally independent of each other. This way, Causal CGMs enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections but also the accuracy of the explanation provided for a specific instance.
Closest to Causal CGMs in concept usage are *post-hoc* causal concept-based explainability techniques, like DiConStruct (de Oliveira Moreira et al., ) and conceptual counterfactual explanations (Abid et al., ). These methods build surrogate causal models to emulate a target black box model’s predictive behaviour. However, as Rudin () notes, convergence in predictive behaviour does not ensure structural similarity in decision-making processes, making surrogate models as explanatory proxies questionable.
Causally interpretable *by-design* architectures, such as Causal CGMs, do not suffer from this issue.

#### Limitations and future works

Our method’s limitations are mainly derived from limitations inherent to CBMs and causal reasoning. The quality of learnt causal graphs mainly depends on the quality of the dataset and its annotations. Missing and noisy labels might lead to suboptimal graphs. Similarly to generalised PGMs, Causal CGMs can be easily unfolded when the final graph is acyclic. Variables in cycles can still be inferred, but require special unfolding techniques which might be explored in future works. Finally, a Causal CGM’s learnt graph does not necessarily represent the causal mechanisms of the data-generating process, but rather that of the model’s inference. This makes our models suitable to be verified and controlled but not necessarily to understand their training data’s distribution.

## 6 Conclusion

Causal opacity represents a key open challenge at the intersection of deep learning, interpretability, and causality. Causal CGMs address this challenge by employing an architecture which makes the decision-making process causally transparent by design. This makes Causal CGMs reliable and verifiable compared to both usual DL architectures and standard (non-causal) CBMs. The results of our experiments show that Causal CGMs support the analysis of interventional and counterfactual scenarios—thereby improving the model’s causal interpretability and supporting the effective verification of its reliability and fairness—and enable human-in-the-loop corrections to mispredicted intermediate reasoning steps, boosting not just downstream accuracy after corrections, but also accuracy of the explanation provided for a specific instance. As a result, advancing this research line could significantly improve the reliability and verifiability of concept-based deep learning models, thus supporting their deployment in real-world applications.

## Appendix A Causal explainability, causal opacity, and causal discovery

Deep Learning (DL) models have a pervasive impact on many areas of contemporary research and society (Baldi, ).
Despite this success, there is growing concern about the widespread real-world application of DL, particularly in sensitive domains (Dignum, ).
This is partly due to lack of *causal explainability* of these models, which undermine their robustness, fairness and generalisability in out-of-distribution context (Schölkopf et al., ).
Causal explainability, in particular, is a multi-faced issue, which includes a variety of different, albeit related, problems (Termine & Primiero, ).
One such problem is that of *causal discovery*
and concerns the possibility of using a model to detect and understand the *causal mechanisms* of the data generating process (Pearl & Mackenzie, ; Pearl, ; Schölkopf et al., ; Kaddour et al., ).
An equally important but distinct problem is that of *causal opacity*, which denotes the difficulty of users to grasp and understand the “hidden” *causal structure* that underlines the predictions delivered by a given model (see Figure 1(a)).
This problem can be better understood in the light of Pearl’s framework of causality (Pearl, ; ), which relates causal understanding to the ability of an agent to answer *what-if* type of questions.
In particular, Pearl identifies three different kinds of what-if questions organised in a hierarchy of levels. At the bottom level we have *observational questions*, which concern the actual state of affairs one can observe in data. At the intermediate level we find *interventional questions*, which concern the effects of intervening on the actual state of affairs and fix the value of some variable to a pre-defined one. Finally, the top level encompasses *counterfactual questions*, which concern an hypothetical state of affairs that could have been occured but it is not the actual one.
The three levels provide a measure of the causal understanding that an agent has of a given target-system. Agents that answer only observational questions do not possess any causal understanding of the target system; agents with an intermediate level of causal understanding can answer interventional questions, whereas a complete causal understanding is required for counterfactual questions.

In this paper, we propose to involve Pearl’s hierarchy to define and measure the causal opacity/transparency of a DNN model based on the ability of its users to answer interventional and counterfactual questions
concerning the structure of its inferential behaviour (e.g., “what happens if I fix the feature *age* to a value greater than $50$?” or “what the model’s prediction would have been if the feature *age* had taken a value greater than $50$ instead of lower then or equals to $50$?”).
In this regard, complete causal transparency require that users can answer both interventional and counterfactual questions concerning the model’s inferential behaviour,
whereas if only observational questions can be addressed (i.e., questions regarding the input-output associations of the model), then the model results completely causally opaque: this is notably the case of the majority of DNNs considered in contemporary AI research.

Notice that causal opacity is fundamentally distinct from the problem of causal discovery, which concerns the causal structure *of the world* and not that of a model’s inference. To clarify this point, consider a simple rules-based model as the one described in Fig. 7.

Figure: Figure 7: Rules-based Model $M_{1}$
Refer to caption: extracted/5913344/RulesBasedModel.jpeg

This model is causally transparent as users can easily compute interventional and counterfactual questions concerning its inferential behaviour (e.g., “what if the age of the patient would had been $\geq 50$ instead of $<50$”?). However, causal transparency does not imply that the causal structure of the model’s inference (Fig.8(a)) resemble the causal structure of the world (Fig. 8(b)), which remains a very distinct animal.

Figure: (a)

To ensure that a model is causally transparent and matches the causal structure of the world, indeed, a suitable combination of causal transparency and causal discovery techniques is usually required. However, causal transparency *per-sé* is a fundamental requirement and sufficient for a variety of tasks, such as
establishing the model’s fairness (especially within the popular framework of *counterfactual fairness*  (Kusner et al., )).

Figure: (a)
Refer to caption: extracted/5913344/RulesBasedModel2.jpeg

Consider the model in Fig. 9, which is a slightly modified version of the model depicted in Fig. 7. The causal transparency of the model makes it easy for users to verify the un-fairness of its inferential behaviour (stemming from the fact that the model uses the variable *gender* to predict the variable *income*), and implement suitable corrections (e.g. remove the causal link between gender and income).

## Appendix B Derivation of Causal CGMs

In the absence of causal priors, Causal CGMs need to learn causal dependencies between high-level features that lead to the model becoming more effective in solving its designated task. As a result, we can formally describe Causal CGMs using a generalised probabilistic graphical model (PGM) that extends a traditional PGM by allowing cycles:

###### Definition B.1 (Generalized Causal Concept Graph Model) .

Given an observed input feature $x$, a set of $k\in\mathbb{N}$ latent factors $u_{i}\in\mathcal{U}$ each associated with a high level interpretable variable $v_{i}\in\mathcal{V}$, a *Generalized Causal Concept Graph Model* is the generalised probabilistic graphical model (PGM) $G=(\mathcal{N},\mathcal{E})$ with nodes $\mathcal{N}=\{x\}\cup\mathcal{U}\cup\mathcal{V}$ and edges $\mathcal{E}=\big{\{}(x,u_{i})\mid u_{i}\in\mathcal{U}\big{\}}\cup\big{\{}(u_{i
},v_{i})\mid i\in\{1,\cdots,k\}\big{\}}\cup\{(v_{i},v_{j})\mid v_{i},v_{j}\in
\mathcal{V},v_{i}\neq v_{j}\}$ which represents the joint conditional distribution $p(v,u\mid x)$:

$$ $x$ $u_{1}$ $u_{2}$ $\dots$ $u_{k}$ $v_{1}$ $v_{2}$ $\dots$ $v_{k}$ (9) $$

The cyclical nature of Causal CGMs comes from the necessity to model all possible dependencies among variables $v_{i}$. Notice that the model is uniquely identified by the set of all conditional probability distributions corresponding to the arrows in the graph. Unfortunately, in generalised PGMs, the model does not easily factorise in terms of such distributions due to the cycles. To deal with cycles while maintaining the independencies induced by the graph structure, we can use an unfolding semantics for cyclical PGMs (Baier et al., ). This semantics is based on the choice of a “cutset” i.e., a specific set of nodes $\mathcal{Q}\subseteq\mathcal{N}$ in the PGM such that every cycle in the PGM contains at least one node in $\mathcal{Q}$. Intuitively, by unfolding the nodes in the cutset, all cycles are broken leaving us with a standard acyclical PGM. The consistency between the semantics of the original cyclical PGM and the unfolded acyclical PGM is only valid in the limit of infinite unfolding (Baier et al., ). However, when computing the likelihood of an observed complete set of variables $v_{i}\in\mathcal{V}$, modelling one single unfolding (i.e., a single transition) suffices for learning the conditional probability distributions among the variables in $\mathcal{V}$, as all the variables become conditionally independent on each other. As a result, we can define a Dissected Causal CGM as the one-step unfolding of the Causal CGM in Definition B.1:

###### Definition B.2 (Dissected Causal CGM) .

Given a Causal CGM, let $\mathcal{V}^{\prime}=\mathcal{V}$ be the cutset. Then, the dissected Causal CGM $\mathcal{G}=(\mathcal{N}\cup\mathcal{V}^{\prime},\mathcal{E}_{\mathcal{V}^{
\prime}})$ is an acyclic PGM obtained by extending the generalised PGM by (i) adding a copy of all cutset nodes $\mathcal{V}^{\prime}=\{v_{i}\mid v_{i}\in\mathcal{V}\}$, (ii) adding a new set of edges directed from parents of cutset’s nodes to the generated copies $\mathcal{V}^{\prime}$ i.e., $\mathcal{E}_{\mathcal{V}^{\prime}}=\{(a,b)\mid(a,b)\in\mathcal{E},b\in\mathcal
{V}^{\prime}\}\cup\{(a,b)\mid(a,b)\in\mathcal{E},b\notin\mathcal{V}^{\prime}\}$, and (iii) defining an initial probability distribution for the new copies $p(v^{\prime}|u)$ given the latent variables.
The resulting PGM, factorised as $p(v,v^{\prime},u\mid x)=p(v\mid v^{\prime},u)p(v^{\prime}\mid u)p(u\mid x)$, is:

$$ $x$ $u_{1}$ $u_{2}$ $\dots$ $u_{k}$ $v_{1}^{\prime}$ $v_{2}^{\prime}$ $\dots$ $v_{k}^{\prime}$ $v_{1}$ $v_{2}$ $\dots$ $v_{k}$ (10) $$

###### Definition B.1 (Generalized Causal Concept Graph Model) .

###### Definition B.2 (Dissected Causal CGM) .

## Appendix C Proof of theorem 3.2

###### Proof.

Let us consider the graph $\mathcal{G}^{*}=(\mathcal{V}\cup{roots(\mathcal{G}^{\prime})},\{(v_{i},v_{j})
\mid v_{i},v_{j}\in\mathcal{V},(v_{i},v_{j})\in\mathcal{E}_{\mathcal{G}^{
\prime}}\}\cup{(v_{i}^{\prime},v_{j})|v_{i}^{\prime}\in roots(\mathcal{G}^{
\prime})})$. We will first prove that this graph is isomorphic to $\mathcal{G}^{\prime}$ for all nodes $\mathcal{V}$ and then that the conditional probabilities are equivalent.

The graph $\mathcal{G}^{\prime}$ is defined with nodes $\mathcal{V}\cup\mathcal{V}^{\prime}$, where $\mathcal{V}$ represents the endogenous variables and $\mathcal{V}^{\prime}$ represents copies of the same high-level variables. Hence, the set of nodes $\mathcal{V}$ is the same for both graphs by definition. In $\mathcal{G}^{\prime}$, the edge set $\mathcal{E}_{\mathcal{G}^{\prime}}$ consists of edges $(v_{i}^{\prime},v_{j})$ between nodes in $\mathcal{V}^{\prime}$ and $\mathcal{V}$. Specifically, each $v_{i}^{\prime}\in\mathcal{V}^{\prime}$ is connected to $v_{j}\in\mathcal{V}$, where $v_{i}^{\prime}\neq v_{j}$. In $\mathcal{G}^{*}$, the edge set $\mathcal{E}_{\mathcal{G}^{*}}$ consists of:

- •
edges $(v_{i},v_{j})$ between nodes in $\mathcal{V}$, which are present if $(v_{i},v_{j})\in\mathcal{E}_{\mathcal{G}^{\prime}}$,
- •
edges $(v_{i}^{\prime},v_{j})$ where $v_{i}^{\prime}\in\text{roots}(G^{\prime})$, representing connections between the root nodes and their corresponding child nodes.

which represent a one-to-one correspondence between the edges in $\mathcal{E}_{\mathcal{G}^{\prime}}$ and $\mathcal{E}_{\mathcal{G}^{*}}$. Thus, the two graphs are isomorphic.

Now, we prove that for all nodes $v_{i}\in\mathcal{V}\setminus(\text{roots}(\mathcal{G}^{\prime})\cup\text{ch}(
\text{roots}(\mathcal{G}^{\prime})))$, the conditional probabilities are equivalent:
$p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})=p(v_{i}
\mid\text{pa}_{\mathcal{V}}(v_{i}),u_{i};\theta_{f})$.
In $\mathcal{G}^{\prime}$, the conditional distribution of a node $v_{i}\in\mathcal{V}$ depends on its parents $\text{pa}_{\mathcal{V}^{\prime}}(v_{i})$, which are nodes in $\mathcal{V}^{\prime}$, as well as the latent factors $u_{i}$. Since $\mathcal{V}=\mathcal{V}^{\prime}$ by Def. 3.1 and the graphs $\mathcal{G}^{\prime}$ and $\mathcal{G}^{*}$ are isomorphic for all $\mathcal{V}$ nodes, the parent set $\text{pa}_{\mathcal{V}^{\prime}}(v_{i})$ in $\mathcal{V}^{\prime}$ corresponds directly to the parent set $\text{pa}_{\mathcal{V}}(v_{i})$ in $\mathcal{V}$, up to the bijection between the two sets.
Under full observability (the training conditions of CGMs), conditional queries of the form $p(v|\text{ all but }v)$ are just inspections of the conditional probability table. As a result, $p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})$ and $p(v_{i}\mid\text{pa}_{\mathcal{V}}(v_{i}),u_{i};\theta_{f})$ correspond to repeating the same query on the same conditional probability table.
Therefore, for all nodes $v_{i}\in\mathcal{V}\setminus(\text{roots}(\mathcal{G}^{\prime})\cup\text{ch}(
\text{roots}(\mathcal{G}^{\prime})))$, we can write:
$p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})=p(v_{i}
\mid\text{pa}_{\mathcal{V}}(v_{i}),u_{i};\theta_{f})$.
∎

###### Proof.

## Appendix D Neural parametrization of Concept Graph Models

We can interpret the factors of a dissected Causal CGM as follows:
$p(u_{j}\mid\mathbf{x};\theta_{\zeta})$ is the exogenous encoder, i.e., a deterministic distribution that is parametrised by a neural network $\zeta:X\to U$. In CBMs, this function represents the input encoder.
The exogenous encoder $\zeta$ generates exogenous variables $u_{i}\in\mathcal{U}$ mapping raw input features $\mathbf{x}$ (e.g., an image’s pixels) to latent embeddings $\hat{\mathbf{u}}_{i}\in\mathbb{R}^{q},\ q\in\mathbb{N}$. In practice, this process mirrors the generation of context vectors in Concept Embedding Models to avoid information bottlenecks which may negatively affect the model’s accuracy (Espinosa Zarlenga et al., ). First, the encoder $\psi:X\rightarrow H$ maps raw features to a latent code $\mathbf{h}\in H$. Then, a pair of neural networks $\{\phi_{i}^{+},\phi_{i}^{-}\}$ map the latent code into two different embeddings whose concatenation $[\phi_{i}^{+}(\mathbf{h}),\phi_{i}^{-}(\mathbf{h})]^{T}$ corresponds to the exogenous variable $\mathcal{U}_{i}$ of the $i$-th concept:

$$ (exogenous variables) $\displaystyle\quad\hat{\mathbf{u}}_{i}=\zeta(\mathbf{x})=[\phi_{i}^{+}(\psi( \mathbf{x})),\phi_{i}^{-}(\psi(\mathbf{x}))]^{T}.$ (11) $$

In contrast, $p(v_{j}^{\prime}\mid u_{j};\theta_{s})$ is the copies predictor. This is the product of $k$ independent Bernoulli distributions whose logits are parameterised by a neural network $s:U\to V$. In CBMs, the composition of the exogenous encoder and the concept predictor is often called *concept encoder* $g=\zeta\circ s$. In causal methods, this function represents a (supervised) *causal feature learner* (Kaddour et al., ).
The copies predictor $s$ generates endogenous copies $v_{i}^{\prime}\in\mathcal{V}$ from latent embeddings $\hat{\mathbf{u}}_{i}$. This is obtained using a neural network classifier $s:U\to V$ as a scoring function as in (Espinosa Zarlenga et al., ):

$$ (endogenous copies) $\displaystyle\quad\hat{v}_{i}^{\prime}=s(\hat{\mathbf{u}}_{i})=\sigma\big{(}W_ {s}\hat{\mathbf{u}}_{i}+\mathbf{b}_{s}\big{)}$ (12) $$

Finally, $p(v_{i}\mid\text{pa}_{\mathcal{V}^{\prime}}(v_{i}),u_{i};\theta_{f})$ is the endogenous predictor. This distribution is the product of $k$ independent Bernoulli distributions whose logits are parameterised by a neural network $f:V^{k}\times U\to V$. The input to this function $\text{pa}_{\mathcal{V}^{\prime}}(v_{i})$ (representing direct causal dependencies) is weighted
by a learnable adjacency matrix $M\subseteq\mathbb{R}^{k\times k}$, where each learnable weight $a_{ij}$ models the strength of the dependency of $v_{i}$ from its parents $v_{j}^{\prime}$. In CBMs this function is called a *task predictor* (Koh et al., )
and intuitively represents the
analogous of the structural equations that model
*causal mechanisms* in SCMs (Kaddour et al., ).
The endogenous predictor generates the endogenous variables $\hat{v}_{i}\in\mathcal{V}$ by considering exogenous variables $\hat{\mathbf{u}}_{j}$ and copies $\hat{v}_{j}^{\prime}$ with $j\neq i$. First, the function $\omega:V\times U\to\mathbb{R}^{z}$ generates endogenous embeddings $\hat{\mathbf{v}}_{j}^{\prime}$ using exogenous variables $\hat{\mathbf{u}}_{j}$ and copies $\hat{v}_{j}^{\prime}$, following (Espinosa Zarlenga et al., ). Then, all endogenous embeddings are weighted by the strength of the dependency $a_{ij}$ and aggregated using a deepset-like neural network $f_{i}:\mathbb{R}^{z\times k}\to[0,1]$ which maps endogenous embeddings to endogenous predictions:

$$ (endogenous embeddings) $\displaystyle\quad\hat{\mathbf{v}}_{j}^{\prime}=\omega(\hat{v}_{j}^{\prime}, \hat{\mathbf{u}}_{j})=\hat{v}_{j}^{\prime}\phi_{j}^{+}(\psi(\mathbf{x}))+(1- \hat{v}_{j}^{\prime})\phi_{j}^{-}(\psi(\mathbf{x}))$ (13) (endogenous variables) $\displaystyle\quad\hat{v}_{i}=f_{i}\left(\{a_{ij}\hat{\mathbf{v}}_{j}^{\prime} \}_{j\in\{1,\dots,k\}}\right).$ (14) $$

In order to learn explicit structural equations, existing logic-based aggregation methods can be used from the concept literature (Barbiero et al., ; ). App. F describes in more detail their adaptation in Causal CGM.

## Appendix E Unfolding CGMs

Notice how learning a DAG together with Definition B.2 allows to unfold a Causal CGM’s endogenous predictor applying a directed message passing on the associated structural causal model $\mathcal{M}$, ensuring that the values of endogenous variables are derived solely from the nodes that are their ancestors on the causal graph (see Figure 10). As a first step, we compute the exogenous variables for all nodes. We then predict the values of endogenous variables in root nodes in the learned DAG from their corresponding exogenous variables. Following this, we can generate the endogenous embeddings for root nodes and aggregate endogenous embeddings to compute the value of endogenous variables of each child node. We repeat this process until all leaf nodes of the graph are reached. We can obtain this by replacing in Eq. 13 the endogenous copies $\hat{v}_{j}^{\prime}$ with the parents of the endogenous variable $v_{i}$:

$$ (unfolding) $\displaystyle\quad\hat{v}_{i}=f_{i}\left(\{a_{ij}\omega(\hat{v}_{j},\hat{ \mathbf{u}}_{j})\}\right),\quad\forall i,j\in\mathcal{V}$ (15) $$

Note that this causal unrolling guarantees two key properties (see Figure 3(a) and Fig. 10): (1) modifying the value of a cause (parent node) will impact the effect (child node) in our model, (2) conversely, intervening upon an effect does not alter the cause. This is because, in our model, information flows sequentially following the graph, mirroring the fundamental nature of causal effects. Consequently, this layer not only facilitates the computation of task predictions but also enables the exploration of causal relationships through do-interventions and counterfactual analysis. In App. H.6 we explain why $1$-step unfolding of the causal graph during training is equivalent to any k-step ($k>1$) and empirically show related results.

Figure: Figure 10: Unfolded Causal CGM’s endogenous predictor.
Refer to caption: extracted/5913344/figs/h_dsprites.png

## Appendix F Architecture

#### Initialisation of adjacency matrix

Causal CGM provides versatile initialisation options for the adjacency matrix $A$, tailored to specific scenarios. In certain instances, weights can be derived from domain expertise or provided along with training data and labels. Without such information, weights can be directly inferred from training labels through causal structural learning algorithms (Kaddour et al., ) as a preliminary step. These approaches guide the model towards a predetermined decision-making pathway. Alternatively, weights can be learnt concurrently during the Causal CGM training phase, as outlined in Section 3.3. These weights may be initialised either randomly or based on the conditional entropy between labels, providing a better starting point. Additionally, a hybrid approach is feasible, where certain elements in $A$ are fixed while others remain trainable. For instance, a causal structural learning algorithm might yield a Partial Ancestral Graph (PAG) with undirected edges, allowing for the definition of directed edges and learning the direction for others to avoid cycle formation.

#### Causal mechanisms

In Causal CGM, the function $f_{i}$ corresponds to a causal mechanism in a SCM. Such mechanisms are typically formalised via structural equations.
For instance, linear models are a common choice for label predictors in Concept Bottleneck Models (Koh et al., ) where endogenous embeddings are aggregated using a permutation invariant aggregator function $\oplus$ (such as the element-wise maximum, or sum):

$$ $\hat{v}_{i}=\sigma\left(W_{i}\bigoplus_{j\in\{1,\dots,k\}}a_{ij}\hat{\mathbf{v }}_{j}+\mathbf{b}_{i}\right)$ (16) $$

However, other options are also available to increase the expressiveness and interpretability of the decision-making process, such as Deep Concept Reasoning (Barbiero et al., ) class predictors, which build logic-based formulae to obtain class label predictions using endogenous embeddings:

$$ $\hat{v}_{i}\leftarrow\bigvee_{\mathbf{x}\in X_{\text{train}}}\bigwedge_{j\in\{ 1,\dots,k\}}l_{j}(\mathbf{x})=\bigvee_{\mathbf{x}\in X_{\text{train}}} \bigwedge_{j\in\{1,\dots,k\}}(\rho_{ij}(a_{ij}\hat{\mathbf{v}}_{j})\iff\hat{v} _{j}^{\prime})$ (17) $$

where $l_{j}$ denotes the literal of relevant $\hat{v}_{j}^{\prime}$ representing the variable’s sign or “polarity” in the logic rule (i.e., either $v_{j}$ or $\neg v_{j}$). For example, given three variables $v_{1},v_{2},v_{3},v_{4}$, DCR can predict the endogenous variable $v_{2}$ using the rule $v_{2}\leftarrow(v_{1}\land\neg v_{3})\lor(\neg v_{1}\land v_{3})$ which highlights the underlying causal mechanism linking $v_{1},v_{3},v_{4}$ to $v_{2}$ (notice how DCR can also learn to remove irrelevant variables such as $v_{4}$).

#### Compositional generalisation

The training procedure of Causal CGMs is highly parallelizable and modular as only direct connections need to be trained together (e.g., $a\rightarrow b$ and $b\rightarrow c$), while the model takes care of distant connections in an indirect way. For instance, the connection $a\rightarrow b\rightarrow c$ can be obtained as a composition of two different independent training procedures for $a\rightarrow b$ and $b\rightarrow c$. As a result, it is trivial for a Causal CGM to make the causal graph grow even at test time (see Figure 11). This can be done by composing two different graphs obtained by independent training procedures, encoders, datasets, or data types. We note that this is not possible in standard CBMs, which need to re-train the task predictor from scratch whenever new concepts or tasks are added to the mix. Moreover, this modularity enables a form of out-of-distribution compositional generalisation as it creates new distant connections between variables that were never part of the same training procedure (e.g., $a$ and $c$ in the previous example).

Figure: Figure 11: Compositional generalisation in Causal CGMs: two different Causal CGMs architectures are trained independently and then composed only at test time, thus creating a larger graph and allowing out-of-distribution causal inference.
Refer to caption: x5.png

## Appendix G Experimental setup

### G.1 Datasets

In our experiments we use three different datasets:

- •
Checkmark — The dataset consists of tabular data with three features, each ranging from $-1$ to $1$ (denoted as $a$, $b$, and $c$). The target variable $d$ can be either 0 or 1. Each feature is annotated with a concept that indicates whether it is positive or negative. The dataset also incorporates causal relationships among the features. For example, feature $c$ is defined as the inverse of feature $b$. The target $d$ is set to 1 when both features $a$ and $b$ are positive. This data set is used to test our hypothesis in a straightforward and controlled setting.
- •
dSprites (Matthey et al., ) — The dataset comprises images featuring one of three objects (square, heart) in various positions and sizes. The defined concepts include: (1) object shape (square or heart), (2) object size (small or large), (3) vertical position (top or bottom), (4) horizontal position (left or right), (5) object colour (red or blue). Based on these, causal relationships and a binary classification task are established: if the object is a heart on the right side, it is large; if a heart is at the top of the image, it is red; the label is positive if the object is both red and large.
- •
CelebA (Liu et al., ) — The CelebA dataset features celebrity images annotated with various attributes, including lipstick presence, gender, facial shape, and hair type. Gender is used as the classification label. This dataset is chosen for the presence of correlations and biases, such as the association between wearing lipstick and being identified as female.
- •
CIFAR-10 (Krizhevsky, ) — The CIFAR-10 dataset consists of 60,000 colour images in 10 different classes, such as cats, dogs, and cars, with no pre-defined concepts. To extract concepts, we follow the Label-Free Concept approach described in Oikarinen et al. (), where the concept set is extracted using auxiliary Vision Language Models (VLS), which are then used to establish causal relationships and perform classification tasks. This dataset is used to test our model’s ability to handle complex visual features in a setting where concepts are not obtained through human annotations.

### G.2 Baselines

We evaluated our approach against three established baselines:

- •
Black Box: This model employs a single predictor that processes the input to simultaneously predict the task label and all relevant concepts. It lacks interpretability and does not differentiate between the importance of task labels and concepts.
- •
Concept Bottleneck Model (CBM) (Koh et al., ): This model first uses a concept predictor to infer concepts from the initial input, followed by a task label prediction based on these concepts. It is designed to be interpretable and treats concepts as significant informational to predict the task label.
- •
Concept Embedding Model (CEM) (Espinosa Zarlenga et al., ): Comprising $n$ context encoders, one for each concept, this model predicts each concept based on its respective context before predicting the final task label. It treats concepts and task labels in the same way as CBM.

### G.3 Experiments

In our experiments, we evaluate our approach by examining four key dimensions: (i) performance accuracy, (ii) influence of ground-truth interventions, (iii) identification of causal structures, and (iv) blocking for the influence of one variable on another.

To evaluate the first dimension, we conducted a comparative analysis of our approach (using both a learned and a predefined graph) against Black Box, CBM and CEM. This was to determine if graph-based inference would decrease model performance. For this assessment, we calculated the model’s accuracy in predicting all concepts and the task. Typically, these metrics are calculated independently; however, in our study, we treated tasks and concepts equivalently, considering them collectively as labels.

In the second aspect, we evaluate our approach by comparing it against CBM and CEM in terms of response to ground-truth interventions. Enhancing the impact of interventions in the Concept-Based Model is crucial for improving the role of humans in the loop. In our experiments, we initially perturbed the inputs to reduce label prediction accuracy, following methodologies established in prior research (Espinosa Zarlenga et al., ). Subsequently, we implemented interventions on the most inaccurately predicted concepts in CEM and CBM. This intervention strategy is considered highly effective, as noted in (Shin et al., ). For the Causal CGM, interventions began with concepts that have a higher number of descendant nodes in the model’s graph, aiming to maximise the intervention’s effectiveness. To assess this dimension, we measured the change in accuracy for non-intervened labels before and after the interventions on $n$ concepts (Delta Label Accuracy).

In the third aspect, we visualise the DAG utilised during the inference stage by Casual CEM and derive the corresponding logic equations. We generate Sum of Product logic rules from a table that lists all possible combinations of input concept values alongside the most frequent prediction for each combination derived from the training set, similarly to what done by (Ciravegna et al., ). It is crucial to note that while these logic rules are general for the model decision-making process, exogenous information may alter predictions for particular instances.

In the final dimension of our analysis, we compare Causal CGM, which operates on a specified graph, against CBM and CEM in terms of their efficacy in mitigating a variable’s influence on the task. Specifically, we perturb the prediction of a concept and then, following the graph structure utilised by Causal CGM, we intervene with the ground truth label on all descendant nodes (blocking). Consequently, in Causal CGM, all links between the altered concepts and the labels are deleted, a feat unachievable in CBM and CEM. To assess this characteristic, we calculated the Residual Concept Causal Effect, ratio of the Concept Causal Effect (Goyal et al., ) post- and pre-application of the blocking techniques. Ideally, this ratio should be zero, indicating that after blocking, the altered node’s value no longer influences the outcome of the task.

### G.4 Implementation details

Additional details
To maximise the efficacy of interventions in Causal CGM, the second term of the loss can be regularised to maximise the average Causal Concept Effect (CaCE) (Goyal et al., ) as follows:

$$ $\mathcal{L}_{\text{CaCE}}=\frac{1}{n}\sum_{i=0}^{n}|p(v_{i}|do(v_{i,r}=1)-p(v_ {i}|do(v_{i,r}=0)|$ $$

Here, $r$ represents a randomly chosen index for each sample, which is used to select one of the concepts following Espinosa Zarlenga et al. (). This regularisation can be weighted using an hyperparameter, $\lambda_{3}$. Moreover, for all the experiments where the graph is learnt end-to-end, we initialise the learnable adjacency matrix with the conditional entropy between each pair of values, extracted from the training set.

Hyperparameters All baseline and proposed models were trained for varying epochs across different datasets: 500 for Checkmark, 200 for dSprites, 30 for CelebA and 25 CIFAR10. The optimal epoch for each was determined based on label accuracy on the validation set. A uniform learning rate of 0.01 was applied across all models and datasets. For the CBM and CEM models, both concept and task losses were equally weighted at 1. This weighting scheme was also applied to the loss terms for endogenous copies’ prediction, endogenous variables’ prediction ($\lambda_{1}$), and graph priors ($\lambda_{2}$). The weight assigned to the loss terms in our models to maximize CaCE is 0.05. Additionally, $\gamma$ was treated as a learnable parameter, initialized at 0.1, and $\beta$ was set to 1. All experiments were conducted using five different seeds (1, 2, 3, 4, 5).

Code, licenses and hardware For our experiments, we implement all baselines and methods in Python 3.9 and relied upon open-source libraries such as PyTorch 2.0 (Paszke et al., ) (BSD license), PytorchLightning v2.1.2 (Apache Licence 2.0), Sklearn 1.2 (Pedregosa et al., ) (BSD license). In addition, we used Matplotlib (Hunter, ) 3.7 (BSD license) to produce the plots shown in this paper. Two datasets we used are freely available on the web with licenses: dSprites (Apache 2.0) and CelebA, which is released for non-commercial use research purposes only. We also introduce the Checkmark dataset and we described it in this section.
We will publicly release the code with all the details used to reproduce all the experiments under an MIT license.
All the experiments except the CIFAR10 ones were performed on a device equipped with an M3 Max and 36GB of RAM, without the use of a GPU. The CIFAR10 experimentswere conducted on a workstation equipped with an NVIDIA RTX A6000 GPU, two AMD EPYC 7513 32-Core processors, and 512 GB of RAM.Approximately 60 hours of computational time were utilized from the start of the project, whereas reproducing the experiments detailed here requires only 8 hours.

## Appendix H Additional results

In this section, we include all the experiments shown in Section 4 for the four datasets in more detail and an ablation study on the value of $\lambda_{3}$.

### H.1 Ground-Truth interventions

Figure 12 illustrates the performance comparison among Causal CGM (using both the provided and learnt graphs), CBM, and CEM regarding the effects of interventions. Delta Label Accuracy, which quantifies the change in label accuracy before and after interventions on a growing number of concepts, is calculated solely for the concepts not directly intervened upon. In particular, Causal CGM demonstrates superior performance when interventions involve fewer concepts. This superior performance is attributed to the propagation of intervention effects through all descendant nodes in Causal CGM, unlike CBM and CEM, where the impact is confined to the final task without adjustments to other concepts. The most significant performance gain, approximately 15 percentage points, is observed in CelebA after seven interventions. This effect is particularly pronounced in scenarios with multiple concepts, such as in CIFAR10 and CelebA. In contrast, for simpler tasks with fewer concepts, like Checkmark, the benefits of our approach are less pronounced. The gap observed in CIFAR-10 between CBM and Causal CGM after all interventions is partly due to our evaluation setup: when perturbing the input prior to intervention, we occasionally generate out-of-distribution (OOD) concept embeddings, which only affected the CIFAR10 experiment. This reduces the effectiveness of interventions for embedding-based models, while scalar-based interventions in CBMs remain unaffected. Despite this challenge, our model still achieved approximately 15% higher accuracy overall, with fewer required interventions. Furthermore, when focusing solely on the effects of interventions on the task label, the causal graph utilised by Causal CGM allows us to identify beforehand the specific subset of concepts influencing the task prediction. This pre-identification significantly decreases the required number of interventions to achieve the desired outcome. Figure 13 demonstrates that Causal CGM attains comparable improvements in task performance but after interventions on only three or four concepts, in contrast to the ten and eleven concepts required by CEM and CBM, respectively. The elevated standard error observed in Causal CGM with the learnt graph is attributed to the variability of the graph structure, which significantly influences the outcomes of interventions.

### H.2 Causal structures

Figure 14 illustrates the adjacency matrices corresponding to the DAGs used by Causal CGM for inference in the first three datasets. On the other hand, Tables 4, 5, and 6 present the logic rules derived from the adjacency matrices depicted in the aforementioned figure. Notably, in the Checkmark dataset, both configurations successfully identified the ground truth graph and the correct logic rules. In the case of dSprites, the DAG identified through causal structural learning (GRaSP (Lam et al., )) accurately discovers the causal graph and associated logic rules. Although the end-to-end model accurately identifies the correct relationships between concepts and tasks, it proposes alternative methods for concept prediction. It is important to note that even though the model did not identify the correct causal graph, the model was still capable of performing causal inference with the existing graph. In the CelebA data set, where there is no ground truth for either the graph or logic rules, the findings by GRaSP and the end-to-end model appear plausible and reveal biases inherent in the dataset, such as the strong correlation between makeup use and gender or potential causal links like smiling and a slightly open mouth. This scenario underscores the benefits of employing Causal CGM, particularly in demonstrating how specific concepts are used to predict other concepts and tasks.

### H.3 Ablation study

In Tables 7, 8, and 9, we present the outcomes of varying the hyperparameter $\lambda_{3}$, which weights the loss term designed to enhance the CaCE effect. The results indicate that optimising this loss term contributes to improved CaCE scores, thereby augmenting the efficacy of the interventions. Nonetheless, excessively high values of $\lambda_{3}$ may lead to diminished model performance, as it tends to prioritise boosting the CaCE score at the expense of accurate predictions.

Figure: (a)
Refer to caption: x6.png

Figure: Figure 13: Impact on the task accuracy of ground-truth interventions performed on CelebA concepts. Causal CGM+CD has received in input a causal graph, discovered with a causal structural learning algorithm (GRaSP (Lam et al., )), while Causal CGM learns it end-to-end.
Refer to caption: x10.png

Figure: (a)
Refer to caption: x11.png

**Table 4: Logic rules extracted for the Checkmark dataset from Causal CGM+CD with a given DAG and from Causal CGM with a learnt DAG. A term which refers to an exogenous variable is omitted for simplicity.**
| Method | Checkmark |
| --- | --- |
| Causal CGM | $\begin{aligned} \text{a}&\leftarrow\epsilon_{0}\\<br>\text{b}&\leftarrow\epsilon_{1}\\<br>\text{c}&\leftarrow\sim\text{b}\\<br>\text{d}&\leftarrow\text{a}\wedge\text{c}\\<br>\end{aligned}$ |
| Causal CGM+CD | $\begin{aligned} \text{a}&\leftarrow\epsilon_{0}\\<br>\text{b}&\leftarrow\epsilon_{1}\\<br>\text{c}&\leftarrow\sim\text{b}\\<br>\text{d}&\leftarrow\text{a}\wedge\text{c}\\<br>\end{aligned}$ |

**Table 5: Logic rules extracted for the Dsprites dataset from Causal CGM+CD with a given DAG and from Causal CGM with a learnt DAG. A term which refers to an exogenous variable is omitted for simplicity.**
| Method | Dsprites |
| --- | --- |
| Causal CGM | $\begin{aligned} \text{Shape}&\leftarrow\text{Size}\\<br>\text{Size}&\leftarrow\epsilon_{1}\\<br>\text{PosY}&\leftarrow\epsilon_{2}\\<br>\text{PosX}&\leftarrow\epsilon_{3}\\<br>\text{Color}&\leftarrow\text{Shape}\\<br>\text{Label}&\leftarrow\text{Size}\wedge\text{Color}\\<br>\end{aligned}$ |
| Causal CGM+CD | $\begin{aligned} \text{Shape}&\leftarrow\epsilon_{0}\\<br>\text{Size}&\leftarrow\text{Shape}\wedge\text{PosX}\\<br>\text{PosY}&\leftarrow\epsilon_{2}\\<br>\text{PosX}&\leftarrow\epsilon_{3}\\<br>\text{Color}&\leftarrow\text{Shape}\wedge\text{PosY}\\<br>\text{Label}&\leftarrow\text{Size}\wedge\text{Color}\\<br>\end{aligned}$ |

**Table 6: Logic rules extracted for the Celeba dataset from Causal CGM+CD with a given DAG and from Causal CGM with a learnt DAG. A term which refers to an exogenous variable is omitted for simplicity.**
| Method | Celeba |
| --- | --- |
| Causal CGM | $\begin{aligned} \text{Smiling (S)}&\leftarrow\epsilon_{0}\\<br>\text{Attractive (A)}&\leftarrow\text{Heavy_Make}\\<br>\text{Mouth_Slig (MS)}&\leftarrow\text{False}\\<br>\text{High_Cheek (HC)}&\leftarrow\epsilon_{3}\\<br>\text{Wearing_Li (WL)}&\leftarrow\text{Attractive}\\<br>\text{Heavy_Make (HM)}&\leftarrow\text{Smiling}\wedge\text{High_Cheek}\\<br>\text{Male}&\leftarrow\sim\text{Wearing_Li}\wedge\sim\text{Heavy_Make}\\<br>\text{Wavy_Hair (WH)}&\leftarrow(\text{HC}\wedge\text{WL}\wedge\sim\text{Male<br>})\vee(\text{HC}\wedge\text{WL}\wedge\sim\text{OF})\\<br>\text{Big_Lips (BL)}&\leftarrow\text{Smiling}\wedge\text{High_Cheek}\wedge<br>\sim\text{Male}\wedge\sim\text{Oval_Face}\\<br>\text{Oval_Face (OF)}&\leftarrow\text{False}\\<br>\text{Makeup (M)}&\leftarrow\text{False}\\<br>\text{Fem_Model}&\leftarrow(\text{M}\wedge\sim\text{S})\vee(\text{M}\wedge<br>\sim\text{Male})\vee(\text{M}\wedge\text{HC}\wedge\sim\text{WH})\vee(\text{WL}<br>\wedge\text{WH}\wedge\text{BL}\wedge\sim\text{S}\wedge\sim\text{HC})\\<br>\end{aligned}$ |
| Causal CGM+CD | $\begin{aligned} \text{Smiling}&\leftarrow\text{Mouth_Slig}\\<br>\text{Attractive}&\leftarrow\text{Wearing_Li}\\<br>\text{Mouth_Slig}&\leftarrow\epsilon_{2}\\<br>\text{High_Cheek}&\leftarrow\text{Smiling}\\<br>\text{Wearing_Li}&\leftarrow\epsilon_{4}\\<br>\text{Heavy_Make}&\leftarrow(\text{Attractive}\wedge\text{Wearing_Li})\vee(<br>\text{Wearing_Li}\wedge\text{Oval_Face})\\<br>\text{Male}&\leftarrow\sim\text{Wearing_Li}\wedge\sim\text{Heavy_Make}\\<br>\text{Wavy_Hair}&\leftarrow\text{Makeup}\vee(\text{Attractive}\wedge\text{<br>Wearing_Li}\wedge\sim\text{Male})\\<br>\text{Big_Lips}&\leftarrow\text{Makeup}\\<br>\text{Oval_Face}&\leftarrow\text{Smiling}\wedge\text{Attractive}\wedge\text{<br>Wearing_Li}\wedge\sim\text{Big_Lips}\\<br>\text{Makeup}&\leftarrow\text{False}\\<br>\text{Fem_Model}&\leftarrow\text{Makeup}\\<br>\end{aligned}$ |

**Table 7: Ablation study regarding $\lambda_{3}$ on the Checkmark dataset.**
|  | $\lambda_{3}$ | Label Accuracy | Average CaCE | min CaCE | max CaCE | CaCE | CaCE block |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Black Box |  | $90.15\pm 0.14$ |  |  |  |  |  |
| CBM |  | $90.34\pm 0.55$ | $4.09\pm 0.80$ | $0.00\pm 0.00$ | $9.45\pm 1.86$ | $28.36\pm 5.58$ | $27.79\pm 5.64$ |
| CEM |  | $89.09\pm 1.98$ | $1.75\pm 1.66$ | $0.00\pm 0.00$ | $4.45\pm 3.78$ | $11.59\pm 12.76$ | $11.89\pm 12.95$ |
| Causal CGM | 0.05 | $88.24\pm 1.30$ | $14.47\pm 2.86$ | $0.00\pm 0.00$ | $44.37\pm 5.58$ | $16.15\pm 11.25$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.05 | $89.43\pm 0.93$ | $13.15\pm 2.31$ | $0.00\pm 0.00$ | $39.53\pm 3.62$ | $20.99\pm 8.19$ | $0.00\pm 0.00$ |
| Causal CGM | 0.2 | $85.88\pm 3.31$ | $11.64\pm 3.21$ | $0.00\pm 0.00$ | $37.55\pm 8.77$ | $16.32\pm 15.36$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.2 | $85.09\pm 2.78$ | $16.39\pm 5.00$ | $0.00\pm 0.00$ | $39.69\pm 10.97$ | $31.15\pm 19.48$ | $0.00\pm 0.00$ |
| Causal CGM | 0 | $87.86\pm 1.66$ | $6.98\pm 2.95$ | $0.00\pm 0.00$ | $18.16\pm 8.29$ | $7.39\pm 4.10$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0 | $87.04\pm 2.72$ | $12.16\pm 2.92$ | $0.00\pm 0.00$ | $33.15\pm 9.97$ | $14.52\pm 14.29$ | $0.00\pm 0.00$ |

**Table 8: Ablation study regarding $\lambda_{3}$ on the dSprites dataset.**
|  | $\lambda_{3}$ | Label Accuracy | Average CaCE | min CaCE | max CaCE | CaCE | CaCE block |
| --- | --- | --- | --- | --- | --- | --- | --- |
| BlackBox |  | $99.53\pm 0.05$ |  |  |  |  |  |
| CBM |  | $99.55\pm 0.07$ | $2.77\pm 0.30$ | $0.00\pm 0.00$ | $8.51\pm 1.03$ | $0.64\pm 0.70$ | $0.64\pm 0.70$ |
| CEM |  | $99.48\pm 0.07$ | $0.46\pm 0.29$ | $0.00\pm 0.00$ | $2.45\pm 1.67$ | $2.43\pm 4.62$ | $2.43\pm 4.62$ |
| CausalCEM | 0.05 | $99.44\pm 0.11$ | $17.53\pm 3.26$ | $0.00\pm 0.00$ | $44.47\pm 10.16$ | $34.13\pm 19.46$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.05 | $99.40\pm 0.15$ | $12.85\pm 0.59$ | $0.00\pm 0.00$ | $27.72\pm 3.66$ | $28.95\pm 13.50$ | $0.00\pm 0.00$ |
| Causal CGM | 0.2 | $98.80\pm 1.24$ | $14.13\pm 3.39$ | $0.00\pm 0.00$ | $37.59\pm 14.20$ | $17.52\pm 13.41$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.2 | $99.30\pm 0.13$ | $12.90\pm 0.51$ | $0.00\pm 0.00$ | $34.01\pm 6.99$ | $16.84\pm 6.79$ | $0.00\pm 0.00$ |
| Causal CGM | 0 | $99.58\pm 0.12$ | $6.89\pm 1.55$ | $0.00\pm 0.00$ | $18.51\pm 5.14$ | $12.11\pm 13.01$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0 | $99.51\pm 0.05$ | $5.67\pm 1.21$ | $0.00\pm 0.00$ | $12.41\pm 1.82$ | $15.15\pm 4.10$ | $0.00\pm 0.00$ |

**Table 9: Ablation study regarding $\lambda_{3}$ on the CelebA dataset.**
|  | $\lambda_{3}$ | Label Accuracy | Average CaCE | min CaCE | max CaCE | CaCE | CaCE block |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Black Box |  | $90.15\pm 1.30$ |  |  |  |  |  |
| CBM |  | $79.00\pm 0.18$ | $0.54\pm 0.03$ | $0.00\pm 0.00$ | $1.67\pm 0.15$ | $5.58\pm 2.36$ | $5.46\pm 2.13$ |
| CEM |  | $79.17\pm 0.26$ | $0.27\pm 0.12$ | $0.00\pm 0.00$ | $1.07\pm 0.56$ | $1.00\pm 0.45$ | $1.06\pm 0.50$ |
| Causal CGM | 0.05 | $78.23\pm 0.45$ | $2.17\pm 1.44$ | $0.00\pm 0.00$ | $8.18\pm 4.38$ | $0.04\pm 0.07$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.05 | $78.42\pm 0.42$ | $5.48\pm 0.34$ | $0.00\pm 0.00$ | $24.17\pm 1.32$ | $1.24\pm 0.62$ | $0.00\pm 0.00$ |
| Causal CGM | 0.2 | $77.49\pm 0.37$ | $1.70\pm 0.98$ | $0.00\pm 0.00$ | $8.95\pm 4.86$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0.2 | $78.08\pm 0.39$ | $6.15\pm 0.31$ | $0.00\pm 0.00$ | $29.57\pm 1.03$ | $0.82\pm 0.46$ | $0.00\pm 0.00$ |
| Causal CGM | 0 | $77.42\pm 1.09$ | $2.85\pm 0.44$ | $0.00\pm 0.00$ | $13.06\pm 2.65$ | $0.00\pm 0.00$ | $0.00\pm 0.00$ |
| Causal CGM + CE | 0 | $78.31\pm 0.36$ | $4.64\pm 0.13$ | $0.00\pm 0.00$ | $18.31\pm 2.61$ | $1.25\pm 0.68$ | $0.00\pm 0.00$ |

### H.4 Concept incompleteness and generalisation with a different number of concepts

Concept incompleteness does not have a strong impact on Causal CGM. As shown in the experiments with CLIP on CIFAR, we can automatically extract new concepts if the provided ones are insufficient. However, to quantify the impact of incompleteness on classification performances, we report an ablation study on a CBM benchmark (Marcinkevičs et al., ) specifically designed to assess concept incompleteness. This benchmark is a synthetic tabular dataset where we can control to what degree concepts explain the variance in the downstream task (see (Marcinkevičs et al., ) for details). In Figure 15 we observe that, as the concept incompleteness $i$ increases from $i=0$, where all relevant concepts are available, to $i=0.9$, where only $10\%$ of relevant concepts are available, the performance of CBMs drop, while that of black boxes, CEMs, and Causal CGMs do not thanks to the use of concept embeddings.

Figure: Figure 15: Label accuracy for different numbers of concepts with increasing concept incompleteness.
Refer to caption: x17.png

### H.5 Scalability and computational complexity

Causal CGMs pays a small cost (at most quadratic in the number of class labels) to address causal transparency. In terms of the number of parameters, the endogenous predictor scales linearly with the number of concepts, as in standard CBMs. Regarding time complexity, the endogenous predictor scales quadratically (both in train and inference) with the number of concepts. This is because, during training, we model all concept dependencies, and, during testing, inference is performed on a DAG (a triangular adjacency matrix in the worst case). This is a small cost in exchange for interpretability (and not unique to our work).
To better communicate our method’s scalability, we report the number of parameters and training and inference times as we increase the number of class labels on the same CBM benchmark used to assess incompleteness (Marcinkevičs et al., ). In Figure 16 we observe that the impact on the number of parameters is negligible as it is mostly due to the encoder. Here the encoder is a relatively small network, but it could be a billion-parameter in a Vision-lLanguage Model (VLM). Training and inference time are slightly higher than baselines, but if training includes a large encoder, say a VLM, the relative impact of the endogenous predictor would be significantly reduced (we did not have time to train a VLM from scratch).

Figure: Figure 16: Number of parameters, train and inference time.
Refer to caption: x18.png

### H.6 k 𝑘 k italic_k -unfolding ablation

In Causal CBM, $1$-step unfolding of the causal graph during training is equivalent to any k-step ($k>1$) unfolding as any step corresponds (theoretically and empirically) to repeating the same query on the same conditional probability table (CPT). Causal CGM learns the conditional probabilities of the variables given its parents (i.e. the CPT of the Bayesian network). Under full observability, conditional queries of the form $p(v|\text{ all but }v)$ are just inspections of the CPT and do not require any reasoning on the cyclical model (which alleviates us from setting a complex $k$-step semantics to unroll the cyclical model). The key observation is that the cyclical model served the purpose of templating multiple (any) acyclical graphs jointly without making assumptions on concept relations. During learning, the probabilities of the CPT (parameterized by NNs) are also subject to an acyclicity objective. After learning, the resulting graph is acyclic thus inference is performed via standard message passing along the directed acyclic graph. To show this experimentally, we report the results on our CBM benchmark using a different number of unfolding steps and considering different levels of concept completeness in Table 10. As expected, the results show that increasing the number of steps does not impact Causal CGM’s classification performances.

**Table 10: Causal CGM’s accuracy with $k=\{1,2,3\}$-unfolding and higher concept incompleteness.**
|  | Incompleteness ratio |  |  |  |
| --- | --- | --- | --- | --- |
|  | 0.0 | 0.1 | 0.5 | 0.9 |
| Unfolding K=1 | $85.00\pm 0.14$ | $83.70\pm 1.10$ | $84.49\pm 0.57$ | $85.51\pm 0.16$ |
| Unfolding K=2 | $83.78\pm 0.81$ | $83.72\pm 0.85$ | $85.26\pm 0.20$ | $84.96\pm 0.14$ |
| Unfolding K=3 | $85.00\pm 0.34$ | $83.10\pm 0.98$ | $84.30\pm 0.47$ | $85.32\pm 0.05$ |

## Appendix I Answering counterfactual queries with Causal Concept Graph Models

In our experiments, we use Causal CGMs to compute Pearl’s Probability of Sufficiency and Necessity (PNS), which is a prototypical example of a counterfactual query (Tian & Pearl, ). In general, the computation of counterfactual queries requires three key ingredients  (Pearl, ): (i) a causal graph, (ii) structural rules predicting an endogenous variable given its parents’ values, and (iii) information about the exogenous variables distribution, which for us is encoded in the embeddings. In causal CGM, all these three ingredients are available, allowing the computation of both identifiable and unidentifiable causal queries. For the latter, as is the case of PNS, the result is given in terms of an interval exactly as it happens in causal inference with structural causal models (with probabilistic quantification over exogenous variables). Figure 17 illustrates the upper boundaries of the PNS tables obtained using Causal CGM across the four datasets. Additionally, Table 11 provides an example of counterfactuals generated on the CelebA dataset, where we first intervene on the lipstick concept, followed by an intervention on the makeup concept.

Figure: (a)
Refer to caption: x19.png

**Table 11: Examples of counterfactuals generated on CelebA obtained via do-interventions (intervened variables are marked in red, changed variables are underlined).**
|  | Endogenous variables’ activations |
| --- | --- |
| Initial Predicted State | Smiling, Not Attractive, Mouth Slig, High Cheek, Not Wearing Li, Not Heavy Make, |
| Not Wavy Hair, Not Big Lips, Not Oval Face, Not Makeup, Not Fem Model |  |
| What if I would wear lipstick? | Smiling, Attractive, Mouth Slig, High Cheek, Wearing Li, Heavy Make, |
| Not Wavy Hair, Not Big Lips, Not Oval Face, Not Makeup, Not Fem Model |  |
| What if I would wear lipstick and also makeup? | Smiling, Attractive, Mouth Slig, High Cheek, Wearing Li, Heavy Make, |
| Not Wavy Hair, Big Lips, Not Oval Face, Makeup, Fem Model |  |

## Appendix J Causal Concept Graph Models address “causal opacity”, not “causal discovery”

Our work is based on a fundamental yet subtle distinction between causal opacity/transparency and causal discovery that is poorly understood in the ML literature. Although related, the two problems are different in nature: Causal opacity/transparency refers to the problem of understanding and manipulating the causal structure of a model’s decision-making process, and it thus is a specific instance of what is usually referred to as the opacity or black-box problem. Causal discovery, on the other hand, is not concerned with how the model is but rather with how the world (Boge, ), i.e., it concerns determining the causal structure of the generating mechanism beyond data. This distinction applies also to the notion of causal understanding. Indeed, we can distinguish between two forms of causal understanding depending on whether the object of one’s understanding is the model’s decision-making process or the world/data-generating mechanism. Notice that causal opacity (and the related form of causal understanding) neither implies nor requires causal discovery. The causal structure of the model’s decision-making process does not necessarily mirror the causal structure of the world. Of course, one could argue that a good model should always resemble the world’s causal structure, but this is not always true or even wanted (e.g., if the world is biased, we may want to construct an unbiased model that deviates from the data-generating process of its training data).

## Appendix K Relations with causal representation learning

If we assume a strict definition of causal representation learning (CLR) as any method that aims at learning high-level representations of the causal-generating mechanisms of data (Schölkopf et al., ), then Causal CGMs are not an example of CRL. Causal CGMs aims not to discover causal mechanisms beyond data but to build models whose decision-making processes follow a causal structure that users can visualize, interpret, and manipulate. This requires the model to use high-level variables (concepts) that are meaningful for a user (as opposed to, say, an image’s pixels) and to clearly display the causal relationships between these variables (e.g., via a casual graph).

Regardless, there are similarities between Causal CGMs and CRL. Both rely on Pearl’s formalism of SCMs and use high-level interpretable variables and causal graphs to model causal dependencies. In the case of Causal CGMs, however, the learned causal structure may not necessarily mirror the causal structure of the data-generating mechanism, as it is required for CRL models.

## Appendix L Comparing different decision making processes with Causal Concept Graph Models

Causal CGMs are not identifiable in the sense that we can obtain different causal graphs by training the model over the same dataset. This is similar to obtaining different parameter values and embeddings when training standard deep-learning black boxes on the same data. Similarly, the training of causal CGMs can result in different causal decision-making structures for the same task. The difference is that in the case of the black box, we cannot compare the different models because they are (causally) opaque. In the case of Causal CGMs, we can inspect the multiple models obtained and verify them against desirable properties (e.g., fairness), hence adopting the best one.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: - > Concept Graph Models duplicate the layer of interpretable variables $V$ by introducing an additional layer of identical copies $V^{′}$ . Using this additional layer, Causal CGMs make predictions of each $v_{i} \in V$ using as possible inputs all $v_{j}' \in V^{′}$ for all $j \neq i$

How do you guarantee the causal graph will be acyclic?
soundness: 3
strengths: -  The paper is well written.
-  The idea of introducing causal transparency in concept-based models is interesting.
-  The experiments are sound.
summary: This paper presents Causal Concept Graph Models (Causal CGMs), a concept-based architecture whose decision-making process is causally transparent by design. It avoids the unrealistic assumption that concepts must be causally independent and direct causes of the class prediction. The paper aims to design Deep Learning  models where each prediction can be traced back to a chain of
semantically meaningful causes.
weaknesses: -  > *Causal opacity refers to the difficulty in understanding the mechanisms behind a model’s decision-making process.* 

  The paper frequently references the concept of causal opacity, but it is too vaguely defined to be fully understood.

- How the model performs abduction to answer counterfactual queries is unclear.

- Minor: The description of Figure 3 is inconsistent with the picture. 

- It seems the proposed model sacrifices accuracy to improve interpretability.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: My questions mainly concern the weaknesses mentioned above. I would kindly like to ask the authors the following questions:

1) How does method guarantee identifiability and non-confoundedness of concept predictions. Could the authors elaborate on the implications of theorem 3.2 towards this question and/or highlight possible limitations that might arise?
2) Regarding point 3: Could the further explain how distributions of Theorem 3.2 can be equal, given, that $v'_j$ are inferred from exogenous variables only?
3) How, does the presented approach compare to other causal deep-learning methods? Could the authors present further insights or explain, why other methods might not be comparable?
soundness: 1
strengths: The authors extend prior approaches by dropping the restriction of causally independent concepts and trying to approximate the underlying true graphical causal relation. This is an important extension, as most real-world settings, feature complex interactions between concepts and allows to inspect the consequences of interventions on the modeled process.

By observing predictive power among variables, the authors are able to (partially) recover the individual parents of the variables. If trained correctly, CCGM are therefore deep models, which are interpretable by design and possibly allow for human-in-the-loop adjustments.

The presented results improve over prior CBM and CEM, while lacking only slightly behind 'causally opaque' black-box models. The authors show good predictive performance under interventions, and are able to extract some underlying logic rules from the learned models.
summary: In their work, the authors construct causal concept bottleneck models (CCBMs), which are interpretable models that aim to reveal the underlying decision process of deep learning models. The models are built with the assumptions that all relevant human-aligned concepts are known a priori and that these concepts are direct causes of the class prediction. The authors propose measure the predictive power of concepts towards each other other to figure out variable parents. The authors leverage the assumption, that distribution predictions between features predicted by noise, and features predicted from the prior noise-predicted values are invariant. The recovered structure is then used to extract logic rules and trace-back decision.

The approach is evaluated on smaller image datasets, including CelebA and CIFAR 10, showing performance gains over other non-causal methods.


[Disclaimer: This is an emergency review]
weaknesses: The weaknesses concern extend to which the presented method 'is actually causal'. While existing works in the field of causality are mainly concerned with providing identifiability results, the presented method mainly leverages predictive power to identify causal relations between variables, which might not coincide with learning the correct causal structure and lead the model to learn spurious associations.

1) Measuring the predictive performance of features towards each other does guarantee to recovery the underlying causal structure. Generally, additional assumptions such as interventions or sufficient variability need to be assumed. In that regard, theorem 3.2 does not state any of these assumptions and is provided without proof.

2) In that regard the paper lacks a comparison and detailed discussion to existing methods in the field of causal representation learning [1], such as differences to other existing deep causal approaches ([2-4]), and -with regard to the prior point- guarantees on the identifiability of (latent) concepts [5-7].
3) Variables $v'_i$ are predicted just from their exogenous noise variables $u_i$. Assuming that, variables are indeed embedded into a causal structure, and assuming that at least for some $v_i$ the parent set is non-empty, it can not be that $p(v'_i\mid pa_{V'}(v_i),u_i) = p(v_i \mid u_i)$.
4) The experimental section compares to only non-causal methods (CEM and CBM) on their predictive performance (accuracy). Given the above discussion, it is unclear whether the models have actually learned the correspond process. No analysis or discussion towards identifiability and possible avoidance of confounding is presented. As a result a rather strong drop-off in performance between CCGM+CD (recovering the graph from ground-truth labels) and CCGM is observed. In particular, when the causal graph is recovered from data, the presented causal CGM+CD approach might by very similar to Neural Causal Models (NCM) [4].



**Minor**

* Table 1 is hard to read. I suggest breaking "semantic transparency" and "causal transparency" into two lines to make the texts bigger.


[1] Schölkopf, Bernhard, et al. "Toward causal representation learning." *Proceedings of the IEEE* 109.5 (2021): 612-634.  
[2] Kladny, Klaus-Rudolf, et al. "Deep backtracking counterfactuals for causally compliant explanations." *arXiv preprint arXiv:2310.07665* (2023).  
[3] Lippe, Phillip, et al. "Biscuit: Causal representation learning from binary interactions." *Uncertainty in Artificial Intelligence*. PMLR, 2023.  
[4] Xia, Kevin, Yushu Pan, and Elias Bareinboim. "Neural causal models for counterfactual identification and estimation." *arXiv preprint arXiv:2210.00035* (2022).  
[5] Brehmer, Johann, et al. "Weakly supervised causal representation learning." *Advances in Neural Information Processing Systems* 35 (2022): 38319-38331.  
[6] Wendong, Liang, et al. "Causal component analysis." *Advances in Neural Information Processing Systems* 36 (2024).  
[7] von Kügelgen, Julius, et al. "Nonparametric identifiability of causal representations from unknown interventions." *Advances in Neural Information Processing Systems* 36 (2024).

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. I do not understand the experiment described in the last paragraph of the experimental section (lines 463-476).  Could the authors please provide an example? How are variables i and j selected for each dataset? Are the results in Table 3. averaged over all possible pairs? How do the authors decide which variables to intervene on (i.e. what structure is used as Ground Truth)? Why is 0 a desirable outcome?  - the authors write that the aim of this procedure is debiasing, to my understanding 0 RCCE means no change.
2. Table 3 has two columns titled “CELEBA”
soundness: 3
strengths: 1. The paper is cleanly written. 
2. The approach is well-placed in the literature. 
3. The experimental section is extensive.
4. The method description is detailed and well-structured.
summary: The authors propose a new class of interpretable models whose decision-making process is causally transparent by design. The method represents relations between high-level concepts, associated with each data point, as a DAG allowing for interventional and counterfactual analysis of predictions and human improvements during the inference process.

The authors demonstrate that their approach matches the generalization performance of SoTA models. Additional experimental results showcase how explicit causal representation can be efficiently used to increase interpretability and improve performance with human interventions.
weaknesses: 1. One bit of experimental evaluation is unclear to me (see Questions)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: Refer to Weaknesses
soundness: 2
strengths: A serious shortcoming of CBMs is that they assume that each human-specified concept independently influences the model's predictions. The authors address this issue by proposing a Causal Graph Concept model that captures the intricacies of the dependencies between the concepts & the target variable and among them.

As the proposed model is causally transparent by design, it allows a human to understand the inner workings of the model by enabling them to ask questions, such as:
- What is the relationship between feature x & label y?
- What happens to the prediction if I set feature x to k?
- What if feature x was k' instead of k?
The proposed model's performance matches or is within the ballpark of the opaque state-of-the-art models on the chosen set of benchmarks.

Theoretically, it also allows humans to intervene and correct erroneous intermediate reasoning steps.
summary: Over the past few years, there has been a lot of interest in designing models that are interpretable by construction. Concept Bottleneck Models are one such approach; they first predict an intermediate set of human-specified concepts and then use the concepts to predict the target. These intermediate concepts allow humans to understand the dynamics of model predictions through interventions. One key drawback of such an approach is that they assume each concept independently influences the predictions, while in practice, there is always a complex relationship between the concepts and how they influence the target variable. The paper addresses the above issue by introducing the Causal Graph Concept Graph Model, which is transparent by design and captures the complex dynamics of a model's decision processes.
weaknesses: - The paper isn't written well, and some sections are hard to follow.
- One of the key weaknesses lies in the experiments.
     - Why only choose simpler benchmarks, like CelebA & CIFAR10, and benchmarks with semantically richer tasks like AwA2, CUB, etc.?
     - Understanding what the authors mean by joint training is a bit hard.
          - Are the concept encoders also trained along with the process of learning CGMs? 
          - What formulation of CBMs do the authors build on? (independent bottleneck, joint bottleneck, or sequential bottleneck?)
          - What is the architecture of the bottleneck layer? Does it influence the quality of CGMs learnt?
     - One contribution is that CGMs allow human-in-the-loop corrections of erroneous intermediate reasoning steps. I don't see any experiments to support this. Did you evaluate whether the learnt causal graphs are sound and valuable? Did you do a user study to assess their utility to users? 
     - Do you have any intuition about why there is a drop in Figures 4 and 5 for a certain number of intervened concepts?
     - Column 4 is Table 3 is CIFAR10?
     - Line 345 says CIFAR10 is an animal classification dataset. Did you use the entire dataset or a subset for evaluation?
- Another issue with CBMs is that they assume that human-specified concepts are readily available, which rarely happens in practice. Much work has been done in the literature to relax this assumption. I wonder why the authors build on the vanilla version of CBMs.

---

confidence: 2
contribution: 1
initial_rating: 8
presentation: 3
questions: line 88: (Minor comment) The authors write the causal CGM is related to XAI. Maybe it is more about interpretability than it is about explainability? To me, XAI methods aim at explaining black-box models. However, causal CGM is rather about constructing the model such that it is interpretable by construction.

line 91: As already explained in weakness 1, I am not convinced that causal CGM is a new architecture. 

line 105: I would challenge that structural causal models are *the* standard framework for causal modeling. Rubin causal models are also quite common. I would suggest rephrasing this. For example: *"Structural causal models are a widely used framework for causal modeling, alongside other approaches like Rubin causal models."*

line 111: *"which determine the values of each endogenous variable $v_i \in V$ by computing the conditional probability ..."* This sentence sounds contradictory to me. According to the SCM definition that I am familiar with, all endogenous variables can be expressed as a deterministic function of all exogenous variables (reduced form expression). Thus, this conditional probability should be just a Dirac measure? I would appreciate if the authors could clarify this.

line 115: I believe this should read *hard interventions*. There also exist other types of interventions where variables are not fixed to a value.

line 121: I am not too familiar with the literature on *concept-based models* (CBM). However, I do not understand how the term *concept* differs from *high-level feature* and how *concept-based modeling* is distinct from *representation learning* (see weakness 2).

line 199: (Minor comment) Doesn't theorem 3.2 follow from d-separation? I could imagine that the proof could be greatly simplified with such a graphical argument.

line 245: How does the training differ from the method proposed by [1] (see weakness 1)? It would be great if the authors could highlight this in detail.

line 257: *maximise the log-likelihood of v, v′* How can the likelihood of unobserved variables be maximized? I suppose you maximize a variational lower bound of $\text{log} \\; p(x)$?

line 481: *Causal CGMs focus on high-level human interpretable concepts* As mentioned in weakness 3, this seems to be at odds with the original motivation of the work.
soundness: 4
strengths: 1. The authors present an interesting perspective on causal representation learning, where the learned abstract variables and mechanisms are used to obtain insights into a trained model (interpretability) rather than aiming at generalizing to unseen domains. The latter is the main motivation of the related work that I am familiar with (e.g., [1, 2, 3]).

2. The manuscript is particularly well-written and the plots are great, too. The authors did an excellent job at presenting their work well.

3. There are plenty of experiments that (somewhat surprisingly) show that causal CGMs perform similarly to black box models that do not enforce a causal bottleneck.

[1] von Kügelgen, Julius, et al. "Nonparametric identifiability of causal representations from unknown interventions." Advances in Neural Information Processing Systems 36 (2024).

[2] Lippe, Phillip, et al. "Biscuit: Causal representation learning from binary interactions." Uncertainty in Artificial Intelligence. PMLR, 2023.

[3] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.
summary: The authors introduce a generalization of *Concept Bottleneck Models* (CBM), called *Causal Concept Graph Models* (causal CGM). In contrast to existing CBMs, causal CGM allows for modeling causal relationships between latent variables. In contrast to causal representation learning, causal CGM aims at addressing causal opacity (interpretability) of deep learning models, rather than identifying specific variables and/or mechanisms. The claimed properties of Causal CGM are experimentally assessed on various data sets and compared with a black-box deep learning baseline, a non-causal CBM baseline and Concept Embedding Models.
weaknesses: 1. In terms of the architecture and training of the proposed method, I am concerned about novelty. The method may seem novel within the CBM literature, where only independent latent variables have been considered so far. However, the proposed method seems to be (almost) identical to [1], which is branded as *causal variational autoencoder* rather than *concept bottleneck model*. Apart from the name and problem motivation, I cannot identify any differences. It would be great if the authors could highlight the differences, if they exist (I did not see them).

2. While the authors start with an interesting perspective on learning latent causal variables (see strength 1), it is unfortunate that this idea is not taken very far. Specifically, I would be interested in questions such as how such a causal graph could be efficiently inferred from any black-box model post training (just an example of the kind of questions that came up to me). Instead, causal CGM seems to be an existing causal representation learning method (see weakness 1), but without identifiability guarantee. I would advise the authors to develop the paper more closely along this original idea (strength 1), as I believe there is a lot of unused potential.

3. The *human-in-the-loop corrections* aspect seems contradictory to the motivation of the work: Causal CGM is introduced from the point of interpretability rather than identifiability of variables and mechanisms. Human-in-the-loop corrections indicate that specific variables (known the human) ought to be identified, which lies at odds with the causal opacity motivation. Therefore, I would suggest moving this aspect entirely to the appendix.

[1] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004.08697, 2020.

Meta-review: The paper shows how to construct causal concept bottleneck models (CCBMs), i.e., interpretable models that aim to reveal the underlying decision process of deep learning models.  Overall, the reviewers lean towards acceptance, and I agree. While the datasets considered are rather small, the results are interesting as they pave the way to deep models where each prediction can be traced back to a chain of semantically meaningful causes. The authors, however, should spend more time on explaining key terms of the approach such as causal opacity,

Decision: Accept (Poster)